Jiaxun Song created AIRFLOW-4171:
------------------------------------
Summary: KubernetesPodOperator / pod_launcher cannot capture
entire log after the task complete
Key: AIRFLOW-4171
URL: https://issues.apache.org/jira/browse/AIRFLOW-4171
Project: Apache Airflow
Issue Type: Bug
Affects Versions: 1.10.2
Reporter: Jiaxun Song
Fix For: 1.10.2
While using KubernetesPodOperator, I found that even though get_logs was set to
be True, only a few lines of log from the task Pod is captured. Following is
the potential explanation.
In the following pod_launcher module
[https://github.com/apache/airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py]
function run_pod returns self._monitor_pod(pod, get_logs). And inside function
_monitor_pod, it only captures the log when the pod starts to run. In other
words, when KubernetesPodOperator is using pod_launcher to spin up a Pod for a
task, it can only capture the log at the very beginning of the Pod. So we
cannot see the entire log from the Pod when the Pod completed.
Please correct me if I am wrong, but I tried everything to capture the entire
log from the task Pod. The only explanation I can find is the above.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
