[
https://issues.apache.org/jira/browse/AIRFLOW-4171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiaxun Song updated AIRFLOW-4171:
---------------------------------
Description:
While using KubernetesPodOperator, I found that even though get_logs was set to
be True, only a few lines of log from the task Pod is captured. Following is
the potential explanation.
In the following pod_launcher module class PodLauncher
[https://github.com/apache/airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py]
function run_pod returns self._monitor_pod(pod, get_logs). And inside function
_monitor_pod, it only captures the log when the pod starts to run. In other
words, when KubernetesPodOperator is using pod_launcher to spin up a Pod for a
task, it can only capture the log at the very beginning of the Pod. So we
cannot see the entire log from the Pod when the Pod completed.
Also, every time I got this message from the task log: '{{failed to watch file
"/var/log/pods/afaa3c1c-5126-11e9-9808-0a91764318bc/base/0.log": no space left
on device}}'. Maybe the problem is from K8S host VM (I am using AKS).
Please correct me if I am wrong, but I tried everything to capture the entire
log from the task Pod. The only explanation I can find is the above.
was:
While using KubernetesPodOperator, I found that even though get_logs was set to
be True, only a few lines of log from the task Pod is captured. Following is
the potential explanation.
In the following pod_launcher module class PodLauncher
[https://github.com/apache/airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py]
function run_pod returns self._monitor_pod(pod, get_logs). And inside function
_monitor_pod, it only captures the log when the pod starts to run. In other
words, when KubernetesPodOperator is using pod_launcher to spin up a Pod for a
task, it can only capture the log at the very beginning of the Pod. So we
cannot see the entire log from the Pod when the Pod completed.
Please correct me if I am wrong, but I tried everything to capture the entire
log from the task Pod. The only explanation I can find is the above.
> KubernetesPodOperator / pod_launcher cannot capture entire log after the task
> complete
> --------------------------------------------------------------------------------------
>
> Key: AIRFLOW-4171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4171
> Project: Apache Airflow
> Issue Type: Bug
> Affects Versions: 1.10.2
> Reporter: Jiaxun Song
> Priority: Major
> Fix For: 1.10.2
>
>
> While using KubernetesPodOperator, I found that even though get_logs was set
> to be True, only a few lines of log from the task Pod is captured. Following
> is the potential explanation.
> In the following pod_launcher module class PodLauncher
> [https://github.com/apache/airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py]
> function run_pod returns self._monitor_pod(pod, get_logs). And inside
> function _monitor_pod, it only captures the log when the pod starts to run.
> In other words, when KubernetesPodOperator is using pod_launcher to spin up a
> Pod for a task, it can only capture the log at the very beginning of the Pod.
> So we cannot see the entire log from the Pod when the Pod completed.
> Also, every time I got this message from the task log: '{{failed to watch
> file "/var/log/pods/afaa3c1c-5126-11e9-9808-0a91764318bc/base/0.log": no
> space left on device}}'. Maybe the problem is from K8S host VM (I am using
> AKS).
> Please correct me if I am wrong, but I tried everything to capture the entire
> log from the task Pod. The only explanation I can find is the above.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
