[ https://issues.apache.org/jira/browse/AIRFLOW-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Miller updated AIRFLOW-2970: ----------------------------------- Attachment: image-2019-09-17-12-19-45-207.png > Kubernetes logging is broken > ---------------------------- > > Key: AIRFLOW-2970 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2970 > Project: Apache Airflow > Issue Type: Bug > Components: executors > Reporter: Jon Davies > Assignee: Daniel Imberman > Priority: Major > Attachments: image-2019-09-17-12-19-45-207.png > > > I'm using Airflow with the Kubernetes executor and pod operator. And my DAGs > are configured to do get_log=True and all my DAGs are set to log to stdout > and I can see all the logs in kubectl logs. > I can see that the scheduler logs things to: > $AIRFLOW_HOME/logs/scheduler/2018-08-28/* > However, this just consists of: > {code:java} > [2018-08-28 13:03:27,695] {jobs.py:385} INFO - Started process (PID=16994) to > work on /home/airflow/dags/dag.py > [2018-08-28 13:03:27,697] {jobs.py:1782} INFO - Processing file > /home/airflow/dags/dag.py for tasks to queue > [2018-08-28 13:03:27,697] {logging_mixin.py:95} INFO - [2018-08-28 > 13:03:27,697] {models.py:258} INFO - Filling up the DagBag from > /home/airflow/dags/dag.py > {code} > If I quickly exec into the executor the scheduler spins up, I can see that > things are properly logged to: > {code:java} > /home/airflow/logs/dag$ tail -f > dag-downloader/2018-08-28T13\:05\:07.704072+00\:00/1.log > [2018-08-28 13:05:24,399] {logging_mixin.py:95} INFO - [2018-08-28 > 13:05:24,399] {pod_launcher.py:112} INFO - Event: dag-downloader-015ca48c had > an event of type Pending > ... > [2018-08-28 13:05:37,193] {logging_mixin.py:95} INFO - [2018-08-28 > 13:05:37,193] {pod_launcher.py:95} INFO - > b'INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting > new HTTPS connection (7): blah-blah.s3.eu-west-1.amazonaws.com\n' > ... > ...all other log lines from pod... > {code} > However, this executor pod only exists for the duration of the lifetime of > the task pod so the logs are lost pretty much immediately after the task > runs. There is nothing that ships the logs back to the scheduler and/or web > UI. -- This message was sent by Atlassian Jira (v8.3.2#803003)