Arthur Vigil created AIRFLOW-1667:
-------------------------------------
Summary: Remote log handlers don't upload logs
Key: AIRFLOW-1667
URL: https://issues.apache.org/jira/browse/AIRFLOW-1667
Project: Apache Airflow
Issue Type: Bug
Components: logging
Affects Versions: 1.9.0, 1.10.0
Reporter: Arthur Vigil
AIRFLOW-1385 revised logging for configurability, but the provided remote log
handlers (S3TaskHandler and GCSTaskHandler) only upload on close (flush is left
at the default implementation provided by `logging.FileHandler`). A handler
will be closed on process exit by `logging.shutdown()`, but depending on the
Executor used worker processes may not regularly shutdown, and can very likely
persist between tasks. This means during normal execution log files are never
uploaded.
Need to find a way to flush remote log handlers in a timely manner, but without
hitting the target resources unnecessarily.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
