Fang-Pen Lin created AIRFLOW-2325:
-------------------------------------
Summary: Task logging with AWS Cloud watch
Key: AIRFLOW-2325
URL: https://issues.apache.org/jira/browse/AIRFLOW-2325
Project: Apache Airflow
Issue Type: New Feature
Components: logging
Reporter: Fang-Pen Lin
In many cases, it's ideal to use remote logging while running Airflow in
production, as the worker could be easily scale down or scale up. Or the worker
is running in containers, where the local storage is not meant to be there
forever. In that case, the S3 task logging handler could be used
[https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/s3_task_handler.py]
However, it comes with drawback. S3 logging handler only uploads the log when
the task completed or failed. For long running tasks, it's hard to know what's
going on with the process until it finishes.
To make more real-time logging, I built a logging handler based on AWS
CloudWatch. It uses a third party python package `watchtower`
[https://github.com/kislyuk/watchtower/tree/master/watchtower]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
