Fang-Pen Lin created AIRFLOW-2325:

             Summary: Task logging with AWS Cloud watch
                 Key: AIRFLOW-2325
             Project: Apache Airflow
          Issue Type: New Feature
          Components: logging
            Reporter: Fang-Pen Lin

In many cases, it's ideal to use remote logging while running Airflow in 
production, as the worker could be easily scale down or scale up. Or the worker 
is running in containers, where the local storage is not meant to be there 
forever. In that case, the S3 task logging handler could be used


However, it comes with drawback. S3 logging handler only uploads the log when 
the task completed or failed. For long running tasks, it's hard to know what's 
going on with the process until it finishes.

To make more real-time logging, I built a logging handler based on AWS 
CloudWatch. It uses a third party python package `watchtower`



This message was sent by Atlassian JIRA

Reply via email to