[ 
https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allison Wang updated AIRFLOW-1325:
----------------------------------
    Summary: Airflow Streaming Log Backed By ElasticSearch  (was: Make Airflow 
Logging Backed By Elasticsearch)

> Airflow Streaming Log Backed By ElasticSearch
> ---------------------------------------------
>
>                 Key: AIRFLOW-1325
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1325
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Allison Wang
>            Assignee: Allison Wang
>
> Currently, Airflow uses S3/GCS as the log storage backend. Workers, when 
> executing the task, flushes logs into local files. When tasks are completed, 
> those log files will be uploaded to the remote storage system like S3 or GCS. 
> This approach makes log streaming and analysis difficult. Also when worker 
> servers are down while executing the task, the entire task log will be lost 
> until worker servers are recovered. It's also considered a bad practice for 
> airflow webserver to communicate directly with worker servers.
> This change adds functionality to use customized logging backend. Users are 
> able to configure logging backend that supports streaming logs and more 
> advanced queries. Currently, Elasticsearch logging backend is implemented.
> This feature will also be backward compatible. It will direct users to the 
> old logging flow if logging_backend_url is not set. A new UI will be created 
> to support above features and old page won't be modified.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to