[
https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allison Wang updated AIRFLOW-1325:
----------------------------------
Summary: Airflow Streaming Log Backed By ElasticSearch (was: Make Airflow
Logging Backed By Elasticsearch)
> Airflow Streaming Log Backed By ElasticSearch
> ---------------------------------------------
>
> Key: AIRFLOW-1325
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1325
> Project: Apache Airflow
> Issue Type: Improvement
> Reporter: Allison Wang
> Assignee: Allison Wang
>
> Currently, Airflow uses S3/GCS as the log storage backend. Workers, when
> executing the task, flushes logs into local files. When tasks are completed,
> those log files will be uploaded to the remote storage system like S3 or GCS.
> This approach makes log streaming and analysis difficult. Also when worker
> servers are down while executing the task, the entire task log will be lost
> until worker servers are recovered. It's also considered a bad practice for
> airflow webserver to communicate directly with worker servers.
> This change adds functionality to use customized logging backend. Users are
> able to configure logging backend that supports streaming logs and more
> advanced queries. Currently, Elasticsearch logging backend is implemented.
> This feature will also be backward compatible. It will direct users to the
> old logging flow if logging_backend_url is not set. A new UI will be created
> to support above features and old page won't be modified.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
