[
https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allison Wang updated AIRFLOW-1325:
----------------------------------
Description: Add Elasticsearch logging backend. (was: Currently, Airflow
uses S3/GCS as the log storage backend. Workers, when executing the task,
flushes logs into local files. When tasks are completed, those log files will
be uploaded to the remote storage system like S3 or GCS. This approach makes
log streaming and analysis difficult. Also when worker servers are down while
executing the task, the entire task log will be lost until worker servers are
recovered. It's also considered a bad practice for airflow webserver to
communicate directly with worker servers.
This change adds functionality to use customized logging backend. Users are
able to configure logging backend that supports streaming logs and more
advanced queries. Currently, Elasticsearch logging backend is implemented.
This feature will also be backward compatible. It will direct users to the old
logging flow if logging_backend_url is not set. A new UI will be created to
support above features and old page won't be modified.)
> Airflow Streaming Log Backed By ElasticSearch
> ---------------------------------------------
>
> Key: AIRFLOW-1325
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1325
> Project: Apache Airflow
> Issue Type: Improvement
> Components: logging
> Reporter: Allison Wang
> Assignee: Allison Wang
>
> Add Elasticsearch logging backend.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
