[jira] [Updated] (AIRFLOW-1325) Airflow streaming log backed by ElasticSearch
[ https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated AIRFLOW-1325: -- Summary: Airflow streaming log backed by ElasticSearch (was: Enable ElasticSearch for Airflow Logs) > Airflow streaming log backed by ElasticSearch > - > > Key: AIRFLOW-1325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1325 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Reporter: Allison Wang >Assignee: Allison Wang > > Add Elasticsearch log handler and reader for querying logs in ES -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1325) Airflow Streaming Log Backed By ElasticSearch
[ https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated AIRFLOW-1325: -- Description: Add Elasticsearch logging backend. (was: Currently, Airflow uses S3/GCS as the log storage backend. Workers, when executing the task, flushes logs into local files. When tasks are completed, those log files will be uploaded to the remote storage system like S3 or GCS. This approach makes log streaming and analysis difficult. Also when worker servers are down while executing the task, the entire task log will be lost until worker servers are recovered. It's also considered a bad practice for airflow webserver to communicate directly with worker servers. This change adds functionality to use customized logging backend. Users are able to configure logging backend that supports streaming logs and more advanced queries. Currently, Elasticsearch logging backend is implemented. This feature will also be backward compatible. It will direct users to the old logging flow if logging_backend_url is not set. A new UI will be created to support above features and old page won't be modified.) > Airflow Streaming Log Backed By ElasticSearch > - > > Key: AIRFLOW-1325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1325 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Reporter: Allison Wang >Assignee: Allison Wang > > Add Elasticsearch logging backend. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1325) Airflow Streaming Log Backed By ElasticSearch
[ https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated AIRFLOW-1325: Component/s: logging > Airflow Streaming Log Backed By ElasticSearch > - > > Key: AIRFLOW-1325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1325 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Reporter: Allison Wang >Assignee: Allison Wang > > Currently, Airflow uses S3/GCS as the log storage backend. Workers, when > executing the task, flushes logs into local files. When tasks are completed, > those log files will be uploaded to the remote storage system like S3 or GCS. > This approach makes log streaming and analysis difficult. Also when worker > servers are down while executing the task, the entire task log will be lost > until worker servers are recovered. It's also considered a bad practice for > airflow webserver to communicate directly with worker servers. > This change adds functionality to use customized logging backend. Users are > able to configure logging backend that supports streaming logs and more > advanced queries. Currently, Elasticsearch logging backend is implemented. > This feature will also be backward compatible. It will direct users to the > old logging flow if logging_backend_url is not set. A new UI will be created > to support above features and old page won't be modified. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1325) Airflow Streaming Log Backed By ElasticSearch
[ https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated AIRFLOW-1325: -- Summary: Airflow Streaming Log Backed By ElasticSearch (was: Make Airflow Logging Backed By Elasticsearch) > Airflow Streaming Log Backed By ElasticSearch > - > > Key: AIRFLOW-1325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1325 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Allison Wang >Assignee: Allison Wang > > Currently, Airflow uses S3/GCS as the log storage backend. Workers, when > executing the task, flushes logs into local files. When tasks are completed, > those log files will be uploaded to the remote storage system like S3 or GCS. > This approach makes log streaming and analysis difficult. Also when worker > servers are down while executing the task, the entire task log will be lost > until worker servers are recovered. It's also considered a bad practice for > airflow webserver to communicate directly with worker servers. > This change adds functionality to use customized logging backend. Users are > able to configure logging backend that supports streaming logs and more > advanced queries. Currently, Elasticsearch logging backend is implemented. > This feature will also be backward compatible. It will direct users to the > old logging flow if logging_backend_url is not set. A new UI will be created > to support above features and old page won't be modified. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
