[jira] [Commented] (AIRFLOW-1325) Make Airflow Logging Backed By Elasticsearch
[ https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061491#comment-16061491 ] Allison Wang commented on AIRFLOW-1325: --- Yes airflow will only use ES if the user configures the logging_backend_url and S3/GCS won't be removed :) > Make Airflow Logging Backed By Elasticsearch > > > Key: AIRFLOW-1325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1325 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Allison Wang >Assignee: Allison Wang > > Currently, Airflow uses S3/GCS as the log storage backend. Workers, when > executing the task, flushes logs into local files. When tasks are completed, > those log files will be uploaded to the remote storage system like S3 or GCS. > This approach makes log streaming and analysis difficult. Also when worker > servers are down while executing the task, the entire task log will be lost > until worker servers are recovered. It's also considered a bad practice for > airflow webserver to communicate directly with worker servers. > This change adds functionality to use customized logging backend. Users are > able to configure logging backend that supports streaming logs and more > advanced queries. Currently, Elasticsearch logging backend is implemented. > Having Elasticsearch as logging backend enables the development of more > advanced logging related features. Those are features that will be > implemented in the future: > - Streaming logs without refresh the page > - Separate logs by attempts > - Filter log with excluded phrases > This feature will also be backward compatible. It will direct users to the > old logging flow if logging_backend_url is not set. A new UI will be created > to support above features and old page won't be modified. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1325) Make Airflow Logging Backed By Elasticsearch
[ https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061439#comment-16061439 ] Chris Riccomini commented on AIRFLOW-1325: -- Ah, I see here: https://github.com/apache/incubator-airflow/pull/2380/files#diff-1c2404a3a60f829127232842250ff406R485 That it continues to behave as expected if logging backend isn't configured. :) > Make Airflow Logging Backed By Elasticsearch > > > Key: AIRFLOW-1325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1325 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Allison Wang >Assignee: Allison Wang > > Currently, Airflow uses S3/GCS as the log storage backend. Workers, when > executing the task, flushes logs into local files. When tasks are completed, > those log files will be uploaded to the remote storage system like S3 or GCS. > This approach makes log streaming and analysis difficult. Also when worker > servers are down while executing the task, the entire task log will be lost > until worker servers are recovered. It's also considered a bad practice for > airflow webserver to communicate directly with worker servers. > This change adds functionality to use customized logging backend. Users are > able to configure logging backend that supports streaming logs and more > advanced queries. Currently, Elasticsearch logging backend is implemented. > Having Elasticsearch as logging backend enables the development of more > advanced logging related features. Those are features that will be > implemented in the future: > - Streaming logs without refresh the page > - Separate logs by attempts > - Filter log with excluded phrases > This feature will also be backward compatible. It will direct users to the > old logging flow if logging_backend_url is not set. A new UI will be created > to support above features and old page won't be modified. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1325) Make Airflow Logging Backed By Elasticsearch
[ https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061429#comment-16061429 ] Chris Riccomini commented on AIRFLOW-1325: -- To be clear, are you planning to REMOVE support for S3/GCS? We are heavy users of GCS for logging, and this would be a show stopper for us. > Make Airflow Logging Backed By Elasticsearch > > > Key: AIRFLOW-1325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1325 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Allison Wang >Assignee: Allison Wang > > Currently, Airflow uses S3/GCS as the log storage backend. Workers, when > executing the task, flushes logs into local files. When tasks are completed, > those log files will be uploaded to the remote storage system like S3 or GCS. > This approach makes log streaming and analysis difficult. Also when worker > servers are down while executing the task, the entire task log will be lost > until worker servers are recovered. It's also considered a bad practice for > airflow webserver to communicate directly with worker servers. > This change adds functionality to use customized logging backend. Users are > able to configure logging backend that supports streaming logs and more > advanced queries. Currently, Elasticsearch logging backend is implemented. > Having Elasticsearch as logging backend enables the development of more > advanced logging related features. Those are features that will be > implemented in the future: > - Streaming logs without refresh the page > - Separate logs by attempts > - Filter log with excluded phrases > This feature will also be backward compatible. It will direct users to the > old logging flow if logging_backend_url is not set. A new UI will be created > to support above features and old page won't be modified. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
