[
https://issues.apache.org/jira/browse/AIRFLOW-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Hallan updated AIRFLOW-4077:
-----------------------------------
Description:
We have been writing our airflow logs to S3 for quite sometime now
successfully, using the worker class 'sync' which is the default. Viewing logs
via the airflow UI works perfectly, in the sense that logs are read from the
remote s3 location.
When switching the worker class to gevent, the airflow UI no longer attempts to
read the logs from the remote s3 location, rather it always attempts to read
the logs from the file and as we are using a containerized environment reading
log files in this way fails.
Switching only the worker class back to sync and leaving all other
configuration untouched resolves this issue.
was:
We have been writing our airflow logs to S3 for quite sometime now
successfully, using the worker class 'sync' which is the default. Viewing logs
via the airflow UI works perfectly, in the sense that logs are read from the
remote s3 location.
When switching the worker class to gevent, the airflow UI no longer attempts to
read the logs from the remote s3 location, rather it always attempts to read
the logs from the file and as we are using a containerized environment read log
files in this was fails.
Switching only the worker class back to sync and leaving all other
configuration untouched resolves this issue.
> Using gevent worker class, breaks reading remote log from s3
> ------------------------------------------------------------
>
> Key: AIRFLOW-4077
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4077
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging, webapp, worker
> Affects Versions: 1.10.2
> Environment: Local - Ubuntu
> Production - AWS Elastic Beanstalk
> Reporter: Sanjay Hallan
> Priority: Major
> Labels: S3, gevent, logging, worker
>
> We have been writing our airflow logs to S3 for quite sometime now
> successfully, using the worker class 'sync' which is the default. Viewing
> logs via the airflow UI works perfectly, in the sense that logs are read from
> the remote s3 location.
> When switching the worker class to gevent, the airflow UI no longer attempts
> to read the logs from the remote s3 location, rather it always attempts to
> read the logs from the file and as we are using a containerized environment
> reading log files in this way fails.
>
> Switching only the worker class back to sync and leaving all other
> configuration untouched resolves this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
