Re: S3 logging not working with docker

2018-09-16 Thread vsr . bhavani
On 2018/09/16 05:53:02, Bhavani Ramasamy wrote: > Hello Team, > I am trying to setup S3 logging with docker & CeleryExecutor. Files are not > written to S3. I have configured in airflow.cfg like below, > > remote_logging = True > > remote_log_conn_id = s3_connection_mine > >

Re: S3 logging not working with docker

2018-09-16 Thread Kyle Hamlin
Are you using the DockerOperator? If so, the issue is here in the create_contianer method: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/docker_operator.py#L207 create_container takes a host_config: https://docker-py.readthedocs.io/en/1.2.3/api/#create_container

S3 logging not working with docker

2018-09-16 Thread Bhavani Ramasamy
Hello Team, I am trying to setup S3 logging with docker & CeleryExecutor. Files are not written to S3. I have configured in airflow.cfg like below, remote_logging = True remote_log_conn_id = s3_connection_mine remote_base_log_folder = s3:// mybucket/airflow/logs/ I have tried with