Bhavani,
I was able to get it to work. I am using a modified version of
https://github.com/puckel/docker-airflow
Here is what I had to do:
Environment variables:
I did not change airflow.cfg, but put the following variables in .env and
modified docker-compose.yml to pass them to the containers:
In .env:
LOGGING_CONFIG_CLASS=log_config.DEFAULT_LOGGING_CONFIG
REMOTE_BASE_LOG_FOLDER=s3://<bucket>/airflow-logs-dev/
REMOTE_LOG_CONN_ID=<my s3 connection>
Then in docker-compose.yml, I passed the following under environment for
all the airflow containers:
environment:
- LOAD_EX=n
- FERNET_KEY=<my key>
- EXECUTOR=Celery
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
- POSTGRES_DB=${POSTGRES_DB}
- POSTGRES_HOST=${POSTGRES_HOST}
- AIRFLOW__CORE__LOGGING_CONFIG_CLASS=${LOGGING_CONFIG_CLASS}
- AIRFLOW__CORE__REMOTE_LOGGING=True
-
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER=${REMOTE_BASE_LOG_FOLDER}
- AIRFLOW__CORE__REMOTE_LOG_CONN_ID=${REMOTE_LOG_CONN_ID}
- AIRFLOW__SMTP__SMTP_MAIL_FROM=${SMTP_MAIL_FROM}
- AIRFLOW__WEBSERVER__BASE_URL=${BASE_URL}
I then created the following files:
__init__.py (empty)
log_config.py (used code from
https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py
)
and I set up my Dockerfile to copy them to the ${AIRFLOW_HOME}/config/
directory:
COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
COPY config/*.py ${AIRFLOW_HOME}/config/
After this, the workers were able to log to s3 successfully.
This StackOverfiow answer helped me but I had to make some tweaks:
https://stackoverflow.com/questions/50222860/airflow-wont-write-logs-to-s3
Let me know if this works for you.
Pedro
On Sun, Sep 16, 2018 at 1:12 AM Bhavani Ramasamy <[email protected]>
wrote:
> I am facing the same issue.. Did you find any solution for this?
>
> On 2018/09/08 01:05:26, Pedro Machado <[email protected]> wrote:
> > I am looking at the documentation here>
> >
> https://airflow.incubator.apache.org/howto/write-logs.html#writing-logs-to-amazon-s3>
>
> > and am wondering if for the s3 configuration it's not necessary to
> create a>
> > log configuration file as it's described under the GCP section>
> >
> https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform>
>
> >
> > I ran a quick test configuring remote_base_log_folder>
> > and remote_log_conn_id through environment variables and it didn't
> work.>
> >
> > Could someone shed some light on this?>
> >
> > Thanks,>
> >
> > Pedro>
> >
