On 2018/09/17 03:03:06, Pedro Machado <[email protected]> wrote:
> Bhavani,
>
> I was able to get it to work. I am using a modified version of
> https://github.com/puckel/docker-airflow
>
> Here is what I had to do:
>
> Environment variables:
>
> I did not change airflow.cfg, but put the following variables in .env and
> modified docker-compose.yml to pass them to the containers:
>
> In .env:
>
> LOGGING_CONFIG_CLASS=log_config.DEFAULT_LOGGING_CONFIG
> REMOTE_BASE_LOG_FOLDER=s3://<bucket>/airflow-logs-dev/
> REMOTE_LOG_CONN_ID=<my s3 connection>
>
> Then in docker-compose.yml, I passed the following under environment for
> all the airflow containers:
>
> environment:
> - LOAD_EX=n
> - FERNET_KEY=<my key>
> - EXECUTOR=Celery
> - POSTGRES_USER=${POSTGRES_USER}
> - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
> - POSTGRES_DB=${POSTGRES_DB}
> - POSTGRES_HOST=${POSTGRES_HOST}
> - AIRFLOW__CORE__LOGGING_CONFIG_CLASS=${LOGGING_CONFIG_CLASS}
> - AIRFLOW__CORE__REMOTE_LOGGING=True
> -
> AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER=${REMOTE_BASE_LOG_FOLDER}
> - AIRFLOW__CORE__REMOTE_LOG_CONN_ID=${REMOTE_LOG_CONN_ID}
> - AIRFLOW__SMTP__SMTP_MAIL_FROM=${SMTP_MAIL_FROM}
> - AIRFLOW__WEBSERVER__BASE_URL=${BASE_URL}
>
> I then created the following files:
>
> __init__.py (empty)
> log_config.py (used code from
> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py
> )
>
> and I set up my Dockerfile to copy them to the ${AIRFLOW_HOME}/config/
> directory:
>
> COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
> COPY config/*.py ${AIRFLOW_HOME}/config/
>
> After this, the workers were able to log to s3 successfully.
>
> This StackOverfiow answer helped me but I had to make some tweaks:
> https://stackoverflow.com/questions/50222860/airflow-wont-write-logs-to-s3
>
> Let me know if this works for you.
>
> Pedro
>
>
> On Sun, Sep 16, 2018 at 1:12 AM Bhavani Ramasamy <[email protected]>
> wrote:
>
> > I am facing the same issue.. Did you find any solution for this?
> >
> > On 2018/09/08 01:05:26, Pedro Machado <[email protected]> wrote:
> > > I am looking at the documentation here>
> > >
> > https://airflow.incubator.apache.org/howto/write-logs.html#writing-logs-to-amazon-s3>
> >
> > > and am wondering if for the s3 configuration it's not necessary to
> > create a>
> > > log configuration file as it's described under the GCP section>
> > >
> > https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform>
> >
> > >
> > > I ran a quick test configuring remote_base_log_folder>
> > > and remote_log_conn_id through environment variables and it didn't
> > work.>
> > >
> > > Could someone shed some light on this?>
> > >
> > > Thanks,>
> > >
> > > Pedro>
> > >
> Hello Pedro,
Thanks for quick response. I am using the similar configurations like you
mentioned. S3 logs are working when i am using LocalExecutor. But when I am
using CeleryExecutor it is not writing to S3. I have the celery result backend
also connected to the same PG database & broker as redis queue. Are you able to
use CeleryExecutor & run airflow in multiple docker containers with S3 logs?
Thanks,
Bhavani
