Re: Tasks in remain in queued status

I see, there are a couple of reasons, for example, you don't have enough celery workers or workers are faulty. You may want to inspect the celery state using the GUI-based flower tool. Alternatively, try delete all queued tasks, Ai

Re: Adding moderators to the email lists for an incubating project

Sid, Count me in. My apache user is 'arthur' Best, Arthur On Fri, Jun 1, 2018 at 4:28 PM Sid Anand wrote: > Which committers would like to be added as dev and commits list moderators > for Airflow? If interested, let me know on this thread and I will open an > infra ticket. I get between 5-10

Re: Adding moderators to the email lists for an incubating project

I did look at the committer SVN repo and those instructions seemed to get me no where. -s On Fri, Jun 1, 2018 at 4:28 PM, Sid Anand wrote: > Which committers would like to be added as dev and commits list moderators > for Airflow? If interested, let me know on this thread and I will open an > in

Re: Tasks in remain in queued status

> > Using postgres and redis running in their containers. The set up is based > on the astronomer open set up: > https://github.com/astronomerio/astronomer/blob/master/examples/airflow-enterprise/docker-compose.yml

Fwd: Adding moderators to the email lists for an incubating project

Which committers would like to be added as dev and commits list moderators for Airflow? If interested, let me know on this thread and I will open an infra ticket. I get between 5-10 emails a day that require moderation. -s -- Forwarded message -- From: Jakob Homan Date: Fri, Jun

Re: Tasks in remain in queued status

What message broker and result backend are you using? Please make sure they are persistent and data can survive between restarts. On Fri, Jun 1, 2018 at 11:13 AM Pedro Machado wrote: > Hi, > > I am sure this must have come up before but I a quick search of the list > archive didn't return any re

Re: Log external module into airflow log

AFAIK, stdout of the task will be included in the task_instance log, you might get what you want by just printing it. Cheers, Kevin Y On Fri, Jun 1, 2018 at 12:20 PM Martin Gauthier wrote: > Here is the output i am getting from the log in airflow UI > > *** Reading local log. > [2018-06-01 15:1

Re: Log external module into airflow log

Here is the output i am getting from the log in airflow UI *** Reading local log. [2018-06-01 15:15:55,979] {{cli.py:374}} INFO - Running on host airflow-webserver [2018-06-01 15:15:56,042] {{models.py:1197}} INFO - Dependencies all met for [2018-06-01 15:15:56,090] {{models.py:1197}} INFO - Depe

Re: Log external module into airflow log

For example this python operator write_polylines_geo_data = PythonOperator( task_id='write_geo_data_polylines', provide_context=True, python_callable=generate_geo_data_files_from_wkt_data_file, op_args=[ polylines_staging_file_loc, 'pickle', WGS54, 'geometry', None, (polylines_geojson_file_locatio

Tasks in remain in queued status

Hi, I am sure this must have come up before but I a quick search of the list archive didn't return any results. I am running airflow 1.9 on docker using the CeleryExecutor. I am seeing tasks that stay in queued status for a long time. This morning, I restarted the scheduler service and the tasks

Re: Log external module into airflow log

From inside a running task these should already show up in the task instance log. Do they not? Or do you want the logging from parsing the DAG itself? These probably get routed to $AIRFLOW_HOME/logs/scheduler/latest/$dag_id.log (path approximate - from memory) -ash > On 1 Jun 2018, at 19:02,

Log external module into airflow log

Hi I would like to know if their is any possibilities where we could use the airflow logger config to handle logging from external modules used into any DAGs Workflow. For example : Let's say I have a DAG which use this lib mylib.utils If mylib.utils has a log = logging.getLogger(__name__

Re: Airflow London Meetup Group

I usually visit London in March for QCon London... Aside from hoping to see an Airflow talk at QCon London someday soon, I'd also like to meet the Airflow community in London town. Thx for starting this! -s On Fri, Jun 1, 2018 at 9:56 AM, Sid Anand wrote: > Woot! > > I've updated : > >- htt

Re: Airflow London Meetup Group

Woot! I've updated : - https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups - https://twitter.com/ApacheAirflow/status/1002594373849120769 - https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements On Fri, Jun 1, 2018 at 5:42 AM, Naik Kaxil wrote: > We have just creat

Re: Airflow London Meetup Group

We have just created the Airflow London meetup and hope to grow the Airflow community in London. Meetup URL: https://www.meetup.com/London-Apache-Airflow-Meetup/ Looking forward to meet Airflow users in London. Regards, Kaxil On 01/06/2018, 10:31, "Tautvydas" wrote: I'd be interested i

Re: Airflow London Meetup Group

I'd be interested in Airflow meetup as well. Cheers, Tautvydas On 1 June 2018 at 08:55, Christian Barra wrote: > > > > Hi there, > > > > > > > > We(me and my colleagues at Data Reply UK) are planning to start a > meetup > > for Airflow users. Me & one of my friend already gave a talk at PyData

Re: Airflow London Meetup Group

> > Hi there, > > > > We(me and my colleagues at Data Reply UK) are planning to start a meetup > for Airflow users. Me & one of my friend already gave a talk at PyData > London last month and people showed huge interest in it. > > > > slightly unrelated but this year EuroPython is going to be in

Re: conn_id breaking change; once more with feeling

Thanks Maxime & Ash, So, I'm glad to hear that minor releases should never invalidate DAGs. With rare exceptions being possible. I like the idea that this is (or was going to be) written up on the wiki as a policy. In it I suppose defining what a minor version is, would help. (I think it's Maj