Avoid sensor sleep by rescheduling task, was: Re: How to wait for external process

I digged a bit into the Airflow code and I think I found a possible solution, see draft at [1]: Add a "reschedule" flag to BaseSensorOperator, when set it doesn't sleep but raises an AirflowRescheduleTask exception. Within the TaskInstance this exception is handled, similar to a failure. The task

Re: Tasks in remain in queued status

It looks like the parallelism setting was set pretty low via an environment variable and I wasn't aware. Could that be the cause? I'll write back of the problem persists. Thanks for your help! Pedro

Re: Adding moderators to the email lists for an incubating project

Hi Sid, Count me in. My Apache id is 'kaxilnaik'. Regards, Kaxil Kaxil Naik Data Reply 2nd Floor, Nova South 160 Victoria Street, Westminster London SW1E 5LB - UK phone: +44 (0)20 7730 6000 k.n...@reply.com www.reply.com [Data Reply]

Re: Airflow London Meetup Group

I would surely try to give a talk at next Qcon event. And would love to see you at the meetup whenever you visit London. Regards, Kaxil Kaxil Naik Data Reply 2nd Floor, Nova South 160 Victoria Street, Westminster London SW1E 5LB - UK phone: +44 (0)20 7730 6000

Re: Adding moderators to the email lists for an incubating project

Sid, Count me in as well. My apache id is 'kaxilnaik'. Kaxil Naik Data Reply 2nd Floor, Nova South 160 Victoria Street, Westminster London SW1E 5LB - UK phone: +44 (0)20 7730 6000 k.n...@reply.com www.reply.com [Data Reply] From: