Avoid sensor sleep by rescheduling task, was: Re: How to wait for external process

[Stefan Seelmann]

I digged a bit into the Airflow code and I think I found a possible
solution, see draft at [1]:
Add a "reschedule" flag to BaseSensorOperator, when set it doesn't sleep
but raises an AirflowRescheduleTask exception. Within the TaskInstance
this exception is handled, similar to a failure. The task state is set
to UP_FOR_RETRY. The task is rescheduled by the scheduler.

Advantages:
* Only small code changes are required
* Leverage the existing retry mechanism, including delay and exponential
backoff

Tradeoff:
* Overhead by running the task again and again


I'd like to ask the community if that is a valueable change to be
included into Airflow?


If so I'll create a Jira, improve the branch, and send a PR:
* Find a solution for timeout and soft_fail
* Add tests
* Add a task_reschedule table (similar to the task_fail table)

Also some open questions:
* Should a separate state (e.g. UP_FOR_RESCHEDULE) be used to
differentiate between error and intended reschedule? Then proably also
different parameters for delay and exponential backoff make sense.
* Maybe it's feasable to execute the sensor poke code directly by the
scheduler to avoid execution of mini task? But that can be done in a
separate change.


Kind Regards,
Stefan

[1]
https://github.com/seelmann/incubator-airflow/commit/379bf0fdf9bbb9f26f74ac5fa6325ba5a0e975a2

Re: Tasks in remain in queued status

[Pedro Machado]

It looks like the parallelism setting was set pretty low via an environment
variable and I wasn't aware. Could that be the cause?

I'll write back of the problem persists.

Thanks for your help!
Pedro

Re: Adding moderators to the email lists for an incubating project

[Naik Kaxil]

Hi Sid,

Count me in. My Apache id is 'kaxilnaik'.

Regards,
Kaxil


Kaxil Naik

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

[Data Reply]

From: Sid Anand 
Sent: Saturday, June 2, 2018 12:28:16 AM
To: dev@airflow.incubator.apache.org
Subject: Fwd: Adding moderators to the email lists for an incubating project

Which committers would like to be added as dev and commits list moderators
for Airflow? If interested, let me know on this thread and I will open an
infra ticket. I get between 5-10 emails a day that require moderation.

-s

-- Forwarded message --
From: Jakob Homan 
Date: Fri, Jun 1, 2018 at 3:49 PM
Subject: Re: Adding moderators to the email lists for an incubating project
To: gene...@incubator.apache.org


Sid-
  Adding new moderators is through an infra ticket:
https://apache.org/dev/committers.html#mailing-list-moderators

   I think the resources there for describing who are the current
moderators may be out of date.
-jg


On 1 June 2018 at 14:55, Sid Anand  wrote:
> Folks!
> I seem to be one of the few committers on Airflow's email moderator list.
> I'd like to add other committers who have voiced interest in moderating
the
> lists. How can I add others and view the current list of moderators?
>
> -s (Sid)

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Airflow London Meetup Group

[Naik Kaxil]

I would surely try to give a talk at next Qcon event. And would love to see you 
at the meetup whenever you visit London.

Regards,
Kaxil


Kaxil Naik

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

[Data Reply]

From: Sid Anand 
Sent: Friday, June 1, 2018 5:57:23 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow London Meetup Group

I usually visit London in March for QCon London... Aside from hoping to see
an Airflow talk at QCon London someday soon, I'd also like to meet the
Airflow community in London town. Thx for starting this!

-s

On Fri, Jun 1, 2018 at 9:56 AM, Sid Anand  wrote:

> Woot!
>
> I've updated :
>
>- https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
>- https://twitter.com/ApacheAirflow/status/1002594373849120769
>- https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements
>
>
> On Fri, Jun 1, 2018 at 5:42 AM, Naik Kaxil  wrote:
>
>> We have just created the Airflow London meetup and hope to grow the
>> Airflow community in London.
>>
>> Meetup URL: https://www.meetup.com/London-Apache-Airflow-Meetup/
>>
>> Looking forward to meet Airflow users in London.
>>
>> Regards,
>> Kaxil
>>
>> On 01/06/2018, 10:31, "Tautvydas"  wrote:
>>
>> I'd be interested in Airflow meetup as well.
>>
>> Cheers,
>> Tautvydas
>>
>> On 1 June 2018 at 08:55, Christian Barra  wrote:
>>
>> > >
>> > > Hi there,
>> >
>> >
>> > > >
>> > > > We(me and my colleagues at Data Reply UK) are planning to start
>> a
>> > meetup
>> > > for Airflow users. Me & one of my friend already gave a talk at
>> PyData
>> > > London last month and people showed huge interest in it.
>> > > >
>> > >
>> >
>> > >
>> > slightly unrelated but this year EuroPython is going to be in
>> Edinburgh
>> > (July).
>> >
>> > What about organizing a sprint there?
>> >
>>
>>
>>
>>
>>
>>
>> Kaxil Naik
>>
>> Data Reply
>> 2nd Floor, Nova South
>> 160 Victoria Street, Westminster
>> London SW1E 5LB - UK
>> phone: +44 (0)20 7730 6000
>> k.n...@reply.com
>> www.reply.com
>>
>
>

Re: Adding moderators to the email lists for an incubating project

[Naik Kaxil]

Sid,

Count me in as well. My apache id is 'kaxilnaik'.


Kaxil Naik

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

[Data Reply]

From: Arthur Wiedmer 
Sent: Saturday, June 2, 2018 12:34:22 AM
To: dev@airflow.incubator.apache.org
Subject: Re: Adding moderators to the email lists for an incubating project

Sid,

Count me in. My apache user is 'arthur'

Best,
Arthur

On Fri, Jun 1, 2018 at 4:28 PM Sid Anand  wrote:

> Which committers would like to be added as dev and commits list moderators
> for Airflow? If interested, let me know on this thread and I will open an
> infra ticket. I get between 5-10 emails a day that require moderation.
>
> -s
>
> -- Forwarded message --
> From: Jakob Homan 
> Date: Fri, Jun 1, 2018 at 3:49 PM
> Subject: Re: Adding moderators to the email lists for an incubating project
> To: gene...@incubator.apache.org
>
>
> Sid-
>   Adding new moderators is through an infra ticket:
> https://apache.org/dev/committers.html#mailing-list-moderators
>
>I think the resources there for describing who are the current
> moderators may be out of date.
> -jg
>
>
> On 1 June 2018 at 14:55, Sid Anand  wrote:
> > Folks!
> > I seem to be one of the few committers on Airflow's email moderator list.
> > I'd like to add other committers who have voiced interest in moderating
> the
> > lists. How can I add others and view the current list of moderators?
> >
> > -s (Sid)
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>