Re: Give it up for Fokko!

[Sid Anand]

+100

On Fri, Apr 13, 2018 at 4:08 PM, Alex Tronchin-James 949-412-7220 <
alex.n.ja...@gmail.com> wrote:

> Bravo!!! Bien fait!
>
> On Fri, Apr 13, 2018 at 3:54 PM Joy Gao  wrote:
>
> >  
> >
> > On Fri, Apr 13, 2018 at 11:47 AM, Naik Kaxil  wrote:
> >
> > > Couldn't agree more. Thanks Fokko
> > >
> > > On 13/04/2018, 17:56, "Maxime Beauchemin"  >
> > > wrote:
> > >
> > > Hey all,
> > >
> > > I wanted to point out the amazing work that Fokko is doing,
> > > reviewing/merging PRs and doing fantastic committer & maintainer
> > work.
> > > It
> > > takes a variety of contributions to make projects like Airflow
> > thrive,
> > > but
> > > without this kind of involvement it wouldn't be possible to keep
> > > shipping
> > > better versions of the product steadily.
> > >
> > > Cheers to that!
> > >
> > > Max
> > >
> > >
> > >
> > >
> > >
> > >
> > > Kaxil Naik
> > >
> > > Data Reply
> > > 38 Grosvenor Gardens
> > > London SW1W 0EB - UK
> > > phone: +44 (0)20 7730 6000 <+44%2020%207730%206000>
> > > k.n...@reply.com
> > > www.reply.com
> > >
> >
>

Re: Give it up for Fokko!

[Alex Tronchin-James 949-412-7220]

Bravo!!! Bien fait!

On Fri, Apr 13, 2018 at 3:54 PM Joy Gao  wrote:

>  
>
> On Fri, Apr 13, 2018 at 11:47 AM, Naik Kaxil  wrote:
>
> > Couldn't agree more. Thanks Fokko
> >
> > On 13/04/2018, 17:56, "Maxime Beauchemin" 
> > wrote:
> >
> > Hey all,
> >
> > I wanted to point out the amazing work that Fokko is doing,
> > reviewing/merging PRs and doing fantastic committer & maintainer
> work.
> > It
> > takes a variety of contributions to make projects like Airflow
> thrive,
> > but
> > without this kind of involvement it wouldn't be possible to keep
> > shipping
> > better versions of the product steadily.
> >
> > Cheers to that!
> >
> > Max
> >
> >
> >
> >
> >
> >
> > Kaxil Naik
> >
> > Data Reply
> > 38 Grosvenor Gardens
> > London SW1W 0EB - UK
> > phone: +44 (0)20 7730 6000 <+44%2020%207730%206000>
> > k.n...@reply.com
> > www.reply.com
> >
>

Re: Give it up for Fokko!

[Joy Gao]

 

On Fri, Apr 13, 2018 at 11:47 AM, Naik Kaxil  wrote:

> Couldn't agree more. Thanks Fokko
>
> On 13/04/2018, 17:56, "Maxime Beauchemin" 
> wrote:
>
> Hey all,
>
> I wanted to point out the amazing work that Fokko is doing,
> reviewing/merging PRs and doing fantastic committer & maintainer work.
> It
> takes a variety of contributions to make projects like Airflow thrive,
> but
> without this kind of involvement it wouldn't be possible to keep
> shipping
> better versions of the product steadily.
>
> Cheers to that!
>
> Max
>
>
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 38 Grosvenor Gardens
> London SW1W 0EB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>

Re: Give it up for Fokko!

[Chris Riccomini]

+1 !

On Fri, Apr 13, 2018 at 10:27 AM, Tao Feng  wrote:

> Thanks Fokko. Really appreciate your great feedback and your hard work on
> helping airflow project thrive:)
>
> On Fri, Apr 13, 2018 at 9:55 AM, Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
>
> > Hey all,
> >
> > I wanted to point out the amazing work that Fokko is doing,
> > reviewing/merging PRs and doing fantastic committer & maintainer work. It
> > takes a variety of contributions to make projects like Airflow thrive,
> but
> > without this kind of involvement it wouldn't be possible to keep shipping
> > better versions of the product steadily.
> >
> > Cheers to that!
> >
> > Max
> >
>

Re: Give it up for Fokko!

[Tao Feng]

Thanks Fokko. Really appreciate your great feedback and your hard work on
helping airflow project thrive:)

On Fri, Apr 13, 2018 at 9:55 AM, Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> Hey all,
>
> I wanted to point out the amazing work that Fokko is doing,
> reviewing/merging PRs and doing fantastic committer & maintainer work. It
> takes a variety of contributions to make projects like Airflow thrive, but
> without this kind of involvement it wouldn't be possible to keep shipping
> better versions of the product steadily.
>
> Cheers to that!
>
> Max
>

Re: Benchmarking of Airflow Scheduler with Celery Executor

[Maxime Beauchemin]

If you're concerned about scheduler scalability I'd go with a bigger box.
The scheduler uses multiprocessing so more CPU power means more throughput.

Also you may want to provision a beefy MySQL box to make sure that doesn't
become the bottleneck. 10k tasks heartbeating to the DB every 30 seconds is
significant load.

Perhaps Airbnb folks chime in about their scale and hardware setup?

Max

On Fri, Apr 13, 2018 at 9:14 AM, ramandu...@gmail.com 
wrote:

> Thanks Ry,
> Just wondering if there is any approximate number on concurrent tasks a
> scheduler can run on say 16 GB RAM and 8 core machine.
> If its already been done that would be useful.
> We did some benchmarking with local executor and observed that each
> TaskInstance was taking ~100MB of memory so we could only run ~130
> concurrent tasks on 16 GB RAM and 8 core machine.
>
> -Raman Gupta
>
>
>
> On 2018/04/12 16:32:37, Ry Walker  wrote:
> > Hi Raman -
> >
> > First, we’d be happy to help you test this out with Airflow. Or you could
> > do it yourself by using http://open.astronomer.io/airflow/ (w/ Docker
> > Engine + Docker Compose) to quickly spin up a test environment.
> Everything
> > is hooked to Prometheus/Grafana to monitor how the system reacts to your
> > workload.
> >
> > -Ry
> > CEO, Astronomer
> >
> > On April 12, 2018 at 12:23:46 PM, ramandu...@gmail.com (
> ramandu...@gmail.com)
> > wrote:
> >
> > Hi All,
> > We have requirement to run 10k(s) of concurrent tasks. We are exploring
> > Airflow's Celery Executor for same. Horizontally Scaling of worker nodes
> > seem possible but it can only have one active scheduler.
> > So will Airflow scheduler be able to handle these many concurrent tasks.
> > Is there any benchmarking number around airflow scheduler's scalability.
> > Thanks,
> > Raman
> >
>

Re: Give it up for Fokko!

[Ace Haidrey]

Couldn't agree more. Appreciate you Fokko! Thanks for all of your hard work

Cheers!
Ace

> On Apr 13, 2018, at 9:55 AM, Maxime Beauchemin  
> wrote:
> 
> Hey all,
> 
> I wanted to point out the amazing work that Fokko is doing,
> reviewing/merging PRs and doing fantastic committer & maintainer work. It
> takes a variety of contributions to make projects like Airflow thrive, but
> without this kind of involvement it wouldn't be possible to keep shipping
> better versions of the product steadily.
> 
> Cheers to that!
> 
> Max

Give it up for Fokko!

[Maxime Beauchemin]

Hey all,

I wanted to point out the amazing work that Fokko is doing,
reviewing/merging PRs and doing fantastic committer & maintainer work. It
takes a variety of contributions to make projects like Airflow thrive, but
without this kind of involvement it wouldn't be possible to keep shipping
better versions of the product steadily.

Cheers to that!

Max

Re: Benchmarking of Airflow Scheduler with Celery Executor

[ramandumcs]

Thanks Ry,
Just wondering if there is any approximate number on concurrent tasks a 
scheduler can run on say 16 GB RAM and 8 core machine.
If its already been done that would be useful.
We did some benchmarking with local executor and observed that each 
TaskInstance was taking ~100MB of memory so we could only run ~130 concurrent 
tasks on 16 GB RAM and 8 core machine.

-Raman Gupta  

 

On 2018/04/12 16:32:37, Ry Walker  wrote: 
> Hi Raman -
> 
> First, we’d be happy to help you test this out with Airflow. Or you could
> do it yourself by using http://open.astronomer.io/airflow/ (w/ Docker
> Engine + Docker Compose) to quickly spin up a test environment. Everything
> is hooked to Prometheus/Grafana to monitor how the system reacts to your
> workload.
> 
> -Ry
> CEO, Astronomer
> 
> On April 12, 2018 at 12:23:46 PM, ramandu...@gmail.com (ramandu...@gmail.com)
> wrote:
> 
> Hi All,
> We have requirement to run 10k(s) of concurrent tasks. We are exploring
> Airflow's Celery Executor for same. Horizontally Scaling of worker nodes
> seem possible but it can only have one active scheduler.
> So will Airflow scheduler be able to handle these many concurrent tasks.
> Is there any benchmarking number around airflow scheduler's scalability.
> Thanks,
> Raman
> 

Re: Cancel a Running dag

[ramandumcs]

Thanks Bolke,
Will it become part of Airflow 1.10 release. Is there any tentative timeline 
for same.
-Raman Gupta

On 2018/04/12 19:19:07, Bolke de Bruin  wrote: 
> This is now fixed in master. Clearing tasks will now properly terminate a 
> running task. If you pause the dag run no new tasks will be scheduled.
> 
> B.
> 
> 
> 
> Verstuurd vanaf mijn iPad
> 
> > Op 12 apr. 2018 om 20:23 heeft Laura Lorenz  het 
> > volgende geschreven:
> > 
> > That won't stop them if they are already running in a celery worker or
> > already in your messaging queue backend (e.g. rabbitmq; redis), but it will
> > prevent the message to do them from being emitted again by the airflow
> > scheduler to your messaging queue backend. To be thorough you have to do
> > both - stop the scheduler from scheduling the tasks anymore (by failing
> > them individually and/or the DagRun in the metadata database) and, if you
> > want to make sure the tasks that already got picked up stop and don't try
> > again, you have to kill their worker processes and make sure your messaging
> > queue is clean of messages of that task type. If you don't care that any
> > already started or queued up tasks finish, you can simply doctor the
> > metadata database.
> > 
> > Laura
> > 
> > On Thu, Apr 12, 2018 at 12:40 PM, ramandu...@gmail.com  >> wrote:
> > 
> >> Thanks Laura,
> >> We are using the CeleryExecutor. Just wondering if marking the
> >> TaskInstances as failed in metadata store would also work.
> >> -Raman
> >> 
> >>> On 2018/04/12 16:27:00, Laura Lorenz  wrote:
> >>> I use the CeleryExecutor and have used a mix of `celery control` and
> >>> messaging queue purges to kill the running tasks and prevent them from
> >>> being picked up by workers again (respectively), and doctor the DagRun to
> >>> failed to stop the scheduler from repopulating the message. I think if
> >> you
> >>> are using the Local or Sequential Executor you'd have to kill the
> >> scheduler
> >>> process.
> >>> 
> >>> Laura
> >>> 
> >>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston 
> >>> wrote:
> >>> 
>  I don't think killing a currently running task is possible today.
>  
>  Of course you can pause it from the CLI or web UI so that future runs
> >> don't
>  get triggered, but it sounds like that's not what you're looking for.
>  
>  Best,
>  Taylor
>  
>  *Taylor Edmiston*
>  Blog  | Stack Overflow CV
>   | LinkedIn
>   | AngelList
>  
>  
>  
>  On Thu, Apr 12, 2018 at 11:26 AM, ramandu...@gmail.com <
>  ramandu...@gmail.com
> > wrote:
>  
> > Hi All,
> > We have a use case to cancel the already running DAG. So is there any
> > recommended way to do so.
> > 
> > Thanks,
> > Raman
> > 
>  
> >>> 
> >> 
> 

Re: Cancel a Running dag

[Prem Prakash Sharma]

I have a similar use case, to support that I have implemented on_kill methods 
for all the operators.

To cancel current run:
Get all the running send a kill signal, in on_kill make sure current run for 
the task terminates gracefully, after the kill signal set state to failed(state 
should be set manually as in case of retries the scheduler will pick the task 
again)
Get all queued tasks and set state to shutdown
Just to be safe scan for tasks with up_for_retry state and set the state to 
shutdown

Task instances have set_state method that can be used to change states 
externally
To get all tasks with given state use get_task_instances method of dagrun
You can do this in loop until dagrun fails to avoid any race conditions for 
setting task states. I had a case when scheduler updated the state after I have 
changed the state.

There might be simpler ways but I feel I shouldn’t modify airflow metastore 
directly when I can use set_state which internally does that for me and don’t 
wanna stop the scheduler as other dagruns will be affected. 

This was the best solution I could come up with. If there is a better approach 
please let me know.

Thanks and Regards,
Prem

On 2018/04/13 08:33:50, "Driesprong, Fokko"  wrote: 
> Like Bolke said, it has been fixed in master. One of the perquisites is
> support by the operator. For example, the Spark operator has implemented
> how to kill the Spark job on YARN, Local and Kubernetes. If you are running
> something else, you might want to check if this is implemented.
> 
> Implemented on_kill: https://github.com/apache/incubator-airflow/pull/3204
> An example of the on_kill:
> https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L485-L534
> 
> Cheers, Fokko
> 
> 2018-04-12 21:19 GMT+02:00 Bolke de Bruin :
> 
> > This is now fixed in master. Clearing tasks will now properly terminate a
> > running task. If you pause the dag run no new tasks will be scheduled.
> >
> > B.
> >
> >
> >
> > Verstuurd vanaf mijn iPad
> >
> > > Op 12 apr. 2018 om 20:23 heeft Laura Lorenz 
> > het volgende geschreven:
> > >
> > > That won't stop them if they are already running in a celery worker or
> > > already in your messaging queue backend (e.g. rabbitmq; redis), but it
> > will
> > > prevent the message to do them from being emitted again by the airflow
> > > scheduler to your messaging queue backend. To be thorough you have to do
> > > both - stop the scheduler from scheduling the tasks anymore (by failing
> > > them individually and/or the DagRun in the metadata database) and, if you
> > > want to make sure the tasks that already got picked up stop and don't try
> > > again, you have to kill their worker processes and make sure your
> > messaging
> > > queue is clean of messages of that task type. If you don't care that any
> > > already started or queued up tasks finish, you can simply doctor the
> > > metadata database.
> > >
> > > Laura
> > >
> > > On Thu, Apr 12, 2018 at 12:40 PM, ramandu...@gmail.com <
> > ramandu...@gmail.com
> > >> wrote:
> > >
> > >> Thanks Laura,
> > >> We are using the CeleryExecutor. Just wondering if marking the
> > >> TaskInstances as failed in metadata store would also work.
> > >> -Raman
> > >>
> > >>> On 2018/04/12 16:27:00, Laura Lorenz  wrote:
> > >>> I use the CeleryExecutor and have used a mix of `celery control` and
> > >>> messaging queue purges to kill the running tasks and prevent them from
> > >>> being picked up by workers again (respectively), and doctor the DagRun
> > to
> > >>> failed to stop the scheduler from repopulating the message. I think if
> > >> you
> > >>> are using the Local or Sequential Executor you'd have to kill the
> > >> scheduler
> > >>> process.
> > >>>
> > >>> Laura
> > >>>
> > >>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston  > >
> > >>> wrote:
> > >>>
> >  I don't think killing a currently running task is possible today.
> > 
> >  Of course you can pause it from the CLI or web UI so that future runs
> > >> don't
> >  get triggered, but it sounds like that's not what you're looking for.
> > 
> >  Best,
> >  Taylor
> > 
> >  *Taylor Edmiston*
> >  Blog  | Stack Overflow CV
> >   | LinkedIn
> >   | AngelList
> >  
> > 
> > 
> >  On Thu, Apr 12, 2018 at 11:26 AM, ramandu...@gmail.com <
> >  ramandu...@gmail.com
> > > wrote:
> > 
> > > Hi All,
> > > We have a use case to cancel the already running DAG. So is there any
> > > recommended way to do so.
> > >
> > > Thanks,
> > > Raman
> > >
> > 
> > >>>
> > >>
> >
> 

Re: Cancel a Running dag

[Driesprong, Fokko]

Like Bolke said, it has been fixed in master. One of the perquisites is
support by the operator. For example, the Spark operator has implemented
how to kill the Spark job on YARN, Local and Kubernetes. If you are running
something else, you might want to check if this is implemented.

Implemented on_kill: https://github.com/apache/incubator-airflow/pull/3204
An example of the on_kill:
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L485-L534

Cheers, Fokko

2018-04-12 21:19 GMT+02:00 Bolke de Bruin :

> This is now fixed in master. Clearing tasks will now properly terminate a
> running task. If you pause the dag run no new tasks will be scheduled.
>
> B.
>
>
>
> Verstuurd vanaf mijn iPad
>
> > Op 12 apr. 2018 om 20:23 heeft Laura Lorenz 
> het volgende geschreven:
> >
> > That won't stop them if they are already running in a celery worker or
> > already in your messaging queue backend (e.g. rabbitmq; redis), but it
> will
> > prevent the message to do them from being emitted again by the airflow
> > scheduler to your messaging queue backend. To be thorough you have to do
> > both - stop the scheduler from scheduling the tasks anymore (by failing
> > them individually and/or the DagRun in the metadata database) and, if you
> > want to make sure the tasks that already got picked up stop and don't try
> > again, you have to kill their worker processes and make sure your
> messaging
> > queue is clean of messages of that task type. If you don't care that any
> > already started or queued up tasks finish, you can simply doctor the
> > metadata database.
> >
> > Laura
> >
> > On Thu, Apr 12, 2018 at 12:40 PM, ramandu...@gmail.com <
> ramandu...@gmail.com
> >> wrote:
> >
> >> Thanks Laura,
> >> We are using the CeleryExecutor. Just wondering if marking the
> >> TaskInstances as failed in metadata store would also work.
> >> -Raman
> >>
> >>> On 2018/04/12 16:27:00, Laura Lorenz  wrote:
> >>> I use the CeleryExecutor and have used a mix of `celery control` and
> >>> messaging queue purges to kill the running tasks and prevent them from
> >>> being picked up by workers again (respectively), and doctor the DagRun
> to
> >>> failed to stop the scheduler from repopulating the message. I think if
> >> you
> >>> are using the Local or Sequential Executor you'd have to kill the
> >> scheduler
> >>> process.
> >>>
> >>> Laura
> >>>
> >>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston  >
> >>> wrote:
> >>>
>  I don't think killing a currently running task is possible today.
> 
>  Of course you can pause it from the CLI or web UI so that future runs
> >> don't
>  get triggered, but it sounds like that's not what you're looking for.
> 
>  Best,
>  Taylor
> 
>  *Taylor Edmiston*
>  Blog  | Stack Overflow CV
>   | LinkedIn
>   | AngelList
>  
> 
> 
>  On Thu, Apr 12, 2018 at 11:26 AM, ramandu...@gmail.com <
>  ramandu...@gmail.com
> > wrote:
> 
> > Hi All,
> > We have a use case to cancel the already running DAG. So is there any
> > recommended way to do so.
> >
> > Thanks,
> > Raman
> >
> 
> >>>
> >>
>