Thanks Bolke, Will it become part of Airflow 1.10 release. Is there any tentative timeline for same. -Raman Gupta
On 2018/04/12 19:19:07, Bolke de Bruin <bdbr...@gmail.com> wrote: > This is now fixed in master. Clearing tasks will now properly terminate a > running task. If you pause the dag run no new tasks will be scheduled. > > B. > > > > Verstuurd vanaf mijn iPad > > > Op 12 apr. 2018 om 20:23 heeft Laura Lorenz <llor...@industrydive.com> het > > volgende geschreven: > > > > That won't stop them if they are already running in a celery worker or > > already in your messaging queue backend (e.g. rabbitmq; redis), but it will > > prevent the message to do them from being emitted again by the airflow > > scheduler to your messaging queue backend. To be thorough you have to do > > both - stop the scheduler from scheduling the tasks anymore (by failing > > them individually and/or the DagRun in the metadata database) and, if you > > want to make sure the tasks that already got picked up stop and don't try > > again, you have to kill their worker processes and make sure your messaging > > queue is clean of messages of that task type. If you don't care that any > > already started or queued up tasks finish, you can simply doctor the > > metadata database. > > > > Laura > > > > On Thu, Apr 12, 2018 at 12:40 PM, ramandu...@gmail.com <ramandu...@gmail.com > >> wrote: > > > >> Thanks Laura, > >> We are using the CeleryExecutor. Just wondering if marking the > >> TaskInstances as failed in metadata store would also work. > >> -Raman > >> > >>> On 2018/04/12 16:27:00, Laura Lorenz <llor...@industrydive.com> wrote: > >>> I use the CeleryExecutor and have used a mix of `celery control` and > >>> messaging queue purges to kill the running tasks and prevent them from > >>> being picked up by workers again (respectively), and doctor the DagRun to > >>> failed to stop the scheduler from repopulating the message. I think if > >> you > >>> are using the Local or Sequential Executor you'd have to kill the > >> scheduler > >>> process. > >>> > >>> Laura > >>> > >>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <tedmis...@gmail.com> > >>> wrote: > >>> > >>>> I don't think killing a currently running task is possible today. > >>>> > >>>> Of course you can pause it from the CLI or web UI so that future runs > >> don't > >>>> get triggered, but it sounds like that's not what you're looking for. > >>>> > >>>> Best, > >>>> Taylor > >>>> > >>>> *Taylor Edmiston* > >>>> Blog <http://blog.tedmiston.com> | Stack Overflow CV > >>>> <https://stackoverflow.com/story/taylor> | LinkedIn > >>>> <https://www.linkedin.com/in/tedmiston/> | AngelList > >>>> <https://angel.co/taylor> > >>>> > >>>> > >>>> On Thu, Apr 12, 2018 at 11:26 AM, ramandu...@gmail.com < > >>>> ramandu...@gmail.com > >>>>> wrote: > >>>> > >>>>> Hi All, > >>>>> We have a use case to cancel the already running DAG. So is there any > >>>>> recommended way to do so. > >>>>> > >>>>> Thanks, > >>>>> Raman > >>>>> > >>>> > >>> > >> >
