That won't stop them if they are already running in a celery worker or already in your messaging queue backend (e.g. rabbitmq; redis), but it will prevent the message to do them from being emitted again by the airflow scheduler to your messaging queue backend. To be thorough you have to do both - stop the scheduler from scheduling the tasks anymore (by failing them individually and/or the DagRun in the metadata database) and, if you want to make sure the tasks that already got picked up stop and don't try again, you have to kill their worker processes and make sure your messaging queue is clean of messages of that task type. If you don't care that any already started or queued up tasks finish, you can simply doctor the metadata database.
Laura On Thu, Apr 12, 2018 at 12:40 PM, [email protected] <[email protected] > wrote: > Thanks Laura, > We are using the CeleryExecutor. Just wondering if marking the > TaskInstances as failed in metadata store would also work. > -Raman > > On 2018/04/12 16:27:00, Laura Lorenz <[email protected]> wrote: > > I use the CeleryExecutor and have used a mix of `celery control` and > > messaging queue purges to kill the running tasks and prevent them from > > being picked up by workers again (respectively), and doctor the DagRun to > > failed to stop the scheduler from repopulating the message. I think if > you > > are using the Local or Sequential Executor you'd have to kill the > scheduler > > process. > > > > Laura > > > > On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <[email protected]> > > wrote: > > > > > I don't think killing a currently running task is possible today. > > > > > > Of course you can pause it from the CLI or web UI so that future runs > don't > > > get triggered, but it sounds like that's not what you're looking for. > > > > > > Best, > > > Taylor > > > > > > *Taylor Edmiston* > > > Blog <http://blog.tedmiston.com> | Stack Overflow CV > > > <https://stackoverflow.com/story/taylor> | LinkedIn > > > <https://www.linkedin.com/in/tedmiston/> | AngelList > > > <https://angel.co/taylor> > > > > > > > > > On Thu, Apr 12, 2018 at 11:26 AM, [email protected] < > > > [email protected] > > > > wrote: > > > > > > > Hi All, > > > > We have a use case to cancel the already running DAG. So is there any > > > > recommended way to do so. > > > > > > > > Thanks, > > > > Raman > > > > > > > > > >
