Changing the state of a dag run in the UI will only stop the scheduler from
evaluating that dag run, it does not sigterm or anything like that.
"Clearing" running task instances (UI/CLI) will result in failing the task
instances. Also as of 1.8.x if the state of the task in changed externally
from
I have a large DAG (32 tasks) with concurrency=2 and max_active_runs=1.
Most of the tasks also use a redshift_pool, and this is running the
LocalExecutor on 1.8.0RC4.
When the DAG kicks off things seem to generally function, but a few of the
tasks get moved to queued status (appropriately) but
Thanks for the share, Max! I really enjoyed working on and co-publishing
the post. I know everyone is quite busy with the 1.8 release, so I
especially appreciate taking the time to spread the word and think about
the big picture.
I'm happy to field any questions on the post or Astronomer. You
Jointly publishing this short Q with Taylor D. Edmiston of Atronomer.io
https://medium.com/the-astronomer-journey/airflow-and-the-future-of-data-engineering-a-q-a-266f68d956a9#.cpast6vjd
Hi folks,
We have been using Airflow to run ETL jobs but sometimes require to cancel a
DAG run. Since I could not find a way to cancel a DAG run (from CLI or UI), we
are using a crude work-around that is to cancel the tasks that are running and
thus failing the DAG run.
What I expect (and I
Here's a follow-up with more information.
I think my suspicion is supported by the Airflow database. It has the task
marked as failed in the task_instance table, and the PID matches the dead job:
airflow=> select job_id, task_id, dag_id, state, pid from task_instance where
job_id=127022;
Hi folks,
We're experiencing an issue where a failing DAG is not properly terminated. The
log file contains a Python stack trace of the failure and then ends with:
{base_task_runner.py:112} INFO - Running: ['bash', '-c', 'airflow run
my_dag my_task 2017-02-22T00:00:00 --job_id 127022
Gotcha:
It works, but then slightly different.
If you added the subdag, do not zoom in, but click on the subdag in the main
dag. Use mark success there. It will then allow you to mark all tasks
successful that are part of the subdag.
Do we still consider this a blocker? Imho, no as a
