roldugin opened a new issue #12959: URL: https://github.com/apache/airflow/issues/12959
**Apache Airflow version**: 1.10.12 (regression, *NOT* reproducible on 1.10.5) **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): N/A **Environment**: local with SequentialExecutor + deployed with CeleryExecutor on Kubernetes - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): macOS 10.15 - **Kernel** (e.g. `uname -a`): Darwin macmac 19.6.0 Darwin Kernel Version 19.6.0: Sun Jul 5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64 x86_64 - **Install tools**: - **Others**: **What happened**: Clearing the `SKIPPED` "dummy" task in the following DAG deadlocks it and returns it to the `SKIPPED` state. <img width="298" alt="dag" src="https://user-images.githubusercontent.com/749698/101660840-e8ef0680-3a47-11eb-81e4-689c27a62113.png";> **What you expected to happen**: I expect the "dummy" task to run and succeed. This is the behaviour in Airflow 1.10.5 and before. **How to reproduce it**: ```python from datetime import datetime import airflow from airflow.operators.dummy_operator import DummyOperator from airflow.operators.python_operator import ShortCircuitOperator with airflow.DAG( dag_id='test_dag', start_date=datetime(2020, 1, 1), schedule_interval="@monthly", catchup=False, ) as dag: short_circuit_returning_false = ShortCircuitOperator( task_id='short_circuit_returning_false', python_callable=lambda **kwargs: False) dummy = DummyOperator(task_id="dummy") short_circuit_returning_false >> dummy ``` ``` airflow backfill test_dag -s 2020-01-01 -e 2020-01-01 # Backfill the DAG - dummy task will be skipped as expected airflow clear test_dag -t dummy -s 2020-01-01 -e 2020-01-01 # Crear the dummy task airflow backfill test_dag -s 2020-01-01 -e 2020-01-01 # Backfill the DAG again - dummy task deadlocks ``` **Detailed logs**: <details><summary>deadlock.log</summary> $ airflow backfill test_dag -s 2020-01-01 -e 2020-01-01 [2020-12-09 17:36:03,580] {__init__.py:50} INFO - Using executor SequentialExecutor [2020-12-09 17:36:03,580] {dagbag.py:417} INFO - Filling up the DagBag from /Users/grol/dev/sc-airflow-dags/dags [2020-12-09 17:36:21,409] {base_executor.py:58} INFO - Adding to queue: ['airflow', 'run', 'test_dag', 'short_circuit_returning_false', '2020-01-01T00:00:00+00:00', '--local', '--pool', 'default_pool', '-sd', '/Users/grol/dev/sc-airflow-dags/dags/rnr/test_dag.py', '--cfg_path', '/var/folders/1z/s9zsm2hx4sj5c674rvkn6q9c0000gp/T/tmppn6x5bv3'] [2020-12-09 17:36:26,342] {sequential_executor.py:54} INFO - Executing command: ['airflow', 'run', 'test_dag', 'short_circuit_returning_false', '2020-01-01T00:00:00+00:00', '--local', '--pool', 'default_pool', '-sd', '/Users/grol/dev/sc-airflow-dags/dags/rnr/test_dag.py', '--cfg_path', '/var/folders/1z/s9zsm2hx4sj5c674rvkn6q9c0000gp/T/tmppn6x5bv3'] [2020-12-09 17:36:28,616] {__init__.py:50} INFO - Using executor SequentialExecutor [2020-12-09 17:36:28,617] {dagbag.py:417} INFO - Filling up the DagBag from /Users/grol/dev/sc-airflow-dags/dags/rnr/test_dag.py Running <TaskInstance: test_dag.short_circuit_returning_false 2020-01-01T00:00:00+00:00 [queued]> on host macmac [2020-12-09 17:36:33,847] {dagrun.py:320} INFO - Marking run <DagRun test_dag @ 2020-01-01 00:00:00+00:00: backfill_2020-01-01T00:00:00+00:00, externally triggered: False> successful [2020-12-09 17:36:33,852] {backfill_job.py:359} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 1 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 1 [2020-12-09 17:36:33,864] {backfill_job.py:359} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 1 | running: 0 | failed: 0 | skipped: 1 | deadlocked: 0 | not ready: 0 [2020-12-09 17:36:33,864] {backfill_job.py:813} INFO - Backfill done. Exiting. $ airflow clear test_dag -t dummy -s 2020-01-01 -e 2020-01-01 [2020-12-09 17:36:45,965] {__init__.py:50} INFO - Using executor SequentialExecutor [2020-12-09 17:36:45,965] {dagbag.py:417} INFO - Filling up the DagBag from /Users/grol/dev/sc-airflow-dags/dags You are about to delete these 1 tasks: <TaskInstance: test_dag.dummy 2020-01-01 00:00:00+00:00 [skipped]> Are you sure? (yes/no): yes $ airflow backfill test_dag -s 2020-01-01 -e 2020-01-01 [2020-12-09 17:37:14,115] {__init__.py:50} INFO - Using executor SequentialExecutor [2020-12-09 17:37:14,116] {dagbag.py:417} INFO - Filling up the DagBag from /Users/grol/dev/sc-airflow-dags/dags [2020-12-09 17:37:34,878] {backfill_job.py:600} WARNING - Deadlock discovered for ti_status.to_run=odict_values([<TaskInstance: test_dag.dummy 2020-01-01 00:00:00+00:00 [skipped]>]) [2020-12-09 17:37:34,881] {dagrun.py:320} INFO - Marking run <DagRun test_dag @ 2020-01-01 00:00:00+00:00: backfill_2020-01-01T00:00:00+00:00, externally triggered: False> successful [2020-12-09 17:37:34,884] {backfill_job.py:359} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 1 | not ready: 1 Traceback (most recent call last): File "/Users/grol/dev/airflow-dev/venv/bin/airflow", line 37, in <module> args.func(args) File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 76, in wrapper return f(*args, **kwargs) File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/bin/cli.py", line 222, in backfill dag.run( File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/models/dag.py", line 1432, in run job.run() File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 218, in run self._execute() File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/jobs/backfill_job.py", line 794, in _execute raise AirflowException(err) airflow.exceptions.AirflowException: BackfillJob is deadlocked. These tasks have succeeded: DAG ID Task ID Execution date Try number -------- ----------------------------- ------------------------- ------------ test_dag short_circuit_returning_false 2020-01-01 00:00:00+00:00 5 These tasks are running: DAG ID Task ID Execution date Try number -------- --------- ---------------- ------------ These tasks have failed: DAG ID Task ID Execution date Try number -------- --------- ---------------- ------------ These tasks are skipped: DAG ID Task ID Execution date Try number -------- --------- ---------------- ------------ These tasks are deadlocked: DAG ID Task ID Execution date Try number -------- --------- ------------------------- ------------ test_dag dummy 2020-01-01 00:00:00+00:00 1 </details> ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/749698/101660840-e8ef0680-3a47-11eb-81e4-689c27a62113.png"](https://user-images.githubusercontent.com/749698/101660840-e8ef0680-3a47-11eb-81e4-689c27a62113.png"){kind=link}