tronlightracer opened a new issue #22626:
URL: https://github.com/apache/airflow/issues/22626
### Apache Airflow version
main (development)
### What happened
When using the command airflow dags test tasks are put in the deadlock state.
### What you think should happen instead
I think the airflow dags test shouldn't deadlock tasks.
### How to reproduce
from airflow.models import DAG
from airflow.decorators import task
from airflow.operators.python import PythonOperator, PythonVirtualenvOperator
from datetime import datetime, timedelta
def dynamic_importer(module1, module2, module3):
import importlib
for module in [module1, module2, module3]:
print(module)
importlib.import_module(module)
def dynamic_imports(imports: list):
from random import choices
return [choices(imports, k=3) for _ in range(10)]
the_imports = ["flask", "flask_restful", "wheel", "click", "numpy",
"cryptography", "pandas"]
with DAG(
dag_id="test_mapped_virtualenv",
start_date=datetime(1970, 1, 1),
schedule_interval=timedelta(days=30 * 365),
) as dag:
PythonVirtualenvOperator.partial(
task_id="venv_experiments",
python_callable=dynamic_importer,
requirements=the_imports,
system_site_packages=False,
).expand(op_args=dynamic_imports(the_imports))
[2022-03-30 09:15:32,872] {taskinstance.py:850} DEBUG - Setting task state
for <TaskInstance: test_mapped_virtualenv.venv_expirements
backfill__2022-03-30T09:15:31+00:00 [None]> to scheduled
[2022-03-30 09:15:32,885] {backfill_job.py:405} DEBUG - *** Clearing out
not_ready list ***
[2022-03-30 09:15:32,890] {taskinstance.py:759} DEBUG - Refreshing
TaskInstance <TaskInstance: test_mapped_virtualenv.venv_expirements
backfill__2022-03-30T09:15:31+00:00 [scheduled]> from DB
[2022-03-30 09:15:32,895] {backfill_job.py:418} DEBUG - Task instance to run
<TaskInstance: test_mapped_virtualenv.venv_expirements
backfill__2022-03-30T09:15:31+00:00 [scheduled]> state scheduled
[2022-03-30 09:15:32,896] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Task Instance State' PASSED: True, Task state
scheduled was valid.
[2022-03-30 09:15:32,896] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Trigger Rule' PASSED: True, The task instance did not
have any upstream tasks.
[2022-03-30 09:15:32,896] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Task Instance Not Running' PASSED: True, Task is not
in running state.
[2022-03-30 09:15:32,896] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Not In Retry Period' PASSED: True, The task instance
was not marked for retrying.
[2022-03-30 09:15:32,897] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Previous Dagrun State' PASSED: True, The context
specified that the state of past DAGs could be ignored.
[2022-03-30 09:15:32,897] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Task has been mapped' PASSED: False, The task has yet
to be mapped!
[2022-03-30 09:15:32,897] {taskinstance.py:1041} DEBUG - Dependencies not
met for <TaskInstance: test_mapped_virtualenv.venv_expirements
backfill__2022-03-30T09:15:31+00:00 [scheduled]>, dependency 'Task has been
mapped' FAILED: The task has yet to be mapped!
[2022-03-30 09:15:32,897] {backfill_job.py:536} DEBUG - Adding
<TaskInstance: test_mapped_virtualenv.venv_expirements
backfill__2022-03-30T09:15:31+00:00 [scheduled]> to not_ready
[2022-03-30 09:15:37,847] {base_job.py:226} DEBUG - [heartbeat]
[2022-03-30 09:15:37,847] {base_executor.py:156} DEBUG - 0 running task
instances
[2022-03-30 09:15:37,848] {base_executor.py:157} DEBUG - 0 in queue
[2022-03-30 09:15:37,848] {base_executor.py:158} DEBUG - 32 open slots
[2022-03-30 09:15:37,848] {base_executor.py:167} DEBUG - Calling the <class
'airflow.executors.debug_executor.DebugExecutor'> sync method
[2022-03-30 09:15:37,848] {backfill_job.py:596} WARNING - Deadlock
discovered for ti_status.to_run=dict_values([<TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]>])
[2022-03-30 09:15:37,853] {dagrun.py:628} DEBUG - number of tis tasks for
<DagRun test_mapped_virtualenv @ 2022-03-30T09:15:31+00:00:
backfill__2022-03-30T09:15:31+00:00, externally triggered: False>: 1 task(s)
[2022-03-30 09:15:37,853] {dagrun.py:644} DEBUG - number of scheduleable
tasks for <DagRun test_mapped_virtualenv @ 2022-03-30T09:15:31+00:00:
backfill__2022-03-30T09:15:31+00:00, externally triggered: False>: 0 task(s)
[2022-03-30 09:15:37,853] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Task has been mapped' PASSED: False, The task has yet
to be mapped!
[2022-03-30 09:15:37,854] {taskinstance.py:1041} DEBUG - Dependencies not
met for <TaskInstance: test_mapped_virtualenv.venv_expirements
backfill__2022-03-30T09:15:31+00:00 [scheduled]>, dependency 'Task has been
mapped' FAILED: The task has yet to be mapped!
[2022-03-30 09:15:37,854] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Not In Retry Period' PASSED: True, The context
specified that being in a retry period was permitted.
[2022-03-30 09:15:37,854] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Previous Dagrun State' PASSED: True, The task did not
have depends_on_past set.
[2022-03-30 09:15:37,854] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Trigger Rule' PASSED: True, The task instance did not
have any upstream tasks.
[2022-03-30 09:15:37,854] {dagrun.py:570} ERROR - Deadlock; marking run
<DagRun test_mapped_virtualenv @ 2022-03-30T09:15:31+00:00:
backfill__2022-03-30T09:15:31+00:00, externally triggered: False> failed
[2022-03-30 09:15:37,854] {dagrun.py:594} INFO - DagRun Finished:
dag_id=test_mapped_virtualenv, execution_date=2022-03-30T09:15:31+00:00,
run_id=backfill__2022-03-30T09:15:31+00:00, run_start_date=2022-03-30
15:15:32.853718+00:00, run_end_date=2022-03-30 15:15:37.854580+00:00,
run_duration=5.000862, state=failed, external_trigger=False, run_type=backfill,
data_interval_start=2022-03-30T09:15:31+00:00,
data_interval_end=2052-03-22T09:15:31+00:00, dag_hash=None
[2022-03-30 09:15:37,854] {backfill_job.py:362} INFO - [backfill progress] |
finished run 1 of 1 | tasks waiting: 0 | succeeded: 0 | running: 0 | failed: 0
| skipped: 0 | deadlocked: 1 | not ready: 1
[2022-03-30 09:15:37,855] {backfill_job.py:376} DEBUG - Finished dag run
loop iteration. Remaining tasks dict_values([])
[2022-03-30 09:15:37,858] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Task has been mapped' PASSED: False, The task has yet
to be mapped!
[2022-03-30 09:15:37,858] {taskinstance.py:1041} DEBUG - Dependencies not
met for <TaskInstance: test_mapped_virtualenv.venv_expirements
backfill__2022-03-30T09:15:31+00:00 [scheduled]>, dependency 'Task has been
mapped' FAILED: The task has yet to be mapped!
[2022-03-30 09:15:37,861] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Not In Retry Period' PASSED: True, The task instance
was not marked for retrying.
[2022-03-30 09:15:37,861] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Previous Dagrun State' PASSED: True, The task did not
have depends_on_past set.
[2022-03-30 09:15:37,861] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Trigger Rule' PASSED: True, The task instance did not
have any upstream tasks.
[2022-03-30 09:15:37,861] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Task has been mapped' PASSED: False, The task has yet
to be mapped!
[2022-03-30 09:15:37,862] {taskinstance.py:1041} DEBUG - Dependencies not
met for <TaskInstance: test_mapped_virtualenv.venv_expirements
backfill__2022-03-30T09:15:31+00:00 [scheduled]>, dependency 'Task has been
mapped' FAILED: The task has yet to be mapped!
[2022-03-30 09:15:37,864] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Not In Retry Period' PASSED: True, The task instance
was not marked for retrying.
[2022-03-30 09:15:37,864] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Previous Dagrun State' PASSED: True, The context
specified that the state of past DAGs could be ignored.
[2022-03-30 09:15:37,864] {taskinstance.py:1061} DEBUG - <TaskInstance:
test_mapped_virtualenv.venv_expirements backfill__2022-03-30T09:15:31+00:00
[scheduled]> dependency 'Trigger Rule' PASSED: True, The task instance did not
have any upstream tasks.
BackfillJob is deadlocked.
These tasks have succeeded:
DAG ID Task ID Run ID Try number
-------- --------- -------- ------------
These tasks are running:
DAG ID Task ID Run ID Try number
-------- --------- -------- ------------
These tasks have failed:
DAG ID Task ID Run ID Try number
-------- --------- -------- ------------
These tasks are skipped:
DAG ID Task ID Run ID Try number
-------- --------- -------- ------------
These tasks are deadlocked:
DAG ID Task ID Run ID
Try number
---------------------- ----------------
----------------------------------- ------------
test_mapped_virtualenv venv_expirements
backfill__2022-03-30T09:15:31+00:00 1
[2022-03-30 09:15:37,870] {cli_action_loggers.py:84} DEBUG - Calling
callbacks: []
[2022-03-30 09:15:37,870] {settings.py:383} DEBUG - Disposing DB connection
pool (PID 221785)
### Operating System
Ubuntu Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
tested with airflow breeze and a local virtualenv with airflow 2.3.0
installed.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
