jens-scheffler-bosch commented on issue #14338:
URL: https://github.com/apache/airflow/issues/14338#issuecomment-1251456102
I believe this is rather a technical limitation where documentation might
need to be improved or we need to have a warning raised in logs. Each task that
is being executed requires a worker slot. A call to a subdag also allocates one
worker slot until the execution of the subdag is completed.
I tried to re-produce the issue with the `example_subdag_operator.py` that
is shipped with the example DAGs in Airflow (2.4.0) and was not able to
re-produce when setting `concurrency=1` in the DAG. Alternatively also with the
non-deprecated option `max_active_tasks=1`.
My setup consisted out of LocalExecutor with a development setup of Airflow.
What I was able to re-produce was forcing the Airflow scheduler to run
`SequentialExecutor` which limits task processing to a single task. In this
case (irrespective of setting of `concurrency`) only one task slot is provided
as executor. In this case SubDagOperator can not be used because the run of the
subdag allocates one task slot and there is no worker slot possible to process
tasks from within the subdag.
To be able to work on this issue, can you please:
- Do a regression on a current Airflow release
- If you can re-produce also try it with the `example_subdag_operator.py`
- If you can re-produce in your environment using a current version, please
describe:
- Which Airflow-Version
- Which Environment
- Which Executor is used
- A copy of the DAG (or a neutralized DAG) which can be used to
re-produce in an external environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
