SameerMesiah97 commented on code in PR #69768:
URL: https://github.com/apache/airflow/pull/69768#discussion_r3609057346
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -161,6 +162,30 @@ def _validate_executor_fields(dag: DAG, bundle_name: str |
None = None) -> None:
)
+def _assign_default_team_pools(
+ dag: DAG,
+ bundle_name: str | None = None,
+) -> None:
+ """Assign the default team pool to tasks that do not explicitly specify a
pool."""
+ dag_team_name = None
+
+ if conf.getboolean("core", "multi_team"):
+ if bundle_name:
+ from airflow.dag_processing.bundles.manager import
DagBundlesManager
+
+ bundle_manager = DagBundlesManager()
+ bundle_config = bundle_manager._bundle_config[bundle_name]
+
+ dag_team_name = bundle_config.team_name
+
+ if not dag_team_name:
+ return
+
+ for task in dag.tasks:
+ if task.pool == Pool.DEFAULT_POOL_NAME:
+ task.pool = Pool.get_default_team_pool_name(dag_team_name)
Review Comment:
@justinpakzad @stephen-bracken
1) I have extended the implementation reconcile the pools for each team when
the `teams sync` command is executed. This should handle the scenario you
mentioned. I have also handled the teams `delete` path as well when in my
latest commit.
2) I agree this PR changes user-facing semantics with regards to the default
pool for a certain task. My view is that 'default_pool' is better treated as a
sentinel for "the default pool" rather than a guarantee that tasks always
execute in a single global pool. In a multi-team deployment, the natural
default is the team's default pool, while users who require a shared pool can
continue to specify one explicitly
3) `pool` is initialized to "default_pool" on `SerializedBaseOperator`, so
by the time we process the DAG we no longer know whether the user omitted pool
or explicitly set `pool="default_pool"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
