udsy19 opened a new pull request, #75918:
URL: https://github.com/apache/airflow/pull/75918
## Why
`process_params` (`task-sdk/src/airflow/sdk/definitions/param.py`) builds a
fresh `ParamsDict` and merges the Dag's and the task's params into it:
```python
params = ParamsDict(suppress_exception=suppress_exception)
with contextlib.suppress(AttributeError):
params.update(dag.params)
if task.params:
params.update(task.params)
```
`ParamsDict.update()` with a `ParamsDict` argument stores the same `Param`
objects rather than copies. Applying the run's conf afterwards calls
`Param.resolve(value=...)`, which sets `self.value` **on those shared
objects**, so the run's conf overwrites the defaults held by `dag.params` and
`task.params` themselves.
This is visible whenever the same Dag/task objects are reused in one
process. For example, `dag.test()` runs tasks in-process on
`self.task_dict[...]`, so a second `dag.test()` call without conf (common in a
pytest suite) sees the first call's conf values instead of the declared
defaults:
```python
dag = ...(params=ParamsDict({"x": Param(1, type="integer")}))
task = ...(params=ParamsDict({"y": Param("a", type="string")}))
process_params(dag, task, {"x": 5, "y": "b"}, suppress_exception=False) #
{'x': 5, 'y': 'b'}
process_params(dag, task, None, suppress_exception=False) #
{'x': 5, 'y': 'b'} (expected {'x': 1, 'y': 'a'})
dag.params.dump(), task.params.dump() #
({'x': 5}, {'y': 'b'})
```
## Fix
Merge deep copies, so resolving the run's conf only touches the per-run
`ParamsDict`. This matches how `BaseOperator` already merges Dag params
(`copy.deepcopy(dag.params)` in `bases/operator.py`). Override precedence and
validation are unchanged.
## Tests
Adds `TestProcessParams.test_dagrun_conf_does_not_mutate_dag_or_task_params`
to `task-sdk/tests/task_sdk/definitions/test_param.py`. It checks that applying
a conf leaves `dag.params`/`task.params` at their defaults and that a later
conf-less call returns the defaults. Verified fail-on-revert: on `main` it
fails with `{'dag_param': 2} != {'dag_param': 1}`; with the fix it passes, as
do the existing `TestParamsDict` tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
