sdaberdaku opened a new pull request, #75764:
URL: https://github.com/apache/airflow/pull/75764
`@task.pyspark` did not render Jinja in `config_kwargs` and `conn_id`, while
`PySparkOperator` renders both.
## Why
`_PySparkDecoratedOperator` does not define `template_fields`.
`DecoratedOperator` comes before `PySparkOperator` in its MRO, so the decorated
operator inherited only `("op_args", "op_kwargs")` and dropped the two
Spark-specific fields that `PySparkOperator` declares. Templated values in
`config_kwargs` therefore reached Spark as literal text on the decorator path.
## What changes
- `_PySparkDecoratedOperator.template_fields =
(*DecoratedOperator.template_fields, "conn_id", "config_kwargs")`, the same
pattern `_KubernetesDecoratedOperator` uses to keep the fields of
`KubernetesPodOperator`.
- `test_pyspark_decorator_renders_config_kwargs`: a task with
`config_kwargs={"spark.app.name": "app-{{ task.task_id }}"}` ends up with
`spark.app.name == "app-f"` in the Spark conf.
## Tests
The reproduction from #75762 run against `apache-airflow==3.3.2` and
`apache-airflow-providers-apache-spark==6.3.4` with this change applied prints
the rendered value for both the decorator and the operator. `ruff check` and
`ruff format --check` pass on both changed files. I did not run the provider
test suite locally with Breeze.
closes: #75762
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Fable 5.1, Opus 5.5)
Generated-by: Claude Code (Fable 5.1, Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
