sdaberdaku opened a new issue, #75762:
URL: https://github.com/apache/airflow/issues/75762
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.2
### What happened and how to reproduce it?
`@task.pyspark` does not render Jinja in `config_kwargs` (or `conn_id`). The
same arguments passed to `PySparkOperator` are rendered.
`PySparkOperator` declares both as template fields:
```
PySparkOperator.template_fields == ("conn_id", "config_kwargs",
"templates_dict", "op_args", "op_kwargs")
```
`_PySparkDecoratedOperator` does not define `template_fields`.
`DecoratedOperator` comes before `PySparkOperator` in its MRO, so it inherits
`DecoratedOperator`'s fields and drops the two Spark-specific ones:
```
_PySparkDecoratedOperator.template_fields == ("op_args", "op_kwargs")
```
Templated values in `config_kwargs` therefore reach Spark as literal text
when the decorator is used.
Reproduction:
```python
from airflow.providers.apache.spark.operators.spark_pyspark import
PySparkOperator
from airflow.sdk import DAG, task
conf = {"spark.app.name": "app-{{ run_id }}"}
with DAG("repro", schedule=None):
@task.pyspark(config_kwargs=conf)
def decorated(spark=None):
pass
decorated_op = decorated().operator
classic_op = PySparkOperator(task_id="classic", python_callable=lambda
spark=None: None, config_kwargs=conf)
context = {"run_id": "manual__1"}
for op in (decorated_op, classic_op):
op.render_template_fields(context)
print(type(op).__name__, op.config_kwargs["spark.app.name"])
```
Output:
```
_PySparkDecoratedOperator app-{{ run_id }}
PySparkOperator app-manual__1
```
### What you think should happen instead?
`@task.pyspark` should template `conn_id` and `config_kwargs` exactly like
`PySparkOperator`. The decorated operator can declare the union of both sets of
fields, the way `_KubernetesDecoratedOperator` keeps the fields of
`KubernetesPodOperator`:
```python
template_fields: Sequence[str] = (*DecoratedOperator.template_fields,
"conn_id", "config_kwargs")
```
### Operating System
Linux
### Deployment
Virtualenv installation
### Apache Airflow Provider(s)
apache-spark
### Versions of Apache Airflow Providers
apache-airflow-providers-apache-spark==6.3.4
apache-airflow-providers-common-compat==1.23.0
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
The decorator never templated these fields: before `PySparkOperator` was
introduced it declared `template_fields = ("op_args", "op_kwargs")` explicitly.
The operator refactor made the inconsistency visible, since the same arguments
now behave differently on the two entry points.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
