LudvigLundberg opened a new issue, #37357:
URL: https://github.com/apache/airflow/issues/37357
### Apache Airflow Provider(s)
apache-livy
### Versions of Apache Airflow Providers
apache-airflow-providers-apache-livy==3.7.1
### Apache Airflow version
2.8.1
### Operating System
macOS Sonoma 14.0 (23A344)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
Template rendering for the livy operator doesn*t work (tested with `args`
and `files`), nothing is rendered in the rendered template tab in the task
instance view and the un-rendered string is sent to livy, e.g.:
`Submitting job {"file": "/spark-examples.jar", "className":
"org.apache.spark.examples.SparkPi", "args": ["{{ task.task_id }}"],
"numExecutors": 1, "conf": {"spark.shuffle.compress": "false"}}`
I verified that template rendering is working in other DAGS by running the
included `example_bash_operator` DAG
### What you think should happen instead
When downgrading the version to
`apache-airflow-providers-apache-livy==3.7.0`, rendered template tab is filled
with values and the rendered string is sent to livy:
`Submitting job {"file": "/spark-examples.jar", "className":
"org.apache.spark.examples.SparkPi", "args": ["pi_java_task"], "numExecutors":
1, "conf": {"spark.shuffle.compress": "false"}}`
### How to reproduce
Following the instructions from
https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html
using docker compose and building the following Dockerfile:
```dockerfile
FROM apache/airflow:2.8.1
ADD requirements.txt .
RUN pip install apache-airflow==2.8.1 -r requirements.txt
```
A custom requirements.txt:
```
apache-airflow-providers-apache-livy==3.7.1
```
Running the following example DAG:
```python
from datetime import datetime
from airflow import DAG
from airflow.providers.apache.livy.operators.livy import LivyOperator
with DAG(
dag_id='example_livy_operator',
default_args={'args': [10]},
schedule_interval='@daily',
start_date=datetime(2021, 1, 1),
catchup=False,
) as dag:
# [START create_livy]
livy_java_task = LivyOperator(
task_id="pi_java_task",
file='/spark-examples.jar',
args=["{{ task.task_id }}"],
num_executors=1,
conf={
'spark.shuffle.compress': 'false',
},
class_name='org.apache.spark.examples.SparkPi',
)
```
To get template rendering working change the requirements.txt to the
following and rebuild/deploy airflow:
```
apache-airflow-providers-apache-livy==3.7.0
```
### Anything else
Seems the issue was introduced in
https://github.com/apache/airflow/pull/36490 to fix
https://github.com/apache/airflow/issues/36484
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
