Taragolis commented on PR #35648:
URL: https://github.com/apache/airflow/pull/35648#issuecomment-1812014159
My initial point was, that this operator won't work well with all cases of
templated fields around of `name`, so better fix it now and test, rather then
keep of "one day in the future"
Example when this operator would fail during initialisation
```python
import pendulum
from airflow.decorators import task
from airflow.models.dag import DAG
from airflow.providers.amazon.aws.operators.emr import
EmrServerlessStartJobOperator
SPARK_JOB_DRIVER = {
"sparkSubmit": {
"entryPoint": "spam",
"entryPointArguments": ["s3://spam/egg/output"],
"sparkSubmitParameters": "--conf spark.executor.cores=1 --conf
spark.executor.memory=4g\
--conf spark.driver.cores=1 --conf spark.driver.memory=4g --conf
spark.executor.instances=1",
}
}
SPARK_CONFIGURATION_OVERRIDES = {
"monitoringConfiguration": {"s3MonitoringConfiguration": {"logUri":
"s3://spam/egg/logs"}}
}
with DAG(
"pr_35648",
start_date=pendulum.datetime(2023, 6, 1, tz="UTC"),
schedule=None,
catchup=False,
tags=["pr", "35648"]
):
@task
def emr_config():
return {
"name": "new_application"
}
config = emr_config()
EmrServerlessStartJobOperator(
task_id="create_emr_serverless_task",
application_id="foo",
aws_conn_id=None,
release_label="emr-6.6.0",
job_type="SPARK",
config=config,
configuration_overrides={},
execution_role_arn="foo-bar",
job_driver=SPARK_JOB_DRIVER
)
```
```console
...
self.name = name or self.config.pop("name",
f"emr_serverless_job_airflow_{uuid4()}")
AttributeError: 'PlainXComArg' object has no attribute 'pop'
```
I have no objection to add only `name` as templated field and keep it for
"better time" in the future. But I think it is a good time to fix this
behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
