vatsrahul1001 commented on issue #40103:
URL: https://github.com/apache/airflow/issues/40103#issuecomment-2162412999
Saying that I updated my DAG to use a xcom value which should return a dict,
however, still I am getting same parsing error
```
SPARK_CONFIGURATION_OVERRIDES = {
"monitoringConfiguration": {"s3MonitoringConfiguration": {"logUri":
f"s3://{bucket_name}/logs"}}
}
def test_xcom():
return SPARK_CONFIGURATION_OVERRIDES
@task()
def test_log():
print(f"data is {type(emr_serverless_app_id)}")
print(f"data is {str(emr_serverless_app_id)}")
return SPARK_CONFIGURATION_OVERRIDES
t1 = PythonOperator(
task_id="t1",
python_callable=test_xcom,
)
# [START howto_sensor_emr_serverless_application]
wait_for_app_creation = EmrServerlessApplicationSensor(
task_id="wait_for_app_creation",
application_id=emr_serverless_app_id,
)
# [END howto_sensor_emr_serverless_application]
wait_for_app_creation.poke_interval = 1
# [START howto_operator_emr_serverless_start_job]
start_job = EmrServerlessStartJobOperator(
task_id="start_emr_serverless_job",
application_id=emr_serverless_app_id,
execution_role_arn=role_arn,
job_driver=SPARK_JOB_DRIVER,
configuration_overrides=t1.output
)
```
Verified
[configuration_overrides](https://github.com/apache/airflow/blob/89b32e6363cdaa8665a9aa0843025c3084d3b291/airflow/providers/amazon/aws/operators/emr.py#L1252)
is a templated_field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
