josh-fell commented on issue #10285:
URL: https://github.com/apache/airflow/issues/10285#issuecomment-870582068
Going through these example DAGs there are a number of `xcom_pull()` uses in
which values from the `XCom` are directly accessed via an index and/or key as
well as a `for loop` in Jinja. Trying to rewrite these instances using the new
syntax is awfully cumbersome IMO. For example:
```python
# Access by dict key
job_id="{{task_instance.xcom_pull('start_python_job_dataflow_runner_async')['dataflow_job_id']}}"
# Access by index
object_name="{{ task_instance.xcom_pull('upload_sheet_to_gcs')[0] }}"
# Using a for loop and accessing by dict key
echo_cmd = """
{% for m in task_instance.xcom_pull('pull_messages') %}
echo "AckID: {{ m.get('ackId') }}, Base64-Encoded: {{ m.get('message')
}}"
{% endfor %}
"""
```
The `output` property of operators doesn't seem to elegantly handle this
type of access currently. I can use a workaround with ugly hacks like these:
```python
# Dict key access workaround (index access would be written similarly)
start_python_job_dataflow_runner_async_output =
str(start_python_job_dataflow_runner_async.output).strip("{ }")
wait_for_python_job_dataflow_runner_async_done = DataflowJobStatusSensor(
task_id="wait-for-python-job-async-done",
job_id="{{{{
{start_python_job_dataflow_runner_async_output}['dataflow_job_id'] }}}}",
expected_statuses={DataflowJobStatus.JOB_STATE_DONE},
project_id=GCP_PROJECT_ID,
location='us-central1',
)
# For loop workaround
echo_cmd = f"""
{{% for m in {pull_messages_output} %}}
echo "AckID: {{{{ m.get('ackId') }}}}, Base64-Encoded: {{{{
m.get('message') }}}}"
{{% endfor %}}
"""
```
I logged #16618 as a feature request to enhance the use of `XComArgs` to
handle such access. I was wondering if you all thought it was worth
implementing these workarounds or to leave them as-is for now. What do you
think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
