ldacey commented on pull request #16823:
URL: https://github.com/apache/airflow/pull/16823#issuecomment-981776479
This worked for me.
- extract.output can be used because the key is `return_value`
- I added a template_field called `input_files` in my custom
DatasetToDatasetOperator
- I used XcomArg to overwrite the key from `return_value` to `filter_list`
- The input_files argument reads the XComArg
```
transform = PrepareParquetOperator(
task_id="transform",
input_files=extract.output,
)
transformed_files = XComArg(transform, "filter_list")
finalize = DatasetToDatasetOperator(
task_id="finalize",
input_files=transformed_files,
)
extract >> transformed_files >> finalize
```
It seems like I have to use `transformed_files` as my upstream task instead
of `transform`. Everything works though.
It _would_ be nice if we could just overwrite the `return_value` key with
anything we want to without requiring intermediate tasks or XComArgs. I will
most likely just replace all of my custom keys with `return_value` instead
though in the database, and remove any of my custom `xcom.push` code in favor
of just returning results though.
Thanks @ephraimbuddy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
