ashb commented on pull request #16823:
URL: https://github.com/apache/airflow/pull/16823#issuecomment-875051253
> Then Xcoms pushed will be
>
> ```
> key=mykey, value=[3,4,5,6]
> key=return_value, value={'mykey':[3,4,56]}
> ```
FWIW that feels like a bug itself -- it shouldn't show up "twice" in xcom,
which is especially important for custom xcom backends and larger data (i.e.
DFs)
> One thing that one can do if he's able to control the xcom_key:
>
> In overriding the `orm_deserialize_value` we can do something like:
>
> ```
> def orm_deserialize_value(self):
> if self.key.starts_with('df'):
> return 'Xcom uploaded to GCS'
> return BaseXCom.deserialize_value(self)
> ```
I need convincing that a) this is the right thing to trigger this behaviour
one, and b) that the interface you have proposed is right for users.
For instance:
```python
@task
def my_task():
return some_large_df()
```
That is very likely to be error prone to and hard to spot the error for
anyone not intimately familiar with XComs -- the opposite direction we want to
head in :)
So lets take an big step back: what feature/functionality are you trying to
achieve?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
