bolkedebruin commented on PR #32319:
URL: https://github.com/apache/airflow/pull/32319#issuecomment-1826883406
Hey @Joffreybvn thanks for responding and apologies for being late to the
game and being a bit rough. It isn't required to return the same during
deserialization as is provided through serialization. In other words it is fine
if you serialize a `pyodbc.Row` and deserialize to a `NamedTuple`.
@potiuk All XCom serialization and deserialization goes through `serde.py`.
So if you put something into XCom and you want to have it returned it goes
through there. The original issue's stack trace said:
```
Traceback (most recent call last):
File "/opt/app-root/lib64/python3.9/site-packages/airflow/utils/json.py",
line 91, in default
return serialize(o)
File
"/opt/app-root/lib64/python3.9/site-packages/airflow/serialization/serde.py",
line 171, in serialize
raise TypeError(f"cannot serialize object of type {cls}")
TypeError: cannot serialize object of type <class 'pyodbc.Row'>
```
By dropping a serializer/deserializer into serializers for `pyodbc.Row` (and
one for DataBricks maybe) this could have been fixed. `_make_serializable`
seems to create an intermediate format at the moment which imho seems
unnecessary.
BUT if you intention is to have a **common** "flattened" format, that by
specification is serializable by the backend (not just JSON btw), then a
renaming of the method might make sense. Then maybe the solution is then
renaming the method and changing the description of the method
`_make_serializable`.
I realize that I might be overzealous and misreading due to all the
'serialize' here, but I did not see any consideration going into implementing
this in the framework that is available for this (`serde`). My question is can
this not be solved a lot simpler and without the need for a change to the
Operators. `serde` handles XCom for you. Why is the common `_make_serializable`
method required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
