potiuk commented on issue #40974: URL: https://github.com/apache/airflow/issues/40974#issuecomment-2252459946
Also (to add to what @bolkedebruin wrote ) in internal-API we used Pydantic for some of that - mostly because it alows to serialize ORM SQLalchemy objects together with their relations. But this is possible to plug-in in serde as well as another serializers and treat it similarly to "to_dict" or "to_json".. Also I think there was a lot of misunderstandings and clashes (and I am one of the guilty ones there) about which and why and how we are using serializers - precisely because we have not agreed on a common strategy we want to take, direction and vision we have. It has never been agreed and writen down, and I think we should start with that. For example right now when I look at `serde` - it's a really good idea and if we extend the concept of it to be able to provide seriaizers from respective providers and make it support all that we need in "core" airflow, that could be a good idea for example (but we had disagreements on that in the past - again mostly because we had no consolidated vision and because in many cases it was just easier and faster (and without the whole picture and vision set) to use another approach than common. And possibly that was even good because that would in some cases break compatibility too much for some Airlfow 2 cases. But with Airflow 3, I'd be personally for making `serde` default and incorporate the 'DAG' serializer (after playing a bit with both) and making it the default `engine` and plugging in whatever serializer implementations are good. That should of course include all the compatibility and migration pieces as needed. We could also possibly to distribute the current serde modules (deltalake, iceberg etc - as "pluggable" / externak serializers in providers). That would be my 1c . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
