michelgoldstein opened a new pull request #4615: (For Vik/Duy) Switching XCom serialization logic (re-do on 1.10.x branch) URL: https://github.com/apache/airflow/pull/4615 This is a redo of https://github.com/SiftScience/airflow/pull/1 based on the 1.10.x branch (v1-10-stable) **Purpose:** - https://github.com/SiftScience/code/issues/31076 - XCom data serialization works fine with all API calls except for when writing from the Admin UI. This should fix it (and, in certain ways, clean up the serialization logic in the SQLAlchemy model) **Technical overview:** - The SQLAlchemy model says that the "data" field in the `xcom` table is a binary field. It is populated with either a pickled version of an object (legacy mode, but the one we use), or an UTF-8-encoded json string. - The way this translation happens is by making sure that all the set and get operations go through some transformation function. And that works for most use cases, except for the Admin UI, which uses `flask-admin` and goes directly from the form to the SQLAlchemy model. That means that it tries to write a string into a binary field. - The actual solution is to not call it a binary field, but some type we control and then provide a serialization/deserialization in the type format itself. This means that we don't have to deal with doing serialization/deserialization in the getters/setters, which have also been removed. - Also bumped the version to `1.10.2-siftpatch1` which will be used for deployment **Testing/deployment plan:** - Tested with our docker setup and a local PyPI repository and all XCom behavior worked (create, update on UI, push and pull from python) - Confirmed that it works with a PyPI repository. Just need to set up one (that is not my laptop). - Tried to run the unit tests, but was unable to do it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
