casassg opened a new issue #8059: [AIP-31] Enable customized XCom class URL: https://github.com/apache/airflow/issues/8059 **Description** - Enable custom XCom backend functionality for custom serialization/deserialization. - XCom backend can be set from airflow.cfg and defaults to existing `airflow.models.XCom`. **Use case / motivation** This comes from a discussion with @turbaszek . We were talking that it would be nice to have a flexible XCom backend that allowed us to serialize/deserialize XCom values using custom logic. A possible extension here would be to enable large objects to be serialized into a custom file system while XCom would only include a pointer (and metadata) of the object in the file system. An example use case would be data processing: ``` @task def process_data(input: pd.DataFrame) -> pd.DataFrame # make some magic ``` Where DataFrame are saved between tasks as `csv`. This is probably a stretch and maybe out of scope for this AIP (or require a new AIP). But wanted to put this out there.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
