Alwaysgaurav1 opened a new pull request, #69432:
URL: https://github.com/apache/airflow/pull/69432
### PR Title
feat(task-sdk): Move XCom serialization and deserialization to Task SDK
### PR Description
#### What does this PR do?
This PR moves the responsibility of XCom serialization and deserialization
from the Airflow API Server to the language-specific client-side **Task SDK**
as part of **AIP-72 (Task Execution Interface)**.
Under this new architecture:
1. The **Task SDK** client-side (Python/Go/Java) handles serializing native
Python objects directly into raw JSON-formatted string literals before sending
them to the API Server via `SetXCom`.
2. The **API Server** now treats XCom values as opaque JSON string literals,
saving them directly to the database without parsing or re-serializing them.
3. Upon retrieval, the **API Server** returns the raw JSON string. The
**Task SDK** parses and deserializes it back to native objects.
4. For backward compatibility, the API Server will automatically convert
legacy database objects (stored as dicts/lists/primitives) into JSON strings if
they are not already stored as JSON string literals.
#### Why is it needed?
- **Decoupled Languages**: Decouples language-specific serialization logic
from the API Server, allowing non-Python SDKs (Go, Java, TypeScript) to use the
same standardized JSON-formatted string contract.
- **Performance**: Eliminates "triple serialization" overhead (Task SDK
serialization -> API Server `XCom.serialize` -> SQLAlchemy JSON column type
serialization).
- **Bug Fix**: Resolves a bug where traditional components pulling XCom
values set by Task Execution tasks retrieved raw serialized dictionary
structures (e.g., `{"__classname__": "builtins.tuple", ...}`) instead of fully
deserialized Python objects (e.g., tuples). By storing them as JSON string
literals, `XComModel.deserialize_value` can now parse and deserialize them
correctly.
#### Changes Made
- **Task SDK Client**:
- `BaseXCom.serialize_value`: Modified to fully JSON-serialize values
using `json.dumps(serialize(value))`.
- `BaseXCom.deserialize_value`: Modified to load raw JSON string values
before calling the `deserialize` utility.
- `_generated.py`: Updated datamodel types (`XComResponse`,
`XComSequenceIndexResponse`, `XComSequenceSliceResponse`) to use `str | None`
for values.
- **Airflow Core API**:
- `api_fastapi/execution_api/datamodels/xcom.py`: Updated types to use
`str | None`.
- `api_fastapi/execution_api/routes/xcoms.py`: Updated GET and POST routes
to support JSON-serialized string handling and preserve backward compatibility
for legacy non-string records.
- `test_xcoms.py` & `test_xcom.py`: Updated tests to assert correct
serialization outputs and verify database-to-native object roundtrip validation.
#### Related Issue
- Closes #45231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
