amoghrajesh opened a new pull request, #58992:
URL: https://github.com/apache/airflow/pull/58992
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
closes: https://github.com/apache/airflow/issues/58887
closes: https://github.com/apache/airflow/issues/58885
### Motivation
This change is part of the broader effort to achieve client-server
separation. The serialization and deserialization utilities (`serde`) are
execution time utilities used primarily during task execution for:
- Serializing/deserializing XComs when communicating with the API server
- Processing deadline alerts
Since these utilities are only needed at execution time (in workers, DAG
processors, and triggerers), they belong in the `task-sdk` rather than
`airflow-core`. This allows the core server components (Scheduler, API Server)
to operate without requiring the task SDK, and also enabled task SDK to not
having to import serde from airflow core, enabling independent deployments and
upgrades.
### Blockers and Solutions
Several blockers were identified and resolved:
1. Migration File: The migration
`0092_3_2_0_replace_deadline_inline_callback_with_fkey.py` uses
`serde.deserialize()`
Effort tracked by @ramitkataria in [Do not use serde library for database
migrations](https://github.com/apache/airflow/issues/58886)
2. XCom API Stringification: The core API's xcom endpoint
(`/api/v2/dags/{dag_id}/dagRuns/{run_id}/taskInstances/{task_id}/xcomEntries`)
used `deserialize(full=False)` to stringify XCom values for UI display.
The solution was to create a dedicated `stringify.py` module in
`airflow-core` that provides _stringification_ without any SDK dependencies.
This module matches the behavior of `deserialize(full=False)` exactly but is
self-contained and independent of any external libraries.
3. XComEncoder/XComDecoder: These JSON encoder/decoder classes in
`airflow-core` directly imported `serde` for full serialization/deserialization
of XCom values.
The fix here was to implement lazy imports that attempt to load `serde` from
the SDK, making the dependency optional. If the SDK is not available,
appropriate error messages are shown. The consumers of this anyways were mostly
fixed / handled by: https://github.com/apache/airflow/pull/58900, which
discovered that the same effect can be achieved without using those encoders.
In a follow-up, I might move it to task SDK as well.
### High Level Changes
1. Serde module is moved to task sdk now
- All serializers moved to
`task-sdk/src/airflow/sdk/serialization/serializers/`
2. Stringification Separation
- Created `airflow-core/src/airflow/serialization/stringify.py` for UI
stringification
- This module is self-contained and does not import from SDK
- Matches `deserialize(full=False)` behavior exactly
3. Deprecation Path
- Old imports (`from airflow.serialization.serde import `) continue to
work
- Redirection to SDK variant with deprecation warnings
- Ensures backward compatibility till we remove it
4. Tests
- Serialize/deserialize tests moved to
`task-sdk/tests/task_sdk/serialization/test_serde.py`
- Stringify tests created in
`airflow-core/tests/unit/serialization/test_stringify.py`
5. Docs
- Updated serializer documentation to reference
`airflow.sdk.serialization.serializers
### What does this mean for new serializers? (Debatable, check open
questions below)
**Important**: New serializers should now be added to the **SDK namespace**:
The old `airflow-core/src/airflow/serialization/serializers/` directory
still exists for backward compatibility but is deprecated. New serializers
should not be added there.
### Open Questions
1. When should we delete
`airflow-core/src/airflow/serialization/serializers/`? Currently kept for
backward compatibility, but should be removed in a future release, no code
reads off of here, shall we just delete it? What for people who have added
their serializers in here?
2. Should XcomEncoder + Decoder stay in airflow core or task sdk?
### Testing
- All serialize/deserialize tests moved to SDK and passing
- New stringify tests created to test all possible situations
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
