developer-rpai opened a new pull request, #73748: URL: https://github.com/apache/airflow/pull/73748
Fixes #72635. ## Problem A naive `datetime` pushed to XCom by one task comes back with a **different value** when the pulling task runs on a worker with a different OS timezone. Nothing errors; the value is just wrong. In real deployments (workers spread across zones) this is silent data corruption of XCom payloads. ## Root cause In `task-sdk/src/airflow/sdk/serde/serializers/datetime.py`: - `serialize` stores `o.timestamp()`, which interprets a naive datetime in the **writing** process's OS local timezone. - `deserialize` calls `datetime.fromtimestamp(ts, tz=None)`, which renders it in the **reading** process's OS local timezone. The two are only the same machine in a single-process test. ## Fix Anchor naive datetimes to the configured default timezone (`core.default_timezone`, exposed to the SDK as the initialized timezone) instead of OS local time, which is exactly what the Airflow timezone docs already define as the meaning of a naive datetime: - `serialize`: naive datetimes are localized with `make_aware()` (uses the SDK-initialized default timezone, DST-fold aware) before taking the epoch. The payload keeps `tz` empty so the value still deserializes as naive. - `deserialize`: when no timezone was stored, the epoch is interpreted in the default timezone via `make_naive()` and returned as a naive datetime, so the round-trip compares equal to what was pushed regardless of reader/writer OS timezones. Timezone-aware datetimes, `date`, and `timedelta` paths are untouched. ## Tests Reproduced the issue's exact scenario (write with `TZ=UTC`, read with `TZ=America/New_York` and `TZ=Asia/Kolkata` in separate processes): - Before: pulled `07:00` / `17:30` instead of `12:00`. - After: pulled `12:00` in all three zones, `pulled == pushed`. Also verified: aware datetimes (UTC, America/New_York) round-trip unchanged; `datetime.now()`-style naive keeps the existing test-suite assertion shape (`input.timestamp() == deserialized.timestamp()`); DST-adjacent wall times; `date`/`timedelta` unaffected; all under a non-UTC default timezone (`America/Chicago`) and a hostile `TZ=America/New_York` process. ## Caveat XCom blobs written by the old code carry an epoch computed in the writer's OS zone; they will now be interpreted under `default_timezone` on read. The old interpretation was undefined/broken, so there is no "correct" old behavior to preserve, but in-flight values serialized before this change may shift on read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
