zhangshenghang opened a new pull request, #11524: URL: https://github.com/apache/seatunnel/pull/11524
## Purpose of the change Fixes #10795. Syncing Iceberg → Doris with both sides in `Asia/Shanghai` shifts every `TIMESTAMP` (`timestamptz`) column by **-8 hours**: the data written into Doris is UTC wall-clock. ## Root cause - The Iceberg source returns `timestamptz` values normalized to UTC (`OffsetDateTime` at offset `Z`) — that part is correct. - The Doris sink serializes `TIMESTAMP_TZ` as *wall-clock* because Doris `DATETIME` has no timezone. But the wall-clock branch in the JSON/text/CSV serializers did `OffsetDateTime.toLocalDateTime()`, which **drops the offset without converting the instant**. For a UTC value this emits the UTC wall-clock, and Doris parses it in its `Asia/Shanghai` session — hence the 8-hour shift. ## Changes All three serializers used by the Doris sink now convert the instant to the session (JVM) timezone before taking the wall-clock: - `RowToJsonConverters` (JSON, `serializeTimestampTzAsLocal`) - `TextSerializationSchema` (`wallClockTimestampTz`) - `CsvSerializationSchema` (`wallClockTimestampTz`) ```java odt.atZoneSameInstant(ZoneId.systemDefault()).toLocalDateTime() ``` Behavioral impact is scoped: - Values already carrying the session offset serialize to exactly the same string as before (no change). - Only values whose offset differs from the session zone (the buggy case, e.g. UTC from Iceberg) change — they now represent the same instant in the sink's timezone. - Both flags are only used by the Doris sink's `SeaTunnelRowSerializer`; nothing else is affected. ## Tests Added serialization tests in all three format modules, pinning the JVM timezone to `Asia/Shanghai`: - `2024-01-01T10:00:00Z` → `2024-01-01 18:00:00` (text/CSV) and `"2024-01-01T18:00:00"` (JSON) — previously `10:00:00`. - `2024-01-01T18:00:00+08:00` → `2024-01-01 18:00:00` — unchanged, proving no regression for values already at the session offset. ## Verification - `./mvnw -pl seatunnel-formats/seatunnel-format-json,seatunnel-formats/seatunnel-format-text,seatunnel-formats/seatunnel-format-csv,seatunnel-connectors-v2/connector-doris test` — all pass (including existing TIMESTAMP_TZ round-trip tests, which use the offset-preserving path and are untouched). ## Notes - Users should keep the engine JVM timezone aligned with the Doris session timezone (standard deployment); with that, source wall-clock == sink wall-clock for the same instant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
