Doris-Breakwater commented on issue #67370: URL: https://github.com/apache/doris/issues/67370#issuecomment-5490657576
Breakwater-GitHub-Analysis-Slot: slot_71f66e5c4174 ## Initial maintainer assessment **Disposition:** Accept as an Arrow Flight SQL interoperability bug. The report contains a complete version and reproduction, and the epoch values plus metadata-only workaround strongly indicate a schema-annotation problem rather than TIMESTAMPTZ value corruption. The impact is still user-visible: a standard Python Arrow conversion fails even though the query data is otherwise correct. The issue is currently unlabeled and unassigned. Suggested labels are `kind/bug` and `area/rpc`; the Iceberg fixture exposes the problem, but the failing code path is the generic Flight SQL Arrow schema builder rather than Iceberg decoding. ## Code-path findings at `31263df4dc1d4d3a27517d264802cd4d6b92c874` Verified facts: 1. The FE fetches the schema generated by the result BE and forwards that schema in `FlightInfo`; it does not reconstruct the TIMESTAMPTZ field itself ([`FlightSqlConnectProcessor.java`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlConnectProcessor.java#L112-L175), [`DorisFlightSqlProducer.java`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java#L235-L283)). 2. The BE result sink passes `RuntimeState::timezone()` into `get_arrow_schema_from_expr_ctxs()` ([`result_sink_operator.cpp`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/exec/operator/result_sink_operator.cpp#L55-L67)). For `TYPE_TIMESTAMPTZ`, `convert_to_arrow_type()` constructs `arrow::TimestampType(time_unit, timezone)` with that string unchanged ([`arrow_row_batch.cpp`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/format/arrow/arrow_row_batch.cpp#L100-L119)). There is no Arrow-specific validation or normalization at this boundary. 3. TIMESTAMPTZ payload encoding is independent of that metadata string: `DataTypeTimeStampTzSerDe::write_column_to_arrow()` converts every non-null value to a Unix epoch using `cctz::utc_time_zone()` ([`data_type_timestamptz_serde.cpp`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/core/data_type_serde/data_type_timestamptz_serde.cpp#L329-L359)). This matches the report's correct raw epochs and supports treating the defect as metadata-only. 4. The FE query coordinator canonicalizes the session timezone using `ZoneId.of(...).getId()` before putting it in `TQueryGlobals` ([`Coordinator.java`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java#L369-L375), [`TimeUtils.java`](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/common/util/TimeUtils.java#L151-L154)). Java canonicalization returns `Z` for the valid Doris fixed-offset session value `+00:00`. Doris BE also recognizes `Z` as Zulu, so that identifier can reach schema construction successfully. 5. Arrow timestamp timezone metadata permits a tz-database name or a numeric `+XX:XX`/`-XX:XX` offset. `Z` is neither form. Consequently, emitting `timestamp[..., tz=Z]` is non-portable and PyArrow's `ZoneInfoNotFoundError` is consistent with the schema ([Arrow timestamp format contract](https://arrow.apache.org/docs/format/Columnar.html#timestamp)). This verifies the root-cause boundary: a timezone identifier valid within Doris/Java can be copied into an Arrow schema even when it is not a valid Arrow timezone identifier. It also proves a deterministic failure path for a session timezone of `+00:00`. One detail remains unverified: at this exact commit, `TimeUtils.getCanonicalTimeZone("UTC")` evaluates to `UTC`, not `Z`. Therefore the report's literal `SET time_zone = 'UTC'` should not be assumed to be the precise origin of `Z` without observing the effective value on the same Flight session. This discrepancy does not invalidate the reported server output or the generic bug, but it matters for identifying every upstream trigger and choosing regression coverage. ## Missing evidence for the exact `UTC -> Z` trigger Please add the following from the same ADBC connection and cursor, immediately before the failing query: 1. The fetched result of `SHOW VARIABLES LIKE 'time_zone'` (not only confirmation that `SET` returned successfully). 2. `repr(table.schema.field(<timestamptz-column>).type.tz)` for a small matrix of session values: `UTC`, `+00:00`, and `Asia/Shanghai`. 3. Exact `pyarrow`, `adbc_driver_flightsql`, and Python versions. These are not needed to accept the server-side metadata bug, but they make the client failure reproducible. 4. FE and participating BE version/commit output, to rule out a mixed-version cluster if the effective session value is confirmed as `UTC` while the wire schema still contains `Z`. A Doris execution profile is not needed for this issue; this is schema construction, not a performance or operator-timing problem. ## Recommended next steps 1. Normalize Arrow timezone metadata at the Arrow schema-construction boundary, rather than changing Doris's general execution-time timezone representation. At minimum, map the zero-offset identifier `Z` to the portable `UTC` (or `+00:00`) while leaving epoch buffers unchanged and preserving valid IANA names/non-zero fixed offsets. 2. Add a focused BE unit test around `TYPE_TIMESTAMPTZ` Arrow type construction covering `UTC`, `Z`, `+00:00`, a non-zero offset, and an IANA zone. Because conversion is recursive, include at least one nested TIMESTAMPTZ case. 3. Add an end-to-end Flight SQL/ADBC regression for both `SET time_zone = 'UTC'` and `SET time_zone = '+00:00'` that asserts the wire-schema timezone and calls `to_pylist()` on nullable TIMESTAMPTZ values. Existing Flight SQL coverage checks JDBC type identity but does not assert timezone metadata portability. 4. Link the eventual fix and regression to tracking issue #65615. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
