Doris-Breakwater commented on issue #69326: URL: https://github.com/apache/doris/issues/69326#issuecomment-6285554892
Breakwater-GitHub-Analysis-Slot: slot_5f9454ba503b **Initial assessment:** the reported `Z` resolution gap is confirmed by inspection of the cited source revisions. It merits BE datetime/timezone and VARIANT correctness triage. The issue is currently open with no labels. The downstream field omission is consistent with the verified conversion mechanism, but attributing that particular run to the missing alias still requires a controlled reproduction. No Doris build, unit test, or end-to-end reproduction was executed for this analysis. **Verified source behavior** - At `cc8a24a13edb46e62359678bb79f79a0a45802ce`, the timezone scan uses relative names as cache keys and installs `z` only from an existing `zulu` key. `etc/zulu` cannot satisfy that lookup. On a cache miss, normalization accepts `utc`, `etc/utc`, and `zulu`, but rejects `z`; the resolver returns false before reaching its UTC fallback. Thus an available UTC file alone cannot rescue `Z`. The same relevant logic exists at `349cd15471d7531a687f71db19d5742b3759d4cc`. See [timezone loading](https://github.com/apache/doris/blob/cc8a24a13edb46e62359678bb79f79a0a45802ce/be/src/util/timezone_utils.cpp#L95), [resolver](https://github.com/apache/doris/blob/cc8a24a13edb46e62359678bb79f79a0a45802ce/be/src/util/timezone_utils.cpp#L162), and [normalization/fallback](https://github.com/apache/doris/blob/cc8a24a13edb46e62359678bb79f79a0a45802ce/be/src/util/timezone_utils.cpp#L249). Treating `Z` as zero offset follows [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.6). - Both [strict](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/exprs/function/cast/cast_to_datetimev2_impl.hpp#L750) and [permissive](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/exprs/function/cast/cast_to_datetimev2_impl.hpp#L1034) datetime parsing pass named suffixes to this resolver. Numeric `+00:00` follows the offset path instead. The [non-strict DATETIMEV2 serde](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp#L283) marks failed conversions NULL and returns `Status::OK()`. - The VARIANT consequence has an explicit source path: typed candidate conversion calls `VariantPathBuilder::convert_to()`, which invokes `promote(..., true)`. Promotion casts the column and removes NULL values **and their path row IDs**, while retaining the logical rows. In ordinary layout, conversion happens before selection into materialized or sparse storage. This supports field omission with successful row ingestion; zero filtered load rows does not establish successful conversion of every field. See [typed candidate conversion and storage selection](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/storage/segment/variant/v2/variant_shredder.cpp#L581), [forced conversion](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/storage/segment/variant/v2/variant_path_builder.cpp#L1235), and [NULL filtering](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/storage/segment/varian t/v2/variant_path_builder.cpp#L1099). One validation detail: this storage cast creates its own default [RuntimeState](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/exec/common/variant_util.cpp#L218), whose timezone initializes to the default `+08:00` ([initialization](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/runtime/runtime_state.cpp#L316)). Record the actual conversion context rather than assuming the querying session controls this path. **Evidence still needed** 1. Exact downstream FE/BE build commits, minimal `SHOW CREATE TABLE`, load command/options and response, and a small input containing matched `Z` and `+00:00` values. Include load/session timezone and strictness settings, typed-path/sparse properties, and typed-path plus full-VARIANT results. 2. The failing BE process's effective `TZDIR`, timezone files visible inside its container/mount namespace at startup, and startup timezone-loading warnings. Post-failure host filesystem inspection does not establish the contents of an already populated process cache. 3. A fresh Apache master reproduction with its exact SHA, comparing values immediately after load and after compaction. Current evidence does not establish a compaction-specific defect or the affected release range. **Recommended next steps** Add `lower == "z"` to the existing UTC normalization branch. The existing UTC fallback then provides a zone without requiring the `Zulu` compatibility file; no change to VARIANT NULL handling or timezone conversion policy is needed for this fix. [PR #68979](https://github.com/apache/doris/pull/68979), inspected at head `8f45f28b1cdeedb0aa0bfa9ecf7d4a97f6c7b01e`, remains open and does not add this normalization, so it should not be treated as resolving this issue. Validate with environment-independent BE tests for direct parsing and resolver lookup of `Z`/`z`, asserting zero offset and including `UTC`/`+00:00` controls. The direct-parser assertions should expose the old behavior even on a host containing `Zulu`; exercise the resolver with named entries absent. Serialize cache-mutating tests and restore state on every exit. Then run stored-column strict/permissive datetime checks and a minimal predefined VARIANT DATETIME load in an isolated timezone environment without top-level `Zulu`. Cover ordinary materialized and sparse typed paths, and compare both field presence and value before/after compaction. In a verified UTC+08:00 conversion context, both timestamp forms must yield `2015-01-01 11:47:41`. For affected ingestion, consider normalizing terminal UTC `Z` to `+00:00` upstream after validating the same minimal load. Keep the explicit offset. A parser fix only prevents future failures; already discarded values require recovery from original input and reloading. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
