Doris-Breakwater commented on issue #68074: URL: https://github.com/apache/doris/issues/68074#issuecomment-5695723687
Breakwater-GitHub-Analysis-Slot: slot_69a5adce867b ## Initial assessment This is a confirmed VARIANT JSON-serialization correctness bug in 4.1.3, not a type-inference problem and not merely a formatting preference. The stored subcolumn can remain `boolean`, but the reconstructed JSON changes the JSON type from boolean to number. There is no evidence of on-disk type corruption; the loss occurs when VARIANT values are rendered as JSON text. Because that text can be consumed by JSON functions and external clients, the result is semantically incorrect and should be backported to 4.1 after a fix is validated. The issue is currently unlabelled. Please add the repository's normal bug and VARIANT/component labels. ## Verified code-path evidence - `CAST(VARIANT AS STRING)` reaches `DataTypeVariantSerDe::serialize_one_cell_to_json()`, which delegates to `ColumnVariant::serialize_one_row_to_string()` ([4.1.3 source](https://github.com/apache/doris/blob/4.1.3/be/src/core/data_type_serde/data_type_variant_serde.cpp#L130-L135)). - A dense VARIANT subcolumn is rendered by `ColumnVariant::Subcolumn::serialize_text_json()`, which delegates to the inferred subcolumn type's SerDe ([4.1.3 source](https://github.com/apache/doris/blob/4.1.3/be/src/core/column/column_variant.cpp#L1399-L1426)). For `TYPE_BOOLEAN`, `DataTypeNumberSerDe` enters the shared `is_int_or_bool(T)` branch and calls `write_number()` on the underlying `UInt8`, producing `1` or `0` ([4.1.3 source](https://github.com/apache/doris/blob/4.1.3/be/src/core/data_type_serde/data_type_number_serde.cpp#L204-L220)). The locally available current `branch-4.1` snapshot still has the same logic, so this is not already fixed there. - `ARRAY<BOOLEAN>` recurses through `DataTypeArraySerDe` into the same nested boolean SerDe, which explains `[1, 0]` ([4.1.3 source](https://github.com/apache/doris/blob/4.1.3/be/src/core/data_type_serde/data_type_array_serde.cpp#L46-L69)). - Sparse values preserve their binary type, but reconstruction decodes each value into a temporary `Subcolumn` and calls the same `serialize_text_json()` path ([4.1.3 source](https://github.com/apache/doris/blob/4.1.3/be/src/core/column/column_variant.cpp#L2011-L2025)). This explains why crossing `variant_max_subcolumns_count` changes JSONB-backed values to `1`/`0` without requiring a separate sparse-column defect. - JSONB is the working comparison because `DataTypeJsonbSerDe` uses `JsonbToJson`, which emits JSON boolean tokens ([4.1.3 source](https://github.com/apache/doris/blob/4.1.3/be/src/core/data_type_serde/data_type_jsonb_serde.cpp#L64-L79)). - `FormatOptions` already documents an `is_bool_value_num` switch whose false form is intended to render `[true]`, but the number SerDe currently does not consult it ([4.1.3 source](https://github.com/apache/doris/blob/4.1.3/be/src/core/data_type_serde/data_type_serde.h#L187-L194)). This is strong evidence for the missing formatting distinction, although the exact patch scope should be reviewed for compatibility. The distinction from #68016 is correct. A boolean-only path returns from the single-type case before mixed numeric supertype selection; changing boolean-plus-number merging cannot affect this serializer path. I did not rerun a Doris cluster locally, but the supplied reproduction is complete and deterministic, and the 4.1.3 source path directly accounts for the dense scalar, nested, array, JSONB, and sparse results. ## Missing information / decision needed No additional logs, profile, or reproduction details are needed to triage this issue. The only design decision needed before implementation is scope: JSON reconstruction for VARIANT must request JSON boolean literals while existing non-VARIANT display behavior for `BOOLEAN` or `ARRAY<BOOLEAN>` should not be changed accidentally. In particular, globally changing the default boolean text format could create unrelated compatibility changes. Operational note: the expected local Develop skill was unavailable at `/mnt/disk4/breakwater/Breakwater/skills/breakwater-develop/SKILL.md`, so no code changes were attempted. ## Recommended next steps 1. Add a failing VARIANT regression test covering dense `true`/`false`, nested booleans, `ARRAY<BOOLEAN>`, `CAST(j['b'] AS STRING)`, and a heterogeneous JSONB control case. 2. Add a sparse-path case using a deliberately small `default_variant_max_subcolumns_count` so the test does not require 2,049 keys. Assert `true`, `false`, and `[true]` on both sides of the dense/sparse boundary. 3. Implement the formatting distinction narrowly: make the boolean number SerDe honor `FormatOptions::is_bool_value_num`, and ensure VARIANT JSON reconstruction passes the JSON-literal setting through dense, nested-array, scalar-root, document-value, and sparse paths. Preserve the existing default for unrelated SQL/MySQL/collection rendering unless maintainers explicitly choose a broader behavior change. 4. Run the relevant BE SerDe/ColumnVariant unit tests and the VARIANT regression suites, including the combined matrix with #68016. Then backport the validated fix to branch-4.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
