eldenmoon opened a new pull request, #69995:
URL: https://github.com/apache/doris/pull/69995
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A (the legacy sparse key order fix below; link to follow)
Problem Summary:
Master stores `{"a.b":1}` and `{"a":{"b":2}}` under the same storage path,
`a.b`. The legacy suite records the first input reading back as
`{"a":{"b":1}}`: pushed-down `v['a.b']` and `v['a']['b']` both return `1`,
while exact in-memory `element_at(v, 'a.b')` returns `NULL`. Putting both
shapes in one object fails the load with `may contains duplicated entry`, as
the existing `duplicate_json_path` suite covers.
The legacy suites also lock ancestor hiding and rewrite behavior for
`{"a":5,"a.b":6}`, plus index-suffix collisions. They land first to protect
existing tables.
Design:
- Add `variant_escape_dots_in_keys` in `variant<properties(...)>`, default
`false`. It is immutable in this phase: ALTER in either direction fails, and
combining it with `variant_enable_nested_group` is rejected. With it enabled,
`v['a.b']` addresses the literal key; `v.a.b` and `v['a']['b']` address the
nested path.
- Use an injective per-component codec: `%` → `%25`, `.` → `%2E`. Encode
once when shredding keys and when scanner/segment entry points build storage
paths from logical query components. Decode only when the assembler emits
object keys; malformed encoded components return `Corruption`.
- Keep path statistics, sparse/doc map keys, bucket hashes, compaction
schemas and index suffixes in storage syntax. Templates, `field_pattern` and
SEARCH field names also use storage syntax. Write escaped sparse map keys in
byte order, as required by the reader's binary search (`%` sorts before `.`).
Key CAST pushdown types by the storage column name.
- Persist `ColumnMetaPB.variant_path_encoding` on the segment root: `0` is
legacy, `1` is escaped. Readers return `Corruption` for unknown markers and
`NotSupported` for a property/marker mismatch; physically absent roots retain
existing NULL/default behavior. The compaction gate does no work for tables
without the property; for escaped columns, every present input root must match
it.
- Escape `%` in index suffixes only for escaped columns, preserving legacy
suffix bytes. Whole-value rewrites—non-vertical and doc compaction, heavy
schema change, INSERT SELECT, MOW partial update and CTAS—decode and re-encode
losslessly because the codec is a bijection.
Protocol: add optional fields **ColumnPB 36**, **ColumnMetaPB 28** and
**TColumn 31**. These ids were checked as unused on apache branch-3.1,
branch-4.0, branch-4.1, branch-4.2 and master.
Compatibility: tables without the property retain existing behavior, locked
by the legacy suites and `duplicate_json_path`. There is no FE configuration
gate: use the property only after **every FE and BE is upgraded**. Tables using
it cannot be downgraded. Extended DESC displays storage paths such as `v.a%2Eb`.
The pre-existing sparse key ordering bug for keys with bytes below `.` in
tables without the property is fixed in a separate PR (link to follow); this PR
changes sparse byte ordering only for escaped columns.
### Release note
Add the opt-in Variant column property `variant_escape_dots_in_keys` to
preserve dotted keys separately from nested paths. Use it only after every FE
and BE is upgraded; tables using the property cannot be downgraded.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Validation:
- ASAN BE referenced tests: **1,616 passed, 5 skipped**; final focused
tests: **203 passed, 2 skipped**. FE: **817 tests across 40 classes passed**
after the final FE change.
- Both new suites passed output generation, line-by-line review and reruns
with `-parallel 1`. Coverage includes one-segment indexes on/off, vertical
widths 1/2 with multiple sparse shards, DUP/MOW partial updates, all rewrite
paths above, CTAS property retention, templates, SEARCH and rejection cases.
- **All five targeted suites passed on the final binary**: the two new
suites, both legacy suites and `duplicate_json_path`. Legacy outputs remained
byte-identical.
- Full `variant_p0`, default parallelism: **176/186 passed initially**. Each
failure was rerun alone with `-parallel 1`; the 8 debug-point suites passed
once debug points were enabled on the test BE. Remaining failures, unrelated to
this change: OSS `403 InvalidAccessKeyId` and the existing schema-change
baseline expecting `"public":1` instead of `true`. The HEAD V2 counterpart
expects `true` and passed; no separate pristine-master rebuild was run.
- ASAN product builds, formatting and build hygiene passed. Final clang-tidy
diagnostics on added lines: **zero**; existing diagnostics were retained.
- Behavior changed:
- [ ] No.
- [x] Yes. New opt-in column property; default behavior is unchanged.
- Does this need documentation?
- [ ] No.
- [x] Yes. Document PR to follow.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
