eldenmoon opened a new pull request, #70498:
URL: https://github.com/apache/doris/pull/70498
### Background
VARIANT segment writers shred object leaves into typed paths and infer a
storage type for array values. Vertical compaction processes a tablet schema in
small column groups; extracted VARIANT paths can store their column metadata in
the segment's external metadata region.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Performance
Problem Summary: Long-array writes repeatedly decode the same container
layout and recompute common scalar types after the result has stabilized.
Wide-schema vertical compaction also rebuilds the entire reader schema and
bulk-loads all external path metadata for groups that need only a few leaves.
This PR reuses array layout information, skips redundant scalar inference,
shares one reader-schema snapshot per merge, and uses existing indexed metadata
lookup for ordinary physical leaves. Storage types, values, corruption checks
and publication semantics are preserved.
### Details
- **Change map:** 124 changed production lines across 6 files; 191 added
unit-test lines; 184 changed benchmark lines; 135 documentation lines; no
generated files. The commits separate array changes, compaction changes, and
tests/benchmarks/documentation.
- **Why it was slow:** Array inference, representability checks and append
each decode the container header/offset layout. Mixed scalar arrays can keep
resolving a common type after JSONB already absorbs additional scalars. For
wide compaction, copying all N schema columns and loading all external path
metadata for each small column group repeats work proportional to N across O(N)
groups.
- **Optimization:**
- Array traversal/inference (60 changed production lines): add a borrowed
`ArrayView` following `ObjectView`; retain child offset/size/boundary checks;
skip redundant JSONB and FLOAT/DOUBLE common-type resolution while still
detecting later object/array children.
- Reader schema (32 changed production lines): copy it once per vertical
merge, keep it separate from the writer schema, and retain a separate projected
`ReadSchema` for each group.
- External metadata (32 changed production lines): use existing exact-path
lookup/cache for ordinary physical leaves. Missing paths keep
full-tree/sparse-statistics planning; root, sparse/doc storage paths and
enabled NestedGroup providers retain their existing route. Selected-path I/O
errors propagate.
- **Results:** Local RELEASE/O3 benchmarks, 2,000 rows per case, five
fresh-input rounds per binary, alternating baseline/fixed order on the same
CPU. Array cases have 8,192 elements; the wide physical case has 2,248
materialized paths and merges ten overlapping input rowsets. CPU medians are:
| Synthetic case | Phase | Baseline CPU | Fixed CPU | Reduction |
|---|---|---:|---:|---:|
| Long integer array | Segment writer | 2.368 s | 2.016 s | 14.9% |
| Long mixed integer/fractional array | Segment writer | 8.280 s | 3.598 s
| 56.5% |
| Long mixed integer/fractional array | Cumulative compaction | 9.667 s |
5.050 s | 47.8% |
| Wide materialized object | Vertical cumulative compaction | 37.796 s |
9.201 s | 75.7% |
- **Applicability and cost:** These are mechanism benchmarks using synthetic
JSON, not a reproduction of the original production dataset. The supplied
profile does not establish array lengths or element-type mixtures, and
production input/DDL/version are unavailable. Segment-writer timing excludes
JSON generation/parsing and output validation; compaction CPU covers the
calling thread. Pure 256 KiB strings and whole-VARIANT wide-object rewriting
are approximately unchanged. Wide-object import shows no consistent benefit or
regression: the full matrix is 1.4% higher CPU, while a separate import-only
control is 1.4% lower with overlapping ranges. No persistent cache,
configuration, storage-format or FE/BE protocol change is introduced.
Production Cloud and parallel-writer throughput are unverified. The complete
matrix, ranges and limitations are in `be/benchmark/variant/LARGE_JSON.md`.
### Testing
All results below are local. Tests and benchmarks ran on baseline
`1f241cf9b451730d06fc2edbba89bef27ba4efda` plus this diff; validated C++ source
hashes match PR head `24907185d94f9ca287b86b21857c36f97182904a`.
- Before product changes: 33 focused ASAN tests passed.
- Final source: 147 ASAN tests selected, 143 passed, zero failures, 4
skipped because NestedGroup support is unavailable. Coverage includes
array/null-map/corruption boundaries, late container fallback, writer/reader
compatibility, decimal/float promotion, sparse/doc layouts, vertical
DUP/UNIQUE/AGG merges, deletes, key ranges, cancellation, and wide compaction
followed by rowset reload with independent field expectations.
- `BUILD_TYPE=RELEASE ./build.sh --be --benchmark`; 9 scenarios x 5 rounds x
2 binaries = 90 validated timed cases. Every case checks persisted complete
values; compaction additionally checks versions, row counts, layout and
physical rewrite. Profiled runs and smoke runs are excluded from timing results.
- ASAN through `run-be-ut.sh --run`, selecting `VariantPathBuilderTest`,
`VariantShredderTest`, `VariantValueTest`, `VerticalCompactionTest`, the
compaction read/schema/util tests, `VariantColumnWriterReaderTest` and
`VariantWriterCompatibilityTest`.
- Repository clang-format 16, build hygiene, changed-line clang-tidy, and
`git diff --check` passed. The local benchmark/UT toolchain required
`GLIBC_COMPATIBILITY=OFF` to avoid duplicate compatibility symbols; product
compatibility settings are unchanged.
- CI results are pending. No SQL-cluster load, full-cluster restart or
production Cloud result is claimed.
### Notes for Reviewers
Start with the three small production changes above. The reader snapshot is
inspected by sequential group readers; each group's projected schema stays
independent. The metadata lookup runs before acquiring the metadata-tree shared
lock and reuses existing reader/cache locking.
The branch merges cleanly with master
`e125ea858678fd2aea5ebf179359650aa77fabb6`; the three commits since the local
test baseline change only CI workflows. This merge check is separate from
runtime validation.
### Release note
Reduce VARIANT writer CPU overhead for long arrays and vertical compaction
overhead for wide schemas without changing stored values or formats.
### Check List (For Author)
- Behavior changed: No (performance only; storage types, fallback and
corruption checks are preserved)
- Does this need documentation: No (benchmark instructions and validation
notes are included)
- Pre-submission status: Partially verified: local ASAN and RELEASE
benchmarks passed; four NestedGroup tests skipped; CI and production Cloud
validation pending
### AI Disclosure
- Tool: Codex (model identifier unavailable in this session)
- Generated: Production changes, unit tests, benchmarks, validation
documentation and this PR description
- Human review: Not recorded; maintainer review pending
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
