freemandealer opened a new pull request, #70140:
URL: https://github.com/apache/doris/pull/70140
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Mostly-NULL value columns still occupy dense column storage in a MemTable.
This can exhaust the write buffer with relatively little non-NULL data and
contribute to excessive small segments during loads.
Add experimental sparse MemTable storage for eligible nullable value
columns. After sampling 4,096 rows, retain non-NULL values and their row IDs
when the non-NULL ratio is at most 10% and the estimated memory saving is at
least 30%. Materialize in batches of at most 4,096 rows through the current
vertical segment writer. Persisted segment encoding stays unchanged.
Keep dense fallback for unsupported schemas and duplicate UNIQUE keys, and
materialize complete Blocks for the shared changelog path. Preserve master’s
adaptive write buffer, batched row metadata and sorted allocated LSNs. Support
both beta rowset writer paths, use the finalized segment count for the rowset
segment-limit check, and expose sparse/peak-memory flush counters.
The BE switch `enable_sparse_memtable` defaults to `false`. The logical
materialization and row limits default to 4 GiB and 10,000,000 rows; each
MemTable snapshots the configuration at construction. This is an experimental
BE gate; a table-level FE property remains future work.
### Release note
Add experimental sparse MemTable storage for mostly-NULL eligible value
columns. Disabled by default; the persisted segment format is unchanged.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [ ] No.
- [x] Yes. Enabling the experimental BE switch changes MemTable memory
representation and flush timing; unsupported cases retain dense output.
- Does this need documentation?
- [x] No. This experimental BE gate is documented above and in the
configuration declarations; no production rollout is enabled.
- [ ] Yes.
Validation on upstream master `7866f5a36c02e15dc014aaef30e537a395f4059d`:
- Full BE ASAN build/package with `build.sh --be -j 16`, including the GLIBC
2.17 compatibility check.
- `run-be-ut.sh --run
--filter='MemTable*:VerticalSegmentWriterWritePathsTest.*' -j 16`: 46 tests
passed. Coverage includes sparse/dense equivalence, capability and error cases,
configuration snapshots, legacy Block consumers, LSN ordering, and DUP/MOW
segment round trips across the 4,096-row batch boundary.
- clang-format 16 and header hygiene passed.
- clang-tidy passed on changed lines in all 21 C++ sources/headers using the
actual target compilation flags, with Unity entries expanded for source
coverage.
The ASAN UT run retains leak detection and uses one precise suppression for
an OpenBLAS/libomp startup leak (`__kmp_affinity_create_x2apicid_map`, one
352-byte allocation). The same allocation reproduces with a zero-test filter.
No live-cluster regression or performance benchmark was run.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
