eldenmoon opened a new pull request, #69914:
URL: https://github.com/apache/doris/pull/69914
### What problem does this PR solve?
Issue Number: None
Related PR: #69761
Problem Summary:
`VariantAssembler` found each row's visible values by scanning every
materialized slot of the row, both in `emit_merged_row()` and in the root
fast-path probe. With wide sparse documents (up to 2048 materialized paths, a
few present per row) this row-major scan over all null maps dominates assembly.
Each hierarchical batch now builds a per-row list of candidate slots, the
slots whose null byte is 0, column by column with `simd::find_byte`. Consumers
walk only the row's candidates, in slot order, and still decide visibility with
the single predicate `is_materialized_value_visible()`, so precedence and the
root/array/JSONB visibility rules are unchanged. When candidates make up at
least half of `rows × slots`, every row shares one identity list, which is the
previous scan, so dense batches keep their cost. The list exists only below
that threshold, so it holds at most `rows × slots / 2` `uint32_t` (16 MB for a
4096-row batch with 2048 slots).
Micro benchmark `BM_VariantAssembler` (new, RELEASE, 4096-row batches of
nullable Int64 slots; master and this PR run interleaved, 6 rounds × 5
repetitions, medians):
| Case | master | this PR |
|---|---|---|
| WideSparse: 2048 slots, 20 visible per row | 98.8 ms | 21.1 ms (4.7×) |
| Dense: 200 slots, all visible | 115.9 ms | 117.4 ms (+1.3%) |
| Narrow: 10 slots, all visible | 4.57 ms | 4.56 ms |
`MediumSparse` (200 slots, 75 visible) sets the threshold: the candidate
list took 65.4 ms and the identity scan 68.7 ms.
End to end on master, whole-VARIANT scans and compaction of sparse data do
not change measurably (`SELECT sum(length(CAST(v AS STRING)))` 9.08 s vs 9.00
s), because their cost is dominated by reading the subcolumns. The gain shows
where wide batches are assembled, such as the VARIANT values-index writer in
#69761. Measured there with an earlier revision of this patch (same sparse
path, before the dense threshold was added), sparse values-index load went from
54.1 s to 35.1 s, rebuild compaction from 75.2 s to 43.1 s, and root MATCH
without the index from 31.2 s to 14.8 s.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
Test results (ASAN build):
- BE UT: new `VariantAssemblerLegacyTest` cases cover the visibility
matrix, sparse and materialized ordering on prefix-related paths, mixed DOC
rows, batch boundaries (15/16/17/33 rows), and both the candidate-list and
identity modes. All VARIANT BE UT pass: 901 passed, 5 skipped (NestedGroup and
streaming-writer cases this build does not enable).
- Regression: `variant_p0` (182 suites) and `variant_p1` (4 suites)
pass, except two:
- `outfile_csv_variant_doc_value` needs S3 credentials that the test
environment lacks.
- `double_write_schema_change_with_variant` fails identically on
master a72dcaa17b1: same output, a key-order difference inside an array of
objects.
- Manual test: `BUILD_TYPE=RELEASE ./build.sh --benchmark`, then
`benchmark_test --benchmark_filter='^BM_VariantAssembler/'`. master's
`be/benchmark/variant/benchmark_variant_segment.hpp` currently does not
compile, so its include in `benchmark_main.cpp` was removed locally for the
measurement.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
