mrhhsg opened a new pull request, #69284:
URL: https://github.com/apache/doris/pull/69284
### What problem does this PR solve?
Issue Number: None
Problem Summary:
`percentile_approx` and `quantile_union` keep one `TDigest` per GROUP BY
group. The
`TDigest(compression, unmerged_size, merged_size)` constructor eagerly
reserved
`2 * compression` processed centroids and `8 * compression + 1` unprocessed
centroids before
any value was stored. At the default `compression = 10000` this costs 800008
bytes per group
(160000 B + 640008 B), independent of how many rows the group actually holds.
Both allocations exceed the ASAN primary allocator's 128 KiB size-class
limit, so each digest
occupied two secondary-allocator chunks. With roughly 524288 live groups the
process exhausted
the secondary allocator's chunk table (`kMaxNumChunks = 1 << 20`) and
aborted in
`sanitizer_allocator_secondary.h:42`. Release builds map about 480 GB of
virtual address space
for 600K singleton groups in the same scenario.
Fix:
- Let both vectors grow on demand in the constructor. `_max_processed` and
`_max_unprocessed`
are still initialized exactly as before: they are compression thresholds
used by
`is_dirty()`, the buffered `add()` path, and serialization, not capacity
caches.
- Remove the exact-size `reserve(total)` from `_merge_unprocessed`. Once the
constructor no
longer preallocates, `reserve(total)` would reallocate and copy the whole
buffer on every
merge of a small digest, degrading many sequential merges to O(n^2)
copies. `insert()` grows
geometrically instead, keeping amortized O(1).
The compression thresholds, the t-digest algorithm, the numeric results, and
the serialized
format are unchanged.
Tests updated/added:
- New unit test: a default-constructed digest has zero buffer capacity,
allocates less than
4 KiB in total, keeps the historical thresholds (20000 / 80000), and still
returns the right
quantile after a single value.
- New unit test: 1000 singleton groups allocate memory proportional to their
data
(< 1 KiB per group) and each returns the correct median.
- Updated an existing accumulator test that previously asserted the 80001
preallocation; it
now asserts that the accumulator retains spare write-buffer capacity
without encoding the
removed preallocation.
### Release note
Fix backend aborts for `percentile_approx` / `quantile_union` queries with a
very high number
of GROUP BY groups by no longer preallocating compression-sized buffers per
group.
### Check List (For Author)
- Test: Unit Test
- BE unit tests under ASAN: 41 tests from 5 suites passed, covering
`TDigestTest`
(including the two new capacity/memory tests), `QuantileStateTest`,
`AggregateFunctionPercentileApproxArrayTest`,
`AggregateFunctionQuantileStateTest`,
and `AggregateFunctionQuantileStateRangeTest`. Command:
`./run-be-ut.sh --run
--filter='TDigestTest.*:QuantileStateTest.*:AggregateFunctionPercentileApproxArrayTest.*:AggregateFunctionQuantileStateTest.*:*AggregateFunctionQuantileStateRangeTest.*'
-j 64`
- A standalone ASAN program holding 600000 live `TDigest` instances now
runs to
completion; the same program aborted at about 524288 digests before
the fix.
- A negative-control build against the old header confirms the new
assertions fail on the
unfixed code (capacities 20000 / 80001 and 800008 bytes per digest;
1000 singleton
groups allocate 800008000 bytes).
- `clang-format` (v16) passes on the changed files, the repository-wide
format check
passes, and the build-hygiene check passes.
- `clang-tidy` could not run on this machine: its toolchain reports
`'stddef.h' file not found` for the changed files, unrelated to this
change.
- No regression test was added: reproducing the abort requires more than
~524288 live
digests, which no regression cluster can hold, and release builds
neither abort nor
return an error row. The new unit tests assert the exact capacity and
allocation
behavior at the layer where the fix lives.
- Behavior changed: No
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
