mrhhsg opened a new pull request, #70819:
URL: https://github.com/apache/doris/pull/70819
### Background
A quantile state keeps small groups as exact samples and converts larger
groups to a TDigest. Its compression value controls the digest's precision and
memory tradeoff; the current default is 2048. BEs exchange and persist these
states in the existing binary format.
### What problem does this PR solve?
Issue Number: TODO(author): link a public issue before merge
Related PR: #69284 (lazy TDigest buffers), #69037 (conversion sample
preservation), #68498 (digest copy-on-write)
PR Type: Feature
Problem Summary:
An explicit compression value of 128 currently falls back to 2048, so a
1,440-sample group remains in the exact representation. This change allows
values down to 128 behind a default-off BE creation setting and switches
representation at a compression-specific sample limit. Mixed-compression
partials also need bounded weighted recompression and preservation of actual
sample endpoints; most of the production diff handles those merge paths, while
most of the total diff is tests. Existing users retain the 2048 default, but a
valid low-compression request now fails clearly until creation is enabled
instead of silently using 2048.
### Details
- **Change map:** +234/-96 production lines in 7 files; +598/-3 test lines
in 6 files; +47 generated expected-output lines in 2 files. The production work
covers state conversion and mixed merging (+153/-89 in `QuantileState`),
bounded TDigest ingestion and endpoints (+39), SQL argument handling and the
creation gate (+24/-5), and small config, declarations, and
restored-accumulator changes (+18/-2).
- **Scope:** `TO_QUANTILE_STATE` creation and BE quantile-state reading,
merging, and serialization. Low compression is `[128, 2048)`; the default
remains 2048 and the maximum remains 10000. FE syntax, `percentile_approx`
behavior, and global TDigest defaults do not change.
- **Design:** Separate the legal minimum from the default and convert after
`min(2048, ceil(compression))` explicit samples. Low-compression digests use a
smaller pending buffer; mixed low/high states select the lower compression and
ingest weighted centroids incrementally, preserving recorded extrema and
releasing spare capacity. The default-off creation gate keeps the existing wire
format and confines rollout to reader-first deployments without adding protocol
fields.
- **Usage:** Upgrade all BE readers first, then set
`enable_low_compression_quantile_state = true` in every BE's startup
configuration and restart them. For example:
```sql
SELECT quantile_state_to_base64(to_quantile_state(1.0, 128));
```
- **Default behavior:** The BE setting is false and is not dynamically
mutable. Valid requests below 2048 return `InvalidArgument` while it is off;
invalid compression values retain the 2048 fallback. Already stored
low-compression states remain readable even when creation is disabled.
- **Compatibility:** New BEs read both existing and low-compression states
in the unchanged layout. Old BEs reject low-compression states, including
exchanged or persisted data, so do not enable creation during a rolling upgrade
or roll back below a reader-capable version after low-compression data exists.
Disabling creation is not a downgrade mechanism.
- **Persistence and failure handling:** The serialized state layout and
existing column/stream paths are reused; no FE metadata, EditLog, image,
Thrift, or Protobuf change is introduced. Existing storage and recovery paths
can carry low-compression states only where all readers understand them.
- **Behavior change:** A low-compression request that previously fell back
to 2048 now errors while the gate is off and uses the requested value when it
is on. Non-NULL nullable FLOAT compression constants now retain their requested
value; a true NULL still uses 2048. An EMPTY merge target now inherits the
first nonempty input's compression, including in high-compression cases.
### Testing
- Local ASAN BE and FE builds succeeded; FE Maven Checkstyle reported no
violations.
- Local focused ASAN BE unit tests passed: 71 tests in 7 suites, covering
low-compression boundaries, mixed weighted merges, extrema, copy-on-write,
serialization, aggregate restore, and constant/nullable arguments. The
nullable-constant case failed on the earlier implementation and passed after
its correction.
- With creation disabled, the new ordinary P0 SQL suite passed (1/1). With
creation enabled, the SQL content of the opt-in suite passed (1/1) on an
isolated local cluster; its expected output was generated by the regression
runner and passed a subsequent comparison run. This was a mirror without the
Docker wrapper, not an execution of the Docker suite.
- Three existing high-compression P0 suites passed on the preceding commit;
their covered production paths were not changed by the final nullable-argument
correction.
- The actual Docker wrapper, representative RELEASE performance and accuracy
benchmarks, multi-BE exchange, and mixed-version upgrade/rollback tests were
not run.
### Notes for Reviewers
Read `QuantileState::merge` alongside `TDigest::merge_bounded`: centroid
means must retain their weights, while true endpoints need separate
preservation. The high-compression fast path remains, except that an EMPTY
target now takes the first nonempty source's compression. The opt-in setting
controls creation, not reading or automatic cluster capability negotiation;
memory and performance improvements have not been quantified on a
representative workload.
### Release note
`TO_QUANTILE_STATE` can use compression values down to 128 after operators
upgrade all BE readers and enable `enable_low_compression_quantile_state`.
Low-compression requests fail while the setting is disabled rather than
silently using the default compression.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: Yes; document the BE setting, reader-first
rollout, and rollback floor in the Doris website.
- Pre-submission status: Partially verified: the actual Docker wrapper,
representative RELEASE performance and accuracy, multi-BE exchange, and
mixed-version deployment were not tested.
### AI Disclosure
- Tool: Codex
- Generated: BE code, unit tests, SQL regression, and this description.
- Human review: Not stated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]