mrhhsg opened a new pull request, #70819:
URL: https://github.com/apache/doris/pull/70819

   ### Background
   
   A quantile state keeps small groups as exact samples and converts larger 
groups to a TDigest. Its compression value controls the digest's precision and 
memory tradeoff; the current default is 2048. BEs exchange and persist these 
states in the existing binary format.
   
   ### What problem does this PR solve?
   
   Issue Number: TODO(author): link a public issue before merge
   Related PR: #69284 (lazy TDigest buffers), #69037 (conversion sample 
preservation), #68498 (digest copy-on-write)
   PR Type: Feature
   
   Problem Summary:
   
   An explicit compression value of 128 currently falls back to 2048, so a 
1,440-sample group remains in the exact representation. This change allows 
values down to 128 behind a default-off BE creation setting and switches 
representation at a compression-specific sample limit. Mixed-compression 
partials also need bounded weighted recompression and preservation of actual 
sample endpoints; most of the production diff handles those merge paths, while 
most of the total diff is tests. Existing users retain the 2048 default, but a 
valid low-compression request now fails clearly until creation is enabled 
instead of silently using 2048.
   
   ### Details
   
   - **Change map:** +234/-96 production lines in 7 files; +598/-3 test lines 
in 6 files; +47 generated expected-output lines in 2 files. The production work 
covers state conversion and mixed merging (+153/-89 in `QuantileState`), 
bounded TDigest ingestion and endpoints (+39), SQL argument handling and the 
creation gate (+24/-5), and small config, declarations, and 
restored-accumulator changes (+18/-2).
   - **Scope:** `TO_QUANTILE_STATE` creation and BE quantile-state reading, 
merging, and serialization. Low compression is `[128, 2048)`; the default 
remains 2048 and the maximum remains 10000. FE syntax, `percentile_approx` 
behavior, and global TDigest defaults do not change.
   - **Design:** Separate the legal minimum from the default and convert after 
`min(2048, ceil(compression))` explicit samples. Low-compression digests use a 
smaller pending buffer; mixed low/high states select the lower compression and 
ingest weighted centroids incrementally, preserving recorded extrema and 
releasing spare capacity. The default-off creation gate keeps the existing wire 
format and confines rollout to reader-first deployments without adding protocol 
fields.
   - **Usage:** Upgrade all BE readers first, then set 
`enable_low_compression_quantile_state = true` in every BE's startup 
configuration and restart them. For example:
   
     ```sql
     SELECT quantile_state_to_base64(to_quantile_state(1.0, 128));
     ```
   
   - **Default behavior:** The BE setting is false and is not dynamically 
mutable. Valid requests below 2048 return `InvalidArgument` while it is off; 
invalid compression values retain the 2048 fallback. Already stored 
low-compression states remain readable even when creation is disabled.
   - **Compatibility:** New BEs read both existing and low-compression states 
in the unchanged layout. Old BEs reject low-compression states, including 
exchanged or persisted data, so do not enable creation during a rolling upgrade 
or roll back below a reader-capable version after low-compression data exists. 
Disabling creation is not a downgrade mechanism.
   - **Persistence and failure handling:** The serialized state layout and 
existing column/stream paths are reused; no FE metadata, EditLog, image, 
Thrift, or Protobuf change is introduced. Existing storage and recovery paths 
can carry low-compression states only where all readers understand them.
   - **Behavior change:** A low-compression request that previously fell back 
to 2048 now errors while the gate is off and uses the requested value when it 
is on. Non-NULL nullable FLOAT compression constants now retain their requested 
value; a true NULL still uses 2048. An EMPTY merge target now inherits the 
first nonempty input's compression, including in high-compression cases.
   
   ### Testing
   
   - Local ASAN BE and FE builds succeeded; FE Maven Checkstyle reported no 
violations.
   - Local focused ASAN BE unit tests passed: 71 tests in 7 suites, covering 
low-compression boundaries, mixed weighted merges, extrema, copy-on-write, 
serialization, aggregate restore, and constant/nullable arguments. The 
nullable-constant case failed on the earlier implementation and passed after 
its correction.
   - With creation disabled, the new ordinary P0 SQL suite passed (1/1). With 
creation enabled, the SQL content of the opt-in suite passed (1/1) on an 
isolated local cluster; its expected output was generated by the regression 
runner and passed a subsequent comparison run. This was a mirror without the 
Docker wrapper, not an execution of the Docker suite.
   - Three existing high-compression P0 suites passed on the preceding commit; 
their covered production paths were not changed by the final nullable-argument 
correction.
   - The actual Docker wrapper, representative RELEASE performance and accuracy 
benchmarks, multi-BE exchange, and mixed-version upgrade/rollback tests were 
not run.
   
   ### Notes for Reviewers
   
   Read `QuantileState::merge` alongside `TDigest::merge_bounded`: centroid 
means must retain their weights, while true endpoints need separate 
preservation. The high-compression fast path remains, except that an EMPTY 
target now takes the first nonempty source's compression. The opt-in setting 
controls creation, not reading or automatic cluster capability negotiation; 
memory and performance improvements have not been quantified on a 
representative workload.
   
   ### Release note
   
   `TO_QUANTILE_STATE` can use compression values down to 128 after operators 
upgrade all BE readers and enable `enable_low_compression_quantile_state`. 
Low-compression requests fail while the setting is disabled rather than 
silently using the default compression.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes; document the BE setting, reader-first 
rollout, and rollback floor in the Doris website.
   - Pre-submission status: Partially verified: the actual Docker wrapper, 
representative RELEASE performance and accuracy, multi-BE exchange, and 
mixed-version deployment were not tested.
   
   ### AI Disclosure
   
   - Tool: Codex
   - Generated: BE code, unit tests, SQL regression, and this description.
   - Human review: Not stated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to