jacktengg opened a new pull request, #69155:
URL: https://github.com/apache/doris/pull/69155
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: percentile_approx_weighted accepts DOUBLE weights but
narrows weights, accumulated mass and interpolation operands to FLOAT in
TDigest. Positive weights such as 1e-100 become zero and can abort BE when
zero-weight centroids are merged; weights such as 1e100 overflow to infinity
and produce invalid results. For values 0 and 10 with equal weights, the median
should be 5 regardless of whether the weights are 1e-100, 1 or 1e100.
Parameterize the existing digest by weight type and use long double weights
and arithmetic for both weighted percentile overloads. This preserves finite
positive DOUBLE weights and handles accumulated mass above DBL_MAX and ranks
below DBL_TRUE_MIN. Serialize wide weights as a DOUBLE significand and a binary
exponent to avoid native long double representation differences on x86 and ARM.
Keep the FLOAT representation and existing serialized layout for unweighted
percentile functions and QUANTILE_STATE. Reject NaN and positive infinite input
weights while retaining the existing behavior of ignoring nonpositive weights.
Introduce BE execution version 18 for the wide weighted state format and
keep both legacy FLOAT overloads for versions 8 through 17. Validate weights
after conversion in the legacy path so underflowed weights cannot create
zero-weight centroids. Report the weighted aggregate's correct function name
and unwrap Nullable function names in compatibility checks to reject
incompatible stored weighted AGG_STATE data across the version 18 boundary.
Add unit and regression coverage for tiny, ordinary and large weights,
DOUBLE range boundaries, merging, serialization and execution-version
compatibility. Both weighted overloads return 5 for equal weights and 7.5 for a
1:3 ratio at each scale. Regenerate affected regression outputs for wider
arithmetic's rounding differences.
Wide weighted digests use more memory: on x86, centroid storage increases
from 8 to 32 bytes and cumulative weights from 4 to 16 bytes. No performance
benchmark was run.
### Release note
Fix BE crashes, incorrect percentiles and NaN results in
percentile_approx_weighted for finite positive DOUBLE weights outside the FLOAT
range. Full weight range support requires execution version 18. Previously
stored weighted AGG_STATE data must be rebuilt for version 18 or read with the
legacy implementation at version 17. Ordinary weighted results may change
slightly due to wider arithmetic. NaN and positive infinite weights are
rejected.
### Check List (For Author)
- Test: Unit Test / Regression Test / Manual test
- Validation recorded in the original commits; code is unchanged by this
squash
- ASAN BE and FE: ./build.sh --be --fe -j 64; existing BE Java
extensions and CDC client reused
- Final BE build after compatibility correction: ./build.sh --be -j 16
- run-be-ut.sh: all 53 selected weighted percentile, TDigest, percentile
array and QuantileState tests passed
- run-regression-test.sh: test_percentile_approx_weighted_underflow,
test_aggregate_percentile_approx_weighted,
test_aggregate_percentile_approx_array and test_agg_state_parameters passed
- Regression outputs generated by run-regression-test.sh -forceGenOut
- Manual SQL: tiny, ordinary and large equal weights return 5 at
be_exec_version=18
- clang-format 16, clang-tidy, BE build hygiene, glibc 2.17 and FE Maven
Checkstyle checks passed
- Squash validation: identical committed tree and preserved local
changes; no build or test rerun
- Behavior changed: Yes; preserve finite positive DOUBLE weights at version
18, prevent legacy underflow crashes, reject NaN and positive infinite weights,
and enforce weighted state compatibility
- Does this need documentation: No; correct existing weighted percentile
semantics
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
