tisonkun opened a new pull request, #765:
URL: https://github.com/apache/datasketches-java/pull/765
Fixes #702.
`TDigestDouble` can still store an infinite centroid mean when merging
finite, same-sign extreme values with large weights. The incremental
calculation overflows and its fallback can also overflow because it
independently rounds both normalized weights. Quantile interpolation has the
same issue, and a deserialized last centroid with weight 2 can produce `NaN`
through division by zero.
Normalize the merge weight before multiplication and use fused interpolation
when the difference is finite. For quantile interpolation, use a sign-aware
weighted average; handle the two-sample right-tail boundary explicitly.
Also complete deserialization validation across native double/float and
reference double/float formats:
- Reject fractional, non-finite, or out-of-range reference weights before
narrowing to `long`.
- Reject overflowing total weights, including buffered values. Reject
overflowing updates and merges before changing the destination.
- Check flags, counts, payload lengths, ordered extrema, centroid ordering,
and centroid/buffer membership in the stored range before accepting the image.
- Preserve valid reference compact images: their float means can round
outside double extrema, so validate in float precision and clamp those means to
the original bounds.
## Rust comparison
This completes the relevant fixes from apache/datasketches-rust#88,
apache/datasketches-rust#173, apache/datasketches-rust#250, and
apache/datasketches-rust#262. Main already rejects non-finite updates and
nonpositive native weights, and already fixes the swapped quantile
interpolation weights and left-tail rank normalization. The regressions above
remained on base commit `90319c208c99ee97b3bbfcb7e3ed911b4909884d`.
## Validation
- Confirmed regression failures against the unchanged main implementation,
including an infinite stored centroid mean, infinite quantile, and `NaN` at the
two-sample right tail.
- `mvn -B clean verify`: 2,255 tests passed, including the new numeric and
malformed-input regressions; Javadoc and RAT checks passed.
- `TDigestCrossLanguageTest`: all 5 selected test methods passed against 32
C++, 16 Go, 8 Java, and 2 reference TDigest snapshots. The cross-language
snapshots came from DataSketches TCK revision
`c0a180708c6e6433e4cba7fba091713eb8af3eaa`.
```sh
mvn -B test -Pcheck_all_language_files \
-Dtest=TDigestCrossLanguageTest \
-Dtestng.all_groups=check_cpp_files,check_go_files,check_java_files,check_cpp_historical_files
\
-Dtest.data.root=/tmp/issue702-compat
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
