jaideeppyne opened a new pull request, #188: URL: https://github.com/apache/datasketches-go/pull/188
`Quantile`'s right tail divides by `lastWeight/2 - 1`. That denominator is zero when the last centroid weighs 2, and the only rank that reaches the branch (one unit of weight below the total) has a zero numerator as well, so the result is NaN. Java and Rust already return the stored maximum in this case. This does the same. The rank weight is also rounded before it is subtracted from the total. On arm64 the compiler otherwise fuses that multiply and subtract, skips the right tail, and returns a value past the maximum (40 for a sketch whose maximum is 20). Tests: `go test ./tdigest/`. The new case builds centroids of weight 10 and 2 and checks `Quantile(11/12)` is the maximum. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
