GitHub user leerho added a comment to the discussion: Discussion about construction of DataSketches library Rust component.
We have had many discussions in the past about how to do cross-language testing, and we are always open to contributions on how to do this better. Nonetheless, we have discovered that it is not simple and has some nuances that are unique to a sketching library. - Some sketches are deterministic (e.g., HLL, Theta, CPC) but some are purely probabilistic (Classic Quantiles, KLL, REQ). - _Deterministic_ means given the same input in the same order, the sketch will produce the same exact result (which is not exact). Order sensitivity is another topic in itself. - The probabilistic sketches are designed that way in order to improve accuracy and reduce sensitivity to data ordering. Attempts to force the probabilistic sketches to be deterministic will undermine the mathematical guarantees of error distribution, which are fundamentally probabilistic. Please don't do that. - This means, especially for the probabilistic sketches, that two equally configured sketches reading the same serialized image will result in slightly different results. Both results should be within the _confidence interval_ of error determined by the test. If not, that is when, and only when, you can declare an error failure of the test. - What this implies about cross-platform testing is that we consider two different language implementations of the same sketch algorithm to be "binary" compatible if both sketches produce a result that is within the defined bounds of the test. - It does not mean that two different language implementations of the same sketch algorithm will produce the same exact serialized image. A byte for byte comparison of the two sketch images will often not be the same, because of floating-point rounding issues, which can be different across platforms and languages, as well as probabilistic behavior. In terms of how to implement this kind of testing we have come up with a somewhat ad-hoc solution: - Each language implements a public _profile_ or script that produces a broad set of serialized images from its own sketches. - Then a GitHub/Actions/workflow in Java, for example, will check out and run the script in DS-CPP that generates the C++ sketch images, copies the files over to a DS-Java implementation that it has also checked out, and runs the DS-java _profile_ that loads and tests those sketch images for compliance. And the reverse is implemented on the C++ side. We haven't really discussed how this should work with a growing number of languages. Perhaps we need to choose a common "master" language? That has issues too. Cheers. GitHub link: https://github.com/apache/datasketches-rust/discussions/4#discussioncomment-15226056 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
