leerho commented on issue #693:
URL:
https://github.com/apache/datasketches-java/issues/693#issuecomment-3652384779
@thomasrebele.
I don't see in your replies that you have explored the possibility of
changing the expectation to allow for slight variance in the computed result.
What is preventing that? Is this a human expectation or a software tool
expectation?
The case that is important to you, "merging n KLL sketches to a single one"
is also important to us as it is the most important case where sketching
requires the probabilistic behavior to maintain the guarantee of accuracy.
Even if you could fix the seed of the RNG, in a large clustered environment
it is hard to control the order that results come in from the various nodes,
and changing the order of the input will also create slight variance in the
output. Unless you force the strict order of inputs, and with thousands of
input nodes, that will be very expensive both processing cost and time cost.
As long as you accept the probabilistic behavior of results being within a
tolerance, sketching is insensitive to order! This means that merging _n_
sketches from thousands of nodes can be performed as fast as the data comes in.
No need to sort or control the order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
