leerho commented on PR #515: URL: https://github.com/apache/datasketches-cpp/pull/515#issuecomment-5472591476
I'm not sure I understand the motivation for this. The role of the UpdateSketch is to allow updating while retaining as much accuracy as possible. Reducing the retained entries to K significantly reduces accuracy. The role of the CompactSketch is to maintain that accuracy in an immutable and smaller form. Compared to the frequency of updates, obtaining a sketch in its compact form is normally far less frequent. Yet you complain that it is slow. That implies to me you must be doing this operation frequently. I am curious, why? Are you keeping a large number of reduced accuracy sketches around? If you subsequently merge these, your merge accuracy will be reduced also. Why in the world would you want to do that? The motivation for reducing the result to K on the ThetaUnion::getResult() is for a whole different reason. After unioning a number of theta sketches and before obtaining any 'result' or answer from the union operation, the retained entries MUST be reduced to K (actually MinK) in order to preserve order insensitivity and accuracy of the merge operation itself. And that is the role of getResult(). Not doing that will result in a dramatic increase in the variance of the estimate, basically throwing the accuracy guarantees out the window. This is why the ThetaUnion does not have a getEstimate(). To attempt to do that without reducing the retained entries to K first, would be a disaster. And why the ThetaSketch (in Java) does not provide a public trim() operation. I do note that there is a public trim() operation in the C++ version of theta_sketch.hpp, but I would argue that it is in the wrong place or should be exclusively used by the Union operations. I'm not convinced that "getResult()" makes any sense for the UpdateSketch, because the UpdateSketch itself IS the result -- for subsequent set operations or for obtaining the current estimate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
