davecromberge commented on code in PR #12042:
URL: https://github.com/apache/pinot/pull/12042#discussion_r1408415739
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/CustomSerDeUtils.java:
##########
@@ -243,9 +243,14 @@ public TDigest deserialize(ByteBuffer byteBuffer) {
@Override
public byte[] serialize(Sketch value) {
- // NOTE: Compact the sketch in unsorted, on-heap fashion for performance
concern.
- // See https://datasketches.apache.org/docs/Theta/ThetaSize.html
for more details.
- return value.compact(false, null).toByteArray();
+ // The serializer should respect existing ordering to enable "early stop"
+ // optimisations on unions.
+ boolean shouldCompact = !value.isCompact();
+ boolean shouldOrder = value.isOrdered();
+ if (shouldCompact) {
Review Comment:
No - this is not the case and it depends on how the ingested sketches have
been processed by the end user. It might be the case that this is true for the
empty and singleton sketch (sketch of a single hashed item), but not for others.
The emphasis on these changes is that the serialiser does not alter the
current ordering on existing sketches. There is a small performance
optimisation to this in that there is no need to re-compact or re-order
sketches that are already in that state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
