davecromberge commented on code in PR #12042:
URL: https://github.com/apache/pinot/pull/12042#discussion_r1414237048
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/CustomSerDeUtils.java:
##########
@@ -243,9 +243,14 @@ public TDigest deserialize(ByteBuffer byteBuffer) {
@Override
public byte[] serialize(Sketch value) {
- // NOTE: Compact the sketch in unsorted, on-heap fashion for performance
concern.
- // See https://datasketches.apache.org/docs/Theta/ThetaSize.html
for more details.
- return value.compact(false, null).toByteArray();
+ // The serializer should respect existing ordering to enable "early stop"
+ // optimisations on unions.
+ boolean shouldCompact = !value.isCompact();
+ boolean shouldOrder = value.isOrdered();
Review Comment:
I think those points are valid - but I thought that it would be best to
leave the decision as to whether to order in the hands of the end user. The
current version respects the input ordering.
Moreover, the sketch is not re-compacted if it is already in compacted form.
This is a rather subjective take on the problem, and I will go with your
recommendation on this one @snleee - what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
