Victsm commented on a change in pull request #29855:
URL: https://github.com/apache/spark/pull/29855#discussion_r498409133
##########
File path:
common/network-common/src/main/java/org/apache/spark/network/protocol/Encoders.java
##########
@@ -44,6 +46,42 @@ public static String decode(ByteBuf buf) {
}
}
+ /** Bitmaps are encoded with their serialization length followed by the
serialization bytes. */
+ public static class Bitmaps {
+ public static int encodedLength(RoaringBitmap b) {
+ // Compress the bitmap before serializing it. Note that since
BlockTransferMessage
+ // needs to invoke encodedLength first to figure out the length for the
ByteBuf, it
+ // guarantees that the bitmap will always be compressed before being
serialized.
+ b.trim();
+ b.runOptimize();
+ return 4 + b.serializedSizeInBytes();
+ }
+
+ public static void encode(ByteBuf buf, RoaringBitmap b) {
+ int encodedLength = b.serializedSizeInBytes();
+ buf.writeInt(encodedLength);
+ // RoaringBitmap requires nio ByteBuffer for serde. We expose the netty
ByteBuf as a nio
+ // ByteBuffer. Here, we need to explicitly manage the index so we can
write into the
+ // ByteBuffer, and the write is reflected in the underneath ByteBuf.
+ b.serialize(buf.nioBuffer(buf.writerIndex(), encodedLength));
+ buf.writerIndex(buf.writerIndex() + encodedLength);
+ }
+
+ public static RoaringBitmap decode(ByteBuf buf) {
+ int length = buf.readInt();
Review comment:
That's a good point.
The position of the underlying nio buffer does not change after
deserialization.
I was previously unsure if we can safely rely on
bitmap.serializedSizeInBytes right after deserialization to get the number of
bytes read from the buffer.
However, according to the RoaringBitmap API doc, that's actually the
suggested way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
