clintropolis commented on a change in pull request #6397: Adds bloom filter
aggregator to 'druid-bloom-filters' extension
URL: https://github.com/apache/incubator-druid/pull/6397#discussion_r249213055
##########
File path:
extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomKFilter.java
##########
@@ -210,6 +220,261 @@ public static void mergeBloomFilterBytes(
}
}
+ public static void serialize(ByteBuffer out, BloomKFilter bloomFilter)
+ {
+ serialize(out, out.position(), bloomFilter);
+ }
+
+ /**
+ * Serialize a bloom filter to a ByteBuffer. Does not mutate buffer position.
+ *
+ * @param out output buffer to write to
+ * @param position output buffer position
+ * @param bloomFilter BloomKFilter that needs to be seralized
+ */
+ public static void serialize(ByteBuffer out, int position, BloomKFilter
bloomFilter)
+ {
+ /**
+ * Serialized BloomKFilter format:
+ * 1 byte for the number of hash functions.
+ * 1 big endian int(to match OutputStream) for the number of longs in the
bitset
+ * big endian longs in the BloomKFilter bitset
+ */
+ ByteBuffer view = out.duplicate().order(ByteOrder.BIG_ENDIAN);
+ view.position(position);
+ view.put((byte) bloomFilter.k);
+ view.putInt(bloomFilter.getBitSet().length);
+ for (long value : bloomFilter.getBitSet()) {
+ view.putLong(value);
+ }
+ }
+
+ public static BloomKFilter deserialize(ByteBuffer in) throws IOException
+ {
+ return deserialize(in, in.position());
+ }
+
+ /**
+ * Deserialize a bloom filter
+ * Read a byte buffer, which was written by {@linkplain
#serialize(OutputStream, BloomKFilter)} or
+ * {@linkplain #serialize(ByteBuffer, int, BloomKFilter)}
+ * into a {@code BloomKFilter}. Does not mutate buffer position.
+ *
+ * @param in input ByteBuffer
+ *
+ * @return deserialized BloomKFilter
+ */
+ public static BloomKFilter deserialize(ByteBuffer in, int position) throws
IOException
+ {
+ if (in == null) {
+ throw new IOException("Input stream is null");
+ }
+
+ try {
+ ByteBuffer dataBuffer = in.duplicate().order(ByteOrder.BIG_ENDIAN);
+ dataBuffer.position(position);
+ int numHashFunc = dataBuffer.get();
+ int bitsetArrayLen = dataBuffer.getInt();
+ long[] data = new long[bitsetArrayLen];
+ for (int i = 0; i < bitsetArrayLen; i++) {
+ data[i] = dataBuffer.getLong();
+ }
+ return new BloomKFilter(data, numHashFunc);
+ }
+ catch (RuntimeException e) {
+ IOException io = new IOException("Unable to deserialize BloomKFilter");
Review comment:
Fixed, I had just copied the instance methods as the basis of the
`ByteBuffer` methods and that's how it did this part.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
