richardstartin edited a comment on issue #7408: upgrade to RoaringBitmap 0.8.0 and serialise directly to ByteBuffer URL: https://github.com/apache/incubator-druid/pull/7408#issuecomment-479694073 I doubt druid bottlenecks on serialising bitmaps, but I wrote this new method in RoaringBitmap for an application that does, and it may be useful here. Here's some quick benchmark evidence, taking a look at how long `WrappedRoaringBitmap.serialize` takes before and after the change. ```java @Benchmark public void serialize(SerializeBenchmarkState state, Blackhole bh) { state.bitmap.serialize(state.buffer); bh.consume(state.buffer); state.buffer.flip(); } @State(Scope.Benchmark) public static class SerializeBenchmarkState { WrappedRoaringBitmap bitmap; ByteBuffer buffer; @Setup public void setup(BitmapIterationBenchmark state) { bitmap = new WrappedRoaringBitmap(); Random random = ThreadLocalRandom.current(); for (int bit = 0; bit < state.size; bit++) { if (random.nextDouble() < state.prob) { bitmap.add(bit); } } buffer = ByteBuffer.allocate(bitmap.getSizeInBytes()); } } ``` ``` 8c104a115c4e08b2df60d940d6d85a7e79bef752 (incubator-druid:master) Benchmark (bitmapAlgo) (prob) (size) Mode Cnt Score Error Units BitmapIterationBenchmark.serialize roaring 0.0 1000000 avgt 10 29.110 ± 0.459 ns/op BitmapIterationBenchmark.serialize roaring 0.001 1000000 avgt 10 10624.144 ± 139.993 ns/op BitmapIterationBenchmark.serialize roaring 0.1 1000000 avgt 10 187265.856 ± 5515.013 ns/op BitmapIterationBenchmark.serialize roaring 0.5 1000000 avgt 10 184694.297 ± 8353.848 ns/op BitmapIterationBenchmark.serialize roaring 0.99 1000000 avgt 10 233570.949 ± 67602.280 ns/op BitmapIterationBenchmark.serialize roaring 1.0 1000000 avgt 10 1083.100 ± 38.245 ns/op 32673c121edea450b84ffef767506b60a49f4a23 (richardstartin:faster-roaring-serialize) Benchmark (bitmapAlgo) (prob) (size) Mode Cnt Score Error Units BitmapIterationBenchmark.serialize roaring 0.0 1000000 avgt 10 10.510 ± 0.182 ns/op BitmapIterationBenchmark.serialize roaring 0.001 1000000 avgt 10 957.783 ± 12.215 ns/op BitmapIterationBenchmark.serialize roaring 0.1 1000000 avgt 10 54361.198 ± 655.209 ns/op BitmapIterationBenchmark.serialize roaring 0.5 1000000 avgt 10 53632.341 ± 377.866 ns/op BitmapIterationBenchmark.serialize roaring 0.99 1000000 avgt 10 16405.135 ± 271.594 ns/op BitmapIterationBenchmark.serialize roaring 1.0 1000000 avgt 10 404.527 ± 3.868 ns/op ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
