leventov opened a new issue #3892: Move from ByteBuffer to Memory URL: https://github.com/apache/incubator-druid/issues/3892 The goal of this issue is to agree on basic design principles and expectations from this refactoring. **Goals** - Speed up critical places in query processing by skipping bound checks, which `ByteBuffer` always performs when making any read or write. Bound checks could be hoisted outside the hot loops manually or enabled via assertions/static final boolean flag, [that is effectively free with some precautions](http://pveentjer.blogspot.mx/2017/01/final-static-boolean-jit.html). - Get rid or 2 GB limitation, use `long` indexes. See #3743. - Don't make a lot of Object copies when a `Memory` object has to be "sliced", "converted to read-only", etc. See https://github.com/druid-io/druid/pull/3716#issuecomment-274658045. - <span>+</span> Eliminate current Endian craziness in Druid by standardizing on native-endian. https://github.com/druid-io/druid/issues/3892#issuecomment-276235357 **Design** - Based on DataSketches's [Memory](https://github.com/DataSketches/sketches-core/blob/master/memory/src/main/java/com/yahoo/memory/Memory.java). - <strike>It is moved to Druid source tree, not used as a dependency.</strike> See https://github.com/druid-io/druid/issues/3892#issuecomment-276589924 - `Memory` object is immutable. "position and limit", if needed, are passed along with `Memory` as two primitive longs. - Upon creation a mutable `Memory` object has a cached immutable "view" object (which implements read methods and throws exception from write methods), this object is always returned when "as read only" is needed. - Explicit `close()` or `free()` on `Memory` is possible, but not strictly required, there is a `sun.misc.Cleaner` safety net, as well as in `DirectByteBuffer`. - While transition from `ByteBuffer` to `Memory` is in progress (<strike>not expected to be done all at once, but subsystem by subsystem, class by class</strike> see https://github.com/druid-io/druid/issues/3892#issuecomment-276548114), and also when we need to interop with external libraries which require `ByteBuffer`, conversion from `Memory` to `ByteBuffer` and vice versa is possible. Likely it requires to - Use `DirectByteBuffer`-compatible format of `Cleaner` inside `Memory` - Be able to access private `DirectByteBuffer` constructors [via `MagicAccessorImpl`](https://github.com/OpenHFT/Java-Lang/blob/7f6ca8aceb4dbcaea83f62d1d85542d6f374515a/lang/src/main/java/net/openhft/lang/io/ByteBufferReuse.java). - <span>+</span> "Global" (immutable) `Memory`'s bounds are checked optionally via assertions/guarded by static final boolean flag, "local" position and limit are check explicitly manually, with helper methods or versions of read and write methods of Memory, which accept read/write position and "local" limits. https://github.com/druid-io/druid/issues/3892#issuecomment-276185704 Objections, additions, corrections, questions are welcome. @leerho @cheddar @weijietong @akashdw @himanshug @fjy @niketh
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
