lidavidm commented on PR #14066: URL: https://github.com/apache/arrow/pull/14066#issuecomment-1239720345
JEP 424 may improve this: https://openjdk.org/jeps/424 The new API lets us optionally use the garbage collector for managing memory: > All of the examples above use non-deterministic deallocation: The memory associated with the allocated segments is deallocated by the garbage collector after the memory segment instance becomes unreachable. We say that such segments are implicitly deallocated. …while still offering precise control: > Memory segments support deterministic deallocation through memory sessions. A memory session models the lifecycle of one or more memory segments. A newly-created memory session is in the alive state, which means that all the segments it manages can be accessed safely. At the client's request a memory session can be closed so that access to the segments managed by the session is no longer allowed. The MemorySession class implements the AutoCloseable interface so that memory sessions work with the [try-with-resources](https://docs.oracle.com/javase/specs/jls/se16/html/jls-14.html#jls-14.20.3) statement This is very similar to our concept of a BufferAllocator. JEP 424 would also let us implement the C Data Interface without JNI: > …there are situations where clients might only have a MemoryAddress instance, as is often the case when interacting with native code…the client can unsafely turn the address into a segment via the MemorySegment::ofAddress factory. This factory attaches fresh spatial and temporal bounds to an otherwise raw memory address in order to allow dereference operations. In the very long term, it feels like it could replace much of the implementation of the memory module. In the near term, it may let us provide an allocator implementation that does not require JVM flags on sufficiently recent versions of Java, if we can figure out how to abstract between `sun.misc.Unsafe` and the new APIs appropriately and without penalty. There are a few important caveats: - The API is a preview and cannot be used without flags yet. It will be a long time before we can use it at all, and a very very long time before we can use it exclusively. - As a replacement for `sun.misc.Unsafe` and the JNI code in the C Data Interface, it will still require JVM flags, or else _every use_ of the API will incur a warning, making it pointless: > …the use of unsafe methods in the FFM API is restricted: Their use is permitted but, by default, every such use causes a warning to be issued at run time. To allow code in a module M to use unsafe methods without warnings, specify the --enable-native-access=M option on the java command line…In a future release, it is likely that this option will be required in order to use unsafe methods. So my conclusion is that JEP 424 will eventually be useful, but we will _always_ be dependent on setting special JVM flags that are not normally needed, and so it is not a fundamental user experience improvement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
