davisusanibar opened a new pull request, #13072: URL: https://github.com/apache/arrow/pull/13072
Proposal to upgrade Arrow Java to [Module System](https://openjdk.java.net/jeps/261). Please note: This PR is not consider to be part of next release 10.0.0, the plan is to test this on 11.0.0 version. **Current status** 1. Apache Arrow Java is compiled and tested by JDK 8, 11, 17, 18 but packaged with source/target JDK8 **Changes implemented by this PR** 1. Apache Arrow Java project are continuing compiled and tested by JDK 8, 11, 17, 18 but packaged with source/target JDK8 2. If Apache Arrow Java project is building with JDK11+ validate the project and test it against the module-info.java definitions 3. There are not CI implementation to deliver Jar library with JDK8 & JDK11+ (we need to define what will be our next step for JDK versions) 4. We probably are breaking APIs because we are changing package name to ensure unique package names on the project required by module-info.java 5. We are added a new project that contains Apache Arrow Netty custom implementation needed for 6. There are remaining modules pending to implement modules with module-info.java files. This PR contains modules-info.java for: Format / Memory / Vector 7. Is pending to review errors at Integration testing **Next steps** 1. Implement cross-compilation with JDK 11+ and packaging with release=8 (continue offering support for JDK8): That is not posible, if we compile with --release=8 we will have problems to access sun.misc.Unsafe that is not as exposed/public/documented API for N >= 9: -source N -target N --system <documented-APIs-from-N>. (https://openjdk.org/jeps/247) 2. We have these options for our next steps: - **Option 1**.- Maintain all as it is now, continue packaging with source/target JDK8 - **Option 2**.- Change/Update/Modify use of sun.misc.* classes dependencies to be able to support cross-compilation (--release 8) to packaging with JDK11+ and continue offering support for JDK8 also ```` jdeps --jdk-internals arrow-memory-unsafe-10.0.0-SNAPSHOT.jar arrow-memory-unsafe-10.0.0-SNAPSHOT.jar -> jdk.unsupported org.apache.arrow.memory.unsafe.UnsafeAllocationManager -> sun.misc.Unsafe JDK internal API (jdk.unsupported) Warning: JDK internal APIs are unsupported and private to JDK implementation that are subject to be removed or changed incompatibly and could break your application. Please modify your code to eliminate dependence on any JDK internal APIs. For the most recent update on JDK internal API replacements, please check: https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool JDK Internal API Suggested Replacement ---------------- --------------------- sun.misc.Unsafe See http://openjdk.java.net/jeps/260 ```` - **Option 3**.- _Drop support for JDK8_ and starting offering support to packaging with JDK11+, this will be the right approach, but, this thread does not have an agreement about to drop JDK8 support ([JAVA] JDK Support Policy - https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj) If you decided to push some of this options please use that [ML](https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj) to move on next steps. - - - - - - - - - - - - - - - - - - - - - - - **Detailed steps for current implementation** JPMS Planning: - [x] Migrate Arrow Java Format - [x] Migrate Arrow Java Memory Core - [x] Migrate Arrow Java Memory Netty Not possible to apply module system for the package io.netty.buffer, it was needed to create another module `memory-netty-buffer-patch` and patch this package with our arrow java io.netty.buffer code. - [x] Migrate Arrow Java Memory Unsafe TODO: Need to refactor protected access methods from unit test. - [x] Migrate Arrow Java Vector TODO: Need to refactor protected access methods from unit test. - [ ] Migrate Arrow Java Flight Blocker: Current grpc-api and grpc-context are not modularized and cannot be used as a module because both of them has the same package named io.grpc - **JPMS Java Platform Module System** Java modules enforced new restrictions on existing code: - Package need to be uniqueness - There are some modules with the same package name (i.e. org.apache.arrow.memory / io.netty.buffer) that is consumed by another module with the same package name to access **protected** methods. - Java Memory module problems: - Core / Netty / Unsafe has the same package name org.apache.arrow.memory and there are not problems to consume protected methods, if we change the name of the package we need to define a way about how Netty module could consume for example Core protected methods. - Java Netty extends io.netty.buffer capabilities: - Is needed to create a new module arrow-memory-netty-buffer-patch to add new Java functionalities added and this need to be consumed by Arrow Memory Netty. - Dependencies also need to be modularized: - Current jar **dependencies** need to be modularized to be consumed as a module if not there are no way to convert our project to module if any dependencies are not modularized before. - Arrow Format: [x] - Arrow Memory: [x] - Arrow Vector: [x] - Arrow Flight: - Blocker: Current grpc-api and grpc-context are not modularized and cannot be used as a module because both of them has the same package named io.grpc. Summarize changes and benefits to migrate Arrow Java with Module System: - JPMS offer Strong encapsulation, Well-defined interfaces, Explicit dependencies. (3)(4) - JPMS offer reliable configuration and security to hide platform internals. - JPMS offer a partial solution to solve problema about read (80%) /write (20%) code. - JPMS offer optimization for readability about read/write ratio (90/10) thru module-info.java. - Consistency logs, JPMS implement consistency logs to really use that to solve the current problem. - Be able to customize JRE needed with only modules needed (not java.desktop for example and others) thru JLink. - Modules is also been implemented by other language such as Javascript (ES2015), C++(C++20), Net (Nuget/NetCore) and this is some bus that at some time we could onboard. - Consider tho take a lokk at this discussion about pro/cons (5). (3): https://nipafx.dev/java-modules-reflection-vs-encapsulation/ (4): https://github.com/nipafx/demo-jigsaw-reflection (5): https://www.reddit.com/r/java/comments/okt3j3/do_you_use_jigsaw_modules_in_your_java_projects/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
