This is an automated email from the ASF dual-hosted git repository.
lidavidm pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new e7ab540ed9 GH-38998: [Java] Build memory-core and memory-unsafe as
JPMS modules (#39011)
e7ab540ed9 is described below
commit e7ab540ed9901d19ebde81f43e8417562ef8a334
Author: James Duong <[email protected]>
AuthorDate: Wed Jan 10 05:30:21 2024 -0800
GH-38998: [Java] Build memory-core and memory-unsafe as JPMS modules
(#39011)
Depends on #39134
### Rationale for this change
Part of modularizing Arrow. Allows these components to be used by tools
that require JPMS modules such as jlink.
### What changes are included in this PR?
- Refactor memory modules so that all netty code is in netty-memory.
- All code being injected into Netty packages and have dependencies on
Netty's package-private code go into a new module memory-netty-buffer-patch.
- The surefire command line has been changed to allow reflection on unsafe
to be used by arrow-memory-core
- Add module-info files for arrow-memory-core and arrow-memory-unsafe
### Are these changes tested?
Yes, existing tests work when run with modules.
### Are there any user-facing changes?
Yes, users now need to put
--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED on
their java command line instead of --add-opens=java.base/java.nio=ALL-UNNAMED
since memory-core is now a named module instead of unnamed.
**This PR includes breaking changes to public APIs.**
Yes, users now need to put
--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED on
their java command line instead of --add-opens=java.base/java.nio=ALL-UNNAMED
since memory-core is now a named module instead of unnamed.
* Closes: #38998
Authored-by: James Duong <[email protected]>
Signed-off-by: David Li <[email protected]>
---
dev/archery/archery/integration/tester_java.py | 5 +-
docs/source/java/install.rst | 6 +-
java/c/pom.xml | 5 +
java/gandiva/pom.xml | 2 +-
.../module-info-compiler-maven-plugin/pom.xml | 4 +-
java/memory/memory-core/pom.xml | 25 +++-
.../src/main/java/module-info.java} | 26 ++--
.../org/apache/arrow/memory/CheckAllocator.java | 39 +++++-
.../memory/DefaultAllocationManagerOption.java | 12 +-
.../org/apache/arrow/memory/util/MemoryUtil.java | 2 +-
.../arrow/memory/CountingAllocationListener.java | 10 ++
.../java/org/apache/arrow/memory/TestArrowBuf.java | 8 +-
.../org/apache/arrow/memory/TestBaseAllocator.java | 149 ++++++++-------------
.../apache/arrow/memory/TestForeignAllocation.java | 7 +
.../pom.xml | 24 +++-
.../java/io/netty/buffer/ExpandableByteBuf.java | 0
.../src/main/java/io/netty/buffer/LargeBuffer.java | 0
.../io/netty/buffer/MutableWrappedByteBuf.java | 0
.../main/java/io/netty/buffer/NettyArrowBuf.java | 16 ++-
.../io/netty/buffer/PooledByteBufAllocatorL.java | 7 +-
.../io/netty/buffer/UnsafeDirectLittleEndian.java | 0
.../arrow/memory/patch}/ArrowByteBufAllocator.java | 4 +-
.../netty/buffer/TestUnsafeDirectLittleEndian.java | 5 +
java/memory/memory-netty/pom.xml | 7 +
.../DefaultAllocationManagerFactory.java | 6 +-
.../memory/{ => netty}/NettyAllocationManager.java | 7 +-
.../io/netty/buffer/TestExpandableByteBuf.java | 4 +
.../java/io/netty/buffer/TestNettyArrowBuf.java | 5 +-
.../memory/{ => netty}/ITTestLargeArrowBuf.java | 5 +-
.../{ => netty}/TestAllocationManagerNetty.java | 4 +-
.../memory/{ => netty}/TestEmptyArrowBuf.java | 9 +-
.../arrow/memory/{ => netty}/TestEndianness.java | 4 +-
.../{ => netty}/TestNettyAllocationManager.java | 15 ++-
.../arrow/memory/netty/TestNettyAllocator.java | 81 +++++++++++
java/memory/memory-unsafe/pom.xml | 4 +
.../src/main/java/module-info.java} | 20 +--
.../DefaultAllocationManagerFactory.java | 6 +-
.../{ => unsafe}/UnsafeAllocationManager.java | 6 +-
.../{ => unsafe}/TestAllocationManagerUnsafe.java | 4 +-
.../{ => unsafe}/TestUnsafeAllocationManager.java | 13 +-
java/memory/pom.xml | 1 +
java/performance/pom.xml | 2 +-
java/pom.xml | 20 ++-
43 files changed, 387 insertions(+), 192 deletions(-)
diff --git a/dev/archery/archery/integration/tester_java.py
b/dev/archery/archery/integration/tester_java.py
index 6cd1afa64f..032ac13e74 100644
--- a/dev/archery/archery/integration/tester_java.py
+++ b/dev/archery/archery/integration/tester_java.py
@@ -40,7 +40,7 @@ def load_version_from_pom():
_JAVA_OPTS = [
"-Dio.netty.tryReflectionSetAccessible=true",
"-Darrow.struct.conflict.policy=CONFLICT_APPEND",
- "--add-opens=java.base/java.nio=ALL-UNNAMED",
+ "--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED",
# GH-39113: avoid failures accessing files in `/tmp/hsperfdata_...`
"-XX:-UsePerfData",
]
@@ -247,7 +247,8 @@ class JavaTester(Tester):
if 'Unrecognized option: --add-opens' not in proc.stderr:
# Java 9+
self._java_opts.append(
- '--add-opens=java.base/java.nio=ALL-UNNAMED')
+ '--add-opens=java.base/java.nio='
+ 'org.apache.arrow.memory.core,ALL-UNNAMED')
def _run(self, arrow_path=None, json_path=None, command='VALIDATE'):
cmd = (
diff --git a/docs/source/java/install.rst b/docs/source/java/install.rst
index 32c121573a..b7484536f2 100644
--- a/docs/source/java/install.rst
+++ b/docs/source/java/install.rst
@@ -33,14 +33,14 @@ Java modules are compatible with JDK 8 and above.
Currently, JDK 8, 11, 17, and 21 are tested in CI.
When using Java 9 or later, some JDK internals must be exposed by
-adding ``--add-opens=java.base/java.nio=ALL-UNNAMED`` to the ``java`` command:
+adding
``--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED`` to
the ``java`` command:
.. code-block:: shell
# Directly on the command line
- $ java --add-opens=java.base/java.nio=ALL-UNNAMED -jar ...
+ $ java
--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -jar ...
# Indirectly via environment variables
- $ env _JAVA_OPTIONS="--add-opens=java.base/java.nio=ALL-UNNAMED" java -jar
...
+ $ env
_JAVA_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED"
java -jar ...
Otherwise, you may see errors like ``module java.base does not "opens
java.nio" to unnamed module``.
diff --git a/java/c/pom.xml b/java/c/pom.xml
index d66379d356..8fc3f36994 100644
--- a/java/c/pom.xml
+++ b/java/c/pom.xml
@@ -53,6 +53,11 @@
<artifactId>arrow-memory-unsafe</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-format</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
diff --git a/java/gandiva/pom.xml b/java/gandiva/pom.xml
index 128fa1508f..e837a09ff8 100644
--- a/java/gandiva/pom.xml
+++ b/java/gandiva/pom.xml
@@ -79,7 +79,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
- <version>2.9.1</version>
+ <version>3.6.3</version>
<executions>
<execution>
<id>attach-javadocs</id>
diff --git a/java/maven/module-info-compiler-maven-plugin/pom.xml
b/java/maven/module-info-compiler-maven-plugin/pom.xml
index 837a78b43b..46c0d563f4 100644
--- a/java/maven/module-info-compiler-maven-plugin/pom.xml
+++ b/java/maven/module-info-compiler-maven-plugin/pom.xml
@@ -125,6 +125,4 @@
</plugin>
</plugins>
</build>
-
-
-</project>
\ No newline at end of file
+</project>
diff --git a/java/memory/memory-core/pom.xml b/java/memory/memory-core/pom.xml
index b914b1fa10..6e411c0cd5 100644
--- a/java/memory/memory-core/pom.xml
+++ b/java/memory/memory-core/pom.xml
@@ -46,7 +46,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
-
<configuration>
<excludes>
<!-- Test is only useful when NOT running with add-opens -->
@@ -58,6 +57,30 @@
</build>
<profiles>
+ <profile>
+ <id>jdk11+</id>
+ <activation>
+ <jdk>[11,]</jdk>
+ <property>
+ <name>!m2e.version</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration combine.self="override">
+
<argLine>--add-reads=org.apache.arrow.memory.core=ch.qos.logback.classic
--add-opens=java.base/java.lang.reflect=org.apache.arrow.memory.core
--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED</argLine>
+ <excludes>
+ <!-- Test is only useful when NOT running with add-opens -->
+ <exclude>**/TestOpens.java</exclude>
+ </excludes>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
<profile>
<id>opens-tests</id>
<!-- Run tests WITHOUT add-opens to make sure we fail-fast -->
diff --git
a/java/memory/memory-netty/src/main/java/io/netty/buffer/LargeBuffer.java
b/java/memory/memory-core/src/main/java/module-info.java
similarity index 67%
copy from
java/memory/memory-netty/src/main/java/io/netty/buffer/LargeBuffer.java
copy to java/memory/memory-core/src/main/java/module-info.java
index 792b3b8148..34ba34e80b 100644
--- a/java/memory/memory-netty/src/main/java/io/netty/buffer/LargeBuffer.java
+++ b/java/memory/memory-core/src/main/java/module-info.java
@@ -15,20 +15,14 @@
* limitations under the License.
*/
-package io.netty.buffer;
-
-/**
- * A MutableWrappedByteBuf that also maintains a metric of the number of huge
buffer bytes and
- * counts.
- */
-public class LargeBuffer extends MutableWrappedByteBuf {
-
- public LargeBuffer(ByteBuf buffer) {
- super(buffer);
- }
-
- @Override
- public ByteBuf copy(int index, int length) {
- return new LargeBuffer(buffer.copy(index, length));
- }
+module org.apache.arrow.memory.core {
+ exports org.apache.arrow.memory;
+ exports org.apache.arrow.memory.rounding;
+ exports org.apache.arrow.memory.util;
+ exports org.apache.arrow.memory.util.hash;
+ exports org.apache.arrow.util;
+ requires transitive jdk.unsupported;
+ requires jsr305;
+ requires org.immutables.value;
+ requires org.slf4j;
}
diff --git
a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/CheckAllocator.java
b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/CheckAllocator.java
index 79b825aa2e..dac4a3fcff 100644
---
a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/CheckAllocator.java
+++
b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/CheckAllocator.java
@@ -31,20 +31,35 @@ import org.slf4j.LoggerFactory;
*/
final class CheckAllocator {
private static final Logger logger =
LoggerFactory.getLogger(CheckAllocator.class);
- private static final String ALLOCATOR_PATH =
"org/apache/arrow/memory/DefaultAllocationManagerFactory.class";
+ // unique package names needed by JPMS module naming
+ private static final String ALLOCATOR_PATH_CORE =
+ "org/apache/arrow/memory/DefaultAllocationManagerFactory.class";
+ private static final String ALLOCATOR_PATH_UNSAFE =
+ "org/apache/arrow/memory/unsafe/DefaultAllocationManagerFactory.class";
+ private static final String ALLOCATOR_PATH_NETTY =
+ "org/apache/arrow/memory/netty/DefaultAllocationManagerFactory.class";
private CheckAllocator() {
-
}
static String check() {
Set<URL> urls = scanClasspath();
URL rootAllocator = assertOnlyOne(urls);
reportResult(rootAllocator);
- return "org.apache.arrow.memory.DefaultAllocationManagerFactory";
+ if (rootAllocator.getPath().contains("memory-core") ||
+ rootAllocator.getPath().contains("/org/apache/arrow/memory/core/")) {
+ return "org.apache.arrow.memory.DefaultAllocationManagerFactory";
+ } else if (rootAllocator.getPath().contains("memory-unsafe") ||
+ rootAllocator.getPath().contains("/org/apache/arrow/memory/unsafe/")) {
+ return "org.apache.arrow.memory.unsafe.DefaultAllocationManagerFactory";
+ } else if (rootAllocator.getPath().contains("memory-netty") ||
+ rootAllocator.getPath().contains("/org/apache/arrow/memory/netty/")) {
+ return "org.apache.arrow.memory.netty.DefaultAllocationManagerFactory";
+ } else {
+ throw new IllegalStateException("Unknown allocation manager type to
infer. Current: " + rootAllocator.getPath());
+ }
}
-
private static Set<URL> scanClasspath() {
// LinkedHashSet appropriate here because it preserves insertion order
// during iteration
@@ -53,9 +68,21 @@ final class CheckAllocator {
ClassLoader allocatorClassLoader = CheckAllocator.class.getClassLoader();
Enumeration<URL> paths;
if (allocatorClassLoader == null) {
- paths = ClassLoader.getSystemResources(ALLOCATOR_PATH);
+ paths = ClassLoader.getSystemResources(ALLOCATOR_PATH_CORE);
+ if (!paths.hasMoreElements()) {
+ paths = ClassLoader.getSystemResources(ALLOCATOR_PATH_UNSAFE);
+ }
+ if (!paths.hasMoreElements()) {
+ paths = ClassLoader.getSystemResources(ALLOCATOR_PATH_NETTY);
+ }
} else {
- paths = allocatorClassLoader.getResources(ALLOCATOR_PATH);
+ paths = allocatorClassLoader.getResources(ALLOCATOR_PATH_CORE);
+ if (!paths.hasMoreElements()) {
+ paths = allocatorClassLoader.getResources(ALLOCATOR_PATH_UNSAFE);
+ }
+ if (!paths.hasMoreElements()) {
+ paths = allocatorClassLoader.getResources(ALLOCATOR_PATH_NETTY);
+ }
}
while (paths.hasMoreElements()) {
URL path = paths.nextElement();
diff --git
a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/DefaultAllocationManagerOption.java
b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/DefaultAllocationManagerOption.java
index d57b72ba41..6d408abcc3 100644
---
a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/DefaultAllocationManagerOption.java
+++
b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/DefaultAllocationManagerOption.java
@@ -19,6 +19,7 @@ package org.apache.arrow.memory;
import java.lang.reflect.Field;
+import org.apache.arrow.util.VisibleForTesting;
import org.checkerframework.checker.nullness.qual.Nullable;
@@ -64,8 +65,13 @@ public class DefaultAllocationManagerOption {
Unknown,
}
+ /**
+ * Returns the default allocation manager type.
+ * @return the default allocation manager type.
+ */
@SuppressWarnings("nullness:argument") //enum types valueOf are implicitly
non-null
- static AllocationManagerType getDefaultAllocationManagerType() {
+ @VisibleForTesting
+ public static AllocationManagerType getDefaultAllocationManagerType() {
AllocationManagerType ret = AllocationManagerType.Unknown;
try {
@@ -122,7 +128,7 @@ public class DefaultAllocationManagerOption {
private static AllocationManager.Factory getUnsafeFactory() {
try {
- return getFactory("org.apache.arrow.memory.UnsafeAllocationManager");
+ return
getFactory("org.apache.arrow.memory.unsafe.UnsafeAllocationManager");
} catch (RuntimeException e) {
throw new RuntimeException("Please add arrow-memory-unsafe to your
classpath," +
" No DefaultAllocationManager found to instantiate an
UnsafeAllocationManager", e);
@@ -131,7 +137,7 @@ public class DefaultAllocationManagerOption {
private static AllocationManager.Factory getNettyFactory() {
try {
- return getFactory("org.apache.arrow.memory.NettyAllocationManager");
+ return
getFactory("org.apache.arrow.memory.netty.NettyAllocationManager");
} catch (RuntimeException e) {
throw new RuntimeException("Please add arrow-memory-netty to your
classpath," +
" No DefaultAllocationManager found to instantiate an
NettyAllocationManager", e);
diff --git
a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
index f79cf79531..2f74a985a3 100644
---
a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
+++
b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java
@@ -148,7 +148,7 @@ public class MemoryUtil {
// the static fields above get initialized
final RuntimeException failure = new RuntimeException(
"Failed to initialize MemoryUtil. You must start Java with " +
- "`--add-opens=java.base/java.nio=ALL-UNNAMED` " +
+
"`--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED` " +
"(See https://arrow.apache.org/docs/java/install.html)", e);
failure.printStackTrace();
throw failure;
diff --git
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/CountingAllocationListener.java
b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/CountingAllocationListener.java
similarity index 87%
rename from
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/CountingAllocationListener.java
rename to
java/memory/memory-core/src/test/java/org/apache/arrow/memory/CountingAllocationListener.java
index 78c78c8ad8..38e1a582b8 100644
---
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/CountingAllocationListener.java
+++
b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/CountingAllocationListener.java
@@ -17,6 +17,11 @@
package org.apache.arrow.memory;
+import org.apache.arrow.memory.AllocationListener;
+import org.apache.arrow.memory.AllocationOutcome;
+import org.apache.arrow.memory.BufferAllocator;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
/**
* Counting allocation listener.
* It counts the number of times it has been invoked, and how much memory
allocation it has seen
@@ -30,6 +35,7 @@ final class CountingAllocationListener implements
AllocationListener {
private long totalMem;
private long currentMem;
private boolean expandOnFail;
+ @Nullable
BufferAllocator expandAlloc;
long expandLimit;
@@ -58,6 +64,10 @@ final class CountingAllocationListener implements
AllocationListener {
@Override
public boolean onFailedAllocation(long size, AllocationOutcome outcome) {
if (expandOnFail) {
+ if (expandAlloc == null) {
+ throw new IllegalStateException("expandAlloc must be non-null because
this " +
+ "listener is set to expand on failure.");
+ }
expandAlloc.setLimit(expandLimit);
return true;
}
diff --git
a/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestArrowBuf.java
b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestArrowBuf.java
index c88768c43b..9ba42abc1c 100644
---
a/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestArrowBuf.java
+++
b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestArrowBuf.java
@@ -167,13 +167,15 @@ public class TestArrowBuf {
fieldDebug.set(null, true);
try (BufferAllocator allocator = new RootAllocator(128)) {
allocator.buffer(2);
- Exception e = assertThrows(IllegalStateException.class, () ->
allocator.close());
- assertTrue(e.getMessage().contains("event log for:")); // JDK8, JDK11
+ Exception e = assertThrows(IllegalStateException.class,
allocator::close);
+ assertTrue("Exception had the following message: " + e.getMessage(),
+ e.getMessage().contains("event log for:")); // JDK8, JDK11
} finally {
fieldDebug.set(null, false);
}
} catch (Exception e) {
- assertTrue(e.toString().contains("java.lang.NoSuchFieldException:
modifiers")); // JDK17+
+ assertTrue("Exception had the following toString(): " + e.toString(),
+ e.toString().contains("java.lang.NoSuchFieldException: modifiers"));
// JDK17+
} finally {
((Logger) LoggerFactory.getLogger("org.apache.arrow")).setLevel(null);
}
diff --git
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestBaseAllocator.java
b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBaseAllocator.java
similarity index 90%
rename from
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestBaseAllocator.java
rename to
java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBaseAllocator.java
index 7613d073f8..535d5c15e8 100644
---
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestBaseAllocator.java
+++
b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBaseAllocator.java
@@ -31,22 +31,16 @@ import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.Iterator;
-import java.util.stream.Collectors;
import org.apache.arrow.memory.AllocationOutcomeDetails.Entry;
import org.apache.arrow.memory.rounding.RoundingPolicy;
import org.apache.arrow.memory.rounding.SegmentRoundingPolicy;
import org.apache.arrow.memory.util.AssertionUtil;
+import org.junit.Assert;
import org.junit.Ignore;
import org.junit.Test;
import org.junit.jupiter.api.Assertions;
-import org.slf4j.LoggerFactory;
-import ch.qos.logback.classic.Level;
-import ch.qos.logback.classic.Logger;
-import ch.qos.logback.classic.spi.ILoggingEvent;
-import ch.qos.logback.core.read.ListAppender;
-import io.netty.buffer.PooledByteBufAllocatorL;
import sun.misc.Unsafe;
public class TestBaseAllocator {
@@ -118,6 +112,7 @@ public class TestBaseAllocator {
}
@Test
+ @Ignore
public void testRootAllocator_getEmpty() throws Exception {
try (final RootAllocator rootAllocator =
new RootAllocator(MAX_ALLOCATION)) {
@@ -447,79 +442,79 @@ public class TestBaseAllocator {
@Test
public void testRootAllocator_listeners() throws Exception {
- CountingAllocationListener l1 = new CountingAllocationListener();
- assertEquals(0, l1.getNumPreCalls());
- assertEquals(0, l1.getNumCalls());
- assertEquals(0, l1.getNumReleaseCalls());
- assertEquals(0, l1.getNumChildren());
- assertEquals(0, l1.getTotalMem());
- CountingAllocationListener l2 = new CountingAllocationListener();
- assertEquals(0, l2.getNumPreCalls());
- assertEquals(0, l2.getNumCalls());
- assertEquals(0, l2.getNumReleaseCalls());
- assertEquals(0, l2.getNumChildren());
- assertEquals(0, l2.getTotalMem());
+ CountingAllocationListener listener1 = new CountingAllocationListener();
+ Assert.assertEquals(0, listener1.getNumPreCalls());
+ Assert.assertEquals(0, listener1.getNumCalls());
+ Assert.assertEquals(0, listener1.getNumReleaseCalls());
+ Assert.assertEquals(0, listener1.getNumChildren());
+ Assert.assertEquals(0, listener1.getTotalMem());
+ CountingAllocationListener listener2 = new CountingAllocationListener();
+ Assert.assertEquals(0, listener2.getNumPreCalls());
+ Assert.assertEquals(0, listener2.getNumCalls());
+ Assert.assertEquals(0, listener2.getNumReleaseCalls());
+ Assert.assertEquals(0, listener2.getNumChildren());
+ Assert.assertEquals(0, listener2.getTotalMem());
// root and first-level child share the first listener
// second-level and third-level child share the second listener
- try (final RootAllocator rootAllocator = new RootAllocator(l1,
MAX_ALLOCATION)) {
+ try (final RootAllocator rootAllocator = new RootAllocator(listener1,
MAX_ALLOCATION)) {
try (final BufferAllocator c1 = rootAllocator.newChildAllocator("c1", 0,
MAX_ALLOCATION)) {
- assertEquals(1, l1.getNumChildren());
+ Assert.assertEquals(1, listener1.getNumChildren());
final ArrowBuf buf1 = c1.buffer(16);
assertNotNull("allocation failed", buf1);
- assertEquals(1, l1.getNumPreCalls());
- assertEquals(1, l1.getNumCalls());
- assertEquals(0, l1.getNumReleaseCalls());
- assertEquals(16, l1.getTotalMem());
+ Assert.assertEquals(1, listener1.getNumPreCalls());
+ Assert.assertEquals(1, listener1.getNumCalls());
+ Assert.assertEquals(0, listener1.getNumReleaseCalls());
+ Assert.assertEquals(16, listener1.getTotalMem());
buf1.getReferenceManager().release();
- try (final BufferAllocator c2 = c1.newChildAllocator("c2", l2, 0,
MAX_ALLOCATION)) {
- assertEquals(2, l1.getNumChildren()); // c1 got a new child, so c1's
listener (l1) is notified
- assertEquals(0, l2.getNumChildren());
+ try (final BufferAllocator c2 = c1.newChildAllocator("c2", listener2,
0, MAX_ALLOCATION)) {
+ Assert.assertEquals(2, listener1.getNumChildren()); // c1 got a new
child, so listener1 is notified.
+ Assert.assertEquals(0, listener2.getNumChildren());
final ArrowBuf buf2 = c2.buffer(32);
assertNotNull("allocation failed", buf2);
- assertEquals(1, l1.getNumCalls());
- assertEquals(16, l1.getTotalMem());
- assertEquals(1, l2.getNumPreCalls());
- assertEquals(1, l2.getNumCalls());
- assertEquals(0, l2.getNumReleaseCalls());
- assertEquals(32, l2.getTotalMem());
+ Assert.assertEquals(1, listener1.getNumCalls());
+ Assert.assertEquals(16, listener1.getTotalMem());
+ Assert.assertEquals(1, listener2.getNumPreCalls());
+ Assert.assertEquals(1, listener2.getNumCalls());
+ Assert.assertEquals(0, listener2.getNumReleaseCalls());
+ Assert.assertEquals(32, listener2.getTotalMem());
buf2.getReferenceManager().release();
try (final BufferAllocator c3 = c2.newChildAllocator("c3", 0,
MAX_ALLOCATION)) {
- assertEquals(2, l1.getNumChildren());
- assertEquals(1, l2.getNumChildren());
+ Assert.assertEquals(2, listener1.getNumChildren());
+ Assert.assertEquals(1, listener2.getNumChildren());
final ArrowBuf buf3 = c3.buffer(64);
assertNotNull("allocation failed", buf3);
- assertEquals(1, l1.getNumPreCalls());
- assertEquals(1, l1.getNumCalls());
- assertEquals(1, l1.getNumReleaseCalls());
- assertEquals(16, l1.getTotalMem());
- assertEquals(2, l2.getNumPreCalls());
- assertEquals(2, l2.getNumCalls());
- assertEquals(1, l2.getNumReleaseCalls());
- assertEquals(32 + 64, l2.getTotalMem());
+ Assert.assertEquals(1, listener1.getNumPreCalls());
+ Assert.assertEquals(1, listener1.getNumCalls());
+ Assert.assertEquals(1, listener1.getNumReleaseCalls());
+ Assert.assertEquals(16, listener1.getTotalMem());
+ Assert.assertEquals(2, listener2.getNumPreCalls());
+ Assert.assertEquals(2, listener2.getNumCalls());
+ Assert.assertEquals(1, listener2.getNumReleaseCalls());
+ Assert.assertEquals(32 + 64, listener2.getTotalMem());
buf3.getReferenceManager().release();
}
- assertEquals(2, l1.getNumChildren());
- assertEquals(0, l2.getNumChildren()); // third-level child removed
+ Assert.assertEquals(2, listener1.getNumChildren());
+ Assert.assertEquals(0, listener2.getNumChildren()); // third-level
child removed
}
- assertEquals(1, l1.getNumChildren()); // second-level child removed
- assertEquals(0, l2.getNumChildren());
+ Assert.assertEquals(1, listener1.getNumChildren()); // second-level
child removed
+ Assert.assertEquals(0, listener2.getNumChildren());
}
- assertEquals(0, l1.getNumChildren()); // first-level child removed
+ Assert.assertEquals(0, listener1.getNumChildren()); // first-level child
removed
- assertEquals(2, l2.getNumReleaseCalls());
+ Assert.assertEquals(2, listener2.getNumReleaseCalls());
}
}
@Test
public void testRootAllocator_listenerAllocationFail() throws Exception {
- CountingAllocationListener l1 = new CountingAllocationListener();
- assertEquals(0, l1.getNumCalls());
- assertEquals(0, l1.getTotalMem());
+ CountingAllocationListener listener1 = new CountingAllocationListener();
+ Assert.assertEquals(0, listener1.getNumCalls());
+ Assert.assertEquals(0, listener1.getTotalMem());
// Test attempts to allocate too much from a child whose limit is set to
half of the max
// allocation. The listener's callback triggers, expanding the child
allocator's limit, so then
// the allocation succeeds.
try (final RootAllocator rootAllocator = new
RootAllocator(MAX_ALLOCATION)) {
- try (final BufferAllocator c1 = rootAllocator.newChildAllocator("c1",
l1, 0,
+ try (final BufferAllocator c1 = rootAllocator.newChildAllocator("c1",
listener1, 0,
MAX_ALLOCATION / 2)) {
try {
c1.buffer(MAX_ALLOCATION);
@@ -527,14 +522,14 @@ public class TestBaseAllocator {
} catch (OutOfMemoryException e) {
// expected
}
- assertEquals(0, l1.getNumCalls());
- assertEquals(0, l1.getTotalMem());
+ Assert.assertEquals(0, listener1.getNumCalls());
+ Assert.assertEquals(0, listener1.getTotalMem());
- l1.setExpandOnFail(c1, MAX_ALLOCATION);
+ listener1.setExpandOnFail(c1, MAX_ALLOCATION);
ArrowBuf arrowBuf = c1.buffer(MAX_ALLOCATION);
assertNotNull("allocation failed", arrowBuf);
- assertEquals(1, l1.getNumCalls());
- assertEquals(MAX_ALLOCATION, l1.getTotalMem());
+ Assert.assertEquals(1, listener1.getNumCalls());
+ Assert.assertEquals(MAX_ALLOCATION, listener1.getTotalMem());
arrowBuf.getReferenceManager().release();
}
}
@@ -1098,42 +1093,6 @@ public class TestBaseAllocator {
}
}
- @Test
- public void testMemoryUsage() {
- ListAppender<ILoggingEvent> memoryLogsAppender = new ListAppender<>();
- Logger logger = (Logger) LoggerFactory.getLogger("arrow.allocator");
- try {
- logger.setLevel(Level.TRACE);
- logger.addAppender(memoryLogsAppender);
- memoryLogsAppender.start();
- try (ArrowBuf buf = new ArrowBuf(ReferenceManager.NO_OP, null,
- 1024, new PooledByteBufAllocatorL().empty.memoryAddress())) {
- buf.memoryAddress();
- }
- boolean result = false;
- long startTime = System.currentTimeMillis();
- while ((System.currentTimeMillis() - startTime) < 10000) { // 10 seconds
maximum for time to read logs
- result = memoryLogsAppender.list.stream()
- .anyMatch(
- log -> log.toString().contains("Memory Usage: \n") &&
- log.toString().contains("Large buffers outstanding: ") &&
- log.toString().contains("Normal buffers outstanding: ") &&
- log.getLevel().equals(Level.TRACE)
- );
- if (result) {
- break;
- }
- }
- assertTrue("Log messages are:\n" +
-
memoryLogsAppender.list.stream().map(ILoggingEvent::toString).collect(Collectors.joining("\n")),
- result);
- } finally {
- memoryLogsAppender.stop();
- logger.detachAppender(memoryLogsAppender);
- logger.setLevel(null);
- }
- }
-
@Test
public void testOverlimit() {
try (BufferAllocator allocator = new RootAllocator(1024)) {
diff --git
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestForeignAllocation.java
b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestForeignAllocation.java
similarity index 93%
rename from
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestForeignAllocation.java
rename to
java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestForeignAllocation.java
index ec049ca692..46e94fad37 100644
---
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestForeignAllocation.java
+++
b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestForeignAllocation.java
@@ -23,6 +23,13 @@ import static org.junit.Assert.assertTrue;
import java.util.ArrayList;
import java.util.List;
+import org.apache.arrow.memory.AllocationListener;
+import org.apache.arrow.memory.AllocationOutcome;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.ForeignAllocation;
+import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.memory.util.MemoryUtil;
import org.junit.After;
import org.junit.Before;
diff --git a/java/memory/memory-unsafe/pom.xml
b/java/memory/memory-netty-buffer-patch/pom.xml
similarity index 60%
copy from java/memory/memory-unsafe/pom.xml
copy to java/memory/memory-netty-buffer-patch/pom.xml
index 9f81373081..1d4407c638 100644
--- a/java/memory/memory-unsafe/pom.xml
+++ b/java/memory/memory-netty-buffer-patch/pom.xml
@@ -9,7 +9,9 @@
License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
OF ANY KIND, either express or implied. See the License for the specific
language governing permissions and limitations under the License. -->
-<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+<project xmlns="http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
<parent>
<artifactId>arrow-memory</artifactId>
<groupId>org.apache.arrow</groupId>
@@ -17,16 +19,26 @@
</parent>
<modelVersion>4.0.0</modelVersion>
- <artifactId>arrow-memory-unsafe</artifactId>
- <name>Arrow Memory - Unsafe</name>
- <description>Allocator and utils for allocating memory in Arrow based on
sun.misc.Unsafe</description>
-
+ <artifactId>arrow-memory-netty-buffer-patch</artifactId>
+ <name>Arrow Memory - Netty Buffer</name>
+ <description>Netty Buffer needed to patch that is consumed by Arrow Memory
Netty</description>
<dependencies>
<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-memory-core</artifactId>
</dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-buffer</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-common</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
</dependencies>
-
</project>
diff --git
a/java/memory/memory-netty/src/main/java/io/netty/buffer/ExpandableByteBuf.java
b/java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/ExpandableByteBuf.java
similarity index 100%
rename from
java/memory/memory-netty/src/main/java/io/netty/buffer/ExpandableByteBuf.java
rename to
java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/ExpandableByteBuf.java
diff --git
a/java/memory/memory-netty/src/main/java/io/netty/buffer/LargeBuffer.java
b/java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/LargeBuffer.java
similarity index 100%
copy from
java/memory/memory-netty/src/main/java/io/netty/buffer/LargeBuffer.java
copy to
java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/LargeBuffer.java
diff --git
a/java/memory/memory-netty/src/main/java/io/netty/buffer/MutableWrappedByteBuf.java
b/java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/MutableWrappedByteBuf.java
similarity index 100%
rename from
java/memory/memory-netty/src/main/java/io/netty/buffer/MutableWrappedByteBuf.java
rename to
java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/MutableWrappedByteBuf.java
diff --git
a/java/memory/memory-netty/src/main/java/io/netty/buffer/NettyArrowBuf.java
b/java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/NettyArrowBuf.java
similarity index 96%
rename from
java/memory/memory-netty/src/main/java/io/netty/buffer/NettyArrowBuf.java
rename to
java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/NettyArrowBuf.java
index 71e4b7cb6d..466444c7d5 100644
--- a/java/memory/memory-netty/src/main/java/io/netty/buffer/NettyArrowBuf.java
+++
b/java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/NettyArrowBuf.java
@@ -17,8 +17,6 @@
package io.netty.buffer;
-import static org.apache.arrow.memory.util.LargeMemoryUtil.checkedCastToInt;
-
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
@@ -29,10 +27,12 @@ import java.nio.channels.GatheringByteChannel;
import java.nio.channels.ScatteringByteChannel;
import org.apache.arrow.memory.ArrowBuf;
-import org.apache.arrow.memory.ArrowByteBufAllocator;
import org.apache.arrow.memory.BoundsChecking;
import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.patch.ArrowByteBufAllocator;
+import org.apache.arrow.memory.util.LargeMemoryUtil;
import org.apache.arrow.util.Preconditions;
+import org.apache.arrow.util.VisibleForTesting;
import io.netty.util.internal.PlatformDependent;
@@ -264,7 +264,7 @@ public class NettyArrowBuf extends AbstractByteBuf
implements AutoCloseable {
* @return ByteBuffer
*/
private ByteBuffer getDirectBuffer(long index) {
- return PlatformDependent.directBuffer(addr(index), checkedCastToInt(length
- index));
+ return PlatformDependent.directBuffer(addr(index),
LargeMemoryUtil.checkedCastToInt(length - index));
}
@Override
@@ -580,11 +580,13 @@ public class NettyArrowBuf extends AbstractByteBuf
implements AutoCloseable {
}
@Override
+ @VisibleForTesting
protected void _setInt(int index, int value) {
setInt(index, value);
}
@Override
+ @VisibleForTesting
protected void _setIntLE(int index, int value) {
this.chk(index, 4);
PlatformDependent.putInt(this.addr(index), Integer.reverseBytes(value));
@@ -620,9 +622,9 @@ public class NettyArrowBuf extends AbstractByteBuf
implements AutoCloseable {
final NettyArrowBuf nettyArrowBuf = new NettyArrowBuf(
buf,
buf.getReferenceManager().getAllocator(),
- checkedCastToInt(buf.capacity()));
- nettyArrowBuf.readerIndex(checkedCastToInt(buf.readerIndex()));
- nettyArrowBuf.writerIndex(checkedCastToInt(buf.writerIndex()));
+ LargeMemoryUtil.checkedCastToInt(buf.capacity()));
+
nettyArrowBuf.readerIndex(LargeMemoryUtil.checkedCastToInt(buf.readerIndex()));
+
nettyArrowBuf.writerIndex(LargeMemoryUtil.checkedCastToInt(buf.writerIndex()));
return nettyArrowBuf;
}
diff --git
a/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
b/java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
similarity index 97%
rename from
java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
rename to
java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
index ba9aba353c..ea84a3258e 100644
---
a/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
+++
b/java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
@@ -17,13 +17,12 @@
package io.netty.buffer;
-import static org.apache.arrow.memory.util.AssertionUtil.ASSERT_ENABLED;
-
import java.lang.reflect.Field;
import java.nio.ByteBuffer;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.arrow.memory.OutOfMemoryException;
+import org.apache.arrow.memory.util.AssertionUtil;
import org.apache.arrow.memory.util.LargeMemoryUtil;
import io.netty.util.internal.OutOfDirectMemoryError;
@@ -51,7 +50,7 @@ public class PooledByteBufAllocatorL {
}
/**
- * Returns a {@linkplain io.netty.buffer.UnsafeDirectLittleEndian} of the
given size.
+ * Returns a {@linkplain UnsafeDirectLittleEndian} of the given size.
*/
public UnsafeDirectLittleEndian allocate(long size) {
try {
@@ -180,7 +179,7 @@ public class PooledByteBufAllocatorL {
fail();
}
- if (!ASSERT_ENABLED) {
+ if (!AssertionUtil.ASSERT_ENABLED) {
return new UnsafeDirectLittleEndian((PooledUnsafeDirectByteBuf)
buf);
}
diff --git
a/java/memory/memory-netty/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java
b/java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java
similarity index 100%
rename from
java/memory/memory-netty/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java
rename to
java/memory/memory-netty-buffer-patch/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java
diff --git
a/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/ArrowByteBufAllocator.java
b/java/memory/memory-netty-buffer-patch/src/main/java/org/apache/arrow/memory/patch/ArrowByteBufAllocator.java
similarity index 97%
rename from
java/memory/memory-netty/src/main/java/org/apache/arrow/memory/ArrowByteBufAllocator.java
rename to
java/memory/memory-netty-buffer-patch/src/main/java/org/apache/arrow/memory/patch/ArrowByteBufAllocator.java
index ff40b49ff6..6ce08b5a59 100644
---
a/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/ArrowByteBufAllocator.java
+++
b/java/memory/memory-netty-buffer-patch/src/main/java/org/apache/arrow/memory/patch/ArrowByteBufAllocator.java
@@ -15,7 +15,9 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.patch;
+
+import org.apache.arrow.memory.BufferAllocator;
import io.netty.buffer.AbstractByteBufAllocator;
import io.netty.buffer.ByteBuf;
diff --git
a/java/memory/memory-netty/src/test/java/io/netty/buffer/TestUnsafeDirectLittleEndian.java
b/java/memory/memory-netty-buffer-patch/src/test/java/io/netty/buffer/TestUnsafeDirectLittleEndian.java
similarity index 95%
rename from
java/memory/memory-netty/src/test/java/io/netty/buffer/TestUnsafeDirectLittleEndian.java
rename to
java/memory/memory-netty-buffer-patch/src/test/java/io/netty/buffer/TestUnsafeDirectLittleEndian.java
index c2bd95bb3d..043c2c1605 100644
---
a/java/memory/memory-netty/src/test/java/io/netty/buffer/TestUnsafeDirectLittleEndian.java
+++
b/java/memory/memory-netty-buffer-patch/src/test/java/io/netty/buffer/TestUnsafeDirectLittleEndian.java
@@ -28,6 +28,11 @@ import java.nio.charset.StandardCharsets;
import org.junit.Test;
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.LargeBuffer;
+import io.netty.buffer.Unpooled;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
public class TestUnsafeDirectLittleEndian {
private static final boolean LITTLE_ENDIAN = ByteOrder.nativeOrder() ==
ByteOrder.LITTLE_ENDIAN;
diff --git a/java/memory/memory-netty/pom.xml b/java/memory/memory-netty/pom.xml
index e625cbeabc..159ab5160c 100644
--- a/java/memory/memory-netty/pom.xml
+++ b/java/memory/memory-netty/pom.xml
@@ -26,9 +26,15 @@
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-memory-core</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-memory-netty-buffer-patch</artifactId>
+ <version>${project.version}</version>
+ </dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-buffer</artifactId>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>io.netty</groupId>
@@ -37,6 +43,7 @@
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
+ <scope>test</scope>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
diff --git
a/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/DefaultAllocationManagerFactory.java
b/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/netty/DefaultAllocationManagerFactory.java
similarity index 87%
rename from
java/memory/memory-netty/src/main/java/org/apache/arrow/memory/DefaultAllocationManagerFactory.java
rename to
java/memory/memory-netty/src/main/java/org/apache/arrow/memory/netty/DefaultAllocationManagerFactory.java
index 10cfb5c164..8ece77178f 100644
---
a/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/DefaultAllocationManagerFactory.java
+++
b/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/netty/DefaultAllocationManagerFactory.java
@@ -15,7 +15,11 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.netty;
+
+import org.apache.arrow.memory.AllocationManager;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
/**
* The default Allocation Manager Factory for a module.
diff --git
a/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/NettyAllocationManager.java
b/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/netty/NettyAllocationManager.java
similarity index 94%
rename from
java/memory/memory-netty/src/main/java/org/apache/arrow/memory/NettyAllocationManager.java
rename to
java/memory/memory-netty/src/main/java/org/apache/arrow/memory/netty/NettyAllocationManager.java
index 2000477830..58354d0c2e 100644
---
a/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/NettyAllocationManager.java
+++
b/java/memory/memory-netty/src/main/java/org/apache/arrow/memory/netty/NettyAllocationManager.java
@@ -15,7 +15,12 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.netty;
+
+import org.apache.arrow.memory.AllocationManager;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.ReferenceManager;
import io.netty.buffer.PooledByteBufAllocatorL;
import io.netty.buffer.UnsafeDirectLittleEndian;
diff --git
a/java/memory/memory-netty/src/test/java/io/netty/buffer/TestExpandableByteBuf.java
b/java/memory/memory-netty/src/test/java/io/netty/buffer/TestExpandableByteBuf.java
index b39cca8e8e..67557b65a6 100644
---
a/java/memory/memory-netty/src/test/java/io/netty/buffer/TestExpandableByteBuf.java
+++
b/java/memory/memory-netty/src/test/java/io/netty/buffer/TestExpandableByteBuf.java
@@ -23,6 +23,10 @@ import org.apache.arrow.memory.RootAllocator;
import org.junit.Assert;
import org.junit.Test;
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.ExpandableByteBuf;
+import io.netty.buffer.NettyArrowBuf;
+
public class TestExpandableByteBuf {
@Test
diff --git
a/java/memory/memory-netty/src/test/java/io/netty/buffer/TestNettyArrowBuf.java
b/java/memory/memory-netty/src/test/java/io/netty/buffer/TestNettyArrowBuf.java
index 45d3b41e8a..f18bccb4c9 100644
---
a/java/memory/memory-netty/src/test/java/io/netty/buffer/TestNettyArrowBuf.java
+++
b/java/memory/memory-netty/src/test/java/io/netty/buffer/TestNettyArrowBuf.java
@@ -20,12 +20,15 @@ package io.netty.buffer;
import java.nio.ByteBuffer;
import org.apache.arrow.memory.ArrowBuf;
-import org.apache.arrow.memory.ArrowByteBufAllocator;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.memory.patch.ArrowByteBufAllocator;
import org.junit.Assert;
import org.junit.Test;
+import io.netty.buffer.CompositeByteBuf;
+import io.netty.buffer.NettyArrowBuf;
+
public class TestNettyArrowBuf {
@Test
diff --git
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/ITTestLargeArrowBuf.java
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/ITTestLargeArrowBuf.java
similarity index 93%
rename from
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/ITTestLargeArrowBuf.java
rename to
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/ITTestLargeArrowBuf.java
index fa8d510e36..71dba73d28 100644
---
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/ITTestLargeArrowBuf.java
+++
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/ITTestLargeArrowBuf.java
@@ -15,10 +15,13 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.netty;
import static org.junit.Assert.assertEquals;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestAllocationManagerNetty.java
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestAllocationManagerNetty.java
similarity index 90%
rename from
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestAllocationManagerNetty.java
rename to
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestAllocationManagerNetty.java
index 2dbd56480b..7f1e34ddc5 100644
---
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestAllocationManagerNetty.java
+++
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestAllocationManagerNetty.java
@@ -15,10 +15,12 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.netty;
import static org.junit.Assert.assertEquals;
+import org.apache.arrow.memory.AllocationManager;
+import org.apache.arrow.memory.DefaultAllocationManagerOption;
import org.junit.Test;
/**
diff --git
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestEmptyArrowBuf.java
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestEmptyArrowBuf.java
similarity index 90%
rename from
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestEmptyArrowBuf.java
rename to
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestEmptyArrowBuf.java
index 3fd7ce74aa..b9948083e6 100644
---
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestEmptyArrowBuf.java
+++
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestEmptyArrowBuf.java
@@ -15,10 +15,13 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.netty;
import static org.junit.Assert.assertEquals;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.ReferenceManager;
+import org.apache.arrow.memory.RootAllocator;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -47,8 +50,8 @@ public class TestEmptyArrowBuf {
public void testZeroBuf() {
// Exercise the historical log inside the empty ArrowBuf. This is
initialized statically, and there is a circular
// dependency between ArrowBuf and BaseAllocator, so if the initialization
happens in the wrong order, the
- // historical log will be null even though BaseAllocator.DEBUG is true.
- allocator.getEmpty().print(new StringBuilder(), 0,
BaseAllocator.Verbosity.LOG_WITH_STACKTRACE);
+ // historical log will be null even though RootAllocator.DEBUG is true.
+ allocator.getEmpty().print(new StringBuilder(), 0,
RootAllocator.Verbosity.LOG_WITH_STACKTRACE);
}
@Test
diff --git
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestEndianness.java
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestEndianness.java
similarity index 92%
rename from
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestEndianness.java
rename to
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestEndianness.java
index dcaeb24889..a782523cbc 100644
---
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestEndianness.java
+++
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestEndianness.java
@@ -15,12 +15,14 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.netty;
import static org.junit.Assert.assertEquals;
import java.nio.ByteOrder;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
import org.junit.Test;
import io.netty.buffer.ByteBuf;
diff --git
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestNettyAllocationManager.java
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestNettyAllocationManager.java
similarity index 87%
rename from
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestNettyAllocationManager.java
rename to
java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestNettyAllocationManager.java
index 1b64cd7336..39692c96ce 100644
---
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/TestNettyAllocationManager.java
+++
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestNettyAllocationManager.java
@@ -15,13 +15,18 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.netty;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertTrue;
+import org.apache.arrow.memory.AllocationManager;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.BufferLedger;
+import org.apache.arrow.memory.RootAllocator;
import org.junit.Test;
/**
@@ -31,8 +36,8 @@ public class TestNettyAllocationManager {
static int CUSTOMIZED_ALLOCATION_CUTOFF_VALUE = 1024;
- private BaseAllocator createCustomizedAllocator() {
- return new RootAllocator(BaseAllocator.configBuilder()
+ private RootAllocator createCustomizedAllocator() {
+ return new RootAllocator(RootAllocator.configBuilder()
.allocationManagerFactory(new AllocationManager.Factory() {
@Override
public AllocationManager create(BufferAllocator accountingAllocator,
long size) {
@@ -65,7 +70,7 @@ public class TestNettyAllocationManager {
@Test
public void testSmallBufferAllocation() {
final long bufSize = CUSTOMIZED_ALLOCATION_CUTOFF_VALUE - 512L;
- try (BaseAllocator allocator = createCustomizedAllocator();
+ try (RootAllocator allocator = createCustomizedAllocator();
ArrowBuf buffer = allocator.buffer(bufSize)) {
assertTrue(buffer.getReferenceManager() instanceof BufferLedger);
@@ -89,7 +94,7 @@ public class TestNettyAllocationManager {
@Test
public void testLargeBufferAllocation() {
final long bufSize = CUSTOMIZED_ALLOCATION_CUTOFF_VALUE + 1024L;
- try (BaseAllocator allocator = createCustomizedAllocator();
+ try (RootAllocator allocator = createCustomizedAllocator();
ArrowBuf buffer = allocator.buffer(bufSize)) {
assertTrue(buffer.getReferenceManager() instanceof BufferLedger);
BufferLedger bufferLedger = (BufferLedger) buffer.getReferenceManager();
diff --git
a/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestNettyAllocator.java
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestNettyAllocator.java
new file mode 100644
index 0000000000..07fdc3f784
--- /dev/null
+++
b/java/memory/memory-netty/src/test/java/org/apache/arrow/memory/netty/TestNettyAllocator.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory.netty;
+
+import static org.junit.Assert.assertTrue;
+
+import java.util.Collections;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.ReferenceManager;
+import org.junit.Test;
+import org.slf4j.LoggerFactory;
+
+import ch.qos.logback.classic.Level;
+import ch.qos.logback.classic.Logger;
+import ch.qos.logback.classic.spi.ILoggingEvent;
+import ch.qos.logback.core.read.ListAppender;
+import io.netty.buffer.PooledByteBufAllocatorL;
+
+/**
+ * Test netty allocators.
+ */
+public class TestNettyAllocator {
+
+ @Test
+ public void testMemoryUsage() {
+ ListAppender<ILoggingEvent> memoryLogsAppender = new ListAppender<>();
+ memoryLogsAppender.list =
Collections.synchronizedList(memoryLogsAppender.list);
+ Logger logger = (Logger) LoggerFactory.getLogger("arrow.allocator");
+ try {
+ logger.setLevel(Level.TRACE);
+ logger.addAppender(memoryLogsAppender);
+ memoryLogsAppender.start();
+ try (ArrowBuf buf = new ArrowBuf(ReferenceManager.NO_OP, null,
+ 1024, new PooledByteBufAllocatorL().empty.memoryAddress())) {
+ buf.memoryAddress();
+ }
+ boolean result = false;
+ long startTime = System.currentTimeMillis();
+ while ((System.currentTimeMillis() - startTime) < 10000) { // 10 seconds
maximum for time to read logs
+ // Lock on the list backing the appender since a background thread
might try to add more logs
+ // while stream() is iterating over list elements. This would throw a
flakey ConcurrentModificationException.
+ synchronized (memoryLogsAppender.list) {
+ result = memoryLogsAppender.list.stream()
+ .anyMatch(
+ log -> log.toString().contains("Memory Usage: \n") &&
+ log.toString().contains("Large buffers outstanding: ") &&
+ log.toString().contains("Normal buffers outstanding: ")
&&
+ log.getLevel().equals(Level.TRACE)
+ );
+ }
+ if (result) {
+ break;
+ }
+ }
+ assertTrue("Log messages are:\n" +
+
memoryLogsAppender.list.stream().map(ILoggingEvent::toString).collect(Collectors.joining("\n")),
+ result);
+ } finally {
+ memoryLogsAppender.stop();
+ logger.detachAppender(memoryLogsAppender);
+ logger.setLevel(null);
+ }
+ }
+}
diff --git a/java/memory/memory-unsafe/pom.xml
b/java/memory/memory-unsafe/pom.xml
index 9f81373081..5ef4e8a914 100644
--- a/java/memory/memory-unsafe/pom.xml
+++ b/java/memory/memory-unsafe/pom.xml
@@ -27,6 +27,10 @@
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-memory-core</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.immutables</groupId>
+ <artifactId>value</artifactId>
+ </dependency>
</dependencies>
</project>
diff --git
a/java/memory/memory-netty/src/main/java/io/netty/buffer/LargeBuffer.java
b/java/memory/memory-unsafe/src/main/java/module-info.java
similarity index 67%
rename from
java/memory/memory-netty/src/main/java/io/netty/buffer/LargeBuffer.java
rename to java/memory/memory-unsafe/src/main/java/module-info.java
index 792b3b8148..aa340d2171 100644
--- a/java/memory/memory-netty/src/main/java/io/netty/buffer/LargeBuffer.java
+++ b/java/memory/memory-unsafe/src/main/java/module-info.java
@@ -15,20 +15,8 @@
* limitations under the License.
*/
-package io.netty.buffer;
-
-/**
- * A MutableWrappedByteBuf that also maintains a metric of the number of huge
buffer bytes and
- * counts.
- */
-public class LargeBuffer extends MutableWrappedByteBuf {
-
- public LargeBuffer(ByteBuf buffer) {
- super(buffer);
- }
-
- @Override
- public ByteBuf copy(int index, int length) {
- return new LargeBuffer(buffer.copy(index, length));
- }
+module org.apache.arrow.memory.unsafe {
+ exports org.apache.arrow.memory.unsafe to org.apache.arrow.memory.core;
+
+ requires org.apache.arrow.memory.core;
}
diff --git
a/java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/DefaultAllocationManagerFactory.java
b/java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/unsafe/DefaultAllocationManagerFactory.java
similarity index 87%
rename from
java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/DefaultAllocationManagerFactory.java
rename to
java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/unsafe/DefaultAllocationManagerFactory.java
index 720c3d02d2..dfb6c70685 100644
---
a/java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/DefaultAllocationManagerFactory.java
+++
b/java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/unsafe/DefaultAllocationManagerFactory.java
@@ -15,7 +15,11 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.unsafe;
+
+import org.apache.arrow.memory.AllocationManager;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
/**
* The default Allocation Manager Factory for a module.
diff --git
a/java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/UnsafeAllocationManager.java
b/java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/unsafe/UnsafeAllocationManager.java
similarity index 89%
rename from
java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/UnsafeAllocationManager.java
rename to
java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/unsafe/UnsafeAllocationManager.java
index b10aba3598..3468a6ec65 100644
---
a/java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/UnsafeAllocationManager.java
+++
b/java/memory/memory-unsafe/src/main/java/org/apache/arrow/memory/unsafe/UnsafeAllocationManager.java
@@ -15,8 +15,12 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.unsafe;
+import org.apache.arrow.memory.AllocationManager;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.ReferenceManager;
import org.apache.arrow.memory.util.MemoryUtil;
/**
diff --git
a/java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/TestAllocationManagerUnsafe.java
b/java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/unsafe/TestAllocationManagerUnsafe.java
similarity index 90%
rename from
java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/TestAllocationManagerUnsafe.java
rename to
java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/unsafe/TestAllocationManagerUnsafe.java
index 33abe92e50..f1ca96eea0 100644
---
a/java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/TestAllocationManagerUnsafe.java
+++
b/java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/unsafe/TestAllocationManagerUnsafe.java
@@ -15,10 +15,12 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.unsafe;
import static org.junit.Assert.assertEquals;
+import org.apache.arrow.memory.AllocationManager;
+import org.apache.arrow.memory.DefaultAllocationManagerOption;
import org.junit.Test;
/**
diff --git
a/java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/TestUnsafeAllocationManager.java
b/java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/unsafe/TestUnsafeAllocationManager.java
similarity index 82%
rename from
java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/TestUnsafeAllocationManager.java
rename to
java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/unsafe/TestUnsafeAllocationManager.java
index c15882a37a..77233e73cb 100644
---
a/java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/TestUnsafeAllocationManager.java
+++
b/java/memory/memory-unsafe/src/test/java/org/apache/arrow/memory/unsafe/TestUnsafeAllocationManager.java
@@ -15,11 +15,16 @@
* limitations under the License.
*/
-package org.apache.arrow.memory;
+package org.apache.arrow.memory.unsafe;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
+import org.apache.arrow.memory.AllocationManager;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.BufferLedger;
+import org.apache.arrow.memory.RootAllocator;
import org.junit.Test;
/**
@@ -27,8 +32,8 @@ import org.junit.Test;
*/
public class TestUnsafeAllocationManager {
- private BaseAllocator createUnsafeAllocator() {
- return new
RootAllocator(BaseAllocator.configBuilder().allocationManagerFactory(UnsafeAllocationManager.FACTORY)
+ private BufferAllocator createUnsafeAllocator() {
+ return new
RootAllocator(RootAllocator.configBuilder().allocationManagerFactory(UnsafeAllocationManager.FACTORY)
.build());
}
@@ -51,7 +56,7 @@ public class TestUnsafeAllocationManager {
@Test
public void testBufferAllocation() {
final long bufSize = 4096L;
- try (BaseAllocator allocator = createUnsafeAllocator();
+ try (BufferAllocator allocator = createUnsafeAllocator();
ArrowBuf buffer = allocator.buffer(bufSize)) {
assertTrue(buffer.getReferenceManager() instanceof BufferLedger);
BufferLedger bufferLedger = (BufferLedger) buffer.getReferenceManager();
diff --git a/java/memory/pom.xml b/java/memory/pom.xml
index c10263b97f..55fbb90353 100644
--- a/java/memory/pom.xml
+++ b/java/memory/pom.xml
@@ -23,6 +23,7 @@
<modules>
<module>memory-core</module>
<module>memory-unsafe</module>
+ <module>memory-netty-buffer-patch</module>
<module>memory-netty</module>
</modules>
diff --git a/java/performance/pom.xml b/java/performance/pom.xml
index d215d856d7..5e0b6c1b54 100644
--- a/java/performance/pom.xml
+++ b/java/performance/pom.xml
@@ -195,7 +195,7 @@
</plugin>
<plugin>
<artifactId>maven-javadoc-plugin</artifactId>
- <version>2.9.1</version>
+ <version>3.6.3</version>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
diff --git a/java/pom.xml b/java/pom.xml
index 62e63d41a9..042488a5b9 100644
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -579,6 +579,17 @@
</lifecycleMappingMetadata>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-javadoc-plugin</artifactId>
+ <version>3.6.3</version>
+ <configuration>
+ <source>8</source>
+ <sourceFileExcludes>
+ <sourceFileExclude>**/module-info.java</sourceFileExclude>
+ </sourceFileExcludes>
+ </configuration>
+ </plugin>
<plugin>
<groupId>org.apache.arrow.maven.plugins</groupId>
<artifactId>module-info-compiler-maven-plugin</artifactId>
@@ -768,7 +779,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
- <version>3.0.0-M1</version>
+ <version>3.6.3</version>
<reportSets>
<reportSet><!-- by default, id = "default" -->
<reports><!-- select non-aggregate reports -->
@@ -783,6 +794,11 @@
</reports>
</reportSet>
</reportSets>
+ <configuration>
+ <sourceFileExcludes>
+ <sourceFileExclude>**/module-info.java</sourceFileExclude>
+ </sourceFileExcludes>
+ </configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
@@ -934,7 +950,7 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
- <argLine>--add-opens=java.base/java.nio=ALL-UNNAMED</argLine>
+
<argLine>--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED</argLine>
</configuration>
</plugin>
</plugins>
