This is an automated email from the ASF dual-hosted git repository.
lidavidm pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new a4a3d3f482 GH-39265: [Java] Make it run well with the netty newest
version 4.1.104 (#39266)
a4a3d3f482 is described below
commit a4a3d3f4825eb025657121e70c9d86e8d6ecff35
Author: panbingkun <[email protected]>
AuthorDate: Fri Dec 22 23:17:58 2023 +0800
GH-39265: [Java] Make it run well with the netty newest version 4.1.104
(#39266)
### Describe the enhancement requested
When I used `netty arrow memory 14.0.1` and `netty 4.1.104.Final` in Spark,
the following error occurred,
After pr: https://github.com/netty/netty/pull/13613, `PoolArena` no longer
extends `SizeClasses`, but instead uses it as one of its fields, as follows:
<img width="1051" alt="image"
src="https://github.com/apache/arrow/assets/15246973/6112757b-b2b7-42aa-b4c1-6ab473b91a09";>
in order to ensure that `netty arrow memory 14.0.1` works well with `netty
4.1.104.Final` version, I suggest making similar modifications here.
1.Compilation errors are as follows:
https://github.com/panbingkun/spark/actions/runs/7237466030/job/19717162391
<img width="1005" alt="image"
src="https://github.com/apache/arrow/assets/15246973/98edb6a1-f0e6-4d4e-b568-fbdbffe612f0";>
2.Some bugs have been fixed in `netty 4.1.104.Final` as follows:
<img width="862" alt="image"
src="https://github.com/apache/arrow/assets/15246973/12354a1e-cddd-4ab8-b168-e92712d84cea";>
<img width="861" alt="image"
src="https://github.com/apache/arrow/assets/15246973/bd7d27e1-3953-451c-8c9b-24ecb0d61efd";>
4.1.104.Final release note:
https://netty.io/news/2023/12/15/4-1-104-Final.html
4.1.103.Final release note:
https://netty.io/news/2023/12/13/4-1-103-Final.html
4.1.101.Final release note:
https://netty.io/news/2023/11/09/4-1-101-Final.html
### Component(s)
Java
* Closes: #39265
Authored-by: panbingkun <[email protected]>
Signed-off-by: David Li <[email protected]>
---
.../src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java | 7 ++-----
java/pom.xml | 2 +-
2 files changed, 3 insertions(+), 6 deletions(-)
diff --git
a/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
b/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
index 06c6669cfd..ba9aba353c 100644
---
a/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
+++
b/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java
@@ -71,7 +71,7 @@ public class PooledByteBufAllocatorL {
}
public int getChunkSize() {
- return allocator.chunkSize;
+ return allocator.chunkSize();
}
public long getHugeBufferSize() {
@@ -137,7 +137,6 @@ public class PooledByteBufAllocatorL {
private final PoolArena<ByteBuffer>[] directArenas;
private final MemoryStatusThread statusThread;
- private final int chunkSize;
public InnerAllocator() {
super(true);
@@ -150,8 +149,6 @@ public class PooledByteBufAllocatorL {
throw new RuntimeException("Failure while initializing allocator.
Unable to retrieve direct arenas field.", e);
}
- this.chunkSize = directArenas[0].chunkSize;
-
if (memoryLogger.isTraceEnabled()) {
statusThread = new MemoryStatusThread(this);
statusThread.start();
@@ -166,7 +163,7 @@ public class PooledByteBufAllocatorL {
if (directArena != null) {
- if (initialCapacity > directArena.chunkSize) {
+ if (initialCapacity > chunkSize()) {
// This is beyond chunk size so we'll allocate separately.
ByteBuf buf =
UnpooledByteBufAllocator.DEFAULT.directBuffer(initialCapacity, maxCapacity);
diff --git a/java/pom.xml b/java/pom.xml
index 75e0946f10..4cca5e7245 100644
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -33,7 +33,7 @@
<dep.junit.jupiter.version>5.10.1</dep.junit.jupiter.version>
<dep.slf4j.version>2.0.9</dep.slf4j.version>
<dep.guava-bom.version>32.1.3-jre</dep.guava-bom.version>
- <dep.netty-bom.version>4.1.100.Final</dep.netty-bom.version>
+ <dep.netty-bom.version>4.1.104.Final</dep.netty-bom.version>
<dep.grpc-bom.version>1.60.0</dep.grpc-bom.version>
<dep.protobuf-bom.version>3.23.1</dep.protobuf-bom.version>
<dep.jackson-bom.version>2.16.0</dep.jackson-bom.version>
