mrmadira opened a new issue, #36713:
URL: https://github.com/apache/arrow/issues/36713
### Describe the enhancement requested
Latest arrow-memory-netty jar fails to work with latest netty-handler.jar
with the following exception.
```
java.lang.NoSuchMethodError:
io/netty/buffer/PooledByteBufAllocatorL$InnerAllocator.threadCache()Lio/netty/buffer/PoolThreadCache;
(loaded from file:/opt/ibm/spark/jars/arrow-memory-netty-7.0.0.jar by
jdk.internal.loader.ClassLoaders$AppClassLoader@5f26aa33) called from class
io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator (loaded from
file:/opt/ibm/spark/jars/arrow-memory-netty-7.0.0.jar by
jdk.internal.loader.ClassLoaders$AppClassLoader@5f26aa33).
at
io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:164)
at
io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:214)
at
io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:58)
at
org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:77)
at
org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:84)
at
org.apache.arrow.memory.NettyAllocationManager$1.create(NettyAllocationManager.java:34)
at
org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:315)
at
org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:310)
at
org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:298)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:276)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:240)
at
org.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:725)
at
org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363)
at
org.apache.spark.sql.execution.arrow.ArrowConverters$.loadBatch(ArrowConverters.scala:189)
at
org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2.nextBatch(ArrowConverters.scala:165)
at
org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2.<init>(ArrowConverters.scala:144)
at
org.apache.spark.sql.execution.arrow.ArrowConverters$.fromBatchIterator(ArrowConverters.scala:143)
at
org.apache.spark.sql.execution.arrow.ArrowConverters$.$anonfun$toDataFrame$1(ArrowConverters.scala:203)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:839)
```
We believe this change in netty-handler is where the integration breaks.:
https://github.com/netty/netty/commit/9412424091eb89f9d55372e38fedbcfcc285dd94
### Component(s)
Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
