[
https://issues.apache.org/jira/browse/SPARK-56955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18082168#comment-18082168
]
Anupam Yadav commented on SPARK-56955:
--------------------------------------
Reproduced locally on JDK 25 (Corretto 25.0.3). Submitted PR #55999 with the
test skip workaround. I will also investigate the root fix (Arrow/Netty
upgrade) and submit a follow-up PR.
> Arrow/Netty memory allocator fails on JDK 25 (affects spark-shell --remote)
> ---------------------------------------------------------------------------
>
> Key: SPARK-56955
> URL: https://issues.apache.org/jira/browse/SPARK-56955
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.2.0
> Reporter: Anupam Yadav
> Priority: Major
>
> On JDK 25, spark-shell --remote fails to start due to Arrow/Netty memory
> allocator incompatibility.
> *Reproduced locally* on JDK 25 (Amazon Corretto 25.0.3) -- test passes on JDK
> 17 and 21 but fails consistently on JDK 25.
> *Error:*
> {code}
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at
> org.apache.arrow.memory.netty.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26)
> ...
> Caused by: java.lang.UnsupportedOperationException
> at
> org.sparkproject.connect.client.io.netty.buffer.EmptyByteBuf.memoryAddress(EmptyByteBuf.java:961)
> at
> org.sparkproject.connect.client.io.netty.buffer.DuplicatedByteBuf.memoryAddress(DuplicatedByteBuf.java:115)
> at
> org.sparkproject.connect.client.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:45)
> at
> org.sparkproject.connect.client.io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:47)
> {code}
> *Environment:*
> - JDK 25 (Amazon Corretto 25.0.3, also OpenJDK 25.0.3 on UBI 10 per
> dongjoon-hyun)
> - JDK 17 and 21: not affected
> *Root cause:* Netty's {{EmptyByteBuf.memoryAddress()}} throws
> {{UnsupportedOperationException}} on JDK 25. Arrow's
> {{DefaultAllocationManagerFactory}} calls this during static initialization
> via {{PooledByteBufAllocatorL}}. This affects any code path that initializes
> Arrow (Spark Connect client, vectorized readers, etc.).
> *Workaround:* PR #55999 skips the affected test on JDK 25+.
> *Root fix:* Requires Arrow/Netty dependency update to a version that supports
> JDK 25.
> Originally reported by dongjoon-hyun on PR #55720.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
