Max Gekk created SPARK-61280:
--------------------------------
Summary: Let the in-process UDF probe JVMs use Unsafe on Java 25
Key: SPARK-61280
URL: https://issues.apache.org/jira/browse/SPARK-61280
Project: Spark
Issue Type: Test
Components: PySpark
Affects Versions: 5.0.0
Reporter: Max Gekk
On Java 25, test_fresh_jvm_retries_configuration_without_spark_home and
test_missing_jep_has_an_initialization_hint in
python/pyspark/sql/tests/test_inprocess_udf.py fail: the fresh JVM each test
starts exits 1 with
java.lang.ClassCastException: class io.netty.buffer.PooledDirectByteBuf cannot
be cast to class io.netty.buffer.PooledUnsafeDirectByteBuf
Both probes start a bare java with only --add-opens and
-Dio.netty.tryReflectionSetAccessible=true. On Java 25+ Netty 4.2 disables its
use of Unsafe unless the JVM runs with --sun-misc-unsafe-memory-access=allow,
and Arrow's PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL always
casts the buffer Netty returns to PooledUnsafeDirectByteBuf. Spark's own JVMs
get the flag from JavaModuleOptions (SPARK-56955); these probes do not, and
both allocate an Arrow buffer.
The tests are skipped where jep is not installed, which includes the Java 25
build (its python-311 image has no jep), so CI has not run them on Java 25 yet.
A Java 25 build with the python-312 image (which installs jep) fails both.
Fix: pass --sun-misc-unsafe-memory-access=allow to the probes, with
-XX:+IgnoreUnrecognizedVMOptions so Java 21, which does not know the flag,
still starts.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
