[
https://issues.apache.org/jira/browse/SPARK-61280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-61280:
-----------------------------
Summary: Start the in-process UDF probe JVMs with Spark's module options
(was: Let the in-process UDF probe JVMs use Unsafe on Java 25)
> Start the in-process UDF probe JVMs with Spark's module options
> ---------------------------------------------------------------
>
> Key: SPARK-61280
> URL: https://issues.apache.org/jira/browse/SPARK-61280
> Project: Spark
> Issue Type: Test
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Max Gekk
> Priority: Major
> Labels: pull-request-available
>
> On Java 25, test_fresh_jvm_retries_configuration_without_spark_home and
> test_missing_jep_has_an_initialization_hint in
> python/pyspark/sql/tests/test_inprocess_udf.py fail: the fresh JVM each test
> starts exits 1 with
> java.lang.ClassCastException: class io.netty.buffer.PooledDirectByteBuf
> cannot be cast to class io.netty.buffer.PooledUnsafeDirectByteBuf
> Both probes start a bare java with only --add-opens and
> -Dio.netty.tryReflectionSetAccessible=true. On Java 25+ Netty 4.2 disables
> its use of Unsafe unless the JVM runs with
> --sun-misc-unsafe-memory-access=allow, and Arrow's
> PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL always casts the
> buffer Netty returns to PooledUnsafeDirectByteBuf. Spark's own JVMs get the
> flag from JavaModuleOptions (SPARK-56955); these probes do not, and both
> allocate an Arrow buffer.
> The tests are skipped where jep is not installed, which includes the Java 25
> build (its python-311 image has no jep), so CI has not run them on Java 25
> yet. A Java 25 build with the python-312 image (which installs jep) fails
> both.
> Fix: pass --sun-misc-unsafe-memory-access=allow to the probes, with
> -XX:+IgnoreUnrecognizedVMOptions so Java 21, which does not know the flag,
> still starts.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
