This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5f392a219de2 [SPARK-47610][CORE] Always set
`io.netty.tryReflectionSetAccessible=true`
5f392a219de2 is described below
commit 5f392a219de29b0856884fb95ff3e313f1047013
Author: Cheng Pan <[email protected]>
AuthorDate: Wed Mar 27 13:16:13 2024 +0800
[SPARK-47610][CORE] Always set `io.netty.tryReflectionSetAccessible=true`
### What changes were proposed in this pull request?
Always set `io.netty.tryReflectionSetAccessible=true`
### Why are the changes needed?
Arrow requires `-Dio.netty.tryReflectionSetAccessible=true` for JDK9+, see
details in ARROW-7223. SPARK-29924 (fixed in 3.0.0) added document to guide
users to add such JavaOpts manually, as Arrow is a Spark built-in component,
and later we added such Java options to the building system(Maven, SBT, and
PySpark test suite) manually. Now Spark requires JDK 17+, I think we can add
such Java options by default to reduce disturbing users.
### Does this PR introduce _any_ user-facing change?
Yes, no impacts for those users who manually added
`io.netty.tryReflectionSetAccessible=true`, but makes life easier for new Spark
users.
### How was this patch tested?
Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45733 from pan3793/SPARK-47610.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
docs/index.md | 2 --
.../src/main/java/org/apache/spark/launcher/JavaModuleOptions.java | 3 ++-
pom.xml | 5 +++--
project/SparkBuild.scala | 4 ++--
python/docs/source/getting_started/install.rst | 5 +----
python/run-tests.py | 2 +-
sql/hive/pom.xml | 2 +-
7 files changed, 10 insertions(+), 13 deletions(-)
diff --git a/docs/index.md b/docs/index.md
index 12c53c40c8f7..57f701316bd0 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -38,8 +38,6 @@ Spark runs on Java 17/21, Scala 2.13, Python 3.8+, and R 3.5+.
When using the Scala API, it is necessary for applications to use the same
version of Scala that Spark was compiled for.
For example, when using Scala 2.13, use Spark compiled for 2.13, and compile
code/applications for Scala 2.13 as well.
-Setting `-Dio.netty.tryReflectionSetAccessible=true` is required for the
Apache Arrow library. This prevents the
`java.lang.UnsupportedOperationException: sun.misc.Unsafe or
java.nio.DirectByteBuffer.(long, int) not available` error when Apache Arrow
uses Netty internally.
-
# Running the Examples and Shell
Spark comes with several sample programs. Python, Scala, Java, and R examples
are in the
diff --git
a/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
b/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
index 8893f4bcb85a..3a8fa6c42d47 100644
--- a/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
+++ b/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
@@ -42,7 +42,8 @@ public class JavaModuleOptions {
"--add-opens=java.base/sun.security.action=ALL-UNNAMED",
"--add-opens=java.base/sun.util.calendar=ALL-UNNAMED",
"--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED",
- "-Djdk.reflect.useDirectMethodHandle=false"};
+ "-Djdk.reflect.useDirectMethodHandle=false",
+ "-Dio.netty.tryReflectionSetAccessible=true"};
/**
* Returns the default Java options related to `--add-opens' and
diff --git a/pom.xml b/pom.xml
index 79f8745e01f8..ffa4b5df36cb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -316,6 +316,7 @@
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false
+ -Dio.netty.tryReflectionSetAccessible=true
</extraJavaTestArgs>
</properties>
<repositories>
@@ -3109,7 +3110,7 @@
<include>**/*Suite.java</include>
</includes>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
- <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs}</argLine>
<environmentVariables>
<!--
Setting SPARK_DIST_CLASSPATH is a simple way to make sure any
child processes
@@ -3163,7 +3164,7 @@
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>SparkTestSuite.txt</filereports>
- <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs}
-Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g
-XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs}</argLine>
<stderr/>
<environmentVariables>
<!--
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 46f6bf130f4f..cfe9a9cd224c 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1563,7 +1563,6 @@ object TestSettings {
(Test / javaOptions) += "-Dhive.conf.validation=false",
(Test / javaOptions) += "-Dsun.io.serialization.extendedDebugInfo=false",
(Test / javaOptions) += "-Dderby.system.durability=test",
- (Test / javaOptions) += "-Dio.netty.tryReflectionSetAccessible=true",
(Test / javaOptions) ++= {
if ("true".equals(System.getProperty("java.net.preferIPv6Addresses"))) {
Seq("-Djava.net.preferIPv6Addresses=true")
@@ -1592,7 +1591,8 @@ object TestSettings {
"--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
"--add-opens=java.base/sun.security.action=ALL-UNNAMED",
"--add-opens=java.base/sun.util.calendar=ALL-UNNAMED",
- "-Djdk.reflect.useDirectMethodHandle=false").mkString(" ")
+ "-Djdk.reflect.useDirectMethodHandle=false",
+ "-Dio.netty.tryReflectionSetAccessible=true").mkString(" ")
s"-Xmx$heapSize -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize
-XX:ReservedCodeCacheSize=128m -Dfile.encoding=UTF-8 $extraTestJavaArgs"
.split(" ").toSeq
},
diff --git a/python/docs/source/getting_started/install.rst
b/python/docs/source/getting_started/install.rst
index ce29cf626be0..6aa89a689480 100644
--- a/python/docs/source/getting_started/install.rst
+++ b/python/docs/source/getting_started/install.rst
@@ -164,7 +164,4 @@ Package Supported version Note
`googleapis-common-protos` >=1.56.4 Required for Spark Connect
========================== =========================
======================================================================================
-Note that PySpark requires Java 17 or later with ``JAVA_HOME`` properly set.
-Need set ``-Dio.netty.tryReflectionSetAccessible=true`` for Arrow related
features and refer
-to |downloading|_.
-
+Note that PySpark requires Java 17 or later with ``JAVA_HOME`` properly set
and refer to |downloading|_.
diff --git a/python/run-tests.py b/python/run-tests.py
index 4cd3569efce3..66514a9216e9 100755
--- a/python/run-tests.py
+++ b/python/run-tests.py
@@ -112,7 +112,7 @@ def run_individual_python_test(target_dir, test_name,
pyspark_python, keep_test_
# Also override the JVM's temp directory by setting driver and executor
options.
java_options = "-Djava.io.tmpdir={0}".format(tmp_dir)
- java_options = java_options + " -Dio.netty.tryReflectionSetAccessible=true
-Xss4M"
+ java_options = java_options + " -Xss4M"
spark_args = [
"--conf", "spark.driver.extraJavaOptions='{0}'".format(java_options),
"--conf", "spark.executor.extraJavaOptions='{0}'".format(java_options),
diff --git a/sql/hive/pom.xml b/sql/hive/pom.xml
index c1c9a44694ba..3895d9dc5a63 100644
--- a/sql/hive/pom.xml
+++ b/sql/hive/pom.xml
@@ -233,7 +233,7 @@
<artifactId>scalatest-maven-plugin</artifactId>
<configuration>
<!-- Specially disable assertions since some Hive tests fail them -->
- <argLine>-da -Xmx4g -XX:ReservedCodeCacheSize=${CodeCacheSize}
${extraJavaTestArgs} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-da -Xmx4g -XX:ReservedCodeCacheSize=${CodeCacheSize}
${extraJavaTestArgs}</argLine>
</configuration>
</plugin>
<plugin>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
