This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/orc.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 3eb258661 ORC-1586: Fix IllegalAccessError when SparkBenchmark runs on
JDK17
3eb258661 is described below
commit 3eb2586614fe723d37f69ca76cb052962f6b6a12
Author: sychen <[email protected]>
AuthorDate: Tue Jan 16 08:03:56 2024 -0800
ORC-1586: Fix IllegalAccessError when SparkBenchmark runs on JDK17
### What changes were proposed in this pull request?
Add java options `--add-opens=java.base/sun.nio.ch=ALL-UNNAMED`
### Why are the changes needed?
```java
java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$
(in unnamed module 0x5b2c883c) cannot access class sun.nio.ch.DirectBuffer (in
module java.base) because module java.base does not export sun.nio.ch to
unnamed module 0x5b2c883c
at
org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
at
org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
at
org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:121)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:358)
at
org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:295)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:344)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:196)
at
org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:284)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:483)
at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
at scala.Option.getOrElse(Option.scala:189)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093)
at
org.apache.orc.bench.spark.SparkBenchmark$InputSource.setup(SparkBenchmark.java:129)
at
org.apache.orc.bench.spark.jmh_generated.SparkBenchmark_fullRead_jmhTest._jmh_tryInit_f_inputsource1_1(SparkBenchmark_fullRead_jmhTest.java:403)
```
### How was this patch tested?
local test
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #1748 from cxzl25/ORC-1586.
Authored-by: sychen <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 5097fac05a68a3602fb9867b067ec50211e0691b)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../bench/spark/src/java/org/apache/orc/bench/spark/SparkBenchmark.java | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/java/bench/spark/src/java/org/apache/orc/bench/spark/SparkBenchmark.java
b/java/bench/spark/src/java/org/apache/orc/bench/spark/SparkBenchmark.java
index 360c4de92..1285875dc 100644
--- a/java/bench/spark/src/java/org/apache/orc/bench/spark/SparkBenchmark.java
+++ b/java/bench/spark/src/java/org/apache/orc/bench/spark/SparkBenchmark.java
@@ -46,6 +46,7 @@ import org.apache.spark.sql.types.StructType;
import org.apache.spark.sql.vectorized.ColumnarBatch;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
@@ -73,6 +74,7 @@ import java.util.concurrent.TimeUnit;
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@AutoService(OrcBenchmark.class)
+@Fork(jvmArgsAppend = "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED")
public class SparkBenchmark implements OrcBenchmark {
private static final Path root = Utilities.getBenchmarkRoot();
