Thanks everyone for giving your feedback. Jvm option "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" resolved the issue "cannot access class sun.nio.ch.DirectBuffer" But still Spark throws some other exception
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (ldap executor driver): java.io.InvalidObjectException: ReflectiveOperationException during deserialization at java.base/java.lang.invoke.SerializedLambda.readResolve(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) Caused by: java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) ... 86 common frames omitted Caused by: java.lang.IllegalArgumentException: too many arguments at java.base/java.lang.invoke.LambdaMetafactory.altMetafactory(Unknown Source) at scala.runtime.LambdaDeserializer$.makeCallSite$1(LambdaDeserializer.scala:105) at scala.runtime.LambdaDeserializer$.deserializeLambda(LambdaDeserializer.scala:114) at scala.runtime.LambdaDeserialize.deserializeLambda(LambdaDeserialize.java:38) Maybe we need to change the subject to say "spark-sql_2.12 doesn't work with jdk17 " or should I open another discussion? *Thanks And RegardsSibi.ArunachalammCruncher* On Wed, Apr 13, 2022 at 10:16 PM Sean Owen <sro...@gmail.com> wrote: > Yes I think that's a change that has caused difficulties, but, these > internal APIs were always discouraged. Hey, one is even called 'unsafe'. > There is an escape hatch, the JVM arg below. > > On Wed, Apr 13, 2022, 9:09 AM Andrew Melo <andrew.m...@gmail.com> wrote: > >> Gotcha. Seeing as there's a lot of large projects who used the unsafe API >> either directly or indirectly (via netty, etc..) it's a bit surprising that >> it was so thoroughly closed off without an escape hatch, but I'm sure there >> was a lively discussion around it... >> >> Cheers >> Andrew >> >> On Wed, Apr 13, 2022 at 09:07 Sean Owen <sro...@gmail.com> wrote: >> >>> It is intentionally closed by the JVM going forward, as direct access is >>> discouraged. But it's still necessary for Spark. In some cases, like direct >>> mem access, there is a new API but it's in Java 17 I think, and we can't >>> assume Java 17 any time soon. >>> >>> On Wed, Apr 13, 2022 at 9:05 AM Andrew Melo <andrew.m...@gmail.com> >>> wrote: >>> >>>> Hi Sean, >>>> >>>> Out of curiosity, will Java 11+ always require special flags to access >>>> the unsafe direct memory interfaces, or is this something that will either >>>> be addressed by the spec (by making an "approved" interface) or by >>>> libraries (with some other workaround)? >>>> >>>> Thanks >>>> Andrew >>>> >>>> On Tue, Apr 12, 2022 at 08:45 Sean Owen <sro...@gmail.com> wrote: >>>> >>>>> In Java 11+, you will need to tell the JVM to allow access to internal >>>>> packages in some cases, for any JVM application. You will need flags like >>>>> "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED", which you can see in >>>>> the pom.xml file for the project. >>>>> >>>>> Spark 3.2 does not necessarily work with Java 17 (3.3 should have >>>>> support), but it may well work after you address those flags. >>>>> >>>>> On Tue, Apr 12, 2022 at 7:05 AM Arunachalam Sibisakkaravarthi < >>>>> arunacha...@mcruncher.com> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> spark-sql_2.12:3.2.1 is used in our application. >>>>>> >>>>>> It throws following exceptions when the app runs using JRE17 >>>>>> >>>>>> java.lang.IllegalAccessError: class >>>>>> org.apache.spark.storage.StorageUtils$ (in unnamed module @0x451f1bd4) >>>>>> cannot access class sun.nio.ch.DirectBuffer (in module java.base) >>>>>> because module java.base does not export sun.nio.ch to unnamed module >>>>>> @0x451f1bd43 at >>>>>> org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)4 >>>>>> at >>>>>> org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)5 at >>>>>> org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)6 >>>>>> at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)7 >>>>>> at >>>>>> org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)8 >>>>>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)9 at >>>>>> org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)10 at >>>>>> org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)11 >>>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)12 >>>>>> at >>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)13 >>>>>> at >>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)14 >>>>>> at scala.Option.getOrElse(Option.scala:189)15 at >>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943) >>>>>> >>>>>> How do we fix this? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *Thanks And RegardsSibi.ArunachalammCruncher* >>>>>> >>>>> -- >>>> It's dark in this basement. >>>> >>> -- >> It's dark in this basement. >> >
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (ldap executor driver): java.io.InvalidObjectException: ReflectiveOperationException during deserialization at java.base/java.lang.invoke.SerializedLambda.readResolve(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.base/java.io.ObjectStreamClass.invokeReadResolve(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.base/java.io.ObjectStreamClass.invokeReadObject(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.base/java.io.ObjectStreamClass.invokeReadObject(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.base/java.io.ObjectStreamClass.invokeReadObject(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:85) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) ... 86 more Caused by: java.lang.IllegalArgumentException: too many arguments at java.base/java.lang.invoke.LambdaMetafactory.altMetafactory(Unknown Source) at scala.runtime.LambdaDeserializer$.makeCallSite$1(LambdaDeserializer.scala:105) at scala.runtime.LambdaDeserializer$.deserializeLambda(LambdaDeserializer.scala:114) at scala.runtime.LambdaDeserialize.deserializeLambda(LambdaDeserialize.java:38) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$deserializeLambda$(JdbcUtils.scala) ... 90 more Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2454) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2403) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2402) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2402) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1160) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1160) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1160) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2642) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2584) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2573) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) Caused by: java.io.InvalidObjectException: ReflectiveOperationException during deserialization at java.base/java.lang.invoke.SerializedLambda.readResolve(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.base/java.io.ObjectStreamClass.invokeReadResolve(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.base/java.io.ObjectStreamClass.invokeReadObject(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.base/java.io.ObjectStreamClass.invokeReadObject(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.base/java.io.ObjectStreamClass.invokeReadObject(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream$FieldValues.<init>(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:85) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) ... 86 common frames omitted Caused by: java.lang.IllegalArgumentException: too many arguments at java.base/java.lang.invoke.LambdaMetafactory.altMetafactory(Unknown Source) at scala.runtime.LambdaDeserializer$.makeCallSite$1(LambdaDeserializer.scala:105) at scala.runtime.LambdaDeserializer$.deserializeLambda(LambdaDeserializer.scala:114) at scala.runtime.LambdaDeserialize.deserializeLambda(LambdaDeserialize.java:38) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$deserializeLambda$(JdbcUtils.scala) ... 90 common frames omitted
--------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
