This was bug in scala-library:2.12.11 and it is fixed in 2.12.15 https://github.com/scala/bug/issues/12419 https://github.com/scala/scala/pull/9676
spark-sql_2.12 v3.2.1 uses scala-library v2.12.15, but somehow an older version of scala-library v2.12.11 which causes a problem and throws the exceptions is bundled in our artifact package Now it is fixed in our application and spark works on jdk17 after setting following jvm options "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" " --add-opens=java.base/sun.util.calendar=ALL-UNNAMED" Thanks everyone for your help. *Thanks And RegardsSibi.ArunachalammCruncher* On Thu, Apr 14, 2022 at 11:22 AM Arunachalam Sibisakkaravarthi < arunacha...@mcruncher.com> wrote: > This was bug in scala-library:2.12.11 and it is fixed in 2.12.15 > https://github.com/scala/bug/issues/12419 > https://github.com/scala/scala/pull/9676 > > spark-sql_2.12 v3.2.1 uses scala-library v2.12.15, but somehow an older > version of scala-library v2.12.11 which causes a problem and throws the > exceptions is bundled in our artifact package > Now it is fixed in our application and spark works on jdk17 after setting > following jvm options > "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" > " --add-opens=java.base/sun.util.calendar=ALL-UNNAMED" > > Thanks everyone for your help. > > > > *Thanks And RegardsSibi.ArunachalammCruncher* > > > On Thu, Apr 14, 2022 at 10:50 AM Sean Owen <sro...@gmail.com> wrote: > >> It is not officially supported, yes. Try Spark 3.3 from the branch if you >> want to try Java 17 >> >> On Wed, Apr 13, 2022, 9:36 PM Arunachalam Sibisakkaravarthi < >> arunacha...@mcruncher.com> wrote: >> >>> Thanks everyone for giving your feedback. >>> Jvm option "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" resolved the >>> issue "cannot access class sun.nio.ch.DirectBuffer" >>> But still Spark throws some other exception >>> >>> org.apache.spark.SparkException: Job aborted due to stage failure: Task >>> 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage >>> 0.0 (TID 0) (ldap executor driver): java.io.InvalidObjectException: >>> ReflectiveOperationException during deserialization >>> at java.base/java.lang.invoke.SerializedLambda.readResolve(Unknown >>> Source) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown >>> Source) >>> at >>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>> Source) >>> >>> Caused by: java.lang.reflect.InvocationTargetException: null >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown >>> Source) >>> at >>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>> Source) >>> at java.base/java.lang.reflect.Method.invoke(Unknown Source) >>> ... 86 common frames omitted >>> >>> Caused by: java.lang.IllegalArgumentException: too many arguments >>> at java.base/java.lang.invoke.LambdaMetafactory.altMetafactory(Unknown >>> Source) >>> at >>> scala.runtime.LambdaDeserializer$.makeCallSite$1(LambdaDeserializer.scala:105) >>> at >>> scala.runtime.LambdaDeserializer$.deserializeLambda(LambdaDeserializer.scala:114) >>> at >>> scala.runtime.LambdaDeserialize.deserializeLambda(LambdaDeserialize.java:38) >>> >>> Maybe we need to change the subject to say "spark-sql_2.12 doesn't work >>> with jdk17 " or should I open another discussion? >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *Thanks And RegardsSibi.ArunachalammCruncher* >>> >>> >>> On Wed, Apr 13, 2022 at 10:16 PM Sean Owen <sro...@gmail.com> wrote: >>> >>>> Yes I think that's a change that has caused difficulties, but, these >>>> internal APIs were always discouraged. Hey, one is even called 'unsafe'. >>>> There is an escape hatch, the JVM arg below. >>>> >>>> On Wed, Apr 13, 2022, 9:09 AM Andrew Melo <andrew.m...@gmail.com> >>>> wrote: >>>> >>>>> Gotcha. Seeing as there's a lot of large projects who used the unsafe >>>>> API either directly or indirectly (via netty, etc..) it's a bit surprising >>>>> that it was so thoroughly closed off without an escape hatch, but I'm sure >>>>> there was a lively discussion around it... >>>>> >>>>> Cheers >>>>> Andrew >>>>> >>>>> On Wed, Apr 13, 2022 at 09:07 Sean Owen <sro...@gmail.com> wrote: >>>>> >>>>>> It is intentionally closed by the JVM going forward, as direct access >>>>>> is discouraged. But it's still necessary for Spark. In some cases, like >>>>>> direct mem access, there is a new API but it's in Java 17 I think, and we >>>>>> can't assume Java 17 any time soon. >>>>>> >>>>>> On Wed, Apr 13, 2022 at 9:05 AM Andrew Melo <andrew.m...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Sean, >>>>>>> >>>>>>> Out of curiosity, will Java 11+ always require special flags to >>>>>>> access the unsafe direct memory interfaces, or is this something that >>>>>>> will >>>>>>> either be addressed by the spec (by making an "approved" interface) or >>>>>>> by >>>>>>> libraries (with some other workaround)? >>>>>>> >>>>>>> Thanks >>>>>>> Andrew >>>>>>> >>>>>>> On Tue, Apr 12, 2022 at 08:45 Sean Owen <sro...@gmail.com> wrote: >>>>>>> >>>>>>>> In Java 11+, you will need to tell the JVM to allow access to >>>>>>>> internal packages in some cases, for any JVM application. You will need >>>>>>>> flags like "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED", which >>>>>>>> you can see in the pom.xml file for the project. >>>>>>>> >>>>>>>> Spark 3.2 does not necessarily work with Java 17 (3.3 should have >>>>>>>> support), but it may well work after you address those flags. >>>>>>>> >>>>>>>> On Tue, Apr 12, 2022 at 7:05 AM Arunachalam Sibisakkaravarthi < >>>>>>>> arunacha...@mcruncher.com> wrote: >>>>>>>> >>>>>>>>> Hi guys, >>>>>>>>> >>>>>>>>> spark-sql_2.12:3.2.1 is used in our application. >>>>>>>>> >>>>>>>>> It throws following exceptions when the app runs using JRE17 >>>>>>>>> >>>>>>>>> java.lang.IllegalAccessError: class >>>>>>>>> org.apache.spark.storage.StorageUtils$ (in unnamed module >>>>>>>>> @0x451f1bd4) cannot access class sun.nio.ch.DirectBuffer (in module >>>>>>>>> java.base) because module java.base does not export sun.nio.ch to >>>>>>>>> unnamed module @0x451f1bd43 at >>>>>>>>> org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)4 >>>>>>>>> at >>>>>>>>> org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)5 >>>>>>>>> at >>>>>>>>> org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)6 >>>>>>>>> at >>>>>>>>> org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)7 >>>>>>>>> at >>>>>>>>> org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)8 >>>>>>>>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)9 at >>>>>>>>> org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)10 >>>>>>>>> at >>>>>>>>> org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)11 >>>>>>>>> at >>>>>>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:460)12 >>>>>>>>> at >>>>>>>>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)13 >>>>>>>>> at >>>>>>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)14 >>>>>>>>> at scala.Option.getOrElse(Option.scala:189)15 at >>>>>>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943) >>>>>>>>> >>>>>>>>> How do we fix this? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Thanks And RegardsSibi.ArunachalammCruncher* >>>>>>>>> >>>>>>>> -- >>>>>>> It's dark in this basement. >>>>>>> >>>>>> -- >>>>> It's dark in this basement. >>>>> >>>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >>
