Yes I think that's a change that has caused difficulties, but, these internal APIs were always discouraged. Hey, one is even called 'unsafe'. There is an escape hatch, the JVM arg below.
On Wed, Apr 13, 2022, 9:09 AM Andrew Melo <andrew.m...@gmail.com> wrote: > Gotcha. Seeing as there's a lot of large projects who used the unsafe API > either directly or indirectly (via netty, etc..) it's a bit surprising that > it was so thoroughly closed off without an escape hatch, but I'm sure there > was a lively discussion around it... > > Cheers > Andrew > > On Wed, Apr 13, 2022 at 09:07 Sean Owen <sro...@gmail.com> wrote: > >> It is intentionally closed by the JVM going forward, as direct access is >> discouraged. But it's still necessary for Spark. In some cases, like direct >> mem access, there is a new API but it's in Java 17 I think, and we can't >> assume Java 17 any time soon. >> >> On Wed, Apr 13, 2022 at 9:05 AM Andrew Melo <andrew.m...@gmail.com> >> wrote: >> >>> Hi Sean, >>> >>> Out of curiosity, will Java 11+ always require special flags to access >>> the unsafe direct memory interfaces, or is this something that will either >>> be addressed by the spec (by making an "approved" interface) or by >>> libraries (with some other workaround)? >>> >>> Thanks >>> Andrew >>> >>> On Tue, Apr 12, 2022 at 08:45 Sean Owen <sro...@gmail.com> wrote: >>> >>>> In Java 11+, you will need to tell the JVM to allow access to internal >>>> packages in some cases, for any JVM application. You will need flags like >>>> "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED", which you can see in >>>> the pom.xml file for the project. >>>> >>>> Spark 3.2 does not necessarily work with Java 17 (3.3 should have >>>> support), but it may well work after you address those flags. >>>> >>>> On Tue, Apr 12, 2022 at 7:05 AM Arunachalam Sibisakkaravarthi < >>>> arunacha...@mcruncher.com> wrote: >>>> >>>>> Hi guys, >>>>> >>>>> spark-sql_2.12:3.2.1 is used in our application. >>>>> >>>>> It throws following exceptions when the app runs using JRE17 >>>>> >>>>> java.lang.IllegalAccessError: class >>>>> org.apache.spark.storage.StorageUtils$ (in unnamed module @0x451f1bd4) >>>>> cannot access class sun.nio.ch.DirectBuffer (in module java.base) because >>>>> module java.base does not export sun.nio.ch to unnamed module >>>>> @0x451f1bd43 at >>>>> org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)4 >>>>> at >>>>> org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)5 at >>>>> org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)6 >>>>> at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)7 >>>>> at >>>>> org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)8 >>>>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)9 at >>>>> org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)10 at >>>>> org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)11 >>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)12 >>>>> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)13 >>>>> at >>>>> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)14 >>>>> at scala.Option.getOrElse(Option.scala:189)15 at >>>>> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943) >>>>> >>>>> How do we fix this? >>>>> >>>>> >>>>> >>>>> >>>>> *Thanks And RegardsSibi.ArunachalammCruncher* >>>>> >>>> -- >>> It's dark in this basement. >>> >> -- > It's dark in this basement. >
