To whom it may concern, We are building a Spark connector for our own database. Due to some technical concerns, the connector jar is only compatible with JAVA 11 and above. We plan to deploy the connector jar in Amazon EMR and use it in a Jupyter notebook. Due to the compatibility, we must set up an EMR cluster and run Spark+YARN+LIVY+HADOOP services with *JAVA 11*.
However, we can't even open an idle Spark session in the Jupyter Notebook with Livy. Below is the exception we got from YARN and Livy > YARN Diagnostics: > Application application_1670453222303_0001 failed 1 times (global limit =2; > local limit is =1) due to AM Container for > appattempt_1670453222303_0001_000001 exited with exitCode: 13 > Failing this attempt.Diagnostics: [2022-12-07 22:56:20.869]Exception from > container-launch. > Container id: container_1670453222303_0001_01_000001 > Exit code: 13 > > [2022-12-07 22:56:20.897]Container exited with a non-zero exit code 13. Error > file: prelaunch.err. > Last 4096 bytes of prelaunch.err : > Last 4096 bytes of stderr : > veMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742) > ) > 22/12/07 22:56:20 ERROR ApplicationMaster: Uncaught exception: > org.apache.spark.SparkException: Exception thrown in awaitResult: > at > org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301) > ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] > at > org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:514) > ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] > at > org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:278) > ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:929) > ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:928) > ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] > at java.security.AccessController.doPrivileged(Native Method) ~[?:?] > at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?] > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) > ~[hadoop-client-api-3.3.3-amzn-0.jar:?] > at > org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:928) > ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] > at > org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) > ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] > Caused by: java.util.concurrent.ExecutionException: Boxed Error > at scala.concurrent.impl.Promise$.resolver(Promise.scala:87) > ~[scala-library-2.12.15.jar:?] > at > scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:79) > ~[scala-library-2.12.15.jar:?] > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > ~[scala-library-2.12.15.jar:?] > at scala.concurrent.Promise.tryFailure(Promise.scala:112) > ~[scala-library-2.12.15.jar:?] > at scala.concurrent.Promise.tryFailure$(Promise.scala:112) > ~[scala-library-2.12.15.jar:?] > at > scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:187) > ~[scala-library-2.12.15.jar:?] > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:761) > ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]*Caused by: > java.lang.IncompatibleClassChangeError: Inconsistent constant pool data in > classfile for class* org/apache/livy/shaded/json4s/DefaultFormats. Method > 'java.text.SimpleDateFormat > $anonfun$df$1(org.apache.livy.shaded.json4s.DefaultFormats)' at index 156 is > CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef > at > org.apache.livy.shaded.json4s.DefaultFormats.$init$(Formats.scala:318) ~[?:?] > at > org.apache.livy.shaded.json4s.DefaultFormats$.<init>(Formats.scala:296) > ~[?:?]. > > Our EMR cluster is configured to have 1 master and 3 workers. There should always be enough resources. The cluster is on *emr-6.9.0, with Spark 3.3.0, JupyterHub 1.4.1, Livy 0.7.1. installed.* We've been blocked by this issue for a while, our investigation tends to believe this is a compatibility problem between Livy and Java 11. Also we only see the exception for interactive sessions; For batch sessions, Livy works as expected with Java 11. Any feedback here can be super helpful to us. Thank you in advance for all your help! regards, Xiaowei
