Thank you, Larry! I will try to see if I can update the Livy install in EMR, if you have any tutorial on upgrading Livy, please let me know. Also top the post here, to see if other thoughts. Thanks a lot!
On Thu, Dec 8, 2022 at 1:44 PM larry mccay <[email protected]> wrote: > It appears to me that the org.json4s.DefaultFormats class in the shaded > jar for Livy 0.7.1 is incompatible. > You could try upgrading that dependency within the shaded jar. > > Others may have more thoughts? > > On Thu, Dec 8, 2022 at 3:14 PM Xiaowei Wang <[email protected]> wrote: > >> To whom it may concern, >> >> We are building a Spark connector for our own database. Due to some >> technical concerns, the connector jar is only compatible with JAVA 11 and >> above. We plan to deploy the connector jar in Amazon EMR and use it in a >> Jupyter notebook. Due to the compatibility, we must set up an EMR cluster >> and run Spark+YARN+LIVY+HADOOP services with *JAVA 11*. >> >> However, we can't even open an idle Spark session in the Jupyter Notebook >> with Livy. Below is the exception we got from YARN and Livy >> >>> YARN Diagnostics: >>> Application application_1670453222303_0001 failed 1 times (global limit =2; >>> local limit is =1) due to AM Container for >>> appattempt_1670453222303_0001_000001 exited with exitCode: 13 >>> Failing this attempt.Diagnostics: [2022-12-07 22:56:20.869]Exception from >>> container-launch. >>> Container id: container_1670453222303_0001_01_000001 >>> Exit code: 13 >>> >>> [2022-12-07 22:56:20.897]Container exited with a non-zero exit code 13. >>> Error file: prelaunch.err. >>> Last 4096 bytes of prelaunch.err : >>> Last 4096 bytes of stderr : >>> veMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>> at >>> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742) >>> ) >>> 22/12/07 22:56:20 ERROR ApplicationMaster: Uncaught exception: >>> org.apache.spark.SparkException: Exception thrown in awaitResult: >>> at >>> org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301) >>> ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] >>> at >>> org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:514) >>> ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] >>> at >>> org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:278) >>> ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] >>> at >>> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:929) >>> ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] >>> at >>> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:928) >>> ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] >>> at java.security.AccessController.doPrivileged(Native Method) ~[?:?] >>> at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?] >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) >>> ~[hadoop-client-api-3.3.3-amzn-0.jar:?] >>> at >>> org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:928) >>> ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] >>> at >>> org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) >>> ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1] >>> Caused by: java.util.concurrent.ExecutionException: Boxed Error >>> at scala.concurrent.impl.Promise$.resolver(Promise.scala:87) >>> ~[scala-library-2.12.15.jar:?] >>> at >>> scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:79) >>> ~[scala-library-2.12.15.jar:?] >>> at >>> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) >>> ~[scala-library-2.12.15.jar:?] >>> at scala.concurrent.Promise.tryFailure(Promise.scala:112) >>> ~[scala-library-2.12.15.jar:?] >>> at scala.concurrent.Promise.tryFailure$(Promise.scala:112) >>> ~[scala-library-2.12.15.jar:?] >>> at >>> scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:187) >>> ~[scala-library-2.12.15.jar:?] >>> at >>> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:761) >>> ~[spark-yarn_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]*Caused by: >>> java.lang.IncompatibleClassChangeError: Inconsistent constant pool data in >>> classfile for class* org/apache/livy/shaded/json4s/DefaultFormats. Method >>> 'java.text.SimpleDateFormat >>> $anonfun$df$1(org.apache.livy.shaded.json4s.DefaultFormats)' at index 156 >>> is CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef >>> at >>> org.apache.livy.shaded.json4s.DefaultFormats.$init$(Formats.scala:318) >>> ~[?:?] >>> at >>> org.apache.livy.shaded.json4s.DefaultFormats$.<init>(Formats.scala:296) >>> ~[?:?]. >>> >>> >> Our EMR cluster is configured to have 1 master and 3 workers. There >> should always be enough resources. The cluster is on *emr-6.9.0, >> with Spark 3.3.0, JupyterHub 1.4.1, Livy 0.7.1. installed.* >> >> We've been blocked by this issue for a while, our investigation tends to >> believe this is a compatibility problem between Livy and Java 11. Also we >> only see the exception for interactive sessions; For batch sessions, Livy >> works as expected with Java 11. Any feedback here can be super helpful to >> us. Thank you in advance for all your help! >> >> regards, >> Xiaowei >> > -- inner peace
