Jacek Laskowski created SPARK-23830: ---------------------------------------
Summary: Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object Key: SPARK-23830 URL: https://issues.apache.org/jira/browse/SPARK-23830 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 2.3.0 Reporter: Jacek Laskowski As reported on StackOverflow in [Why does Spark on YARN fail with “Exception in thread ”Driver“ java.lang.NullPointerException”?|https://stackoverflow.com/q/49564334/1305344] the following Spark application fails with {{Exception in thread "Driver" java.lang.NullPointerException}} with Spark on YARN in cluster deploy mode: {code} class MyClass { def main(args: Array[String]): Unit = { val c = new MyClass() c.process() } def process(): Unit = { val sparkConf = new SparkConf().setAppName("my-test") val sparkSession: SparkSession = SparkSession.builder().config(sparkConf).getOrCreate() import sparkSession.implicits._ .... } ... } {code} The exception is as follows: {code} 18/03/29 20:07:52 INFO ApplicationMaster: Starting the user application in a separate Thread 18/03/29 20:07:52 INFO ApplicationMaster: Waiting for spark context initialization... Exception in thread "Driver" java.lang.NullPointerException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637) {code} I think the reason for the exception {{Exception in thread "Driver" java.lang.NullPointerException}} is due to [the following code|https://github.com/apache/spark/blob/v2.3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L700-L701]: {code} val mainMethod = userClassLoader.loadClass(args.userClass) .getMethod("main", classOf[Array[String]]) {code} So when {{mainMethod}} is used in [the following code|https://github.com/apache/spark/blob/v2.3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L706] it simply gives NPE. {code} mainMethod.invoke(null, userArgs.toArray) {code} That could be easily avoided with an extra check if the {{mainMethod}} is initialized and give a user a message what may have been a reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org