Jacek Laskowski created SPARK-23830:
---------------------------------------
Summary: Spark on YARN in cluster deploy mode fail with
NullPointerException when a Spark application is a Scala class not object
Key: SPARK-23830
URL: https://issues.apache.org/jira/browse/SPARK-23830
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 2.3.0
Reporter: Jacek Laskowski
As reported on StackOverflow in [Why does Spark on YARN fail with “Exception in
thread ”Driver“
java.lang.NullPointerException”?|https://stackoverflow.com/q/49564334/1305344]
the following Spark application fails with {{Exception in thread "Driver"
java.lang.NullPointerException}} with Spark on YARN in cluster deploy mode:
{code}
class MyClass {
def main(args: Array[String]): Unit = {
val c = new MyClass()
c.process()
}
def process(): Unit = {
val sparkConf = new SparkConf().setAppName("my-test")
val sparkSession: SparkSession =
SparkSession.builder().config(sparkConf).getOrCreate()
import sparkSession.implicits._
....
}
...
}
{code}
The exception is as follows:
{code}
18/03/29 20:07:52 INFO ApplicationMaster: Starting the user application in a
separate Thread
18/03/29 20:07:52 INFO ApplicationMaster: Waiting for spark context
initialization...
Exception in thread "Driver" java.lang.NullPointerException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
{code}
I think the reason for the exception {{Exception in thread "Driver"
java.lang.NullPointerException}} is due to [the following
code|https://github.com/apache/spark/blob/v2.3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L700-L701]:
{code}
val mainMethod = userClassLoader.loadClass(args.userClass)
.getMethod("main", classOf[Array[String]])
{code}
So when {{mainMethod}} is used in [the following
code|https://github.com/apache/spark/blob/v2.3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L706]
it simply gives NPE.
{code}
mainMethod.invoke(null, userArgs.toArray)
{code}
That could be easily avoided with an extra check if the {{mainMethod}} is
initialized and give a user a message what may have been a reason.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
