German Schiavon Matteo created SPARK-22574:
----------------------------------------------
Summary: Wrong request causing Spark Dispatcher going inactive
Key: SPARK-22574
URL: https://issues.apache.org/jira/browse/SPARK-22574
Project: Spark
Issue Type: Bug
Components: Mesos, Spark Submit
Affects Versions: 2.2.0
Reporter: German Schiavon Matteo
Priority: Minor
Fix For: 2.2.0, 2.1.0, 2.0.0
When submitting a wrong _CreateSubmissionRequest_ to Spark Dispatcher is
causing a bad state of Dispatcher and making it inactive as a mesos framework.
The class CreateSubmissionRequest initialise its arguments to null as follows:
{code:title=CreateSubmissionRequest.scala|borderStyle=solid}
var appResource: String = null
var mainClass: String = null
var appArgs: Array[String] = null
var sparkProperties: Map[String, String] = null
var environmentVariables: Map[String, String] = null
{code}
There are some checks of this variable but not in all of them, for example in
appArgs and environmentVariables.
If you don't set _appArgs_ it will cause the following error:
{code:title=error|borderStyle=solid}
17/11/21 14:37:24 INFO MesosClusterScheduler: Reviving Offers.
Exception in thread "Thread-22" java.lang.NullPointerException
at
org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.getDriverCommandValue(MesosClusterScheduler.scala:444)
at
org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.buildDriverCommand(MesosClusterScheduler.scala:451)
at
org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.org$apache$spark$scheduler$cluster$mesos$MesosClusterScheduler$$createTaskInfo(MesosClusterScheduler.scala:538)
at
org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:570)
at
org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:555)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.scheduleTasks(MesosClusterScheduler.scala:555)
at
org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.resourceOffers(MesosClusterScheduler.scala:621)
{code}
Because it's trying to access to it without checking whether is null or not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
