Shixiong Zhu created SPARK-9479: ----------------------------------- Summary: ReceiverTrackerSuite fails for maven build Key: SPARK-9479 URL: https://issues.apache.org/jira/browse/SPARK-9479 Project: Spark Issue Type: Bug Components: Streaming, Tests Reporter: Shixiong Zhu
The test failure is here: https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/3109/ I saw the following exception in the log: {code} org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.NullPointerException org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63) org.apache.spark.SparkContext.broadcast(SparkContext.scala:1297) org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:834) {code} This exception is because SparkEnv.get returns null. I found the maven build is different from the sbt build. The maven build will create all Suite classes at the beginning. `ReceiverTrackerSuite` creates StreamingContext (SparkContext) in the constructor. That means SparkContext is created very early. And the global SparkEnv will be set to null in the previous test. Therefore we saw the above exception when running `Receiver tracker - propagates rate limit` in `ReceiverTrackerSuite`. This test was added recently. Note: the previous tests in `ReceiverTrackerSuite` didn't use SparkContext actually, that's why we didn't see such failure before. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org