[ https://issues.apache.org/jira/browse/SPARK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946457#comment-15946457 ]
Saisai Shao commented on SPARK-20128: ------------------------------------- Here the exception is from MasterSource, which only exists in Standalone Master, I think it should not be related to SparkContext, may be the Master is not cleanly stopped. Also as I remembered by default we will not enable ConsoleReporter, not sure how this could be happened. > MetricsSystem not always killed in SparkContext.stop() > ------------------------------------------------------ > > Key: SPARK-20128 > URL: https://issues.apache.org/jira/browse/SPARK-20128 > Project: Spark > Issue Type: Test > Components: Spark Core, Tests > Affects Versions: 2.2.0 > Reporter: Imran Rashid > Labels: flaky-test > > One Jenkins run failed due to the MetricsSystem never getting killed after a > failed test, which led that test to hang and the tests to timeout: > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75176 > {noformat} > 17/03/24 13:44:19.537 dag-scheduler-event-loop ERROR > DAGSchedulerEventProcessLoop: DAGSchedulerEventProcessLoop failed; shutting > down SparkContext > java.lang.ArrayIndexOutOfBoundsException: -1 > at > org.apache.spark.MapOutputTrackerMaster$$anonfun$getEpochForMapOutput$1.apply(MapOutputTracker.scala:431) > at > org.apache.spark.MapOutputTrackerMaster$$anonfun$getEpochForMapOutput$1.apply(MapOutputTracker.scala:430) > at scala.Option.flatMap(Option.scala:171) > at > org.apache.spark.MapOutputTrackerMaster.getEpochForMapOutput(MapOutputTracker.scala:430) > at > org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1298) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1731) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1689) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1678) > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > 17/03/24 13:44:19.540 dispatcher-event-loop-11 INFO > MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! > 17/03/24 13:44:19.546 stop-spark-context INFO MemoryStore: MemoryStore cleared > 17/03/24 13:44:19.546 stop-spark-context INFO BlockManager: BlockManager > stopped > 17/03/24 13:44:19.546 stop-spark-context INFO BlockManagerMaster: > BlockManagerMaster stopped > 17/03/24 13:44:19.546 dispatcher-event-loop-16 INFO > OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: > OutputCommitCoordinator stopped! > 17/03/24 13:44:19.547 stop-spark-context INFO SparkContext: Successfully > stopped SparkContext > 17/03/24 14:02:19.934 metrics-console-reporter-1-thread-1 ERROR > ScheduledReporter: RuntimeException thrown from ConsoleReporter#report. > Exception was suppressed. > java.lang.NullPointerException > at > org.apache.spark.deploy.master.MasterSource$$anon$2.getValue(MasterSource.scala:35) > at > org.apache.spark.deploy.master.MasterSource$$anon$2.getValue(MasterSource.scala:34) > at > com.codahale.metrics.ConsoleReporter.printGauge(ConsoleReporter.java:239) > ... > {noformat} > unfortunately I didn't save the entire test logs, but what happens is the > initial IndexOutOfBoundsException is a real bug, which causes the > SparkContext to stop, and the test to fail. However, the MetricsSystem > somehow stays alive, and since its not a daemon thread, it just hangs, and > every 20 mins we get that NPE from within the metrics system as it tries to > report. > I am totally perplexed at how this can happen, it looks like the metric > system should always get stopped by the time we see > {noformat} > 17/03/24 13:44:19.547 stop-spark-context INFO SparkContext: Successfully > stopped SparkContext > {noformat} > I don't think I've ever seen this in a real spark use, but it doesn't look > like something which is limited to tests, whatever the cause. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org