HeartSaVioR commented on issue #25790: [SPARK-29046] Fix NPE in SQLConf.get when active SparkContext is stopping URL: https://github.com/apache/spark/pull/25790#issuecomment-531517800 https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-jdk-11-ubuntu-testing/1554/consoleFull There's leakage on threads after running previous test: ``` - SPARK-29046: SQLConf.get shouldn't throw NPE when active SparkContext is stopping 19:20:51.356 WARN org.apache.spark.sql.internal.SQLConfSuite: ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.internal.SQLConfSuite, thread names: executor-heartbeater, Timer-684, dag-scheduler-event-loop, Timer-685 ===== ``` and it affects other suites. ``` - determining the number of reducers: complex query 2 19:21:03.377 WARN org.apache.spark.rpc.netty.NettyRpcEnv: Ignored failure: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@56f4464f[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5c0c2d3f[Wrapped task = org.apache.spark.rpc.netty.NettyRpcEnv$$anon$1@52cbd8ce]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@13f6d2f5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] 19:21:03.378 WARN org.apache.spark.executor.Executor: Issue communicating with driver in heartbeater org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:228) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:103) at org.apache.spark.executor.Executor.reportHeartBeat(Executor.scala:911) at org.apache.spark.executor.Executor.$anonfun$heartbeater$1(Executor.scala:205) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1960) at org.apache.spark.Heartbeater$$anon$1.run(Heartbeater.scala:46) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.spark.rpc.RpcEnvStoppedException: RpcEnv already stopped. at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:160) at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:137) at org.apache.spark.rpc.netty.NettyRpcEnv.askAbortable(NettyRpcEnv.scala:240) at org.apache.spark.rpc.netty.NettyRpcEndpointRef.askAbortable(NettyRpcEnv.scala:546) at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:550) at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:102) ... 11 more ``` and ``` - Union two datasets with different pre-shuffle partition number 19:21:06.093 WARN org.apache.spark.sql.execution.ReduceNumShufflePartitionsSuite: ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.execution.ReduceNumShufflePartitionsSuite, thread names: QueryStageCreator-21, QueryStageCreator-26, QueryStageCreator-19, QueryStageCreator-25, QueryStageCreator-22, QueryStageCreator-29, QueryStageCreator-18, QueryStageCreator-17, QueryStageCreator-23, QueryStageCreator-24, QueryStageCreator-31, QueryStageCreator-28, QueryStageCreator-27, QueryStageCreator-16, QueryStageCreator-30, QueryStageCreator-20 ===== 19:21:06.118 WARN org.apache.spark.sql.SparkSession: An existing Spark session exists as the active or default session. This probably means another suite leaked it. Attempting to stop it before continuing. This existing Spark session was created at: scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) org.scalatest.Transformer.apply(Transformer.scala:22) org.scalatest.Transformer.apply(Transformer.scala:20) org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186) org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149) org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184) org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196) org.scalatest.SuperEngine.runTestImpl(Engine.scala:289) org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196) org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178) org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56) org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221) org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214) org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56) org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229) org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396) scala.collection.immutable.List.foreach(List.scala:392) ``` That was actually due to not stop dagScheduler before assigning null. But even I fixed it, it didn't resolve other side-effects coming from Spark's ground rule - Spark doesn't support multiple SparkContext instances. (Other things were broken.) I just learned more for this time, it doesn't work even one is being used actively and other one is sitting down to wait. Never have two SparkContext instances in JVM.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
