sunpe commented on pull request #33154: URL: https://github.com/apache/spark/pull/33154#issuecomment-872026868
hello @kotlovs , @mridulm . Issue [SPARK-34674](https://issues.apache.org/jira/browse/SPARK-34674) said the spark context could not close on k8s. But this [pr](https://github.com/apache/spark/commit/c625eb4f9f970108d93bf3342c7ccb7ec873dc27), will close context only not shell or thrift server. In my case, I use springboot and spark together to create a web app. This app wait for user's request, and do some job on spark. I regist 'SparkSession' object as a spring bean. Like this. ``` @Bean @ConditionalOnMissingBean(SparkSession.class) public SparkSession sparkSession(SparkConf conf) { return SparkSession.builder() .enableHiveSupport() .config(conf) .getOrCreate(); } ``` And then Pack the spring application as a jar. Use the `spark submit` command to start my application. While application is started up, the spark context will stop. soonly. As testing, I add 3 log in core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala. ``` try { logInfo("## app start ") app.start(childArgs.toArray, sparkConf) logInfo("## app start over ") } catch { case t: Throwable => throw findCause(t) } finally { logInfo("## finally clean spark context ") if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) && !isThriftServer(args.mainClass)) { try { SparkContext.getActive.foreach(_.stop()) } catch { case e: Throwable => logError(s"Failed to close SparkContext: $e") } } } ``` While application started, app logs will like this. While the app started, the spark context will stop soonly. ``` ## app start . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.5.2) ... 21/07/01 15:05:10 INFO YarnClientSchedulerBackend: Application application_1619358582322_6529891 has started running. 21/07/01 15:05:10 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40461. ... 21/07/01 15:05:10 INFO SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 21/07/01 15:05:10 INFO Utils: Using initial executors = 1, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances 21/07/01 15:05:10 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM) 21/07/01 15:05:11 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000000000(ns) 21/07/01 15:05:11 INFO WelcomePageHandlerMapping: Adding welcome page template: index 21/07/01 15:05:12 INFO Http11NioProtocol: Starting ProtocolHandler ["http-nio-9000"] 21/07/01 15:05:12 INFO TomcatWebServer: Tomcat started on port(s): 9000 (http) with context path '' 21/07/01 15:05:12 INFO SpringApplication: Started application in 2939.592 seconds (JVM running for 2942.937) ## app start over ## finally clean spark context 21/07/01 15:05:12 INFO AbstractConnector: Stopped Spark@23cd4246{HTTP/1.1, (http/1.1)}{0.0.0.0:4040} 21/07/01 15:05:12 INFO SparkUI: Stopped Spark web UI at http://xxxxxxxxx:4040 21/07/01 15:05:12 INFO YarnClientSchedulerBackend: Interrupting monitor thread 21/07/01 15:05:12 INFO YarnClientSchedulerBackend: Shutting down all executors 21/07/01 15:05:12 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down 21/07/01 15:05:12 INFO YarnClientSchedulerBackend: YARN client scheduler backend Stopped 21/07/01 15:05:12 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 21/07/01 15:05:13 INFO MemoryStore: MemoryStore cleared 21/07/01 15:05:13 INFO BlockManager: BlockManager stopped 21/07/01 15:05:13 INFO BlockManagerMaster: BlockManagerMaster stopped 21/07/01 15:05:13 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 21/07/01 15:05:13 INFO SparkContext: Successfully stopped SparkContext ``` And after fixes. ``` ## app start . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.5.2) .... 21/07/01 16:05:51 INFO YarnClientSchedulerBackend: Application application_1619358582322_6532933 has started running. 21/07/01 16:05:51 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42549. 21/07/01 16:05:51 INFO NettyBlockTransferService: Server created on xxxxxxxxx:42549 21/07/01 16:05:51 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 21/07/01 16:05:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, xxxxxxx, 42549, None) 21/07/01 16:05:51 INFO BlockManagerMasterEndpoint: Registering block manager xxxxxxxxx:42549 with 5.2 GiB RAM, BlockManagerId(driver, xxxxxxx, 42549, None) 21/07/01 16:05:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, xxxxxxxx, 42549, None) 21/07/01 16:05:51 INFO BlockManager: external shuffle service port = 7337 21/07/01 16:05:51 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, xxxxxxxx, 42549, None) 21/07/01 16:05:51 INFO ServerInfo: Adding filter to /metrics/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 21/07/01 16:05:51 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@60510791{/metrics/json,null,AVAILABLE,@Spark} 21/07/01 16:05:52 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM) 21/07/01 16:05:54 INFO Utils: Using initial executors = 1, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances 21/07/01 16:05:54 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000000000(ns) 21/07/01 16:05:55 INFO WelcomePageHandlerMapping: Adding welcome page template: index 21/07/01 16:05:55 INFO Http11NioProtocol: Starting ProtocolHandler ["http-nio-9000"] 21/07/01 16:05:55 INFO TomcatWebServer: Tomcat started on port(s): 9000 (http) with context path '' 21/07/01 16:05:55 INFO SpringApplication: Started application in 848.362 seconds (JVM running for 851.343) ## app start over ## finally clean spark context ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
