[
https://issues.apache.org/jira/browse/SPARK-35949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SunPeng updated SPARK-35949:
----------------------------
Description:
Spark 3.1.2 with springboot, on client mode, the spark context while stopped
while the application is started.
{code:java}
//代码占位符
@Bean
@ConditionalOnMissingBean(SparkSession.class)
public SparkSession sparkSession(SparkConf conf) {
return SparkSession.builder()
.enableHiveSupport()
.config(conf)
.getOrCreate();
}{code}
{quote} 21/06/30 12:03:38 INFO YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
21/06/30 12:03:38 INFO Utils: Using initial executors = 1, max of
spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors
and spark.executor.instances
21/06/30 12:03:38 INFO YarnClientSchedulerBackend: SchedulerBackend is ready
for scheduling beginning after waiting maxRegisteredResourcesWaitingTime:
30000000000(ns)
21/06/30 12:03:39 INFO WelcomePageHandlerMapping: Adding welcome page
template: index
21/06/30 12:03:40 INFO Http11NioProtocol: Starting ProtocolHandler
["http-nio-9000"]
21/06/30 12:03:40 INFO TomcatWebServer: Tomcat started on port(s): 9000 (http)
with context path ''
21/06/30 12:03:40 INFO SpringApplication: Started application in 525.411
seconds (JVM running for 529.958)
21/06/30 12:03:40 INFO AbstractConnector: Stopped Spark@3e1d19ea\{HTTP/1.1,
(http/1.1)}
Unknown macro: \{0.0.0.0}
21/06/30 12:03:40 INFO SparkUI: Stopped Spark web UI at
21/06/30 12:03:40 INFO YarnClientSchedulerBackend: Interrupting monitor thread
21/06/30 12:03:40 INFO YarnClientSchedulerBackend: Shutting down all executors
21/06/30 12:03:40 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each
executor to shut down
21/06/30 12:03:40 INFO YarnClientSchedulerBackend: YARN client scheduler
backend Stopped
21/06/30 12:03:40 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
21/06/30 12:03:40 INFO MemoryStore: MemoryStore cleared
21/06/30 12:03:40 INFO BlockManager: BlockManager stopped
21/06/30 12:03:40 INFO BlockManagerMaster: BlockManagerMaster stopped
21/06/30 12:03:40 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
21/06/30 12:03:40 INFO SparkContext: Successfully stopped SparkContext
21/06/30 12:03:40 INFO [/]: Initializing Spring DispatcherServlet
'dispatcherServlet'
21/06/30 12:03:40 INFO DispatcherServlet: Initializing Servlet
'dispatcherServlet'
21/06/30 12:03:40 INFO DispatcherServlet: Completed initialization in 1 ms
{quote}
was:
In spark 3.1.2, on client mode, the spark context while stopped while the
application is started.
{quote} 21/06/30 12:03:38 INFO YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
21/06/30 12:03:38 INFO Utils: Using initial executors = 1, max of
spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors
and spark.executor.instances
21/06/30 12:03:38 INFO YarnClientSchedulerBackend: SchedulerBackend is ready
for scheduling beginning after waiting maxRegisteredResourcesWaitingTime:
30000000000(ns)
21/06/30 12:03:39 INFO WelcomePageHandlerMapping: Adding welcome page
template: index
21/06/30 12:03:40 INFO Http11NioProtocol: Starting ProtocolHandler
["http-nio-9000"]
21/06/30 12:03:40 INFO TomcatWebServer: Tomcat started on port(s): 9000 (http)
with context path ''
21/06/30 12:03:40 INFO SpringApplication: Started application in 525.411
seconds (JVM running for 529.958)
21/06/30 12:03:40 INFO AbstractConnector: Stopped Spark@3e1d19ea\{HTTP/1.1,
(http/1.1)}
Unknown macro: \{0.0.0.0}
21/06/30 12:03:40 INFO SparkUI: Stopped Spark web UI at
21/06/30 12:03:40 INFO YarnClientSchedulerBackend: Interrupting monitor thread
21/06/30 12:03:40 INFO YarnClientSchedulerBackend: Shutting down all executors
21/06/30 12:03:40 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each
executor to shut down
21/06/30 12:03:40 INFO YarnClientSchedulerBackend: YARN client scheduler
backend Stopped
21/06/30 12:03:40 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
21/06/30 12:03:40 INFO MemoryStore: MemoryStore cleared
21/06/30 12:03:40 INFO BlockManager: BlockManager stopped
21/06/30 12:03:40 INFO BlockManagerMaster: BlockManagerMaster stopped
21/06/30 12:03:40 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
21/06/30 12:03:40 INFO SparkContext: Successfully stopped SparkContext
21/06/30 12:03:40 INFO [/]: Initializing Spring DispatcherServlet
'dispatcherServlet'
21/06/30 12:03:40 INFO DispatcherServlet: Initializing Servlet
'dispatcherServlet'
21/06/30 12:03:40 INFO DispatcherServlet: Completed initialization in 1 ms
{quote}
> working with spring boot, spark context will stopped while application is
> started.
> ----------------------------------------------------------------------------------
>
> Key: SPARK-35949
> URL: https://issues.apache.org/jira/browse/SPARK-35949
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.1.2
> Reporter: SunPeng
> Priority: Critical
>
> Spark 3.1.2 with springboot, on client mode, the spark context while stopped
> while the application is started.
>
>
> {code:java}
> //代码占位符
> @Bean
> @ConditionalOnMissingBean(SparkSession.class)
> public SparkSession sparkSession(SparkConf conf) {
> return SparkSession.builder()
> .enableHiveSupport()
> .config(conf)
> .getOrCreate();
> }{code}
>
> {quote} 21/06/30 12:03:38 INFO YarnSchedulerBackend$YarnSchedulerEndpoint:
> ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
> 21/06/30 12:03:38 INFO Utils: Using initial executors = 1, max of
> spark.dynamicAllocation.initialExecutors,
> spark.dynamicAllocation.minExecutors and spark.executor.instances
> 21/06/30 12:03:38 INFO YarnClientSchedulerBackend: SchedulerBackend is ready
> for scheduling beginning after waiting maxRegisteredResourcesWaitingTime:
> 30000000000(ns)
> 21/06/30 12:03:39 INFO WelcomePageHandlerMapping: Adding welcome page
> template: index
> 21/06/30 12:03:40 INFO Http11NioProtocol: Starting ProtocolHandler
> ["http-nio-9000"]
> 21/06/30 12:03:40 INFO TomcatWebServer: Tomcat started on port(s): 9000
> (http) with context path ''
> 21/06/30 12:03:40 INFO SpringApplication: Started application in 525.411
> seconds (JVM running for 529.958)
> 21/06/30 12:03:40 INFO AbstractConnector: Stopped Spark@3e1d19ea\{HTTP/1.1,
> (http/1.1)}
> Unknown macro: \{0.0.0.0}
> 21/06/30 12:03:40 INFO SparkUI: Stopped Spark web UI at
> 21/06/30 12:03:40 INFO YarnClientSchedulerBackend: Interrupting monitor
> thread
> 21/06/30 12:03:40 INFO YarnClientSchedulerBackend: Shutting down all
> executors
> 21/06/30 12:03:40 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each
> executor to shut down
> 21/06/30 12:03:40 INFO YarnClientSchedulerBackend: YARN client scheduler
> backend Stopped
> 21/06/30 12:03:40 INFO MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 21/06/30 12:03:40 INFO MemoryStore: MemoryStore cleared
> 21/06/30 12:03:40 INFO BlockManager: BlockManager stopped
> 21/06/30 12:03:40 INFO BlockManagerMaster: BlockManagerMaster stopped
> 21/06/30 12:03:40 INFO
> OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!
> 21/06/30 12:03:40 INFO SparkContext: Successfully stopped SparkContext
> 21/06/30 12:03:40 INFO [/]: Initializing Spring DispatcherServlet
> 'dispatcherServlet'
> 21/06/30 12:03:40 INFO DispatcherServlet: Initializing Servlet
> 'dispatcherServlet'
> 21/06/30 12:03:40 INFO DispatcherServlet: Completed initialization in 1 ms
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
