This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 3c5f6a2ebf8 [SPARK-44625][CONNECT][FOLLOWUP] Make initialization of SparkConnectExecutionManager lazy 3c5f6a2ebf8 is described below commit 3c5f6a2ebf8aaa1fa9081793ced9a1a76c66c371 Author: Juliusz Sompolski <ju...@databricks.com> AuthorDate: Sat Aug 12 08:48:02 2023 +0900 [SPARK-44625][CONNECT][FOLLOWUP] Make initialization of SparkConnectExecutionManager lazy ### What changes were proposed in this pull request? SparkConnectExecutionManager is a val field in SparkConnectService, so it gets initialized the first time SparkConnectService object is touched. In some scenarios, this can accidentally happen too early, before anything is initialized. SparkConnectExecutionManager wants to read configs from SparkEnv, and when it's not initialized yet it can fail. Make SparkConnectExecutionManager lazy. Since it is `private[connect]` it should not then be initialized before the service is started. Raised https://issues.apache.org/jira/browse/SPARK-44779 as a followup to look at the Spark Connect initialization in general. ### Why are the changes needed? Fix initialization order. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Got one test failure in one specific environment that failed because of wrong initialization order. Closes #42457 from juliuszsompolski/SPARK-44625-fup. Authored-by: Juliusz Sompolski <ju...@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- .../org/apache/spark/sql/connect/service/SparkConnectService.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala index a7b5fbdcec0..fe773e4b704 100644 --- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala +++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala @@ -278,7 +278,7 @@ object SparkConnectService extends Logging { private val userSessionMapping = cacheBuilder(CACHE_SIZE, CACHE_TIMEOUT_SECONDS).build[SessionCacheKey, SessionHolder]() - private[connect] val executionManager = new SparkConnectExecutionManager() + private[connect] lazy val executionManager = new SparkConnectExecutionManager() private[connect] val streamingSessionManager = new SparkConnectStreamingQueryCache() --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org