This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 343471dac4b9 [SPARK-49409][CONNECT] Adjust the default value of
CONNECT_SESSION_PLAN_CACHE_SIZE
343471dac4b9 is described below
commit 343471dac4b96b43a09763d759b6c30760fb626e
Author: Changgyoo Park <[email protected]>
AuthorDate: Fri Aug 30 08:04:04 2024 -0400
[SPARK-49409][CONNECT] Adjust the default value of
CONNECT_SESSION_PLAN_CACHE_SIZE
### What changes were proposed in this pull request?
Set the default value of CONNECT_SESSION_PLAN_CACHE_SIZE to 16 which was
previously 5.
### Why are the changes needed?
There is a rare use case where a data frame is built from multiple
dependent data frames, and while creating the last dependent data frame, all
the other dependent data frames are invalidated from the plan cache, causing
them to be analysed again. For this particular case, the performance of
creating the data frame was improved from 41 minutes to 1 minute.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Performance test.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47937 from changgyoopark-db/SPARK-49409.
Authored-by: Changgyoo Park <[email protected]>
Signed-off-by: Herman van Hovell <[email protected]>
---
.../src/main/scala/org/apache/spark/sql/connect/config/Connect.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
index dc45684a75eb..92709ff29a1c 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
@@ -289,7 +289,7 @@ object Connect {
.version("4.0.0")
.internal()
.intConf
- .createWithDefault(5)
+ .createWithDefault(16)
val CONNECT_SESSION_PLAN_CACHE_ENABLED =
buildConf("spark.connect.session.planCache.enabled")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
