[spark] branch branch-3.3 updated: [SPARK-40682][SQL][TESTS] Set `spark.driver.maxResultSize` to 3g in `SqlBasedBenchmark`

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
     new 9f8eef8bc7f [SPARK-40682][SQL][TESTS] Set `spark.driver.maxResultSize` 
to 3g in `SqlBasedBenchmark`
9f8eef8bc7f is described below

commit 9f8eef8bc7fbb5f9a0fe7a4f5c99da0b59b74c07
Author: Dongjoon Hyun <dongj...@apache.org>
AuthorDate: Thu Oct 6 18:33:19 2022 -0700

    [SPARK-40682][SQL][TESTS] Set `spark.driver.maxResultSize` to 3g in 
`SqlBasedBenchmark`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to set `spark.driver.maxResultSize` to `3g` from the default 
value, `1g`,  in `SqlBasedBenchmark`.
    
    ### Why are the changes needed?
    
    Apache Spark benchmark is using `4g` JVM memory. We can utilize it for 
`spark.driver.maxResultSize` in some cases after this PR.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a test case and unblocks only the cases which failed before.
    
    ### How was this patch tested?
    
    N/A
    
    Closes #38136 from dongjoon-hyun/SPARK-40682.
    
    Authored-by: Dongjoon Hyun <dongj...@apache.org>
    Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
    (cherry picked from commit e02d518f11c3a83780170e6b7221520639393729)
    Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
 .../org/apache/spark/sql/execution/benchmark/SqlBasedBenchmark.scala    | 2 ++
 1 file changed, 2 insertions(+)

diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/SqlBasedBenchmark.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/SqlBasedBenchmark.scala
index f84172278be..78d6b015803 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/SqlBasedBenchmark.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/SqlBasedBenchmark.scala
@@ -18,6 +18,7 @@
 package org.apache.spark.sql.execution.benchmark
 
 import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
+import org.apache.spark.internal.config.MAX_RESULT_SIZE
 import org.apache.spark.internal.config.UI.UI_ENABLED
 import org.apache.spark.sql.{Dataset, SparkSession}
 import org.apache.spark.sql.SaveMode.Overwrite
@@ -41,6 +42,7 @@ trait SqlBasedBenchmark extends BenchmarkBase with SQLHelper {
       .config(SQLConf.SHUFFLE_PARTITIONS.key, 1)
       .config(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key, 1)
       .config(UI_ENABLED.key, false)
+      .config(MAX_RESULT_SIZE.key, "3g")
       .getOrCreate()
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org