This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ee4e7077bbf3 [SPARK-55832][SQL][TESTS] Reduce periodic GC interval in
ThriftServerQueryTestSuite to mitigate OOM
ee4e7077bbf3 is described below
commit ee4e7077bbf38bd665a2db3d75d6c28eea6d6591
Author: Kent Yao <[email protected]>
AuthorDate: Thu Mar 5 18:11:00 2026 +0800
[SPARK-55832][SQL][TESTS] Reduce periodic GC interval in
ThriftServerQueryTestSuite to mitigate OOM
### What changes were proposed in this pull request?
Reduces `spark.cleaner.periodicGC.interval` from the default 30 minutes to
60 seconds for `ThriftServerQueryTestSuite`.
### Why are the changes needed?
`ThriftServerQueryTestSuite` runs 800+ SQL golden file tests sequentially
in a single SparkSession over ~50 minutes. The default 30-minute periodic GC
interval means only one GC fires during the entire run, allowing memory to
accumulate.
This causes intermittent OOM (`java.lang.OutOfMemoryError: Java heap
space`) during memory-intensive tests like scalar subquery evaluation, with
100+ failed 64MB page allocations. For example:
https://github.com/apache/spark/actions/runs/22659394682/job/65676090341
With 60-second intervals, GC runs ~50 times during the suite, preventing
the memory buildup.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Compilation verified. The change only affects test configuration.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with GitHub Copilot.
Closes #54619 from yaooqinn/SPARK-55832.
Authored-by: Kent Yao <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
---
.../spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
index 9bbd0ae711db..10458a865ffe 100644
---
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
+++
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
@@ -24,8 +24,9 @@ import java.util.{Locale, MissingFormatArgumentException}
import scala.util.control.NonFatal
-import org.apache.spark.SparkException
+import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.CLEANER_PERIODIC_GC_INTERVAL
import org.apache.spark.sql.SQLQueryTestSuite
import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
import org.apache.spark.sql.execution.HiveResult.{getBinaryFormatter,
getTimeFormatters, toHiveString, BinaryFormatter, TimeFormatters}
@@ -70,6 +71,12 @@ import org.apache.spark.util.Utils
// scalastyle:on line.size.limit
class ThriftServerQueryTestSuite extends SQLQueryTestSuite with
SharedThriftServer with Logging {
+ // SPARK-55832: Reduce periodic GC interval to mitigate OOM. This suite runs
800+ SQL golden
+ // file tests sequentially in a single SparkSession, and the default
30-minute interval is
+ // too infrequent for the memory pressure during the ~50-minute run.
+ override protected def sparkConf: SparkConf = {
+ super.sparkConf.set(CLEANER_PERIODIC_GC_INTERVAL, 60L)
+ }
override def mode: ServerMode.Value = ServerMode.binary
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
