This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new a8e6edc3e4a6 [SPARK-58623][SQL] Extend ExpandBenchmark with
CSE/rewrite matrix
a8e6edc3e4a6 is described below
commit a8e6edc3e4a66351e42fb717e7bb784018325d89
Author: James Xu <[email protected]>
AuthorDate: Wed Aug 12 16:30:43 2026 +0800
[SPARK-58623][SQL] Extend ExpandBenchmark with CSE/rewrite matrix
### What changes were proposed in this pull request?
Extend the subexpression-elimination benchmark in `ExpandBenchmark` (added
in
SPARK-58501) from two cases to a full 2x2 matrix over
`spark.sql.optimizer.rewriteCountDistinctConditional` and
`spark.sql.subexpressionElimination`:
| Case | Expand branches | daysSince evals/row |
|---|---|---|
| rewrite off, CSE off (base) | 10 | 18 |
| rewrite on, CSE off | 2 | 18 |
| rewrite on, CSE on | 2 | 1 |
| rewrite off, CSE on | 10 | 1 |
The four cases attribute the benefit of each optimization independently: the
conditional COUNT(DISTINCT) rewrite cuts data amplification (10x to 2x),
while
subexpression elimination cuts per-row subexpression evaluations (18 to 1).
Also add an optional section filter (`ratio`, `char`, `subexpr`) and a
`smoke`
mode to `runBenchmarkSuite` so a single section can be measured without
editing
the file:
```
build/sbt "sql/Test/runMain
org.apache.spark.sql.execution.benchmark.ExpandBenchmark subexpr"
build/sbt "sql/Test/runMain
org.apache.spark.sql.execution.benchmark.ExpandBenchmark smoke"
```
Regenerate `sql/core/benchmarks/ExpandBenchmark-jdk21-results.txt`.
### Why are the changes needed?
SPARK-58501 made `ExpandExec` participate in whole-stage subexpression
elimination, so a subexpression shared across Expand branches is evaluated
once
per input row instead of once per branch occurrence. The previous benchmark
only
measured the CSE toggle on the 2-branch plan produced by the conditional
rewrite. The missing cell (rewrite off, CSE on) shows the CSE benefit holds
even
without the data-amplification reduction and lets reviewers attribute the
combined speedup to each optimization.
On Apple M3 Pro / JDK 21, 5M rows:
| Case | Best (ms) | Avg (ms) | Relative |
|---|---|---|---|
| rewrite off, CSE off | 84,450 | 84,850 | 1.0X |
| rewrite on, CSE off | 41,354 | 41,544 | 2.0X |
| rewrite on, CSE on | 10,712 | 10,843 | 7.9X |
| rewrite off, CSE on | 53,281 | 54,021 | 1.6X |
### Does this PR introduce _any_ user-facing change?
No. Benchmark/test-only change.
### How was this patch tested?
- `build/sbt "sql/Test/compile"` and `build/sbt "sql/Test/scalastyle"` pass.
- Smoke run (`... ExpandBenchmark smoke`) verifies all sections run with a
single iteration on tiny data.
- Section filter verified: `... ExpandBenchmark subexpr` runs only the
subexpression-elimination section.
- Full results committed in
`sql/core/benchmarks/ExpandBenchmark-jdk21-results.txt`
(Apple M3 Pro, JDK 21). Results for other JDKs can be regenerated via the
GitHub Actions benchmark workflow:
https://spark.apache.org/developer-tools.html#github-workflow-benchmarks
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: pi 0.84.0
Closes #57949 from xumingming/expand-benchmark-followup-4.x.
Authored-by: James Xu <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
.../benchmarks/ExpandBenchmark-jdk21-results.txt | 79 ++++++-----
.../sql/execution/benchmark/ExpandBenchmark.scala | 144 ++++++++++++++++-----
2 files changed, 155 insertions(+), 68 deletions(-)
diff --git a/sql/core/benchmarks/ExpandBenchmark-jdk21-results.txt
b/sql/core/benchmarks/ExpandBenchmark-jdk21-results.txt
index 919cdd29abb5..9cc85852fafc 100644
--- a/sql/core/benchmarks/ExpandBenchmark-jdk21-results.txt
+++ b/sql/core/benchmarks/ExpandBenchmark-jdk21-results.txt
@@ -2,58 +2,69 @@
Expand: varying number of COUNT(DISTINCT)
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.17.0-1008-azure
-Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Mac OS X 26.5.2
+Apple M3 Pro
2 distinct aggregates: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-with sum - baseline (ratio 3) 3017 3070
49 3.5 287.8 1.0X
-with sum - optimized (ratio 3) 2932 2982
44 3.6 279.6 1.0X
-pure distinct - baseline (ratio 2) 1848 1902
52 5.7 176.2 1.6X
-pure distinct - optimized (ratio 2) 669 678
6 15.7 63.8 4.5X
+with sum - baseline (ratio 3) 2518 2821
425 4.2 240.1 1.0X
+with sum - optimized (ratio 3) 2363 2429
52 4.4 225.4 1.1X
+pure distinct - baseline (ratio 2) 1382 1471
149 7.6 131.8 1.8X
+pure distinct - optimized (ratio 2) 400 409
11 26.2 38.2 6.3X
-OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.17.0-1008-azure
-Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Mac OS X 26.5.2
+Apple M3 Pro
4 distinct aggregates: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-with sum - baseline (ratio 5) 5480 5526
37 1.9 522.6 1.0X
-with sum - optimized (ratio 5) 5395 5484
81 1.9 514.5 1.0X
-pure distinct - baseline (ratio 4) 4040 4144
79 2.6 385.3 1.4X
-pure distinct - optimized (ratio 4) 826 861
31 12.7 78.8 6.6X
+with sum - baseline (ratio 5) 4569 4650
60 2.3 435.8 1.0X
+with sum - optimized (ratio 5) 4584 4644
41 2.3 437.2 1.0X
+pure distinct - baseline (ratio 4) 3393 3471
64 3.1 323.6 1.3X
+pure distinct - optimized (ratio 4) 471 510
44 22.3 44.9 9.7X
-OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.17.0-1008-azure
-Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Mac OS X 26.5.2
+Apple M3 Pro
6 distinct aggregates: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-with sum - baseline (ratio 7) 8417 8451
21 1.2 802.7 1.0X
-with sum - optimized (ratio 7) 8369 8428
38 1.3 798.1 1.0X
-pure distinct - baseline (ratio 6) 6586 6618
32 1.6 628.1 1.3X
-pure distinct - optimized (ratio 6) 841 875
32 12.5 80.2 10.0X
+with sum - baseline (ratio 7) 6311 6527
288 1.7 601.8 1.0X
+with sum - optimized (ratio 7) 5659 5853
206 1.9 539.7 1.1X
+pure distinct - baseline (ratio 6) 5145 5184
28 2.0 490.6 1.2X
+pure distinct - optimized (ratio 6) 506 518
10 20.7 48.2 12.5X
-OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.17.0-1008-azure
-Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Mac OS X 26.5.2
+Apple M3 Pro
8 distinct aggregates: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-with sum - baseline (ratio 9) 12404 12546
147 0.8 1182.9 1.0X
-with sum - optimized (ratio 9) 12481 12552
90 0.8 1190.3 1.0X
-pure distinct - baseline (ratio 8) 10836 10950
68 1.0 1033.4 1.1X
-pure distinct - optimized (ratio 8) 1774 1965
145 5.9 169.2 7.0X
+with sum - baseline (ratio 9) 8226 8286
36 1.3 784.5 1.0X
+with sum - optimized (ratio 9) 8218 8272
46 1.3 783.8 1.0X
+pure distinct - baseline (ratio 8) 7086 7134
40 1.5 675.8 1.2X
+pure distinct - optimized (ratio 8) 955 976
12 11.0 91.1 8.6X
================================================================================================
Expand: varying data characteristics (pure distinct)
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.17.0-1008-azure
-Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Mac OS X 26.5.2
+Apple M3 Pro
6 pure distinct aggs with varying data: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-1K groups, moderate card - baseline 6537 6561
24 1.6 623.4 1.0X
-1K groups, moderate card - optimized 963 984
21 10.9 91.8 6.8X
-100K groups, moderate card - baseline 13314 13441
143 0.8 1269.7 0.5X
-100K groups, moderate card - optimized 4020 4144
145 2.6 383.4 1.6X
-1K groups, low card (5 vals) - baseline 6387 6415
29 1.6 609.1 1.0X
-1K groups, low card (5 vals) - optimized 815 829
11 12.9 77.7 8.0X
-no grouping key - baseline 4072 4141
55 2.6 388.4 1.6X
-no grouping key - optimized 339 353
19 31.0 32.3 19.3X
+1K groups, moderate card - baseline 5175 5218
37 2.0 493.6 1.0X
+1K groups, moderate card - optimized 497 507
10 21.1 47.4 10.4X
+100K groups, moderate card - baseline 9323 9420
130 1.1 889.1 0.6X
+100K groups, moderate card - optimized 1993 2069
139 5.3 190.1 2.6X
+1K groups, low card (5 vals) - baseline 5277 5315
23 2.0 503.2 1.0X
+1K groups, low card (5 vals) - optimized 466 474
8 22.5 44.5 11.1X
+no grouping key - baseline 3577 3610
23 2.9 341.1 1.4X
+no grouping key - optimized 218 221
5 48.2 20.8 23.8X
+================================================================================================
+Expand: subexpression elimination across branches
+================================================================================================
+OpenJDK 64-Bit Server VM 21.0.3+9-LTS on Mac OS X 26.5.2
+Apple M3 Pro
+9 conditional COUNT(DISTINCT) + 9 conditional SUM sharing one subexpression:
Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-----------------------------------------------------------------------------------------------------------------------------------------------------------
+rewrite off, CSE off (10x amplify, 18 evals/row)
84450 84850 382 0.1 16107.6 1.0X
+rewrite on, CSE off (2x amplify, 18 evals/row)
41354 41544 312 0.1 7887.7 2.0X
+rewrite on, CSE on (2x amplify, 1 eval/row)
10712 10843 120 0.5 2043.2 7.9X
+rewrite off, CSE on (10x amplify, 1 eval/row)
53281 54021 722 0.1 10162.6 1.6X
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExpandBenchmark.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExpandBenchmark.scala
index 586dfec19620..42644708ab5b 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExpandBenchmark.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExpandBenchmark.scala
@@ -33,10 +33,22 @@ import org.apache.spark.sql.internal.SQLConf
* OptimizeExpand rule that pre-aggregates data before the Expand. Controlled
by
* spark.sql.optimizer.optimizeExpandRatio (default -1 = disabled).
*
- * It also measures subexpression elimination in Expand: conditional-aggregate
- * queries whose conditions all share the same expensive subexpression, which
- * ends up in every branch of the Expand. Controlled by
- * spark.sql.subexpressionElimination.enabled (default true).
+ * It also measures two optimizations that target a shared expensive
+ * subexpression landing in the branches of the Expand, using a traffic/BI
+ * "N-day active users" rollup (conditional COUNT(DISTINCT) and conditional SUM
+ * aggregates whose conditions all share the same datetime subexpression):
+ * - rewriteCountDistinctConditional collapses N conditional COUNT(DISTINCT)
+ * on the same base column into one distinct group, shrinking the Expand
+ * fan-out from Nx to 1x (cuts data amplification, not the per-row
+ * subexpression evaluation cost);
+ * - subexpressionElimination (in Expand, via whole-stage codegen) evaluates
+ * the shared subexpression once per input row instead of once per Expand
+ * branch occurrence.
+ * The four cases form a 2x2 matrix over the two optimizations (base, +rewrite,
+ * +rewrite+CSE, and +CSE alone), so the benefit of each optimization can be
+ * attributed independently. Controlled by
+ * spark.sql.optimizer.rewriteCountDistinctConditional.enabled (default true)
+ * and spark.sql.subexpressionElimination.enabled (default true).
*
* To run this benchmark:
* {{{
@@ -45,6 +57,14 @@ import org.apache.spark.sql.internal.SQLConf
* SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/Test/runMain <this
class>"
* Results will be written to "benchmarks/ExpandBenchmark-results.txt".
* }}}
+ *
+ * Optional arguments select benchmark sections (default: all):
+ * {{{
+ * <this class> ratio # Expand: varying number of COUNT(DISTINCT)
+ * <this class> char # Expand: varying data characteristics (pure
distinct)
+ * <this class> subexpr # Expand: subexpression elimination across branches
+ * <this class> smoke # all sections with tiny data and a single iteration
+ * }}}
*/
object ExpandBenchmark extends SqlBasedBenchmark {
@@ -79,7 +99,7 @@ object ExpandBenchmark extends SqlBasedBenchmark {
private def expandRatioBenchmark(
title: String, N: Long, numDistinct: Int,
- table: String): Unit = {
+ table: String, numIters: Int = 5): Unit = {
val benchmark = new Benchmark(title, N, output = output)
val sqlWithSum = countDistinctQuery(
@@ -94,28 +114,28 @@ object ExpandBenchmark extends SqlBasedBenchmark {
benchmark.addCase(
s"with sum - baseline (ratio $ratioWithSum)",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "-1") {
spark.sql(sqlWithSum).noop()
}
}
benchmark.addCase(
s"with sum - optimized (ratio $ratioWithSum)",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "2") {
spark.sql(sqlWithSum).noop()
}
}
benchmark.addCase(
s"pure distinct - baseline (ratio $ratioPure)",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "-1") {
spark.sql(sqlPure).noop()
}
}
benchmark.addCase(
s"pure distinct - optimized (ratio $ratioPure)",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "2") {
spark.sql(sqlPure).noop()
}
@@ -143,8 +163,9 @@ object ExpandBenchmark extends SqlBasedBenchmark {
* whose conditions all share the same expensive datetime subexpression
* (parse `pt`, format it back, then `datediff` against a fixed date).
* `RewriteDistinctAggregates` places those conditions in the branches of an
- * Expand, so the shared subexpression is codegen'd once per branch without
- * subexpression elimination.
+ * Expand, so without subexpression elimination the shared subexpression is
+ * codegen'd once per occurrence: 9 in the distinct branch(es) and 9 in the
+ * SUM branch, i.e. 18 evaluations per input row.
*/
private def conditionalAggsQuery(table: String, windows: Seq[Int]): String =
{
val daysSince = "datediff(" +
@@ -161,18 +182,63 @@ object ExpandBenchmark extends SqlBasedBenchmark {
|GROUP BY user_id""".stripMargin
}
+ /**
+ * Measures a traffic/BI "N-day active users" rollup (see
[[conditionalAggsQuery]])
+ * under all four combinations of the two optimizations that affect the
shared
+ * `daysSince` subexpression in the Expand branches:
+ * - base: rewriteCountDistinctConditional off, subexpression elimination
off
+ * -> 10 Expand branches (one per conditional COUNT(DISTINCT) plus one for
+ * the SUMs), daysSince evaluated 18 times per input row;
+ * - +rewrite: rewriteCountDistinctConditional on, CSE off -> the 9
+ * conditional COUNT(DISTINCT) expressions collapse into one distinct
group, so the
+ * Expand has 2 branches (data amplification drops from 10x to 2x), but
+ * daysSince is still evaluated 18 times per input row;
+ * - +rewrite+CSE: both on -> the Expand runs the standard whole-stage
+ * subexpression elimination, evaluating daysSince once per input row
+ * before the branch loop;
+ * - +CSE alone: rewriteCountDistinctConditional off, subexpression
+ * elimination on -> the same 10-branch Expand as the base case, but
+ * daysSince is evaluated once per input row, isolating the CSE benefit
+ * without the data-amplification reduction.
+ */
private def subExprEliminationBenchmark(
- title: String, N: Long, table: String): Unit = {
+ title: String, N: Long, table: String, numIters: Int = 3): Unit = {
val benchmark = new Benchmark(title, N, output = output)
val sql = conditionalAggsQuery(table, Seq(1, 7, 14, 30, 60, 90, 180, 365,
730))
- benchmark.addCase("CSE disabled (re-evaluate per branch)", numIters = 3) {
_ =>
- withSQLConf(SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> "false") {
+ benchmark.addCase(
+ "rewrite off, CSE off (10x amplify, 18 evals/row)",
+ numIters = numIters) { _ =>
+ withSQLConf(
+ SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> "false",
+ SQLConf.REWRITE_COUNT_DISTINCT_CONDITIONAL_ENABLED.key -> "false") {
spark.sql(sql).noop()
}
}
- benchmark.addCase("CSE enabled (evaluate once per row)", numIters = 3) { _
=>
- withSQLConf(SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> "true") {
+ benchmark.addCase(
+ "rewrite on, CSE off (2x amplify, 18 evals/row)",
+ numIters = numIters) { _ =>
+ withSQLConf(
+ SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> "false",
+ SQLConf.REWRITE_COUNT_DISTINCT_CONDITIONAL_ENABLED.key -> "true") {
+ spark.sql(sql).noop()
+ }
+ }
+ benchmark.addCase(
+ "rewrite on, CSE on (2x amplify, 1 eval/row)",
+ numIters = numIters) { _ =>
+ withSQLConf(
+ SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> "true",
+ SQLConf.REWRITE_COUNT_DISTINCT_CONDITIONAL_ENABLED.key -> "true") {
+ spark.sql(sql).noop()
+ }
+ }
+ benchmark.addCase(
+ "rewrite off, CSE on (10x amplify, 1 eval/row)",
+ numIters = numIters) { _ =>
+ withSQLConf(
+ SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> "true",
+ SQLConf.REWRITE_COUNT_DISTINCT_CONDITIONAL_ENABLED.key -> "false") {
spark.sql(sql).noop()
}
}
@@ -181,23 +247,33 @@ object ExpandBenchmark extends SqlBasedBenchmark {
}
override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
- val N = 10L << 20 // ~10M rows
+ val smoke = mainArgs.contains("smoke")
+ val sectionFilter = mainArgs.filterNot(_ == "smoke").toSet
+
+ def runSection(key: String, name: String)(body: => Unit): Unit = {
+ if (sectionFilter.isEmpty || sectionFilter.contains(key)) {
+ runBenchmark(name)(body)
+ }
+ }
+
+ val numIters = if (smoke) 1 else 5
+ val N = if (smoke) 1000L else 10L << 20 // ~10M rows
- runBenchmark("Expand: varying number of COUNT(DISTINCT)") {
+ runSection("ratio", "Expand: varying number of COUNT(DISTINCT)") {
prepareTable("expand_bench", N, keyMod = 1000,
colMods = Seq(100, 200, 300, 400, 500, 600, 700, 800))
expandRatioBenchmark(
- "2 distinct aggregates", N, 2, "expand_bench")
+ "2 distinct aggregates", N, 2, "expand_bench", numIters)
expandRatioBenchmark(
- "4 distinct aggregates", N, 4, "expand_bench")
+ "4 distinct aggregates", N, 4, "expand_bench", numIters)
expandRatioBenchmark(
- "6 distinct aggregates", N, 6, "expand_bench")
+ "6 distinct aggregates", N, 6, "expand_bench", numIters)
expandRatioBenchmark(
- "8 distinct aggregates", N, 8, "expand_bench")
+ "8 distinct aggregates", N, 8, "expand_bench", numIters)
}
- runBenchmark("Expand: varying data characteristics (pure distinct)") {
+ runSection("char", "Expand: varying data characteristics (pure distinct)")
{
// Default: 1K groups, moderate distinct cardinality
prepareTable("expand_default", N, keyMod = 1000,
colMods = Seq(100, 200, 300, 400, 500, 600))
@@ -223,49 +299,49 @@ object ExpandBenchmark extends SqlBasedBenchmark {
"6 pure distinct aggs with varying data", N, output = output)
benchDataChar.addCase("1K groups, moderate card - baseline",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "-1") {
spark.sql(sql6d).noop()
}
}
benchDataChar.addCase("1K groups, moderate card - optimized",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "2") {
spark.sql(sql6d).noop()
}
}
benchDataChar.addCase("100K groups, moderate card - baseline",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "-1") {
spark.sql(sql6dHighKey).noop()
}
}
benchDataChar.addCase("100K groups, moderate card - optimized",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "2") {
spark.sql(sql6dHighKey).noop()
}
}
benchDataChar.addCase("1K groups, low card (5 vals) - baseline",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "-1") {
spark.sql(sql6dLowCard).noop()
}
}
benchDataChar.addCase("1K groups, low card (5 vals) - optimized",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "2") {
spark.sql(sql6dLowCard).noop()
}
}
benchDataChar.addCase("no grouping key - baseline",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "-1") {
spark.sql(sql6dGlobal).noop()
}
}
benchDataChar.addCase("no grouping key - optimized",
- numIters = 5) { _ =>
+ numIters = numIters) { _ =>
withSQLConf(OPTIMIZE_EXPAND_RATIO -> "2") {
spark.sql(sql6dGlobal).noop()
}
@@ -274,12 +350,12 @@ object ExpandBenchmark extends SqlBasedBenchmark {
benchDataChar.run()
}
- runBenchmark("Expand: subexpression elimination across branches") {
- val numRows = 5L << 20 // ~5M rows
+ runSection("subexpr", "Expand: subexpression elimination across branches")
{
+ val numRows = if (smoke) 1000L else 5L << 20 // ~5M rows
prepareTrafficTable("expand_traffic", numRows, userMod = 1000000)
subExprEliminationBenchmark(
"9 conditional COUNT(DISTINCT) + 9 conditional SUM sharing one
subexpression",
- numRows, "expand_traffic")
+ numRows, "expand_traffic", numIters = if (smoke) 1 else 3)
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
