GitHub user wangyum opened a pull request:
https://github.com/apache/spark/pull/22522
[SPARK-25510][TEST] Create new trait replace BenchmarkWithCodegen
## What changes were proposed in this pull request?
We need create a new trait to replace `BenchmarkWithCodegen` as
`BenchmarkWithCodegen` extends from `SparkFunSuite`.
For example. when doing `AggregateBenchmark` refactor.
Before this change, it should be:
```scala
object AggregateBenchmark extends BenchmarkBase {
lazy val sparkSession = SparkSession.builder
.master("local[1]")
.appName(this.getClass.getSimpleName)
.config("spark.sql.shuffle.partitions", 1)
.config("spark.sql.autoBroadcastJoinThreshold", 1)
.getOrCreate()
/** Runs function `f` with whole stage codegen on and off. */
def runBenchmark(name: String, cardinality: Long)(f: => Unit): Unit = {
val benchmark = new Benchmark(name, cardinality, output = output)
benchmark.addCase(s"$name wholestage off", numIters = 2) { iter =>
sparkSession.conf.set("spark.sql.codegen.wholeStage", value = false)
f
}
benchmark.addCase(s"$name wholestage on", numIters = 5) { iter =>
sparkSession.conf.set("spark.sql.codegen.wholeStage", value = true)
f
}
benchmark.run()
}
override def benchmark(): Unit = {
runBenchmark("aggregate without grouping") {
val N = 500L << 22
runBenchmark("agg w/o group", N) {
sparkSession.range(N).selectExpr("sum(id)").collect()
}
}
...
```
After this change:
```scala
object AggregateBenchmark extends BenchmarkBase with
RunBenchmarkWithCodegen {
override def benchmark(): Unit = {
runBenchmark("aggregate without grouping") {
val N = 500L << 22
runBenchmark("agg w/o group", N) {
sparkSession.range(N).selectExpr("sum(id)").collect()
}
}
...
```
All affect benchmarks:
```
AggregateBenchmark
BenchmarkWideTable
JoinBenchmark
MiscBenchmark
ObjectHashAggregateExecBenchmark
SortBenchmark
UnsafeArrayDataBenchmark
```
## How was this patch tested?
manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangyum/spark SPARK-25510
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22522.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22522
----
commit 275cc6c5f8f106eb339c7ed01734e279a223705e
Author: Yuming Wang <yumwang@...>
Date: 2018-09-21T17:36:57Z
Create new BenchmarkWithCodegen trait doesn't extends SparkFunSuite
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
