[GitHub] spark pull request #22488: [SPARK-25479][TEST] Refactor DatasetBenchmark to ...

2018-10-04 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/22488


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22488: [SPARK-25479][TEST] Refactor DatasetBenchmark to ...

2018-10-01 Thread wangyum 
Github user wangyum commented on a diff in the pull request:

https://github.com/apache/spark/pull/22488#discussion_r221780844
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala ---
@@ -242,75 +249,22 @@ object DatasetBenchmark {
 benchmark
   }
 
-  def main(args: Array[String]): Unit = {
-val spark = SparkSession.builder
+  override def getSparkSession: SparkSession = {
--- End diff --

Need override default `SparkSession` as default `SparkSession` is:
```scala
SparkSession.builder()
  .master("local[1]")
  .appName(this.getClass.getCanonicalName)
  .config(SQLConf.SHUFFLE_PARTITIONS.key, 1)
  .config(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key, 1)
  .getOrCreate()
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22488: [SPARK-25479][TEST] Refactor DatasetBenchmark to ...

2018-09-23 Thread dongjoon-hyun 
Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/22488#discussion_r219729655
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala ---
@@ -242,75 +248,20 @@ object DatasetBenchmark {
 benchmark
   }
 
-  def main(args: Array[String]): Unit = {
-val spark = SparkSession.builder
-  .master("local[*]")
-  .appName("Dataset benchmark")
-  .getOrCreate()
+  val spark = SparkSession.builder
+.master("local[*]")
+.appName("Dataset benchmark")
+.getOrCreate()
--- End diff --

Can we move this SparkSession building part into `benchmark()` function and 
before `runBenchmark("Dataset Benchmark")`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22488: [SPARK-25479][TEST] Refactor DatasetBenchmark to ...

2018-09-20 Thread wangyum 
GitHub user wangyum opened a pull request:

https://github.com/apache/spark/pull/22488

[SPARK-25479][TEST] Refactor DatasetBenchmark to use main method

## What changes were proposed in this pull request?

Refactor `DatasetBenchmark` to use main method.
Generate benchmark result:
```sh
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain 
org.apache.spark.sql.DatasetBenchmark"
```

## How was this patch tested?

manual tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangyum/spark SPARK-25479

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22488


commit 21b623aad6a84cca2ab5f89f1c29d3b3b1b82d80
Author: Yuming Wang 
Date:   2018-09-20T09:46:19Z

Refactor DatasetBenchmark




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org