[GitHub] spark pull request #22664: [SPARK-25662][TEST] Refactor DataSourceReadBenchm...

2018-10-10 Thread dongjoon-hyun 
Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/22664#discussion_r224272542
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
 ---
@@ -34,10 +34,15 @@ import org.apache.spark.sql.vectorized.ColumnVector
 
 /**
  * Benchmark to measure data source read performance.
- * To run this:
- *  spark-submit --class  
+ * To run this benchmark:
+ * {{{
+ *   1. without sbt: bin/spark-submit --class  --jars , 
--- End diff --

Could you run `dev/scalastyle` and fix this in your branch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22664: [SPARK-25662][TEST] Refactor DataSourceReadBenchm...

2018-10-08 Thread dongjoon-hyun 
Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/22664#discussion_r223476989
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
 ---
@@ -34,10 +34,15 @@ import org.apache.spark.sql.vectorized.ColumnVector
 
 /**
  * Benchmark to measure data source read performance.
- * To run this:
- *  spark-submit --class  
+ * To run this benchmark:
+ * {{{
+ *   1. without sbt: bin/spark-submit --class  
--- End diff --

Yes. We noticed that the required jars are introduced during refactoring. 
So, we start to fix this guide in recent PRs like 
[this](https://github.com/apache/spark/commit/669ade3a8eed0016b5ece57d776cea0616417088).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22664: [SPARK-25662][TEST] Refactor DataSourceReadBenchm...

2018-10-07 Thread peter-toth 
Github user peter-toth commented on a diff in the pull request:

https://github.com/apache/spark/pull/22664#discussion_r22366
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
 ---
@@ -34,10 +34,15 @@ import org.apache.spark.sql.vectorized.ColumnVector
 
 /**
  * Benchmark to measure data source read performance.
- * To run this:
- *  spark-submit --class  
+ * To run this benchmark:
+ * {{{
+ *   1. without sbt: bin/spark-submit --class  
--- End diff --

```
bin/spark-submit --class 
org.apache.spark.sql.execution.benchmark.DataSourceReadBenchmark --jars 
core/target/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar,sql/catalyst/target/spark-catalyst_2.11-3.0.0-SNAPSHOT-tests.jar
 sql/core/target/spark-sql_2.11-3.0.0-SNAPSHOT-tests.jar
```
does work for me, but I checked in `FilterPushdownBenchmark` and it seems 
we don't mention other required jars.
Shall I modify the command?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22664: [SPARK-25662][TEST] Refactor DataSourceReadBenchm...

2018-10-07 Thread dongjoon-hyun 
Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/22664#discussion_r223220320
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
 ---
@@ -34,10 +34,15 @@ import org.apache.spark.sql.vectorized.ColumnVector
 
 /**
  * Benchmark to measure data source read performance.
- * To run this:
- *  spark-submit --class  
+ * To run this benchmark:
+ * {{{
+ *   1. without sbt: bin/spark-submit --class  
--- End diff --

Hi, @peter-toth . Could you run this command actually?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22664: [SPARK-25662][TEST] Refactor DataSourceReadBenchm...

2018-10-07 Thread peter-toth 
GitHub user peter-toth opened a pull request:

https://github.com/apache/spark/pull/22664

[SPARK-25662][TEST] Refactor DataSourceReadBenchmark to use main method

## What changes were proposed in this pull request?

1. Refactor DataSourceReadBenchmark

## How was this patch tested?

Manually tested and regenerated results.
```
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain 
org.apache.spark.sql.execution.benchmark.DataSourceReadBenchmark"
```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peter-toth/spark SPARK-25662

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22664.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22664


commit 0f82794c67749c4dad65d5672a853d04096a9785
Author: Peter Toth 
Date:   2018-10-07T15:01:05Z

[SPARK-25662][TEST] Refactor DataSourceReadBenchmark to use main method

Change-Id: Icfd0484c8e0fef2ed0b184e09e52db9432e0a250




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org