[GitHub] spark pull request #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Ben...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22513 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Ben...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22513#discussion_r219414758 --- Diff: core/src/main/scala/org/apache/spark/sql/execution/benchmark/BenchmarkBase.scala --- @@ -15,7 +15,7 @@ * limitations under the License. */ -package org.apache.spark.util +package org.apache.spark.sql.execution.benchmark --- End diff -- ditto --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Ben...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22513#discussion_r219414641 --- Diff: core/src/main/scala/org/apache/spark/sql/execution/benchmark/Benchmark.scala --- @@ -15,7 +15,7 @@ * limitations under the License. */ -package org.apache.spark.util +package org.apache.spark.sql.execution.benchmark --- End diff -- this class is in core and we should not have `sql` in the package name --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Ben...
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22513#discussion_r219397646
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala
---
@@ -27,7 +27,7 @@ import
org.apache.spark.sql.functions.monotonically_increasing_id
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.ParquetOutputTimestampType
import org.apache.spark.sql.types.{ByteType, Decimal, DecimalType,
TimestampType}
-import org.apache.spark.util.{Benchmark, BenchmarkBase =>
FileBenchmarkBase, Utils}
+import org.apache.spark.util.Utils
/**
* Benchmark to measure read performance with Filter pushdown.
--- End diff --
Thanks, I have updated the doc ð
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Ben...
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/22513#discussion_r219388085
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala
---
@@ -27,7 +27,7 @@ import
org.apache.spark.sql.functions.monotonically_increasing_id
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.ParquetOutputTimestampType
import org.apache.spark.sql.types.{ByteType, Decimal, DecimalType,
TimestampType}
-import org.apache.spark.util.{Benchmark, BenchmarkBase =>
FileBenchmarkBase, Utils}
+import org.apache.spark.util.Utils
/**
* Benchmark to measure read performance with Filter pushdown.
--- End diff --
How about change scala doc to below to fix **fails to generate
documentation**?
```scala
* To run this benchmark:
* {{{
* 1. without sbt: bin/spark-submit --class
* 2. build/sbt "sql/test:runMain "
* 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt
"sql/test:runMain "
* Results will be written to
"benchmarks/FilterPushdownBenchmark-results.txt".
* }}}
```
fails to generate documentation error message:
```java
/home/jenkins/workspace/SparkPullRequestBuilder@2/target/javaunidoc/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.html...
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/mllib/target/java/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.java:5:
error: unknown tag: this
[error] * 1. without sbt: bin/spark-submit --class
[error] ^
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/mllib/target/java/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.java:5:
error: unknown tag: spark
[error] * 1. without sbt: bin/spark-submit --class
[error] ^
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/mllib/target/java/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.java:6:
error: unknown tag: this
[error] * 2. build/sbt "mllib/test:runMain "
[error] ^
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/mllib/target/java/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.java:7:
error: unknown tag: this
[error] * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt
"mllib/test:runMain "
[error]
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Ben...
GitHub user gengliangwang opened a pull request: https://github.com/apache/spark/pull/22513 [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark ## What changes were proposed in this pull request? Currently there are two classes with the same naming BenchmarkBase: 1. `org.apache.spark.util.BenchmarkBase` 2. `org.apache.spark.sql.execution.benchmark.BenchmarkBase` This is very confusing. And the benchmark object `org.apache.spark.sql.execution.benchmark.FilterPushdownBenchmark` is using the one in `org.apache.spark.util.BenchmarkBase`, while there is another class `BenchmarkBase` in the same package of it. Here I propose: 1. the package org.apache.spark.util.BenchmarkBase should be in test package, move to org.apache.spark.sql.execution.benchmark . 2. Rename the org.apache.spark.sql.execution.benchmark.BenchmarkBase as BenchmarkWithCodegen 3. Move org.apache.spark.util.Benchmark to test package of org.apache.spark.sql.execution.benchmark ## How was this patch tested? Unit test You can merge this pull request into a Git repository by running: $ git pull https://github.com/gengliangwang/spark refactorBenchmarkBase Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22513.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22513 commit 89bd8300405a6c7f2ed4d756db66b2d1cc3f7389 Author: Gengliang Wang Date: 2018-09-21T05:07:01Z refactor BenchmarkBase --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
