spark git commit: [SPARK-9557][SQL] Refactor ParquetFilterSuite and remove old ParquetFilters code
Repository: spark
Updated Branches:
refs/heads/branch-1.6 b9adfdf9c -> c42433d02
[SPARK-9557][SQL] Refactor ParquetFilterSuite and remove old ParquetFilters code
Actually this was resolved by https://github.com/apache/spark/pull/8275.
But I found the JIRA issue for this is not marked as resolved since the PR
above was made for another issue but the PR above resolved both.
I commented that this is resolved by the PR above; however, I opened this PR as
I would like to just add
a little bit of corrections.
In the previous PR, I refactored the test by not reducing just collecting
filters; however, this would not test properly `And` filter (which is not
given to the tests). I unintentionally changed this from the original way
(before being refactored).
In this PR, I just followed the original way to collect filters by reducing.
I would like to close this if this PR is inappropriate and somebody would like
this deal with it in the separate PR related with this.
Author: hyukjinkwon
Closes #9554 from HyukjinKwon/SPARK-9557.
(cherry picked from commit 9565c246eadecf4836d247d0067f2200f061d25f)
Signed-off-by: Michael Armbrust
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c42433d0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c42433d0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c42433d0
Branch: refs/heads/branch-1.6
Commit: c42433d0272d43217c0247b03bc6684df1eabfec
Parents: b9adfdf
Author: hyukjinkwon
Authored: Mon Nov 9 15:20:50 2015 -0800
Committer: Michael Armbrust
Committed: Mon Nov 9 15:21:06 2015 -0800
--
.../execution/datasources/parquet/ParquetFilterSuite.scala | 8
1 file changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/c42433d0/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
--
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
index c24c9f0..579dabf 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
@@ -54,12 +54,12 @@ class ParquetFilterSuite extends QueryTest with ParquetTest
with SharedSQLContex
.select(output.map(e => Column(e)): _*)
.where(Column(predicate))
- val analyzedPredicate = query.queryExecution.optimizedPlan.collect {
+ val maybeAnalyzedPredicate = query.queryExecution.optimizedPlan.collect {
case PhysicalOperation(_, filters, LogicalRelation(_: ParquetRelation,
_)) => filters
- }.flatten
- assert(analyzedPredicate.nonEmpty)
+ }.flatten.reduceLeftOption(_ && _)
+ assert(maybeAnalyzedPredicate.isDefined)
- val selectedFilters =
analyzedPredicate.flatMap(DataSourceStrategy.translateFilter)
+ val selectedFilters =
maybeAnalyzedPredicate.flatMap(DataSourceStrategy.translateFilter)
assert(selectedFilters.nonEmpty)
selectedFilters.foreach { pred =>
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-9557][SQL] Refactor ParquetFilterSuite and remove old ParquetFilters code
Repository: spark
Updated Branches:
refs/heads/master fcb57e9c7 -> 9565c246e
[SPARK-9557][SQL] Refactor ParquetFilterSuite and remove old ParquetFilters code
Actually this was resolved by https://github.com/apache/spark/pull/8275.
But I found the JIRA issue for this is not marked as resolved since the PR
above was made for another issue but the PR above resolved both.
I commented that this is resolved by the PR above; however, I opened this PR as
I would like to just add
a little bit of corrections.
In the previous PR, I refactored the test by not reducing just collecting
filters; however, this would not test properly `And` filter (which is not
given to the tests). I unintentionally changed this from the original way
(before being refactored).
In this PR, I just followed the original way to collect filters by reducing.
I would like to close this if this PR is inappropriate and somebody would like
this deal with it in the separate PR related with this.
Author: hyukjinkwon
Closes #9554 from HyukjinKwon/SPARK-9557.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9565c246
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9565c246
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9565c246
Branch: refs/heads/master
Commit: 9565c246eadecf4836d247d0067f2200f061d25f
Parents: fcb57e9
Author: hyukjinkwon
Authored: Mon Nov 9 15:20:50 2015 -0800
Committer: Michael Armbrust
Committed: Mon Nov 9 15:20:50 2015 -0800
--
.../execution/datasources/parquet/ParquetFilterSuite.scala | 8
1 file changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/9565c246/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
--
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
index c24c9f0..579dabf 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
@@ -54,12 +54,12 @@ class ParquetFilterSuite extends QueryTest with ParquetTest
with SharedSQLContex
.select(output.map(e => Column(e)): _*)
.where(Column(predicate))
- val analyzedPredicate = query.queryExecution.optimizedPlan.collect {
+ val maybeAnalyzedPredicate = query.queryExecution.optimizedPlan.collect {
case PhysicalOperation(_, filters, LogicalRelation(_: ParquetRelation,
_)) => filters
- }.flatten
- assert(analyzedPredicate.nonEmpty)
+ }.flatten.reduceLeftOption(_ && _)
+ assert(maybeAnalyzedPredicate.isDefined)
- val selectedFilters =
analyzedPredicate.flatMap(DataSourceStrategy.translateFilter)
+ val selectedFilters =
maybeAnalyzedPredicate.flatMap(DataSourceStrategy.translateFilter)
assert(selectedFilters.nonEmpty)
selectedFilters.foreach { pred =>
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
