Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153290147
Overall LGTM. Once we update the `FilteredScanSuite`, we are good to go.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153382900
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153382866
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153382067
**[Test build #44927 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44927/consoleFull)**
for PR 9399 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153395800
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153397117
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153397162
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153386962
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153396956
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153386964
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153395771
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153395899
**[Test build #44933 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44933/consoleFull)**
for PR 9399 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153424036
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153424038
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153434867
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153439292
Thanks! Merging!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153416340
**[Test build #44928 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44928/consoleFull)**
for PR 9399 at commit
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153440296
Let's also have some test cases that having a column that is used in
handled filters as well as in unhandled/unconvertible filters.
---
If your project is set up for it,
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9399
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153416822
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153431665
I will merge it once it passes jenkins. Let's have a test to make sure
those handled filters will not show up in the Filter operator.
---
If your project is set up for
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153434863
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153442775
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153423795
**[Test build #44927 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44927/consoleFull)**
for PR 9399 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153442505
**[Test build #44934 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44934/consoleFull)**
for PR 9399 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153442781
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153434574
**[Test build #44933 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44933/consoleFull)**
for PR 9399 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153416824
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43755700
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -266,47 +267,75 @@ private[sql] object
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43755741
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala ---
@@ -202,51 +232,60 @@ class FilteredScanSuite extends
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43755716
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -409,7 +439,48 @@ private[sql] object
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153377634
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153377650
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153379346
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153379397
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153379820
**[Test build #44928 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44928/consoleFull)**
for PR 9399 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153398208
**[Test build #44934 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44934/consoleFull)**
for PR 9399 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153012486
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153050846
**[Test build #44814 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44814/consoleFull)**
for PR 9399 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153012488
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153012481
**[Test build #44811 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44811/consoleFull)**
for PR 9399 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153048329
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153048363
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153011716
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153011694
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153012036
**[Test build #44811 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44811/consoleFull)**
for PR 9399 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153095366
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153095363
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153095199
**[Test build #44814 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44814/consoleFull)**
for PR 9399 at commit
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43719128
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -409,7 +439,48 @@ private[sql] object
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43719436
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala ---
@@ -202,51 +232,60 @@ class FilteredScanSuite extends DataSourceTest
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43716445
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -266,47 +267,75 @@ private[sql] object
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153203737
`unhandledFilter` will not see filters using partitioning columns.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153204617
Ok, thanks for explanation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-153202825
Ok, actually I was planning to optimize the expression with partition key,
which will introduce the `ConstantFolding`, as the partition key will be a
constant
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152865071
**[Test build #44768 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44768/consoleFull)**
for PR 9399 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152872738
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152872862
**[Test build #44776 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44776/consoleFull)**
for PR 9399 at commit
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43590253
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -266,26 +267,39 @@ private[sql] object
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43590460
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -295,18 +309,20 @@ private[sql] object
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43590582
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala ---
@@ -44,16 +44,46 @@ case class SimpleFilteredScan(from: Int, to:
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43590585
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/sources/ParquetHadoopFsRelationSuite.scala
---
@@ -145,14 +145,16 @@ class ParquetHadoopFsRelationSuite
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43590547
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala ---
@@ -101,6 +130,10 @@ object FiltersPushed {
var list:
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152892964
One more consideration for this improvement, as we probably need to
optimize the filters by folding the expression, as the partition keys are
actually are the
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152884826
**[Test build #44776 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44776/consoleFull)**
for PR 9399 at commit
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43594749
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/sources/ParquetHadoopFsRelationSuite.scala
---
@@ -145,14 +145,16 @@ class
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152884914
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152884911
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43594470
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -295,18 +309,20 @@ private[sql] object
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152893179
@chenghao-intel Can you give an example showing `unhandledFilters` is
insufficient? Also, regarding "So I am wondering if we can leave the
unhandledFilters and
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152910627
Actually I am talking that it probably give us some troubles in getting the
`unhandledFilters` if we planned to optimize the cases where partition keys
combined
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152865193
**[Test build #44768 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44768/consoleFull)**
for PR 9399 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152865196
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152865194
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152872733
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43590334
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -295,18 +309,20 @@ private[sql] object
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/9399
[SPARK-10978] [SQL] Allow data sources to eliminate filters
This PR adds a new method `unhandledFilters` to `BaseRelation`. Data
sources which implement this method properly may avoid the
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152864572
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152864583
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43590263
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -266,26 +267,39 @@ private[sql] object
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43590278
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -295,18 +309,20 @@ private[sql] object
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43590281
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -295,18 +309,20 @@ private[sql] object
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9399#discussion_r43590284
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -266,26 +267,39 @@ private[sql] object
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152931881
Sorry, I am challenge this as it's about the API, which probably difficult
to change back once it's released, and we'd better think further, by adding the
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9399#issuecomment-152930901
Oh, for example: let's say we have the table src (key, value) partition (p1)
For the query like "SELECT value FROM src WHERE key > p1",
And we assume
85 matches
Mail list logo
