Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/13893
Ping, @jiangxb1987 .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: r
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/13893
@heary-cao tried to resolve the same issue in
https://github.com/apache/spark/pull/18969
ping @jiangxb1987
---
-
T
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/13893
ya, this still exists. Let me find some time to resolve this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13893
@jiangxb1987 do we still have this bug?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user sameeragarwal commented on the issue:
https://github.com/apache/spark/pull/13893
ping @jiangxb1987 @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/13893
@cloud-fan I've send a PR to add `Scanner` operator in #14619 , please have
a look at it when you have time, thanks!
---
If your project is set up for it, you can reply to this email and have y
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/13893
@cloud-fan Do you mean something like adding in `basicLogicalOperators` the
following:
`case class Scanner(
projectionList: Seq[NamedExpression],
filters: Seq[Expression],
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/13893
ping @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13893
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62596/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13893
**[Test build #62596 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62596/consoleFull)**
for PR 13893 at commit
[`db28228`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13893
**[Test build #62596 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62596/consoleFull)**
for PR 13893 at commit
[`db28228`](https://github.com/apache/spark/commit/d
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13893
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/13893
With [PR#14012](https://github.com/apache/spark/pull/14012) the order
between deterministic and non-deterministic predicates would not be changed
arbitrarily, so I think we could apply this impr
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/13893
@jiangxb1987 Please feel free to create a new JIRA ticket and PR for this,
thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/13893
If PushDownPredicate should be improved, I would like to send a PR in one
or two days. Is a new task need to be created?@cloud-fan
---
If your project is set up for it, you can reply to this em
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13893
It's a good point, looks like we can also improve the `PushDownPredicate`
rule according to this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on G
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/13893
Predicates should not be reordered if a condition contains
non-deterministic parts, for example, 'rand() < 0.1 AND a=1' should not be
reordered to 'a=1 AND rand() < 0.1' as the number of calls
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13893
no, the predicates order doesn't matter. Our optimizer can reorder the
predicates to run them more efficient.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/13893
@cloud-fan I pushed a commit to apply predicate pushdown on deterministic
parts placed before any non-deterministic predicates, should it be safe to do
this optimizationï¼
---
If your project
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/13893
@AlekseiS , I think we should always consider the risk and take care of it
even it's not a common case. We can't assume what users expect, and prepare for
the worst case.
---
If your project is
Github user AlekseiS commented on the issue:
https://github.com/apache/spark/pull/13893
@liancheng I think partition predicates are a bit different. If you
explicitly specify a partition predicate, like "date=2016-06-27", do you really
expect other partitions being scanned regardless
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/13893
@cloud-fan could you please have a look at this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13893
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
24 matches
Mail list logo
