Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220855166
All issues are resolved now. I'll update this PR now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220855050
Thank you for sharing this, @linbojin .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user linbojin commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220837854
@dongjoon-hyun @cloud-fan @marmbrus Thanks for your discussions about my
reported issue: [SPARK-15282](https://issues.apache.org/jira/browse/SPARK-15282)
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220796055
Thank you. @cloud-fan .
By the way, to be clear with this, should we revert the change on
`PushDownPredicate`, too?
I think it's another separate issue.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220784640
I like the idea to document `ScalaUDF` and say that the scala function must
be deterministic. If users don't follow, they are on their own responsibility.
---
If
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220721052
For the documentation, sure, I will. We had better declare UDF should be
deterministic as @thunterdb and you said from now.
For the runtime, I don't have
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220700839
My main questions remains: does this actually make a difference in runtime?
or is execution smart enough already to do this optimization (even if to the
user it
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220685543
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220685544
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220685159
**[Test build #59009 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59009/consoleFull)**
for PR 13087 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220667139
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220667137
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220666864
**[Test build #59006 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59006/consoleFull)**
for PR 13087 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220661982
**[Test build #59009 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59009/consoleFull)**
for PR 13087 at commit
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220662788
@marmbrus @markhamstra @thunterdb .
Now, this code and description of this PR is rollbacked my second commit 7
days ago.
For `common subexpression
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/13087#discussion_r64002341
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1025,7 +1025,8 @@ object PushDownPredicate
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/13087#discussion_r64000518
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1025,7 +1025,8 @@ object PushDownPredicate extends
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/13087#discussion_r63997574
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1025,7 +1025,8 @@ object PushDownPredicate
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/13087#discussion_r63992511
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1025,7 +1025,8 @@ object PushDownPredicate extends
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220503129
cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user linbojin commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220502400
Hi, @marmbrus
Could you review this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220416841
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220416839
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220416508
**[Test build #58874 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58874/consoleFull)**
for PR 13087 at commit
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220394299
Hi, @rxin .
Could you review this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-220390918
**[Test build #58874 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58874/consoleFull)**
for PR 13087 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-219502671
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-219502672
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-219502377
**[Test build #58647 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58647/consoleFull)**
for PR 13087 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-219479059
**[Test build #58647 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58647/consoleFull)**
for PR 13087 at commit
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-218949073
Hi, @liancheng and @cloud-fan .
This PR is similar with your commit, `[SPARK-13473][SQL] Don't push
predicate through project with nondeterministic field(s)`.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-218948867
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-218948865
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-218948774
**[Test build #58535 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58535/consoleFull)**
for PR 13087 at commit
Github user dongjoon-hyun commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-218948335
The reported error scenario is the following.
```scala
scala> val df = sc.parallelize(Seq(("a", "b"), ("a1",
"b1"))).toDF("old","old1")
scala> val
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13087#issuecomment-218940532
**[Test build #58535 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58535/consoleFull)**
for PR 13087 at commit
36 matches
Mail list logo
