Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155537739
I think I can resolve the conflicts manually. Merging to master and 1.6.
Thanks!
---
If your project is set up for it, you can reply to this email and have
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9480
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155258413
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155259441
**[Test build #45468 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45468/consoleFull)**
for PR 9480 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155143818
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155145415
**[Test build #45384 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45384/consoleFull)**
for PR 9480 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155143767
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155185089
**[Test build #45384 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45384/consoleFull)**
for PR 9480 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155185293
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155259218
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155259185
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155236703
**[Test build #45443 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45443/consoleFull)**
for PR 9480 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155236534
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-155236517
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154570059
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44201615
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression extends
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44201576
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression extends
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44202414
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -135,6 +147,7 @@ abstract class Expression extends
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44205755
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44200916
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44202061
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala
---
@@ -275,8 +275,9 @@ object
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44201792
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
---
@@ -0,0 +1,108 @@
+/*
+ * Licensed
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154586827
**[Test build #45263 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45263/consoleFull)**
for PR 9480 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154586930
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154567189
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154590257
**[Test build #1999 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1999/consoleFull)**
for PR 9480 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154570265
**[Test build #1999 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1999/consoleFull)**
for PR 9480 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154567121
**[Test build #45259 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45259/consoleFull)**
for PR 9480 at commit
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154590883
I don't think the most recent failure is your fault.
@rxin, did you patch break the build?
```
File
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154599464
>
I'm still not sure how you are suggesting we execute the version you are
proposing. I'd guess you do that by adding an additional project that creates a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154562314
**[Test build #45263 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45263/consoleFull)**
for PR 9480 at commit
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44197514
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154585715
This is awesome. LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44177613
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala
---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44178151
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -341,7 +440,18 @@ abstract class
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44177472
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class
Github user nongli commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154518082
> In general, I think we'd better add more IR (Intermediate Representation,
should be the Expression or Logical Plan node here) for common sub expression
cases, and we
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154518695
This looks pretty good to me, just a few minor comments. Should we have a
benchmark to see is there any regression for tiny common subexpression? If not
much, then we
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44190675
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44190096
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -341,7 +440,18 @@ abstract class
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154551545
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154551510
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44191412
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class
Github user nongli commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154553190
I ran this benchmark:
```
val data = sqlContext.range(20 * 1024 *
1024).toDF("v").registerTempTable("t")
for (i <- 0 until 4) {
val t1
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154554001
**[Test build #45259 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45259/consoleFull)**
for PR 9480 at commit
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154554772
Thanks for the numbers, that's cool.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44194583
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -203,6 +203,10 @@ case class
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154561748
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154561767
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44043807
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression extends
Github user nongli commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154137436
@chenghao-intel I'm not sure what you are suggesting. How do you suggest we
execute the expressions? The transformation you have as an example makes sense
but how does
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154288464
I took one pass. Overall pretty good! @davies Can you also take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44106783
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-154316345
In general, I think we'd better add more IR (Intermediate Representation,
should be the Expression or Logical Plan node here) for common sub expression
cases,
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44108481
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44105110
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression extends
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r44099965
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression
GitHub user nongli opened a pull request:
https://github.com/apache/spark/pull/9480
[SPARK-10371] [SQL] Implement subexpr elimination for UnsafeProjections
This patch adds the building blocks for codegening subexpr elimination and
implements
it end to end for UnsafeProjection.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/9480#discussion_r43966655
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
---
@@ -151,6 +164,30 @@ abstract class Expression
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153924543
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153924518
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153925305
**[Test build #45075 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45075/consoleFull)**
for PR 9480 at commit
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153926546
My 2 cents, Common Expression Elimination is a very interesting/useful
optimization, and I did the same thing for Shark quite long time ago, for my
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153948155
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9480#issuecomment-153948080
**[Test build #45075 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45075/consoleFull)**
for PR 9480 at commit
65 matches
Mail list logo
