Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49188774
You are right that this rule does more than null propagation now. I'm not
sure what a better name would be. `DegenerateExpressionSimplification`?
Regarding
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49205341
It doesn't bring much benefit right now, but what we are doing here is
creating patterns in NullPropagation to specify the semantics of each
individual expression ... not
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49210382
I'm not sure I agree with that. This is a pretty niche optimization not
something fundamental about the expressions that is required for correct
evaluation (and the
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49211681
That is exactly the argument I made when the folding logic was added. :) I
suggested that we add `deterministic` instead and then have a rule that folds
things that are
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49211732
I would be supportive of changing it to match my original proposal...
---
If your project is set up for it, you can reply to this email and have your
reply appear on
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/1428
[SPARK-2509][SQL] Add optimization for Substring.
`Substring` including `null` literal cases could be added to
`NullPropagation`.
You can merge this pull request into a Git repository by running:
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49120808
QA tests have started for PR 1428. This patch merges cleanly. brView
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16705/consoleFull
---
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1428#discussion_r14980614
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -171,6 +171,9 @@ object NullPropagation extends
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1428#discussion_r14980864
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -171,6 +171,9 @@ object NullPropagation extends
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49125284
QA results for PR 1428:br- This patch PASSES unit tests.br- This patch
merges cleanlybr- This patch adds no public classesbrbrFor more
information see test
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49125317
Thanks. Merging this in master branch-1.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1428#discussion_r14981973
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -171,6 +171,9 @@ object NullPropagation extends
Github user ueshin commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49125413
@rxin Ah, wait for a moment.
What do you think about foldability of the `Substring` I mentioned above?
---
If your project is set up for it, you can reply to this
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1428
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user ueshin commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49125499
I'll open another issue about foldability of `Substring`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49125623
I've already merged this. I think we should re-evaluate in the future
whether we should push some stuff into foldable itself, because as I see it a
lot of the pattern
Github user ueshin commented on the pull request:
https://github.com/apache/spark/pull/1428#issuecomment-49125889
@rxin I agree that we need the better way for `NullPropagation ` instead of
too much pattern matching.
---
If your project is set up for it, you can reply to this email
17 matches
Mail list logo
