GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/11926
[SPARK-14111][SQL] Correct output nullability with constraints for logical
plans
## What changes were proposed in this pull request?
JIRA: https://issues.apache.org/jira/browse/SPARK-14111
We use output of logical plan as schema. Output nullablity is important and
we should keep its correctness.
With constraints and optimization, in fact we will change output
nullability of logical plans. But we don't reflect such changes in the output
attributes. So the output nullablity is not correct now.
## How was this patch tested?
Modified `InferFiltersFromConstraintsSuite` and `AnalysisSuite`. Existing
tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1
output-nullable-with-constraint3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11926.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11926
----
commit fb6c8cd182dae1aded07baf59cb185f1afde84e7
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-15T09:01:04Z
Modify output nullable with constraint for Join.
commit 2e4eca4c8336da790321569709475eaff8f193b5
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-15T09:51:11Z
Replace attributes in condition with correct ones.
commit aef73d5e71b9c997ac98a7172036c7c79f9e9b1c
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-16T04:32:41Z
Refactor.
commit 5bf4b4b544ef2aa25d93c974e94f8314a6626ef7
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-16T04:47:27Z
Refactor.
commit 93a73b79fbeee9c1bd722f5aa66257409d3c2512
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-16T04:59:28Z
Modify output nullability with constraints for Filter operator.
commit d85e8381f9b61d68288f8009ffa35b2cfd1ed771
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-16T07:12:43Z
Merge remote-tracking branch 'upstream/master' into
output-nullable-with-constraint
commit c7d54a0fb78c826903c0db8f1b1ac7b0d54bb303
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-16T07:30:08Z
Fix a bug.
commit 76a8566e9506e6a41107c8cb244a76d4525b7a44
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-16T10:04:32Z
Fix test.
commit bbced69407e1d400fb66b684c9c6529b96f15d88
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-17T03:07:05Z
Merge remote-tracking branch 'upstream/master' into
output-nullable-with-constraint
Conflicts:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/InferFiltersFromConstraintsSuite.scala
commit 6b4a98cdd44a447e6ab8c3aee908647243e62449
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-17T07:54:05Z
fix bug.
commit 665bf50880b3f87353c5037946db28f8dda7d1c2
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-17T11:41:35Z
Fix scala style.
commit ac795610e2091a1534b80e7eea01630c4cb5deb4
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-18T07:45:12Z
Fix it.
commit 7f68967eeb3f303c552dadc760788d3fe9d090f5
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-19T04:33:06Z
Modify attribute nullability for filter pushdown.
commit a7b8daef9e82e184226f101a5fd81fcc070dc25c
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-21T04:32:25Z
Reset nullabilty for project and filter list in preparing scaning in memory
relation.
commit 23b328d1c01806841943ad8dd0ab3eed8963d7e2
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-21T05:02:21Z
Unnecessary change removed.
commit da3f35b4d315cc3c2576ac781cd3e8beef5eb774
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-21T08:16:21Z
Fix python test.
commit cdc5878e4ccb98f87b4496b85a2ef95e1722a1f6
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-24T03:09:45Z
Correct output nullability of logical plans.
commit 5912fd3794e0f62817a4c44025de09bfcec4c944
Author: Liang-Chi Hsieh <[email protected]>
Date: 2016-03-24T03:31:16Z
Merge remote-tracking branch 'upstream/master' into
output-nullable-with-constraint3
Conflicts:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
