GitHub user eyalfa opened a pull request:
https://github.com/apache/spark/pull/16043
[SPARK-18601][SQL] Simplify Create/Get complex expression pairs in optimizer
## What changes were proposed in this pull request?
It often happens that a complex object (struct/map/array) is created only
to get elements from it in an subsequent expression. We can add an optimizer
rule for this.
## How was this patch tested?
unit-tests
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/eyalfa/spark SPARK-18601
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16043.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16043
----
commit b49a644dd4fb77251d1349366e38c51f3e1bd237
Author: eyal farago <[email protected]>
Date: 2016-11-22T15:59:20Z
getOnCreateStruct: added tests to show the problem, and an initial
implementation.
commit 27afcaf1fffe3cd2927cacaa1c9a6168d26ab4fe
Author: eyal farago <[email protected]>
Date: 2016-11-22T19:09:18Z
getOnCreateStruct: move the new ruleinto an existing batch.
commit 7dd168f41c10ebf79b5ed56e53b697e9250eb633
Author: eyal farago <[email protected]>
Date: 2016-11-26T16:21:37Z
getOnCreateStruct: added opt rules for createArray and createMap, found a
bug in arr.getField.
commit 6d42111d4f094d35634883dea2fac2f3b4eb50fa
Author: eyal farago <[email protected]>
Date: 2016-11-28T20:18:26Z
SPARK-18601: removed the CreateNamedStructLike extractor.
commit 0f6823907ec862e479f4cacdcd1869ffa8ce7858
Author: eyal farago <[email protected]>
Date: 2016-11-28T20:19:51Z
SPARK-18601: test suite for spark-18601
commit 07a35ea11a635a87c7e32f5fda8f60c0815d9479
Author: eyal farago <[email protected]>
Date: 2016-11-28T20:21:00Z
SPARK-18601: optimizer rules for simplifying operations on complex type
constructors.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
