GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/11563
[SPARK-12719][SQL] SQL generation support for generators, including UDTF
## What changes were proposed in this pull request?
This PR is to convert SQL from analyzed logical plans containing Generate
operator.
Sample Plan :
```
GlobalLimit 3
+- LocalLimit 3
+- Project [gencol2#204]
+- Generate explode(gencol1#203), true, false, Some(gentab2),
[gencol2#204]
+- Generate explode(array(array(1, 2, 3))), true, false,
Some(gentab1), [gencol1#203]
+- MetastoreRelation default, t4, None
```
Generated Query:
```
SELECT `gentab2`.`gencol2` FROM `default`.`t4` LATERAL VIEW
explode(array(array(1, 2, 3))) `gentab1` AS `gencol1` LATERAL VIEW
explode(`gentab1`.`gencol1`) `gentab2` AS `gencol2` LIMIT 3
```
- Generators can be specified in either projection list or in the lateral
view clause.
- First case is handled in projectToSQL
- The second case is handled in generateToSQL
- Also contains the fix for spark-13698. As soon as it it merged , i will
rebase and submit again.
## How was this patch tested?
Added test cases in LogicalPlanToSQLSuite
Have also run HiveCompatibilitySuite to look for any failure in generation
of Generate plans.
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dilipbiswal/spark SPARK-12719
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11563.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11563
----
commit d2aa88ae294fc76e256080b6f55a3b11b65f6de6
Author: Dilip Biswal <[email protected]>
Date: 2016-03-05T09:49:58Z
[SPARK-13698] Fix Analysis Exceptions when Using Backticks in Generate
commit d49d5667cebe2c1319159dd4470467349e8cf027
Author: Dilip Biswal <[email protected]>
Date: 2016-03-07T07:42:00Z
[SPARK-12719] SQL generation support for generators (including UDTF)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
