GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/11596
[SPARK-12719][SQL] [WIP] SQL generation support for generators, including
UDTF
## What changes were proposed in this pull request?
This is a alternate way to convert SQL from analyzed logical plans
containing Generate operator.
In this PR , generators in projection list are expressed as LATERAL VIEW.
Sample Plan :
```
GlobalLimit 3
+- LocalLimit 3
+- Project [gencol2#204]
+- Generate explode(gencol1#203), true, false, Some(gentab2),
[gencol2#204]
+- Generate explode(array(array(1, 2, 3))), true, false,
Some(gentab1), [gencol1#203]
+- MetastoreRelation default, t4, None
```
Generated Query:
```
SELECT `gentab2`.`gencol2` FROM `default`.`t4` LATERAL VIEW
explode(array(array(1, 2, 3))) `gentab1` AS `gencol1` LATERAL VIEW
explode(`gentab1`.`gencol1`) `gentab2` AS `gencol2` LIMIT 3
```
## How was this patch tested?
Tests added to LogicalPlanToSQLSuite
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dilipbiswal/spark SPARK-12719-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11596.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11596
----
commit fe8b0642994fd976681d7e675712738bb6f7076b
Author: Dilip Biswal <[email protected]>
Date: 2016-03-05T09:49:58Z
[SPARK-13698] Fix Analysis Exceptions when Using Backticks in Generate
commit 1e9b683febba74192d5a882bcd38d0a47dd630bf
Author: Dilip Biswal <[email protected]>
Date: 2016-03-07T07:42:00Z
[SPARK-12719] SQL generation support for generators (including UDTF)
commit 865f5a5afaaff2c1eb101630328c362dc63e0ddf
Author: Dilip Biswal <[email protected]>
Date: 2016-03-07T21:28:48Z
Code review comments
commit c037d8ccc008bcba518a4be62da385d1fd037756
Author: Dilip Biswal <[email protected]>
Date: 2016-03-08T02:36:21Z
Review comments from Reynold
commit e10883a720c7e10d5a5294093c473ff6ff358d0c
Author: Dilip Biswal <[email protected]>
Date: 2016-03-08T07:35:46Z
Alternative approach.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
