Github user dilipbiswal commented on the pull request:
https://github.com/apache/spark/pull/11563#issuecomment-193689584
@cloud-fan Thanks..I understand now from Reynold's and your feedback. I
wrongly assumed the goal to make the best effort to retain the original sql
layout.
Regarding the pattern of logical plans between the two cases here is
what comes to my mind now. I will update if i remember more.
* Projection List
* The generate node has no qualifier.
* There can be one generator in the projection List
* Join is not always true.
* LATERAL VIEW
* Generate has a qualifier (column names optional)
* There can be multiple generators. So we need preserve the order of
from clause.
* Join is always true as there has to be a table before the LATERAL
VIEW. This
can cause ambiguity of column resolution if we express the
projection list generators
as lateral views and don't qualify the projection list properly.
```
+- Project [gencol2#204]
+- Generate explode(gencol1#203), true, false, Some(gentab2),
[gencol2#204]
+- Generate explode(array(array(1, 2, 3))), true, false,
Some(gentab1), [gencol1#203]
+- MetastoreRelation default, t4, None
```
In the above plan, we have two LATERAL VIEW clauses with qualifier gentab1
and gentab2 respectively.
```
Project [value#53]
+- Generate explode(array(1, 2, 3)), false, false, None, [value#53]
+- MetastoreRelation default, src, None
```
In the above case, its a generator in the projection list.
There is a test which uses generator but has no table in the FROM clause
like following.
```SQL
select explode(array(1,2,3)) AS gencol
```
I suspect this may cause problem if we need to express the generator as a
LATERAL VIEW.
Please let me know if you need any other info.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
