Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2405#issuecomment-57566803
Okay here are some thoughts and questions:
- I don't think it really matters that we can't handle `f1.f11 > f2.f22`
because we already don't know what do to if a user does `[1,2] > [0,3]` even
without this new syntax.
- Am I correct in saying that hive doesn't support this syntax at all and
that we are inventing new functionality? I'm not strictly opposed to this, but
we should be careful as once we support something we can't get rid of it later.
- I'm not convinced that we need to handle arbitrary array nesting here.
The case of getting all of one field from an array (which i guess makes this
SQL short hand for `array.map(_.fieldName)`) seems reasonable, but is there a
use case for the arbitrary nesting version?
- This ends up complicating `GetField` quite a bit. What about creating a
new expression type `ArrayGetField` and adding something to the analyzer that
switches expression types when an array is detected. The idea here is to keep
each expression simple so we can code-gen on a case by case basis.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
