Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9497#issuecomment-154154315
I do want to simplify this, but I'm not sure that this is the right
approach. Specifically, field reordering is a very important feature. Imagine
the case where you are reading from JSON and you only want some of the fields.
Furthermore, as we try to make this competitive with/faster than with RDDs I
think it will be useful to be able to extract from arbitrary schemata without
having to do an extra projection.
One thing that I think we can simplify is to have only two steps that
happen at well defined places:
- **Resove** - happens when you pass an encoder into a Dataset, at the end
it should only have AttributeReferences (i.e. this a combination of unbind and
resolve)
- **Bind** - Right before you are actually going to use an encoder you
bind it to the row's schema.
Hopefully this approach could be simpler and easier to reason about.
Either way, since this is an internal detail, I'd like to table this until
after 1.6. I'm way more worried about fleshing out the API and making sure
that there is good Java support.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
