Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/11443#issuecomment-194691016
@yhuai `HiveCompatibilitySuite` passed both locally and on Jenkins.
@marmbrus @rxin OK, then I'll merge this one as soon as Jenkins passes. An
update of the most recent changes and planned follow-ups:
Changes:
- Dataset eager analysis
At first I tried to fix this issue by passing encoder by name, then
workaround this issue via lazy vals and customized serialization. But later on
I felt that this debugging feature probably isn't worth messing up Dataset
constructor. Since this debugging feature is only used in
`PlanTest.checkAnswer`, I resorted to `QueryExecution.assertAnalyzed`. Related
changes are described in the updated PR description. Details can be found in
[this commit][1].
Follow-ups:
- As Michael commented, I'll remove `collectRows` and `takeRows` and use
either `collectAsList`/`takeAsList` or casting when appropriate through out the
code base.
- (Other TODO entries listed in the PR description)
[1]:
https://github.com/liancheng/spark/commit/bf7a17a68eb0f41a4cf5c926860449beddc0622c
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
