Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/1439#issuecomment-49338675
I think we are not clear on the boundary between a `TableReader` and a
physical `TableScan` operator (e.g. `HiveTableScan`). Seems we just want
`TableReader` to create `RDD`s (general-purpose work) and inside a `TableScan`
operator, we create Catalyst `Row`s (table-specific work). However, when we
look at `HadoopTableReader`, it is actually a `HiveTableReader`. For every Hive
partition, we create a `HadoopRDD` (requiring Hive-specific code) and
deserialize Hive rows. I am not sure if `TableReader` is a good abstraction.
I think it makes sense to remove the trait of `TableReader` and add a
abstract `TableScan` class (inheriting `LeafNode`). All existing TableScan
operators will inherit this abstract `TableScan` class. If we think it is the
right approach. I can do it in another PR.
@marmbrus, @liancheng, @rxin, @chenghao-intel thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
