[
https://issues.apache.org/jira/browse/HIVE-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongqiang He updated HIVE-461:
------------------------------
Attachment: hive-461-2009-06-26.patch
Modifications:
1) change RCFile to accept column id list of read columns instead of skip
columns.
2) move the util method, which sets read column id list in conf, from RCFile to
RCFileInputFormat
3) fetch the read column id list from table scan operator in HiveInputFormat
4) store the read coliumn id list in table scan when does column pruner
> Optimize RCFile reading by using column pruning results
> -------------------------------------------------------
>
> Key: HIVE-461
> URL: https://issues.apache.org/jira/browse/HIVE-461
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.4.0
> Reporter: Zheng Shao
> Assignee: Yongqiang He
> Attachments: hive-461-2009-05-26.patch, hive-461-2009-06-26.patch
>
>
> RCFile is a column-based file format introduced in HIVE-352. Column-based
> storage has shown better compression ratio. On our internal data set (30
> columns, most of them are short integer strings), we are seeing
> gzip-compressed RCFile to be 20%+ smaller than gzip-compressed SequenceFile.
> RCFIle also has the potential to improve the reading efficiency a lot since
> it compresses each column separately.
> We should integrate RCFile with the column pruning results from Hive to make
> the reading faster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.