[
https://issues.apache.org/jira/browse/HIVE-13828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15297816#comment-15297816
]
Gopal V commented on HIVE-13828:
--------------------------------
-0 on the idea in general.
Mostly found that this caused OOMs during split generation, since the AMs which
open > 1000+ files spend a significant amount of memory serializing these
footer objects and that ends up in the recovery information for Tez.
> Enable hive.orc.splits.include.file.footer by default
> -----------------------------------------------------
>
> Key: HIVE-13828
> URL: https://issues.apache.org/jira/browse/HIVE-13828
> Project: Hive
> Issue Type: Improvement
> Components: Hive
> Reporter: Rajesh Balamohan
> Priority: Minor
>
> As a part of setting up the OrcInputFormat.getRecordReader in the task side,
> hive ends up opening the file path and reads the metadata information. If
> hive.orc.splits.include.file.footer=true, this metadata info can be passed on
> to task side which can help reduce the overhead. It would be good to
> consider enabling this parameter by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
