[
https://issues.apache.org/jira/browse/HADOOP-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666589#action_12666589
]
Tom White commented on HADOOP-4565:
-----------------------------------
Do you know if this new input format will work well with Hadoop Archives (HAR
files)? I wonder because the main use for HARs is to pack lots of small files
into a few HDFS files, to reduce memory resource pressure on the namenode, so
it seems like a good candidate for combining multiple files into splits.
I had a quick look at the patch and had one comment (sorry to come to this
late). The test should use asserts to check that the split locations are as
expected rather than printing to System.out.
> MultiFileInputSplit can use data locality information to create splits
> ----------------------------------------------------------------------
>
> Key: HADOOP-4565
> URL: https://issues.apache.org/jira/browse/HADOOP-4565
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: CombineMultiFile.patch, CombineMultiFile2.patch,
> CombineMultiFile3.patch, CombineMultiFile4.patch, CombineMultiFile5.patch,
> CombineMultiFile7.patch, CombineMultiFile8.patch
>
>
> The MultiFileInputFormat takes a set of paths and creates splits based on
> file sizes. Each splits contains a few files an each split are roughly equal
> in size. It would be efficient if we can extend this InputFormat to create
> splits such each all the blocks in one split and either node-local or
> rack-local.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
