[
https://issues.apache.org/jira/browse/CRUNCH-263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Wills updated CRUNCH-263:
------------------------------
Attachment: CRUNCH-263.patch
Here's a patch that sets the default max split size based on the dfs.block.size
for HDFS.
> Create intelligent defaults for the CrunchCombineFileInputFormat
> ----------------------------------------------------------------
>
> Key: CRUNCH-263
> URL: https://issues.apache.org/jira/browse/CRUNCH-263
> Project: Crunch
> Issue Type: Bug
> Components: Core, IO
> Affects Versions: 0.8.0
> Reporter: Josh Wills
> Assignee: Josh Wills
> Fix For: 0.8.0
>
> Attachments: CRUNCH-263.patch
>
>
> The CombineFileInputFormat is too aggressive at combining file blocks unless
> a sensible mapred.max.split.size value is set in the Configuration or is
> explicitly configured on the CombineFileInputFormat class itself. This JIRA
> seeks to set intelligent defaults for the max split size in the case that a
> default is not already provided and provide a way for Crunch clients to
> override the defaults via a configuration parameter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
