[
https://issues.apache.org/jira/browse/HAMA-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462373#comment-13462373
]
Yuesheng Hu commented on HAMA-647:
----------------------------------
hi Edward,
If files.length == numSplits == 1, the method will return, the "else if"
clause will not be executed.
But when files.length/2 > numSplits, the goalSize will be negative.
Another problem is : if user setNumTasks to be zero, the spliter can work! I
think it should stop the submit and
print a error message, ranther than set the numSplits to be 1 and let the job
continue to run.
Am I missing something?
> Make the input spliter robustly
> --------------------------------
>
> Key: HAMA-647
> URL: https://issues.apache.org/jira/browse/HAMA-647
> Project: Hama
> Issue Type: Improvement
> Components: bsp core
> Affects Versions: 0.5.0, 0.6.0
> Reporter: Yuesheng Hu
> Assignee: Yuesheng Hu
> Priority: Critical
> Fix For: 0.6.0
>
> Attachments: HAMA-647-2.patch, HAMA-647.patch
>
>
> Currently, the spliter in FileInputFormat is based on the Mapreduce's
> spliter. But, Hama is different from Mapreduce, Hama's task can not be
> pended until the slot becomes free. So, the current spliter is not suitable
> for Hama. When input file is small, it may be ok, but when input is very
> large, the number of splits will be very large too, even our cluster is
> powerful enough to handle the input. More details, please see the comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
