[
https://issues.apache.org/jira/browse/HAMA-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090007#comment-13090007
]
ChiaHung Lin commented on HAMA-413:
-----------------------------------
For the issue of `File does not exist', can we let client (PiEstimator,
SerializePrinting) use zk checking if the data has been written to hdfs or not
at the moment (as temporarily solution)? Or before set taskStatus's phase to
cleanup, we can check if the execution has finished or not.
The problem I can think of with child process reporting to master directly is
the system may end up with 3 times rpc requests (e.g. 6,000 groom servers with
12,000 rpc execution). The master may simply be busy dealing with such trivial
tasks. This would have impact on the performance because if I remember
correctly, one of reason the birth of mapreduce 2.0 is too many rpc, including
tasktracker and clients.
> Remove limitation on the number of tasks
> ----------------------------------------
>
> Key: HAMA-413
> URL: https://issues.apache.org/jira/browse/HAMA-413
> Project: Hama
> Issue Type: Sub-task
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-413_v01.patch, HAMA-413_v02.patch,
> HAMA-413_v03.patch, HAMA_413_v04.patch
>
>
> By HAMA-410 patch, BSPPeer object will be constructed at child process. Now
> we can just remove limitation on the number of tasks.
> Here's TODO list:
> 1. The number of tasks per groom should be configurable e.g.,
> 'bsp.local.tasks.maximum'.
> 2. The 'totalTaskCapacity' should be calculated at
> BSPMaster.getClusterStatus().
> 3. When scheduling tasks, consider how to allocate them.
> 4. Each BSPPeer should know all created peers of Hama cluster by job. It can
> be listed based on actions of GroomServer.
> 5. In examples, 'cluster.getGroomServers()' can be changed to
> 'cluster.getMaxTasks()'.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
