[
https://issues.apache.org/jira/browse/HAMA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867790#action_12867790
]
Hyunsik Choi commented on HAMA-251:
-----------------------------------
1) How to programming BSP with HAMA to process data on HDFS?
A task is paired with an input split. The input split is a HDFS block. The more
HDFS block is close to the reader, the more network cost is reduced. Thus, the
scheduler should assign as many tasks paired with local block as possible. For
this, BSPMaster needs to hold network topology and gets groom servers (with a
data node) in which certain DFS blocks reside during assigning tasks to groom
servers. This explain is scope of this issue.
2) How to configure its BSP job?
In my progress, job configuration is performed in BSPJob and JobContext. BSPjob
is a subclass of JobContext. JobContext has a Configuration instance. Like new
MapReduce API, BSPJob is a writable job instance, and JobContext is readonly
view. Parameters that users set are contained in the Configuration instance in
JobContext. As you know, A Configuration instance can be written into a xml
file, and it can be restored to a Configuration instance.
You can check the writeXml() in Configuration, BSPJob and JobContext.
3) How to submit Job through client interface?
Do you mean command line interface as the below comment?
https://issues.apache.org/jira/browse/HAMA-244?focusedCommentId=12857607&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12857607
If so, it is somewhat trivial. If we have internal implementation, we can make
any type interfaces. In addition, I made a really simple job submission
interface for test as you can see BSPJob and BSPExample. Right now, it is
enough to test job submission.
If you mean how jobs is disseminated, the job submission is through
BSPJob.java. In order to submit job, a user firstly needs to set some
parameters (e.g., Input Path, Output Path, and class names for java reflection)
to a BSPJob instance; my progress supports only to set a class name for BSP
computation. When the user submit the BSPJob instance, BSPJob contacts the
BSPMaster, and then it get new job id from BSPMaster. The BSPJob packs
configurations with JobID to a xml file and copies it to BSPMaster's local disk
via DFS remote copy. A Jar file that contains the job class is also copied to
BSPMaster's local disk.
Upon receiving the submitted job, BSPMaster adds the job to a simple scheduler.
The scheduler assigns tasks to groom servers via responses of heartbeat. So
far, I have designed that each groom server has only one task. For optimal
scheduling, BSPMaster needs status of groom servers. The status of groom
servers have to include the max task capacity and the number of running task
for each groom servers. For this, BSPMaster collects the statuses of all groom
servers that join the BSP cluster through heartbeat. Finally, the task
information is transmitted to groom servers via the response of heartbeat.
You can check the heartbeat(), submitJob() in BSPJob, and submitJobInternal()
in JobClient.
4) Then, Who manage the Job execution?
In overall, BSPMaster controls all groom servers. BSPMaster sends a lunch
action to groom servers through responses of heartbeat. Then, each groom runs
BSPPeer with a job class. During processing job, each groom sends status of a
running task to BSPMaster via heartbeat. BSPMaster keeps status of all running
tasks in a map data structure. Later, this information can be provided to users
via any kind interfaces, such as terminal and web. If a user kills certain job,
the user can send a kill action via BSPJob in the same way. These
communications are performed with Hadoop RPC. The patch already contains some
of these interaction.
You can check the heartbeat () in BSPMaster and transmitHeartBeat in
GroomServer.
> Add job manager to BSPMaster
> ----------------------------
>
> Key: HAMA-251
> URL: https://issues.apache.org/jira/browse/HAMA-251
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.2.0
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.2.0
>
> Attachments: HAMA-251_01.patch
>
>
> In regard to BSPMaster, I plan to design as follows:
> * BSPMaster is responsible to manage jobs submitted by clients.
> * BSPMaster is responsible to disseminate submitted jobs to BSPPeers.
> * BSPMaster is responsible to control job progress and BSPPeers actions.
> It will be similar to JobTracker. In this issue, let's discuss the above
> features of BSPMaster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
