[
https://issues.apache.org/jira/browse/HAMA-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205365#comment-13205365
]
Suraj Menon commented on HAMA-511:
----------------------------------
Nice to see some views expressed. There is no way we should be giving up the
simple or rather the new simple fault tolerant BSP API.
The reason, I rushed to express this idea, is for us to keep it in mind while
we design and implement fault tolerance, which is our current focus and there
are people already working on it.
As an example, when I am making the checkpointing configurable with a simple
modulo logic today, I was making it a modulo function of a counter for number
of times sync() function is called. Now with selective superstep
synchronization in mind, I have to make the checkpointing logic a function of
current superstep number.
Regarding the changes to be made, I was encouraged by the design that Thomas
had in his github repo.
https://github.com/thomasjungblut/thomasjungblut-common/blob/master/src/de/jungblut/bsp/ft/FaultTolerantBSP.java
I think It already necessitates sending the Superstep array to BSPPeer. For
selective synchronization, we would need multidimensional array to be sent with
the column of the array to be executed. The column number would also have to be
part of the identity of Zookeeper node for synchronization. I agree this is not
a small task.
In offline mode, we can always implement task precedence constraints with
multiple batch processing (like Oozie for hadoop-mapreduce). However, I think
having this flexibility would be really useful for real-time Hama tasks. I feel
this would give Hama capability to be a framework for implementing distributed
real-time computation tasks as well. We can evaluate the design of S4(Y!),
EarlyBird(Twitter) and others to verify this.
> Submitting heterogenous supersteps with precedence constraints on Hama
> ----------------------------------------------------------------------
>
> Key: HAMA-511
> URL: https://issues.apache.org/jira/browse/HAMA-511
> Project: Hama
> Issue Type: New Feature
> Reporter: Suraj Menon
> Priority: Minor
> Attachments: Defining supersteps for BSP.pdf
>
>
> Hama should support submission of jobs with support for:
> 1) Skipping unwanted superstep synchronization.
> 2) Run supersteps with heterogenous nature of computation
> 3) Scheduling supersteps with precedence constraints.
> An explanation of these is provided in the attachment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
