This sounds like a perfect application of Zookeeper.
On Fri, Apr 24, 2009 at 11:47 PM, Arun C Murthy <a...@yahoo-inc.com> wrote: > On Apr 22, 2009, at 7:17 PM, Samprita Hegde wrote: > > > Currently all the information seems to be sent via the heartbeat message in >> the InterTracker Protocol. Is there a way where I can decouple only some >> part of heartbeat message and put it on to the space? (Especially the >> TaskCompletionEvent and TaskStatus). Using this the task completion events >> can be exchanged directly among the Task Trackers adn not through the Job >> Tracker. >> > > You would need to fix the TaskTrackers to update the shared-space rather > than send it to the JobTracker. > > One pertinent point to remember is that you do need some global arbitration > for e.g. deciding which among the concurrently successful speculative tasks > are to be declared as 'successful' etc. > -- Ted Dunning, CTO DeepDyve
