Hi Peter, That part is taken care of by the hadoop framework itself, more specifically, my the jobtracker.
Hari On Thu, Jan 6, 2011 at 1:25 AM, Peter Veentjer <[email protected]>wrote: > Hi Tylen, > > I'm not worried about the scheduling since there are tons of mechanisms > available for that. But what I want to prevent is that every node in the > cluster starts to schedule the same map/reduce jobs. Only one machine is > sufficient and when that machine fails, another one should take over. > > On Wed, Jan 5, 2011 at 7:31 PM, Tyler Coffin <[email protected]> wrote: > > > Cron works great and is probably already on your systems. > > > > -----Original Message----- > > From: Peter Veentjer [mailto:[email protected]] > > Sent: January 5, 2011 12:35 > > To: [email protected] > > Subject: Scheduling map/reduce jobs > > > > He Guys, > > > > although it isn't completely related to HBase. Is there support for > > scheduling map reduce jobs? > > > > E.g. I want to do a map reduce job that automatically removes certain > > elements from hbase and perhaps some additional cleanup (I know that > there > > is support for lease times). > > > > I could have every node in the cluster schedule this job every hour, but > if > > there are n nodes I don't want this job to be running n times. Only 1 > time > > would be sufficient. I can create some checking logic for scheduling a > map > > reduce job where a job that should not be run is rejected/ignored, but if > > there is something out of the box. > > > > Another question: > > > > We are building an environment where client provided plugins can be > > deployed > > and obey some kind of SLA (e.g. we want it to be running on 2 nodes at > > least). So the old plugin needs to be undeployed, the jars of the new > > plugin > > need to be copied on demand to the nodes and the new plugin needs to be > > deployed. Is there functionality for this behavior out of the box or does > > it > > need to be implemented by ourselves? > > > > --------------------------------------------------------------------- > > This transmission (including any attachments) may contain confidential > > information, privileged material (including material protected by the > > solicitor-client or other applicable privileges), or constitute > non-public > > information. Any use of this information by anyone other than the > intended > > recipient is prohibited. If you have received this transmission in error, > > please immediately reply to the sender and delete this information from > your > > system. Use, dissemination, distribution, or reproduction of this > > transmission by unintended recipients is not authorized and may be > unlawful. > > >
