Hi Tylen, I'm not worried about the scheduling since there are tons of mechanisms available for that. But what I want to prevent is that every node in the cluster starts to schedule the same map/reduce jobs. Only one machine is sufficient and when that machine fails, another one should take over.
On Wed, Jan 5, 2011 at 7:31 PM, Tyler Coffin <[email protected]> wrote: > Cron works great and is probably already on your systems. > > -----Original Message----- > From: Peter Veentjer [mailto:[email protected]] > Sent: January 5, 2011 12:35 > To: [email protected] > Subject: Scheduling map/reduce jobs > > He Guys, > > although it isn't completely related to HBase. Is there support for > scheduling map reduce jobs? > > E.g. I want to do a map reduce job that automatically removes certain > elements from hbase and perhaps some additional cleanup (I know that there > is support for lease times). > > I could have every node in the cluster schedule this job every hour, but if > there are n nodes I don't want this job to be running n times. Only 1 time > would be sufficient. I can create some checking logic for scheduling a map > reduce job where a job that should not be run is rejected/ignored, but if > there is something out of the box. > > Another question: > > We are building an environment where client provided plugins can be > deployed > and obey some kind of SLA (e.g. we want it to be running on 2 nodes at > least). So the old plugin needs to be undeployed, the jars of the new > plugin > need to be copied on demand to the nodes and the new plugin needs to be > deployed. Is there functionality for this behavior out of the box or does > it > need to be implemented by ourselves? > > --------------------------------------------------------------------- > This transmission (including any attachments) may contain confidential > information, privileged material (including material protected by the > solicitor-client or other applicable privileges), or constitute non-public > information. Any use of this information by anyone other than the intended > recipient is prohibited. If you have received this transmission in error, > please immediately reply to the sender and delete this information from your > system. Use, dissemination, distribution, or reproduction of this > transmission by unintended recipients is not authorized and may be unlawful. >
