Hi Mehmet, Are you using MR1 or MR2?
The fair scheduler, present in both versions, but configured slightly differently, allows you to limit the number of map and reduce tasks in a queue. The configuration can be updated at runtime by modifying the scheduler's allocations file. It also has a feature that automatically maps jobs to queues based on the user submitted them. Here are links to documentation in MR1 and MR2: http://hadoop.apache.org/docs/stable/fair_scheduler.html http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html -Sandy On Tue, May 21, 2013 at 2:43 PM, Mehmet Belgin <mehmet.bel...@oit.gatech.edu > wrote: > Hi Everyone, > > I was wondering if there is a way for limiting the number of tasks > (map+reduce) *per user* at runtime? Using an environment variable perhaps? > I am asking this from a resource provisioning perspective. I am trying to > come up with a N-token licensing system for multiple users to use our > limited hadoop resources simultaneously. That is, when user A checks out 6 > tokens, he/she can only run 6 hadoop tasks. > > If there is no such thing in hadoop, has anyone tried to integrate hadoop > with torque/moab (or any other RM or scheduler)? Any advice in that > direction will be appreciated :) > > Thanks in advance, > -Mehmet > > > > > > >
