I thank the best option would be able to set the max per node in its config file I thank someone is or has worked on this I seen something in jira.
for the new option I would thank a job over ride would work something like this 1) Check node config if job over ride is lower then node then use job override but if local node max is lower then use it this way we could slow down the number of task for a job if needed on nodes that has excess compacty. or 2) Jobconf would over ride all nodes local settings on this. Billy "Arun C Murthy" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > On Thu, Jan 03, 2008 at 10:12:04AM +0530, Arun C Murthy wrote: >>On Wed, Jan 02, 2008 at 12:08:53PM -0800, Jason Venner wrote: >>>In our case, we have specific jobs that due to resource constraints can >>>only be run serially (ie: 1 instance per machine). >> >>I see, at this point there isn't anything in Hadoop which can help you out >>here... >> > > Given that, please file a jira for this enhancement anyway... Thanks! > > I'd imagine we should consider features such as: > a) Max simultaneous tasks per node per job (current ask). > b) Max concurrent tasks per job cluster-wide (i.e. don't run more than 25, > or an absolute number, of maps of a given job simultaneously on the > cluster) - this should help jobs which need to respect SLAs of external > services regardless of cluster sizes - don't open more than 150 > simultaneous db-connections. > > Arun > >>Having said that, could you consider running another Map-Reduce cluster >>with mapred.tasktracker.map.tasks.maximum set to 1 for these special jobs? >>Run this cluster on the same machines simultaneously with the your >>_regular_ cluster; just pick different ports etc. >> >>hth, >>Arun >> >>>Most of our jobs are more normal and can be run in parallel on the >>>machines. >>> >>>Arun C Murthy wrote: >>>>Billy, >>>> >>>> >>>>On Wed, Jan 02, 2008 at 01:38:06PM -0600, Billy wrote: >>>> >>>>>If I add this to a command line as a -jobconf should it be enforced? >>>>> >>>>> >>>> >>>>This is a property of the TaskTracker and hence cannot be set on a >>>>per-job >>>>basis... >>>> >>>> >>>>>Say I have a job that I want to run only 1 map at a time per server >>>>> >>>>> >>>> >>>>Could you describe your reasons? >>>> >>>>Arun >>>> >>>> >>>>>I have tried this and look in the job.xml file and its set correctly >>>>>but >>>>>not enforced. >>>>> >>>>> >>>> >>>> >>>> >>>>>Billy >>>>> >>>>> >>>>> >>>>> >
