I noticed it is not possible to run simultaneously two jobs by the same user, the second job gets stuck at map 0% reduce 0%
On 10/01/2008, Jeff Hammerbacher <[EMAIL PROTECTED]> wrote: > > it's a stopgap and doesn't seem to be working well for y!: > https://issues.apache.org/jira/browse/HADOOP-2510 > > On Jan 9, 2008 5:30 PM, Ted Dunning <[EMAIL PROTECTED]> wrote: > > > > > What is the status of Hadoop on Demand? Is it ready for prime time? > > > > > > On 1/9/08 4:58 PM, "Aaron Kimball" <[EMAIL PROTECTED]> wrote: > > > > > I will add to the discussion that the ability to have multiple tasks > of > > > equal priority all making progress simultaneously is important in > > > academic environments. There are a number of undergraduate programs > > > which are starting to use Hadoop in code labs for students. > > > > > > Multiple students should be able to submit jobs and if one student's > > > poorly-written task is grinding up a lot of cycles on a shared > cluster, > > > other students still need to be able to test their code in the > meantime; > > > ideally, they would not need to enter a lengthy job queue. ... I'd say > > > that this actually applies to development clusters in general, where > > > individual task performance is less important than the ability of > > > multiple developers to test code concurrently. > > > > > > - Aaron > > > > > > > > > > > > Joydeep Sen Sarma wrote: > > >>> that can run(per job) at any given time. > > >> > > >> not possible afaik - but i will be happy to hear otherwise. > > >> > > >> priorities are a good substitute though. there's no point needlessly > > >> restricting concurrency if there's nothing else to run. if there is > > something > > >> else more important to run - then in most cases, assigning a higher > > priority > > >> to that other thing would make the right thing happen. > > >> > > >> except with long running tasks (usually reducers) that cannot be > > preempted. > > >> (Hadoop does not seem to use OS process priorities at all. I wonder > if > > >> process priorities can be used as a substitute for pre-emption.) > > >> > > >> HOD is another solution that you might want to look into - my > > understanding > > >> is that with HOD u can restrict the number of machines used by a job. > > >> > > >> ________________________________ > > >> > > >> From: Xavier Stevens [mailto:[EMAIL PROTECTED] > > >> Sent: Wed 1/9/2008 2:57 PM > > >> To: hadoop-user@lucene.apache.org > > >> Subject: RE: Question on running simultaneous jobs > > >> > > >> > > >> > > >> This doesn't work to solve this issue because it sets the total > number > > >> of map/reduce tasks. When setting the total number of map tasks I get > > an > > >> ArrayOutOfBoundsException within Hadoop; I believe because of the > input > > >> dataset size (around 90 million lines). > > >> > > >> I think it is important to make a distinction between setting total > > >> number of map/reduce tasks and the number that can run(per job) at > any > > >> given time. I would like only to restrict the later, while allowing > > >> Hadoop to divide the data into chunks as it sees fit. > > >> > > >> > > >> -----Original Message----- > > >> From: Ted Dunning [mailto:[EMAIL PROTECTED] > > >> Sent: Wednesday, January 09, 2008 1:50 PM > > >> To: hadoop-user@lucene.apache.org > > >> Subject: Re: Question on running simultaneous jobs > > >> > > >> > > >> You may need to upgrade, but 15.1 does just fine with multiple jobs > in > > >> the cluster. Use conf.setNumMapTasks(int) and > > >> conf.setNumReduceTasks(int). > > >> > > >> > > >> On 1/9/08 11:25 AM, "Xavier Stevens" <[EMAIL PROTECTED]> wrote: > > >> > > >>> Does Hadoop support running simultaneous jobs? If so, what > parameters > > >> > > >>> do I need to set in my job configuration? We basically want to give > a > > >> > > >>> job that takes a really long time, half of the total resources of > the > > >>> cluster so other jobs don't queue up behind it. > > >>> > > >>> I am using Hadoop 0.14.2 currently. I tried setting > > >>> mapred.tasktracker.tasks.maximum to be half of the maximum specified > > >>> in mapred-default.xml. This shows the change in the web > > >>> administration page for the job, but it has no effect on the actual > > >>> numbers of tasks running. > > >>> > > >>> Thanks, > > >>> > > >>> Xavier > > >>> > > >> > > >> > > >> > > >> > > >> > > >> > > > > > -- --------------------------------------------------------- شهر مبارك كريم كل عام و أنتم بخير --------------------------------------------------------- Honsali Khalil − 本査理 カリル Academic>Japan>NIT>Grad. Sc. Eng.>Dept. CS>Matsuo&Tsumura Lab. http://www.matlab.nitech.ac.jp/~k-hon/ +81 (zero-)eight-zero 5134 8119 [EMAIL PROTECTED] (instant reply mail)
