Ramon, If your doing this all from command line, you would launch the job with jar and then take note of the Running job: <id> in the output. After you get the Running job ID, you would launch "./hadoop job -set-priority <jobID> HIGH" to change the priority. If your doing this all through the Hadoop Java API's, you will need to launch the job using Job (new) or JobClient (old) and iterate find the JobStatus for the job that you submitted. After you found the JobStatus, you would .setJobPriority(JobPriority.HIGH); on the JobStatus. I think its a little different with the newer Job API. OceanSync.com does do this and will be released in the coming days, its in BETA now. They allow you to change job priorities in our job management section. The software will have a freeware version available. Alternative (and not recommended) to the Java API's, you can launch the job using the Java Runtime.exec ProcessBuilder so that you can still use the hadoop binary to launch the job and then set the priority after finding the subString for the jobID.
[doovel@dovix bin]$ ./hadoop jar wordCount.jar 11/11/21 02:30:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 11/11/21 02:30:22 INFO mapred.FileInputFormat: Total input paths to process : 1 11/11/21 02:30:22 INFO mapred.JobClient: Running job: job_201111202344_0002 11/11/21 02:30:23 INFO mapred.JobClient: map 0% reduce 0% [doovel@dovix bin]$ ./hadoop job -set-priority job_201111202344_0002 HIGH Changed job priority. [doovel@dovix bin]$ ________________________________ Ronnie Dove Ocean Sync Hadoop Management Software Development Leader http://www.oceansync.com ________________________________ From: WangRamon <[email protected]> To: [email protected] Sent: Monday, November 21, 2011 1:26 AM Subject: RE: How to manage hadoop job submit? Thanks Richard, but i still have a question, if i'm using hadoop jar to submit a job, how can i set the priority for each job? I do see something like this "mapred.job.priority", but when i put it in my Job's configuration like this: conf.set("mapred.job.priority", "HIGH"), it seems it doesn't work. Thanks Cheers Ramon > Date: Sun, 20 Nov 2011 09:42:43 -0800 > From: [email protected] > Subject: Re: How to manage hadoop job submit? > To: [email protected] > > Ramon, > > You might issue an ./hadoop -list all to get the jobs and then -set-priority > <id> <priority>. I know that someone from that Ocean Sync > (http://www.oceansync.com) Hadoop management project is working with > interacting with MapReduce jobs through a GUI, to set priorities through > that, but they are still in beta. > > > job > Command to interact with Map Reduce Jobs. > Usage: hadoop job [GENERIC_OPTIONS] [-submit <job-file>] | [-status > <job-id>] | [-counter <job-id> <group-name> <counter-name>] | [-kill > <job-id>] | [-events <job-id> <from-event-#> <#-of-events>] | [-history > [all] <jobOutputDir>] | [-list [all]] | [-kill-task <task-id>] | [-fail-task > <task-id>] | [-set-priority <job-id> <priority>] > > > ----- Original Message ----- > From: WangRamon <[email protected]> > To: [email protected] > Cc: > Sent: Sunday, November 20, 2011 4:44 AM > Subject: How to manage hadoop job submit? > > > Hi All > I'm new to hadoop, I know I can use "haddop jar" to submit my M/R job, but we > need to submit a lot of jobs in my real environment, there is priority > requirement for each jobs, so is there any way to manage how to submit jobs? > Any Java API? Or we can only use the hadoop command line with shell or python > to do the job submit? > Thanks Ramon
