from the error looks like you have some incorrect hive settings which are failing the job initialization.
this is the error >java.io.IOException: Number of maps in JobConf doesn't match number of > recieved splits for job job_201210051717_0015! numMapTasks=10 can you tell us if you are setting in hive variables before firing up the query? something like split size or # no maps etc On Mon, Oct 8, 2012 at 5:30 PM, Ajit Kumar Shreevastava <ajit.shreevast...@hcl.com> wrote: > Hi Nitin, > > > > Job tracker log:--> > > > > 2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configuration: > /usr/local/hadoopstorage/mapred/local/jobTracker/job_201210051717_0015.xml:a > attempt to override fina > > l parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configuration: > /usr/local/hadoopstorage/mapred/local/jobTracker/job_201210051717_0015.xml:a > attempt to override fina > > l parameter: mapred.map.tasks; Ignoring. > > 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobInProgress: > job_201210051717_0015: nMaps=10 nReduces=2 max=-1 > > 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker: Job > job_201210051717_0015 added successfully for user 'hadoop' to queue > 'default' > > 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.AuditLogger: > USER=hadoop IP=10.99.42.9 OPERATION=SUBMIT_JOB > TARGET=job_201210051717_0015 RESULT=SUCCESS > > 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker: > Initializing job_201210051717_0015 > > 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobInProgress: > Initializing job_201210051717_0015 > > 2012-10-08 15:43:29,350 INFO org.apache.hadoop.mapred.JobInProgress: > jobToken generated and stored with users keys in > /usr/local/hadoopstorage/mapred/system/job_2012100 > > 51717_0015/jobToken > > 2012-10-08 15:43:29,352 ERROR org.apache.hadoop.mapred.JobTracker: Job > initialization failed: > > java.io.IOException: Number of maps in JobConf doesn't match number of > recieved splits for job job_201210051717_0015! numMapTasks=10, #splits=1 > > at > org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:703) > > at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207) > > at > org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > > 2012-10-08 15:43:29,352 INFO org.apache.hadoop.mapred.JobTracker: Failing > job job_201210051717_0015 > > 2012-10-08 15:43:29,352 INFO > org.apache.hadoop.mapred.JobInProgress$JobSummary: > jobId=job_201210051717_0015,submitTime=1349691209316,launchTime=0,,finishTime=1349691209 > > 352,numMaps=0,numSlotsPerMap=1,numReduces=0,numSlotsPerReduce=1,user=hadoop,queue=default,status=FAILED,mapSlotSeconds=0,reduceSlotsSeconds=0,clusterMapCapacity=14,clus > > terReduceCapacity=14,jobName=select count(1) from pokes(Stage-1) > > 2012-10-08 15:43:29,353 INFO org.apache.hadoop.mapred.JobHistory: Moving > file:/var/log/hadoop/history/job_201210051717_0015_1349691209316_hadoop_select+count%281%29+fro > > m+pokes%28Stage-1%29 to > file:/var/log/hadoop/history/done/version-1/NHCLT-PC44-2_1349437647390_/2012/10/08/000000 > > 2012-10-08 15:43:29,356 INFO org.apache.hadoop.mapred.JobHistory: Moving > file:/var/log/hadoop/history/job_201210051717_0015_conf.xml to > file:/var/log/hadoop/history/don > > e/version-1/NHCLT-PC44-2_1349437647390_/2012/10/08/000000 > > 2012-10-08 15:43:30,362 INFO org.apache.hadoop.mapred.JobTracker: Killing > job job_201210051717_0015 > > 2012-10-08 17:17:35,641 INFO > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > Updating the current master key for generating delegation > > tokens > > > > > > and hive log:--> > > > > 2012-10-08 15:43:08,597 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:08,606 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:08,606 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:09,000 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:09,004 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:09,004 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:09,014 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:09,018 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:09,018 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:17,309 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:17,319 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:17,320 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:17,759 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:17,766 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:17,766 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:17,783 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:17,786 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:17,787 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.core.resources" but it cannot be resolved. > > 2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.core.resources" but it cannot be resolved. > > 2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.core.runtime" but it cannot be resolved. > > 2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.core.runtime" but it cannot be resolved. > > 2012-10-08 15:43:24,120 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.text" but it cannot be resolved. > > 2012-10-08 15:43:24,120 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.text" but it cannot be resolved. > > 2012-10-08 15:43:24,734 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:24,750 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:24,751 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:28,535 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:28,537 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:28,537 WARN conf.Configuration > (Configuration.java:loadResource(1245)) - > file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final > parameter: mapred.reduce.tasks; Ignoring. > > 2012-10-08 15:43:29,113 WARN mapred.JobClient > (JobClient.java:copyAndConfigureFiles(667)) - Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the same. > > 2012-10-08 15:43:29,243 WARN snappy.LoadSnappy > (LoadSnappy.java:<clinit>(46)) - Snappy native library not loaded > > 2012-10-08 15:43:30,354 ERROR exec.Task (SessionState.java:printError(400)) > - Ended Job = job_201210051717_0015 with errors > > 2012-10-08 15:43:30,356 ERROR exec.Task (SessionState.java:printError(400)) > - Error during job, obtaining debugging information... > > 2012-10-08 15:43:30,369 ERROR ql.Driver (SessionState.java:printError(400)) > - FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.MapRedTask > > > > > > Regards, > > Ajit > > > > -----Original Message----- > From: Nitin Pawar [mailto:nitinpawar...@gmail.com] > Sent: Monday, October 08, 2012 4:19 PM > To: user@hive.apache.org > Subject: Re: hive mapred problem > > > > Can you provide with job logs and failed task logs? > > > > > > On Mon, Oct 8, 2012 at 4:13 PM, Ajit Kumar Shreevastava > > <ajit.shreevast...@hcl.com> wrote: > >> Hi, > >> > >> > >> > >> When I run the query "select count(1) from pokes;" it fails with the >> message > >> as below. > >> > >> > >> > >> hive> select count(1) from pokes; > >> > >> Total MapReduce jobs = 1 > >> > >> Launching Job 1 out of 1 > >> > >> Number of reduce tasks determined at compile time: 1 > >> > >> In order to change the average load for a reducer (in bytes): > >> > >> set hive.exec.reducers.bytes.per.reducer=<number> > >> > >> In order to limit the maximum number of reducers: > >> > >> set hive.exec.reducers.max=<number> > >> > >> In order to set a constant number of reducers: > >> > >> set mapred.reduce.tasks=<number> > >> > >> Starting Job = job_201210051717_0015, Tracking URL = > >> http://NHCLT-PC44-2:50030/jobdetails.jsp?jobid=job_201210051717_0015 > >> > >> Kill Command = /home/hadoop/hadoop-1.0.3/bin/hadoop job -kill > >> job_201210051717_0015 > >> > >> Hadoop job information for Stage-1: number of mappers: 0; number of > >> reducers: 0 > >> > >> 2012-10-08 15:43:30,351 Stage-1 map = 100%, reduce = 100% > >> > >> Ended Job = job_201210051717_0015 with errors > >> > >> Error during job, obtaining debugging information... > >> > >> FAILED: Execution Error, return code 2 from > >> org.apache.hadoop.hive.ql.exec.MapRedTask > >> > >> MapReduce Jobs Launched: > >> > >> Job 0: HDFS Read: 0 HDFS Write: 0 FAIL > >> > >> Total MapReduce CPU Time Spent: 0 msec > >> > >> > >> > >> Thanks and Regards > >> > >> Ajit Kumar Shreevastava > >> > >> ADCOE (App Development Center Of Excellence ) > >> > >> Mobile: 9717775634 > >> > >> > >> > >> > >> > >> ::DISCLAIMER:: > >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------- > >> > >> The contents of this e-mail and any attachment(s) are confidential and > >> intended for the named recipient(s) only. > >> E-mail transmission is not guaranteed to be secure or error-free as > >> information could be intercepted, corrupted, > >> lost, destroyed, arrive late or incomplete, or may contain viruses in > >> transmission. The e mail and its contents > >> (with or without referred errors) shall therefore not attach any liability > >> on the originator or HCL or its affiliates. > >> Views or opinions, if any, presented in this email are solely those of the > >> author and may not necessarily reflect the > >> views or opinions of HCL or its affiliates. Any form of reproduction, > >> dissemination, copying, disclosure, modification, > >> distribution and / or publication of this message without the prior >> written > >> consent of authorized representative of > >> HCL is strictly prohibited. If you have received this email in error >> please > >> delete it and notify the sender immediately. > >> Before opening any email and/or attachments, please check them for viruses > >> and other defects. > >> > >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------- > > > > > > > > -- > > Nitin Pawar -- Nitin Pawar
