I'm not using Yarn, But I'll give that a shot.. Is hive 11 dependent on Yarn?
JC On Aug 23, 2013, at 1:19 PM, Justin Workman <justinjwork...@gmail.com> wrote: > This looks similar to the problem you get when you don't set > yarn.resourcemanager.address. > > Sent from my iPhone > > On Aug 23, 2013, at 11:12 AM, Jim Colestock <j...@ramblingredneck.com> wrote: > >> Hello Justin, >> >> Thanks so much for the write up.. I'm compiling now.. I did find one error >> in your instructions: >> >> patch -p0 > HIVE-4619.D10971.1.path >> >> I believe it should be: >> >> patch -p0 < HIVE-4619.D10971.1.patch >> >> >> But alas, I still doesn't work for me. I managed to get it to compile after >> using maven to install a zookeeper dependency.. But when I try to do >> anything in hive that requires an M/R job, I get the following: >> >> 2013-08-23 13:07:07,865 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 >> 2013-08-23 13:07:07,867 WARN org.apache.hadoop.mapred.Child: Error running >> child >> java.lang.RuntimeException: java.lang.NullPointerException >> at >> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230) >> at >> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255) >> at >> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381) >> at >> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374) >> at >> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540) >> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >> at org.apache.hadoop.mapred.Child.main(Child.java:262) >> Caused by: java.lang.NullPointerException >> at >> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215) >> ... 11 more >> 2013-08-23 13:07:07,871 INFO org.apache.hadoop.mapred.Task: Runnning cleanup >> for the task >> >> Any thoughts? >> >> Thanks again, >> >> JC >> >> >> >> >> >> >> >> On Aug 22, 2013, at 11:15 PM, Justin Workman <justinjwork...@gmail.com> >> wrote: >> >>> For anyone interested, I have finished my blog on getting Hive-0.11.0 >>> working with CDH4.3.0. It is a hack and can be made better but works like a >>> charm. >>> >>> http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/ >>> >>> >>> >>> On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <j...@ramblingredneck.com> >>> wrote: >>> Justin, >>> >>> That would be much appreciated! >>> >>> Thanks Again, >>> >>> JC >>> >>> On Aug 22, 2013, at 9:18 AM, Justin Workman <justinjwork...@gmail.com> >>> wrote: >>> >>>> I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing >>>> up a blog today explaining what was needed to get it working. I will reply >>>> with the link later today. >>>> >>>> Justin >>>> >>>> Sent from my iPhone >>>> >>>> On Aug 22, 2013, at 6:17 AM, Jim Colestock <j...@ramblingredneck.com> >>>> wrote: >>>> >>>>> Hello All, >>>>> >>>>> Sorry for not replying to the thread, but I never received any of your >>>>> responses.. Hopefully this info will help: >>>>> >>>>> @kulkarni.swarnim >>>>> >>>>> Here is the error I'm getting on task tracker for the mapper: >>>>> >>>>> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >>>>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 >>>>> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error >>>>> running child >>>>> java.lang.RuntimeException: java.lang.NullPointerException >>>>> at >>>>> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230) >>>>> at >>>>> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255) >>>>> at >>>>> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381) >>>>> at >>>>> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374) >>>>> at >>>>> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540) >>>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394) >>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) >>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at javax.security.auth.Subject.doAs(Subject.java:396) >>>>> at >>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >>>>> at org.apache.hadoop.mapred.Child.main(Child.java:262) >>>>> Caused by: java.lang.NullPointerException >>>>> at >>>>> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215) >>>>> ... 11 more >>>>> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning >>>>> cleanup for the task >>>>> >>>>> @Srinivas Surasani >>>>> Hopefully the above helps.. >>>>> >>>>> @Lars Francke >>>>> Thanks, I'll take a peek at those 2 patches.. But I've just recently >>>>> pulled down fresh code and recompiled everything and I'm still getting >>>>> the above error. >>>>> >>>>> And for reference, below is my original email to the list.. Thanks all >>>>> for you help! >>>>> >>>>> >>>>> JC >>>>> >>>>> >>>>> >>>>> Hello All, >>>>> >>>>> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3? >>>>> >>>>> I've been able to get hive to connect to my metadb (which is in >>>>> Postgres). Verified by doing a show tables.. I can run explain and >>>>> describes on tables, but when I try to run anything that fires off an M/R >>>>> job, I get the following error: >>>>> >>>>> hive>select count(*) from tableA; >>>>> Total MapReduce jobs = 1 >>>>> Launching Job 1 out of 1 >>>>> Number of reduce tasks determined at compile time: 1 >>>>> In order to change the average load for a reducer (in bytes): >>>>> set hive.exec.reducers.bytes.per.reducer=<number> >>>>> In order to limit the maximum number of reducers: >>>>> set hive.exec.reducers.max=<number> >>>>> In order to set a constant number of reducers: >>>>> set mapred.reduce.tasks=<number> >>>>> Starting Job = job_201307112247_13816, Tracking URL = >>>>> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816 >>>>> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill >>>>> job_201307112247_13816 >>>>> Hadoop job information for Stage-1: number of mappers: 1; number of >>>>> reducers: 1 >>>>> 2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0% >>>>> 2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100% >>>>> Ended Job = job_201307112247_13816 with errors >>>>> Error during job, obtaining debugging information... >>>>> Job Tracking URL: >>>>> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816 >>>>> Examining task ID: task_201307112247_13816_m_000002 (and more) from job >>>>> job_201307112247_13816 >>>>> Exception in thread "Thread-19" java.lang.NoClassDefFoundError: >>>>> org/apache/hadoop/mapreduce/util/HostUtil >>>>> at >>>>> org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61) >>>>> at >>>>> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186) >>>>> at >>>>> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> Caused by: java.lang.ClassNotFoundException: >>>>> org.apache.hadoop.mapreduce.util.HostUtil >>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >>>>> ... 4 more >>>>> FAILED: Execution Error, return code 2 from >>>>> org.apache.hadoop.hive.ql.exec.MapRedTask >>>>> MapReduce Jobs Launched: >>>>> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL >>>>> Total MapReduce CPU Time Spent: 0 msec >>>>> >>>>> I'm using my configs from hive 0.10, which works with no issues and this >>>>> was pretty much a drop in replacement on the machine that hadoop 0.10 was >>>>> running on.. >>>>> >>>>> I've done a bunch of googling around and have found a bunch of other >>>>> folks that have have had the same issue, but no solid answers.. >>>>> >>>>> Thanks in advance for any help.. >>>>> >>>>> JC >>>>> >>>>> >>> >>> >>
