Can you help reduce the test case a bit. Does it happpen if you just run the subquery as well?
Ashish ________________________________ From: Eva Tse [mailto:[email protected]] Sent: Thursday, August 27, 2009 7:08 PM To: [email protected] Subject: Assertions in reducer when running hive query We are running a Hive query like the following: from (select sum (c.countrow) as countrow, c.c_id as c_id, f.dt_id as dt_id, f.xid as xid, f.esn as esn, f.hour as hour from session_facts f join session_cds c on (f.xid = c.xid and f.esn = c.esn) where c.dateint = 20090825 and f.dateint = 20090825 and c.hour = f.hour and c.c_id is not null group by c.c_id, f.dt_id, f.xid, f.esn, f.hour) x insert overwrite table temp_a select count (1), x.c_id, x.dt_id, x.hour where x.countrow > 0 group by x.c_id, x.dt_id, x.hour insert overwrite table temp_b select count (1), x.c_id, x.dt_id, x.hour group by x.c_id, x.dt_id, x.hour We are getting the following exceptions in our reducers after we turn on the flag -enableassertions in mapred.child.java.opts. If we don't turn on the flag, it is fine. Have anyone seen this before and do we know why? We are using Hive truck version # 795559 with Hadoop 0.20. Thanks, Eva. 2009-08-27 18:20:39,782 INFO org.apache.hadoop.mapred.Merger: Merging 5 sorted segments 2009-08-27 18:20:39,794 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 5 segments left of total size: 264428308 bytes 2009-08-27 18:20:39,830 INFO ExecReducer: conf classpath = [file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/classes, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/attempt_200908212037_4391_r_000000_1/] 2009-08-27 18:20:39,830 INFO ExecReducer: thread classpath = [file:/etc/hadoop-0.20/conf.empty/, file:/usr/java/jdk1.6.0_07/lib/tools.jar, file:/usr/lib/hadoop-0.20/, file:/usr/lib/hadoop-0.20/hadoop-0.20.0~17-1.cloudera-core.jar, file:/usr/lib/hadoop-0.20/lib/commons-cli-2.0-SNAPSHOT.jar, file:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar, file:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar, file:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar, file:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar, file:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar, file:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar, file:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar, file:/usr/lib/hadoop-0.20/lib/hadoop-0.20.0~17-1.cloudera-fairscheduler.jar, file:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar, file:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar, file:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar, file:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar, file:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar, file:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar, file:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar, file:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar, file:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar, file:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar, file:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar, file:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar, file:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar, file:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar, file:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar, file:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/attempt_200908212037_4391_r_000000_1/work/, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/classes, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/attempt_200908212037_4391_r_000000_1/work/] 2009-08-27 18:20:40,697 INFO ExecReducer: <JOIN>Id =6 <Children> <FIL>Id =7 <Children> <SEL>Id =8 <Children> <GBY>Id =9 <Children> <FS>Id =10 <Parent>Id = 9 <\Parent> <\FS> <\Children> <Parent>Id = 8 <\Parent> <\GBY> <\Children> <Parent>Id = 7 <\Parent> <\SEL> <\Children> <Parent>Id = 6 <\Parent> <\FIL> <\Children> <\JOIN> 2009-08-27 18:20:40,698 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: Initializing Self 6 JOIN 2009-08-27 18:20:40,700 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator: COMMONJOIN struct<key:struct<joinkey0:string,joinkey1:string>,value:struct<_col0:string,_col1:string,_col2:int,_col24:string,_col25:string>,alias:tinyint> 2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator: JOIN struct<_col0:string,_col1:string,_col2:int,_col24:string,_col25:string,_col28:int,_col33:int,_col34:string,_col35:string> totalsz = 9 2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: Operator 6 JOIN initialized 2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: Initializing children of 6 JOIN 2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing child 7 FIL 2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing Self 7 FIL 2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Operator 7 FIL initialized 2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing children of 7 FIL 2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing child 8 SEL 2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing Self 8 SEL 2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT struct<_col0:string,_col1:string,_col2:int,_col24:string,_col25:string,_col28:int,_col33:int,_col34:string,_col35:string> 2009-08-27 18:20:40,717 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Operator 8 SEL initialized 2009-08-27 18:20:40,717 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing children of 8 SEL 2009-08-27 18:20:40,717 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator: Initializing child 9 GBY 2009-08-27 18:20:40,717 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator: Initializing Self 9 GBY 2009-08-27 18:20:40,733 WARN org.apache.hadoop.mapred.TaskTracker: Error running child java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:428) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) ... 5 more Caused by: java.lang.RuntimeException: Reduce operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:122) ... 10 more Caused by: java.lang.AssertionError at org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:140) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:286) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:323) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:308) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:58) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:286) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:323) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:308) at org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOperator.java:61) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:286) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:323) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:308) at org.apache.hadoop.hive.ql.exec.JoinOperator.initializeOp(JoinOperator.java:40) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:286) at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:115) ... 10 more 2009-08-27 18:20:40,737 INFO org.apache.hadoop.mapred.TaskRunner: Runnning cleanup for the task
