[
https://issues.apache.org/jira/browse/HIVE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13791589#comment-13791589
]
jeff little commented on HIVE-5245:
-----------------------------------
Task Logs: 'attempt_201308241420_3665_m_000000_0'
syslog logs
2013-10-10 23:25:14,771 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded
the native-hadoop library
2013-10-10 23:25:14,841 INFO org.apache.hadoop.mapred.TaskRunner: Creating
symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/distcache/-6415303628198910415_1127677245_586502988/namenode/tmp/hive-hadoop/hive_2013-10-10_23-21-25_380_3054038040192061682/-mr-10010/319df19a-bb7f-44d8-8bc5-e2cc1b034bb2
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/HIVE_PLAN319df19a-bb7f-44d8-8bc5-e2cc1b034bb2
2013-10-10 23:25:14,846 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/org
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/org
2013-10-10 23:25:14,847 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/javolution
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/javolution
2013-10-10 23:25:14,847 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/META-INF
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/META-INF
2013-10-10 23:25:14,848 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/.job.jar.crc
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/.job.jar.crc
2013-10-10 23:25:14,849 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/javaewah
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/javaewah
2013-10-10 23:25:14,850 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/com
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/com
2013-10-10 23:25:14,850 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/hive-exec-log4j.properties
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/hive-exec-log4j.properties
2013-10-10 23:25:14,851 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/javax
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/javax
2013-10-10 23:25:14,852 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/job.jar
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/job.jar
2013-10-10 23:25:15,089 INFO org.apache.hadoop.util.ProcessTree: setsid exited
with exit code 0
2013-10-10 23:25:15,091 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@28bb494b
2013-10-10 23:25:15,493 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader:
Loaded native gpl library
2013-10-10 23:25:15,495 INFO com.hadoop.compression.lzo.LzoCodec: Successfully
loaded & initialized native-lzo library [hadoop-lzo rev
6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
2013-10-10 23:25:15,503 INFO
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file
hdfs://namenode:9000/user/hive/warehouse/test.db/test_02/record_day=20130813/test_02.txt
2013-10-10 23:25:15,503 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
2013-10-10 23:25:15,511 INFO ExecMapper: maximum memory = 200998912
2013-10-10 23:25:15,511 INFO ExecMapper: conf classpath =
[file:/home/hadoop/package/hadoop-1.0.4/conf.cluster/,
file:/usr/lib/jvm/java-1.6.0-sun-1.6.0.30.x86_64/lib/tools.jar,
file:/home/hadoop/package/hadoop-1.0.4/build/classes/,
file:/home/hadoop/package/hadoop-1.0.4/build/test/classes/,
file:/home/hadoop/package/hadoop-1.0.4/,
file:/home/hadoop/package/hadoop-1.0.4/hadoop-core-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/asm-3.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/aspectjrt-1.6.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/aspectjtools-1.6.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-beanutils-1.7.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-beanutils-core-1.8.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-cli-1.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-codec-1.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-collections-3.2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-configuration-1.6.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-daemon-1.0.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-digester-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-el-1.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-httpclient-3.0.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-io-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-lang-2.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-logging-1.1.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-logging-api-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-math-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-net-1.4.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/core-3.1.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-capacity-scheduler-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-fairscheduler-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-lzo-0.4.15.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-thriftfs-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hsqldb-1.8.0.10.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jackson-core-asl-1.8.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jackson-mapper-asl-1.8.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jasper-compiler-5.5.12.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jasper-runtime-5.5.12.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jdeb-0.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-core-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-json-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-server-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jets3t-0.6.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jetty-6.1.26.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jetty-util-6.1.26.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsch-0.1.42.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/junit-4.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/kfs-0.2.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/log4j-1.2.15.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/mockito-all-1.8.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/oro-2.0.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/servlet-api-2.5-20081211.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/slf4j-api-1.4.3.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/xmlenc-0.52.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsp-2.1/jsp-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsp-2.1/jsp-api-2.1.jar,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/classes,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/distcache/-7362175280706972069_54005676_586503270/namenode/home/hadoop/hdfs/tmp/mapred/staging/hadoop/.staging/job_201308241420_3665/libjars/hive-contrib-0.11.0.jar/,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/]
2013-10-10 23:25:15,511 INFO ExecMapper: thread classpath =
[file:/home/hadoop/package/hadoop-1.0.4/conf.cluster/,
file:/usr/lib/jvm/java-1.6.0-sun-1.6.0.30.x86_64/lib/tools.jar,
file:/home/hadoop/package/hadoop-1.0.4/build/classes/,
file:/home/hadoop/package/hadoop-1.0.4/build/test/classes/,
file:/home/hadoop/package/hadoop-1.0.4/,
file:/home/hadoop/package/hadoop-1.0.4/hadoop-core-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/asm-3.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/aspectjrt-1.6.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/aspectjtools-1.6.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-beanutils-1.7.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-beanutils-core-1.8.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-cli-1.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-codec-1.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-collections-3.2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-configuration-1.6.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-daemon-1.0.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-digester-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-el-1.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-httpclient-3.0.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-io-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-lang-2.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-logging-1.1.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-logging-api-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-math-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-net-1.4.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/core-3.1.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-capacity-scheduler-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-fairscheduler-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-lzo-0.4.15.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-thriftfs-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hsqldb-1.8.0.10.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jackson-core-asl-1.8.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jackson-mapper-asl-1.8.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jasper-compiler-5.5.12.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jasper-runtime-5.5.12.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jdeb-0.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-core-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-json-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-server-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jets3t-0.6.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jetty-6.1.26.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jetty-util-6.1.26.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsch-0.1.42.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/junit-4.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/kfs-0.2.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/log4j-1.2.15.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/mockito-all-1.8.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/oro-2.0.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/servlet-api-2.5-20081211.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/slf4j-api-1.4.3.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/xmlenc-0.52.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsp-2.1/jsp-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsp-2.1/jsp-api-2.1.jar,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/classes,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/distcache/-7362175280706972069_54005676_586503270/namenode/home/hadoop/hdfs/tmp/mapred/staging/hadoop/.staging/job_201308241420_3665/libjars/hive-contrib-0.11.0.jar/,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000000_0/work/]
2013-10-10 23:25:15,548 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding
alias b to work list for file
hdfs://namenode:9000/user/hive/warehouse/test.db/test_02/record_day=20130812
2013-10-10 23:25:15,549 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding
alias b to work list for file
hdfs://namenode:9000/user/hive/warehouse/test.db/test_02/record_day=20130813
2013-10-10 23:25:15,551 INFO org.apache.hadoop.hive.ql.exec.MapOperator: dump
TS struct<id:int,course:string,score:string,record_day:string>
2013-10-10 23:25:15,551 INFO ExecMapper:
<MAP>Id =5
<Children>
<TS>Id =0
<Children>
<MAPJOIN>Id =1
<Children>
<SEL>Id =2
<Children>
<FS>Id =3
<Parent>Id = 2 null<\Parent>
<\FS>
<\Children>
<Parent>Id = 1 null<\Parent>
<\SEL>
<\Children>
<Parent>Id = 4
<HASHTABLEDUMMY>Id =4
<Children>null
<\Children>
<\HASHTABLEDUMMY>Id = 0 null<\Parent>
<\MAPJOIN>
<\Children>
<Parent>Id = 5 null<\Parent>
<\TS>
<\Children>
<\MAP>
2013-10-10 23:25:15,552 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Initializing Self 5 MAP
2013-10-10 23:25:15,552 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
Initializing Self 0 TS
2013-10-10 23:25:15,552 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
Operator 0 TS initialized
2013-10-10 23:25:15,552 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
Initializing children of 0 TS
2013-10-10 23:25:15,552 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Initializing child 1 MAPJOIN
2013-10-10 23:25:15,552 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
Initialization Done 0 TS
2013-10-10 23:25:15,552 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Initialization Done 5 MAP
2013-10-10 23:25:15,552 INFO ExecMapper: Initializing dummy operator
2013-10-10 23:25:15,552 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: Initializing Self 4
HASHTABLEDUMMY
2013-10-10 23:25:15,564 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: Operator 4
HASHTABLEDUMMY initialized
2013-10-10 23:25:15,565 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: Initializing children of
4 HASHTABLEDUMMY
2013-10-10 23:25:15,565 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Initializing child 1 MAPJOIN
2013-10-10 23:25:15,565 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Initializing Self 1 MAPJOIN
2013-10-10 23:25:15,575 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
JOIN struct<_col0:int,_col1:string,_col2:string,_col3:string> totalsz = 4
2013-10-10 23:25:15,576 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Operator 1 MAPJOIN initialized
2013-10-10 23:25:15,576 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Initializing children of 1 MAPJOIN
2013-10-10 23:25:15,576 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initializing child 2 SEL
2013-10-10 23:25:15,576 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initializing Self 2 SEL
2013-10-10 23:25:15,576 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
SELECT struct<_col0:int,_col1:string,_col2:string,_col3:string>
2013-10-10 23:25:15,576 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Operator 2 SEL initialized
2013-10-10 23:25:15,576 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initializing children of 2 SEL
2013-10-10 23:25:15,576 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing child 3 FS
2013-10-10 23:25:15,576 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing Self 3 FS
2013-10-10 23:25:15,580 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Operator 3 FS initialized
2013-10-10 23:25:15,580 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initialization Done 3 FS
2013-10-10 23:25:15,580 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initialization Done 2 SEL
2013-10-10 23:25:15,584 INFO
org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper: maximum memory:
200998912
2013-10-10 23:25:15,584 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Initialization Done 1 MAPJOIN
2013-10-10 23:25:15,584 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: Initialization Done 4
HASHTABLEDUMMY
2013-10-10 23:25:15,588 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Processing alias b for file
hdfs://namenode:9000/user/hive/warehouse/test.db/test_02/record_day=20130813
2013-10-10 23:25:15,589 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
******* Load from HashTable File: input :
hdfs://namenode:9000/user/hive/warehouse/test.db/test_02/record_day=20130813/test_02.txt
2013-10-10 23:25:15,590 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Load back 1 hashtable file from tmp file
uri:/home/hadoop/hdfs/mapred/local/taskTracker/distcache/-8026335471844171088_-517161708_586502978/namenode/tmp/hive-hadoop/hive_2013-10-10_23-21-25_380_3054038040192061682/-mr-10006/HashTable-Stage-6/Stage-6.tar.gz/MapJoin-mapfile480--.hashtable
2013-10-10 23:25:15,605 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 5
forwarding 1 rows
2013-10-10 23:25:15,605 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
0 forwarding 1 rows
2013-10-10 23:25:15,606 WARN org.apache.hadoop.hive.serde2.lazy.LazyStruct:
Extra bytes detected at the end of the row! Ignoring similar problems.
2013-10-10 23:25:15,606 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: 1
forwarding 1 rows
2013-10-10 23:25:15,606 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
forwarding 1 rows
2013-10-10 23:25:15,606 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Final Path: FS
hdfs://namenode:9000/tmp/hive-hadoop/hive_2013-10-10_23-21-25_380_3054038040192061682/_tmp.-ext-10001/000000_0
2013-10-10 23:25:15,607 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Writing to temp file: FS
hdfs://namenode:9000/tmp/hive-hadoop/hive_2013-10-10_23-21-25_380_3054038040192061682/_task_tmp.-ext-10001/_tmp.000000_0
2013-10-10 23:25:15,607 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
New Final Path: FS
hdfs://namenode:9000/tmp/hive-hadoop/hive_2013-10-10_23-21-25_380_3054038040192061682/_tmp.-ext-10001/000000_0
2013-10-10 23:25:15,657 INFO ExecMapper: ExecMapper: processing 1 rows: used
memory = 31440456
2013-10-10 23:25:15,657 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: 1
forwarding 10 rows
2013-10-10 23:25:15,657 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
forwarding 10 rows
2013-10-10 23:25:15,658 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 5
finished. closing...
2013-10-10 23:25:15,658 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 5
forwarded 8 rows
2013-10-10 23:25:15,658 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
DESERIALIZE_ERRORS:0
2013-10-10 23:25:15,658 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
0 finished. closing...
2013-10-10 23:25:15,658 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
0 forwarded 8 rows
2013-10-10 23:25:15,658 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
0 Close done
2013-10-10 23:25:15,658 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 5
Close done
2013-10-10 23:25:15,658 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: 4 finished. closing...
2013-10-10 23:25:15,658 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: 4 forwarded 0 rows
2013-10-10 23:25:15,658 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: 1
finished. closing...
2013-10-10 23:25:15,658 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: 1
forwarded 12 rows
2013-10-10 23:25:15,662 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
finished. closing...
2013-10-10 23:25:15,662 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
forwarded 12 rows
2013-10-10 23:25:15,662 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: 3
finished. closing...
2013-10-10 23:25:15,662 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: 3
forwarded 0 rows
2013-10-10 23:25:15,719 ERROR
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher: Error during
instantiating JDBC driver org.apache.derby.jdbc.EmbeddedDriver.
java.lang.ClassNotFoundException: org.apache.derby.jdbc.EmbeddedDriver
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.connect(JDBCStatsPublisher.java:68)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:963)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:872)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:201)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-10-10 23:25:15,720 ERROR org.apache.hadoop.hive.ql.exec.FileSinkOperator:
StatsPublishing error: cannot connect to database
2013-10-10 23:25:15,720 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
TABLE_ID_1_ROWCOUNT:12
2013-10-10 23:25:15,720 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
Close done
2013-10-10 23:25:15,720 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: 1
Close done
2013-10-10 23:25:15,720 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: 4 Close done
2013-10-10 23:25:15,720 INFO ExecMapper: ExecMapper: processed 8 rows: used
memory = 34940376
2013-10-10 23:25:15,724 INFO org.apache.hadoop.mapred.Task:
Task:attempt_201308241420_3665_m_000000_0 is done. And is in the process of
commiting
2013-10-10 23:25:18,083 INFO org.apache.hadoop.mapred.Task: Task
'attempt_201308241420_3665_m_000000_0' done.
2013-10-10 23:25:18,098 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-10-10 23:25:18,115 INFO org.apache.hadoop.io.nativeio.NativeIO:
Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2013-10-10 23:25:18,115 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
UserName hadoop for UID 500 from the native implementation
Task Logs: 'attempt_201308241420_3665_m_000001_0'
2013-10-10 23:25:14,771 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded
the native-hadoop library
2013-10-10 23:25:14,841 INFO org.apache.hadoop.mapred.TaskRunner: Creating
symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/distcache/-6415303628198910415_1127677245_586502988/namenode/tmp/hive-hadoop/hive_2013-10-10_23-21-25_380_3054038040192061682/-mr-10010/319df19a-bb7f-44d8-8bc5-e2cc1b034bb2
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/HIVE_PLAN319df19a-bb7f-44d8-8bc5-e2cc1b034bb2
2013-10-10 23:25:14,845 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/org
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/org
2013-10-10 23:25:14,846 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/javolution
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/javolution
2013-10-10 23:25:14,847 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/META-INF
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/META-INF
2013-10-10 23:25:14,848 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/.job.jar.crc
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/.job.jar.crc
2013-10-10 23:25:14,849 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/javaewah
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/javaewah
2013-10-10 23:25:14,849 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/com
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/com
2013-10-10 23:25:14,850 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/hive-exec-log4j.properties
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/hive-exec-log4j.properties
2013-10-10 23:25:14,851 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/javax
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/javax
2013-10-10 23:25:14,852 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink:
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/job.jar
<-
/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/job.jar
2013-10-10 23:25:15,089 INFO org.apache.hadoop.util.ProcessTree: setsid exited
with exit code 0
2013-10-10 23:25:15,092 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5d2412e7
2013-10-10 23:25:15,493 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader:
Loaded native gpl library
2013-10-10 23:25:15,495 INFO com.hadoop.compression.lzo.LzoCodec: Successfully
loaded & initialized native-lzo library [hadoop-lzo rev
6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
2013-10-10 23:25:15,503 INFO
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file
hdfs://namenode:9000/user/hive/warehouse/test.db/test_02/record_day=20130812/test_02.txt
2013-10-10 23:25:15,503 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
2013-10-10 23:25:15,510 INFO ExecMapper: maximum memory = 200998912
2013-10-10 23:25:15,510 INFO ExecMapper: conf classpath =
[file:/home/hadoop/package/hadoop-1.0.4/conf.cluster/,
file:/usr/lib/jvm/java-1.6.0-sun-1.6.0.30.x86_64/lib/tools.jar,
file:/home/hadoop/package/hadoop-1.0.4/build/classes/,
file:/home/hadoop/package/hadoop-1.0.4/build/test/classes/,
file:/home/hadoop/package/hadoop-1.0.4/,
file:/home/hadoop/package/hadoop-1.0.4/hadoop-core-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/asm-3.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/aspectjrt-1.6.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/aspectjtools-1.6.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-beanutils-1.7.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-beanutils-core-1.8.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-cli-1.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-codec-1.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-collections-3.2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-configuration-1.6.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-daemon-1.0.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-digester-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-el-1.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-httpclient-3.0.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-io-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-lang-2.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-logging-1.1.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-logging-api-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-math-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-net-1.4.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/core-3.1.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-capacity-scheduler-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-fairscheduler-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-lzo-0.4.15.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-thriftfs-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hsqldb-1.8.0.10.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jackson-core-asl-1.8.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jackson-mapper-asl-1.8.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jasper-compiler-5.5.12.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jasper-runtime-5.5.12.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jdeb-0.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-core-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-json-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-server-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jets3t-0.6.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jetty-6.1.26.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jetty-util-6.1.26.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsch-0.1.42.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/junit-4.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/kfs-0.2.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/log4j-1.2.15.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/mockito-all-1.8.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/oro-2.0.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/servlet-api-2.5-20081211.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/slf4j-api-1.4.3.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/xmlenc-0.52.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsp-2.1/jsp-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsp-2.1/jsp-api-2.1.jar,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/classes,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/distcache/-7362175280706972069_54005676_586503270/namenode/home/hadoop/hdfs/tmp/mapred/staging/hadoop/.staging/job_201308241420_3665/libjars/hive-contrib-0.11.0.jar/,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/]
2013-10-10 23:25:15,511 INFO ExecMapper: thread classpath =
[file:/home/hadoop/package/hadoop-1.0.4/conf.cluster/,
file:/usr/lib/jvm/java-1.6.0-sun-1.6.0.30.x86_64/lib/tools.jar,
file:/home/hadoop/package/hadoop-1.0.4/build/classes/,
file:/home/hadoop/package/hadoop-1.0.4/build/test/classes/,
file:/home/hadoop/package/hadoop-1.0.4/,
file:/home/hadoop/package/hadoop-1.0.4/hadoop-core-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/asm-3.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/aspectjrt-1.6.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/aspectjtools-1.6.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-beanutils-1.7.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-beanutils-core-1.8.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-cli-1.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-codec-1.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-collections-3.2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-configuration-1.6.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-daemon-1.0.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-digester-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-el-1.0.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-httpclient-3.0.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-io-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-lang-2.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-logging-1.1.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-logging-api-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-math-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/commons-net-1.4.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/core-3.1.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-capacity-scheduler-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-fairscheduler-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-lzo-0.4.15.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hadoop-thriftfs-1.0.4.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/hsqldb-1.8.0.10.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jackson-core-asl-1.8.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jackson-mapper-asl-1.8.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jasper-compiler-5.5.12.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jasper-runtime-5.5.12.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jdeb-0.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-core-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-json-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jersey-server-1.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jets3t-0.6.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jetty-6.1.26.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jetty-util-6.1.26.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsch-0.1.42.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/junit-4.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/kfs-0.2.2.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/log4j-1.2.15.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/mockito-all-1.8.5.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/oro-2.0.8.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/servlet-api-2.5-20081211.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/slf4j-api-1.4.3.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/xmlenc-0.52.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsp-2.1/jsp-2.1.jar,
file:/home/hadoop/package/hadoop-1.0.4/lib/jsp-2.1/jsp-api-2.1.jar,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/classes,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/jars/,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/distcache/-7362175280706972069_54005676_586503270/namenode/home/hadoop/hdfs/tmp/mapred/staging/hadoop/.staging/job_201308241420_3665/libjars/hive-contrib-0.11.0.jar/,
file:/home/hadoop/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201308241420_3665/attempt_201308241420_3665_m_000001_0/work/]
2013-10-10 23:25:15,547 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding
alias b to work list for file
hdfs://namenode:9000/user/hive/warehouse/test.db/test_02/record_day=20130812
2013-10-10 23:25:15,549 INFO org.apache.hadoop.hive.ql.exec.MapOperator: dump
TS struct<id:int,course:string,score:string,record_day:string>
2013-10-10 23:25:15,550 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding
alias b to work list for file
hdfs://namenode:9000/user/hive/warehouse/test.db/test_02/record_day=20130813
2013-10-10 23:25:15,550 INFO ExecMapper:
<MAP>Id =5
<Children>
<TS>Id =0
<Children>
<MAPJOIN>Id =1
<Children>
<SEL>Id =2
<Children>
<FS>Id =3
<Parent>Id = 2 null<\Parent>
<\FS>
<\Children>
<Parent>Id = 1 null<\Parent>
<\SEL>
<\Children>
<Parent>Id = 4
<HASHTABLEDUMMY>Id =4
<Children>null
<\Children>
<\HASHTABLEDUMMY>Id = 0 null<\Parent>
<\MAPJOIN>
<\Children>
<Parent>Id = 5 null<\Parent>
<\TS>
<\Children>
<\MAP>
2013-10-10 23:25:15,551 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Initializing Self 5 MAP
2013-10-10 23:25:15,551 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
Initializing Self 0 TS
2013-10-10 23:25:15,551 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
Operator 0 TS initialized
2013-10-10 23:25:15,551 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
Initializing children of 0 TS
2013-10-10 23:25:15,551 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Initializing child 1 MAPJOIN
2013-10-10 23:25:15,551 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
Initialization Done 0 TS
2013-10-10 23:25:15,551 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Initialization Done 5 MAP
2013-10-10 23:25:15,551 INFO ExecMapper: Initializing dummy operator
2013-10-10 23:25:15,551 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: Initializing Self 4
HASHTABLEDUMMY
2013-10-10 23:25:15,564 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: Operator 4
HASHTABLEDUMMY initialized
2013-10-10 23:25:15,564 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: Initializing children of
4 HASHTABLEDUMMY
2013-10-10 23:25:15,564 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Initializing child 1 MAPJOIN
2013-10-10 23:25:15,564 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Initializing Self 1 MAPJOIN
2013-10-10 23:25:15,575 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
JOIN struct<_col0:int,_col1:string,_col2:string,_col3:string> totalsz = 4
2013-10-10 23:25:15,575 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Operator 1 MAPJOIN initialized
2013-10-10 23:25:15,575 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Initializing children of 1 MAPJOIN
2013-10-10 23:25:15,575 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initializing child 2 SEL
2013-10-10 23:25:15,575 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initializing Self 2 SEL
2013-10-10 23:25:15,575 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
SELECT struct<_col0:int,_col1:string,_col2:string,_col3:string>
2013-10-10 23:25:15,575 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Operator 2 SEL initialized
2013-10-10 23:25:15,575 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initializing children of 2 SEL
2013-10-10 23:25:15,575 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing child 3 FS
2013-10-10 23:25:15,576 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing Self 3 FS
2013-10-10 23:25:15,580 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Operator 3 FS initialized
2013-10-10 23:25:15,580 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initialization Done 3 FS
2013-10-10 23:25:15,580 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initialization Done 2 SEL
2013-10-10 23:25:15,583 INFO
org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper: maximum memory:
200998912
2013-10-10 23:25:15,583 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Initialization Done 1 MAPJOIN
2013-10-10 23:25:15,583 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: Initialization Done 4
HASHTABLEDUMMY
2013-10-10 23:25:15,587 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Processing alias b for file
hdfs://namenode:9000/user/hive/warehouse/test.db/test_02/record_day=20130812
2013-10-10 23:25:15,588 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
******* Load from HashTable File: input :
hdfs://namenode:9000/user/hive/warehouse/test.db/test_02/record_day=20130812/test_02.txt
2013-10-10 23:25:15,589 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator:
Load back 1 hashtable file from tmp file
uri:/home/hadoop/hdfs/mapred/local/taskTracker/distcache/-8026335471844171088_-517161708_586502978/namenode/tmp/hive-hadoop/hive_2013-10-10_23-21-25_380_3054038040192061682/-mr-10006/HashTable-Stage-6/Stage-6.tar.gz/MapJoin-mapfile480--.hashtable
2013-10-10 23:25:15,605 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 5
forwarding 1 rows
2013-10-10 23:25:15,605 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
0 forwarding 1 rows
2013-10-10 23:25:15,605 WARN org.apache.hadoop.hive.serde2.lazy.LazyStruct:
Extra bytes detected at the end of the row! Ignoring similar problems.
2013-10-10 23:25:15,605 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: 1
forwarding 1 rows
2013-10-10 23:25:15,605 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
forwarding 1 rows
2013-10-10 23:25:15,606 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Final Path: FS
hdfs://namenode:9000/tmp/hive-hadoop/hive_2013-10-10_23-21-25_380_3054038040192061682/_tmp.-ext-10001/000001_0
2013-10-10 23:25:15,606 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Writing to temp file: FS
hdfs://namenode:9000/tmp/hive-hadoop/hive_2013-10-10_23-21-25_380_3054038040192061682/_task_tmp.-ext-10001/_tmp.000001_0
2013-10-10 23:25:15,606 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
New Final Path: FS
hdfs://namenode:9000/tmp/hive-hadoop/hive_2013-10-10_23-21-25_380_3054038040192061682/_tmp.-ext-10001/000001_0
2013-10-10 23:25:15,649 INFO ExecMapper: ExecMapper: processing 1 rows: used
memory = 31506032
2013-10-10 23:25:15,649 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: 1
forwarding 10 rows
2013-10-10 23:25:15,649 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
forwarding 10 rows
2013-10-10 23:25:15,649 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 5
finished. closing...
2013-10-10 23:25:15,650 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 5
forwarded 8 rows
2013-10-10 23:25:15,650 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
DESERIALIZE_ERRORS:0
2013-10-10 23:25:15,650 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
0 finished. closing...
2013-10-10 23:25:15,650 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
0 forwarded 8 rows
2013-10-10 23:25:15,650 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator:
0 Close done
2013-10-10 23:25:15,650 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 5
Close done
2013-10-10 23:25:15,650 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: 4 finished. closing...
2013-10-10 23:25:15,650 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: 4 forwarded 0 rows
2013-10-10 23:25:15,650 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: 1
finished. closing...
2013-10-10 23:25:15,650 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: 1
forwarded 12 rows
2013-10-10 23:25:15,653 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
finished. closing...
2013-10-10 23:25:15,654 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
forwarded 12 rows
2013-10-10 23:25:15,654 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: 3
finished. closing...
2013-10-10 23:25:15,654 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: 3
forwarded 0 rows
2013-10-10 23:25:15,713 ERROR
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher: Error during
instantiating JDBC driver org.apache.derby.jdbc.EmbeddedDriver.
java.lang.ClassNotFoundException: org.apache.derby.jdbc.EmbeddedDriver
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at
org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.connect(JDBCStatsPublisher.java:68)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:963)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:872)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:201)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-10-10 23:25:15,714 ERROR org.apache.hadoop.hive.ql.exec.FileSinkOperator:
StatsPublishing error: cannot connect to database
2013-10-10 23:25:15,714 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
TABLE_ID_1_ROWCOUNT:12
2013-10-10 23:25:15,714 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2
Close done
2013-10-10 23:25:15,714 INFO org.apache.hadoop.hive.ql.exec.MapJoinOperator: 1
Close done
2013-10-10 23:25:15,714 INFO
org.apache.hadoop.hive.ql.exec.HashTableDummyOperator: 4 Close done
2013-10-10 23:25:15,714 INFO ExecMapper: ExecMapper: processed 8 rows: used
memory = 35005952
2013-10-10 23:25:15,718 INFO org.apache.hadoop.mapred.Task:
Task:attempt_201308241420_3665_m_000001_0 is done. And is in the process of
commiting
2013-10-10 23:25:18,085 INFO org.apache.hadoop.mapred.Task: Task
'attempt_201308241420_3665_m_000001_0' done.
2013-10-10 23:25:18,100 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-10-10 23:25:18,117 INFO org.apache.hadoop.io.nativeio.NativeIO:
Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2013-10-10 23:25:18,117 INFO org.apache.hadoop.io.nativeio.NativeIO: Got
UserName hadoop for UID 500 from the native implementation
Hi,Yin Huai, this are the logs of 2 tasks for this CTAS. Otherwise, the hive
version is 0.11!
> hive create table as select(CTAS) can not work(not support) with join on
> operator
> ---------------------------------------------------------------------------------
>
> Key: HIVE-5245
> URL: https://issues.apache.org/jira/browse/HIVE-5245
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 0.11.0
> Reporter: jeff little
> Labels: CTAS, hive
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> hello everyone, recently i came across one hive problem as below:
> hive (test)> create table test_09 as
> > select a.* from test_01 a
> > join test_02 b
> > on (a.id=b.id);
> Automatically selecting local only mode for query
> Total MapReduce jobs = 2
> setting HADOOP_USER_NAME hadoop
> 13/09/09 17:22:36 WARN conf.Configuration:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10008/jobconf.xml:a
> attempt to override final parameter: mapred.system.dir; Ignoring.
> 13/09/09 17:22:36 WARN conf.Configuration:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10008/jobconf.xml:a
> attempt to override final parameter: mapred.local.dir; Ignoring.
> Execution log at: /tmp/hadoop/.log
> 2013-09-09 05:22:36 Starting to launch local task to process map join;
> maximum memory = 932118528
> 2013-09-09 05:22:37 Processing rows: 4 Hashtable size: 4
> Memory usage: 113068056 rate: 0.121
> 2013-09-09 05:22:37 Dump the hashtable into file:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10005/HashTable-Stage-6/MapJoin-mapfile90--.hashtable
> 2013-09-09 05:22:37 Upload 1 File to:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10005/HashTable-Stage-6/MapJoin-mapfile90--.hashtable
> File size: 788
> 2013-09-09 05:22:37 End of local task; Time Taken: 0.444 sec.
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Launching Job 1 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> 13/09/09 17:22:38 WARN conf.Configuration:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10009/jobconf.xml:a
> attempt to override final parameter: mapred.system.dir; Ignoring.
> 13/09/09 17:22:38 WARN conf.Configuration:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10009/jobconf.xml:a
> attempt to override final parameter: mapred.local.dir; Ignoring.
> Execution log at: /tmp/hadoop/.log
> Job running in-process (local Hadoop)
> Hadoop job information for null: number of mappers: 0; number of reducers: 0
> 2013-09-09 17:22:41,807 null map = 0%, reduce = 0%
> 2013-09-09 17:22:44,814 null map = 100%, reduce = 0%
> Ended Job = job_local_0001
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Stage-7 is filtered out by condition resolver.
> OK
> Time taken: 13.138 seconds
> hive (test)> select * from test_09;
> FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'test_09'
> hive (test)>
> Problem:
> I can't get the created table, namely this CTAS is nonavailable, and this
> table is not created by this hql sentence at all.who can explain for
> me.Thanks.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
