Hi,
I have this map class that is accepting input files with a key as
LongWritable and a value of Text.
The input file is in [1]. Here we can see that it contains a key as a
Long (I think) and bytes as value.
In [2], it is my map class. The goal of the map class is to read the
input data, and sent it out as it is.
I was expecting that the key was the index number, and the value was the
gibberish. But, I am having a strange error [3], as the key is getting
there a s Text, and not the index.
Can anyone tell me why I get this error, and how I can solve it?
[1] Input file
|xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat
/output1-1434970707/part-m-00000
15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
0
SEQ"org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.BytesWritableI���#HA��u��BZ��n��tX eX����zZi��ndFw�{>�H�\� ��Bi�'�gb�Y�a�*4�7;���uL_��gs���`��*
187
����tAj#�5*nX�b"��>�˥k�c2�)�1z�1���6�l���Q�be3'$�G��!{�� u����z@�t����v���r�gj8fq�P��.�m ���y��n��!���_w�U8e�����v0���S���2c�_�ջ�R\x*��
�܁���|x
U�3���l���qaq� )�N��k�L��a�2�ն���k NL
404 ��)i�g
A�2)L��W,���ܺ�V�w��M7ڭCϣ����ZI�� be��$�f[���W�7��5�6.}�ѽb�
����C��%|�h�lj�D}@F�"�(K��xh�'6
|
[2] My map class
|
/** Identity mapper set by the user. */
public static class MyFullyIndentityMapperWebdataScan
extends Mapper<LongWritable, Text, LongWritable, Text>{
// private LongWritable word = new LongWritable();
private LongWritable word = new LongWritable();
private Text rvalue = new Text();
public void map(LongWritable key, Text value, Context context
) throws IOException, InterruptedException {
System.out.println("1: " + key.getClass().getName() + " " +
value.getClass().getName());
System.out.println("2: " + context.getCurrentKey().getClass() + " "
+ context.getCurrentValue().getClass());
context.write(key, value);
}
}
|
[3] Output of execution
|
Log Type: stdout
Log Upload Time: 26-Jun-2015 11:58:53
Log Length: 138
1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class
org.apache.hadoop.io.LongWritable class org.apache.hadoop.io.Text
Log Type: syslog
Log Upload Time: 26-Jun-2015 11:58:53
Log Length: 4367
Showing 4096 bytes of 4367 total. Click here
for the full log.
operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at
10 second(s). 2015-06-26 11:58:32,118 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
started 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild:
Executing with tokens: 2015-06-26 11:58:32,128 INFO [main]
org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service:
job_1435332416394_0009, Ident:
(org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d7657)
2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChild:
Sleeping for 0ms before retrying again. Got null now. 2015-06-26 11:58:32,439
INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for
child:
/tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1435332416394_0009
2015-06-26 11:58:32,611 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id 2015-06-26 11:58:33,038 INFO [main]
org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2015-06-26 11:58:33,180 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split:
hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00000:0+17853 2015-06-26
11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi
26214396(104857584) 2015-06-26 11:58:33,252 INFO [main]
org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-06-26
11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at
83886080 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask:
bufstart = 0; bufvoid = 104857600 2015-06-26 11:58:33,252 INFO [main]
org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
2015-06-26 11:58:33,255 INFO [main] org.apache.hadoop.mapred.MapTask: Map
output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-06-26 11:58:33,269 INFO [main] org.apache.hadoop.mapred.MapTask: Starting
flush of map output 2015-06-26 11:58:33,276 WARN [main]
org.apache.hadoop.mapred.YarnChild: Exception running child :
java.io.IOException: Type mismatch in key from map: expected
org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:144)
at
org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:131)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
|
--
--
Thanks,
