You need to show the driver class as well. Are you using textinputformat? Are you aware that this standard inputformat will take as a value the complete line (until newline separator), the key in that case is the bitoffset in the file and definitely not the number you assume it will be.
2015-06-26 18:13 GMT+02:00 xeonmailinglist-gmail <[email protected]> : > Hi, > > I have this map class that is accepting input files with a key as > LongWritable and a value of Text. > > The input file is in [1]. Here we can see that it contains a key as a Long > (I think) and bytes as value. > In [2], it is my map class. The goal of the map class is to read the input > data, and sent it out as it is. > I was expecting that the key was the index number, and the value was the > gibberish. But, I am having a strange error [3], as the key is getting > there a s Text, and not the index. > > Can anyone tell me why I get this error, and how I can solve it? > > [1] Input file > > xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat > /output1-1434970707/part-m-00000 > 15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 0 SEQ > "org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.BytesWritableI���#HA��u��BZ > � � n�� tX eX����zZi��n dFw�{>�H� \� ��Bi�'�g b�Y�a �*4�7;���uL_��gs�� �`�� * > 187 ���� tAj#�5*nX�b " ��>�˥k�c2�)�1z�1� ��6�l��� Q �be3'$ �G��!{� � > u����z@�t����v���r�gj8fq�P��.�m ��� y > ��n��!���_w�U8e�����v0���S���2c�_�ջ�R\x *�� > > > �܁���|x > > > U�3���l���qa q� )�N ��k�L��a�2�ն���k NL > 404 ��)i �g A�2 )L��W,���ܺ�V�w� �M7ڭCϣ����ZI�� be��$�f [�� � W� > 7��5�6.}�ѽ b� ����C��%|�h�lj�D } @ F�"�(K ��xh�'6 > > [2] My map class > > > /** Identity mapper set by the user. */ > public static class MyFullyIndentityMapperWebdataScan > extends Mapper<LongWritable, Text, LongWritable, Text>{ > > // private LongWritable word = new LongWritable(); > private LongWritable word = new LongWritable(); > private Text rvalue = new Text(); > > public void map(LongWritable key, Text value, Context context > ) throws IOException, InterruptedException { > > System.out.println("1: " + key.getClass().getName() + " " + > value.getClass().getName()); > System.out.println("2: " + context.getCurrentKey().getClass() + " > " + context.getCurrentValue().getClass()); > context.write(key, value); > } > } > > [3] Output of execution > > > Log Type: stdout > > Log Upload Time: 26-Jun-2015 11:58:53 > > Log Length: 138 > > 1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class > org.apache.hadoop.io.LongWritable class org.apache.hadoop.io.Text > > > > Log Type: syslog > > Log Upload Time: 26-Jun-2015 11:58:53 > > Log Length: 4367 > > Showing 4096 bytes of 4367 total. Click here > for the full log. > > operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [main] > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period > at 10 second(s). 2015-06-26 11:58:32,118 INFO [main] > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system > started 2015-06-26 11:58:32,128 INFO [main] > org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2015-06-26 > 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: > mapreduce.job, Service: job_1435332416394_0009, Ident: > (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d7657) > 2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChild: > Sleeping for 0ms before retrying again. Got null now. 2015-06-26 11:58:32,439 > INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir > for child: > /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1435332416394_0009 > 2015-06-26 11:58:32,611 INFO [main] > org.apache.hadoop.conf.Configuration.deprecation: > session.id is deprecated. Instead, use dfs.metrics.session-id 2015-06-26 > 11:58:33,038 INFO [main] org.apache.hadoop.mapred.Task: Using > ResourceCalculatorProcessTree : [ ] 2015-06-26 11:58:33,180 INFO [main] > org.apache.hadoop.mapred.MapTask: Processing split: > hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00000:0+17853 2015-06-26 > 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi > 26214396(104857584) 2015-06-26 11:58:33,252 INFO [main] > org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-06-26 > 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at > 83886080 2015-06-26 11:58:33,252 INFO [main] > org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 > 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart > = 26214396; length = 6553600 2015-06-26 11:58:33,255 INFO [main] > org.apache.hadoop.mapred.MapTask: Map output collector class = > org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2015-06-26 11: > 5 > 8:33,269 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map > output 2015-06-26 11:58:33,276 WARN [main] > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: Type mismatch in key from map: expected > org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712) > at > org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) > at > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) > at > org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:144) > at > org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:131) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at > org.apache. > h > adoop.mapred.MapTask.runNewMapper(MapTask.java:784) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > > > > -- > -- > Thanks, > >
