I wrote a mapreduce program which need to use proto buf for parsing.the program
is list below:
import com.baidu.mirror.CommonCollectorOuterClass.*;
import com.baidu.mirror.*;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapred.lib.*;
import org.apache.hadoop.util.*;
public class PvStat {
public static class Map extends MapReduceBase implements
Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text ip = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text,
IntWritable> output, Reporter reporter) throws IOException {
System.out.println("value is : " + value);
byte[] rawData = value.getBytes();
CommonCollector cc = null;
try{
cc = CommonCollector.parseFrom(rawData);
}catch(Exception e){
e.printStackTrace();
}
ip.set(cc.getMachineIp());
output.collect(ip,one);
}
}
public static class Reduce extends MapReduceBase implements Reducer<Text,
IntWritable, Text, IntWritable> {
private IntWritable totalCount = new IntWritable();
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output, Reporter reporter) throws
IOException{
int count = 0;
while (values.hasNext()) {
count += values.next().get();
}
totalCount.set(count);
output.collect(key, totalCount);
}
}
public static void main(String[] args) throws Exception {
// only headle one file once, Aug 7,2015
String inputPath = args[0];
String outputPath = args[1];
JobConf conf = new JobConf(PvStat.class);
conf.setJobName("PvStat");
conf.setNumReduceTasks(1);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(Map.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(NLineInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.setInputPaths(conf, new Path(inputPath));
FileOutputFormat.setOutputPath(conf, new Path(outputPath));
JobClient.runJob(conf);
}
}
BTW,my proto buf version is 2.6.1.And the file what I need to parse is put by
myself also deal with same version.But I don't know which version hadoop use(I
think it shouldn't effect the result).
the detail of error like :
java.lang.VerifyError: class
com.baidu.mirror.CommonCollectorOuterClass$CommonCollector overrides final
method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at PvStat$Map.map(PvStat.java:23)
at PvStat$Map.map(PvStat.java:14)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:63)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:381)
at org.apache.hadoop.mapred.Child.main(Child.java:212)
I couldn't find solution and have to ask for help.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
