I wrote a mapreduce program which need to use proto buf for parsing.the program
is list below:
import com.baidu.mirror.CommonCollectorOuterClass.*;
import com.baidu.mirror.*;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapred.lib.*;
import org.apache.hadoop.util.*;
public class PvStat {
public static class Map extends MapReduceBase implements
Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text ip = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text,
IntWritable> output, Reporter reporter) throws IOException {
System.out.println("value is : " + value);
byte[] rawData = value.getBytes();
CommonCollector cc = null;
try{
cc = CommonCollector.parseFrom(rawData);
}catch(Exception e){
e.printStackTrace();
}
ip.set(cc.getMachineIp());
output.collect(ip,one);
}
}
public static class Reduce extends MapReduceBase implements Reducer<Text,
IntWritable, Text, IntWritable> {
private IntWritable totalCount = new IntWritable();
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output, Reporter reporter) throws
IOException{
int count = 0;
while (values.hasNext()) {
count += values.next().get();
}
totalCount.set(count);
output.collect(key, totalCount);
}
}
public static void main(String[] args) throws Exception {
// only headle one file once, Aug 7,2015
String inputPath = args[0];
String outputPath = args[1];
JobConf conf = new JobConf(PvStat.class);
conf.setJobName("PvStat");
conf.setNumReduceTasks(1);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(Map.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(NLineInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.setInputPaths(conf, new Path(inputPath));
FileOutputFormat.setOutputPath(conf, new Path(outputPath));
JobClient.runJob(conf);
}
}
BTW,my proto buf version is 2.6.1.And the file what I need to parse is put by
myself also deal with same version.But I don't know which version hadoop use(I
think it shouldn't effect the result).
the detail of error like :
java.lang.VerifyError: class
com.baidu.mirror.CommonCollectorOuterClass$CommonCollector overrides final
method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at PvStat$Map.map(PvStat.java:23)
at PvStat$Map.map(PvStat.java:14)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:63)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:381)
at org.apache.hadoop.mapred.Child.main(Child.java:212)
I couldn't find solution and have to ask for help.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
