Ok this things looks like a map-reduce api issue:
I went back to the old style of map-reduce api: now I get a good avro header
but no datums. Sheesh! can someone please help !
The main function:
final static Schema IN_SCHEMA =
LogshedCollectorUtils.getResourceSchema();
final static Schema OUT_SCHEMA =
LogshedCollectorUtils.getResourceSchema();
final static ReflectData reflectData = ReflectData.get();
final static Schema KEY_SCHEMA =
reflectData.getSchema(LogKeyWritable.class);
final static Schema MAP_OUT_SCHEMA = Pair.getPairSchema(KEY_SCHEMA,
OUT_SCHEMA);
Configuration conf =
LogshedCollectorUtils.getLocalHadoopConfiguartion();
JobConf jobConf = new
JobConf(LogshedCollectorUtils.getLocalHadoopConfiguartion(),
MuxDemuxJob.class);
jobConf.setJobName("muxdemux");
jobConf.setJarByClass(MuxDemuxJob.class);
jobConf.setInputFormat(AvroInputFormat.class);
jobConf.setOutputFormat(AvroOutputFormat.class);
AvroJob.setInputSchema(jobConf, IN_SCHEMA);
AvroJob.setMapOutputSchema(jobConf, MAP_OUT_SCHEMA);
AvroJob.setOutputSchema(jobConf, OUT_SCHEMA);
AvroJob.setMapperClass(jobConf, LogshedMapper.class);
AvroJob.setReducerClass(jobConf, LogshedReducer.class);
//Job job = new Job(jobConf, "muxdemux");
FileInputFormat.setInputPaths(jobConf, new Path(args[0]));
Path outPath = new Path(args[1]);
FileOutputFormat.setOutputPath(jobConf, outPath);
JobClient.runJob(jobConf);
return 0;
Nikhil
--
View this message in context:
http://apache-avro.679487.n3.nabble.com/Avro-Map-Reduce-Question-GenericRecord-renaming-reduce-output-tp4025105p4025126.html
Sent from the Avro - Users mailing list archive at Nabble.com.
