I created my own Writable class to store 3 pieces of information. In my mapreducer.Reducer class I collect all of them and then process as a group, ie:

reduce(key, values, context) {
  List<Foo> myFoos =new ArrayList();
  for (Foo value : values) {
   myFoos.add(value);
  }
}

I was perplexed when entries in the list changed underneath me so I put a freeze() method on it:

boolean m_frozen = false;
public void freeze() {
  m_frozen = true;
}
@Override
public void readFields(DataInput in) throws IOException {       
  if (m_frozen) {
     throw new IllegalStateException();
  }
}

And noted that the exception was thrown:
        at Foo.readFields(Foo.java:169)
at org.apache.hadoop.io.serializer.WritableSerialization $WritableDeserializer.deserialize(WritableSerialization.java:67) at org.apache.hadoop.io.serializer.WritableSerialization $WritableDeserializer.deserialize(WritableSerialization.java:40) at org .apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java: 116) at org.apache.hadoop.mapreduce.ReduceContext $ValueIterator.next(ReduceContext.java:163)

Am I doing something wrong? Should I expect this VALUEIN object to change from underneath me? I'm using hadoop 0.20.1 (from a cloudera tarball)

Chris

Reply via email to