It's easiest to consider write as a function that converts your
record to
bytes and readFields as a function restoring your record from
bytes. So it
should be the case that:
MyWritable i = new MyWritable();
i.initWithData(some_data);
i.write(byte_stream);
...
MyWritable j = new MyWritable();
j.initWithData(some_other_data); // (1)
j.readFields(byte_stream);
assert i.equals(j);
Note that the assert should be true whether or not (1) is present,
i.e. a
call to readFields should be deterministic and without hysteresis
(it should
make no difference whether the Writable is newly created or if it
formally
held some other state). readFields must also consume the entire
record, so
for example, if write outputs three integers, readFields must
consume three
integers. Variable-sized Writables are common, but any optional/
variably
sized fields must be encoded to satisfy the preceding.
So if your MyBigWritable record held two ints (integerA, integerB)
and a
MyWritable (my_writable), its write method might look like:
out.writeInt(integerA);
out.writeInt(integerB);
my_writable.write(out);
and readFields would restore:
integerA = in.readInt(in);
integerB = in.readInt(in);
my_writable.readFields(in);
There are many examples in the source of simple, compound, and
variably-sized Writables.
Your RecordReader is responsible for providing a key and value to
your map.
Most generic formats rely on Writables or another mode of
serialization to
write and restore objects to/from structured byte sequences, but less
generic InputFormats will create Writables from byte streams.
TextInputFormat, for example, will create Text objects from CR-
delimited
files, though Text objects are not, themselves, encoded in the
file. In
constrast, a SequenceFile storing the same data will encode the
Text object
(using its write method) and will restore that object as encoded.
The critical difference is that the framework needs to convert your
record
to a byte stream at various points- hence the Writable interface-
while you
may be more particular about the format from which you consume and
the
format to which you need your output to conform. Note that you can
elect to
use a different serialization framework if you prefer.
If your data structure will be used as a key (implementing
WritableComparable), it's strongly recommended that you implement a
RawComparator, which can compare the serialized bytes directly
without
deserializing both arguments. -C
On Jul 14, 2008, at 3:39 PM, Kylie McCormick wrote:
Hi There!
I'm currently working on code for my own Writable object (called
ServiceWritable) and I've been working off LongWritable for this
one. I
was
wondering, however, about the following two functions:
public void readFields(java.io.DataInput in)
and
public void write(java.io.DataOutput out)
I have my own RecordReader object to read in the complex type
Service, and
I
also have my own Writer object to write my complex type ResultSet
for
output. In LongWritable, the code is very simple:
value = in.readLong()
and
out.writeLong(value);
Since I am dealing with more complex objects, the ObjectWritable
won't
help
me. I'm a little confused with the interaction here between my
RecordReader,
and Writer objects--because there does not seem to be any
directly. Can
someone help me out here?
Thanks,
Kylie
