Re: [protobuf] Re: How to fed PB's generated serialized file to map reduce framework

2010-04-23 Thread Kenton Varda
On Fri, Apr 23, 2010 at 1:53 PM, Owen O'Malley wrote: > > On Apr 23, 2010, at 12:00 PM, Kenton Varda wrote: > > If you are assuming that the serialization is canonical, can't you just >> compare the raw bytes? >> > > Clearly that is my fallback plan. Using memcmp as the compare function is > eas

Re: [protobuf] Re: How to fed PB's generated serialized file to map reduce framework

2010-04-23 Thread Kenton Varda
On Fri, Apr 23, 2010 at 12:12 PM, Christopher Smith wrote: > PB serialization is deterministic, but not necessarily canonical depending > on your canonicalization rules. I imagine that can cause issues. In > particular, fields with default values look different depending on whether > the default v

Re: [protobuf] Re: How to fed PB's generated serialized file to map reduce framework

2010-04-23 Thread Christopher Smith
PB serialization is deterministic, but not necessarily canonical depending on your canonicalization rules. I imagine that can cause issues. In particular, fields with default values look different depending on whether the default value was actually set, and of course the ordering of repeated fie

Re: [protobuf] Re: How to fed PB's generated serialized file to map reduce framework

2010-04-23 Thread Kenton Varda
If you are assuming that the serialization is canonical, can't you just compare the raw bytes? On Fri, Apr 23, 2010 at 6:08 AM, Owen O'Malley wrote: > On Apr 22, 8:33 pm, stuti awasthi wrote: > > > I wanted to pass the Protocol Buffer generated serialized file > > directly to map reduce. > > I

[protobuf] Re: How to fed PB's generated serialized file to map reduce framework

2010-04-23 Thread Owen O'Malley
On Apr 22, 8:33 pm, stuti awasthi wrote: > I wanted to pass the Protocol Buffer generated serialized file > directly to map reduce. I actually have a patch for Hadoop that does this. When my work load on security calms down, I'll clean it up and post it on Hadoop's jira. The one spot that Proto