We do stuff very much like this. Generally we try to encode the PB's as sequence files so that they are splittable. If that doesn't make sense, then we use non-splittable coded streams.
--Chris On Apr 19, 2010, at 9:22 PM, stuti awasthi <stutic...@gmail.com> wrote: > > Thanks Jason, > > Yes map reduce works with the raw bytes of data. I was planning to > have log analysis over hadoop using map reduce. Suppose I have ever > increasing log files and I want to serialize its data and then pass > over the Hadoop map reduce to get it analyzed. If this is the use case > then do we have any mechanism through which we can feed the serialized > data into the map reduce? > > Bye :) > > On Apr 19, 10:32 pm, Jason Hsueh <jas...@google.com> wrote: >> As far as I recall, mapreduce just works with raw bytes. If you want to use >> protocol buffers within map reduce, you just need to use the serialization >> routines to convert from the raw bytes to your proto. >> >> >> >> On Mon, Apr 19, 2010 at 4:06 AM, stuti awasthi <stutic...@gmail.com> wrote: >>> Hi all, >> >>> I just started looking at the Protocol Buffer for serialization of >>> data structure. I wanted to use this with the map reduce framework. I >>> searched on web but do not get any specific pointers on how to do it. >> >>> If any body can please tell me how can we use PB with the map reduce >>> framework. Im new to this concept of distributed serialization. >> >>> Any pointers will be great.. Thanks in advance. >> >>> Bye >> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Protocol Buffers" group. >>> To post to this group, send email to proto...@googlegroups.com. >>> To unsubscribe from this group, send email to >>> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com> >>> . >>> For more options, visit this group at >>> http://groups.google.com/group/protobuf?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To post to this group, send email to proto...@googlegroups.com. >> To unsubscribe from this group, send email to >> protobuf+unsubscr...@googlegroups.com. >> For more options, visit this group >> athttp://groups.google.com/group/protobuf?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
