On Tue, Feb 8, 2011 at 5:47 AM, Evan Jones <[email protected]> wrote: > I read this proposal somewhat carefully, and thought about it for a couple > days.
Thanks for the feedback! > * It seems to me that this will solve the problem for people who know > statically at compile time what types they need to handle from a stream, so > they can define the "stream type" appropriately. Will users find themselves > running into the case where they need to handle "generic" messages, and end > up needing to "roll their own" stream support anyway? > > I ask this question because I built my own RPC system on top of protocol > buffers, and in this domain it is useful to be able to pass "unknown" > messages around, typically as unparsed byte strings. Hence, this streams > proposal wouldn't be useful to me, so I'm just wondering: am I an anomaly > here, or could it be that many applications will find themselves needing to > handle "any" protocol buffer message in their streams? In fact, a large part of my motivation for writing this was so that I can use it in my own RPC implementation, Captain Proto. Here's the Captain Proto protocol, which already works in this "streaming" fashion: http://code.google.com/p/capnproto/source/browse/proto/capnproto.proto I handle user messages by passing them as "bytes", embedded in my own outer message. This is somewhat inefficient currently, as it will require an extra copy of all those bytes. However, it seems likely that future improvements to protocol buffers will allow "bytes" fields to share memory with the original buffer, which will eliminate this concern. > The Visitor class has two standard implementations: "Writer" and >> "Filler". MyStream::Writer writes the visited fields to a >> CodedOutputStream, using the same wire format as would be used to encode >> MyStream as one big message. >> > > Imagine I wanted a different protocol. Eg. I want something that checksums > each message, or maybe compresses them, etc. Will I need to subclass > MessageType::Visitor for each stream that I want to encode? Or will I need > to change the code generator? To do these things generically, we'd need to introduce some sort of equivalent of Reflection for streams. This certainly seems like it could be a useful addition to the family, but I wanted to get the basic functionality out there first and then see if this is needed. Note that I expect people will generally only "stream" their top-level message. Although the proposal allows for streaming sub-messages as well, I expect that people will normally want to parse them into message objects which are handled whole. So, you only have to manually implement the top-level stream, and then you can invoke some reflective algorithm from there. > features like being able to detect broken streams and "resume" in the > middle are useful. > I'm not sure how this relates. This seems like it should be handled at a lower layer, like in the InputStream -- if the connection is lost, it can re-establish and resume, without the parser ever knowing what happened. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
