On Thu, May 5, 2011 at 4:14 PM, David Leimbach <[email protected]> wrote:
> Thanks! > > On Thu, May 5, 2011 at 3:11 PM, Mike Oxford <[email protected]> wrote: > >> Protobuf does not do inter-message delimiters. You have to handle that >> aspect yourself. >> >> In the streambuf, put in the length of your protobuf message (in network >> format), then the stream the message into it. >> On the other end they should extract the length, convert to host format >> and then read the data in and ship to protobuf. >> > > Ah this seems like some part I hadn't read up to yet... It's difficult to > do this stuff without contiguous blocks of time :-). > > Some of this makes me want to also investigate using BERT or BERT-RPC > straight to erlang's "native" client for C++ bindings and just skip protocol > buffers altogether :-). > > Yes I know protocol buffers are trendy and all, but even Google's Go > programming language doesn't support them natively (without external > packages). They did their own encoding as well (gob). > > https://github.com/blog/531-introducing-bert-and-bert-rpc > > http://bert-rpc.org/ > > Dave > But I suppose that would be pretty wasteful given that I think the erlang riak client is all protocol buffers too? Dave > > >> >> That's the 10,000 foot view. >> >> http://code.google.com/apis/protocolbuffers/docs/techniques.html >> >> --snip >> If you want to write multiple messages to a single file or stream, it is >> up to you to keep track of where one message ends and the next begins. The >> Protocol Buffer wire format is not self-delimiting, so protocol buffer >> parsers cannot determine where a message ends on their own. The easiest way >> to solve this problem is to write the size of each message before you write >> the message itself. When you read the messages back in, you read the size, >> then read the bytes into a separate buffer, then parse from that buffer. (If >> you want to avoid copying bytes to a separate buffer, check out the >> CodedInputStream class (in both C++ and Java) which can be told to limit >> reads to a certain number of bytes.) >> >> --end snip >> >> You do have to parse a message in its entirety; there is no incremental >> parsing. If you have a message which wraps 10,000 messages you'll have to >> load the whole thing in before you can parse. >> >> HTH. >> >> -mox >> >> >> >> >> >> On Thu, May 5, 2011 at 3:02 PM, David Leimbach <[email protected]> wrote: >> >>> I haven't completely abandoned mine yet either, but I've been finding >>> that some of the I wanted to use with streambufs aren't going to pan out >>> due to the nature of a socket backed stream and the way protocol buffers >>> expects it to work. Protocol Buffers expects the stream to be terminated >>> when done receiving a message *THEN* parse it rather than building the >>> parsing in (I think that's what happened anyway... I really need to check my >>> notes I took when I ran into this). This does appear to make a simple >>> socket fd backed streambuf way to plug into any protocol buffers stream a >>> bit more difficult than I was hoping. >>> >>> Then again, I may just not be overriding a particular member function >>> that I should be either. >>> >>> Might get some time on the weekend to poke at this some more. >>> >>> On the plus side I definitely had some Riak RPC binary protocol bits >>> working. I was encoding/decoding Ping/Pong just fine (and they don't need >>> any Protocol Buffer anything to work). >>> >>> Dave >>> >>> On Sat, Apr 9, 2011 at 8:47 AM, Sean Cribbs <[email protected]> wrote: >>> >>>> I didn't permanently abandon it, but it was much more fiddly than doing >>>> the same thing in pure Ruby. I have plans to deliver separate "native" >>>> Protocol Buffers libraries for MRI and JRuby (at least) in 1.0 of the Ruby >>>> client. >>>> >>>> Because it's being confused in this conversation, I think it merits >>>> clarification -- the "protocol" that is used to talk to Riak and Google's >>>> Protocol Buffers are NOT the same thing. Riak uses a simple length- and >>>> message-code-prefixed binary protocol, in which the complex messages (ones >>>> that have bodies and not just the message code) are serialized via Google's >>>> Protocol Buffers. So, while we don't use the RPC facilities in Google's >>>> library, the *serialization format* DOES use Protocol Buffers. >>>> >>>> Sorry for the confusion, we'll work to make that clearer in the wiki. >>>> >>>> Sean Cribbs <[email protected]> >>>> Developer Advocate >>>> Basho Technologies, Inc. >>>> http://basho.com/ >>>> >>>> On Apr 8, 2011, at 9:17 PM, Scott Gonyea wrote: >>>> >>>> They are the same and you can actually see me plugging into the C++ code >>>> here: >>>> >>>> https://github.com/sgonyea/pabst/tree/master/ext >>>> >>>> But as part of an Objective-C library (called ObjFW). So, the code is >>>> actually an Objective-C++ wrapper around the C++ PB code, that exchanges >>>> messages with Objective-C code (that hooks into Ruby). >>>> >>>> I believe Sean Cribbs has some initial C++-wrapper code in his Ripple >>>> repo... Though he eventually abandoned it after C++ left him permanently >>>> cross-eyed (I think that's why). >>>> >>>> Scott >>>> >>>> On Apr 8, 2011, at 5:20 PM, Mike Oxford wrote: >>>> >>>> Be careful here.. >>>> >>>> I do not thing Riak's "protocol buffers" are the same as Google's >>>> protocol buffers. >>>> Google's does bit-level packing and some other tricks that Riak does >>>> not do, even though they both use the ".proto" file extension and very very >>>> similar proto semantics. >>>> >>>> That said, if they ARE the same, then you can take the .proto files and >>>> generate C++ classes, and use the secondary library "protobuf-c" to >>>> generate >>>> C structs for the wire format. >>>> >>>> -mox >>>> >>>> On Fri, Apr 8, 2011 at 4:43 PM, David Leimbach <[email protected]>wrote: >>>> >>>>> Spent a little time poking at this today... Kind of surprised that >>>>> there was no message defined for PingReq or for listing buckets. >>>>> >>>>> I realize these messages really have no usable payload, and just sort >>>>> of have a tag and length, but for completeness it kind of feels like they >>>>> should be there. >>>>> >>>>> Of course I'm not a Protocol Buffers expert in any sense, so I can't >>>>> say whether this is a normal kind of choice or not. >>>>> >>>>> Dave >>>>> >>>>> >>>>> On Fri, Apr 8, 2011 at 2:49 PM, Scott Gonyea <[email protected]> wrote: >>>>> >>>>>> If we had this then a C-wrapper would be that much more attainable. >>>>>> So, the author of such a lib would be a superstar in my book :). >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On Apr 8, 2011, at 1:46 PM, David Leimbach <[email protected]> wrote: >>>>>> >>>>>> > I've been writing a bit of code in Haskell to push data to Riak, and >>>>>> the bindings are pretty easy to use (Thanks Brian!), but getting >>>>>> penetration >>>>>> at my company for Haskell is going to take a little time. >>>>>> > >>>>>> > As such I'm just wondering if anyone knows of anyone working on a >>>>>> protocol buffers version of a Riak client in C++, or if this is going to >>>>>> be >>>>>> something I'll have to take on. >>>>>> > >>>>>> > I've found a few generic looking C++ projects that use Boost's >>>>>> asynchronous IO stuff with protocol buffers to make an RPC system, but >>>>>> I'm >>>>>> not sure if any of those are implicitly compatible. >>>>>> > >>>>>> > Guess I'm just looking for a pointer... >>>>>> > >>>>>> > Dave >>>>>> > _______________________________________________ >>>>>> > riak-users mailing list >>>>>> > [email protected] >>>>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> [email protected] >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> [email protected] >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> [email protected] >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> [email protected] >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> [email protected] >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
