> We had previously talked about an even tighter binary protocol, but perhaps > harder to generalize a parser around. This doesn't seem different > enough from the existing binary protocol to warrant introducing an > incompatibility.
I honestly can't remember what else was removed in our old discussions. Old man is old. Something about the "reserved bytes" turning into a longer flag set maybe? This proposal just drops those bytes. What I may have mentioned was that uh... size length encoding, similar to what MySQL uses. So a key length of < 252 bytes would be a 1b length. a body length of just under 2^16 would be 3 bytes (253 as a flag + 2 bytes), and over would be 4 bytes (254 as a flag + 3 bytes). Could be nice for getting CAS down in size, but you end up inflating an extra byte for larger values. > Couple notes: > > * We can't omit the vbucket in the response if we're doing *K type operations > (which I still don't think are necessary). okay, so vbucket and "response" stay as separate flags. > * I like the idea of a separate quiet flag from the opcode (but will happily > settle for having the bit in a consistent position) Using the last spare flag in the magic byte? Even if we bump up a whole byte for flags, that only gives one extra. Originally I had it listed in the magic byte, but moved it to the opcode bit to free up a flag. I'd accept a strong argument to the contrary, though. > * Tap has a variable length extras, but we could possibly just push that into > the body. I'm too blind at a glance to see where that is in the wiki page you wrote. Would it be a huge pain to change? > * I don't quite understand the position of CAS in that diagram -- it looks > like it's not aligned, but that doesn't sound right. I didn't spend a ton of time re-aligning the bytes after removing a chunk of them. Given the flags you send, CAS may not end up fully aligned in the header anyway. That's not necessarily how the API end has to implement it however, that's just how it has to be read off the wire. > * I like the ASCII stuff, but I would like to see more stuff around quiet > that make it harder to get wrong. Can you expand on this? The way I tried to write it was direct compat with the way binprot is handled. ie; you can shove commands down the barrel and read them back asynchronously, matching up opaques or keys to specific errors. See the (admittedly barely notated) "response" section for ASCII. > I'd like to re-propose all of the arguments I lost in the original binary > protocol definition that have come back to suck for me and have another > chance to lose, but with more experience now so I can actually say *where* > it's sucked: > > * Untagged responses. > * Signed incr (and no decr -- again) > * Quiet as a flag. > > To expand: > > # Untagged Response > > Like IMAP, I'd like to have the ability to send messages from the server that > are not responses to requests the client made. That is, messages can > be sent *between* other messages. This allows for things like fully > asynchronous responses interleaved with synchronous responses: > > 1> something that will take a while > 1< doing > 2> something quick > 2< done > 3> something else quick > .< Hey, that thing from before is done > 3< done > > Clients can ignore (or log or whatever) unsolicited messages they don't know > how to handle, but some operations get *very* simple (observing > internal memory restructuring, "jobs" over keys, persistence operations, > proxy operations, etc...) How does binprot stop you from doing this presently (besides saying you can't)? > # Signed Incr (and no decr) > > 5 + -1 = 4 > > One operation, one thing to worry about, one thing to test. We've had bugs > in each of them separately. Counters aren't likely to actually go up > above 2^63 (though I have had people say they slice a single counter up into > multiple smaller counters and address the individual parts, so there's > at least one reason, though, you know, bah). This one is so annoying. I think it *may* be fine to change. If a user wishes to treat it as an unsigned value, they can "simply" do unsigned casts with the numbers. That may end up with clients doing horrific signed/unsigned incr/decr emulation layers though. Argh. So much hate. > # Quiet as a Flag > > Again, happy to take it as a consistent bit, but it's not a different > command, just different handling of the command both in the server and the > client. I wrote my justification above. Hit me back with a stronger response, or is it fine? Thanks! -Dormando
