On Sun, Jan 11, 2009 at 6:10 PM, Erich <[email protected]> wrote: > > Hey everyone. In my recent work, I've had to do a lot of work with > protocols, layer 2 and up. This has me thinking about the beanstalk > protocol and evaluating it in a new light. There are a couple of > changes I'd like to suggest, which break compatability, and should be > considered for v2 or some other far off point. > > Without further ado, here are my ideas: > > 1. Framing: I think beanstalk should have a framing mechanism for its > messages. Such a mechanism would simplify the parsing of messages by > a lot, and reduce some potential DOS situations (both intentional, and > more importantly accidental). More on that below. The reason I think > framing would work so well, is that beanstalk has 2 atomic units of > communication, the message and data. Data is already framed, since the > message describing the job (eg OK...) includes a job size argument. > So really all I am proposing is framing the messages. > > The framing policy could be as simple as: [length of message] message > [\r\n] > > Lenght of messages could be a binary in, a set of ascii hex values, or > even (fitting with the current protocol) numbers as ascii, howver > these last two would need to be a fixed length (so preceeded by 0 or > something) to get the benefits of framing. > > So why does this matter? Well for a few reasons: > A. Simplicity of code -- In every client I've looked at, the > socket.recv code would be much simpler. In the case of a binary length > value framing, and a blocking client... it could be replaced with: > > lenraw = socket.recv(4) > len = ntoh(lenraw) > messg = socket.recv(len+2) > if not messg.endswith('\r\n'): do_error() > > This is avoids having to do reads in a while True... , waiting for a > condition that may never happen (\r\n) or a pre calculated max message > length. > > 2. Out of band communications... There are some states which we may > wish to communicate to clients as soon as possible. Examples include > Draining mode, Disk full, etc. These could be included by way of > flags attatched to the protocol messages (responses) from beanstalkd. > It would avoid a full stats round trip message set, but would increase > the per message overhead. It would also allow for finer grained > control for the clients. > > 3. Binary protocol. This one I'm not sure about. Im both for and > against it :) The title explains it all, so instead I present pros > and cons: > > Pros: > - fast > -compact (less bandwidth) > - not much different than already exists, it just enumerates > acceptable, instead of leaving them as text. > > Cons: > - client writing is a bit more complex (parsing etc) > - not telnet debuggable (some will argue that writing a reference > telent like client is a good first step anyway) > > 4. Message batching. By this I mean allowing more than one message > per packet. Im pretty sure this would be OK now, but I think it should > be explicitly stated. This allows for some performance tweaks, > particularly client end (when combined with "stacking"). > > > Now the last three are all performance related. They should be > carefully considered, since beanstalkd is already killer fast, and > we've heard here of memory related bounds, and bandwidth related > bounds, but not cpu related bounds. So maybe we need to evaluate > interms of those. > > Overall Im not married to any of these of course, I just want to bring > the discussion up. I will push pretty hard for #1 (framing) because > it simplifies my code :). > > What do the rest of you guys think about this? > > Regards, > Erich > > > Hi, I'd a like to request a feature in the next version, which is a special kind of messages/tasks that are guaranteed to be received once by every worker. This feature is very useful to control worker's behavior. e.g. upgrading worker to newer version, direct workers to ignore some task. If workers are programed in scripting languages like python or ruby, the feature can be used to update small chunk of codes.
-- Best regards Linan Wang --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en -~----------~----~----~----~------~----~------~--~---
