On Jan 11, 1:51 pm, "Keith Rarick" <[email protected]> wrote:
> On Sun, Jan 11, 2009 at 10:16 AM, Erich <[email protected]> wrote:
> > 1. Should we create a proxy server for beanstalkd? It could be a true
> > proxy (think lighttpd in proxy mode) or a redirecting broker (a
> > beanstalk 302 equivelent).
>
> It would be neat to explore this, but I'm afraid either of these would
> cut into beanstalk's speed significantly. Maybe it wouldn't be too
> much of a hit for persistent mode.
>
In situations where the constraint is memory this wont be a big
problem, a small amount of latency for big jobs will not really matter
that much, since throughput will still be high. In cases of per-
server bandwidth being the problem, then yeah, this could cut into it.
> > 4. Should we create a new project (for those that care to) which is a
> > generic "server broker" (I actually kinda like this idea.. i may do it
> > anyway), and would be usable with beanstalk, rabitmq, memcache,
> > etc...?
>
> This is an awesome idea. I think it would be best done at the IP
> layer, though. A smart switch or router could do it transparently. I
> don't know if such a thing exists already, but I wouldn't be surprised
> if IOS has such a feature somewhere. Or maybe some fancy iptables
> rules and some glue to automate them. This could also be used for a
> super-fast http load balancer.
Thats one way to approach it, and I believe a few projects exist for
similar stuff. I would base it on IPTables, using layer7 matching and
some an NFQueue connection to userland code for making decisions, and
FW marking them, so that fancy routing rules will do their magic. If
it doesn't exist, that would be a really good project, but its too
much like my last day job for me to be interested right now... :) I
see 2 problems with this approach:
1. It creates a bottleneck... all traffic needs to pass thru this box.
2. it needs a fairly deep understanding of each protocol
My thoughts were a fast server and super lightweight protocol, similar
to memcache and beanstalk. Clients would connect to the server and
communication would look something like this:
C: I have data who's md5 is $A for protocol $Pa
S: Send it to server $PaS1
C: i have data who's md5 is $B for protocol $Pa
S: Send it to server $PaS2
C: i have data who's md5 is $C for protocol $Pb
S: Error! No $Pb servers
$PaSN: I now exist!
S: OK will send you data...
And so on. The above was a sort of clock-hashing algorithm done
distributed style, but Im sure other algorithms exist. This is just
one piece of a whole system of course... not the main engine by any
means. Im actually trying to solve 2 problems here:
1. having every application (possibly in multiple languages w/
multiple implementations) correctly pick which server to use
2. Communicating server availability to every instance of the
application.
This is not without it's downsides too:
1. it requires special wrapper scripts for all relevant servers
(unless they natively understand...)
2. more latency for requests.
3. prolly something subtle that im totally flaking on
Regards,
Erich
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"beanstalk-talk" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/beanstalk-talk?hl=en
-~----------~----~----~----~------~----~------~--~---
