I'm still kicking around ideas here. I'm trying to think of a solution
which is workable in the presence of larger numbers of backends,
without sacrificing performance or configurability of smaller setups.
I imagine there are others running into the same problems.
There are, I think, two main problems to solve:
1) Managing large numbers of dynamically changing backends.
2) Improving performance of request to service mapping when the number of
services is large. (Or is there a nice way to do it I'm not aware of?)
One option would be to introduce another shared-memory area for use
between a management daemon and varnishd. I imagine this area would
contain data-structures to be updated by the management daemon
periodically, and read by varnishd when processing requests.
A reader-writer locking scheme should give reasonable performance,
since writes to the structure by the management daemon would be very
infrequent.
Here is how I think it should work:
* shared hashtable of services
* each service has a list of backends (and assigned weights, strategy
params etc)
* default service or 4xx/5xx configurable for hashtable misses
Instead of deciding on a backend name, the VCL code should decide on a
service name. It should be able to either specify a literal name, or
generate a name string based on the Host/URL/other parameters.
For example: sub vcl_recv { req.service = req.http.host; } would give
good performance on large service maps, without precluding long lists
of regular expressions for those that really need them. (is this legal
VCL?)
Initial values for the shared data-structures could be set by the VCL
file to enable varnishd to operate as usual if an external management
daemon is not present.
I guess another (somewhat hacky) solution would be to continually
generate and update VCL on the fly.
Does anyone have further thoughts on this?
On 9/25/07, Poul-Henning Kamp <[EMAIL PROTECTED]> wrote:
> In message <[EMAIL PROTECTED]>, "Luke
> Macpherson" writes:
> >I am looking to deploy varnish in a configuration which has a large
> >number of backends, and integrate it with our existing management
> >infrastructure. I'd like to solicit your opinions on the best way to
> >do that.
> >
> >What I'm thinking of doing is using replacing the compiled VCL
> >configuration object with a custom-written object. This C library
> >could talk to our existing infrastructure to keep track of live
> >backends and choose the appropriate one at run-time based on
> >availability, load and other factors.
>
> VCL has a mechanism for including C language, so in theory you can
> do that from a VCL program.
>
> >Is this a reasonable approach? Is the VCL api stable and well-defined
> >enough? Is there a better place to plug-in this kind of functionality?
>
> VCL doesn't guarantee stable internal APIs, that's one of the reasons
> why we go to such great lengths to make it one single executable that
> contains everything.
>
> That said, I don't see anything very hard or impossible in what
> you are proposing, but it takes a good programmer to get it right.
>
> --
> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
> [EMAIL PROTECTED] | TCP/IP since RFC 956
> FreeBSD committer | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
_______________________________________________
varnish-dev mailing list
varnish-dev@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev
