Michael (and list), First, thanks much for the response. It's much appreciated.
> I had the same thought when we migrated from squid + jesred to Varnish; > we had several thousand patterns across a few sites. I did simplify > things a little by implementing virtual-hosting type behaviour within > Varnish, so it only had to process redirects for the particular site the > request was actually for. If you can make a similar optimisation you > might find the amount of processing per request drops considerably. Not sure I follow here - sounds like you mean putting different sites on diffing IPs and/or different varnishes? We do that with squid to match each app cluster but there is no squid-centric reason for this. We've 13 major sites that we'd front with varnish initially, and if that works well, then we'd use it for more, so possibly 100 disparate sites - each with more than one backend. > While the if/elsif ladder looks a bit ugly and like a lot of work, it's > actually pretty much exactly what squirm is already doing. So I think > you'll find the performance to be about the same; possibly a bit faster > since if you implement it within VCL you won't have the overhead of > communicating over a pipe. Guess I wasn't clear :) It's more of a configuration query since we're used to defining behavior in one place (the patterns) versus two (patterns and a preconfigured backend). > The only issue you'd have with doing it in Varnish is if your backend > hosts are pretty much arbitrary; Varnish needs each origin to be > explicitly defined. This requires a slight change to the logic, in that > you need to set req.backend appropriately, in addition to req.host > and/or req.url. But, it's not really complex. That was my initial concern. After reading your email, I realized I could define all the backends we'd need with deterministic names and possibly write some VCL to select the backend from the "rewritten" hostname. I will have to play with this and see if it works as I think it does. Luckily, we don't rewrite the Host header in squid, so that's one less thing varnish has to do :) Some of our patterns don't specify a hostname component; e.g. http://([^/]+)/images(/.*) ==> http://static.\1/images\2. I know about Varnish's regex substitution, so I'm hopeful that I can do the rewrites in VCL. > I do think that abstracting it out of the VCL so you don't have to > actually manage the if/elsif ladder directly is probably a good idea. > It'd certainly be workable but you've probably got better things to do > with your time. I guess it depends how frequently you make changes or > additions to your redirections. Changes are made by the dev team a few times a week and rolled out weekly via cfengine grabbing the latest from their VCS, pushing the patterns out to the caching boxes and poking squid/squirm. > When I moved to Varnish, I took the opportunity to place all the > redirects and rewrites into our "DNS management system", which is just > an in-house hodge-podge of Python and perl. The redirects are specified > in a similar format to squirm/jesred, and I have a script that parses > them and spits out appropriate VCL. That file is then rsynced to each of > the proxies, and included from the appropriate site's configuration. Heh - I've used this idea elsewhere but never thought of writing something to generate VCL. Thanks for the reminder ;) > If you are happy editing it directly, then there's no real issue. One > nice thing about Varnish is you can tell it to load a new config while > it's running, and if it can't compile it, it'll just tell you to rack > off and keep running the existing one. So even if you break the config > the site keeps running without a hiccup. jesred liked to just stop doing > any redirects if I broke its pattern file, and the comment about "Dodo > mode" makes me think squirm may well do the same thing. Heh - this happened once or twice, then the dev team wrote a test harness that calls squirm with a bunch of URLs and ensures that the expected rewrites are output. We may have to substitute with curls against a non-prod varnish (assuming the config compiles). > I don't think I directly answered your question, so in case you didn't > infer an answer: I personally don't think you'd benefit from doing your > own custom squirm-like (or other) handler if performance is your concern. > I think you'd be better off doing a quick hackish mass-conversion of as > many patterns as you can and seeing how Varnish performs. My hunch is > that'll alleviate any concerns you've got. Thanks much for the ideas and for telling me that someone else was (ab)using squid redirecters ;) -- HTH, YMMV, HANW :) Jason The path to enlightenment is /usr/bin/enlightenment. _______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
