-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello Leif!
W dniu 7/31/2013 02:45, Leif Pedersen pisze: > Interesting. So you're saying that you need 30k rps to the DB, and > the request rate to the front end of Varnish is much higher? If the > Varnish front end is "only" getting 30k rps, then perhaps there's > something you can do to improve the cacheability of objects. > Otherwise, that's impressive even by my standards. > You're sure your DB can keep up in a useful way with 30k rps? That's not a single DB, that's *all the subsystems* for which we cache the results: 30+ mysql and oracle instances (all replicated into additional set of mirrors for HA course, 3 DBAs babysitting all this rubbish), several other specialized subsystems. If you aggregate traffic for all those, it's about 30K rps, more in peak hrs and under some circumstances actually. That those cannot keep up is precisely why we're developing caching layer (actually we have one, but it's spaghetti C developed years ago, basically unmaintainable by now). The load on those machines is high at all times, as queries are not very expensive but they're not trivially cheap either. That's not > the real bottleneck, is it? I've made that mistake myself, so I > feel compelled to ask. :) If so, also quite impressive that your DB > can keep up. > > Have you tried implementing this sort of middleware in node.js? > Sounds like you've used node.js a lot, We haven't, that's the problem: we're a Python shop (with some C skills available too). Other divisions have used node.js a lot (hence that's the next solution under consideration as we can get some assistance). Still not good enough. Let's face it: caching server is what C / C++ / binary-compiled static-typing high-performance system-programming language is built for. In ideal world I'd build this thing in D (that's unavailable for obvious "chicken" lack of solutions and "egg" lack of human skills dilemma). but not for this particular problem if I > understand correctly. Perhaps it would be worth the experiment. > > Here's a great read of WSGI servers with surprising performance > differences. Tornado looks okay, but there is better. > http://nichol.as/benchmark-of-python-web-servers Erm, I do not want to sound ungrateful but that's exactly one of the pages I started with... My colleagues investigated gevent. It fell by the waysides, mostly bc of some Python's C-based extensions leaking memory under so much load. I have investigated FAPWS3 with surprisingly good results: 3K rps, no leaking (at least in my application..). It's little known but works suprisingly well. I'm not sure if credit goes to libev or good FAPWS3 coding but there it is. Still, not good enough. Etc etc. Memory leaks, crashes, lots of failed requests, etc etc. Heck I modified hello world example from Cowboy (Erlang-based http framework) to do stuff like talking over memcached protocol (another thing on working pile for this caching server) and got 5K rps but it's not like Erlang programmers are available in numbers and we're not going to hire one for this project alone. Oh well. I thought: "what the heck I'll give varnish a try" (on caching http-interfaced subsystem backend, we have some of those apart from rdbmses). Result: 7K rps at 1K concurrent connections. And stays approximately at this rate with increasing number of concurrent connections, basically up to 5K concurrent connections and grand total of 4 failed requests (0 at lower rates). Woohoo! Other solutions were basically crushed like bugs under this number of concurrent connections. That's why you should not trust those blog pages with high benchmark results: it's all fine and dandy to get those serially. But if the solution has to handle large number of concurrent connections at any moment - that's where things turn sour. I had more than one high-serial-performance Python solution fail miserably under such circumstances. But now I have a problem: how to plug mysql or oracle into varnish? I've done some C coding but replacing http-oriented backend handling in varnish is a little ambitious for me. > > It sounds like the middleware is really trivial, and compared to > the entry bar for adapting Varnish, probably worth trying several > WSGI servers and/or node.js if need be. > > I am indeed intrigued by connecting directly to the DB, but you > can probably see my skepticism between the lines here in bright > orange. :) If I understand you correctly, you would not connect to DB directly either? We tried that in the past (in small scale). It works for the moment and that's the problem: WI you need to say, upgrade mysql? Your entire client logic layer changes, in most/all of the infrastructure. Tight coupling. Sucks. > Seems like a much more flexible approach to use middleware, worth a > bit of extra hardware...on the other hand, maybe not worth it if > the cost difference really is 10x. That's what we're trying to reduce: infrastructure, power, maintenance costs. Regards, MK -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJR+NAhAAoJEFMgHzhQQ7hOPfYH/iWRxNhVICFZw2I7F/hIUHTv fL+vk4a6mXLGdK4S8tjDqo9xPXBLpdBHUmQiPqVLZouwz/zy+E15l0zteMcj08Qg LPrq8/m9E2smcFvPwKTTVpUtq0VmE+MoZqq289VbLxxoxN8v9mwzPy2C/iDBwMu9 hp939RCBTkATJ7XP+ilXvumKsMhRFVCfbdkpbQbSjNifEiDYplwGLV4FheuMOa9F T4k3M0kFu3BOJmqIvkGrtV5n8ygRtOEn+aK+C/Kq8M2wUsumsLHVnfk+KFmoASev sh3fRK7AV0lhXoxNM6TG994UqzFZ9zh3yZFTPwYED1ck4VYX8/D1xWuGjrYzN8s= =VHHc -----END PGP SIGNATURE----- _______________________________________________ varnish-dev mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
