-------- In message <[email protected]>, Nils Goroll writes:
So before I reply in detail, let me do a quick sketch of what the heck I have been hacking the last couple of weeks. I was taken quite by surprise when various people started talking about backends in multiple of 100k, not *backend connections* but *backends*. In a number of places our code doesn't scale at that level, for instance we have a thread per backend for healthchecks and just weeks before I had added an entire waiter per backend-tcp-pool. Multiplied by 100k that simply doesn't scale. So instead I went the other way. Waiters are now a general purpose event-driver for filedescriptors, and we have a single waiter per thread-pool, which is used both for client connections and backend connections. Next step was to add a global "wait on this in any suitable thread-pool" API, for things which do not have pool-affinity[1]. After that I want to change the backend health-polling to be event driven, so that a single global thread kicks the polls off and the waiters finish them using regular worker threads (still thinking this through). In particular I want the tcp connect(2) to be asynchronous because the healthpolls will see into network problems (in order that backend fetches should not). With that background: >* We should not include waiter.h from cache.h as vmods requiring cache.h > should not need the waiter interface (otherwise we'd need to install > waiter.h) I think I may have added waiter.h the wrong place in Makefile.am. The waiter interface is/can/will be useful for VMODS, as it allows them to do event-driven processing, so it should not be hidden from them. >* For sessions, temporarily alloc struct waited on the ws - we don't need > the workspace while we are waiting, and we don't need struct waited unless > we are waiting Good idea. >* Add Waiter_Done to invalidate struct waited and change the signature of > Wait_Call / waiter_handle_f to struct waited ** to allow for invalidation > of the struct waited pointer. Not so sure about this one, seems a tad strict to me. (It's obviously a good idea to clear it if/when it has no permanence (ie: on ws), but I fail to see a particular reason to mandate clearing it ? Poul-Henning [1] This is a NUMA thing: The client fd's we have accepted should and in the future undoubtedly will have NUMA pool affinity. Backend connections and even less so backend health poll connections will not, so we need to distribute those over the thread pools[2] in some fair fashion. [2] I've been really tempted to rename to concept of thread pools to "clans", because the NUMA aspect is about so much more than just the threads. Clans would have a (just a) pool of threads, acceptor sockets, waiter, etc. I may do this for 5.0. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [email protected] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. _______________________________________________ varnish-dev mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
