On Wed, Jun 08, 2011 at 07:32:29AM +0200, Willy Tarreau wrote: > Hi Simon, > > On Wed, Jun 08, 2011 at 09:55:21AM +0900, Simon Horman wrote: > (...) > > I should note that I'm not entirely sure about the performance implications > > of looping through all sessions like this. Perhaps a per-server list is > > needed. > > That's exactly what I wanted to respond to your mail :-) > I always refused to implement that until we have a per-server session > list, but I also know there it increasing demand for this feature due to > long-running sessions (eg: WebSockets). > > There are sites that are running with more than 100k concurrent connections > and the cost can be huge there. One could say that a site running at 100k > conns will have a decent hardware which will not take too long to scan all > the table though. > > I think that what could be done is to document the feature as possibly > very expensive when servers go down or when servers are flapping. If we > explain all the risks in the doc, users will be able to check the impact > for their usage.
I'm happy with documenting things as you suggest. Do you have any feeling of how difficult it would be to add a per-server session list? > Another point is that a flapping server can definitely make a service > totally unusable and that must be indicated too. You can check any stats > page you have access to, and consider that for each occurrence of a Down > event on a server, all those server's users will be kicked out. Right now, > when checks have too short timeouts, it can happen that a server goes down > for a short period without much impact. It does not get any new connections > for a few seconds but still serve the current ones. With this feature, all > connections will be killed for each such event. Flapping servers are a > common thing in environments where servers are overloaded or support too > few concurrent connections. I think that a warning about this in the doc > is absolutely necessary too. I considered that too. My idea was that perhaps connections should only be closed when they receive their next packet. That is, when a read is made from the client for a session whose server is down. But it it seems to me that there demand for the feature I have implemented (modulo bugs and performance issues). And that alternative schemes such as the one I describe above could be separate on-markeddown actions. > > I'm also unsure of the performance implications of waking up a potentially > > large number of sessions. > > Yes it will cost a lot too. And it will flood the logs at extreme speed. > Think that you're possibly sending 10-100k logs within a few hundreds of > milliseconds, that can be up to one million log messages per second. Most > likely, the time spent in sprintf() will sensibly extend the processing > time (eg: about 100k messages per second, meaning a freeze for up to one > second) and most of the logs will be lost. That can be documented too :-) > > > Perhaps their priority should be reduced? > > That's a good idea. At first read I was against this because there is no > way to restore the priority after that, but since the tasks are dying, > it's not an issue :-) Haha. I was worried about not being able to restore their priority too. But as you point out, its a moot point. > Well, maybe we should still consider the possibility that we only kill the > connection on the server side. For current state of affairs, it will not > change anything, but when we later support server-side keep-alive and later > connection pools, it will make it easier to retry the request over another > connection. Sure, perhaps that could be another on-markedown action? > Oh, BTW, I think you'd better use TASK_WOKEN_OTHER than TASK_WOKEN_RES, > because the later is to indicate that a previously unavailable resource > has become available (reason why you see it in queue processing). Thanks, I was unsure of which value to use. > Another point, please check that it works with servers put into maintenance > mode, I'm not sure this part calls set_server_down(). Thanks, will do.
