Hi Willy, sorry for not responding earlier.
On Mon, Jul 18, 2011 at 04:40:56PM +0200, Willy Tarreau wrote: > Hi Simon ! > > Last week I could find some time with a quiet place at work to dig again > into the master-worker patch series. > > I went back to what we discussed a few weeks ago concerning the pid management > and the cmdline parsing which must be performed only once, until I realized > that I was having trouble again with pid management due to two things that > we cannot easily cover : > - a new process starting with -sf which would replace the old processes > running in master/worker mode. > > - binding issues within the master process when reloading a configuration > > The first point causes a simple issue : if we start a new process which wants > to replace the old ones that were working in master-worker mode, it will send > them a SIGTTOU+SIGUSR1. The issue there is that the master should release all > of its listening sockets, forward the signal to all children then passively > wait for all of them to leave (including old ones). This is not undoable but > adds a bit of complexity, which I was planning on implementing anyway. I agree that this may be a bit tricky, but it does sound like a reasonable approach. > The second point is more complex. I realized that there normally is the same > config in the master process and in all of its active children. What happens > if the master fails to reload a configuration ? I'm thinking about the worst > issues, the one related to binding IP:ports, which can only be detected by an > active process. If the issue is just a conflict with another socket, then > surely the new process can restart with -sf instead and get rid of the issue > (back to point 1). But still, the issue of keeping a master process alive with > a config that has nothing to do with what its children are doing is messy at > best, and very dangerous. I agree that keeping a master going with a new (broken) configuration that has an unknown relationship to what the slaves are using is messy. But I suspect that as things stand it is actually quite safe. Still, it would be nice not to be messy. > My long-term solution would be to have everything related to the configuration > behind a pointer, and have that pointer referenced in sessions, health checks > etc... That way, each session would use the config it was instanciated with > and it would be much easier to allow an old and a new conf to coexist. It > would not completely fix the issues with some global settings though (eg: > nbproc, pollers, tuning options, ...) but it would be a gain. That way, if > the new config fails, we can switch pointers back to old config and forget > everything that was attempted. I think such a pointer would be rather nice, even if it turns out not to solve the problem at hand. > Then my thinking went a bit further : before doing what is described above, > we could have the current master fork a new master which would handle the > new config and issue new processes. If for any reason that new process fails > to start, it simply disappears and nothing changes. What I like with this > method is that it also implicitly allows changing many global settings, even > the master-worker mode may be changed. New sessions would simply use the new > config from the new process and old sessions would remain on the old one. The > only downside I can think of is that it will never provide any possibility to > maintain stats or any state between the old and the new process, but that's > secondary as the master-worker model is not meant for that either. I think that such an approach could work quite well. Its really just an extension of the way the (non master-worker) soft restart mechanism works. Of course, its a bit complex, but I suspect that it can be made to work in a clean manner. The only slight complication that springs to mind is the case where haproxy isn't detached from the terminal. Using a parent process of the master (both old and new) attached to the terminal should be able to handle that. Actually, I think the current code uses such a parent-process (grand-master?) for such purposes anyway. > Another advantage I was seeing to forking a new master from the old one was > that we could probably keep the soft-restart semantics for the situations > where the new process cannot bind : it could send a SIGTTOU to old processes > to relinquish the ports and send a SIGTTIN in case of failure. Yes I agree. Your suggested approach does sounds like it should fit well with the soft-restart semantics that are currently implemented for non-master-worker mode. > That thinking gave me another idea that I have not developped yet. The core > of your work is the socket cache and is what makes the system reliable. One > possibility would that the master doesn't own any configuration at all, just > the sockets. It would be the new processes that would connect to the socket > cache to grab some ports, then fork the new workers as it is done today. > I must say I'm not completely at ease with such a model because I think that > an instanciator is needed, but I like the idea of an autonomous socket cache, > which becomes sort of an interface between haproxy and the kernel. It will > also help if one day we want to implement FTP support, as we'll have to be > able to bind outgoing sockets to local port 20, and that could be performed > by the central socket cache. While a cache server sounds fun, I think it adds additional complexity. And my gut-feeling is that with that complexity will come more problems than are solved. On the other hand, I think it is also worth considering that there may be other caches needed. For instance, it would be nice if stick table entries could survive a soft restart. And as that data lives in processes that are not persistent across soft-restart (even when using master-worker) your suggestion of a cache or caches in a separate process may be a good solution. > With all this in mind, I think we need to discuss a bit more before going > back to the keyboard. Given that a number of bugs have been fixed since > 1.5-dev6, I'll probably issue -dev7 soon and that should not stop us from > trying to elaborate a model that suits all needs. Understood, thanks for keeping me up to date.
