On 04/13/2017 11:31 AM, Olivier Houchard wrote:
> On Thu, Apr 13, 2017 at 11:17:45AM +0200, Conrad Hoffmann wrote:
>> Hi Olivier,
>>
>> On 04/12/2017 06:09 PM, Olivier Houchard wrote:
>>> On Wed, Apr 12, 2017 at 05:50:54PM +0200, Olivier Houchard wrote:
>>>> On Wed, Apr 12, 2017 at 05:30:17PM +0200, Conrad Hoffmann wrote:
>>>>> Hi again,
>>>>>
>>>>> so I tried to get this to work, but didn't manage yet. I also don't quite
>>>>> understand how this is supposed to work. The first haproxy process is
>>>>> started _without_ the -x option, is that correct? Where does that instance
>>>>> ever create the socket for transfer to later instances?
>>>>>
>>>>> I have it working now insofar that on reload, subsequent instances are
>>>>> spawned with the -x option, but they'll just complain that they can't get
>>>>> anything from the unix socket (because, for all I can tell, it's not
>>>>> there?). I also can't see the relevant code path where this socket gets
>>>>> created, but I didn't have time to read all of it yet.
>>>>>
>>>>> Am I doing something wrong? Did anyone get this to work with the
>>>>> systemd-wrapper so far?
>>>>>
>>>>> Also, but this might be a coincidence, my test setup takes a huge
>>>>> performance penalty just by applying your patches (without any reloading
>>>>> whatsoever). Did this happen to anybody else? I'll send some numbers and
>>>>> more details tomorrow.
>>>>>
>>>>
>>>> Ok I can confirm the performance issues, I'm investigating.
>>>>
>>>
>>> Found it, I was messing with SO_LINGER when I shouldn't have been.
>>
>> <removed code for brevity>
>>
>> thanks a lot, I can confirm that the performance regression seems to be gone!
>>
>> I am still having the other (conceptual) problem, though. Sorry if this is
>> just me holding it wrong or something, it's been a while since I dug
>> through the internals of haproxy.
>>
>> So, as I mentioned before, we use nbproc (12) and the systemd-wrapper,
>> which in turn starts haproxy in daemon mode, giving us a process tree like
>> this (path and file names shortened for brevity):
>>
>> \_ /u/s/haproxy-systemd-wrapper -f ./hap.cfg -p /v/r/hap.pid
>> \_ /u/s/haproxy-master
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>> \_ /u/s/haproxy -f ./hap.cfg -p /v/r/hap.pid -Ds
>>
>> Now, in our config file, we have something like this:
>>
>> # expose admin socket for each process
>> stats socket ${STATS_ADDR} level admin process 1
>> stats socket ${STATS_ADDR}-2 level admin process 2
>> stats socket ${STATS_ADDR}-3 level admin process 3
>> stats socket ${STATS_ADDR}-4 level admin process 4
>> stats socket ${STATS_ADDR}-5 level admin process 5
>> stats socket ${STATS_ADDR}-6 level admin process 6
>> stats socket ${STATS_ADDR}-7 level admin process 7
>> stats socket ${STATS_ADDR}-8 level admin process 8
>> stats socket ${STATS_ADDR}-9 level admin process 9
>> stats socket ${STATS_ADDR}-10 level admin process 10
>> stats socket ${STATS_ADDR}-11 level admin process 11
>> stats socket ${STATS_ADDR}-12 level admin process 12
>>
>> Basically, we have a dedicate admin socket for each ("real") process, as we
>> need to be able to talk to each process individually. So I was wondering:
>> which admin socket should I pass as HAPROXY_STATS_SOCKET? I initially
>> thought it would have to be a special stats socket in the haproxy-master
>> process (which we currently don't have), but as far as I can tell from the
>> output of `lsof` the haproxy-master process doesn't even hold any FDs
>> anymore. Will this setup currently work with your patches at all? Do I need
>> to add a stats socket to the master process? Or would this require a list
>> of stats sockets to be passed, similar to the list of PIDs that gets passed
>> to new haproxy instances, so that each process can talk to the one from
>> which it is taking over the socket(s)? In case I need a stats socket for
>> the master process, what would be the directive to create it?
>>
>
> Hi Conrad,
>
> Any of those sockets will do. Each process are made to keep all the
> listening sockets opened, even if the proxy is not bound to that specific
> process, justly so that it can be transferred via the unix socket.
>
> Regards,
>
> Olivier
Thanks, I am finally starting to understand, but I think there still might
be a problem. I didn't see that initially, but when I use one of the
processes existing admin sockets it still fails, with the following messages:
2017-04-13_10:27:46.95005 [WARNING] 102/102746 (14101) : We didn't get the
expected number of sockets (expecting 48 got 37)
2017-04-13_10:27:46.95007 [ALERT] 102/102746 (14101) : Failed to get the
sockets from the old process!
I have a suspicion about the possible reason. We have a two-tier setup, as
is often recommended here on the mailing list: 11 processes do (almost)
only SSL termination, then pass to a single process that does most of the
heavy lifting. These process use different sockets of course (we use
`bind-process 1` and `bind-process 2-X` in frontends). The message above is
from the first process, which is the non-SSL one. When using an admin
socket from any of the other processes, the message changes to "(expecting
48 got 17)".
I assume the patches are incompatible with such a setup at the moment?
Thanks once more :)
Conrad
--
Conrad Hoffmann
Traffic Engineer
SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B
