Re: Seamless reloads: file descriptors utilization in LUA
Hello, On Mon, Jan 21, 2019 at 06:53:12AM +0300, Wert wrote: > Hi, > > I'm talking only about performance ways) > > About socket. > I use UDP for sending, there are no reasons for delays. > However, my bad - I misunderstood some FDs in "lsof". It is not related to > that UDP-sending, that is OK. > > About file system. > I open file from disk for GeoIP, but finally it cached in memory. I don't > think that there will be difference if I move it to ramdisk. > > Example for Redis (works same as for GeoIP): > 1) luarocks install redis-lua > 2) At the top in lua-file (or inside core.register_init()): > redisClient = pcall(redis.connect,'unix:///run/redis/redis.sock') > 3) For cfg just "lua-load" param should be enough. > > For each reload "lsof" would show one additional unix-socket on master and > worker. > > As I understand: > - The LUA initialization executes for master and creates FD. > - During reload "re-executed" master-process keeps old FD and gets one more > by new lua-init. > - Workers inherit everything from master. To avoid FD leaks during a reload of the master process, those FDs must be marked FD_CLOEXEC. I don't know how you are opening these file but you should try to do a fcntl on the FD. -- William Lallemand
Re: Seamless reloads: file descriptors utilization in LUA
Hi, I'm talking only about performance ways) About socket. I use UDP for sending, there are no reasons for delays. However, my bad - I misunderstood some FDs in "lsof". It is not related to that UDP-sending, that is OK. About file system. I open file from disk for GeoIP, but finally it cached in memory. I don't think that there will be difference if I move it to ramdisk. Example for Redis (works same as for GeoIP): 1) luarocks install redis-lua 2) At the top in lua-file (or inside core.register_init()): redisClient = pcall(redis.connect,'unix:///run/redis/redis.sock') 3) For cfg just "lua-load" param should be enough. For each reload "lsof" would show one additional unix-socket on master and worker. As I understand: - The LUA initialization executes for master and creates FD. - During reload "re-executed" master-process keeps old FD and gets one more by new lua-init. - Workers inherit everything from master. > Hi, > First, using: > - fd for file system access > - fd for tcp/udp connections throught Lua Socket > are a bad ideas because these actions block HAProxy. While the > fd/socket is waiting for data, HAProxy does nothing, and it not process any > other connections. > With other words: If you tcp connection reach a timeout of 30s, > HAProxy will be blocked during 30s, and all the incomming or > established connections wille be block during 30s. > - For filesystem access, you can use a ramdisk FS (like /dev/shm > on most distro), or preload files. > - For tcp connections, you can user core.tcp() > - For udp connection, I don’t have workaround. > 1) > When you reload haproxy, the old processes are destroyed, and news > process are created, so the FD are closed with the process. If the > main process own FD, childrens inherit these socket, but, the main > process can’t open more sockets, so you can’t reach limits. It is strange. > Do you have simple conf+lua code which reproduce the problem ? > 2) > As I wrote, using socket with HAProxy + Lua is not a good idea. > I think that it is a bad idea to increase limits for preventing > something linked to usage of a dangerous method. > The mindset of HAProxy is performance and resources saving first ! > BR, > Thierry >> On 16 Jan 2019, at 08:59, Wert wrote: >> >>> CC'ing Thierry: as this has come on this discourse, can we have your >>> opinion about the FD's in LUA and howto best handle ulimit? >> >> >>> Apologies for the duplicate mail. >> >> >>> Thanks, >>> Lukas >> >> 1. FD >> I don't know your architecture too much. From user-side I just see no >> reasons to keep FD that created in LUA. >> For cases when I make redis-connection, open GEO-DB file or some socket to >> send, there is no reason to keep such FDs for new instance. >> It might be option or default policy to completely stop transferring LUA-FDs >> to new master. >> >> If it is difficult, probably there could be some ways to make ability for >> checking and cleaning it manually with CLI. >> >> 2. Ulimit >> It is impossible to know - how many FDs would be used by LUA even after >> fixing infinite grow. >> I use "ulimit-n 1000". Of cause it looks like dirty thing but I can't >> imagine the case when it can make real harm, while low limit can a lot. >> If there is some harm, at least you may adjust current auto-calculated limit >> with "+100","*2" or some similar modifier and it will cover many real cases. >> >> --- >> Wert >>
Re: Seamless reloads: file descriptors utilization in LUA
Hi, First, using: - fd for file system access - fd for tcp/udp connections throught Lua Socket are a bad ideas because these actions block HAProxy. While the fd/socket is waiting for data, HAProxy does nothing, and it not process any other connections. With other words: If you tcp connection reach a timeout of 30s, HAProxy will be blocked during 30s, and all the incomming or established connections wille be block during 30s. - For filesystem access, you can use a ramdisk FS (like /dev/shm on most distro), or preload files. - For tcp connections, you can user core.tcp() - For udp connection, I don’t have workaround. 1) When you reload haproxy, the old processes are destroyed, and news process are created, so the FD are closed with the process. If the main process own FD, childrens inherit these socket, but, the main process can’t open more sockets, so you can’t reach limits. It is strange. Do you have simple conf+lua code which reproduce the problem ? 2) As I wrote, using socket with HAProxy + Lua is not a good idea. I think that it is a bad idea to increase limits for preventing something linked to usage of a dangerous method. The mindset of HAProxy is performance and resources saving first ! BR, Thierry > On 16 Jan 2019, at 08:59, Wert wrote: > >> CC'ing Thierry: as this has come on this discourse, can we have your >> opinion about the FD's in LUA and howto best handle ulimit? > > >> Apologies for the duplicate mail. > > >> Thanks, >> Lukas > > 1. FD > I don't know your architecture too much. From user-side I just see no reasons > to keep FD that created in LUA. > For cases when I make redis-connection, open GEO-DB file or some socket to > send, there is no reason to keep such FDs for new instance. > It might be option or default policy to completely stop transferring LUA-FDs > to new master. > > If it is difficult, probably there could be some ways to make ability for > checking and cleaning it manually with CLI. > > 2. Ulimit > It is impossible to know - how many FDs would be used by LUA even after > fixing infinite grow. > I use "ulimit-n 1000". Of cause it looks like dirty thing but I can't > imagine the case when it can make real harm, while low limit can a lot. > If there is some harm, at least you may adjust current auto-calculated limit > with "+100","*2" or some similar modifier and it will cover many real cases. > > --- > Wert >
Re: Seamless reloads: file descriptors utilization in LUA
> CC'ing Thierry: as this has come on this discourse, can we have your > opinion about the FD's in LUA and howto best handle ulimit? > Apologies for the duplicate mail. > Thanks, > Lukas 1. FD I don't know your architecture too much. From user-side I just see no reasons to keep FD that created in LUA. For cases when I make redis-connection, open GEO-DB file or some socket to send, there is no reason to keep such FDs for new instance. It might be option or default policy to completely stop transferring LUA-FDs to new master. If it is difficult, probably there could be some ways to make ability for checking and cleaning it manually with CLI. 2. Ulimit It is impossible to know - how many FDs would be used by LUA even after fixing infinite grow. I use "ulimit-n 1000". Of cause it looks like dirty thing but I can't imagine the case when it can make real harm, while low limit can a lot. If there is some harm, at least you may adjust current auto-calculated limit with "+100","*2" or some similar modifier and it will cover many real cases. --- Wert
Re: Seamless reloads: file descriptors utilization in LUA
Hello, On Wed, 2 Jan 2019 at 14:54, Lukas Tribus wrote: > > Hello, > > On Sun, 15 Jul 2018 at 07:19, Wert wrote: > > > > Hello, > > > > 1. When in LUA > > - I open some socket and left it unclosed (even UDP-sender socket) > > - Or open some files (for example, I use LUA-maxmind lib that opens GEO-DB > > file) > > > > It is never destroyed. With each reload amount of used descriptors grows > > and finally reaches limits. > > According to "lsof", all sockets and descriptors are belongs to master > > process and all new worker processes. > > > > Should be some way to destroy it during reload or to really use advantages > > of such transfers. > > > > Tested with Haproxy 1.8.12 > > > > 2. Since haproxy has LUA, user could have needs for file descriptors that > > is impossible to count. > > Is there any real reason to keep "auto-calculated" ulimit-n option with > > very low values, based just on connection limits? > > > > Of cause, it is easy to set (for those who read docs very carefully =)), > > but some extra value could cover a few more cases "from the box", also > > making a bit less critical FD-related bugs. > > > > At least some warning in docs for this option should be useful. CC'ing Thierry: as this has come on this discourse, can we have your opinion about the FD's in LUA and howto best handle ulimit? Apologies for the duplicate mail. Thanks, Lukas
Re: Seamless reloads: file descriptors utilization in LUA
Hello, On Sun, 15 Jul 2018 at 07:19, Wert wrote: > > Hello, > > 1. When in LUA > - I open some socket and left it unclosed (even UDP-sender socket) > - Or open some files (for example, I use LUA-maxmind lib that opens GEO-DB > file) > > It is never destroyed. With each reload amount of used descriptors grows and > finally reaches limits. > According to "lsof", all sockets and descriptors are belongs to master > process and all new worker processes. > > Should be some way to destroy it during reload or to really use advantages of > such transfers. > > Tested with Haproxy 1.8.12 > > 2. Since haproxy has LUA, user could have needs for file descriptors that is > impossible to count. > Is there any real reason to keep "auto-calculated" ulimit-n option with very > low values, based just on connection limits? > > Of cause, it is easy to set (for those who read docs very carefully =)), but > some extra value could cover a few more cases "from the box", also making a > bit less critical FD-related bugs. > > At least some warning in docs for this option should be useful.
Seamless reloads: file descriptors utilization in LUA
Hello, 1. When in LUA - I open some socket and left it unclosed (even UDP-sender socket) - Or open some files (for example, I use LUA-maxmind lib that opens GEO-DB file) It is never destroyed. With each reload amount of used descriptors grows and finally reaches limits. According to "lsof", all sockets and descriptors are belongs to master process and all new worker processes. Should be some way to destroy it during reload or to really use advantages of such transfers. Tested with Haproxy 1.8.12 2. Since haproxy has LUA, user could have needs for file descriptors that is impossible to count. Is there any real reason to keep "auto-calculated" ulimit-n option with very low values, based just on connection limits? Of cause, it is easy to set (for those who read docs very carefully =)), but some extra value could cover a few more cases "from the box", also making a bit less critical FD-related bugs. At least some warning in docs for this option should be useful. P.s. Looks like FD-limit was the reason for 503 with NOSRV and SC-flag that I've described here - https://www.mail-archive.com/haproxy@formilux.org/msg30405.html -- Wert Revon
