Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files
Dear Gert, thx. We also studied the situation from point of view of usage of resources. We beliefe, not to have run into a real resource problem, since we are far away fromthe different max-values, such as eg. max open files, max file-descriptor, max i-nodes per partition. We still can not imagine, why we saw "TOO MAN Y OPEN..." and what was the root cause. Anyway, pls find below the result of "$ ulimit -a" of that machine 3 OpenVpn Processes are running on, whereas there is no "nofile" -line to find: - 172-16-128-100 ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 29678 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 655350 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Normally, this value might be read out by using: less /proc/sys/fs/file-max 1527073 # Defintion of max open files less /proc/sys/fs/file-nr 36800 1527073 # Read out value of current situation We are unsure, if a file would be locked and O-VPN would try to write to it, whether in this situation we also would see as well " TOO MAN Y OPEN..." ??? regards Arno From: Gert Doering <g...@greenie.muc.de> To: arno.oderm...@ch.schindler.com, Cc: Gert Doering <g...@greenie.muc.de>, Eric Crist <ecr...@secure-computing.net>, Jan Just Keijser <janj...@nikhef.nl>, openvpn-devel@lists.sourceforge.net List-Post: openvpn-devel@lists.sourceforge.net Date: 24.07.2014 20:13 Subject: Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" Too many open files Hi, On Thu, Jul 24, 2014 at 05:22:37PM +0200, arno.oderm...@ch.schindler.com wrote: > I did the first part: > > NO, we are not using any Plugins, only client -connects scripts Mmmh, ok. > lsof -n | wc -l 4405 That doesn't tell much, except "the total number of open files in the system is 4405". > lsof -p 25211 > openvpn1.txt > lsof -p 25232 > openvpn2.txt > lsof -p 25252 > openvpn3.txt > > It looks like, we got much more then just some lines: > > openvpn3.txtopenvpn2.txtopenvpn1.txt If you look at the files (in attachment), you'll see that the large bulk of it is "TCP" - so your openvpn processes are using up the amount of file descriptors the system is willing to give them for TCP connects, as every TCP client needs to have it's own socket. If you run "ulimit -a" from the same environment where you start the OpenVPN processes, you'll see a line that looks like this: $ ulimit -a ... nofile (-n) 1024 that's the maximum number of file descriptors - subtract some 20-odd, and you have ~1000 left for about 1000 clients. $ ulimit -n 2000 can usually be used to raise that limit to 2000 (if run as root, in the same shell that starts openvpn later)... I'm not a Fedora expert, so maybe they have some other limitations, or ways to control the limits. gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany g...@greenie.muc.de fax: +49-89-35655025 g...@net.informatik.tu-muenchen.de [attachment "attpc7ko.dat" deleted by Arno Odermatt/R/SCH/SCHINDLER] ** Notice: The information contained in this message is intended only for use of the individual(s) named above and may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you are not the intended recipient of this message you are hereby notified that you must not use, disseminate , copy it in any form or take any action in reliance of it. If you have received this message in error please delete it and any copies of it and notify the sender immediately. ***
Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files
Hi, On Thu, Jul 24, 2014 at 05:22:37PM +0200, arno.oderm...@ch.schindler.com wrote: > I did the first part: > > NO, we are not using any Plugins, only client -connects scripts Mmmh, ok. > lsof -n | wc -l 4405 That doesn't tell much, except "the total number of open files in the system is 4405". > lsof -p 25211 > openvpn1.txt > lsof -p 25232 > openvpn2.txt > lsof -p 25252 > openvpn3.txt > > It looks like, we got much more then just some lines: > > openvpn3.txtopenvpn2.txtopenvpn1.txt If you look at the files (in attachment), you'll see that the large bulk of it is "TCP" - so your openvpn processes are using up the amount of file descriptors the system is willing to give them for TCP connects, as every TCP client needs to have it's own socket. If you run "ulimit -a" from the same environment where you start the OpenVPN processes, you'll see a line that looks like this: $ ulimit -a ... nofile (-n) 1024 that's the maximum number of file descriptors - subtract some 20-odd, and you have ~1000 left for about 1000 clients. $ ulimit -n 2000 can usually be used to raise that limit to 2000 (if run as root, in the same shell that starts openvpn later)... I'm not a Fedora expert, so maybe they have some other limitations, or ways to control the limits. gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany g...@greenie.muc.de fax: +49-89-35655025g...@net.informatik.tu-muenchen.de pgpR0YHSeqWHZ.pgp Description: PGP signature
Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files
Hi, On Wed, Jul 23, 2014 at 11:43:27PM +0200, Jan Just Keijser wrote: > >Gert, we are sure, there was not a problem with the resources (eg.: > >max open files, max filed descriptors, etc.) on the system. > >What else can I do about it? > > > try debugging this by adding some statements to the client-connect > script, e.g. > > ls -l /proc/self/fd/* | wc -l >> /tmp/debug.log > sysctl fs.file-nr >> /tmp/debug.log > > to find out if there really are no more filehandles available. > For each connection a new client-connect script is started, so I doubt > that any non-closed file handles will accumulate and cause this problem. The error message came from OpenVPN, so if there is a leak, it's in OpenVPN, not in the script. OTOH, we use openvpn (git master) with client-connect scripts and per-user generated configs on a moderately-loaded system, and have not run into fd exhaustion issues - and lsof doesn't print anything interesting either, so it must be something more interesting than a forgotten close() call... Arno, are you using plugins? (A plugin runs in the OpenVPN context, and if a plugin leaks FDs, it will affect OpenVPN as well) How does a "lsof -p $processid" for the OpenVPN process look like after it has run for a few hours? Mine shows about 20 lines, and most of them are shared libraries (/lib/...*.so.*) - if it is significantly more for you, there is the problem. gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany g...@greenie.muc.de fax: +49-89-35655025g...@net.informatik.tu-muenchen.de pgpM_z8KSwnNg.pgp Description: PGP signature
Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files
Hi, On 23/07/14 08:19, arno.oderm...@ch.schindler.com wrote: Dear both, thank you for your reply. Yes, we are using the "--client-connect" and according to 2.3 OpenVPN manual (see section below) it does create files by writing to "file named by $1." Gert, we are sure, there was not a problem with the resources (eg.: max open files, max filed descriptors, etc.) on the system. What else can I do about it? try debugging this by adding some statements to the client-connect script, e.g. ls -l /proc/self/fd/* | wc -l >> /tmp/debug.log sysctl fs.file-nr >> /tmp/debug.log to find out if there really are no more filehandles available. For each connection a new client-connect script is started, so I doubt that any non-closed file handles will accumulate and cause this problem. HTH, JJK --client-connect script Run script on client connection. The script is passed the common name and IP address of the just-authenticated client as environmental variables (see environmental variable section below). The script is also passed the pathname of a not-yet-created temporary file as $1 (i.e. the first command line argument), to be used by the script to pass dynamically generated config file directives back to OpenVPN. If the script wants to generate a dynamic config file to be applied on the server when the client connects, it should write it to the file named by $1. thank you again Arno From: Gert Doering <g...@greenie.muc.de> To: Eric Crist <ecr...@secure-computing.net>, Cc: arno.oderm...@ch.schindler.com, openvpn-devel@lists.sourceforge.net Date: 22.07.2014 21:34 Subject: Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" Too many open files Hi, On Tue, Jul 22, 2014 at 12:37:19PM -0500, Eric Crist wrote: > This isn't an OpenVPN problem, directly. It appears you have a client connect script, or are storing connection information in temp files. You can increase the maximum allowed open files in Fedora (you'll have to research that yourself). Alternatively, stop storing connection data in a temp file for new connections. OpenVPN, by itself, does not create these temporary files. Uh, this is not fully correct. If you use --client-connect (or any of the other up scripts that enable passing of config values back), OpenVPN will create the temp file to be used for that, to avoid race conditions. I'm not sure when this was added, but "recentish" (2.2.x) Now the sema-files do not ring a bell, but we need to check whether we properly clean up the other files - but that should be visible in "lsof" while OpenVPN is running and after a few connections have completed. gert
Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files
Am 23.07.2014 08:19, schrieb arno.oderm...@ch.schindler.com: > Dear both, > > thank you for your reply. > Yes, we are using the "--client-connect" and according to 2.3 OpenVPN > manual (see section below) it does create files by writing to "file > named by $1." > > Gert, we are sure, there was not a problem with the resources (eg.: max > open files, max filed descriptors, etc.) on the system. > What else can I do about it? Do you impose lower per-process or per-group limits? Did you exhaust the number of inodes on the /var partition (although you should normally get ENOSPC in that situation). Other than that, the proposal is to check lsof output with a few connections there and also a few gone already, in order to see if there are leaks.
Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files
Dear both, thank you for your reply. Yes, we are using the "--client-connect" and according to 2.3 OpenVPN manual (see section below) it does create files by writing to "file named by $1." Gert, we are sure, there was not a problem with the resources (eg.: max open files, max filed descriptors, etc.) on the system. What else can I do about it? --client-connect script Run script on client connection. The script is passed the common name and IP address of the just-authenticated client as environmental variables (see environmental variable section below). The script is also passed the pathname of a not-yet-created temporary file as $1 (i.e. the first command line argument), to be used by the script to pass dynamically generated config file directives back to OpenVPN. If the script wants to generate a dynamic config file to be applied on the server when the client connects, it should write it to the file named by $1. thank you again Arno From: Gert Doering <g...@greenie.muc.de> To: Eric Crist <ecr...@secure-computing.net>, Cc: arno.oderm...@ch.schindler.com, openvpn-devel@lists.sourceforge.net List-Post: openvpn-devel@lists.sourceforge.net Date: 22.07.2014 21:34 Subject: Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" Too many open files Hi, On Tue, Jul 22, 2014 at 12:37:19PM -0500, Eric Crist wrote: > This isn't an OpenVPN problem, directly. It appears you have a client connect script, or are storing connection information in temp files. You can increase the maximum allowed open files in Fedora (you'll have to research that yourself). Alternatively, stop storing connection data in a temp file for new connections. OpenVPN, by itself, does not create these temporary files. Uh, this is not fully correct. If you use --client-connect (or any of the other up scripts that enable passing of config values back), OpenVPN will create the temp file to be used for that, to avoid race conditions. I'm not sure when this was added, but "recentish" (2.2.x) Now the sema-files do not ring a bell, but we need to check whether we properly clean up the other files - but that should be visible in "lsof" while OpenVPN is running and after a few connections have completed. gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany g...@greenie.muc.de fax: +49-89-35655025 g...@net.informatik.tu-muenchen.de [attachment "attp7yyb.dat" deleted by Arno Odermatt/R/SCH/SCHINDLER] ** Notice: The information contained in this message is intended only for use of the individual(s) named above and may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you are not the intended recipient of this message you are hereby notified that you must not use, disseminate , copy it in any form or take any action in reliance of it. If you have received this message in error please delete it and any copies of it and notify the sender immediately. ***
Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files
Hi, On Tue, Jul 22, 2014 at 12:37:19PM -0500, Eric Crist wrote: > This isn't an OpenVPN problem, directly. It appears you have a client > connect script, or are storing connection information in temp files. You can > increase the maximum allowed open files in Fedora (you'll have to research > that yourself). Alternatively, stop storing connection data in a temp file > for new connections. OpenVPN, by itself, does not create these temporary > files. Uh, this is not fully correct. If you use --client-connect (or any of the other up scripts that enable passing of config values back), OpenVPN will create the temp file to be used for that, to avoid race conditions. I'm not sure when this was added, but "recentish" (2.2.x) Now the sema-files do not ring a bell, but we need to check whether we properly clean up the other files - but that should be visible in "lsof" while OpenVPN is running and after a few connections have completed. gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany g...@greenie.muc.de fax: +49-89-35655025g...@net.informatik.tu-muenchen.de pgpAH7R1Bc6x0.pgp Description: PGP signature
Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files
This isn't an OpenVPN problem, directly. It appears you have a client connect script, or are storing connection information in temp files. You can increase the maximum allowed open files in Fedora (you'll have to research that yourself). Alternatively, stop storing connection data in a temp file for new connections. OpenVPN, by itself, does not create these temporary files. - Eric F Crist On Jul 22, 2014, at 11:24:54, arno.oderm...@ch.schindler.com wrote: > Dear all, > > we are driving O-VPN 2.3.2 on Fedora20. > Since we have quit many permanently connected O-VPN clients, we have started > three O-VPN processes, listening on three different ports and setting up > three different tap interfaces: > > Today, all three O-VPN processes crashed suddenly, whereas we found following > error: > > ip-172-16-128-101 openvpn[654]: /172.16.253.10:44214 Could not create > temporary file '/var/tmp/openvpn_cc_1bd37815cbacd70936015e40e25198aa.tmp': > Too many open files > > We did not find any helpful information, neither in the the mail-archives, > nor in other forums/panels, beside something related to user/password > authentication (openvpn-auth-pam), which we are not using (using TLS-server) > and also lsof did not provide any helpful information to correlate this error > to a (file-) resource problem > https://forums.openvpn.net/topic13474.html > https://community.openvpn.net/openvpn/ticket/201 > > > > After this happened, we found: > > - in /tmp:-rw-r--r--. 1 root0 Jul 18 10:51 vpn3_sema_15198 > #sema files laying around > > - in /var/tmp-rw---. 1 root0 Jul 18 10:51 > openvpn_cc_0e211df697b9f5620da89bd05f44ef48.tmp > > > Deleting of the sema-files and restarting O-VPN brought back everything to > life. > > Has anybody ever experienced something similar, can this be a bug and what > could be the corrective action to overcome, this to repeat again? > > Thank you for any help in this > > Ar > > > ** > Notice: The information contained in this message is intended only for use of > the individual(s) named above and may contain confidential, proprietary or > legally privileged information. No confidentiality or privilege is waived or > lost by any mistransmission. If you are not the intended recipient of this > message you are hereby notified that you must not use, disseminate , copy it > in any form or take any action in reliance of it. If you have received this > message in error please delete it and any copies of it and notify the sender > immediately. > *** > -- > Want fast and easy access to all the code in your enterprise? Index and > search up to 200,000 lines of code with a free copy of Black Duck > Code Sight - the same software that powers the world's largest code > search on Ohloh, the Black Duck Open Hub! Try it now. > http://p.sf.net/sfu/bds___ > Openvpn-devel mailing list > Openvpn-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/openvpn-devel signature.asc Description: Message signed with OpenPGP using GPGMail
