Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files

[arno . odermatt]

Dear Gert,

thx.
We also studied the situation from point of view of usage of resources. We 
beliefe, not to have run into a real resource problem, since we are far 
away fromthe different max-values, such as eg. max open files, max 
file-descriptor, max i-nodes per partition.
We still can not imagine, why we saw "TOO MAN Y OPEN..." and what was the 
root cause.

Anyway, pls find below the result of   "$ ulimit -a" of that machine 3 
OpenVpn Processes are running on, whereas there is no "nofile" -line to 
find:

- 

172-16-128-100 ~]$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 29678
max locked memory   (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files  (-n) 655350
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 1024
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

Normally, this value might be read out by using: 

less /proc/sys/fs/file-max  1527073 # Defintion of max open 
files 
less  /proc/sys/fs/file-nr   36800   1527073  
# Read out value of current situation

We are unsure, if a file would be locked and O-VPN would try to write to 
it, whether in this situation we also would see as well " TOO MAN Y 
OPEN..." ???

regards

Arno






From:   Gert Doering <g...@greenie.muc.de>
To: arno.oderm...@ch.schindler.com, 
Cc: Gert Doering <g...@greenie.muc.de>, Eric Crist 
<ecr...@secure-computing.net>, Jan Just Keijser <janj...@nikhef.nl>, 
openvpn-devel@lists.sourceforge.net
List-Post: openvpn-devel@lists.sourceforge.net
Date:   24.07.2014 20:13
Subject:Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create 
temporary file" Too many open files



Hi,

On Thu, Jul 24, 2014 at 05:22:37PM +0200, arno.oderm...@ch.schindler.com 
wrote:
> I did the first part:
> 
> NO, we are not using any Plugins, only client -connects scripts

Mmmh, ok.

> lsof -n | wc -l   4405 

That doesn't tell much, except "the total number of open files in
the system is 4405".

> lsof -p 25211 > openvpn1.txt
> lsof -p 25232 > openvpn2.txt
> lsof -p 25252 > openvpn3.txt
> 
> It looks like, we got much more then just some lines:
> 
> openvpn3.txtopenvpn2.txtopenvpn1.txt

If you look at the files (in attachment), you'll see that the large bulk
of it is "TCP" - so your openvpn processes are using up the amount of file
descriptors the system is willing to give them for TCP connects, as every
TCP client needs to have it's own socket.

If you run "ulimit -a" from the same environment where you start the
OpenVPN processes, you'll see a line that looks like this:

$ ulimit -a
...
nofile (-n)  1024

that's the maximum number of file descriptors - subtract some 20-odd,
and you have ~1000 left for about 1000 clients.

$ ulimit -n 2000

can usually be used to raise that limit to 2000 (if run as root, in the 
same shell that starts openvpn later)...  I'm not a Fedora expert, so 
maybe they have some other limitations, or ways to control the limits.

gert


-- 
USENET is *not* the non-clickable part of WWW!
 //www.muc.de/~gert/
Gert Doering - Munich, Germany g...@greenie.muc.de
fax: +49-89-35655025 g...@net.informatik.tu-muenchen.de
[attachment "attpc7ko.dat" deleted by Arno Odermatt/R/SCH/SCHINDLER] 


**
Notice: The information contained in this message is intended only for use of 
the individual(s) named above and may contain confidential, proprietary or 
legally privileged information. No confidentiality or privilege is waived or 
lost by any mistransmission. If you are not the intended recipient of this 
message you are hereby notified that you must not use, disseminate , copy it in 
any form or take any action in reliance of it. If you have received this 
message in error please delete it and any copies of it and notify the sender 
immediately.
***

Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files

[arno . odermatt]

Dear both,

thank you for your reply.
Yes, we are using the "--client-connect" and according to 2.3 OpenVPN 
manual (see section below) it does create files by writing to "file named 
by $1."

Gert, we are sure, there was not a problem with the resources (eg.: max 
open files, max filed descriptors, etc.) on the system.
What else can I do about it?


--client-connect script
Run script on client connection. The script is passed the common name and 
IP address of the just-authenticated client as environmental variables 
(see environmental variable section below). The script is also passed the 
pathname of a not-yet-created temporary file as $1 (i.e. the first command 
line argument), to be used by the script to pass dynamically generated 
config file directives back to OpenVPN.
If the script wants to generate a dynamic config file to be applied on the 
server when the client connects, it should write it to the file named by 
$1.


thank you again

Arno






From:   Gert Doering <g...@greenie.muc.de>
To: Eric Crist <ecr...@secure-computing.net>, 
Cc: arno.oderm...@ch.schindler.com, 
openvpn-devel@lists.sourceforge.net
List-Post: openvpn-devel@lists.sourceforge.net
Date:   22.07.2014 21:34
Subject:Re: [Openvpn-devel] Openvpn 2.3.2: "Could not create 
temporary file" Too many open files



Hi,

On Tue, Jul 22, 2014 at 12:37:19PM -0500, Eric Crist wrote:
> This isn't an OpenVPN problem, directly.  It appears you have a client 
connect script, or are storing connection information in temp files.  You 
can increase the maximum allowed open files in Fedora (you'll have to 
research that yourself).  Alternatively, stop storing connection data in a 
temp file for new connections.  OpenVPN, by itself, does not create these 
temporary files.

Uh, this is not fully correct.  If you use --client-connect (or any of
the other up scripts that enable passing of config values back), OpenVPN
will create the temp file to be used for that, to avoid race conditions.

I'm not sure when this was added, but "recentish" (2.2.x)

Now the sema-files do not ring a bell, but we need to check whether we
properly clean up the other files - but that should be visible in "lsof"
while OpenVPN is running and after a few connections have completed.

gert

-- 
USENET is *not* the non-clickable part of WWW!
 //www.muc.de/~gert/
Gert Doering - Munich, Germany g...@greenie.muc.de
fax: +49-89-35655025 g...@net.informatik.tu-muenchen.de
[attachment "attp7yyb.dat" deleted by Arno Odermatt/R/SCH/SCHINDLER] 


**
Notice: The information contained in this message is intended only for use of 
the individual(s) named above and may contain confidential, proprietary or 
legally privileged information. No confidentiality or privilege is waived or 
lost by any mistransmission. If you are not the intended recipient of this 
message you are hereby notified that you must not use, disseminate , copy it in 
any form or take any action in reliance of it. If you have received this 
message in error please delete it and any copies of it and notify the sender 
immediately.
***

[Openvpn-devel] Openvpn 2.3.2: "Could not create temporary file" ....Too many open files

[arno . odermatt]

Dear all,

we are driving O-VPN 2.3.2 on Fedora20.
Since we have quit many permanently connected O-VPN clients, we have 
started three O-VPN processes, listening on three different ports and 
setting up three different tap interfaces:

Today, all three O-VPN processes crashed suddenly, whereas we found 
following error:

 ip-172-16-128-101 openvpn[654]: /172.16.253.10:44214 Could not create 
temporary file '/var/tmp/openvpn_cc_1bd37815cbacd70936015e40e25198aa.tmp': 
Too many open files

We did not find any helpful information, neither in the the mail-archives, 
nor in other forums/panels, beside something related to user/password 
authentication (openvpn-auth-pam), which we are not using (using 
TLS-server) and also lsof did not provide any helpful information to 
correlate this error to a (file-) resource problem

https://forums.openvpn.net/topic13474.html 
https://community.openvpn.net/openvpn/ticket/201 


After this happened, we found:

- in /tmp:  -rw-r--r--.  1 root0 Jul 18 10:51 vpn3_sema_15198 
#sema files laying around

- in /var/tmp   -rw---.  1 root0 Jul 18 10:51 
openvpn_cc_0e211df697b9f5620da89bd05f44ef48.tmp


Deleting of the sema-files and restarting O-VPN brought back everything to 
life.

Has anybody ever experienced something similar, can this be a bug and what 
could be the corrective action to overcome, this to repeat again?

Thank you for any help in this

Ar



**
Notice: The information contained in this message is intended only for use of 
the individual(s) named above and may contain confidential, proprietary or 
legally privileged information. No confidentiality or privilege is waived or 
lost by any mistransmission. If you are not the intended recipient of this 
message you are hereby notified that you must not use, disseminate , copy it in 
any form or take any action in reliance of it. If you have received this 
message in error please delete it and any copies of it and notify the sender 
immediately.
***

[Openvpn-devel] Q: Multi threading and roadmap

[arno . odermatt]

Hi,

I found the roadmap https://community.openvpn.net/openvpn/wiki/RoadMap, 
where the thoughts about Monolithic architecture vs. "scaling OpenVPN 
across threads " have been discussed.

Will the aspects of OpenVPN (3.x??)  to be capable to run in multi thread 
mode be available some when soon?

Thank you for any reply








**
Notice: The information contained in this message is intended only for use of 
the individual(s) named above and may contain confidential, proprietary or 
legally privileged information. No confidentiality or privilege is waived or 
lost by any mistransmission. If you are not the intended recipient of this 
message you are hereby notified that you must not use, disseminate , copy it in 
any form or take any action in reliance of it. If you have received this 
message in error please delete it and any copies of it and notify the sender 
immediately.
***

Re: [Openvpn-devel] Does openvpn 2.3.2 has memory leak? (addendum3 and 4)

[arno . odermatt]

Dear Brad,

thank you. Yes, it is acceptable.
>From our point of view, it still seems, memory not to be released 
properly, when a disconnection script has been run.
In our environment with O-VPN 2.3.2, when we start to disconnect large 
amounts of client sessions, using the disconnect script, memory seems not 
to become freed, and therefore can not be re-used by other sessions. This 
is a problem, since the app might  run out of memory again.

freundlich grüsst,
kind regards,

_
_
_
_
_
 

Arno Odermatt 
Phone +41414454053 
arno.oderm...@ch.schindler.com 

Schindler Elevator Ltd. | Corporate Research & Development | CRD/CO-BAT 
Zugerstr. 13 | CH-6030 Ebikon, Switzerland 
http://www.schindler.com 
_
_
_
_
_
 

Please consider your environment. 

Schindler supports sustainable urban development with
safe, reliable and ecologically sound mobility solutions.





From:   Brad Zhang <hebei5...@gmail.com>
To: Gert Doering <g...@greenie.muc.de>, 
arno.oderm...@ch.schindler.com, 
Cc: openvpn-devel@lists.sourceforge.net
List-Post: openvpn-devel@lists.sourceforge.net
Date:   05.12.2013 09:11
Subject:Re: [Openvpn-devel] Does openvpn 2.3.2 has memory leak? 
(addendum3 and 4)



Hi Arno,

It is acceptable for me that memory can be used again in new connection. 


Hi Gert,

It is very helpful, thank you so much for your help. Thanks to all of you 
for developing  so excellent and powerful  open source project (also 
complicated for me:) ). Look forward to openvpn 2.3.3. 

Thanks,
Brad Zhang


On Fri, Nov 22, 2013 at 6:15 PM, Gert Doering <g...@greenie.muc.de> wrote:
Hi,

On Fri, Nov 22, 2013 at 10:51:27AM +0100, arno.oderm...@ch.schindler.com 
wrote:
> Dear Gert, Brad all,
>
> as I also had reported to have recognized, that memory was not freed, 
even
>
> though the openvpn disconnect-script was disconnecting all of the 3400
> Openvpn-sessions (pls see mail below).
> Would you think, this is also improved by the change from malloc to
> calloc, so we could update to 2.3.3 to improve the memory-leak 
situation?

Using malloc() or calloc() will neither affect the total memory used,
nor any fragmentation issues or mem leaks.  It's just about guaranteeing
that allocated memory is zeroed-out.

gert
--
USENET is *not* the non-clickable part of WWW!
   //
www.muc.de/~gert/
Gert Doering - Munich, Germany 
g...@greenie.muc.de
fax: +49-89-35655025
g...@net.informatik.tu-muenchen.de


**
Notice: The information contained in this message is intended only for use of 
the individual(s) named above and may contain confidential, proprietary or 
legally privileged information. No confidentiality or privilege is waived or 
lost by any mistransmission. If you are not the intended recipient of this 
message you are hereby notified that you must not use, disseminate , copy it in 
any form or take any action in reliance of it. If you have received this 
message in error please delete it and any copies of it and notify the sender 
immediately.
***

Re: [Openvpn-devel] Openvpn 2.3.2 potential memory-leak after 24hours (addendum3)

[arno . odermatt]

Dear Gert,


thank you so much.
Meanwhile, we can define the substantial memory growth for the phase of 
the first renegotiation cycle.
Memory is growing by almost double.
Surely, if one had only 50 to 100 concurrent sessions, it is not quit 
hurting anything. But driving the amount of concurrent session as we got 
it in our scenario, memory treatment seems to become key.


kind regards,

_
 

Arno 




From:   Gert Doering <g...@greenie.muc.de>
To: arno.oderm...@ch.schindler.com, 
Cc: Gert Doering <g...@greenie.muc.de>, 
openvpn-devel@lists.sourceforge.net
List-Post: openvpn-devel@lists.sourceforge.net
Date:   22.10.2013 13:43
Subject:Re: Openvpn 2.3.2 potential memory-leak after 24hours 
(addendum3)



Hi,

On Mon, Oct 21, 2013 at 10:34:27AM +0200, arno.oderm...@ch.schindler.com 
wrote:
> thank you for replying.
> Do you see any time window, you potentially could have time for this?

I just gave it an initial try (using the "master" branch but that code
should have the same issues, if any, as no (known) fixes have been 
commited).

Valgrind isn't telling me about any clear mem-leaks yet...

==1953== HEAP SUMMARY:
==1953== in use at exit: 2,560 bytes in 52 blocks
==1953==   total heap usage: 52,139 allocs, 52,087 frees, 15,723,723 bytes 
allocated

... but I could see memory go up in "while sleep 1; do ps wwup 1953 ; 
done" 
listings at exactly the point in time where renegotiation was taking 
place,
with two clients connected to it.

I could use some ideas on how to debug this further - if valgrind isn't
complaining, normally our memory management should be OK, but why is
VSZ/RSZ still growing, then...?

gert
-- 
USENET is *not* the non-clickable part of WWW!
 //www.muc.de/~gert/
Gert Doering - Munich, Germany g...@greenie.muc.de
fax: +49-89-35655025 g...@net.informatik.tu-muenchen.de
[attachment "attmxcg3.dat" deleted by Arno Odermatt/R/SCH/SCHINDLER] 


**
Notice: The information contained in this message is intended only for use of 
the individual(s) named above and may contain confidential, proprietary or 
legally privileged information. No confidentiality or privilege is waived or 
lost by any mistransmission. If you are not the intended recipient of this 
message you are hereby notified that you must not use, disseminate , copy it in 
any form or take any action in reliance of it. If you have received this 
message in error please delete it and any copies of it and notify the sender 
immediately.
***

[Openvpn-devel] Openvpn 2.3.2 potential memory-leak after 24hours (addendum3)

[arno . odermatt]

Good Morning,

thank you for replying.
Do you see any time window, you potentially could have time for this?

Meanwhile, we have been able to rise the memory limits of the system be 
adding to  /etc/sysctl.conf:
   "vm.overcommit_ratio = 80"
   "vm.overcommit_memory = 2"

and by adding "virtual memory" by increasing the amount of swap space.

Using this setup, we now were able to successfully walk through the first 
re-negotiation cycle, which really seems to double memory consumption, 
which never is being freed, after the re-neg process is completed.


 regards,

Arno






From:   Gert Doering 
To: arno.oderm...@ch.schindler.com, 
Cc: openvpn-devel@lists.sourceforge.net
List-Post: openvpn-devel@lists.sourceforge.net
Date:   17.10.2013 16:19
Subject:Re: [Openvpn-devel] Openvpn 2.3.2 potential memory-leak 
after 24hours (addendum2)



Hi,

On Thu, Oct 17, 2013 at 03:47:44PM +0200, arno.oderm...@ch.schindler.com 
wrote:
> Pls. can anyone provide me with a response, whether this mailing list is 

> the right place to deal with this kind of problem?

It is the right place to deal with this kind of problems.

The problem is that nobody seems to have time to start debugging this 
into more detail yet.  What I'd try (if I had time) is to compile 
OpenVPN with valgrind (a memory leak searching tool), run it through
a few client key renegotiations, and then check the output to see
whether it shows anything "obvious".

In any case, thanks for tracing this so far and making it clear to us
that we have an issue there.  Even if it looks like nobody is caring.

gert

-- 
USENET is *not* the non-clickable part of WWW!
 //www.muc.de/~gert/
Gert Doering - Munich, Germany g...@greenie.muc.de
fax: +49-89-35655025 g...@net.informatik.tu-muenchen.de



**
Notice: The information contained in this message is intended only for use of 
the individual(s) named above and may contain confidential, proprietary or 
legally privileged information. No confidentiality or privilege is waived or 
lost by any mistransmission. If you are not the intended recipient of this 
message you are hereby notified that you must not use, disseminate , copy it in 
any form or take any action in reliance of it. If you have received this 
message in error please delete it and any copies of it and notify the sender 
immediately.
***

[Openvpn-devel] Openvpn 2.3.2 potential memory-leak after 24hours (addendum2)

[arno . odermatt]

Dear all,

unfortunately, the potential memory leak, first reported "Brad Zhang", 
Sept. 27 2013, proves to be a real memory leak:

pls. see earlier parts of my story in attached eMails below, thank you.

We now have enlarged the time-window for the renegotiation process "
reneg-sec 86400" ->  172800 and additionally added "reneg-bytes 1024000" 
to randomize the renegotiation phase of individual O-VPN clients and to 
prevent the situation, all connected clients to start to renegotiate 
within a time window of 5 minutes.

We still recongized the memory consumption of the openvpn process to start 

to steadily increase, until the memory limit per process has become 
reached and openvpn crashed as following:

Oct 15 12:13:15 ip-10-56-61-152 openvpn[21742]: openvpn_execve: unable to 
fork: Cannot allocate memory (errno=12)
Oct 15 12:13:16 ip-10-56-61-152 openvpn[21742]: openvpn_execve: unable to 
fork: Cannot allocate memory (errno=12)
Oct  15 12:13:16 ip-10-56-61-152 openvpn[21742]: Exiting due to fatal 
error
Unfortunately, we never recognized, reserved memory for openvpn to start 
to decreased, which is another sign for a memory leak.

Pls. can anyone provide me with a response, whether this mailing list is 
the right place to deal with this kind of problem?
Thank you very much


freundlich grüsst,
kind regards,

_
_
_
_
_
 

Arno Odermatt 
Phone +41414454053 
arno.oderm...@ch.schindler.com 

Schindler Elevator Ltd. | Corporate Research & Development | CRD/CO-BAT 
Zugerstr. 13 | CH-6030 Ebikon, Switzerland 
http://www.schindler.com 
_
_
_
_
_
 

Please consider your environment. 

Schindler supports sustainable urban development with
safe, reliable and ecologically sound mobility solutions.


- Forwarded by Arno Odermatt/R/SCH/SCHINDLER on 17.10.2013 14:50 
-

From:   Arno Odermatt/R/SCH/SCHINDLER
To: openvpn-devel@lists.sourceforge.net, 
List-Post: openvpn-devel@lists.sourceforge.net
Date:   10.10.2013 11:11
Subject:Openvpn 2.3.2 potential memory-leak after 24hours 
(addendum1)


Dear all,

we found our certificate renegotiating timer to be set to 86400secs = 24h.
Therefore, we strongly assume, potentially a memory leak occurs in the 
renegotiating phase of the keys (passwords) as already earlier described 
in the mailing-list by "Brad Zhang" (Sept. 27. 2013)  and 
treated/discussed by David Sommerseth.


http://sourceforge.net/mailarchive/forum.php?thread_name=52456F61.3000104%40topphemmelig.net_name=openvpn-devel


our server side config-file:
reneg-sec 86400


 more /etc/openvpn/server.cnf 

;log /var/log/openvpn_cgwvpn.log

# Set server mode
mode server
float
management tunnel 7505
port 443
proto tcp-server
max-clients 5000

# Use Ethernet tunnel
dev tap

# The IPv4 addresses are used to establish the tunnel
# All traffic is routed using IPv6 addresses
ifconfig 172.16.0.1 255.255.128.0
ifconfig-pool 172.16.1.1 172.16.127.254 255.255.128.0

# Server certificate settings
tls-server
dh /etc/openvpn/servervpn/keys/dh1024.pem
ca /etc/openvpn/servervpn/keys/ca.crt
cert /etc/openvpn/servervpn/keys/CLTVpnSrv.crt
key /etc/openvpn/servervpn/keys/CLTVpnSrv.key

# Keep tunnel open
persist-tun
persist-key

# keep-alive ping
# -> keepalive 10 60
# will also be pushed to the client
# if mode server:
#   ping 10
#   ping-restart 120
#   push "ping 10"
#   push "ping-restart 60"
# else
#   ping 10
#   ping-restart 60
keepalive 240 600

# Renegotiate data channel key once per day
reneg-sec 86400

# optimize timout of slow GPRS connections
tcp-queue-limit 256
tcp-nodelay
tls-timeout 60
hand-window 120
sndbuf 262144
txqueuelen 400
bcast-buffers 4096

# enable LZO compression
comp-lzo

# moderate verbosity
verb 0
mute 10

#Execute scripts
script-security 2 system
up /etc/openvpn/servervpn/vpn-up.sh
down /etc/openvpn/servervpn/vpn-down.sh
client-connect /etc/openvpn/servervpn/client-connect.sh
client-disconnect /etc/openvpn/servervpn/client-disconnect.sh
tmp-dir /var/tmp
[root@localhost ~]

Any reply on this and idea for a soonest fix would be appreciated

thx and regards, 

AO





From:   arno.oderm...@ch.schindler.com
To: openvpn-devel@lists.sourceforge.net, 
List-Post: openvpn-devel@lists.sourceforge.net
Date:   09.10.2013 19:19
Subject:[Openvpn-devel] Openvpn 2.3.2 potential memory-leak after 
24hours



All, 

we are running Openvpn 2.3.2 on Fedora17/64Bit with 8GByte RAM total (EoL, 

yes we are aware) and seem to run into a memory leak situation, as 
following:

- around 2955 Openvpn session running OK for 23'58"
- memory allocated 3247mByte, memory reserved 2.5gByte, CPU utilization 
~1% (see "status-prior-o-vpnrestart-131009.rtf")