Dear all, unfortunately, the potential memory leak, first reported "Brad Zhang", Sept. 27 2013, proves to be a real memory leak:
pls. see earlier parts of my story in attached eMails below, thank you. We now have enlarged the time-window for the renegotiation process " reneg-sec 86400" -> 172800 and additionally added "reneg-bytes 1024000" to randomize the renegotiation phase of individual O-VPN clients and to prevent the situation, all connected clients to start to renegotiate within a time window of 5 minutes. We still recongized the memory consumption of the openvpn process to start to steadily increase, until the memory limit per process has become reached and openvpn crashed as following: Oct 15 12:13:15 ip-10-56-61-152 openvpn[21742]: openvpn_execve: unable to fork: Cannot allocate memory (errno=12) Oct 15 12:13:16 ip-10-56-61-152 openvpn[21742]: openvpn_execve: unable to fork: Cannot allocate memory (errno=12) Oct 15 12:13:16 ip-10-56-61-152 openvpn[21742]: Exiting due to fatal error Unfortunately, we never recognized, reserved memory for openvpn to start to decreased, which is another sign for a memory leak. Pls. can anyone provide me with a response, whether this mailing list is the right place to deal with this kind of problem? Thank you very much freundlich grüsst, kind regards, _________________________ _________________________ _________________________ _________________________ _________________________ ________________ Arno Odermatt Phone +41414454053 arno.oderm...@ch.schindler.com Schindler Elevator Ltd. | Corporate Research & Development | CRD/CO-BAT Zugerstr. 13 | CH-6030 Ebikon, Switzerland http://www.schindler.com _________________________ _________________________ _________________________ _________________________ _________________________ ________________ Please consider your environment. Schindler supports sustainable urban development with safe, reliable and ecologically sound mobility solutions. ----- Forwarded by Arno Odermatt/R&D/SCH/SCHINDLER on 17.10.2013 14:50 ----- From: Arno Odermatt/R&D/SCH/SCHINDLER To: openvpn-devel@lists.sourceforge.net, List-Post: openvpn-devel@lists.sourceforge.net Date: 10.10.2013 11:11 Subject: Openvpn 2.3.2 potential memory-leak after 24hours (addendum1) Dear all, we found our certificate renegotiating timer to be set to 86400secs = 24h. Therefore, we strongly assume, potentially a memory leak occurs in the renegotiating phase of the keys (passwords) as already earlier described in the mailing-list by "Brad Zhang" (Sept. 27. 2013) and treated/discussed by David Sommerseth. http://sourceforge.net/mailarchive/forum.php?thread_name=52456F61.3000104%40topphemmelig.net&forum_name=openvpn-devel our server side config-file: reneg-sec 86400 more /etc/openvpn/server.cnf ;log /var/log/openvpn_cgwvpn.log # Set server mode mode server float management tunnel 7505 port 443 proto tcp-server max-clients 5000 # Use Ethernet tunnel dev tap # The IPv4 addresses are used to establish the tunnel # All traffic is routed using IPv6 addresses ifconfig 172.16.0.1 255.255.128.0 ifconfig-pool 172.16.1.1 172.16.127.254 255.255.128.0 # Server certificate settings tls-server dh /etc/openvpn/servervpn/keys/dh1024.pem ca /etc/openvpn/servervpn/keys/ca.crt cert /etc/openvpn/servervpn/keys/CLTVpnSrv.crt key /etc/openvpn/servervpn/keys/CLTVpnSrv.key # Keep tunnel open persist-tun persist-key # keep-alive ping # -> keepalive 10 60 # will also be pushed to the client # if mode server: # ping 10 # ping-restart 120 # push "ping 10" # push "ping-restart 60" # else # ping 10 # ping-restart 60 keepalive 240 600 # Renegotiate data channel key once per day reneg-sec 86400 # optimize timout of slow GPRS connections tcp-queue-limit 256 tcp-nodelay tls-timeout 60 hand-window 120 sndbuf 262144 txqueuelen 400 bcast-buffers 4096 # enable LZO compression comp-lzo # moderate verbosity verb 0 mute 10 #Execute scripts script-security 2 system up /etc/openvpn/servervpn/vpn-up.sh down /etc/openvpn/servervpn/vpn-down.sh client-connect /etc/openvpn/servervpn/client-connect.sh client-disconnect /etc/openvpn/servervpn/client-disconnect.sh tmp-dir /var/tmp [root@localhost ~] Any reply on this and idea for a soonest fix would be appreciated thx and regards, AO From: arno.oderm...@ch.schindler.com To: openvpn-devel@lists.sourceforge.net, List-Post: openvpn-devel@lists.sourceforge.net Date: 09.10.2013 19:19 Subject: [Openvpn-devel] Openvpn 2.3.2 potential memory-leak after 24hours All, we are running Openvpn 2.3.2 on Fedora17/64Bit with 8GByte RAM total (EoL, yes we are aware) and seem to run into a memory leak situation, as following: - around 2955 Openvpn session running OK for 23'58" - memory allocated 3247mByte, memory reserved 2.5gByte, CPU utilization ~1% (see "status-prior-o-vpnrestart-131009.rtf") - script-security 2 (in openvpn.conf) After exactly 24hours, allocated and reserved memory of the openvpn-process start to increase in 1-2Mbyte steps, without seeing an increase in the amount of Openvpn sessions and steps up to around 3970mByte for the allocated and 3.2gByte for the reserved memory. Then Openvpn crashes! (see bottom "status-prior-o-vpnrestart-131009.rtf") The behavior of this symptom has been recognized in Openvpn 2.2.2-7 and 2.3.2 as following: > Openvpn 2.2.2-7: keeps reporting "external program fork failed" 2.2.2-7 endlessly keeps reporting a fork failed, when the client-connect script should be executed, no new openvpn sessions can be established but the process stays alive. WARNING: Failed running command (--client-connect): external program fork failed WARNING: Failed running command (--client-connect): external program fork failed WARNING: Failed running command (--client-connect): external program fork failed > Openvpn 2.3.2: exiting due to fatal error (openvpn is killed) When a certain memory stage from a perspective of allocated and reserved memory is reached, following scenario happens: Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: openvpn_execve: unable to fork: Cannot allocate memory (errno=12) Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: Exiting due to fatal error Please, can you analyze the issue and send us a feedback about the possible root cause and mitigation? snip log: ------------------------------ Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: openvpn_execve: unable to fork: Cannot allocate memory (errno=12) Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: Exiting due to fatal error Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: Closing TUN/TAP interface Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: /usr/sbin/ip addr del dev tap1 172.16.0.1/17 Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: openvpn_execve: unable to fork: Cannot allocate memory (errno=12) Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: Exiting due to fatal error ^C[ec2-user@localhost ~]$ [ec2-user@localhost ~]$ sudo tail -f /var/log/messages Oct 9 15:40:27 ip-10-56-61-152 openvpn[25964]: NL-200000134777/109.37.153.117:41299 NOTE: --mute triggered... Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: DE-2024460/139.7.160.82:42187 40 variation(s) on previous 10 message(s) suppressed by --mute Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: DE-2024460/139.7.160.82:42187 Connection reset, restarting [0] Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: DE-2024460/139.7.160.82:42187 SIGUSR1[soft,connection-reset] received, client-instance restarting Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: openvpn_execve: unable to fork: Cannot allocate memory (errno=12) Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: Exiting due to fatal error Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: Closing TUN/TAP interface Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: /usr/sbin/ip addr del dev tap1 172.16.0.1/17 Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: openvpn_execve: unable to fork: Cannot allocate memory (errno=12) Oct 9 15:40:29 ip-10-56-61-152 openvpn[25964]: Exiting due to fatal error thx and regards, AO _________________________ _________________________ _________________________ _________________________ _________________________ ________________ ****************************************************** Notice: The information contained in this message is intended only for use of the individual(s) named above and may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you are not the intended recipient of this message you are hereby notified that you must not use, disseminate , copy it in any form or take any action in reliance of it. If you have received this message in error please delete it and any copies of it and notify the sender immediately. *******************************************************[attachment "status-prior-o-vpnrestart-131009.rtf" deleted by Arno Odermatt/R&D/SCH/SCHINDLER] ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk _________________________ ______________________ Openvpn-devel mailing list Openvpn-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-devel ****************************************************** Notice: The information contained in this message is intended only for use of the individual(s) named above and may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you are not the intended recipient of this message you are hereby notified that you must not use, disseminate , copy it in any form or take any action in reliance of it. If you have received this message in error please delete it and any copies of it and notify the sender immediately. *******************************************************
