Doing some more research on the issue led to the conclusion that the
number of clients are increasing the frequency of the crash.
The crash seems not related to a special client openssl version,
although many clients are openwrt linux boxes with a recent openssl
version. Consequently, the chance to see a openwrt client being
handshaking is high.
Mar 2 18:32:34 astakos openvpn[12886]: alexandh_home/89.12.82.13:46888
VERIFY OK: depth=1, CN=Weimar OpenVPN
Mar 2 18:32:35 astakos openvpn[3644]: OpenVPN 2.3.6
x86_64-unknown-openbsd5.7 [SSL (OpenSSL)] [LZO] [MH] [IPv6] built on Feb
27 2015
Mar 2 18:32:35 astakos openvpn[3644]: library versions: LibreSSL 2.1,
LZO 2.08
I recompiled the OpenVPN version with debug flags and hope to see a more
complete backtrace soon.
The backtrace from the openbsd package looks like this:
#0 0x000012d1fb25d170 in memcpy (dst0=0x12d1e9f18348,
src0=0x12d153579000, length=Variable "length" is not available. ) at
/usr/src/lib/libc/string/memcpy.c:96 96
/usr/src/lib/libc/string/memcpy.c: No such file or directory. in
/usr/src/lib/libc/string/memcpy.c (gdb) bt
#0 0x000012d1fb25d170 in memcpy (dst0=0x12d1e9f18348,
src0=0x12d153579000, length=Variable "length" is not available. ) at
/usr/src/lib/libc/string/memcpy.c:96
#1 0x000012cf2676d1fe in pem_password_callback () from
/usr/local/sbin/openvpn
#2 0x000012cf2676b162 in pem_password_callback () from
/usr/local/sbin/openvpn
#3 0x000012cf2676d547 in verify_callback () from
/usr/local/sbin/openvpn
#4 0x000012d1f8238d8c in internal_verify (ctx=0x7f7ffffef260) at
/usr/src/lib/libcrypto/crypto/../../libssl/src/crypto/x509/x509_vfy.c:1613
#5 0x000012d1f823a435 in X509_verify_cert (ctx=0x7f7ffffef260) at
/usr/src/lib/libcrypto/crypto/../../libssl/src/crypto/x509/x509_vfy.c:374
#6 0x000012d1fd9aa8d0 in ssl_verify_cert_chain (s=0x12d19ee9bc00,
sk=Variable "sk" is not available. ) at
/usr/src/lib/libssl/ssl/../../libssl/src/ssl/ssl_cert.c:459
#7 0x000012d1fd983260 in ssl3_get_client_certificate (s=0x12d19ee9bc00)
at /usr/src/lib/libssl/ssl/../../libssl/src/ssl/s3_srvr.c:2540
#8 0x000012d1fd986c68 in ssl3_accept (s=0x12d19ee9bc00) at
/usr/src/lib/libssl/ssl/../../libssl/src/ssl/s3_srvr.c:511
#9 0x000012d1fd993f75 in ssl3_read_bytes (s=0x12d19ee9bc00, type=23,
buf=0x12d1fec9a800 "", len=2048, peek=0) at
/usr/src/lib/libssl/ssl/../../libssl/src/ssl/s3_pkt.c:896
#10 0x000012d1fd9a754e in ssl3_read_internal (s=0x12d19ee9bc00,
buf=0x12d1fec9a800, len=2048, peek=0) at
/usr/src/lib/libssl/ssl/../../libssl/src/ssl/s3_lib.c:2784
#11 0x000012d1fd97b761 in ssl_read (b=0x12d1b40c8880,
out=0x12d1fec9a800 "", outl=2048) at
/usr/src/lib/libssl/ssl/../../libssl/src/ssl/bio_ssl.c:156
#12 0x000012d1f827b43f in BIO_read (b=0x12d1b40c8880,
out=0x12d1fec9a800, outl=2048) at
/usr/src/lib/libcrypto/crypto/../../libssl/src/crypto/bio/bio_lib.c:217
#13 0x000012cf2676820b in pem_password_callback () from
/usr/local/sbin/openvpn
#14 0x000012cf26765d1a in pem_password_callback () from
/usr/local/sbin/openvpn
#15 0x000012cf267679fc in pem_password_callback () from
/usr/local/sbin/openvpn
#16 0x000012cf267153a6 in ?? () from /usr/local/sbin/openvpn
#17 0x000012cf26715af3 in ?? () from /usr/local/sbin/openvpn
#18 0x000012cf26732d1f in mroute_addr_hash_function () from
/usr/local/sbin/openvpn
#19 0x000012cf26734b0d in mroute_addr_hash_function () from
/usr/local/sbin/openvpn
#20 0x000012cf2672f4ad in mroute_addr_hash_function () from
/usr/local/sbin/openvpn
#21 0x000012cf26736f5d in mroute_addr_hash_function () from
/usr/local/sbin/openvpn
#22 0x000012cf267093d1 in ?? () from /usr/local/sbin/openvpn
#23 0x0000000000000000 in ?? () (gdb)
thanks
On 2015-02-23 18:46, Stuart Henderson wrote:
On 2015/02/23 15:55, Alexander Haensch wrote:
>Synopsis: OpenVPN process crashes from time to time
>Category: OpenVPN
>Environment:
System : OpenBSD 5.7
Details : OpenBSD 5.7-beta (GENERIC.MP) #828: Thu Jan 29 14:28:19
MST 2015
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64
Machine : amd64
>Description:
The OpenVPN Server process configured with a tap device is crashing
during Handshake after some days of operation.
The exact reason is not yet clear to me, but i can maybe gather some
log files as this happens pretty much every 10 days.
I think that something is accumulating and finally ending the
process.
>How-To-Repeat:
Let the server run and wait until the process crashes.
>Fix:
I use monit to automatically start OpenVPN again. I think this is a
bad idea.
Log entries from around the time of the crash (both from openvpn and
OS messages from /var/log/messages) would be helpful.
Is there a core dump anywhere? If not, you might be able to get one in
/var/crash by setting sysctl kern.nosuidcoredump=2, but it would be
helpful to run a build of openvpn from ports done with 'make clean=all;
make DEBUG="-O0 -g" repackage && sudo make reinstall' as this will
increase the amount of information available if there is a coredump.
(If you can get one, use gdb to obtain a backtrace and send that,
rather than sending the file itself).
dmesg:
OpenBSD 5.7-beta (GENERIC.MP) #828: Thu Jan 29 14:28:19 MST 2015
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
(real/avail mem lines snipped to protect the innocent ;)
