Quick notes from offlist mails to save time for anyone else reading,
Ted was previously running 6.8-beta Fri Sep 4 22:46:14 MDT 2020,
before the new validator was enabled in libressl.
There was an icinga update in the window, but only a fairly minor
update, and a boost update though that is unlikely to interfere with
certs at all. I'm not running icinga clustering myself so wouldn't have
run into problems with it.
Ted please ignore the "switch to openssl" diff I sent you for now, try
this instead:
Index: patches/patch-lib_base_tlsutility_cpp
===================================================================
RCS file: patches/patch-lib_base_tlsutility_cpp
diff -N patches/patch-lib_base_tlsutility_cpp
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ patches/patch-lib_base_tlsutility_cpp 24 Nov 2020 00:48:51 -0000
@@ -0,0 +1,13 @@
+$OpenBSD$
+
+Index: lib/base/tlsutility.cpp
+--- lib/base/tlsutility.cpp.orig
++++ lib/base/tlsutility.cpp
+@@ -812,6 +812,7 @@ bool VerifyCertificate(const std::shared_ptr<X509>& ca
+
+ X509_STORE_CTX *csc = X509_STORE_CTX_new();
+ X509_STORE_CTX_init(csc, store, certificate.get(), nullptr);
++ X509_STORE_CTX_set_flags(csc, X509_V_FLAG_LEGACY_VERIFY);
+
+ int rc = X509_verify_cert(csc);
+
I can let you have binary packages built with that tomorrow if
that's easier.
On 2020/11/23 17:13, Theodore Wynnychenko wrote:
> Hello
>
> The other day I updated to current (6.8 GENERIC.MP#188).
>
> I then updated packages.
>
> I have been using Icinga2 since about OpenBSD 5.6, and everything was
> fine.
>
> A few hours after the update, I got a warning that my /var/log filesystem
> on the icinga2 master was full.
>
> Then, I noticed warnings in icinga2 for pretty much every check that
> state:
>
> "Error: Function call 'pipe2' failed with error code 24, 'Too many
> open files'"
>
> I only have a couple of dozen endpoints, and have never had this issue
> before. I tried increasing the file limits, but that only increased the
> time before icinga2 crashed into the limit with too many open files.
>
> I then noticed that icinga2 was now throwing a warning about self signed
> certificates.
>
> Specifically, I was getting log messages on the endpoints "New client
> connection for identity 'master.my.tld' to [172.xx.xx.99]:5665
> (certificate validation failed: code 19: self signed certificate in
> certificate chain)".
>
> On the master, I was getting the same, but inverted, error: "New client
> connection for identity 'endpoint.my.tld' from [172.xx.xx.1]:3621
> (certificate validation failed: code 19: self signed certificate in
> certificate chain)".
>
> So, I decided to add the icinga CA certificate to the list of trusted
> certificates in /etc/ssl/cert.pem on both master and endpoint.
>
> Now, when I connect (either from master to endpoint, or the reverse) using
> s_client, I see:
>
> openssl s_client -connect endpoint.my.tld:5665
>
> CONNECTED(00000003)
> depth=1 CN = Icinga CA
> verify return:1
> depth=1 CN = Icinga CA
> verify error:num=19:self signed certificate in certificate chain
> verify return:0
> depth=1 CN = Icinga CA
> verify return:1
> depth=0 CN = endpoint.my.tld
> verify return:1
> depth=0 CN = endpoint.my.tld
> verify return:1
> write W BLOCK
> ---
> Certificate chain
> 0 s:/CN=endpoint.my.tld
> i:/CN=Icinga CA
> 1 s:/CN=Icinga CA
> i:/CN=Icinga CA
> ---
> Server certificate
> -----BEGIN CERTIFICATE-----
> MIIFAjCCAuqgAwIBAgIUZuaLfnjnY6dLkKTQ3J6GGGcCYnowDQYJKoZIhvcNAQEL
> BQAwFDESMBAGA1UEAwwJSWNpbmdhIENBMB4XDTE4MDUxNjE2MzQzN1oXDTMzMDUx
> MjE2MzQzN1owJzElMCMGA1UEAwwccGlwcGluLnNhbWJhLnd5bm55Y2hlbmtvLmNv
> bTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAJzPMaQdWkhb/YMy242C
> jmbRVcwOqZrCRPwscYhAfhfSwNJfSN7k5I9UCFszgvAU3QU3RhEElrJYOQY7UKp1
> V7D4MO88S5NMh6rLrjyCuojxVnbwCA4WwIXMA0zNY6EEG8/1LbcfA8utSy1Y1X0c
> xb4FJKv3ar02j9A5XleZf0p9bKQezysxB3TT17L4AhWsoE1w/7GCfU715OEch+Dw
> WI9TusrJXKLFwHvk1j+ZCjiNM49F7gFDpw3m/Asekt5B3M6L7ZkPM9jI1ThX4etj
> kEC1C2371lP9OdFExqScLudHCL8+2IILd3i+/7YxWFTnxhFszyWMDiMll/omcpot
> qDbFhQdVrfnTo4lczK42EfhQuDMdESkmvay8UVGyUe90AEaQ38V3w8B0iDzgGQTX
> UZNxpurotW2zU1XPhzTOawRlp1POFQ1tnFJzH+iyBCxZZZfZsQchJbMJz/JCGIiW
> qkhTN5bAKoAgO6GglTmqktSZmixa38fZ8qIJ8Y0UZsnH5zRQWcbzKr49u3FhH2rH
> pC9Dh+zlRj/LiMM4c/UF2LzwuIQ3fjZMEff+q3vzs6cXgs8FICX7uvpB4yqqt4Vn
> /ZSRrq++dUAav+0JhTYlW2m+sa/ga4HVAB+zNN7+l92PWSJ0b3DDzZt6CHLp/lWX
> zCPDpo7ABbk+xrg+NEoym8ejAgMBAAGjOTA3MAwGA1UdEwEB/wQCMAAwJwYDVR0R
> BCAwHoIccGlwcGluLnNhbWJhLnd5bm55Y2hlbmtvLmNvbTANBgkqhkiG9w0BAQsF
> AAOCAgEAGh1goJVNy4Ltpv0+x1okod55g7ob6/l2hAwrq0jXBca4zIGncQcdl0jg
> +z6TDMiq+2UUoKB80k5D947t2VHtp+d/wuSTNwpzESNplh5GWpqkdpOHcAN1lkku
> ZgCUnQH/ZFa4Q2V01rPHSaf1znMpaqaYTjAKwNwZY9qRxjXUZg62/D/y9pfmy+VC
> yvZype5rETXjLxr0WN6LABRgda41wiLszMWLAonHRHRVkhdyUYC+brmDNhfByqGJ
> L5oHpvCq9Oywk4zKO02y3wrhL1+JHt0TH/5RmaalWQRsB0vJY9699cnLk6DK/+CD
> YyHecKplsnwnfvwau6aMlwg6zilCZ+YMl3Jwj0vQeG/h8DyTw6t9HtknlnRfcfQ/
> eyTHZtdyH1y1O5v4BQJNt5Ewc7y144IP2g/9Y7g3n7GFlvd68TQjJmI6I4nGsJ+u
> iNu+DGzD6Ih9UhFCWMrR57r9utdarBWYV4NJaXldNnqXrL099Hu3CLzdjikabeHE
> J56gkXOHQdQi2xIyIgIuMHMF+niJkqAmxyGaMfnBOOv6Gvt7TAMLrt4dIFSgomKe
> Rd6xSyCh66H/AYF8b4NlwVCjDmvQf31OtGrl8xVlXDnOeaUHoZsu/ECO9f7E5yuN
> od5dNX4mJoSCkESbwQ67IumOLxSEw5extwTNplG/oEqgl6YalJI=
> -----END CERTIFICATE-----
> subject=/CN=endpoint.my.tld
> issuer=/CN=Icinga CA
> ---
> No client certificate CA names sent
> Server Temp Key: ECDH, X25519, 253 bits
> ---
> SSL handshake has read 3436 bytes and written 392 bytes
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
> Server public key is 4096 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-GCM-SHA384
> Session-ID:
> AF10B5FA058B109699E87151A6BFCE6E9AD4968C04F6E8C1EFE24C8830AE7D5F
> Session-ID-ctx:
> Master-Key:
> 6509DF9A604E5FB4C7F3BD55DC4666FDD93315CA00AA8E373E8C41BD93E3E1D91961AEA356
> 42A684F45DA530C4FDF260
> TLS session ticket lifetime hint: 7200 (seconds)
> TLS session ticket:
> 0000 - b1 ef 85 f6 29 57 ee 81-8f d7 af 12 de 8e 19 1e
> ....)W..........
> 0010 - c4 3d b8 5d 68 1e d0 87-9a 88 09 b8 e5 b8 fd 7d
> .=.]h..........}
> 0020 - 6e 48 ea 63 5f df 83 54-9a d3 b4 3e e6 3a 30 a4
> nH.c_..T...>.:0.
> 0030 - 83 0b df 4d 3e 7b da a2-47 a0 c2 2b 2c 4e 4e f6
> ...M>{..G..+,NN.
> 0040 - f3 b5 6c 24 da 6a f1 c8-bf 27 08 23 1e 37 21 9a
> ..l$.j...'.#.7!.
> 0050 - 93 dd 87 a5 95 b8 72 3c-14 07 33 a1 e4 23 b7 2d
> ......r<..3..#.-
> 0060 - 16 0e b8 ad c4 f9 be 72-a0 44 1f 09 c9 47 47 8a
> .......r.D...GG.
> 0070 - a6 97 10 55 77 a3 fe 7c-0f 2f 33 6c 40 9f 5a 76
> ...Uw..|./3l@.Zv
> 0080 - 43 1b 17 21 44 d9 6a 15-82 b0 9e 42 da 14 78 4e
> C..!D.j....B..xN
> 0090 - 50 5c 19 2a a5 09 61 72-0a f5 11 11 6a 75 4c 67
> P\.*..ar....juLg
>
> Start Time: 1606170242
> Timeout : 7200 (sec)
> Verify return code: 0 (ok)
>
>
> The same is true if I connect from an endpoint to the master:
>
> openssl s_client -connect master.my.tld:5665
> CONNECTED(00000003)
> depth=1 CN = Icinga CA
> verify return:1
> depth=1 CN = Icinga CA
> verify error:num=19:self signed certificate in certificate chain
> verify return:0
> depth=1 CN = Icinga CA
> verify return:1
> depth=0 CN = master.my.tld
> verify return:1
> depth=0 CN = master.my.tld
> verify return:1
> write W BLOCK
> ---
> Certificate chain
> 0 s:/CN= master.my.tld
> i:/CN=Icinga CA
> 1 s:/CN=Icinga CA
> i:/CN=Icinga CA
> ---
> Server certificate
> -----BEGIN CERTIFICATE-----
> MIIExTCCAq2gAwIBAgIBAjANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDDAlJY2lu
> Z2EgQ0EwHhcNMTYwMzI5MTg1MzA1WhcNMzEwMzI2MTg1MzA1WjAmMSQwIgYDVQQD
> DBttZXJyeS5zYW1iYS53eW5ueWNoZW5rby5jb20wggIiMA0GCSqGSIb3DQEBAQUA
> A4ICDwAwggIKAoICAQC6Day5I9w3MeKVE9kNYtiH/N1prcEpq/v5UtsHXXi6rHAA
> 8Q9HrvOdEVZq3gNGb60tJ03yQQ7/hcyDuU/Rt0lpRiRpi4+pZx9TlTKCt8zT8zCy
> bghFxL7F110yf68emwy90MHzsd+piaEsa/ccXfzKC3IgzA9+OXYLk+D/nWYk+qf+
> 5PWbo4EjTz/uMRJSJXYoNsHBHg14kwNOFHI/ewiF4GGLO5X2y0uoLHb945Kri2MF
> 6Dq7Hh1apX0PgHxW8bF17TCBXKqRZ+U9qOL5KFWaaxX3Fo84yIX1n8iHRGymS0fH
> Jq54G0B8IwB1blPGdr36zbZREc0+m9ywLcw/DmgvIEKuz6UEIGCo0KwXGWsM57DD
> butUGNa58782kl5vCf/Ol8ogLEBSWxA5C9tg8eyBYRn705EkLMkSFD3blV1yLAqr
> E5VPFqNhA18O0iALRdUEWwNTFFttL81yLO6c3jnG8FVLhCv4PTA39Wbp/K5gshH3
> 2KHmeXE0jlaOWHFuTod8PEXSl4Ix/HqXPbXZyMrRufLVx/9Tpl95VkEw6euGCz8k
> qumdSQP1SqsDOk5a4sU8s22/B+24k61SwObsCpxulFzOEPqfwAYxRfs3vrKssiys
> M4EJeey5+f3Xm/sWxf6dfilqXJAywweIJkhuGg4RLMwinuAEmvcAzGSeMDzVZwID
> AQABoxAwDjAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQAeFDD7wRUd
> MwgDoooHTzD1Bt6jFLHgY/0ICTLRPmDxl/WrT67x+NDgwqcWRTc4H81G6RuGBl6M
> q5Hh96gaXq7en1JC9r1pg5IzA9fQE0ijNpp4uH5X/7RGD54ca12KSbA+roOPOHX8
> fCXGI3xsWW/4LkRhCM9jfHhkle1G3rBvqBQ/WST2ONuj7EJFcXhBBuF2WavvOobU
> zoS3RYMOq7S3SEtFyY1VvMrZpKvvOGcjYyTDaZxuT7sJ0O8qynr1rTd19dLMq1Wo
> Jn6rILWNoZsXhFYJhBhAeKLak7qNpNlfeIdtCyvybgE4mksfMOrxd9+IhUdzbLuK
> 8/w/2rK5DJbI2jghHUR8wXrgBFaGCgFbx/rUamDuY09VQP3bCmH+WfvBid/7+M5s
> l3VAXnSwI6Ez7KbbeUowznyruaeRiq74UMGa9qEwdnTd8+emu6clRwqPAT1te1Ji
> 2Tq0KiHu/2xlqKGOTvzY0Im207YpMGWO5Irnaf5o/EYPEWUFG6ngpryYwgoftZPZ
> oL/5quNIFjGmAaspHnYFiHQF2KWhg7QUz5nQhm0V7YSybNsRUiYq+jVbgYFyGYIm
> GJQOb71D3BiFlTbOXqn2nNK5qvIZNXMCyO7DQu5wAv291O+a9STqqjnScPHSvqZ+
> fVMWrnq4BFolyOA9NowZwTpyI43oLffzOw==
> -----END CERTIFICATE-----
> subject=/CN= master.my.tld
> issuer=/CN=Icinga CA
> ---
> No client certificate CA names sent
> Server Temp Key: ECDH, X25519, 253 bits
> ---
> SSL handshake has read 3375 bytes and written 392 bytes
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
> Server public key is 4096 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-GCM-SHA384
> Session-ID:
> AC7F7986150D6886FDFE2CD6B26B20B527D3857D0FDAF56DB85719C53A00DC04
> Session-ID-ctx:
> Master-Key:
> 6639461A0943FD8A84812B1E2D8C5C02958271502A0A9CD61D20D8306BF85F11B287F2A437
> A7EE7E4E8A4BFE03813C0C
> TLS session ticket lifetime hint: 7200 (seconds)
> TLS session ticket:
> 0000 - 55 22 c6 0c 47 72 ed 57-a3 2e 68 32 36 51 3e 7c
> U"..Gr.W..h26Q>|
> 0010 - 62 56 7d 88 82 f7 5c 6f-cf 38 c3 66 b1 ea cb 0a
> bV}...\o.8.f....
> 0020 - 7c c8 be bd 5f 8b b6 9a-2d c5 8f 01 80 79 88 52
> |..._...-....y.R
> 0030 - a2 22 5e 8c b0 57 4a db-16 be 3c b4 a2 9e 49 d7
> ."^..WJ...<...I.
> 0040 - 63 d8 39 d5 a8 ee 2a b2-2f d2 60 25 b2 79 58 90
> c.9...*./.`%.yX.
> 0050 - 0f 71 f7 6f 68 cc 78 dc-90 7f 1d f1 04 66 7f 00
> .q.oh.x......f..
> 0060 - 60 bc ae 2a 0e ff 26 03-44 e1 fc b7 c5 a3 99 e6
> `..*..&.D.......
> 0070 - 8f d8 a8 8f 28 ce aa 92-88 32 16 02 6a c9 81 11
> ....(....2..j...
> 0080 - 64 de f9 c5 f5 05 6f 40-a8 4a eb 16 ac 7b 93 d1
> d.....o@.J...{..
> 0090 - 4b 93 e2 97 71 0c 3d 67-a1 42 d5 15 aa 94 c2 9c
> K...q.=g.B......
>
> Start Time: 1606170371
> Timeout : 7200 (sec)
> Verify return code: 0 (ok)
>
>
> But, despite the fact that s_client returns a "OK" when connecting to the
> icinga2 port, the icinga2 process continues to fail with the "certificate
> validation failed: code 19: self signed certificate in certificate chain"
> messages.
>
> I think this is the issue (as a minimally informed observer), and that the
> failures in validation leave processes in icinga2 hanging until there are
> too many open.
>
> But, I have no idea of how to explore this further.
>
> Please let me know if there is anything I should try, or any other
> information that may be helpful in identifying what the issue may be.
>
> As I said, this has been working for (what (6.8-5.6)/2 years), and I made
> no changes after the update.
>
> Thanks again
> Ted
>
>
>
>
