自动回复: Re: OpenSSL 1.1.1 Windows dependencies
这是一封自动回复邮件。已经收到您的来信，我会尽快回复。
Re: OpenSSL 1.1.1 Windows dependencies
On Wed, Oct 26, 2022 at 11:50:16AM -0400, Viktor Dukhovni wrote: > On Wed, Oct 26, 2022 at 11:15:25AM +0100, Matt Caswell wrote: > > > > I'm not promising anything. But if you send me the captures I can take a > > > look at them. > > > > I've taken a look at the captures for the working and non-working scenarios. > > > > Do I understand correctly that your application is acting as the server > > in this setup? > > > > I have compared the working and non-working captures. In both cases the > > ClientHello is successfully received, and the server responds with a > > ServerHello, Certificate, ServerKeyExchange and ServerHelloDone message. > > Aside from normal variations between one session and another, AFAICT, > > the ClientHello and the server's response messages all look identical > > other than the server obviously has a different Certificate. The > > Certificates themselves also look identical to each other other than the > > subject/subjectaltname being for a different server. The intermediate > > certs are the same in both cases. > > > > Following the server's ServerHelloDone the client continues with a > > ClientKeyExchange message in the working case. In the non-working case > > the the client immediately closes the TCP connection without sending any > > kind of alert. > > See longish thread at: > > https://marc.info/?l=postfix-users=166584042429636=2 > > which describes a remarkably similar set of symptoms observed after a > Microsoft patch update. Today the OP posted that a more follow-on patch > appears to have resolved the problem. TL;DR: progress on identifying the issue begins with: https://marc.info/?l=postfix-users=166585652703462=2 -- Viktor.
自动回复: Re: OpenSSL 1.1.1 Windows dependencies
这是一封自动回复邮件。已经收到您的来信，我会尽快回复。
Re: OpenSSL 1.1.1 Windows dependencies
On Wed, Oct 26, 2022 at 11:15:25AM +0100, Matt Caswell wrote: > > I'm not promising anything. But if you send me the captures I can take a > > look at them. > > I've taken a look at the captures for the working and non-working scenarios. > > Do I understand correctly that your application is acting as the server > in this setup? > > I have compared the working and non-working captures. In both cases the > ClientHello is successfully received, and the server responds with a > ServerHello, Certificate, ServerKeyExchange and ServerHelloDone message. > Aside from normal variations between one session and another, AFAICT, > the ClientHello and the server's response messages all look identical > other than the server obviously has a different Certificate. The > Certificates themselves also look identical to each other other than the > subject/subjectaltname being for a different server. The intermediate > certs are the same in both cases. > > Following the server's ServerHelloDone the client continues with a > ClientKeyExchange message in the working case. In the non-working case > the the client immediately closes the TCP connection without sending any > kind of alert. See longish thread at: https://marc.info/?l=postfix-users=166584042429636=2 which describes a remarkably similar set of symptoms observed after a Microsoft patch update. Today the OP posted that a more follow-on patch appears to have resolved the problem. > This really looks like a problem on the client side to me. Yes, the client just hangs up. A known. Disabling session tickets on server appears to help in some cases (for no obvious reason). Applying the follow-on update is a better solution if applicable. -- Viktor.
Re: OpenSSL 1.1.1 Windows dependencies
On 24/10/2022 10:17, Matt Caswell wrote: On 22/10/2022 16:02, David Harris wrote: On 21 Oct 2022 at 13:50, Michael Wojcik via openssl-users wrote: That was my initial thought too, except that if it were firewall-related, the initial port 587 connection would be blocked, and it isn't - the failure doesn't happen until after STARTTLS has been issued. Not necessarily. That's true for a first-generation port-blocking firewall, but not for a packet-inspecting one. There are organizations which use packet-inspecting firewalls to block STARTTLS because they enforce their own TLS termination, in order to inspect all incoming traffic for malicious content and outgoing traffic for exfiltration. I now have wireshark captures showing the exchanges between the working instance and the non-working instance respectively; the problem is definitely happening after STARTTLS has been issued and during the TLS handshake. I'm not high-level enough to be able to make any sense of the negotiation data though. The wireshark capture is quite short (22 items in the list) and I don't mind making it available if it would be useful to anyone. I'm not promising anything. But if you send me the captures I can take a look at them. I've taken a look at the captures for the working and non-working scenarios. Do I understand correctly that your application is acting as the server in this setup? I have compared the working and non-working captures. In both cases the ClientHello is successfully received, and the server responds with a ServerHello, Certificate, ServerKeyExchange and ServerHelloDone message. Aside from normal variations between one session and another, AFAICT, the ClientHello and the server's response messages all look identical other than the server obviously has a different Certificate. The Certificates themselves also look identical to each other other than the subject/subjectaltname being for a different server. The intermediate certs are the same in both cases. Following the server's ServerHelloDone the client continues with a ClientKeyExchange message in the working case. In the non-working case the the client immediately closes the TCP connection without sending any kind of alert. This really looks like a problem on the client side to me. Or at least that's where attention will have to be focused for now to figure out what went wrong. The client side has made the decision to close the connection (or plausibly some middlebox has, but this seems unlikely given that the handshakes up until that point look virtually identical). Is there any possibility of getting any logs from the client side to see why it has decided to abort the connection? Matt
Re: OpenSSL 1.1.1 Windows dependencies
On 22/10/2022 16:02, David Harris wrote: On 21 Oct 2022 at 13:50, Michael Wojcik via openssl-users wrote: That was my initial thought too, except that if it were firewall-related, the initial port 587 connection would be blocked, and it isn't - the failure doesn't happen until after STARTTLS has been issued. Not necessarily. That's true for a first-generation port-blocking firewall, but not for a packet-inspecting one. There are organizations which use packet-inspecting firewalls to block STARTTLS because they enforce their own TLS termination, in order to inspect all incoming traffic for malicious content and outgoing traffic for exfiltration. I now have wireshark captures showing the exchanges between the working instance and the non-working instance respectively; the problem is definitely happening after STARTTLS has been issued and during the TLS handshake. I'm not high-level enough to be able to make any sense of the negotiation data though. The wireshark capture is quite short (22 items in the list) and I don't mind making it available if it would be useful to anyone. I'm not promising anything. But if you send me the captures I can take a look at them. Matt Furthermore, the OpenSSL configuration is identical between the systems/combinations of OpenSSL that work and those that don't. Do you know that for certain? There's no openssl.cnf from some other source being picked up on the non-working system? I'm pretty certain, but I'll get the customer to double-check. Cheers! -- David --
RE: OpenSSL 1.1.1 Windows dependencies
> From: openssl-users On Behalf Of David > Harris > Sent: Saturday, 22 October, 2022 09:02 > > I now have wireshark captures showing the exchanges between the working > instance and the non-working instance respectively; the problem is definitely > happening after STARTTLS has been issued and during the TLS handshake. A packet-inspecting firewall can monitor a TLS handshake (for TLS prior to 1.3) and terminate the conversation if it sees something in the unencrypted messages - ClientHello, ServerHello, ServerCertificate, etc - that it doesn't like. It's not beyond imagining that an organization would have a packet-inspecting firewall that terminates conversations using particular cipher suites, for example. > I'm not high-level enough to be able to make any sense of the negotiation > data though. The wireshark capture is quite short (22 items in the list) > and I don't mind making it available if it would be useful to anyone. Someone might be able to tell something from it. Not much else is coming to mind, I'm afraid. It would help to know what system call is failing, with what errno value, but that's going to be a bit tough to determine on Windows. ProcMon, maybe? And it's curious that the OpenSSL error stack is empty, but without being able to debug you probably couldn't track that down, short of instrumenting a bunch of the OpenSSL code. -- Michael Wojcik
Re: OpenSSL 1.1.1 Windows dependencies
On 21 Oct 2022 at 13:50, Michael Wojcik via openssl-users wrote: > > That was my initial thought too, except that if it were > > firewall-related, the initial port 587 connection would be blocked, > > and it isn't - the failure doesn't happen until after STARTTLS has > > been issued. > > Not necessarily. That's true for a first-generation port-blocking > firewall, but not for a packet-inspecting one. There are organizations > which use packet-inspecting firewalls to block STARTTLS because they > enforce their own TLS termination, in order to inspect all incoming > traffic for malicious content and outgoing traffic for exfiltration. I now have wireshark captures showing the exchanges between the working instance and the non-working instance respectively; the problem is definitely happening after STARTTLS has been issued and during the TLS handshake. I'm not high-level enough to be able to make any sense of the negotiation data though. The wireshark capture is quite short (22 items in the list) and I don't mind making it available if it would be useful to anyone. > > Furthermore, the OpenSSL > > configuration is identical between the systems/combinations of > > OpenSSL that work and those that don't. > > Do you know that for certain? There's no openssl.cnf from some other > source being picked up on the non-working system? I'm pretty certain, but I'll get the customer to double-check. Cheers! -- David --
RE: OpenSSL 1.1.1 Windows dependencies
> From: David Harris > Sent: Friday, 21 October, 2022 01:42 > > On 20 Oct 2022 at 20:04, Michael Wojcik wrote: > > > I think more plausible causes of this failure are things like OpenSSL > > configuration and interference from other software such as an endpoint > > firewall. Getting SYSCALL from SSL_accept *really* looks like > > network-stack-level interference, from a firewall or similar > > mechanism. > > That was my initial thought too, except that if it were firewall-related, the > initial port 587 connection would be blocked, and it isn't - the failure > doesn't > happen until after STARTTLS has been issued. Not necessarily. That's true for a first-generation port-blocking firewall, but not for a packet-inspecting one. There are organizations which use packet-inspecting firewalls to block STARTTLS because they enforce their own TLS termination, in order to inspect all incoming traffic for malicious content and outgoing traffic for exfiltration. > Furthermore, the OpenSSL > configuration is identical between the systems/combinations of OpenSSL that > work and those that don't. Do you know that for certain? There's no openssl.cnf from some other source being picked up on the non-working system? -- Michael Wojcik
Re: OpenSSL 1.1.1 Windows dependencies
On 21 Oct 2022 at 7:27, Richard Levitte wrote: > Let me ask you this: on what Windows version was your application > built? Common wisdom would be to build on the oldest version... My application is a very traditional Win32 application, and at the moment (and until circumstances *force* me to change) I build it in a Windows 7 SP2 VM. There's really no build-related reason why it shouldn't work on Server 2012, especially since the 1.1.1g build (which DOES work on the affected system) is built in exactly the same VM. It's a puzzle, for sure. Thanks for taking the time to look into this for me Richard. Cheers! -- David --
Re: OpenSSL 1.1.1 Windows dependencies
On 20 Oct 2022 at 20:04, Michael Wojcik wrote: > OpenSSL 1.1.1 uses Windows cryptographic routines in two areas I'm > aware of: rand_win.c and the CAPI engine. I don't offhand see a way > that a problem with the calls in rand_win.c would cause the particular > symptom you described. My guess is that you're not using the CAPI > engine, but you might check your OpenSSL configuration on the failing > system. For a variety of reasons to do with redistributables, I build OpenSSL as no-shared, and because of the compiler I prefer to use (an older build of Visual C), I have to compile with no-capi as well, so CAPI clearly isn't an issue in this case. But to be sure, I tried rebuilding OpenSSL with Visual C 2022 (using Visual C 2019 as the compile unit) and according to the customer, the result was the same. > I think more plausible causes of this failure are things like OpenSSL > configuration and interference from other software such as an endpoint > firewall. Getting SYSCALL from SSL_accept *really* looks like > network-stack-level interference, from a firewall or similar > mechanism. That was my initial thought too, except that if it were firewall-related, the initial port 587 connection would be blocked, and it isn't - the failure doesn't happen until after STARTTLS has been issued. Furthermore, the OpenSSL configuration is identical between the systems/combinations of OpenSSL that work and those that don't. > Personally, if I ran into this, I'd just build OpenSSL for debug and > debug into it. But I know that's not everyone's cup of tea. Unfortunately, I don't have that level of access to the customer's systems. I was really just wondering if the combination of factors rang any bells with anyone before I started digging much deeper; it's altogether possible that I might just have to write this one off to experience and tell the user to use a 1.1.1g build of OpenSSL (which I build exactly the same way, and which works correctly in the same setup). Thanks for the help - appreciated. Cheers! -- David --
Re: OpenSSL 1.1.1 Windows dependencies
Hi David, I just did a check to see what Windows libraries the openssl.exe app depends on, going back to look in 1.0.2, and looking at the current development branch (master). 1.0.2: ws2_32.lib(cond: no-sock) gdi32.lib advapi32.lib crypt32.lib user32.lib bufferoverflowu.lib (cond: cl version 14+) unicows.lib (cond: win32 && -DUNICODE) master: ws2_32.lib(cond: no-sock) gdi32.lib advapi32.lib crypt32.lib user32.lib bufferoverflowu.lib (cond: cl version 14+) As you can see, it hasn't changed much (the unicows.lib dependency was there for Win9x compatibility, according to comments). But, it's quite possible that those libraries have dependencies of their own that have changed in newer Windows versions. Let me ask you this: on what Windows version was your application built? Common wisdom would be to build on the oldest version... Cheers, Richard On Thu, 20 Oct 2022 02:54:19 +0200, David Harris wrote: > > Up front, I'd like to apologize if this is an FAQ or has been answered > elsewhere > on this list: my workload means that I simply can't keep as up-to-date as I > would > like. > > I have a situation where my application fails to accept an incoming SSL > handshake on Windows Server 2012, but the identical software running on > Server 2019 accepts the same connection from the same remote client without > a problem. Other types of client software (such as Thunderbird) connect to > either system without any problems. The connecting client is a Windows Cash > Register using Window's built-in crypto facilities. If I downgrade my app to > OpenSSL 1.1.1g or earlier, the problem doesn't happen. With 1.1.1k or 1.1.1q, > I > get the error (I haven't built any versions of OpenSSL between k and q). In > case > it helps, the connection is an incoming SMTP connection on port 587, and > STARTTLS is used to begin SSL negotiation. > > SSL_accept returns -1, with an extended error of "SSL_ERROR_SYSCALL" (5), > which I understand to be largely what it returns when it doesn't have a clear > idea > of what's gone wrong. The error queue is completely empty in this situation. > The > cert is a LetsEncrypt cert that loads without errors and works fine with > other > clients. > > Do recent versions of OpenSSL 1.1.1 have dependencies on some Windows > facility (winsock and wincrypt seem likely candidates) that might work on > Server > 2019 but fail on Server 2012? > > The version of my application that is in public release uses 1.1.1g, so isn't > affected by this issue, but I'm slightly worried that I'm going to see an > uptick in > this type of problem if I release builds based on later versions of 1.1.1. > > Does this ring any bells with anyone? Again, apologies if this is answered > elsewhere - I *did* spend some time in Google but couldn't find anything that > seemed relevant. > > Thanks in advance for any advice. > > Cheers! > > -- David -- > -- Richard Levitte levi...@openssl.org OpenSSL Project http://www.openssl.org/~levitte/
RE: OpenSSL 1.1.1 Windows dependencies
> From: openssl-users On Behalf Of David > Harris > Sent: Wednesday, 19 October, 2022 18:54 > > Do recent versions of OpenSSL 1.1.1 have dependencies on some Windows > facility (winsock and wincrypt seem likely candidates) that might work on > Server 2019 but fail on Server 2012? OpenSSL on Windows has always had a dependency on Winsock/Winsock2 (see b_sock.c, e_os.h, sockets.h) for supporting socket BIOs. Obviously OpenSSL used for TLS is going to be interacting with Winsock. I can't think of any difference between Server 2012 and Server 2019 that would be relevant to the issue you describe. OpenSSL 1.1.1 uses Windows cryptographic routines in two areas I'm aware of: rand_win.c and the CAPI engine. I don't offhand see a way that a problem with the calls in rand_win.c would cause the particular symptom you described. My guess is that you're not using the CAPI engine, but you might check your OpenSSL configuration on the failing system. I think more plausible causes of this failure are things like OpenSSL configuration and interference from other software such as an endpoint firewall. Getting SYSCALL from SSL_accept *really* looks like network-stack-level interference, from a firewall or similar mechanism. Personally, if I ran into this, I'd just build OpenSSL for debug and debug into it. But I know that's not everyone's cup of tea. -- Michael Wojcik
