Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-15 Thread Viktor Dukhovni
> On May 15, 2020, at 11:05 PM, Alexander Vasarab > wrote: > > Yes, after a full day of observation today, the issue appears to have > been resolved. > > Thank you for your time and effort. No worries. Many thanks for sticking with this to the bitter end! Finding this bug required much

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-15 Thread Alexander Vasarab
On 14/05/20 23:15 -0400, Viktor Dukhovni wrote: That was enough of a clue for me. Please build and test the latest commit from the tls-debug branch in my git repo, you should find it works regardless of how you access Postgres... Yes, after a full day of observation today, the issue appears

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-14 Thread Alexander Vasarab
On 14/05/20 01:40 -0400, Viktor Dukhovni wrote: I'f you're comfortable with gdb, and willing to build both Postfix and OpenSSL from source with debugging symbols, then you could add a "-D" flag to the "smtpd" entry in the /opt/postfix/etc/master.cf file, and attach to a "screen" running a

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-14 Thread Viktor Dukhovni
On Thu, May 14, 2020 at 06:20:17PM -0700, Alexander Vasarab wrote: > The plot thickens. > > In a nutshell, the problem is "solved" by changing the virtual(8) hosts [ That's virtual(5) by the way, but not important. ] > to use UNIX domain sockets (rather than e.g. 127.0.0.1). In both cases, >

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-14 Thread Scott Kitterman
On Thursday, May 14, 2020 1:40:38 AM EDT Viktor Dukhovni wrote: > On Wed, May 13, 2020 at 10:01:24PM -0700, Alexander Vasarab wrote: > > May 13 21:56:38 vasaconsulting postfix/smtpd[25599]: tls_bio: > > hsfunc=(nil), rfunc=0x7f310ef36dd0, wfunc=(nil), SSL_get_error(36) = 0 > > May 13 21:56:38

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-14 Thread Alexander Vasarab
On 13/05/20 21:58 -0400, Viktor Dukhovni wrote: Please rebuild, and post another similar set of logs. Thanks. Attached. Alexander May 13 21:56:38 vasaconsulting postfix/smtpd[25599]: connect from [] May 13 21:56:38 vasaconsulting postfix/smtpd[25599]: tls_bio: hsfunc=0x7f310ef3a780,

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Viktor Dukhovni
On Wed, May 13, 2020 at 10:01:24PM -0700, Alexander Vasarab wrote: > May 13 21:56:38 vasaconsulting postfix/smtpd[25599]: tls_bio: hsfunc=(nil), > rfunc=0x7f310ef36dd0, wfunc=(nil), SSL_get_error(36) = 0 > May 13 21:56:38 vasaconsulting postfix/smtpd[25599]: tls_bio: TLS success > May 13

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Alexander Vasarab
On 13/05/20 20:40 -0400, Viktor Dukhovni wrote: Your OpenSSL library looks busted. Do you have more than one set of OpenSSL libraries installed on your system? What ldd report for the "smtpd" executable? Is this the stock OpenSSL for your system, or your own build? There's just one OpenSSL

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Alexander Vasarab
On 13/05/20 16:20 -0400, Viktor Dukhovni wrote: Try the below. Note, if build as below, it will not replace your system The output is attached. Alexander May 13 16:31:24 vasaconsulting postfix/smtpd[14216]: connect from [] May 13 16:31:24 vasaconsulting postfix/smtpd[14216]: tls_bio:

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Viktor Dukhovni
On Wed, May 13, 2020 at 04:37:29PM -0700, Alexander Vasarab wrote: > On 13/05/20 16:20 -0400, Viktor Dukhovni wrote: > >Try the below. Note, if build as below, it will not replace your system > > The output is attached. I pushed one more commit on the tsl-debug branch, that shows the address of

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Viktor Dukhovni
On Wed, May 13, 2020 at 04:37:29PM -0700, Alexander Vasarab wrote: > The output is attached. > > May 13 16:31:24 vasaconsulting postfix/smtpd[14216]: tls_bio: > SSL_get_error(-1) = 2 > May 13 16:31:24 vasaconsulting postfix/smtpd[14216]: tls_bio: waiting for > readable socket > May 13 16:31:24

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Viktor Dukhovni
On Wed, May 13, 2020 at 12:05:24PM -0700, Alexander Vasarab wrote: > On 13/05/20 13:56 -0400, Viktor Dukhovni wrote: > >If you're willing to rebuild Postfix from source, then I can provide > >a patch that would log more details. > > Yes, absolutely willing. Thank you. Try the below. Note, if

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Alexander Vasarab
On 13/05/20 00:34 -0400, Viktor Dukhovni wrote: an SSL_ERROR_WANT_READ. You need to try an updated OpenSSL. Thanks for your insights. I'm trying new things to try to improve my understanding of the issue. I juggled around some versions. Bumped to libssl 1.1.1g, restarted postfix, problem

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Alexander Vasarab
On 13/05/20 13:56 -0400, Viktor Dukhovni wrote: If you're willing to rebuild Postfix from source, then I can provide a patch that would log more details. Yes, absolutely willing. Thank you. Alexander

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Viktor Dukhovni
On Wed, May 13, 2020 at 08:45:49AM -0700, Alexander Vasarab wrote: > On 13/05/20 00:34 -0400, Viktor Dukhovni wrote: > >an SSL_ERROR_WANT_READ. You need to try an updated OpenSSL. > > Thanks for your insights. I'm trying new things to try to improve my > understanding of the issue. > > I

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Wietse Venema
One example of 'stateful' behavior is the way that modern operating systems cache file data in memory. If a bit goes bad in userland, then that may persist across executions by different processes that get their code and initial data from the same file, until the memory page is reloaded from the

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Matus UHLAR - fantomas
On 12/05/20 05:40 -0400, Viktor Dukhovni wrote: Indeed the server slams the TCP socket closed after receiving the client's RCPT command. Unclear why. You might try debug_peer_list next, to see whether the server can log enough cleartext traffic to expose the SMTP traffic inside TLS. On

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-13 Thread Alexander Vasarab
On 12/05/20 23:27 -0400, Viktor Dukhovni wrote: Once again out of the blue, a lost connection. The SMTP server is trying to read the next command after sending "RCPT TO" and encounters an EOF condition, for no apparent reason. At this point, I'd guess your SSL library is broken... I was able

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-12 Thread Viktor Dukhovni
On Tue, May 12, 2020 at 08:59:15PM -0700, Alexander Vasarab wrote: > On 12/05/20 23:27 -0400, Viktor Dukhovni wrote: > >Once again out of the blue, a lost connection. The SMTP server is > >trying to read the next command after sending "RCPT TO" and encounters > >an EOF condition, for no apparent

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-12 Thread Viktor Dukhovni
On Tue, May 12, 2020 at 07:31:02PM -0700, Alexander Vasarab wrote: > I wish I could offer an interpretation, but I'm ignorant to the arcana > that's on full display in this log. > May 12 19:04:06 vasaconsulting postfix/smtpd[3301]: > [ scrubbed>]: 250 2.1.5 Ok > May 12 19:04:06 vasaconsulting

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-12 Thread Alexander Vasarab
On 12/05/20 21:07 -0400, Viktor Dukhovni wrote: At this point, clone your submission service onto port 588, and configure that copy with "smtpd -vvv" + all the other options. Then use your client to connect to port 588, and there should now be voluminous logging from the Postfix I/O layer

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-12 Thread Viktor Dukhovni
On Tue, May 12, 2020 at 03:08:16PM -0700, Alexander Vasarab wrote: > May 12 14:27:22 vasaconsulting postfix/smtpd[3482]: > [ scrubbed>]: 250 2.1.0 Ok > May 12 14:27:22 vasaconsulting postfix/smtpd[3482]: watchdog_pat: > 0x55bec82e41e0 > May 12 14:27:22 vasaconsulting postfix/smtpd[3482]: < [

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-12 Thread Alexander Vasarab
On 12/05/20 05:40 -0400, Viktor Dukhovni wrote: Indeed the server slams the TCP socket closed after receiving the client's RCPT command. Unclear why. You might try debug_peer_list next, to see whether the server can log enough cleartext traffic to expose the SMTP traffic inside TLS. Thanks.

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-12 Thread Andreas Schamanek
FWIW: On Mon, 11 May 2020, at 21:17, Alexander Vasarab wrote: When I send mail, it fails the first time, and then as long as I immediately resend, it succeeds (over TLS, thankfully). I have recently come along this very same pattern when using command line tools like curl or lynx to access

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-12 Thread Viktor Dukhovni
On Mon, May 11, 2020 at 09:17:16PM -0700, Alexander Vasarab wrote: > On 11/05/20 23:35 -0400, Viktor Dukhovni wrote: > >Attaching it is fine, if you're willing to disclose the IP addresses and > >hostnames of the two servers. > > Okay, I've attached two files; the PCAP and the postfix log.

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-11 Thread Alexander Vasarab
On 11/05/20 23:35 -0400, Viktor Dukhovni wrote: Attaching it is fine, if you're willing to disclose the IP addresses and hostnames of the two servers. Okay, I've attached two files; the PCAP and the postfix log. To clarify my earlier email, the unencrypted session scenario only arises when I

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-11 Thread Viktor Dukhovni
On May 11, 2020, at 11:19 PM, Alexander Vasarab wrote: > I've captured the relevant conversation. In doing so, it became clear to > me that when the message succeeds after immediately trying again, it > does so because the subsequent connection does not try to use TLS. So > the pattern is:

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-11 Thread Alexander Vasarab
The real problem is that the connection was terminated mid-transaction. The "shutdown while in init" is I think a distraction, Postfix was cleaning up the TLS session, when it was not yet, or is no longer in a state that is valid for calling SSL_shutdown(). If you manage to collect a PCAP

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-11 Thread Wietse Venema
Viktor Dukhovni: > On Mon, May 11, 2020 at 03:52:27PM -0400, Wietse Venema wrote: > > > Viktor Dukhovni: > > > > > That said, it perhaps possible that Postfix is calling SSL_shutdown() on > > > a connection that never progressed enough to complete the handshake. > > > I'll check what

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-11 Thread Viktor Dukhovni
On Mon, May 11, 2020 at 12:51:55PM -0700, Alexander Vasarab wrote: > May 11 12:20:56 vasaconsulting postfix/smtpd[28652]: connect from scrubbed>[] > May 11 12:20:56 vasaconsulting postfix/tlsmgr[8390]: put smtpd session >

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-11 Thread Viktor Dukhovni
On Mon, May 11, 2020 at 03:52:27PM -0400, Wietse Venema wrote: > Viktor Dukhovni: > > > That said, it perhaps possible that Postfix is calling SSL_shutdown() on > > a connection that never progressed enough to complete the handshake. > > I'll check what preconditions are needed for OpenSSL to

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-11 Thread Wietse Venema
Viktor Dukhovni: > On Mon, May 11, 2020 at 11:43:41AM -0700, Alexander Vasarab wrote: > > > I recently upgraded postfix and OpenSSL to 3.4.10 and 1.1.1d, > > respectively. These versions align with Debian GNU/Linux 10 (buster). > > Since the upgrade I've begun receiving regular log entries that

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-11 Thread Alexander Vasarab
The remote peer sent a TLS shutdown message during the TLS handshake. There is no way to 'continue' the handshake. Maybe the remote peer times out - you could find out by looking at the TIME STAMPS in your logs. Causes for timeout: your server is slow, or your network has packet loss. The

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-11 Thread Viktor Dukhovni
On Mon, May 11, 2020 at 11:43:41AM -0700, Alexander Vasarab wrote: > I recently upgraded postfix and OpenSSL to 3.4.10 and 1.1.1d, > respectively. These versions align with Debian GNU/Linux 10 (buster). > Since the upgrade I've begun receiving regular log entries that look > like this: > > May

Re: "SSL_Shutdown:shutdown while in init" while sending and receiving

2020-05-11 Thread Wietse Venema
Alexander Vasarab: > Greetings, > > I recently upgraded postfix and OpenSSL to 3.4.10 and 1.1.1d, > respectively. These versions align with Debian GNU/Linux 10 (buster). > Since the upgrade I've begun receiving regular log entries that look > like this: > > May 11 11:23:54 vasaconsulting