TLS problem with Lotus Domino v8.5.1 - mutual handshake fails

2011-11-16 Thread gmx Ralf Hauser
Hi,

In our postfix server, we see

SSL_accept error from hgrs-mail01.hgrs.tld.dom[161.x.y.z]: 0
Nov 16 08:54:52 ernesto postfix2cc/smtpd[18662]: warning: TLS library
problem: 18662:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1053:SSL alert number 0:

This error message apparently means that the client aborted the handshake
just after receiving the server certificate (see step 14 in the upper half
of the wireshark sessions screenshot - a successful handshake example in the
lower part - there step 17 is how it would continue)
If anybody is interested, I am happy to bilaterally send the .pcap files for
the wireshark session and a screenshot of such wireshark sessions.

The domino-side log can be found below.

One hypothesis is that there is a Lotus Notes Domino bug (LO41163:
IMPROPERLY BUILDING CERT CHAIN WHEN FOREIGN HOST PRESENTS JUST LEAF CERT)
but the problem continued even when not just the leaf but also the leaf +
intermediate or incl. root respectively were sent by the postfix server. So
there must also be another problem.

Any hints how to do a client certificate authentication TLS-handshake
between IBM's v8.51 as the client and openssl on the server side would be
highly appreciated.

Many thanks in advance

Ralf

15.11.2011 14:36:07   [2114:0011-176C] SMTPClient: Connection successful
 Checking keyfile certificates:
15.11.2011 14:36:07.45 [2114:0011-176C] SSLCheckCertChain Valid certificate
chain received
15.11.2011 14:36:07.45 [2114:0011-176C] int_MapSSLError Mapping SSL error 0
to 0
15.11.2011 14:36:07.45 [2114:0011-176C] SSL_Handshake Enter
15.11.2011 14:36:07.45 [2114:0011-176C] SSL_Handshake Current Cipher 0x
(Unknown Cipher)
15.11.2011 14:36:07.45 [2114:0011-176C] SSL_Handshake SSL Undetermined
attempt
15.11.2011 14:36:07.45 [2114:0011-176C] SSLAdvanceHandshake Enter Processed
: 0 State: 4
15.11.2011 14:36:07.45 [2114:0011-176C] SSLAdvanceHandshake Enter Processed
: SSL_hello_request
15.11.2011 14:36:07.45 [2114:0011-176C] SSLAdvanceHandshake calling
SSLPrepareAndQueueMessage SSLEncodeClientHello
15.11.2011 14:36:07.45 [2114:0011-176C] SSLAdvanceHandshake Exit State : 5
15.11.2011 14:36:07.45 [2114:0011-176C] S_Write Enter len = 102
Xmt buffer: 
...001'..'
15.11.2011 14:36:07.45 [2114:0011-176C] S_Write Switching Endpoint to sync
15.11.2011 14:36:07.45 [2114:0011-176C] S_Write Posting a nti_snd for 102
bytes
15.11.2011 14:36:07.45 [2114:0011-176C] SSL_EncryptData SSL not init exit
15.11.2011 14:36:07.45 [2114:0011-176C] S_Write Switching Endpoint to async
15.11.2011 14:36:07.45 [2114:0011-176C] SSL_EncryptDataCleanup SSL not init
exit
15.11.2011 14:36:07.45 [2114:0011-176C] S_Write nti_done return 102 bytes
rc = 0
15.11.2011 14:36:07.45 [2114:0011-176C] S_Write Exit, wrote 102 bytes
15.11.2011 14:36:07.45 [2114:0011-176C] S_Read Enter len = 5
15.11.2011 14:36:07.45 [2114:0011-176C] S_Read Switching Endpoint to sync
15.11.2011 14:36:07.45 [2114:0011-176C] S_Read Posting a nti_rcv for 5
bytes
15.11.2011 14:36:07.45 [2114:0011-176C] SSL_RcvSetup SSL not init exit
15.11.2011 14:36:07.47 [2114:0011-176C] S_Read Switching Endpoint to async
15.11.2011 14:36:07.47 [2114:0011-176C] S_Read nti_done return 5 bytes rc =
0
Rcv buffer: 
:     00'.'
15.11.2011 14:36:07.47 [2114:0011-176C] S_Read Exit, read 5 bytes
15.11.2011 14:36:07.47 [2114:0011-176C] S_Read Enter len = 74
15.11.2011 14:36:07.47 [2114:0011-176C] S_Read Switching Endpoint to sync
15.11.2011 14:36:07.47 [2114:0011-176C] S_Read Posting a nti_rcv for 74
bytes
15.11.2011 14:36:07.47 [2114:0011-176C] SSL_RcvSetup SSL not init exit
15.11.2011 14:36:07.47 [2114:0011-176C] S_Read Switching Endpoint to async
15.11.2011 14:36:07.47 [2114:0011-176C] S_Read nti_done return 74 bytes rc
= 0
Rcv buffer: 
-- 64 (0x0040) bytes of 0 --
15.11.2011 14:36:07.47 [2114:0011-176C] S_Read Exit, read 74 bytes
15.11.2011 14:36:07.47 [2114:0011-176C] SSLProcessProtocolMessage Record
Content: 22
15.11.2011 14:36:07.47 [2114:0011-176C] SSLProcessHandshakeMessage Enter
Message: 2 State: 5 Key Exchange: 0 Cipher: 0x (Unknown Cipher)
15.11.2011 14:36:07.47 [2114:0011-176C] SSLProcessHandshakeMessage Enter
Message: SSL_server_hello
15.11.2011 14:36:07.47 [2114:0011-176C] SSLProcessHandshakeMessage Exit
Message: 2 State: 5 Key Exchange: 1 Cipher: 0x0004 (RSA_WITH_RC4_128_MD5)
15.11.2011 14:36:07.47 [2114:0011-176C] SSLAdvanceHandshake Enter Processed
: 2 State: 5
15.11.2011 14:36:07.47 [2114:0011-176C] SSLAdvanceHandshake Enter Processed
: SSL_server_hello
15.11.2011 14:36:07.47 [2114:0011-176C] SSLAdvanceHandshake Exit State : 8
15.11.2011 14:36:07.47 [2114:0011-176C] SSL_Handshake After handshake
state= 8 Status= -5000
15.11.2011 14:36:07.47 [2114:0011-176C] SSL_Handshake Exit Status = -5000
15.11.2011 14:36:07.47 [2114:0011-176C] int_MapSSLError Mapping SSL error
-5000 to 4176
15.11.2011 14:36:07.47 [2114:0011-176C] SSL_Handshake Enter
15.11.2011 14:36:07.47 

Lotus Domino server v8 fails mutual TLS handshake

2011-11-15 Thread gmx Ralf Hauser
Hi Steve,

A Lotus Domino server v8 wants to connect to my postfix like several others
do with client certificate authentication over TLS.

Unfortunately, it fails early on with
warning: TLS library problem: 25785:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1053:SSL alert number 0

In the mailing list archive, there is one mention of the problem that didn't
really help me any further.
(http://marc.info/?l=openssl-usersm=104885831305761w=2)

Many thanks for any hints on that error message in advance!

   Ralf
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   majord...@openssl.org