Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
Thanks, Jakob. I was under the impression that in my environment, Diffie Hellman key exchange would be in use, and that it would prevent the use of decryption, even with private key. Is that wrong? -- View this message in context: http://openssl.6102.n7.nabble.com/OpenSSL-non-blocking-epoll-hanging-on-data-receiving-tp66355p66466.html Sent from the OpenSSL - User mailing list archive at Nabble.com. -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
On 27/05/2016 18:29, Matt Caswell wrote: On 27/05/16 16:20, counterpoint wrote: Thanks Matt, good points. Not easy to implement though! In the problem case, my code is the server (it is a proxy), and the standard MariaDB command line client is the client. Yes, it does look as if everything is happening as it should, except that the process stops before all the data has been handled. The client is sending a large query (about 500 KB, using "load data local infile '/root/bigdata.txt' into table upload;"). If the client is connected directly to the database, using SSL, the query runs successfully. If the client is connected through the proxy without SSL (most of the logic exactly the same), the query runs successfully. If a shorter query is chosen, it works with SSL. Looking at the data flows with Wireshark, it looks about right, but I can't see the data in detail because of the SSL :) Perhaps using an eNULL ciphersuite might help? Matt For future reference, another way is to load your private key into WireShark. This works for all but the EDH/ECDH suites that provide PFS security against enemies who steal your private key and then decrypt previously recorded TLS/SSL sessions. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
Fixed the problem in the end, using eNULL was useful to get the full picture. The issue my assumption that if there was data available to process (after read ahead was turned off) then SSL_pending would tell me so. But it seems that when the data extends beyond a single block (with OpenSSL imposing a 16K block size limit) it is necessary to keep reading after a successful read, as there may be more data available from the next block. I haven't seen any more positive way to know if there is data to process than simply repeatedly reading until no data is received. -- View this message in context: http://openssl.6102.n7.nabble.com/OpenSSL-non-blocking-epoll-hanging-on-data-receiving-tp66355p66452.html Sent from the OpenSSL - User mailing list archive at Nabble.com. -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
> Perhaps using an eNULL ciphersuite might help? > Matt Good idea, I'll give it a try. -- View this message in context: http://openssl.6102.n7.nabble.com/OpenSSL-non-blocking-epoll-hanging-on-data-receiving-tp66355p66398.html Sent from the OpenSSL - User mailing list archive at Nabble.com. -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
On 27/05/16 16:20, counterpoint wrote: > Thanks Matt, good points. Not easy to implement though! > > In the problem case, my code is the server (it is a proxy), and the standard > MariaDB command line client is the client. Yes, it does look as if > everything is happening as it should, except that the process stops before > all the data has been handled. > > The client is sending a large query (about 500 KB, using "load data local > infile '/root/bigdata.txt' into table upload;"). > > If the client is connected directly to the database, using SSL, the query > runs successfully. > > If the client is connected through the proxy without SSL (most of the logic > exactly the same), the query runs successfully. > > If a shorter query is chosen, it works with SSL. > > Looking at the data flows with Wireshark, it looks about right, but I can't > see the data in detail because of the SSL :) Perhaps using an eNULL ciphersuite might help? Matt -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
Thanks Matt, good points. Not easy to implement though! In the problem case, my code is the server (it is a proxy), and the standard MariaDB command line client is the client. Yes, it does look as if everything is happening as it should, except that the process stops before all the data has been handled. The client is sending a large query (about 500 KB, using "load data local infile '/root/bigdata.txt' into table upload;"). If the client is connected directly to the database, using SSL, the query runs successfully. If the client is connected through the proxy without SSL (most of the logic exactly the same), the query runs successfully. If a shorter query is chosen, it works with SSL. Looking at the data flows with Wireshark, it looks about right, but I can't see the data in detail because of the SSL :) I thought the read ahead issue would have fixed both sides, but somehow the interface to the client still doesn't work. So it's hard to find things to diagnose, leaving me scratching round for inspiration and something to change that might fix things. And ideally, I need a quick result! Such is life. -- View this message in context: http://openssl.6102.n7.nabble.com/OpenSSL-non-blocking-epoll-hanging-on-data-receiving-tp66355p66396.html Sent from the OpenSSL - User mailing list archive at Nabble.com. -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
On 27/05/16 15:51, counterpoint wrote:
> Seems to always be zero, Matt, as it should be. This gives some idea of
> what is going on:
>
> Breakpoint 4, dcb_basic_read_SSL (dcb=0x7fffdc0158d0,
> nsingleread=0x7fff86fc) at /root/MaxScale/server/core/dcb.c:1218
> 1218switch (SSL_get_error(dcb->ssl, *nsingleread))
> $352 = 16384
>
> Breakpoint 5, dcb_basic_read_SSL (dcb=0x7fffdc0158d0,
> nsingleread=0x7fff86fc) at /root/MaxScale/server/core/dcb.c:1222
> 1222MXS_DEBUG("%lu [%s] Read %d bytes from dcb %p in state %s "
> Fri May 27 16:47:12 BST 2016
> $353 = "Successful SSL read"
>
> Breakpoint 1, dcb_read_SSL (dcb=0x7fffdc0158d0, head=0x7fff87d0)
> at /root/MaxScale/server/core/dcb.c:1191
> 1191ss_dassert(gwbuf_length(*head) == (start_length + nreadtotal));
> $354 = 0
>
> Breakpoint 9, dcb_write (dcb=0x678ef0, queue=0x67d300)
> at /root/MaxScale/server/core/dcb.c:1370
> 1370below_water = (dcb->high_water && dcb->writeqlen <
> dcb->high_water);
> Fri May 27 16:47:12 BST 2016
> $355 = 16384
> $356 = "Writing to the client"
>
> Breakpoint 4 shows the number of bytes read. Breakpoint 5 shows the case
> for analysis of the return code from the read. Breakpoint 1 shows the
> value of s->s3->rbuf.left Breakpoint 9 shows the number of bytes being
> passed to the write function.
So, if I understand you correctly, s->s3->rbuf.left is always 0
(indicating that there is no unprocessed data that OpenSSL has buffered)
and SSL_pending() returns 0 (indicating that OpenSSL has no processed
data bufferred). And this is the point where the hang in epoll occurs?
It sounds to me like OpenSSL on the server is behaving as expected. Its
processed all the data it has received and sent it on to your
application. This looks like either:
- the client hasn't sent the data that you're expecting to receive
(perhaps because of a problem on the client side)
or
- your server application hasn't correctly responded to something that
the client sent you.
Either way it looks to me like an application protocol level issue
rather than an SSL/TLS level issue. So the question is what data are you
waiting for from the client at this point? Does the client think it sent
it? Perhaps its sitting in some buffer somewhere. You need to work out
at the application protocol level what state things are in, and why the
client and server are apparently out of sync with each other.
Matt
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
Seems to always be zero, Matt, as it should be. This gives some idea of what
is going on:
Breakpoint 4, dcb_basic_read_SSL (dcb=0x7fffdc0158d0,
nsingleread=0x7fff86fc) at /root/MaxScale/server/core/dcb.c:12181218
switch (SSL_get_error(dcb->ssl, *nsingleread))$352 = 16384Breakpoint 5,
dcb_basic_read_SSL (dcb=0x7fffdc0158d0,nsingleread=0x7fff86fc) at
/root/MaxScale/server/core/dcb.c:12221222MXS_DEBUG("%lu [%s]
Read %d bytes from dcb %p in state %s "Fri May 27 16:47:12 BST 2016$353 =
"Successful SSL read"Breakpoint 1, dcb_read_SSL (dcb=0x7fffdc0158d0,
head=0x7fff87d0)at /root/MaxScale/server/core/dcb.c:11911191
ss_dassert(gwbuf_length(*head) == (start_length + nreadtotal));$354 =
0Breakpoint 9, dcb_write (dcb=0x678ef0, queue=0x67d300)at
/root/MaxScale/server/core/dcb.c:13701370below_water =
(dcb->high_water && dcb->writeqlen < dcb->high_water);Fri May 27 16:47:12
BST 2016$355 = 16384$356 = "Writing to the client"
Breakpoint 4 shows the number of bytes read.Breakpoint 5 shows the case for
analysis of the return code from the read.Breakpoint 1 shows the value of
s->s3->rbuf.leftBreakpoint 9 shows the number of bytes being passed to the
write function.
--
View this message in context:
http://openssl.6102.n7.nabble.com/OpenSSL-non-blocking-epoll-hanging-on-data-receiving-tp66355p66394.html
Sent from the OpenSSL - User mailing list archive at Nabble.com.--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
On 27/05/16 13:23, counterpoint wrote: > Thanks for the comments, Matt. > >> read_ahead and SSL_pending() do not play nicely together unfortunately. >> See the master (1.1.0) version of the SSL_pending() documentation which >> discusses this issue and introduced the new function SSL_has_pending() >> which addresses it: > >> https://www.openssl.org/docs/manmaster/ssl/SSL_pending.html > > I looked at SSL_has_pending, but can't easily use it as the software needs > to build on standard distributions, as far as possible, and it isn't in e.g. > CentOS 6.5 > >> So it sounds like this is during reading of application data? Has >> SSL_accept() returned successfully, and you are now wanting to call >> SSL_read()? > > It's getting a lot further than that. The SSL_accept returns success, and a > whole series of SSL_read and SSL_write calls look to be operating fine. It > appears that most of the 500KB of data used in the test is transferred, but > (probably near the end) the process hangs. Presumably SSL_pending is > returning 0 and there are no further EPOLLIN events. Or something like > that. > > I expected that turning off read ahead would fix both server and client, but > it seems not. I've read all the configuration options I can find, but so far > haven't found a solution to the server side application. It would be interesting to know what the value of "s->s3->rbuf.left" is (where "s" is your SSL object) when it gets to this point. SSL_pending() tells you how much buffered and processed data is left that OpenSSL can provide. The above value tells you how much buffered and raw *unprocessed* data is left. If you've turned read_ahead off it should be zero. I wouldn't recommend looking at that in a production app (you won't be able to access it in 1.1.0) but for debugging purposes it would be interesting. Matt -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
Thanks for the comments, Matt. > read_ahead and SSL_pending() do not play nicely together unfortunately. > See the master (1.1.0) version of the SSL_pending() documentation which > discusses this issue and introduced the new function SSL_has_pending() > which addresses it: > https://www.openssl.org/docs/manmaster/ssl/SSL_pending.html I looked at SSL_has_pending, but can't easily use it as the software needs to build on standard distributions, as far as possible, and it isn't in e.g. CentOS 6.5 > So it sounds like this is during reading of application data? Has > SSL_accept() returned successfully, and you are now wanting to call > SSL_read()? It's getting a lot further than that. The SSL_accept returns success, and a whole series of SSL_read and SSL_write calls look to be operating fine. It appears that most of the 500KB of data used in the test is transferred, but (probably near the end) the process hangs. Presumably SSL_pending is returning 0 and there are no further EPOLLIN events. Or something like that. I expected that turning off read ahead would fix both server and client, but it seems not. I've read all the configuration options I can find, but so far haven't found a solution to the server side application. -- View this message in context: http://openssl.6102.n7.nabble.com/OpenSSL-non-blocking-epoll-hanging-on-data-receiving-tp66355p66383.html Sent from the OpenSSL - User mailing list archive at Nabble.com. -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
On 27/05/16 07:32, counterpoint wrote: > Hmm, some progress, but still puzzled. When my code is acting as the client, > it seems that the problem can be overcome by calling SSL_set_read_ahead with > a zero parameter, to turn off reading ahead. This is done just before > calling SSL_connect. The application now seems able to read megabytes of > data from the server without hanging. read_ahead and SSL_pending() do not play nicely together unfortunately. See the master (1.1.0) version of the SSL_pending() documentation which discusses this issue and introduced the new function SSL_has_pending() which addresses it: https://www.openssl.org/docs/manmaster/ssl/SSL_pending.html > > However, adding a similar call just before SSL_accept does not solve the > problem when the application is the server, and is reading a lot of data > from the client. It looks as if the data is read (certainly hundreds of KB > in packets of 16384 bytes) but the application then hangs. Presumably for > lack of any trigger (such as EPOLLIN) to generate any further activity. So it sounds like this is during reading of application data? Has SSL_accept() returned successfully, and you are now wanting to call SSL_read()? Matt -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] OpenSSL non-blocking epoll hanging on data receiving
Hmm, some progress, but still puzzled. When my code is acting as the client, it seems that the problem can be overcome by calling SSL_set_read_ahead with a zero parameter, to turn off reading ahead. This is done just before calling SSL_connect. The application now seems able to read megabytes of data from the server without hanging. However, adding a similar call just before SSL_accept does not solve the problem when the application is the server, and is reading a lot of data from the client. It looks as if the data is read (certainly hundreds of KB in packets of 16384 bytes) but the application then hangs. Presumably for lack of any trigger (such as EPOLLIN) to generate any further activity. Any suggestions please? -- View this message in context: http://openssl.6102.n7.nabble.com/OpenSSL-non-blocking-epoll-hanging-on-data-receiving-tp66355p66370.html Sent from the OpenSSL - User mailing list archive at Nabble.com. -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
