> On 19 Oct 2021, at 04:54, Maxim Dounin <mdou...@mdounin.ru> wrote: > > Hello! > > On Mon, Oct 18, 2021 at 06:26:47PM +0300, Sergey Kandaurov wrote: > >>> On 27 Sep 2021, at 16:18, Maxim Dounin <mdou...@mdounin.ru> wrote: >>> >>> # HG changeset patch >>> # User Maxim Dounin <mdou...@mdounin.ru> >>> # Date 1632717779 -10800 >>> # Mon Sep 27 07:42:59 2021 +0300 >>> # Node ID ff514bf17f7f2257dcf036c5c973b74672cefa9a >>> # Parent 8f0fd60c33c106fba5f1ce3cafe990f15fcccc0c >>> SSL: SSL_sendfile() support with kernel TLS. >>> >>> Requires OpenSSL 3.0 compiled with "enable-ktls" option. Further, KTLS >>> needs to be enabled in kernel, and in OpenSSL, either via OpenSSL >>> configuration file or with "ssl_conf_command Options KTLS;" in nginx >>> configuration. >>> >>> On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and >>> can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf". >> >> I am not sure about mentioning ktls_ocf.ko in the commit message. >> The module is only present in FreeBSD 13.0, it was removed post 13.0, >> and the functionality is now always present in kernels with KERN_TLS: >> https://cgit.freebsd.org/src/commit/?id=21e3c1fbe246 >> Further, it is one of many options to enable KTLS. >> It could be better to refer to man ktls(4), instead: >> >> : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and >> : can be enabled with "sysctl kern.ipc.tls.enable=1", see man ktls(4). >> >> (but I don't insist) > > I would rather keep it explicitly mentioned, since it is a > required step on FreeBSD 13, and this is the only FreeBSD release > with KTLS so far. I don't object adding ktls(4) reference though, > updated with the following: > > : On FreeBSD, kernel TLS is available starting with FreeBSD 13.0, and > > : can be enabled with "sysctl kern.ipc.tls.enable=1" and "kldload ktls_ocf" > > : to load a software backend, see man ktls(4) for details. > >
It looks good, thanks. >>> On Linux, kernel TLS is available starting with kernel 4.13 (at least 5.2 >>> is recommended), and needs kernel compiled with CONFIG_TLS=y (with >>> CONFIG_TLS=m, which is used at least on Ubuntu 21.04 by default, >>> the tls module needs to be loaded with "modprobe tls"). >> >> On Linux I observe a problem sending data with short socket buffer space. >> It is Ubuntu 20.04 (5.4.0) and 21.04 (5.11.0), with epoll and select >> event methods. As per tcpdump traces, it looks like the buffer cannot >> be pushed to the network, although it is reported as if it was sent. >> The simplest I could grab (see below) with ssl_buffer_size 4k and sndbuf 8k >> (note that unlike SSL_write(), buffers aren't limited with ssl_buffer_size). > > You mean records? SSL buffer size limits buffering, and as a side > effect it limits maximum size of SSL records generated, since > nginx always uses the buffer to call SSL_write(). With > SSL_sendfile(), it does not limit records generated by sendfile, > since the buffer is not used for SSL_sendfile(). > > (Just for the record, as of now there is no way to limit maximum > record size with SSL_sendfile() (except may be by calling > SSL_sendfile() many times with small file fragments, but this > approach looks awful), as there are no kernel interfaces to > control maximum record size. Further, OpenSSL disables KTLS if > SSL_CTX_set_max_send_fragment() is used with anything other than > 16384, see tls1_change_cipher_state(), "ktls supports only the > maximum fragment size". I don't think this is a major problem > though.) Ok, it was useful to know. > >> It doesn't stuck starting with sndbuf 16k, so it might have something >> with how KTLS send buffers correspond with TCP send buffers. >> (In contrast, the FreeBSD sendfile is strictly constrained by the available >> send buffer space and hence shouldn't have this problem.) >> So it doesn't look like a major issue. > > I was able to reproduce this with sndbuf=32k over localhost on > Ubuntu 21.04 (5.11.0-18-generic). Does not seem to happen with > larger buffers, but might be I'm just not testing it hard enough. > > Over (emulated) network I was able to reproduce this with > sndbuf=24k, but not with larger buffers. > > [...] > >> I've added additional debugging to SSL_sendfile() >> to see that sendfile() returns EBUSY (11). > > Nitpicking: EAGAIN, not EBUSY. EBUSY on Linux is 16, and > sendfile() on Linux shouldn't return EBUSY. Yes, surely EAGAIN. Thanks for noticing. > > Overall, this looks like an issue in Linux KTLS implementation, > probably related to the socket buffer size. While it would be > good to mitigate this on our side if possible, I don't see > anything obvious (I've tried tcp_nodelay, but it doesn't help). > > [...] I think so, too. -- Sergey Kandaurov _______________________________________________ nginx-devel mailing list nginx-devel@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-devel
