Re: svn commit: r330884 - in head/sys: dev/cxgbe dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/tom

[John Baldwin]

On Tuesday, March 13, 2018 11:05:51 PM John Baldwin wrote:
> Author: jhb
> Date: Tue Mar 13 23:05:51 2018
> New Revision: 330884
> URL: https://svnweb.freebsd.org/changeset/base/330884
> 
> Log:
>   Support for TLS offload of TOE connections on T6 adapters.
>   
>   The TOE engine in Chelsio T6 adapters supports offloading of TLS
>   encryption and TCP segmentation for offloaded connections.  Sockets
>   using TLS are required to use a set of custom socket options to upload
>   RX and TX keys to the NIC and to enable RX processing.  Currently
>   these socket options are implemented as TCP options in the vendor
>   specific range.  A patched OpenSSL library will be made available in a
>   port / package for use with the TLS TOE support.

Note that making use of this requires a patched SSL library.  There is not
yet a port for one, but I'm working with brnrd@ to make a package of a
patched OpenSSL.

-- 
John Baldwin
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r330884 - in head/sys: dev/cxgbe dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/tom

[O. Hartmann]

On Wed, 14 Mar 2018 06:25:10 +0100
"O. Hartmann"  wrote:

> On Tue, 13 Mar 2018 23:05:51 + (UTC)
> John Baldwin  wrote:
> 
> > Author: jhb
> > Date: Tue Mar 13 23:05:51 2018
> > New Revision: 330884
> > URL: https://svnweb.freebsd.org/changeset/base/330884
> > 
> > Log:
> >   Support for TLS offload of TOE connections on T6 adapters.
> >   
> >   The TOE engine in Chelsio T6 adapters supports offloading of TLS
> >   encryption and TCP segmentation for offloaded connections.  Sockets
> >   using TLS are required to use a set of custom socket options to upload
> >   RX and TX keys to the NIC and to enable RX processing.  Currently
> >   these socket options are implemented as TCP options in the vendor
> >   specific range.  A patched OpenSSL library will be made available in a
> >   port / package for use with the TLS TOE support.
> >   
> >   TOE sockets can either offload both transmit and reception of TLS
> >   records or just transmit.  TLS offload (both RX and TX) is enabled by
> >   setting the dev.t6nex..tls sysctl to 1 and requires TOE to be
> >   enabled on the relevant interface.  Transmit offload can be used on
> >   any "normal" or TLS TOE socket by using the custom socket option to
> >   program a transmit key.  This permits most TOE sockets to
> >   transparently offload TLS when applications use a patched SSL library
> >   (e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
> >   library).  Receive offload can only be used with TOE sockets using the
> >   TLS mode.  The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
> >   list of TCP port numbers.  Any connection with either a local or
> >   remote port number in that list will be created as a TLS socket rather
> >   than a plain TOE socket.  Note that although this sysctl accepts an
> >   arbitrary list of port numbers, the sysctl(8) tool is only able to set
> >   sysctl nodes to a single value.  A TLS socket will hang without
> >   receiving data if used by an application that is not using a patched
> >   SSL library.  Thus, the tls_rx_ports node should be used with care.
> >   For a server mostly concerned with offloading TLS transmit, this node
> >   is not needed as plain TOE sockets will fall back to software crypto
> >   when using an unpatched SSL library.
> >   
> >   New per-interface statistics nodes are added giving counts of TLS
> >   packets and payload bytes (payload bytes do not include TLS headers or
> >   authentication tags/MACs) offloaded via the TOE engine, e.g.:
> >   
> >   dev.cc.0.stats.rx_tls_octets: 149
> >   dev.cc.0.stats.rx_tls_records: 13
> >   dev.cc.0.stats.tx_tls_octets: 26501823
> >   dev.cc.0.stats.tx_tls_records: 1620
> >   
> >   TLS transmit work requests are constructed by a new variant of
> >   t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.
> >   
> >   TLS transmit work requests require a buffer containing IVs.  If the
> >   IVs are too large to fit into the work request, a separate buffer is
> >   allocated when constructing a work request.  This buffer is associated
> >   with the transmit descriptor and freed when the descriptor is ACKed by
> >   the adapter.
> >   
> >   Received TLS frames use two new CPL messages.  The first message is a
> >   CPL_TLS_DATA containing the decryped payload of a single TLS record.
> >   The handler places the mbuf containing the received payload on an
> >   mbufq in the TOE pcb.  The second message is a CPL_RX_TLS_CMP message
> >   which includes a copy of the TLS header and indicates if there were
> >   any errors.  The handler for this message places the TLS header into
> >   the socket buffer followed by the saved mbuf with the payload data.
> >   Both of these handlers are contained in tom/t4_tls.c.
> >   
> >   A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
> >   including send_rx_credits(), a new send_rx_modulate(), and
> >   t4_close_conn().
> >   
> >   TLS keys for both transmit and receive are stored in onboard memory
> >   in the NIC in the "TLS keys" memory region.
> >   
> >   In some cases a TLS socket can hang with pending data available in the
> >   NIC that is not delivered to the host.  As a workaround, TLS sockets
> >   are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
> >   any data is read from a TLS socket.  In addition, a fallback timer will
> >   periodically send CPL_RX_DATA_ACK messages to the NIC for connections
> >   that are still in the handshake phase.  Once the connection has
> >   finished the handshake and programmed RX keys via the socket option,
> >   the timer is stopped.
> >   
> >   A new function select_ulp_mode() is used to determine what sub-mode a
> >   given TOE socket should use (plain TOE, DDP, or TLS).  The existing
> >   set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
> >   handles initialization of TLS-specific state when necessary in
> >   addition to DDP-specific state.
> >   

Re: svn commit: r330884 - in head/sys: dev/cxgbe dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/tom

[O. Hartmann]

On Tue, 13 Mar 2018 23:05:51 + (UTC)
John Baldwin  wrote:

> Author: jhb
> Date: Tue Mar 13 23:05:51 2018
> New Revision: 330884
> URL: https://svnweb.freebsd.org/changeset/base/330884
> 
> Log:
>   Support for TLS offload of TOE connections on T6 adapters.
>   
>   The TOE engine in Chelsio T6 adapters supports offloading of TLS
>   encryption and TCP segmentation for offloaded connections.  Sockets
>   using TLS are required to use a set of custom socket options to upload
>   RX and TX keys to the NIC and to enable RX processing.  Currently
>   these socket options are implemented as TCP options in the vendor
>   specific range.  A patched OpenSSL library will be made available in a
>   port / package for use with the TLS TOE support.
>   
>   TOE sockets can either offload both transmit and reception of TLS
>   records or just transmit.  TLS offload (both RX and TX) is enabled by
>   setting the dev.t6nex..tls sysctl to 1 and requires TOE to be
>   enabled on the relevant interface.  Transmit offload can be used on
>   any "normal" or TLS TOE socket by using the custom socket option to
>   program a transmit key.  This permits most TOE sockets to
>   transparently offload TLS when applications use a patched SSL library
>   (e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
>   library).  Receive offload can only be used with TOE sockets using the
>   TLS mode.  The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
>   list of TCP port numbers.  Any connection with either a local or
>   remote port number in that list will be created as a TLS socket rather
>   than a plain TOE socket.  Note that although this sysctl accepts an
>   arbitrary list of port numbers, the sysctl(8) tool is only able to set
>   sysctl nodes to a single value.  A TLS socket will hang without
>   receiving data if used by an application that is not using a patched
>   SSL library.  Thus, the tls_rx_ports node should be used with care.
>   For a server mostly concerned with offloading TLS transmit, this node
>   is not needed as plain TOE sockets will fall back to software crypto
>   when using an unpatched SSL library.
>   
>   New per-interface statistics nodes are added giving counts of TLS
>   packets and payload bytes (payload bytes do not include TLS headers or
>   authentication tags/MACs) offloaded via the TOE engine, e.g.:
>   
>   dev.cc.0.stats.rx_tls_octets: 149
>   dev.cc.0.stats.rx_tls_records: 13
>   dev.cc.0.stats.tx_tls_octets: 26501823
>   dev.cc.0.stats.tx_tls_records: 1620
>   
>   TLS transmit work requests are constructed by a new variant of
>   t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.
>   
>   TLS transmit work requests require a buffer containing IVs.  If the
>   IVs are too large to fit into the work request, a separate buffer is
>   allocated when constructing a work request.  This buffer is associated
>   with the transmit descriptor and freed when the descriptor is ACKed by
>   the adapter.
>   
>   Received TLS frames use two new CPL messages.  The first message is a
>   CPL_TLS_DATA containing the decryped payload of a single TLS record.
>   The handler places the mbuf containing the received payload on an
>   mbufq in the TOE pcb.  The second message is a CPL_RX_TLS_CMP message
>   which includes a copy of the TLS header and indicates if there were
>   any errors.  The handler for this message places the TLS header into
>   the socket buffer followed by the saved mbuf with the payload data.
>   Both of these handlers are contained in tom/t4_tls.c.
>   
>   A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
>   including send_rx_credits(), a new send_rx_modulate(), and
>   t4_close_conn().
>   
>   TLS keys for both transmit and receive are stored in onboard memory
>   in the NIC in the "TLS keys" memory region.
>   
>   In some cases a TLS socket can hang with pending data available in the
>   NIC that is not delivered to the host.  As a workaround, TLS sockets
>   are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
>   any data is read from a TLS socket.  In addition, a fallback timer will
>   periodically send CPL_RX_DATA_ACK messages to the NIC for connections
>   that are still in the handshake phase.  Once the connection has
>   finished the handshake and programmed RX keys via the socket option,
>   the timer is stopped.
>   
>   A new function select_ulp_mode() is used to determine what sub-mode a
>   given TOE socket should use (plain TOE, DDP, or TLS).  The existing
>   set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
>   handles initialization of TLS-specific state when necessary in
>   addition to DDP-specific state.
>   
>   Since TLS sockets do not receive individual TCP segments but always
>   receive full TLS records, they can receive more data than is available
>   in the current window (e.g. if a 16k TLS record is received but the
>   socket buffer is itself 16k).  To