Re: svn commit: r330884 - in head/sys: dev/cxgbe dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/tom

[John Baldwin]

On Tuesday, March 13, 2018 11:05:51 PM John Baldwin wrote:
> Author: jhb
> Date: Tue Mar 13 23:05:51 2018
> New Revision: 330884
> URL: https://svnweb.freebsd.org/changeset/base/330884
> 
> Log:
>   Support for TLS offload of TOE connections on T6 adapters.
>   
>   The TOE engine in Chelsio T6 adapters supports offloading of TLS
>   encryption and TCP segmentation for offloaded connections.  Sockets
>   using TLS are required to use a set of custom socket options to upload
>   RX and TX keys to the NIC and to enable RX processing.  Currently
>   these socket options are implemented as TCP options in the vendor
>   specific range.  A patched OpenSSL library will be made available in a
>   port / package for use with the TLS TOE support.

Note that making use of this requires a patched SSL library.  There is not
yet a port for one, but I'm working with brnrd@ to make a package of a
patched OpenSSL.

-- 
John Baldwin
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r330884 - in head/sys: dev/cxgbe dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/tom

[O. Hartmann]

On Wed, 14 Mar 2018 06:25:10 +0100
"O. Hartmann"  wrote:

> On Tue, 13 Mar 2018 23:05:51 + (UTC)
> John Baldwin  wrote:
> 
> > Author: jhb
> > Date: Tue Mar 13 23:05:51 2018
> > New Revision: 330884
> > URL: https://svnweb.freebsd.org/changeset/base/330884
> > 
> > Log:
> >   Support for TLS offload of TOE connections on T6 adapters.
> >   
> >   The TOE engine in Chelsio T6 adapters supports offloading of TLS
> >   encryption and TCP segmentation for offloaded connections.  Sockets
> >   using TLS are required to use a set of custom socket options to upload
> >   RX and TX keys to the NIC and to enable RX processing.  Currently
> >   these socket options are implemented as TCP options in the vendor
> >   specific range.  A patched OpenSSL library will be made available in a
> >   port / package for use with the TLS TOE support.
> >   
> >   TOE sockets can either offload both transmit and reception of TLS
> >   records or just transmit.  TLS offload (both RX and TX) is enabled by
> >   setting the dev.t6nex..tls sysctl to 1 and requires TOE to be
> >   enabled on the relevant interface.  Transmit offload can be used on
> >   any "normal" or TLS TOE socket by using the custom socket option to
> >   program a transmit key.  This permits most TOE sockets to
> >   transparently offload TLS when applications use a patched SSL library
> >   (e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
> >   library).  Receive offload can only be used with TOE sockets using the
> >   TLS mode.  The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
> >   list of TCP port numbers.  Any connection with either a local or
> >   remote port number in that list will be created as a TLS socket rather
> >   than a plain TOE socket.  Note that although this sysctl accepts an
> >   arbitrary list of port numbers, the sysctl(8) tool is only able to set
> >   sysctl nodes to a single value.  A TLS socket will hang without
> >   receiving data if used by an application that is not using a patched
> >   SSL library.  Thus, the tls_rx_ports node should be used with care.
> >   For a server mostly concerned with offloading TLS transmit, this node
> >   is not needed as plain TOE sockets will fall back to software crypto
> >   when using an unpatched SSL library.
> >   
> >   New per-interface statistics nodes are added giving counts of TLS
> >   packets and payload bytes (payload bytes do not include TLS headers or
> >   authentication tags/MACs) offloaded via the TOE engine, e.g.:
> >   
> >   dev.cc.0.stats.rx_tls_octets: 149
> >   dev.cc.0.stats.rx_tls_records: 13
> >   dev.cc.0.stats.tx_tls_octets: 26501823
> >   dev.cc.0.stats.tx_tls_records: 1620
> >   
> >   TLS transmit work requests are constructed by a new variant of
> >   t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.
> >   
> >   TLS transmit work requests require a buffer containing IVs.  If the
> >   IVs are too large to fit into the work request, a separate buffer is
> >   allocated when constructing a work request.  This buffer is associated
> >   with the transmit descriptor and freed when the descriptor is ACKed by
> >   the adapter.
> >   
> >   Received TLS frames use two new CPL messages.  The first message is a
> >   CPL_TLS_DATA containing the decryped payload of a single TLS record.
> >   The handler places the mbuf containing the received payload on an
> >   mbufq in the TOE pcb.  The second message is a CPL_RX_TLS_CMP message
> >   which includes a copy of the TLS header and indicates if there were
> >   any errors.  The handler for this message places the TLS header into
> >   the socket buffer followed by the saved mbuf with the payload data.
> >   Both of these handlers are contained in tom/t4_tls.c.
> >   
> >   A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
> >   including send_rx_credits(), a new send_rx_modulate(), and
> >   t4_close_conn().
> >   
> >   TLS keys for both transmit and receive are stored in onboard memory
> >   in the NIC in the "TLS keys" memory region.
> >   
> >   In some cases a TLS socket can hang with pending data available in the
> >   NIC that is not delivered to the host.  As a workaround, TLS sockets
> >   are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
> >   any data is read from a TLS socket.  In addition, a fallback timer will
> >   periodically send CPL_RX_DATA_ACK messages to the NIC for connections
> >   that are still in the handshake phase.  Once the connection has
> >   finished the handshake and programmed RX keys via the socket option,
> >   the timer is stopped.
> >   
> >   A new function select_ulp_mode() is used to determine what sub-mode a
> >   given TOE socket should use (plain TOE, DDP, or TLS).  The existing
> >   set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
> >   handles initialization of TLS-specific state when necessary in
> >   addition to DDP-specific state.
> >   

Re: svn commit: r330884 - in head/sys: dev/cxgbe dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/tom

[O. Hartmann]

On Tue, 13 Mar 2018 23:05:51 + (UTC)
John Baldwin  wrote:

> Author: jhb
> Date: Tue Mar 13 23:05:51 2018
> New Revision: 330884
> URL: https://svnweb.freebsd.org/changeset/base/330884
> 
> Log:
>   Support for TLS offload of TOE connections on T6 adapters.
>   
>   The TOE engine in Chelsio T6 adapters supports offloading of TLS
>   encryption and TCP segmentation for offloaded connections.  Sockets
>   using TLS are required to use a set of custom socket options to upload
>   RX and TX keys to the NIC and to enable RX processing.  Currently
>   these socket options are implemented as TCP options in the vendor
>   specific range.  A patched OpenSSL library will be made available in a
>   port / package for use with the TLS TOE support.
>   
>   TOE sockets can either offload both transmit and reception of TLS
>   records or just transmit.  TLS offload (both RX and TX) is enabled by
>   setting the dev.t6nex..tls sysctl to 1 and requires TOE to be
>   enabled on the relevant interface.  Transmit offload can be used on
>   any "normal" or TLS TOE socket by using the custom socket option to
>   program a transmit key.  This permits most TOE sockets to
>   transparently offload TLS when applications use a patched SSL library
>   (e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
>   library).  Receive offload can only be used with TOE sockets using the
>   TLS mode.  The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
>   list of TCP port numbers.  Any connection with either a local or
>   remote port number in that list will be created as a TLS socket rather
>   than a plain TOE socket.  Note that although this sysctl accepts an
>   arbitrary list of port numbers, the sysctl(8) tool is only able to set
>   sysctl nodes to a single value.  A TLS socket will hang without
>   receiving data if used by an application that is not using a patched
>   SSL library.  Thus, the tls_rx_ports node should be used with care.
>   For a server mostly concerned with offloading TLS transmit, this node
>   is not needed as plain TOE sockets will fall back to software crypto
>   when using an unpatched SSL library.
>   
>   New per-interface statistics nodes are added giving counts of TLS
>   packets and payload bytes (payload bytes do not include TLS headers or
>   authentication tags/MACs) offloaded via the TOE engine, e.g.:
>   
>   dev.cc.0.stats.rx_tls_octets: 149
>   dev.cc.0.stats.rx_tls_records: 13
>   dev.cc.0.stats.tx_tls_octets: 26501823
>   dev.cc.0.stats.tx_tls_records: 1620
>   
>   TLS transmit work requests are constructed by a new variant of
>   t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.
>   
>   TLS transmit work requests require a buffer containing IVs.  If the
>   IVs are too large to fit into the work request, a separate buffer is
>   allocated when constructing a work request.  This buffer is associated
>   with the transmit descriptor and freed when the descriptor is ACKed by
>   the adapter.
>   
>   Received TLS frames use two new CPL messages.  The first message is a
>   CPL_TLS_DATA containing the decryped payload of a single TLS record.
>   The handler places the mbuf containing the received payload on an
>   mbufq in the TOE pcb.  The second message is a CPL_RX_TLS_CMP message
>   which includes a copy of the TLS header and indicates if there were
>   any errors.  The handler for this message places the TLS header into
>   the socket buffer followed by the saved mbuf with the payload data.
>   Both of these handlers are contained in tom/t4_tls.c.
>   
>   A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
>   including send_rx_credits(), a new send_rx_modulate(), and
>   t4_close_conn().
>   
>   TLS keys for both transmit and receive are stored in onboard memory
>   in the NIC in the "TLS keys" memory region.
>   
>   In some cases a TLS socket can hang with pending data available in the
>   NIC that is not delivered to the host.  As a workaround, TLS sockets
>   are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
>   any data is read from a TLS socket.  In addition, a fallback timer will
>   periodically send CPL_RX_DATA_ACK messages to the NIC for connections
>   that are still in the handshake phase.  Once the connection has
>   finished the handshake and programmed RX keys via the socket option,
>   the timer is stopped.
>   
>   A new function select_ulp_mode() is used to determine what sub-mode a
>   given TOE socket should use (plain TOE, DDP, or TLS).  The existing
>   set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
>   handles initialization of TLS-specific state when necessary in
>   addition to DDP-specific state.
>   
>   Since TLS sockets do not receive individual TCP segments but always
>   receive full TLS records, they can receive more data than is available
>   in the current window (e.g. if a 16k TLS record is received but the
>   socket buffer is itself 16k).  To 

svn commit: r330884 - in head/sys: dev/cxgbe dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/tom

[John Baldwin]

Author: jhb
Date: Tue Mar 13 23:05:51 2018
New Revision: 330884
URL: https://svnweb.freebsd.org/changeset/base/330884

Log:
  Support for TLS offload of TOE connections on T6 adapters.
  
  The TOE engine in Chelsio T6 adapters supports offloading of TLS
  encryption and TCP segmentation for offloaded connections.  Sockets
  using TLS are required to use a set of custom socket options to upload
  RX and TX keys to the NIC and to enable RX processing.  Currently
  these socket options are implemented as TCP options in the vendor
  specific range.  A patched OpenSSL library will be made available in a
  port / package for use with the TLS TOE support.
  
  TOE sockets can either offload both transmit and reception of TLS
  records or just transmit.  TLS offload (both RX and TX) is enabled by
  setting the dev.t6nex..tls sysctl to 1 and requires TOE to be
  enabled on the relevant interface.  Transmit offload can be used on
  any "normal" or TLS TOE socket by using the custom socket option to
  program a transmit key.  This permits most TOE sockets to
  transparently offload TLS when applications use a patched SSL library
  (e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
  library).  Receive offload can only be used with TOE sockets using the
  TLS mode.  The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
  list of TCP port numbers.  Any connection with either a local or
  remote port number in that list will be created as a TLS socket rather
  than a plain TOE socket.  Note that although this sysctl accepts an
  arbitrary list of port numbers, the sysctl(8) tool is only able to set
  sysctl nodes to a single value.  A TLS socket will hang without
  receiving data if used by an application that is not using a patched
  SSL library.  Thus, the tls_rx_ports node should be used with care.
  For a server mostly concerned with offloading TLS transmit, this node
  is not needed as plain TOE sockets will fall back to software crypto
  when using an unpatched SSL library.
  
  New per-interface statistics nodes are added giving counts of TLS
  packets and payload bytes (payload bytes do not include TLS headers or
  authentication tags/MACs) offloaded via the TOE engine, e.g.:
  
  dev.cc.0.stats.rx_tls_octets: 149
  dev.cc.0.stats.rx_tls_records: 13
  dev.cc.0.stats.tx_tls_octets: 26501823
  dev.cc.0.stats.tx_tls_records: 1620
  
  TLS transmit work requests are constructed by a new variant of
  t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.
  
  TLS transmit work requests require a buffer containing IVs.  If the
  IVs are too large to fit into the work request, a separate buffer is
  allocated when constructing a work request.  This buffer is associated
  with the transmit descriptor and freed when the descriptor is ACKed by
  the adapter.
  
  Received TLS frames use two new CPL messages.  The first message is a
  CPL_TLS_DATA containing the decryped payload of a single TLS record.
  The handler places the mbuf containing the received payload on an
  mbufq in the TOE pcb.  The second message is a CPL_RX_TLS_CMP message
  which includes a copy of the TLS header and indicates if there were
  any errors.  The handler for this message places the TLS header into
  the socket buffer followed by the saved mbuf with the payload data.
  Both of these handlers are contained in tom/t4_tls.c.
  
  A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
  including send_rx_credits(), a new send_rx_modulate(), and
  t4_close_conn().
  
  TLS keys for both transmit and receive are stored in onboard memory
  in the NIC in the "TLS keys" memory region.
  
  In some cases a TLS socket can hang with pending data available in the
  NIC that is not delivered to the host.  As a workaround, TLS sockets
  are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
  any data is read from a TLS socket.  In addition, a fallback timer will
  periodically send CPL_RX_DATA_ACK messages to the NIC for connections
  that are still in the handshake phase.  Once the connection has
  finished the handshake and programmed RX keys via the socket option,
  the timer is stopped.
  
  A new function select_ulp_mode() is used to determine what sub-mode a
  given TOE socket should use (plain TOE, DDP, or TLS).  The existing
  set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
  handles initialization of TLS-specific state when necessary in
  addition to DDP-specific state.
  
  Since TLS sockets do not receive individual TCP segments but always
  receive full TLS records, they can receive more data than is available
  in the current window (e.g. if a 16k TLS record is received but the
  socket buffer is itself 16k).  To cope with this, just drop the window
  to 0 when this happens, but track the overage and "eat" the overage as
  it is read from the socket buffer not opening the window (or adding
  rx_credits) for the overage bytes.
  
  Reviewed by:  np (earlier version)