On Wed, 14 Mar 2018 06:25:10 +0100
"O. Hartmann" <ohartm...@walstatt.org> wrote:

> On Tue, 13 Mar 2018 23:05:51 +0000 (UTC)
> John Baldwin <j...@freebsd.org> wrote:
> 
> > Author: jhb
> > Date: Tue Mar 13 23:05:51 2018
> > New Revision: 330884
> > URL: https://svnweb.freebsd.org/changeset/base/330884
> > 
> > Log:
> >   Support for TLS offload of TOE connections on T6 adapters.
> >   
> >   The TOE engine in Chelsio T6 adapters supports offloading of TLS
> >   encryption and TCP segmentation for offloaded connections.  Sockets
> >   using TLS are required to use a set of custom socket options to upload
> >   RX and TX keys to the NIC and to enable RX processing.  Currently
> >   these socket options are implemented as TCP options in the vendor
> >   specific range.  A patched OpenSSL library will be made available in a
> >   port / package for use with the TLS TOE support.
> >   
> >   TOE sockets can either offload both transmit and reception of TLS
> >   records or just transmit.  TLS offload (both RX and TX) is enabled by
> >   setting the dev.t6nex.<x>.tls sysctl to 1 and requires TOE to be
> >   enabled on the relevant interface.  Transmit offload can be used on
> >   any "normal" or TLS TOE socket by using the custom socket option to
> >   program a transmit key.  This permits most TOE sockets to
> >   transparently offload TLS when applications use a patched SSL library
> >   (e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
> >   library).  Receive offload can only be used with TOE sockets using the
> >   TLS mode.  The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
> >   list of TCP port numbers.  Any connection with either a local or
> >   remote port number in that list will be created as a TLS socket rather
> >   than a plain TOE socket.  Note that although this sysctl accepts an
> >   arbitrary list of port numbers, the sysctl(8) tool is only able to set
> >   sysctl nodes to a single value.  A TLS socket will hang without
> >   receiving data if used by an application that is not using a patched
> >   SSL library.  Thus, the tls_rx_ports node should be used with care.
> >   For a server mostly concerned with offloading TLS transmit, this node
> >   is not needed as plain TOE sockets will fall back to software crypto
> >   when using an unpatched SSL library.
> >   
> >   New per-interface statistics nodes are added giving counts of TLS
> >   packets and payload bytes (payload bytes do not include TLS headers or
> >   authentication tags/MACs) offloaded via the TOE engine, e.g.:
> >   
> >   dev.cc.0.stats.rx_tls_octets: 149
> >   dev.cc.0.stats.rx_tls_records: 13
> >   dev.cc.0.stats.tx_tls_octets: 26501823
> >   dev.cc.0.stats.tx_tls_records: 1620
> >   
> >   TLS transmit work requests are constructed by a new variant of
> >   t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.
> >   
> >   TLS transmit work requests require a buffer containing IVs.  If the
> >   IVs are too large to fit into the work request, a separate buffer is
> >   allocated when constructing a work request.  This buffer is associated
> >   with the transmit descriptor and freed when the descriptor is ACKed by
> >   the adapter.
> >   
> >   Received TLS frames use two new CPL messages.  The first message is a
> >   CPL_TLS_DATA containing the decryped payload of a single TLS record.
> >   The handler places the mbuf containing the received payload on an
> >   mbufq in the TOE pcb.  The second message is a CPL_RX_TLS_CMP message
> >   which includes a copy of the TLS header and indicates if there were
> >   any errors.  The handler for this message places the TLS header into
> >   the socket buffer followed by the saved mbuf with the payload data.
> >   Both of these handlers are contained in tom/t4_tls.c.
> >   
> >   A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
> >   including send_rx_credits(), a new send_rx_modulate(), and
> >   t4_close_conn().
> >   
> >   TLS keys for both transmit and receive are stored in onboard memory
> >   in the NIC in the "TLS keys" memory region.
> >   
> >   In some cases a TLS socket can hang with pending data available in the
> >   NIC that is not delivered to the host.  As a workaround, TLS sockets
> >   are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
> >   any data is read from a TLS socket.  In addition, a fallback timer will
> >   periodically send CPL_RX_DATA_ACK messages to the NIC for connections
> >   that are still in the handshake phase.  Once the connection has
> >   finished the handshake and programmed RX keys via the socket option,
> >   the timer is stopped.
> >   
> >   A new function select_ulp_mode() is used to determine what sub-mode a
> >   given TOE socket should use (plain TOE, DDP, or TLS).  The existing
> >   set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
> >   handles initialization of TLS-specific state when necessary in
> >   addition to DDP-specific state.
> >   
> >   Since TLS sockets do not receive individual TCP segments but always
> >   receive full TLS records, they can receive more data than is available
> >   in the current window (e.g. if a 16k TLS record is received but the
> >   socket buffer is itself 16k).  To cope with this, just drop the window
> >   to 0 when this happens, but track the overage and "eat" the overage as
> >   it is read from the socket buffer not opening the window (or adding
> >   rx_credits) for the overage bytes.
> >   
> >   Reviewed by:      np (earlier version)
> >   Sponsored by:     Chelsio Communications
> >   Differential Revision:    https://reviews.freebsd.org/D14529
> > 
> > Added:
> >   head/sys/dev/cxgbe/tom/t4_tls.c   (contents, props changed)
> >   head/sys/dev/cxgbe/tom/t4_tls.h   (contents, props changed)
> > Modified:
> >   head/sys/dev/cxgbe/adapter.h
> >   head/sys/dev/cxgbe/firmware/t6fw_cfg.txt
> >   head/sys/dev/cxgbe/offload.h
> >   head/sys/dev/cxgbe/t4_main.c
> >   head/sys/dev/cxgbe/tom/t4_connect.c
> >   head/sys/dev/cxgbe/tom/t4_cpl_io.c
> >   head/sys/dev/cxgbe/tom/t4_listen.c
> >   head/sys/dev/cxgbe/tom/t4_tom.c
> >   head/sys/dev/cxgbe/tom/t4_tom.h
> >   head/sys/modules/cxgbe/tom/Makefile
> > 
> > Modified: head/sys/dev/cxgbe/adapter.h
> > ==============================================================================
> > --- head/sys/dev/cxgbe/adapter.h    Tue Mar 13 22:54:29 2018
> > (r330883) +++ head/sys/dev/cxgbe/adapter.h  Tue Mar 13 23:05:51
> > 2018        (r330884) @@ -297,6 +297,10 @@ struct port_info {
> >     struct port_stats stats;
> >     u_int tnl_cong_drops;
> >     u_int tx_parse_error;
> > +   u_long  tx_tls_records;
> > +   u_long  tx_tls_octets;
> > +   u_long  rx_tls_records;
> > +   u_long  rx_tls_octets;
> >  
> >     struct callout tick;
> >  };
> > 
> > Modified: head/sys/dev/cxgbe/firmware/t6fw_cfg.txt
> > ==============================================================================
> > --- head/sys/dev/cxgbe/firmware/t6fw_cfg.txt        Tue Mar 13 22:54:29
> > 2018        (r330883) +++ head/sys/dev/cxgbe/firmware/t6fw_cfg.txt
> > Tue Mar 13 23:05:51 2018    (r330884) @@ -163,10 +163,12 @@
> >     nserver = 512
> >     nhpfilter = 0
> >     nhash = 16384
> > -   protocol = ofld, rddp, rdmac, iscsi_initiator_pdu,
> > iscsi_target_pdu, iscsi_t10dif, crypto_lookaside
> > +   protocol = ofld, rddp, rdmac, iscsi_initiator_pdu,
> > iscsi_target_pdu, iscsi_t10dif, tlskeys, crypto_lookaside tp_l2t = 4096
> >     tp_ddp = 2
> >     tp_ddp_iscsi = 2
> > +   tp_tls_key = 3
> > +   tp_tls_mxrxsize = 17408    # 16384 + 1024, governs max rx data, pm
> > max xfer len, rx coalesce sizes tp_stag = 2
> >     tp_pbl = 5
> >     tp_rq = 7
> > @@ -273,7 +275,7 @@
> >  
> >  [fini]
> >     version = 0x1
> > -   checksum = 0x7191019f
> > +   checksum = 0x9e8952d2
> >  #
> >  # $FreeBSD$
> >  #
> > 
> > Modified: head/sys/dev/cxgbe/offload.h
> > ==============================================================================
> > --- head/sys/dev/cxgbe/offload.h    Tue Mar 13 22:54:29 2018
> > (r330883) +++ head/sys/dev/cxgbe/offload.h  Tue Mar 13 23:05:51
> > 2018        (r330884) @@ -151,6 +151,9 @@ struct tom_tunables {
> >     int sndbuf;
> >     int ddp;
> >     int rx_coalesce;
> > +   int tls;
> > +   int *tls_rx_ports;
> > +   int num_tls_rx_ports;
> >     int tx_align;
> >     int tx_zcopy;
> >  };
> > 
> > Modified: head/sys/dev/cxgbe/t4_main.c
> > ==============================================================================
> > --- head/sys/dev/cxgbe/t4_main.c    Tue Mar 13 22:54:29 2018
> > (r330883) +++ head/sys/dev/cxgbe/t4_main.c  Tue Mar 13 23:05:51
> > 2018        (r330884) @@ -591,6 +591,7 @@ static int
> > sysctl_wcwr_stats(SYSCTL_HANDLER_ARGS); static int
> > sysctl_tc_params(SYSCTL_HANDLER_ARGS); #endif
> >  #ifdef TCP_OFFLOAD
> > +static int sysctl_tls_rx_ports(SYSCTL_HANDLER_ARGS);
> >  static int sysctl_tp_tick(SYSCTL_HANDLER_ARGS);
> >  static int sysctl_tp_dack_timer(SYSCTL_HANDLER_ARGS);
> >  static int sysctl_tp_timer(SYSCTL_HANDLER_ARGS);
> > @@ -1390,6 +1391,7 @@ t4_detach_common(device_t dev)
> >     free(sc->sge.iqmap, M_CXGBE);
> >     free(sc->sge.eqmap, M_CXGBE);
> >     free(sc->tids.ftid_tab, M_CXGBE);
> > +   free(sc->tt.tls_rx_ports, M_CXGBE);
> >     t4_destroy_dma_tag(sc);
> >     if (mtx_initialized(&sc->sc_lock)) {
> >             sx_xlock(&t4_list_lock);
> > @@ -5433,6 +5435,14 @@ t4_sysctls(struct adapter *sc)
> >             SYSCTL_ADD_INT(ctx, children, OID_AUTO, "rx_coalesce",
> >                 CTLFLAG_RW, &sc->tt.rx_coalesce, 0, "receive
> > coalescing"); 
> > +           sc->tt.tls = 0;
> > +           SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tls", CTLFLAG_RW,
> > +               &sc->tt.tls, 0, "Inline TLS allowed");
> > +
> > +           SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "tls_rx_ports",
> > +               CTLTYPE_INT | CTLFLAG_RW, sc, 0, sysctl_tls_rx_ports,
> > +               "I", "TCP ports that use inline TLS+TOE RX");
> > +
> >             sc->tt.tx_align = 1;
> >             SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_align",
> >                 CTLFLAG_RW, &sc->tt.tx_align, 0, "chop and align
> > payload"); @@ -5836,6 +5846,19 @@ cxgbe_sysctls(struct port_info *pi)
> >         "# of buffer-group 3 truncated packets");
> >  
> >  #undef SYSCTL_ADD_T4_PORTSTAT
> > +
> > +   SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "tx_tls_records",
> > +       CTLFLAG_RD, &pi->tx_tls_records,
> > +       "# of TLS records transmitted");
> > +   SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "tx_tls_octets",
> > +       CTLFLAG_RD, &pi->tx_tls_octets,
> > +       "# of payload octets in transmitted TLS records");
> > +   SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "rx_tls_records",
> > +       CTLFLAG_RD, &pi->rx_tls_records,
> > +       "# of TLS records received");
> > +   SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "rx_tls_octets",
> > +       CTLFLAG_RD, &pi->rx_tls_octets,
> > +       "# of payload octets in received TLS records");
> >  }
> >  
> >  static int
> > @@ -8257,6 +8280,68 @@ done:
> >  #endif
> >  
> >  #ifdef TCP_OFFLOAD
> > +static int
> > +sysctl_tls_rx_ports(SYSCTL_HANDLER_ARGS)
> > +{
> > +   struct adapter *sc = arg1;
> > +   int *old_ports, *new_ports;
> > +   int i, new_count, rc;
> > +
> > +   if (req->newptr == NULL && req->oldptr == NULL)
> > +           return (SYSCTL_OUT(req, NULL, imax(sc->tt.num_tls_rx_ports,
> > 1) *
> > +               sizeof(sc->tt.tls_rx_ports[0])));
> > +
> > +   rc = begin_synchronized_op(sc, NULL, SLEEP_OK | INTR_OK,
> > "t4tlsrx");
> > +   if (rc)
> > +           return (rc);
> > +
> > +   if (sc->tt.num_tls_rx_ports == 0) {
> > +           i = -1;
> > +           rc = SYSCTL_OUT(req, &i, sizeof(i));
> > +   } else
> > +           rc = SYSCTL_OUT(req, sc->tt.tls_rx_ports,
> > +               sc->tt.num_tls_rx_ports *
> > sizeof(sc->tt.tls_rx_ports[0]));
> > +   if (rc == 0 && req->newptr != NULL) {
> > +           new_count = req->newlen / sizeof(new_ports[0]);
> > +           new_ports = malloc(new_count * sizeof(new_ports[0]),
> > M_CXGBE,
> > +               M_WAITOK);
> > +           rc = SYSCTL_IN(req, new_ports, new_count *
> > +               sizeof(new_ports[0]));
> > +           if (rc)
> > +                   goto err;
> > +
> > +           /* Allow setting to a single '-1' to clear the list. */
> > +           if (new_count == 1 && new_ports[0] == -1) {
> > +                   ADAPTER_LOCK(sc);
> > +                   old_ports = sc->tt.tls_rx_ports;
> > +                   sc->tt.tls_rx_ports = NULL;
> > +                   sc->tt.num_tls_rx_ports = 0;
> > +                   ADAPTER_UNLOCK(sc);
> > +                   free(old_ports, M_CXGBE);
> > +           } else {
> > +                   for (i = 0; i < new_count; i++) {
> > +                           if (new_ports[i] < 1 ||
> > +                               new_ports[i] > IPPORT_MAX) {
> > +                                   rc = EINVAL;
> > +                                   goto err;
> > +                           }
> > +                   }
> > +
> > +                   ADAPTER_LOCK(sc);
> > +                   old_ports = sc->tt.tls_rx_ports;
> > +                   sc->tt.tls_rx_ports = new_ports;
> > +                   sc->tt.num_tls_rx_ports = new_count;
> > +                   ADAPTER_UNLOCK(sc);
> > +                   free(old_ports, M_CXGBE);
> > +                   new_ports = NULL;
> > +           }
> > +   err:
> > +           free(new_ports, M_CXGBE);
> > +   }
> > +   end_synchronized_op(sc, 0);
> > +   return (rc);
> > +}
> > +
> >  static void
> >  unit_conv(char *buf, size_t len, u_int val, u_int factor)
> >  {
> > 
> > Modified: head/sys/dev/cxgbe/tom/t4_connect.c
> > ==============================================================================
> > --- head/sys/dev/cxgbe/tom/t4_connect.c     Tue Mar 13 22:54:29
> > 2018        (r330883) +++ head/sys/dev/cxgbe/tom/t4_connect.c       Tue Mar
> > 13 23:05:51 2018    (r330884) @@ -142,6 +142,10 @@
> > do_act_establish(struct sge_iq *iq, const struct rss_h }
> >  
> >     make_established(toep, cpl->snd_isn, cpl->rcv_isn, cpl->tcp_opt);
> > +
> > +   if (toep->ulp_mode == ULP_MODE_TLS)
> > +           tls_establish(toep);
> > +
> >  done:
> >     INP_WUNLOCK(inp);
> >     CURVNET_RESTORE();
> > @@ -268,6 +272,11 @@ calc_opt2a(struct socket *so, struct toepcb *toep)
> >     if (toep->ulp_mode == ULP_MODE_TCPDDP)
> >             opt2 |= F_RX_FC_VALID | F_RX_FC_DDP;
> >  #endif
> > +   if (toep->ulp_mode == ULP_MODE_TLS) {
> > +           opt2 |= F_RX_FC_VALID;
> > +           opt2 &= ~V_RX_COALESCE(M_RX_COALESCE);
> > +           opt2 |= F_RX_FC_DISABLE;
> > +   }
> >  
> >     return (htobe32(opt2));
> >  }
> > @@ -378,10 +387,7 @@ t4_connect(struct toedev *tod, struct socket *so, stru
> >             DONT_OFFLOAD_ACTIVE_OPEN(ENOMEM);
> >  
> >     toep->vnet = so->so_vnet;
> > -   if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0)
> > -           set_tcpddp_ulp_mode(toep);
> > -   else
> > -           toep->ulp_mode = ULP_MODE_NONE;
> > +   set_ulp_mode(toep, select_ulp_mode(so, sc));
> >     SOCKBUF_LOCK(&so->so_rcv);
> >     /* opt0 rcv_bufsiz initially, assumes its normal meaning later */
> >     toep->rx_credits = min(select_rcv_wnd(so) >> 10, M_RCV_BUFSIZ);
> > 
> > Modified: head/sys/dev/cxgbe/tom/t4_cpl_io.c
> > ==============================================================================
> > --- head/sys/dev/cxgbe/tom/t4_cpl_io.c      Tue Mar 13 22:54:29 2018
> > (r330883) +++ head/sys/dev/cxgbe/tom/t4_cpl_io.c    Tue Mar 13 23:05:51
> > 2018        (r330884) @@ -73,9 +73,6 @@ __FBSDID("$FreeBSD$");
> >  #include "tom/t4_tom_l2t.h"
> >  #include "tom/t4_tom.h"
> >  
> > -#define
> > IS_AIOTX_MBUF(m)                                            \
> > -   ((m)->m_flags & M_EXT && (m)->m_ext.ext_flags & EXT_FLAG_AIOTX)
> > -
> >  static void        t4_aiotx_cancel(struct kaiocb *job);
> >  static void        t4_aiotx_queue_toep(struct toepcb *toep);
> >  
> > @@ -106,7 +103,7 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
> >  {
> >     struct wrqe *wr;
> >     struct fw_flowc_wr *flowc;
> > -   unsigned int nparams = ftxp ? 8 : 6, flowclen;
> > +   unsigned int nparams, flowclen, paramidx;
> >     struct vi_info *vi = toep->vi;
> >     struct port_info *pi = vi->pi;
> >     struct adapter *sc = pi->adapter;
> > @@ -116,6 +113,15 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
> >     KASSERT(!(toep->flags & TPF_FLOWC_WR_SENT),
> >         ("%s: flowc for tid %u sent already", __func__, toep->tid));
> >  
> > +   if (ftxp != NULL)
> > +           nparams = 8;
> > +   else
> > +           nparams = 6;
> > +   if (toep->ulp_mode == ULP_MODE_TLS)
> > +           nparams++;
> > +   if (toep->tls.fcplenmax != 0)
> > +           nparams++;
> > +
> >     flowclen = sizeof(*flowc) + nparams * sizeof(struct
> > fw_flowc_mnemval); 
> >     wr = alloc_wrqe(roundup2(flowclen, 16), toep->ofld_txq);
> > @@ -131,39 +137,45 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
> >     flowc->flowid_len16 = htonl(V_FW_WR_LEN16(howmany(flowclen, 16)) |
> >         V_FW_WR_FLOWID(toep->tid));
> >  
> > -   flowc->mnemval[0].mnemonic = FW_FLOWC_MNEM_PFNVFN;
> > -   flowc->mnemval[0].val = htobe32(pfvf);
> > -   flowc->mnemval[1].mnemonic = FW_FLOWC_MNEM_CH;
> > -   flowc->mnemval[1].val = htobe32(pi->tx_chan);
> > -   flowc->mnemval[2].mnemonic = FW_FLOWC_MNEM_PORT;
> > -   flowc->mnemval[2].val = htobe32(pi->tx_chan);
> > -   flowc->mnemval[3].mnemonic = FW_FLOWC_MNEM_IQID;
> > -   flowc->mnemval[3].val = htobe32(toep->ofld_rxq->iq.abs_id);
> > +#define FLOWC_PARAM(__m, __v) \
> > +   do { \
> > +           flowc->mnemval[paramidx].mnemonic = FW_FLOWC_MNEM_##__m; \
> > +           flowc->mnemval[paramidx].val = htobe32(__v); \
> > +           paramidx++; \
> > +   } while (0)
> > +
> > +   paramidx = 0;
> > +
> > +   FLOWC_PARAM(PFNVFN, pfvf);
> > +   FLOWC_PARAM(CH, pi->tx_chan);
> > +   FLOWC_PARAM(PORT, pi->tx_chan);
> > +   FLOWC_PARAM(IQID, toep->ofld_rxq->iq.abs_id);
> >     if (ftxp) {
> >             uint32_t sndbuf = min(ftxp->snd_space, sc->tt.sndbuf);
> >  
> > -           flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDNXT;
> > -           flowc->mnemval[4].val = htobe32(ftxp->snd_nxt);
> > -           flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_RCVNXT;
> > -           flowc->mnemval[5].val = htobe32(ftxp->rcv_nxt);
> > -           flowc->mnemval[6].mnemonic = FW_FLOWC_MNEM_SNDBUF;
> > -           flowc->mnemval[6].val = htobe32(sndbuf);
> > -           flowc->mnemval[7].mnemonic = FW_FLOWC_MNEM_MSS;
> > -           flowc->mnemval[7].val = htobe32(ftxp->mss);
> > +           FLOWC_PARAM(SNDNXT, ftxp->snd_nxt);
> > +           FLOWC_PARAM(RCVNXT, ftxp->rcv_nxt);
> > +           FLOWC_PARAM(SNDBUF, sndbuf);
> > +           FLOWC_PARAM(MSS, ftxp->mss);
> >  
> >             CTR6(KTR_CXGBE,
> >                 "%s: tid %u, mss %u, sndbuf %u, snd_nxt 0x%x, rcv_nxt
> > 0x%x", __func__, toep->tid, ftxp->mss, sndbuf, ftxp->snd_nxt,
> >                 ftxp->rcv_nxt);
> >     } else {
> > -           flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDBUF;
> > -           flowc->mnemval[4].val = htobe32(512);
> > -           flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_MSS;
> > -           flowc->mnemval[5].val = htobe32(512);
> > +           FLOWC_PARAM(SNDBUF, 512);
> > +           FLOWC_PARAM(MSS, 512);
> >  
> >             CTR2(KTR_CXGBE, "%s: tid %u", __func__, toep->tid);
> >     }
> > +   if (toep->ulp_mode == ULP_MODE_TLS)
> > +           FLOWC_PARAM(ULP_MODE, toep->ulp_mode);
> > +   if (toep->tls.fcplenmax != 0)
> > +           FLOWC_PARAM(TXDATAPLEN_MAX, toep->tls.fcplenmax);
> > +#undef FLOWC_PARAM
> >  
> > +   KASSERT(paramidx == nparams, ("nparams mismatch"));
> > +
> >     txsd->tx_credits = howmany(flowclen, 16);
> >     txsd->plen = 0;
> >     KASSERT(toep->tx_credits >= txsd->tx_credits && toep->txsd_avail >
> > 0, @@ -421,7 +433,7 @@ make_established(struct toepcb *toep, uint32_t
> > snd_isn soisconnected(so);
> >  }
> >  
> > -static int
> > +int
> >  send_rx_credits(struct adapter *sc, struct toepcb *toep, int credits)
> >  {
> >     struct wrqe *wr;
> > @@ -443,6 +455,23 @@ send_rx_credits(struct adapter *sc, struct toepcb *toe
> >  }
> >  
> >  void
> > +send_rx_modulate(struct adapter *sc, struct toepcb *toep)
> > +{
> > +   struct wrqe *wr;
> > +   struct cpl_rx_data_ack *req;
> > +
> > +   wr = alloc_wrqe(sizeof(*req), toep->ctrlq);
> > +   if (wr == NULL)
> > +           return;
> > +   req = wrtod(wr);
> > +
> > +   INIT_TP_WR_MIT_CPL(req, CPL_RX_DATA_ACK, toep->tid);
> > +   req->credit_dack = htobe32(F_RX_MODULATE_RX);
> > +
> > +   t4_wrq_tx(sc, wr);
> > +}
> > +
> > +void
> >  t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
> >  {
> >     struct adapter *sc = tod->tod_softc;
> > @@ -459,8 +488,18 @@ t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
> >         ("%s: sb %p has more data (%d) than last time (%d).",
> >         __func__, sb, sbused(sb), toep->sb_cc));
> >  
> > -   toep->rx_credits += toep->sb_cc - sbused(sb);
> > +   credits = toep->sb_cc - sbused(sb);
> >     toep->sb_cc = sbused(sb);
> > +   if (toep->ulp_mode == ULP_MODE_TLS) {
> > +           if (toep->tls.rcv_over >= credits) {
> > +                   toep->tls.rcv_over -= credits;
> > +                   credits = 0;
> > +           } else {
> > +                   credits -= toep->tls.rcv_over;
> > +                   toep->tls.rcv_over = 0;
> > +           }
> > +   }
> > +   toep->rx_credits += credits;
> >  
> >     if (toep->rx_credits > 0 &&
> >         (tp->rcv_wnd <= 32 * 1024 || toep->rx_credits >= 64 * 1024 ||
> > @@ -471,7 +510,8 @@ t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
> >             toep->rx_credits -= credits;
> >             tp->rcv_wnd += credits;
> >             tp->rcv_adv += credits;
> > -   }
> > +   } else if (toep->flags & TPF_FORCE_CREDITS)
> > +           send_rx_modulate(sc, toep);
> >  }
> >  
> >  void
> > @@ -489,8 +529,8 @@ t4_rcvd(struct toedev *tod, struct tcpcb *tp)
> >  /*
> >   * Close a connection by sending a CPL_CLOSE_CON_REQ message.
> >   */
> > -static int
> > -close_conn(struct adapter *sc, struct toepcb *toep)
> > +int
> > +t4_close_conn(struct adapter *sc, struct toepcb *toep)
> >  {
> >     struct wrqe *wr;
> >     struct cpl_close_con_req *req;
> > @@ -691,6 +731,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep
> >  
> >     KASSERT(toep->ulp_mode == ULP_MODE_NONE ||
> >         toep->ulp_mode == ULP_MODE_TCPDDP ||
> > +       toep->ulp_mode == ULP_MODE_TLS ||
> >         toep->ulp_mode == ULP_MODE_RDMA,
> >         ("%s: ulp_mode %u for toep %p", __func__, toep->ulp_mode,
> > toep)); 
> > @@ -905,7 +946,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep
> >  
> >     /* Send a FIN if requested, but only if there's no more data to
> > send */ if (m == NULL && toep->flags & TPF_SEND_FIN)
> > -           close_conn(sc, toep);
> > +           t4_close_conn(sc, toep);
> >  }
> >  
> >  static inline void
> > @@ -1097,7 +1138,7 @@ t4_push_pdus(struct adapter *sc, struct toepcb *toep, 
> >  
> >     /* Send a FIN if requested, but only if there are no more PDUs to
> > send */ if (mbufq_first(pduq) == NULL && toep->flags & TPF_SEND_FIN)
> > -           close_conn(sc, toep);
> > +           t4_close_conn(sc, toep);
> >  }
> >  
> >  int
> > @@ -1116,6 +1157,8 @@ t4_tod_output(struct toedev *tod, struct tcpcb *tp)
> >  
> >     if (toep->ulp_mode == ULP_MODE_ISCSI)
> >             t4_push_pdus(sc, toep, 0);
> > +   else if (tls_tx_key(toep))
> > +           t4_push_tls_records(sc, toep, 0);
> >     else
> >             t4_push_frames(sc, toep, 0);
> >  
> > @@ -1140,6 +1183,8 @@ t4_send_fin(struct toedev *tod, struct tcpcb *tp)
> >     if (tp->t_state >= TCPS_ESTABLISHED) {
> >             if (toep->ulp_mode == ULP_MODE_ISCSI)
> >                     t4_push_pdus(sc, toep, 0);
> > +           else if (tls_tx_key(toep))
> > +                   t4_push_tls_records(sc, toep, 0);
> >             else
> >                     t4_push_frames(sc, toep, 0);
> >     }
> > @@ -1772,6 +1817,10 @@ do_fw4_ack(struct sge_iq *iq, const struct
> > rss_header credits -= txsd->tx_credits;
> >             toep->tx_credits += txsd->tx_credits;
> >             plen += txsd->plen;
> > +           if (txsd->iv_buffer) {
> > +                   free(txsd->iv_buffer, M_CXGBE);
> > +                   txsd->iv_buffer = NULL;
> > +           }
> >             txsd++;
> >             toep->txsd_avail++;
> >             KASSERT(toep->txsd_avail <= toep->txsd_total,
> > @@ -1797,6 +1846,8 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header 
> >             CURVNET_SET(toep->vnet);
> >             if (toep->ulp_mode == ULP_MODE_ISCSI)
> >                     t4_push_pdus(sc, toep, plen);
> > +           else if (tls_tx_key(toep))
> > +                   t4_push_tls_records(sc, toep, plen);
> >             else
> >                     t4_push_frames(sc, toep, plen);
> >             CURVNET_RESTORE();
> > @@ -1826,6 +1877,12 @@ do_fw4_ack(struct sge_iq *iq, const struct
> > rss_header tid, plen);
> >  #endif
> >                     sbdrop_locked(sb, plen);
> > +                   if (tls_tx_key(toep)) {
> > +                           struct tls_ofld_info *tls_ofld =
> > &toep->tls; +
> > +                           MPASS(tls_ofld->sb_off >= plen);
> > +                           tls_ofld->sb_off -= plen;
> > +                   }
> >                     if (!TAILQ_EMPTY(&toep->aiotx_jobq))
> >                             t4_aiotx_queue_toep(toep);
> >                     sowwakeup_locked(so);   /* unlocks so_snd */
> > @@ -2298,6 +2355,9 @@ t4_aio_queue_aiotx(struct socket *so, struct kaiocb *j
> >             return (EOPNOTSUPP);
> >  
> >     if (!sc->tt.tx_zcopy)
> > +           return (EOPNOTSUPP);
> > +
> > +   if (is_tls_offload(toep) || tls_tx_key(toep))
> >             return (EOPNOTSUPP);
> >  
> >     SOCKBUF_LOCK(&so->so_snd);
> > 
> > Modified: head/sys/dev/cxgbe/tom/t4_listen.c
> > ==============================================================================
> > --- head/sys/dev/cxgbe/tom/t4_listen.c      Tue Mar 13 22:54:29 2018
> > (r330883) +++ head/sys/dev/cxgbe/tom/t4_listen.c    Tue Mar 13 23:05:51
> > 2018        (r330884) @@ -1056,6 +1056,11 @@ calc_opt2p(struct adapter *sc,
> > struct port_info *pi, i if (ulp_mode == ULP_MODE_TCPDDP)
> >             opt2 |= F_RX_FC_VALID | F_RX_FC_DDP;
> >  #endif
> > +   if (ulp_mode == ULP_MODE_TLS) {
> > +           opt2 |= F_RX_FC_VALID;
> > +           opt2 &= ~V_RX_COALESCE(M_RX_COALESCE);
> > +           opt2 |= F_RX_FC_DISABLE;
> > +   }
> >  
> >     return htobe32(opt2);
> >  }
> > @@ -1347,11 +1352,15 @@ found:
> >  
> >             INIT_TP_WR_MIT_CPL(rpl5, CPL_PASS_ACCEPT_RPL, tid);
> >     }
> > -   if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0) {
> > -           ulp_mode = ULP_MODE_TCPDDP;
> > +   ulp_mode = select_ulp_mode(so, sc);
> > +   switch (ulp_mode) {
> > +   case ULP_MODE_TCPDDP:
> >             synqe->flags |= TPF_SYNQE_TCPDDP;
> > -   } else
> > -           ulp_mode = ULP_MODE_NONE;
> > +           break;
> > +   case ULP_MODE_TLS:
> > +           synqe->flags |= TPF_SYNQE_TLS;
> > +           break;
> > +   }
> >     rpl->opt0 = calc_opt0(so, vi, e, mtu_idx, rscale, rx_credits,
> > ulp_mode); rpl->opt2 = calc_opt2p(sc, pi, rxqid, &cpl->tcpopt, &th,
> > ulp_mode); 
> > @@ -1407,8 +1416,8 @@ found:
> >             REJECT_PASS_ACCEPT();
> >     }
> >  
> > -   CTR5(KTR_CXGBE, "%s: stid %u, tid %u, lctx %p, synqe %p, SYNACK",
> > -       __func__, stid, tid, lctx, synqe);
> > +   CTR6(KTR_CXGBE, "%s: stid %u, tid %u, lctx %p, synqe %p, SYNACK
> > mode %d",
> > +       __func__, stid, tid, lctx, synqe, ulp_mode);
> >  
> >     INP_WLOCK(inp);
> >     synqe->flags |= TPF_SYNQE_HAS_L2TE;
> > @@ -1557,9 +1566,11 @@ reset:
> >     toep->tid = tid;
> >     toep->l2te = &sc->l2t->l2tab[synqe->l2e_idx];
> >     if (synqe->flags & TPF_SYNQE_TCPDDP)
> > -           set_tcpddp_ulp_mode(toep);
> > +           set_ulp_mode(toep, ULP_MODE_TCPDDP);
> > +   else if (synqe->flags & TPF_SYNQE_TLS)
> > +           set_ulp_mode(toep, ULP_MODE_TLS);
> >     else
> > -           toep->ulp_mode = ULP_MODE_NONE;
> > +           set_ulp_mode(toep, ULP_MODE_NONE);
> >     /* opt0 rcv_bufsiz initially, assumes its normal meaning later */
> >     toep->rx_credits = synqe->rcv_bufsize;
> >  
> > 
> > Added: head/sys/dev/cxgbe/tom/t4_tls.c
> > ==============================================================================
> > --- /dev/null       00:00:00 1970   (empty, because file is newly
> > added) +++ head/sys/dev/cxgbe/tom/t4_tls.c  Tue Mar 13 23:05:51 2018
> > (r330884) @@ -0,0 +1,1642 @@
> > +/*-
> > + * SPDX-License-Identifier: BSD-2-Clause-FreeBSD
> > + *
> > + * Copyright (c) 2017-2018 Chelsio Communications, Inc.
> > + * All rights reserved.
> > + * Written by: John Baldwin <j...@freebsd.org>
> > + *
> > + * Redistribution and use in source and binary forms, with or without
> > + * modification, are permitted provided that the following conditions
> > + * are met:
> > + * 1. Redistributions of source code must retain the above copyright
> > + *    notice, this list of conditions and the following disclaimer.
> > + * 2. Redistributions in binary form must reproduce the above copyright
> > + *    notice, this list of conditions and the following disclaimer in the
> > + *    documentation and/or other materials provided with the distribution.
> > + *
> > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
> > PURPOSE
> > + * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
> > CONSEQUENTIAL
> > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
> > STRICT
> > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY
> > WAY
> > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> > + * SUCH DAMAGE.
> > + */
> > +
> > +#include "opt_inet.h"
> > +
> > +#include <sys/cdefs.h>
> > +__FBSDID("$FreeBSD$");
> > +
> > +#include <sys/param.h>
> > +#include <sys/sglist.h>
> > +#include <sys/socket.h>
> > +#include <sys/socketvar.h>
> > +#include <sys/systm.h>
> > +#include <netinet/in.h>
> > +#include <netinet/in_pcb.h>
> > +#include <netinet/tcp_var.h>
> > +#include <netinet/toecore.h>
> > +
> > +#ifdef TCP_OFFLOAD
> > +#include "common/common.h"
> > +#include "common/t4_tcb.h"
> > +#include "tom/t4_tom_l2t.h"
> > +#include "tom/t4_tom.h"
> > +
> > +/*
> > + * The TCP sequence number of a CPL_TLS_DATA mbuf is saved here while
> > + * the mbuf is in the ulp_pdu_reclaimq.
> > + */
> > +#define    tls_tcp_seq     PH_loc.thirtytwo[0]
> > +
> > +/*
> > + * Handshake lock used for the handshake timer.  Having a global lock
> > + * is perhaps not ideal, but it avoids having to use callout_drain()
> > + * in tls_uninit_toep() which can't block.  Also, the timer shouldn't
> > + * actually fire for most connections.
> > + */
> > +static struct mtx tls_handshake_lock;
> > +
> > +static void
> > +t4_set_tls_tcb_field(struct toepcb *toep, uint16_t word, uint64_t mask,
> > +    uint64_t val)
> > +{
> > +   struct adapter *sc = td_adapter(toep->td);
> > +
> > +   t4_set_tcb_field(sc, toep->ctrlq, toep->tid, word, mask, val, 0, 0,
> > +       toep->ofld_rxq->iq.abs_id);
> > +}
> > +
> > +/* TLS and DTLS common routines */
> > +int
> > +tls_tx_key(struct toepcb *toep)
> > +{
> > +   struct tls_ofld_info *tls_ofld = &toep->tls;
> > +
> > +   return (tls_ofld->tx_key_addr >= 0);
> > +}
> > +
> > +int
> > +tls_rx_key(struct toepcb *toep)
> > +{
> > +   struct tls_ofld_info *tls_ofld = &toep->tls;
> > +
> > +   return (tls_ofld->rx_key_addr >= 0);
> > +}
> > +
> > +static int
> > +key_size(struct toepcb *toep)
> > +{
> > +   struct tls_ofld_info *tls_ofld = &toep->tls;
> > +
> > +   return ((tls_ofld->key_location ==
> > TLS_SFO_WR_CONTEXTLOC_IMMEDIATE) ?
> > +           tls_ofld->k_ctx.tx_key_info_size : KEY_IN_DDR_SIZE);
> > +}
> > +
> > +/* Set TLS Key-Id in TCB */
> > +static void
> > +t4_set_tls_keyid(struct toepcb *toep, unsigned int key_id)
> > +{
> > +
> > +   t4_set_tls_tcb_field(toep, W_TCB_RX_TLS_KEY_TAG,
> > +                    V_TCB_RX_TLS_KEY_TAG(M_TCB_RX_TLS_BUF_TAG),
> > +                    V_TCB_RX_TLS_KEY_TAG(key_id));
> > +}
> > +
> > +/* Clear TF_RX_QUIESCE to re-enable receive. */
> > +static void
> > +t4_clear_rx_quiesce(struct toepcb *toep)
> > +{
> > +
> > +   t4_set_tls_tcb_field(toep, W_TCB_T_FLAGS, V_TF_RX_QUIESCE(1), 0);
> > +}
> > +
> > +static void
> > +tls_clr_ofld_mode(struct toepcb *toep)
> > +{
> > +
> > +   tls_stop_handshake_timer(toep);
> > +
> > +   /* Operate in PDU extraction mode only. */
> > +   t4_set_tls_tcb_field(toep, W_TCB_ULP_RAW,
> > +       V_TCB_ULP_RAW(M_TCB_ULP_RAW),
> > +       V_TCB_ULP_RAW(V_TF_TLS_ENABLE(1)));
> > +   t4_clear_rx_quiesce(toep);
> > +}
> > +
> > +static void
> > +tls_clr_quiesce(struct toepcb *toep)
> > +{
> > +
> > +   tls_stop_handshake_timer(toep);
> > +   t4_clear_rx_quiesce(toep);
> > +}
> > +
> > +/*
> > + * Calculate the TLS data expansion size
> > + */
> > +static int
> > +tls_expansion_size(struct toepcb *toep, int data_len, int full_pdus_only,
> > +    unsigned short *pdus_per_ulp)
> > +{
> > +   struct tls_ofld_info *tls_ofld = &toep->tls;
> > +   struct tls_scmd *scmd = &tls_ofld->scmd0;
> > +   int expn_size = 0, frag_count = 0, pad_per_pdu = 0,
> > +       pad_last_pdu = 0, last_frag_size = 0, max_frag_size = 0;
> > +   int exp_per_pdu = 0;
> > +   int hdr_len = TLS_HEADER_LENGTH;
> > +
> > +   do {
> > +           max_frag_size = tls_ofld->k_ctx.frag_size;
> > +           if (G_SCMD_CIPH_MODE(scmd->seqno_numivs) ==
> > +              SCMD_CIPH_MODE_AES_GCM) {
> > +                   frag_count = (data_len / max_frag_size);
> > +                   exp_per_pdu = GCM_TAG_SIZE +
> > AEAD_EXPLICIT_DATA_SIZE +
> > +                           hdr_len;
> > +                   expn_size =  frag_count * exp_per_pdu;
> > +                   if (full_pdus_only) {
> > +                           *pdus_per_ulp = data_len / (exp_per_pdu +
> > +                                   max_frag_size);
> > +                           if (*pdus_per_ulp > 32)
> > +                                   *pdus_per_ulp = 32;
> > +                           else if(!*pdus_per_ulp)
> > +                                   *pdus_per_ulp = 1;
> > +                           expn_size = (*pdus_per_ulp) * exp_per_pdu;
> > +                           break;
> > +                   }
> > +                   if ((last_frag_size = data_len % max_frag_size) >
> > 0) {
> > +                           frag_count += 1;
> > +                           expn_size += exp_per_pdu;
> > +                   }
> > +                   break;
> > +           } else if (G_SCMD_CIPH_MODE(scmd->seqno_numivs) !=
> > +                      SCMD_CIPH_MODE_NOP) {
> > +                   /* Calculate the number of fragments we can make */
> > +                   frag_count  = (data_len / max_frag_size);
> > +                   if (frag_count > 0) {
> > +                           pad_per_pdu = (((howmany((max_frag_size +
> > +
> > tls_ofld->mac_length),
> > +                                                 CIPHER_BLOCK_SIZE)) *
> > +                                           CIPHER_BLOCK_SIZE) -
> > +                                          (max_frag_size +
> > +                                           tls_ofld->mac_length));
> > +                           if (!pad_per_pdu)
> > +                                   pad_per_pdu = CIPHER_BLOCK_SIZE;
> > +                           exp_per_pdu = pad_per_pdu +
> > +                                   tls_ofld->mac_length +
> > +                                   hdr_len + CIPHER_BLOCK_SIZE;
> > +                           expn_size = frag_count * exp_per_pdu;
> > +                   }
> > +                   if (full_pdus_only) {
> > +                           *pdus_per_ulp = data_len / (exp_per_pdu +
> > +                                   max_frag_size);
> > +                           if (*pdus_per_ulp > 32)
> > +                                   *pdus_per_ulp = 32;
> > +                           else if (!*pdus_per_ulp)
> > +                                   *pdus_per_ulp = 1;
> > +                           expn_size = (*pdus_per_ulp) * exp_per_pdu;
> > +                           break;
> > +                   }
> > +                   /* Consider the last fragment */
> > +                   if ((last_frag_size = data_len % max_frag_size) >
> > 0) {
> > +                           pad_last_pdu = (((howmany((last_frag_size +
> > +
> > tls_ofld->mac_length),
> > +                                                  CIPHER_BLOCK_SIZE))
> > *
> > +                                            CIPHER_BLOCK_SIZE) -
> > +                                           (last_frag_size +
> > +                                            tls_ofld->mac_length));
> > +                           if (!pad_last_pdu)
> > +                                   pad_last_pdu = CIPHER_BLOCK_SIZE;
> > +                           expn_size += (pad_last_pdu +
> > +                                         tls_ofld->mac_length +
> > hdr_len +
> > +                                         CIPHER_BLOCK_SIZE);
> > +                   }
> > +           }
> > +   } while (0);
> > +
> > +   return (expn_size);
> > +}
> > +
> > +/* Copy Key to WR */
> > +static void
> > +tls_copy_tx_key(struct toepcb *toep, void *dst)
> > +{
> > +   struct tls_ofld_info *tls_ofld = &toep->tls;
> > +   struct ulptx_sc_memrd *sc_memrd;
> > +   struct ulptx_idata *sc;
> > +
> > +   if (tls_ofld->k_ctx.tx_key_info_size <= 0)
> > +           return;
> > +
> > +   if (tls_ofld->key_location == TLS_SFO_WR_CONTEXTLOC_DDR) {
> > +           sc = dst;
> > +           sc->cmd_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_NOOP));
> > +           sc->len = htobe32(0);
> > +           sc_memrd = (struct ulptx_sc_memrd *)(sc + 1);
> > +           sc_memrd->cmd_to_len =
> > htobe32(V_ULPTX_CMD(ULP_TX_SC_MEMRD) |
> > +               V_ULP_TX_SC_MORE(1) |
> > +               V_ULPTX_LEN16(tls_ofld->k_ctx.tx_key_info_size >> 4));
> > +           sc_memrd->addr = htobe32(tls_ofld->tx_key_addr >> 5);
> > +   } else if (tls_ofld->key_location ==
> > TLS_SFO_WR_CONTEXTLOC_IMMEDIATE) {
> > +           memcpy(dst, &tls_ofld->k_ctx.tx,
> > +               tls_ofld->k_ctx.tx_key_info_size);
> > +   }
> > +}
> > +
> > +/* TLS/DTLS content type  for CPL SFO */
> > +static inline unsigned char
> > +tls_content_type(unsigned char content_type)
> > +{
> > +   /*
> > +    * XXX: Shouldn't this map CONTENT_TYPE_APP_DATA to DATA and
> > +    * default to "CUSTOM" for all other types including
> > +    * heartbeat?
> > +    */
> > +   switch (content_type) {
> > +   case CONTENT_TYPE_CCS:
> > +           return CPL_TX_TLS_SFO_TYPE_CCS;
> > +   case CONTENT_TYPE_ALERT:
> > +           return CPL_TX_TLS_SFO_TYPE_ALERT;
> > +   case CONTENT_TYPE_HANDSHAKE:
> > +           return CPL_TX_TLS_SFO_TYPE_HANDSHAKE;
> > +   case CONTENT_TYPE_HEARTBEAT:
> > +           return CPL_TX_TLS_SFO_TYPE_HEARTBEAT;
> > +   }
> > +   return CPL_TX_TLS_SFO_TYPE_DATA;
> > +}
> > +
> > +static unsigned char
> > +get_cipher_key_size(unsigned int ck_size)
> > +{
> > +   switch (ck_size) {
> > +   case AES_NOP: /* NOP */
> > +           return 15;
> > +   case AES_128: /* AES128 */
> > +           return CH_CK_SIZE_128;
> > +   case AES_192: /* AES192 */
> > +           return CH_CK_SIZE_192;
> > +   case AES_256: /* AES256 */
> > +           return CH_CK_SIZE_256;
> > +   default:
> > +           return CH_CK_SIZE_256;
> > +   }
> > +}
> > +
> > +static unsigned char
> > +get_mac_key_size(unsigned int mk_size)
> > +{
> > +   switch (mk_size) {
> > +   case SHA_NOP: /* NOP */
> > +           return CH_MK_SIZE_128;
> > +   case SHA_GHASH: /* GHASH */
> > +   case SHA_512: /* SHA512 */
> > +           return CH_MK_SIZE_512;
> > +   case SHA_224: /* SHA2-224 */
> > +           return CH_MK_SIZE_192;
> > +   case SHA_256: /* SHA2-256*/
> > +           return CH_MK_SIZE_256;
> > +   case SHA_384: /* SHA384 */
> > +           return CH_MK_SIZE_512;
> > +   case SHA1: /* SHA1 */
> > +   default:
> > +           return CH_MK_SIZE_160;
> > +   }
> > +}
> > +
> > +static unsigned int
> > +get_proto_ver(int proto_ver)
> > +{
> > +   switch (proto_ver) {
> > +   case TLS1_2_VERSION:
> > +           return TLS_1_2_VERSION;
> > +   case TLS1_1_VERSION:
> > +           return TLS_1_1_VERSION;
> > +   case DTLS1_2_VERSION:
> > +           return DTLS_1_2_VERSION;
> > +   default:
> > +           return TLS_VERSION_MAX;
> > +   }
> > +}
> > +
> > +static void
> > +tls_rxkey_flit1(struct tls_keyctx *kwr, struct tls_key_context *kctx)
> > +{
> > +
> > +   if (kctx->state.enc_mode == CH_EVP_CIPH_GCM_MODE) {
> > +           kwr->u.rxhdr.ivinsert_to_authinsrt =
> > +               htobe64(V_TLS_KEYCTX_TX_WR_IVINSERT(6ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_AADSTRTOFST(1ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_AADSTOPOFST(5ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_AUTHSRTOFST(14ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_AUTHSTOPOFST(16ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_CIPHERSRTOFST(14ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST(0ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_AUTHINSRT(16ULL));
> > +           kwr->u.rxhdr.ivpresent_to_rxmk_size &=
> > +                   ~(V_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT(1));
> > +           kwr->u.rxhdr.authmode_to_rxvalid &=
> > +                   ~(V_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL(1));
> > +   } else {
> > +           kwr->u.rxhdr.ivinsert_to_authinsrt =
> > +               htobe64(V_TLS_KEYCTX_TX_WR_IVINSERT(6ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_AADSTRTOFST(1ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_AADSTOPOFST(5ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_AUTHSRTOFST(22ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_AUTHSTOPOFST(0ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_CIPHERSRTOFST(22ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_CIPHERSTOPOFST(0ULL) |
> > +                   V_TLS_KEYCTX_TX_WR_AUTHINSRT(0ULL));
> > +   }
> > +}
> > +
> > +/* Rx key */
> > +static void
> > +prepare_rxkey_wr(struct tls_keyctx *kwr, struct tls_key_context *kctx)
> > +{
> > +   unsigned int ck_size = kctx->cipher_secret_size;
> > +   unsigned int mk_size = kctx->mac_secret_size;
> > +   int proto_ver = kctx->proto_ver;
> > +
> > +   kwr->u.rxhdr.flitcnt_hmacctrl =
> > +           ((kctx->tx_key_info_size >> 4) << 3) | kctx->hmac_ctrl;
> > +
> > +   kwr->u.rxhdr.protover_ciphmode =
> > +           V_TLS_KEYCTX_TX_WR_PROTOVER(get_proto_ver(proto_ver)) |
> > +           V_TLS_KEYCTX_TX_WR_CIPHMODE(kctx->state.enc_mode);
> > +
> > +   kwr->u.rxhdr.authmode_to_rxvalid =
> > +           V_TLS_KEYCTX_TX_WR_AUTHMODE(kctx->state.auth_mode) |
> > +           V_TLS_KEYCTX_TX_WR_CIPHAUTHSEQCTRL(1) |
> > +           V_TLS_KEYCTX_TX_WR_SEQNUMCTRL(3) |
> > +           V_TLS_KEYCTX_TX_WR_RXVALID(1);
> > +
> > +   kwr->u.rxhdr.ivpresent_to_rxmk_size =
> > +           V_TLS_KEYCTX_TX_WR_IVPRESENT(0) |
> > +           V_TLS_KEYCTX_TX_WR_RXOPAD_PRESENT(1) |
> > +           V_TLS_KEYCTX_TX_WR_RXCK_SIZE(get_cipher_key_size(ck_size))
> > |
> > +           V_TLS_KEYCTX_TX_WR_RXMK_SIZE(get_mac_key_size(mk_size));
> > +
> > +   tls_rxkey_flit1(kwr, kctx);
> > +
> > +   /* No key reversal for GCM */
> > +   if (kctx->state.enc_mode != CH_EVP_CIPH_GCM_MODE) {
> > +           t4_aes_getdeckey(kwr->keys.edkey, kctx->rx.key,
> > +                            (kctx->cipher_secret_size << 3));
> > +           memcpy(kwr->keys.edkey + kctx->cipher_secret_size,
> > +                  kctx->rx.key + kctx->cipher_secret_size,
> > +                  (IPAD_SIZE + OPAD_SIZE));
> > +   } else {
> > +           memcpy(kwr->keys.edkey, kctx->rx.key,
> > +                  (kctx->tx_key_info_size - SALT_SIZE));
> > +           memcpy(kwr->u.rxhdr.rxsalt, kctx->rx.salt, SALT_SIZE);
> > +   }
> > +}
> > +
> > +/* Tx key */
> > +static void
> > +prepare_txkey_wr(struct tls_keyctx *kwr, struct tls_key_context *kctx)
> > +{
> > +   unsigned int ck_size = kctx->cipher_secret_size;
> > +   unsigned int mk_size = kctx->mac_secret_size;
> > +
> > +   kwr->u.txhdr.ctxlen =
> > +           (kctx->tx_key_info_size >> 4);
> > +   kwr->u.txhdr.dualck_to_txvalid =
> > +           V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(1) |
> > +           V_TLS_KEYCTX_TX_WR_SALT_PRESENT(1) |
> > +           V_TLS_KEYCTX_TX_WR_TXCK_SIZE(get_cipher_key_size(ck_size))
> > |
> > +           V_TLS_KEYCTX_TX_WR_TXMK_SIZE(get_mac_key_size(mk_size)) |
> > +           V_TLS_KEYCTX_TX_WR_TXVALID(1);
> > +
> > +   memcpy(kwr->keys.edkey, kctx->tx.key, HDR_KCTX_SIZE);
> > +   if (kctx->state.enc_mode == CH_EVP_CIPH_GCM_MODE) {
> > +           memcpy(kwr->u.txhdr.txsalt, kctx->tx.salt, SALT_SIZE);
> > +           kwr->u.txhdr.dualck_to_txvalid &=
> > +                   ~(V_TLS_KEYCTX_TX_WR_TXOPAD_PRESENT(1));
> > +   }
> > +   kwr->u.txhdr.dualck_to_txvalid =
> > htons(kwr->u.txhdr.dualck_to_txvalid); +}
> > +
> > +/* TLS Key memory management */
> > +int
> > +tls_init_kmap(struct adapter *sc, struct tom_data *td)
> > +{
> > +
> > +   td->key_map = vmem_create("T4TLS key map", sc->vres.key.start,
> > +       sc->vres.key.size, 8, 0, M_FIRSTFIT | M_NOWAIT);
> > +   if (td->key_map == NULL)
> > +           return (ENOMEM);
> > +   return (0);
> > +}
> > +
> > +void
> > +tls_free_kmap(struct tom_data *td)
> > +{
> > +
> > +   if (td->key_map != NULL)
> > +           vmem_destroy(td->key_map);
> > +}
> > +
> > +static int
> > +get_new_keyid(struct toepcb *toep, struct tls_key_context *k_ctx)
> > +{
> > +   struct tom_data *td = toep->td;
> > +   vmem_addr_t addr;
> > +
> > +   if (vmem_alloc(td->key_map, TLS_KEY_CONTEXT_SZ, M_NOWAIT |
> > M_FIRSTFIT,
> > +       &addr) != 0)
> > +           return (-1);
> > +
> > +   return (addr);
> > +}
> > +
> > +static void
> > +free_keyid(struct toepcb *toep, int keyid)
> > +{
> > +   struct tom_data *td = toep->td;
> > +
> > +   vmem_free(td->key_map, keyid, TLS_KEY_CONTEXT_SZ);
> > +}
> > +
> > 
> > *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
> > _______________________________________________
> > svn-src-h...@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/svn-src-head
> > To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"  
> 
> Building a kernel fails on CURRENT  revision 330908 with the follwoing error:
> 
> [...]
> ===> gpio/gpioiic (all)  
> --- all_subdir_cxgbe ---
> --- cm.o ---
> In file included from /usr/src/sys/dev/cxgbe/iw_cxgbe/cm.c:68:
> /usr/src/sys/dev/cxgbe/tom/t4_tom.h:327:28: error: use of undeclared
> identifier 'ULP_MODE_TLS' return (toep->ulp_mode == ULP_MODE_TLS);
>                                   ^
> /usr/src/sys/dev/cxgbe/tom/t4_tom.h:333:12: error: incomplete definition of
> type 'struct adapter' return (sc->tt.tls && sc->cryptocaps &
> FW_CAPS_CONFIG_TLSKEYS); ~~^
> /usr/src/sys/dev/cxgbe/offload.h:139:8: note: forward declaration of 'struct
> adapter' struct adapter;
> 
> 
> 
> Kind regards,
> 
> oh


problem occurs when

WITH_OFED=YES

is set in /etc/src.conf.
_______________________________________________
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Reply via email to