Re: [RFC PATCH 0/2] net:setup XPS mapping for each online CPU

[Paolo Abeni]

Hi,

On Thu, 2018-03-15 at 15:59 +, Eric Dumazet wrote:
> On Thu, Mar 15, 2018 at 8:51 AM Paolo Abeni  wrote:
> > I'm sorry, I do not follow. AFAICS with unconnected sockets without XPS
> > we always hit the netdev_pick_tx()/skb_tx_hash()/skb_flow_dissect()
> > overhead in xmit path.
> 
> Then fix this if you want, instead of fixing one NIC only, or by enforcing
> XPS by all NIC.
> 
> For unconnected sockets, picking the TX queue based on current cpu is good,
> we do not have to enforce ordering as much as possible.
> 
> (pfifo_fast no longer can enforce it anyway)

Thank you for the prompt reply. 

I'm double checking to avoid misinterpretation on my side: are you
suggesting to plug a CPU-based selection logic for unconnected sockets
in netdev_pick_tx() or to cook patches like 2/2 for all the relevant
NICs? 

Thanks!

Paolo

Re: [RFC PATCH 0/2] net:setup XPS mapping for each online CPU

[Eric Dumazet]

On Thu, Mar 15, 2018 at 8:51 AM Paolo Abeni  wrote:

> Hi,

> On Thu, 2018-03-15 at 15:31 +, Eric Dumazet wrote:
> > On Thu, Mar 15, 2018 at 8:08 AM Paolo Abeni  wrote:
> >
> > > Currently, most MQ netdevice setup the default XPS configuration
mapping
> >
> > 1-1
> > > the first real_num_tx_queues queues and CPUs and no mapping is
created for
> > > the CPUs with id greater then real_num_tx_queues, if any.
> > > As a result, the xmit path for unconnected sockets on such cores
> >
> > experiences a
> > > relevant overhead in netdev_pick_tx(), which needs to dissect each
packet
> >
> > and
> > > compute its hash.
> > > Such scenario is easily triggered. e.g. from DNS server under relevant
> >
> > load, as
> > > the user-space process is moved away from the CPUs serving the
softirqs
> >
> > (note:
> > > this is beneficial for the overall DNS server performances).
> > > This series introduces an helper to easily setup up XPS mapping for
all
> >
> > the
> > > online CPUs, and use it in the ixgbe driver, demonstrating a relevant
> > > performance improvement in the above scenario.
> > > Paolo Abeni (2):
> > >net: introduce netif_set_xps()
> > >ixgbe: setup XPS via netif_set_xps()
> >
> >
> > Resent, not HTML this time, sorry for duplication.
> >
> > I truly believe XPS should not be setup by devices.
> >
> > XPS is policy, and policy does belong to user space.

> Thank you for your comments!

> As general principle, I agree policies should be in user-space, but I
> also think that the kernel should provide a reasonable default. Many MQ
> devices already configure XPS and their default is AFAICS sub-optimal.

> > Note that if XPS is not setup, MQ queue selection is just fine by
default ;

> I'm sorry, I do not follow. AFAICS with unconnected sockets without XPS
> we always hit the netdev_pick_tx()/skb_tx_hash()/skb_flow_dissect()
> overhead in xmit path.


Then fix this if you want, instead of fixing one NIC only, or by enforcing
XPS by all NIC.

For unconnected sockets, picking the TX queue based on current cpu is good,
we do not have to enforce ordering as much as possible.

(pfifo_fast no longer can enforce it anyway)

Re: [RFC PATCH 0/2] net:setup XPS mapping for each online CPU

[Paolo Abeni]

Hi,

On Thu, 2018-03-15 at 15:31 +, Eric Dumazet wrote:
> On Thu, Mar 15, 2018 at 8:08 AM Paolo Abeni  wrote:
> 
> > Currently, most MQ netdevice setup the default XPS configuration mapping
> 
> 1-1
> > the first real_num_tx_queues queues and CPUs and no mapping is created for
> > the CPUs with id greater then real_num_tx_queues, if any.
> > As a result, the xmit path for unconnected sockets on such cores
> 
> experiences a
> > relevant overhead in netdev_pick_tx(), which needs to dissect each packet
> 
> and
> > compute its hash.
> > Such scenario is easily triggered. e.g. from DNS server under relevant
> 
> load, as
> > the user-space process is moved away from the CPUs serving the softirqs
> 
> (note:
> > this is beneficial for the overall DNS server performances).
> > This series introduces an helper to easily setup up XPS mapping for all
> 
> the
> > online CPUs, and use it in the ixgbe driver, demonstrating a relevant
> > performance improvement in the above scenario.
> > Paolo Abeni (2):
> >net: introduce netif_set_xps()
> >ixgbe: setup XPS via netif_set_xps()
> 
> 
> Resent, not HTML this time, sorry for duplication.
> 
> I truly believe XPS should not be setup by devices.
> 
> XPS is policy, and policy does belong to user space.

Thank you for your comments!

As general principle, I agree policies should be in user-space, but I
also think that the kernel should provide a reasonable default. Many MQ
devices already configure XPS and their default is AFAICS sub-optimal.

> Note that if XPS is not setup, MQ queue selection is just fine by default ;

I'm sorry, I do not follow. AFAICS with unconnected sockets without XPS
we always hit the netdev_pick_tx()/skb_tx_hash()/skb_flow_dissect()
overhead in xmit path.

Cheers,

Paolo

Re: [RFC PATCH 0/2] net:setup XPS mapping for each online CPU

[Eric Dumazet]

On Thu, Mar 15, 2018 at 8:08 AM Paolo Abeni  wrote:

> Currently, most MQ netdevice setup the default XPS configuration mapping
1-1
> the first real_num_tx_queues queues and CPUs and no mapping is created for
> the CPUs with id greater then real_num_tx_queues, if any.

> As a result, the xmit path for unconnected sockets on such cores
experiences a
> relevant overhead in netdev_pick_tx(), which needs to dissect each packet
and
> compute its hash.

> Such scenario is easily triggered. e.g. from DNS server under relevant
load, as
> the user-space process is moved away from the CPUs serving the softirqs
(note:
> this is beneficial for the overall DNS server performances).

> This series introduces an helper to easily setup up XPS mapping for all
the
> online CPUs, and use it in the ixgbe driver, demonstrating a relevant
> performance improvement in the above scenario.

> Paolo Abeni (2):
>net: introduce netif_set_xps()
>ixgbe: setup XPS via netif_set_xps()


Resent, not HTML this time, sorry for duplication.

I truly believe XPS should not be setup by devices.

XPS is policy, and policy does belong to user space.

User space knows that CPU 2  and 3 (pure examples) are reserved for storage
interrupts, not NIC ones.

Note that if XPS is not setup, MQ queue selection is just fine by default ;