Re: [ovs-dev] [PATCH v3] netdev-dpdk: Allow configurable queue sizes for 'dpdk' ports
On 21.09.2016 20:37, Loftus, Ciara wrote:
>>
>> Few comments inline.
>
> Thanks for the feedback Ilya.
>
>>
>>> The 'options:n_rxq_desc' and 'n_txq_desc' fields allow the number of rx
>>> and tx descriptors for dpdk ports to be modified. By default the values
>>> are set to 2048, but can be modified to an integer between 1 and 4096
>>> that is a power of two. The values can be modified at runtime, however
>>> require the NIC to restart when changed.
>>>
>>> Signed-off-by: Ciara Loftus
>>>
>>> ---
>>> v3:
>>> * Make queue sizes per-port rather than global
>>> * Check if queue size is power of 2 - fail if so.
>>>
>>> v2:
>>> * Rebase
>>>
>>> INSTALL.DPDK-ADVANCED.md | 16 ++--
>>> NEWS | 2 ++
>>> lib/netdev-dpdk.c| 48
>> +++-
>>> vswitchd/vswitch.xml | 22 ++
>>> 4 files changed, 81 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/INSTALL.DPDK-ADVANCED.md b/INSTALL.DPDK-ADVANCED.md
>>> index d7b9873..488e84f 100644
>>> --- a/INSTALL.DPDK-ADVANCED.md
>>> +++ b/INSTALL.DPDK-ADVANCED.md
>>> @@ -257,7 +257,19 @@ needs to be affinitized accordingly.
>>>The rx queues are assigned to pmd threads on the same NUMA node in a
>>>round-robin fashion.
>>>
>>> -### 4.4 Exact Match Cache
>>> +### 4.4 DPDK Physical Port Queue Sizes
>>> + `ovs-vsctl set Interface dpdk0 options:n_rxq_desc=`
>>> + `ovs-vsctl set Interface dpdk0 options:n_txq_desc=`
>>> +
>>> + The command above sets the number of rx/tx descriptors that the NIC
>>> + associated with dpdk0 will be initialised with.
>>> +
>>> + Different 'n_rxq_desc' and 'n_txq_desc' configurations yield different
>>> + benefits in terms of throughput and latency for different scenarios.
>>> + Generally, smaller queue sizes can have a positive impact for latency at
>> the
>>> + expense of throughput. The opposite is often true for larger queue sizes.
>>
>> Here we can mention that increasing the number of rx descriptors may lead
>> to performance degradation because of using non-vectorized rx functions.
>> At least this is true for i40e and maybe ixgbe dpdk drivers. Setting
>> 'n_rxq_desc=4096' for them will lead to disabling of vectorized rx.
>
> It seems the same applies for ixgbe (IXGBE_MAX_RING_DESC=4096)
> http://dpdk.org/doc/guides/nics/ixgbe.html#rx-constraints
> I will include this info in the next version.
>
>>
>>> +
>>> +### 4.5 Exact Match Cache
>>>
>>>Each pmd thread contains one EMC. After initial flow setup in the
>>>datapath, the EMC contains a single table and provides the lowest level
>>> @@ -274,7 +286,7 @@ needs to be affinitized accordingly.
>>>avoiding datapath classifier lookups is to have multiple pmd threads
>>>running. This can be done as described in section 4.2.
>>>
>>> -### 4.5 Rx Mergeable buffers
>>> +### 4.6 Rx Mergeable buffers
>>>
>>>Rx Mergeable buffers is a virtio feature that allows chaining of multiple
>>>virtio descriptors to handle large packet sizes. As such, large packets
>>> diff --git a/NEWS b/NEWS
>>> index 21ab538..901886d 100644
>>> --- a/NEWS
>>> +++ b/NEWS
>>> @@ -125,6 +125,8 @@ v2.6.0 - xx xxx
>>> * Remove dpdkvhostcuse port type.
>>> * OVS client mode for vHost and vHost reconnect (Requires QEMU 2.7)
>>> * 'dpdkvhostuserclient' port type.
>>> + * New option 'n_rxq_desc' and 'n_txq_desc' fields for DPDK interfaces
>>> + which set the number of rx and tx descriptors to use for the given
>> port.
>>> - Increase number of registers to 16.
>>> - ovs-benchmark: This utility has been removed due to lack of use and
>>> bitrot.
>>> diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
>>> index 89bdc4d..228993f 100644
>>> --- a/lib/netdev-dpdk.c
>>> +++ b/lib/netdev-dpdk.c
>>> @@ -132,8 +132,9 @@ BUILD_ASSERT_DECL((MAX_NB_MBUF /
>> ROUND_DOWN_POW2(MAX_NB_MBUF/MIN_NB_MBUF))
>>>
>>> #define SOCKET0 0
>>>
>>> -#define NIC_PORT_RX_Q_SIZE 2048 /* Size of Physical NIC RX Queue,
>> Max (n+32<=4096)*/
>>> -#define NIC_PORT_TX_Q_SIZE 2048 /* Size of Physical NIC TX Queue,
>> Max (n+32<=4096)*/
>>> +#define NIC_PORT_DEFAULT_RXQ_SIZE 2048 /* Default size of Physical
>> NIC RXQ */
>>> +#define NIC_PORT_DEFAULT_TXQ_SIZE 2048 /* Default size of Physical
>> NIC TXQ */
>>> +#define NIC_PORT_MAX_Q_SIZE 4096 /* Maximum size of Physical
>> NIC Queue */
>>>
>>> #define OVS_VHOST_MAX_QUEUE_NUM 1024 /* Maximum number of
>> vHost TX queues. */
>>> #define OVS_VHOST_QUEUE_MAP_UNKNOWN (-1) /* Mapping not
>> initialized. */
>>> @@ -372,6 +373,12 @@ struct netdev_dpdk {
>>> int requested_mtu;
>>> int requested_n_txq;
>>> int requested_n_rxq;
>>> +int requested_rxq_size;
>>> +int requested_txq_size;
>>> +
>>> +/* Number of rx/tx descriptors for physical devices */
>>> +int rxq_size;
>>> +int txq_size;
>>>
>>> /* Socket ID detected when vHost device is brought up */
>>> int requested_socket_id;
>>
Re: [ovs-dev] [PATCH v3] netdev-dpdk: Allow configurable queue sizes for 'dpdk' ports
>
> Few comments inline.
Thanks for the feedback Ilya.
>
> > The 'options:n_rxq_desc' and 'n_txq_desc' fields allow the number of rx
> > and tx descriptors for dpdk ports to be modified. By default the values
> > are set to 2048, but can be modified to an integer between 1 and 4096
> > that is a power of two. The values can be modified at runtime, however
> > require the NIC to restart when changed.
> >
> > Signed-off-by: Ciara Loftus
> >
> > ---
> > v3:
> > * Make queue sizes per-port rather than global
> > * Check if queue size is power of 2 - fail if so.
> >
> > v2:
> > * Rebase
> >
> > INSTALL.DPDK-ADVANCED.md | 16 ++--
> > NEWS | 2 ++
> > lib/netdev-dpdk.c| 48
> +++-
> > vswitchd/vswitch.xml | 22 ++
> > 4 files changed, 81 insertions(+), 7 deletions(-)
> >
> > diff --git a/INSTALL.DPDK-ADVANCED.md b/INSTALL.DPDK-ADVANCED.md
> > index d7b9873..488e84f 100644
> > --- a/INSTALL.DPDK-ADVANCED.md
> > +++ b/INSTALL.DPDK-ADVANCED.md
> > @@ -257,7 +257,19 @@ needs to be affinitized accordingly.
> >The rx queues are assigned to pmd threads on the same NUMA node in a
> >round-robin fashion.
> >
> > -### 4.4 Exact Match Cache
> > +### 4.4 DPDK Physical Port Queue Sizes
> > + `ovs-vsctl set Interface dpdk0 options:n_rxq_desc=`
> > + `ovs-vsctl set Interface dpdk0 options:n_txq_desc=`
> > +
> > + The command above sets the number of rx/tx descriptors that the NIC
> > + associated with dpdk0 will be initialised with.
> > +
> > + Different 'n_rxq_desc' and 'n_txq_desc' configurations yield different
> > + benefits in terms of throughput and latency for different scenarios.
> > + Generally, smaller queue sizes can have a positive impact for latency at
> the
> > + expense of throughput. The opposite is often true for larger queue sizes.
>
> Here we can mention that increasing the number of rx descriptors may lead
> to performance degradation because of using non-vectorized rx functions.
> At least this is true for i40e and maybe ixgbe dpdk drivers. Setting
> 'n_rxq_desc=4096' for them will lead to disabling of vectorized rx.
It seems the same applies for ixgbe (IXGBE_MAX_RING_DESC=4096)
http://dpdk.org/doc/guides/nics/ixgbe.html#rx-constraints
I will include this info in the next version.
>
> > +
> > +### 4.5 Exact Match Cache
> >
> >Each pmd thread contains one EMC. After initial flow setup in the
> >datapath, the EMC contains a single table and provides the lowest level
> > @@ -274,7 +286,7 @@ needs to be affinitized accordingly.
> >avoiding datapath classifier lookups is to have multiple pmd threads
> >running. This can be done as described in section 4.2.
> >
> > -### 4.5 Rx Mergeable buffers
> > +### 4.6 Rx Mergeable buffers
> >
> >Rx Mergeable buffers is a virtio feature that allows chaining of multiple
> >virtio descriptors to handle large packet sizes. As such, large packets
> > diff --git a/NEWS b/NEWS
> > index 21ab538..901886d 100644
> > --- a/NEWS
> > +++ b/NEWS
> > @@ -125,6 +125,8 @@ v2.6.0 - xx xxx
> > * Remove dpdkvhostcuse port type.
> > * OVS client mode for vHost and vHost reconnect (Requires QEMU 2.7)
> > * 'dpdkvhostuserclient' port type.
> > + * New option 'n_rxq_desc' and 'n_txq_desc' fields for DPDK interfaces
> > + which set the number of rx and tx descriptors to use for the given
> port.
> > - Increase number of registers to 16.
> > - ovs-benchmark: This utility has been removed due to lack of use and
> > bitrot.
> > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
> > index 89bdc4d..228993f 100644
> > --- a/lib/netdev-dpdk.c
> > +++ b/lib/netdev-dpdk.c
> > @@ -132,8 +132,9 @@ BUILD_ASSERT_DECL((MAX_NB_MBUF /
> ROUND_DOWN_POW2(MAX_NB_MBUF/MIN_NB_MBUF))
> >
> > #define SOCKET0 0
> >
> > -#define NIC_PORT_RX_Q_SIZE 2048 /* Size of Physical NIC RX Queue,
> Max (n+32<=4096)*/
> > -#define NIC_PORT_TX_Q_SIZE 2048 /* Size of Physical NIC TX Queue,
> Max (n+32<=4096)*/
> > +#define NIC_PORT_DEFAULT_RXQ_SIZE 2048 /* Default size of Physical
> NIC RXQ */
> > +#define NIC_PORT_DEFAULT_TXQ_SIZE 2048 /* Default size of Physical
> NIC TXQ */
> > +#define NIC_PORT_MAX_Q_SIZE 4096 /* Maximum size of Physical
> NIC Queue */
> >
> > #define OVS_VHOST_MAX_QUEUE_NUM 1024 /* Maximum number of
> vHost TX queues. */
> > #define OVS_VHOST_QUEUE_MAP_UNKNOWN (-1) /* Mapping not
> initialized. */
> > @@ -372,6 +373,12 @@ struct netdev_dpdk {
> > int requested_mtu;
> > int requested_n_txq;
> > int requested_n_rxq;
> > +int requested_rxq_size;
> > +int requested_txq_size;
> > +
> > +/* Number of rx/tx descriptors for physical devices */
> > +int rxq_size;
> > +int txq_size;
> >
> > /* Socket ID detected when vHost device is brought up */
> > int requested_socket_id;
> > @@ -646,7 +653,7 @@ dpdk_eth_dev_queue_setup(struct netdev_dpdk
> *dev,
Re: [ovs-dev] [PATCH v3] netdev-dpdk: Allow configurable queue sizes for 'dpdk' ports
Few comments inline.
> The 'options:n_rxq_desc' and 'n_txq_desc' fields allow the number of rx
> and tx descriptors for dpdk ports to be modified. By default the values
> are set to 2048, but can be modified to an integer between 1 and 4096
> that is a power of two. The values can be modified at runtime, however
> require the NIC to restart when changed.
>
> Signed-off-by: Ciara Loftus
>
> ---
> v3:
> * Make queue sizes per-port rather than global
> * Check if queue size is power of 2 - fail if so.
>
> v2:
> * Rebase
>
> INSTALL.DPDK-ADVANCED.md | 16 ++--
> NEWS | 2 ++
> lib/netdev-dpdk.c| 48
> +++-
> vswitchd/vswitch.xml | 22 ++
> 4 files changed, 81 insertions(+), 7 deletions(-)
>
> diff --git a/INSTALL.DPDK-ADVANCED.md b/INSTALL.DPDK-ADVANCED.md
> index d7b9873..488e84f 100644
> --- a/INSTALL.DPDK-ADVANCED.md
> +++ b/INSTALL.DPDK-ADVANCED.md
> @@ -257,7 +257,19 @@ needs to be affinitized accordingly.
>The rx queues are assigned to pmd threads on the same NUMA node in a
>round-robin fashion.
>
> -### 4.4 Exact Match Cache
> +### 4.4 DPDK Physical Port Queue Sizes
> + `ovs-vsctl set Interface dpdk0 options:n_rxq_desc=`
> + `ovs-vsctl set Interface dpdk0 options:n_txq_desc=`
> +
> + The command above sets the number of rx/tx descriptors that the NIC
> + associated with dpdk0 will be initialised with.
> +
> + Different 'n_rxq_desc' and 'n_txq_desc' configurations yield different
> + benefits in terms of throughput and latency for different scenarios.
> + Generally, smaller queue sizes can have a positive impact for latency at
> the
> + expense of throughput. The opposite is often true for larger queue sizes.
Here we can mention that increasing the number of rx descriptors may lead
to performance degradation because of using non-vectorized rx functions.
At least this is true for i40e and maybe ixgbe dpdk drivers. Setting
'n_rxq_desc=4096' for them will lead to disabling of vectorized rx.
> +
> +### 4.5 Exact Match Cache
>
>Each pmd thread contains one EMC. After initial flow setup in the
>datapath, the EMC contains a single table and provides the lowest level
> @@ -274,7 +286,7 @@ needs to be affinitized accordingly.
>avoiding datapath classifier lookups is to have multiple pmd threads
>running. This can be done as described in section 4.2.
>
> -### 4.5 Rx Mergeable buffers
> +### 4.6 Rx Mergeable buffers
>
>Rx Mergeable buffers is a virtio feature that allows chaining of multiple
>virtio descriptors to handle large packet sizes. As such, large packets
> diff --git a/NEWS b/NEWS
> index 21ab538..901886d 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -125,6 +125,8 @@ v2.6.0 - xx xxx
> * Remove dpdkvhostcuse port type.
> * OVS client mode for vHost and vHost reconnect (Requires QEMU 2.7)
> * 'dpdkvhostuserclient' port type.
> + * New option 'n_rxq_desc' and 'n_txq_desc' fields for DPDK interfaces
> + which set the number of rx and tx descriptors to use for the given
> port.
> - Increase number of registers to 16.
> - ovs-benchmark: This utility has been removed due to lack of use and
> bitrot.
> diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
> index 89bdc4d..228993f 100644
> --- a/lib/netdev-dpdk.c
> +++ b/lib/netdev-dpdk.c
> @@ -132,8 +132,9 @@ BUILD_ASSERT_DECL((MAX_NB_MBUF /
> ROUND_DOWN_POW2(MAX_NB_MBUF/MIN_NB_MBUF))
>
> #define SOCKET0 0
>
> -#define NIC_PORT_RX_Q_SIZE 2048 /* Size of Physical NIC RX Queue, Max
> (n+32<=4096)*/
> -#define NIC_PORT_TX_Q_SIZE 2048 /* Size of Physical NIC TX Queue, Max
> (n+32<=4096)*/
> +#define NIC_PORT_DEFAULT_RXQ_SIZE 2048 /* Default size of Physical NIC RXQ */
> +#define NIC_PORT_DEFAULT_TXQ_SIZE 2048 /* Default size of Physical NIC TXQ */
> +#define NIC_PORT_MAX_Q_SIZE 4096 /* Maximum size of Physical NIC Queue
> */
>
> #define OVS_VHOST_MAX_QUEUE_NUM 1024 /* Maximum number of vHost TX queues.
> */
> #define OVS_VHOST_QUEUE_MAP_UNKNOWN (-1) /* Mapping not initialized. */
> @@ -372,6 +373,12 @@ struct netdev_dpdk {
> int requested_mtu;
> int requested_n_txq;
> int requested_n_rxq;
> +int requested_rxq_size;
> +int requested_txq_size;
> +
> +/* Number of rx/tx descriptors for physical devices */
> +int rxq_size;
> +int txq_size;
>
> /* Socket ID detected when vHost device is brought up */
> int requested_socket_id;
> @@ -646,7 +653,7 @@ dpdk_eth_dev_queue_setup(struct netdev_dpdk *dev, int
> n_rxq, int n_txq)
> }
>
> for (i = 0; i < n_txq; i++) {
> -diag = rte_eth_tx_queue_setup(dev->port_id, i,
> NIC_PORT_TX_Q_SIZE,
> +diag = rte_eth_tx_queue_setup(dev->port_id, i, dev->txq_size,
>dev->socket_id, NULL);
> if (diag) {
> VLOG_INFO("Interface %s txq(%d) setup er
Re: [ovs-dev] [PATCH v3] netdev-dpdk: Allow configurable queue sizes for 'dpdk' ports
On Tue, 20 Sep 2016 15:13:38 +0100 Ciara Loftus wrote: > The 'options:n_rxq_desc' and 'n_txq_desc' fields allow the number of > rx and tx descriptors for dpdk ports to be modified. By default the > values are set to 2048, but can be modified to an integer between 1 > and 4096 that is a power of two. The values can be modified at > runtime, however require the NIC to restart when changed. > > Signed-off-by: Ciara Loftus > Ciaria, thanks for the patch. Acked-by: Yunhong Jiang ___ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
