Re: [lng-odp] [RFC API-NEXT PATCH 0/6] Multiple queue packet IO API

[Nikita Kalyazin]

To sum up, we need the following things for proper multiqueue ingress support
in ODP:
 - a way to assign multiple queues to a CoS (up to the number of queues the
specific NIC supports), e.g., odp_cos_queue_set() -> odp_cos_queues_set();
 - a way to define multiple input default queues (in the case classification
is not available);
 - a way to distribute the packets across those queues (e.g., RSS hashing);
 - remove recv/send-related stuff and migrate the code that uses it.
I suppose, the linux-generic should also be updated, so that odp_queue_deq/enq
don't rely on recv/send, but do all the NIC specific interaction by their own.
Apparently it would require exposing deq/enq callbacks in pktio_if_ops_t for
NIC drivers, similar to recv/send ones.
Do I miss anything?

As for egress, I didn't find anything related to multiple HW Tx queues in the
traffic manager code.  We'll probably need some hashing API for egress too.
Should be shaping/scheduling algorithms be aware of HW Tx queues or they work
at the level of pktio/spigot as a whole?

-- 

Best regards,

Nikita Kalyazin,
n.kalya...@samsung.com

Software Engineer
Virtualization Group
Samsung R Institute Russia
Tel: +7 (495) 797-25-00 #3816
Tel: +7 (495) 797-25-03
Office #1501, 12-1, Dvintsev str.,
Moscow, 127018, Russia

On Thu, Oct 22, 2015 at 06:53:03PM +0100, Stuart Haslam wrote:
> On Thu, Oct 22, 2015 at 06:06:58PM +0300, Nikita Kalyazin wrote:
> > Hi,
> > 
> > > [Alex] ODP queues are neither software nor hardware by definition, each
> > > implementation is free to implement them as they see fit. Also PacketI/O
> > > abstraction is not an abstraction for a NIC device. The ODP way to use 
> > > multiple
> > > queues for ingress is through the classifier. There was a former proposal 
> > > of
> > > enabling RSS at pktio level but it was not accepted. I also noticed that 
> > > this
> > > patch tries to introduce multiple default queues which is kind of 
> > > confusing -
> > > default traffic is all traffic which does not satisfy classifier 
> > > criteria. For
> > > output, multiple queues are used by the means of TM api. 
> > As I understand, ODP implies three options of ingress processing:
> >  - "recv" (poll pktio with raw recv() method);
> 
> I think we should be deprecating this method, and users of it move over
> to using polled (or scheduled) queues. There should be no performance
> penalty and the presence of recv mode just causes confusion.
> 
> >  - "poll" (poll poll pktio via default input queue associated with the 
> > pktio);
> 
> It's also possible to poll an input queue which is associated with a
> class of service. Although, looking at the validation test and example
> just now I see that they don't do this and they work exclusively with
> scheduled queues, this should be changed.
> 
> >  - "sched" (call schedule() which makes use of classifier and queue 
> > priorities).
> > Looks like each ODP implementation should support all the three options.
> > 
> > For "sched" option, classification may be implemented in hardware, and ODP
> > queues, in their turn, can also be implemented in hardware.  The hardware 
> > would
> > distribute packets to its hardware queues according to PMRs or L2/L3 QoS.
> 
> There's no way to do hash based distribution with the classification API
> currently, it would need to be extended. The initial design included
> some APIs for this but they weren't fully formed and didn't make it into
> the release. There has been some debate since then about whether support
> should be added as part of the pktio or classification API, but consensus
> seems to be that the classification API is the right place.
> 
> The rough idea would be to use the classification API to assign a CoS
> and then packets matching that CoS are distributed to a number of queues
> based on a hash of a set of preconfigured fields. At the minute packets
> can only be delivered to a single queue (assigned using odp_cos_set_queue())
> so this would need to be extended. If the hardware is only capable of
> RSS type distribution then you'd have no PMR or L2/L3 based CoS and
> everything would be distributed within the default CoS.
> 
> > For "recv" option, the only way possible to access NIC's hardware queues
> > (if any), is to call recv() on a specified queue explicitly.  So there 
> > should be
> > an API for it.  In my patch I proposed such APIs (odp_pktio_recv_queue()).
> > I admit, this might be not the best API name possible.  However I don't see 
> > any
> > other way to access NIC's hardware queues at "recv" level.
> 
> The right way to do this is to expose the hardware queues as ODP queues
> (odp_queue_t).
> 
> ODP queues should map directly to hardware queues wherever possible, so
> calling odp_queue_deq[_multi]() on a packet input queue reads packets
> directly from the hardware queue.
> 
> > We can add
> > hw_queue_id parameter to existing odp_pktio_recv() alternatively (same way 
> > as
> > DPDK does with rte_eth_rx_burst()).  
> 
> No, the 

Re: [lng-odp] [RFC API-NEXT PATCH 0/6] Multiple queue packet IO API

[Nikita Kalyazin]

Hi Alex,

> I understand your reasoning but ODP Packet I/O (pktio) is _not_ (yet?) a NIC
> abstraction. The fact is that in current ODP design ingress queues are
> "connected" to the pktio device by the configuration of classifier or as
> default/error CoS queues. Classifier can be changed dynamically, e.g, adding 
> or
> removing CoSes or queues without stopping/re-configuring the pktio device. I
> don't see in DPDK such features in PMD, all queues are configured and setup at
> initialization time and they don't change thereafter.
Sorry, could you clarify what do you mean by "ODP pktio is not a NIC
abstraction"?  I hear this second time, but not sure I fully understand the
meaning.  Does it mean, a pktio is not just a pipe to network with send/recv
operations defined on it, but also a bunch of higher level mechanisms like
cls/scheduling/shaping/dropping?

-- 

Best regards,

Nikita Kalyazin,
n.kalya...@samsung.com

Software Engineer
Virtualization Group
Samsung R Institute Russia
Tel: +7 (495) 797-25-00 #3816
Tel: +7 (495) 797-25-03
Office #1501, 12-1, Dvintsev str.,
Moscow, 127018, Russia

On Fri, Oct 23, 2015 at 10:26:02AM +0300, Alexandru Badicioiu wrote:
> 
> 
> On 22 October 2015 at 18:06, Nikita Kalyazin <[1]n.kalya...@samsung.com> 
> wrote:
> 
> Hi,
>
> > [Alex] ODP queues are neither software nor hardware by definition, each
> > implementation is free to implement them as they see fit. Also PacketI/O
> > abstraction is not an abstraction for a NIC device. The ODP way to use
> multiple
> > queues for ingress is through the classifier. There was a former 
> proposal
> of
> > enabling RSS at pktio level but it was not accepted. I also noticed that
> this
> > patch tries to introduce multiple default queues which is kind of
> confusing -
> > default traffic is all traffic which does not satisfy classifier
> criteria. For
> > output, multiple queues are used by the means of TM api. 
> As I understand, ODP implies three options of ingress processing:
>  - "recv" (poll pktio with raw recv() method);
>  - "poll" (poll poll pktio via default input queue associated with the
> pktio);
>  - "sched" (call schedule() which makes use of classifier and queue
> priorities).
> Looks like each ODP implementation should support all the three options.
> 
> For "sched" option, classification may be implemented in hardware, and ODP
> queues, in their turn, can also be implemented in hardware.  The hardware
> would
> distribute packets to its hardware queues according to PMRs or L2/L3 QoS.
> 
> For "recv" option, the only way possible to access NIC's hardware queues
> (if any), is to call recv() on a specified queue explicitly.  So there
> should be
> an API for it.  In my patch I proposed such APIs (odp_pktio_recv_queue()).
> I admit, this might be not the best API name possible.  However I don't 
> see
> any
> other way to access NIC's hardware queues at "recv" level.  We can add
> hw_queue_id parameter to existing odp_pktio_recv() alternatively (same way
> as
> DPDK does with rte_eth_rx_burst()).  In case of using such an explicit
> hardware
> queue polling, RSS should be configured somehow to distribute packets
> across them.
> 
> For "poll" option, it's not completely clear to me how it should relate to
> NIC's
> hardware queues.  It looks like classification is not involved here, and
> this
> option isn't really different from "recv".  If this is correct, presence 
> of
> multiple default input queues is reasonable, since they're directly mapped
> to
> NIC's hardware queues, and odp_queue_deq() simply extracts a packet from
> the
> appropriate HW queue.
> [Alex]
> 
> Hi Nikita, 
> I understand your reasoning but ODP Packet I/O (pktio) is _not_ (yet?) a NIC
> abstraction. The fact is that in current ODP design ingress queues are
> "connected" to the pktio device by the configuration of classifier or as
> default/error CoS queues. Classifier can be changed dynamically, e.g, adding 
> or
> removing CoSes or queues without stopping/re-configuring the pktio device. I
> don't see in DPDK such features in PMD, all queues are configured and setup at
> initialization time and they don't change thereafter.
>    
> 
> --
> 
> Best regards,
> 
> Nikita Kalyazin,
> [2]n.kalya...@samsung.com
> 
> Software Engineer
> Virtualization Group
> Samsung R Institute Russia
> Tel: [3]+7 (495) 797-25-00 #3816
> Tel: [4]+7 (495) 797-25-03
> Office #1501, 12-1, Dvintsev str.,
> Moscow, 127018, Russia
> 
> On Mon, Oct 19, 2015 at 08:17:17PM +0300, Alexandru Badicioiu wrote:
> >
> >
> > On 19 October 2015 at 19:16, Nikita Kalyazin <[1][5]
> n.kalya...@samsung.com> wrote:
> >
> >     Hi Stuart,
> >
> >
> >     Thanks for your feedback.
> >
> >     > One 

Re: [lng-odp] [RFC API-NEXT PATCH 0/6] Multiple queue packet IO API

[Alexandru Badicioiu]

On 22 October 2015 at 18:06, Nikita Kalyazin  wrote:

> Hi,
>
> > [Alex] ODP queues are neither software nor hardware by definition, each
> > implementation is free to implement them as they see fit. Also PacketI/O
> > abstraction is not an abstraction for a NIC device. The ODP way to use
> multiple
> > queues for ingress is through the classifier. There was a former
> proposal of
> > enabling RSS at pktio level but it was not accepted. I also noticed that
> this
> > patch tries to introduce multiple default queues which is kind of
> confusing -
> > default traffic is all traffic which does not satisfy classifier
> criteria. For
> > output, multiple queues are used by the means of TM api.
> As I understand, ODP implies three options of ingress processing:
>  - "recv" (poll pktio with raw recv() method);
>  - "poll" (poll poll pktio via default input queue associated with the
> pktio);
>  - "sched" (call schedule() which makes use of classifier and queue
> priorities).
> Looks like each ODP implementation should support all the three options.
>
> For "sched" option, classification may be implemented in hardware, and ODP
> queues, in their turn, can also be implemented in hardware.  The hardware
> would
> distribute packets to its hardware queues according to PMRs or L2/L3 QoS.
>
> For "recv" option, the only way possible to access NIC's hardware queues
> (if any), is to call recv() on a specified queue explicitly.  So there
> should be
> an API for it.  In my patch I proposed such APIs (odp_pktio_recv_queue()).
> I admit, this might be not the best API name possible.  However I don't
> see any
> other way to access NIC's hardware queues at "recv" level.  We can add
> hw_queue_id parameter to existing odp_pktio_recv() alternatively (same way
> as
> DPDK does with rte_eth_rx_burst()).  In case of using such an explicit
> hardware
> queue polling, RSS should be configured somehow to distribute packets
> across them.
>
> For "poll" option, it's not completely clear to me how it should relate to
> NIC's
> hardware queues.  It looks like classification is not involved here, and
> this
> option isn't really different from "recv".  If this is correct, presence of
> multiple default input queues is reasonable, since they're directly mapped
> to
> NIC's hardware queues, and odp_queue_deq() simply extracts a packet from
> the
> appropriate HW queue.
> [Alex]
>
Hi Nikita,
I understand your reasoning but ODP Packet I/O (pktio) is _not_ (yet?) a
NIC abstraction. The fact is that in current ODP design ingress queues are
"connected" to the pktio device by the configuration of classifier or as
default/error CoS queues. Classifier can be changed dynamically, e.g,
adding or removing CoSes or queues without stopping/re-configuring the
pktio device. I don't see in DPDK such features in PMD, all queues are
configured and setup at initialization time and they don't change
thereafter.


> --
>
> Best regards,
>
> Nikita Kalyazin,
> n.kalya...@samsung.com
>
> Software Engineer
> Virtualization Group
> Samsung R Institute Russia
> Tel: +7 (495) 797-25-00 #3816
> Tel: +7 (495) 797-25-03
> Office #1501, 12-1, Dvintsev str.,
> Moscow, 127018, Russia
>
> On Mon, Oct 19, 2015 at 08:17:17PM +0300, Alexandru Badicioiu wrote:
> >
> >
> > On 19 October 2015 at 19:16, Nikita Kalyazin <[1]n.kalya...@samsung.com>
> wrote:
> >
> > Hi Stuart,
> >
> >
> > Thanks for your feedback.
> >
> > > One thing that is missing here is a method for the application to
> > > configure how packets are distributed, i.e. which fields in the
> packet
> > > are used when calculating the flow hash (not necessarily the hash
> > > algorithm itself, that can be implementation defined).
> > >
> > > For example with the netmap implementation here, if you have a
> > > compatible interface, I assume the user can control distribution
> to RX
> > > queues using something like;
> > >
> > >   ethtool --config-ntuple rx-flow-hash esp4
> > >
> > > But it would be better if this were configurable via the ODP API.
> Petri
> > > had a suggestion a while back related to this to add a structure
> to the
> > > odp_pktio_params_t;
> > >
> > > enum odp_pktio_input_hash {
> > >   /** No specific fields defined */
> > >   ODP_PKTIN_HASH_NONE = 0,
> > >   /** IPv4/v6 addresses */
> > >   ODP_PKTIN_HASH_IP,
> > >   /** UDP ports and IPv4/v6 addresses */
> > >   ODP_PKTIN_HASH_UDP_IP,
> > >   /** TCP ports and IPv4/v6 addresses */
> > >   ODP_PKTIN_HASH_TCP_IP
> > > };
> > Thanks, I plan to think about it.
> > Btw, NICs provide wide variety of settings regarding distribution of
> > packets across the queues.  Is it supposed to extend the proposed
> > structure to support more flexible configuration?
> >
> > > How about:
> > >
> > > int odp_pktio_inq_create(odp_pktio_t pktio, const char *name,
> > >  

Re: [lng-odp] [RFC API-NEXT PATCH 0/6] Multiple queue packet IO API

[Stuart Haslam]

On Thu, Oct 22, 2015 at 06:06:58PM +0300, Nikita Kalyazin wrote:
> Hi,
> 
> > [Alex] ODP queues are neither software nor hardware by definition, each
> > implementation is free to implement them as they see fit. Also PacketI/O
> > abstraction is not an abstraction for a NIC device. The ODP way to use 
> > multiple
> > queues for ingress is through the classifier. There was a former proposal of
> > enabling RSS at pktio level but it was not accepted. I also noticed that 
> > this
> > patch tries to introduce multiple default queues which is kind of confusing 
> > -
> > default traffic is all traffic which does not satisfy classifier criteria. 
> > For
> > output, multiple queues are used by the means of TM api. 
> As I understand, ODP implies three options of ingress processing:
>  - "recv" (poll pktio with raw recv() method);

I think we should be deprecating this method, and users of it move over
to using polled (or scheduled) queues. There should be no performance
penalty and the presence of recv mode just causes confusion.

>  - "poll" (poll poll pktio via default input queue associated with the pktio);

It's also possible to poll an input queue which is associated with a
class of service. Although, looking at the validation test and example
just now I see that they don't do this and they work exclusively with
scheduled queues, this should be changed.

>  - "sched" (call schedule() which makes use of classifier and queue 
> priorities).
> Looks like each ODP implementation should support all the three options.
> 
> For "sched" option, classification may be implemented in hardware, and ODP
> queues, in their turn, can also be implemented in hardware.  The hardware 
> would
> distribute packets to its hardware queues according to PMRs or L2/L3 QoS.

There's no way to do hash based distribution with the classification API
currently, it would need to be extended. The initial design included
some APIs for this but they weren't fully formed and didn't make it into
the release. There has been some debate since then about whether support
should be added as part of the pktio or classification API, but consensus
seems to be that the classification API is the right place.

The rough idea would be to use the classification API to assign a CoS
and then packets matching that CoS are distributed to a number of queues
based on a hash of a set of preconfigured fields. At the minute packets
can only be delivered to a single queue (assigned using odp_cos_set_queue())
so this would need to be extended. If the hardware is only capable of
RSS type distribution then you'd have no PMR or L2/L3 based CoS and
everything would be distributed within the default CoS.

> For "recv" option, the only way possible to access NIC's hardware queues
> (if any), is to call recv() on a specified queue explicitly.  So there should 
> be
> an API for it.  In my patch I proposed such APIs (odp_pktio_recv_queue()).
> I admit, this might be not the best API name possible.  However I don't see 
> any
> other way to access NIC's hardware queues at "recv" level.

The right way to do this is to expose the hardware queues as ODP queues
(odp_queue_t).

ODP queues should map directly to hardware queues wherever possible, so
calling odp_queue_deq[_multi]() on a packet input queue reads packets
directly from the hardware queue.

> We can add
> hw_queue_id parameter to existing odp_pktio_recv() alternatively (same way as
> DPDK does with rte_eth_rx_burst()).  

No, the queue_id passed to rte_eth_rx_burst() should be seen as being
equivalent to an ODP queue. So these APIs are logically equivalent;

uint16_t rte_eth_rx_burst(uint8_t port_id, uint16_t queue_id, struct rte_mbuf 
**rx_pkts, const uint16_t nb_pkts);

int odp_queue_deq_multi(odp_queue_t queue, odp_event_t events[], int num);

> In case of using such an explicit hardware
> queue polling, RSS should be configured somehow to distribute packets across 
> them.
>
> For "poll" option, it's not completely clear to me how it should relate to 
> NIC's
> hardware queues.  It looks like classification is not involved here, and this
> option isn't really different from "recv".  If this is correct, presence of
> multiple default input queues is reasonable, since they're directly mapped to
> NIC's hardware queues, and odp_queue_deq() simply extracts a packet from the
> appropriate HW queue.
> 
> -- 
> 
> Best regards,
> 
> Nikita Kalyazin,
> n.kalya...@samsung.com
> 
> Software Engineer
> Virtualization Group
> Samsung R Institute Russia
> Tel: +7 (495) 797-25-00 #3816
> Tel: +7 (495) 797-25-03
> Office #1501, 12-1, Dvintsev str.,
> Moscow, 127018, Russia
> 
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

Re: [lng-odp] [RFC API-NEXT PATCH 0/6] Multiple queue packet IO API

[Alexandru Badicioiu]

On 19 October 2015 at 19:16, Nikita Kalyazin  wrote:

> Hi Stuart,
>
>
> Thanks for your feedback.
>
> > One thing that is missing here is a method for the application to
> > configure how packets are distributed, i.e. which fields in the packet
> > are used when calculating the flow hash (not necessarily the hash
> > algorithm itself, that can be implementation defined).
> >
> > For example with the netmap implementation here, if you have a
> > compatible interface, I assume the user can control distribution to RX
> > queues using something like;
> >
> >   ethtool --config-ntuple rx-flow-hash esp4
> >
> > But it would be better if this were configurable via the ODP API. Petri
> > had a suggestion a while back related to this to add a structure to the
> > odp_pktio_params_t;
> >
> > enum odp_pktio_input_hash {
> >   /** No specific fields defined */
> >   ODP_PKTIN_HASH_NONE = 0,
> >   /** IPv4/v6 addresses */
> >   ODP_PKTIN_HASH_IP,
> >   /** UDP ports and IPv4/v6 addresses */
> >   ODP_PKTIN_HASH_UDP_IP,
> >   /** TCP ports and IPv4/v6 addresses */
> >   ODP_PKTIN_HASH_TCP_IP
> > };
> Thanks, I plan to think about it.
> Btw, NICs provide wide variety of settings regarding distribution of
> packets across the queues.  Is it supposed to extend the proposed
> structure to support more flexible configuration?
>
> > How about:
> >
> > int odp_pktio_inq_create(odp_pktio_t pktio, const char *name,
> >  const odp_queue_param_t *param,
> >  odp_queue_t queues[], int num);
> >
> > Which would return the number of queues created, and ensure that all
> > queues had the same parameters. Or, maybe it's time to start using
> > odp_queue_group_t, which we have always intended to be a group of queues
> > that all have the same properties.
> Good point.  I'll consider the cleaner approach.
>
> > I'm not keen on the odp_pktio_recv/send_queue() APIs as the queues
> > referred to there are not ODP queues so it makes things a bit less clear,
> > and in general we should be moving away from using the direct _send/_recv
> > APIs. If we're explicitly using queues then we should use the queue
> and/or
> > scheduler APIs.
> Right, these API names look a bit confusing, because same word is used to
> refer to both hardware NIC queues and ODP (software) queues.
>
[Alex] ODP queues are neither software nor hardware by definition, each
implementation is free to implement them as they see fit. Also PacketI/O
abstraction is not an abstraction for a NIC device. The ODP way to use
multiple queues for ingress is through the classifier. There was a former
proposal of enabling RSS at pktio level but it was not accepted. I also
noticed that this patch tries to introduce multiple default queues which is
kind of confusing - default traffic is all traffic which does not satisfy
classifier criteria. For output, multiple queues are used by the means of
TM api.

> However, I think it would be useful to preserve explicit send/recv APIs,
> since in many cases they might appear more efficient (odp-ovs is an example
> of such applications), and rename those to make them more obvious.
>
> > Are you able to join the meeting on Tuesday to discuss?
> >
> > http://www.opendataplane.org/meetings/
> Thanks, I'm going to join the meeting.
>
> --
>
> Best regards,
>
> Nikita Kalyazin,
> n.kalya...@samsung.com
>
> Software Engineer
> Virtualization Group
> Samsung R Institute Russia
> Tel: +7 (495) 797-25-00 #3816
> Tel: +7 (495) 797-25-03
> Office #1501, 12-1, Dvintsev str.,
> Moscow, 127018, Russia
>
> On Fri, Oct 16, 2015 at 07:15:43PM +0100, Stuart Haslam wrote:
> > On Wed, Oct 14, 2015 at 02:32:37PM +0300, Nikita Kalyazin wrote:
> > > This series is a proposal of multiqueue API for ODP.
> > > Modern network adapters (both physical and virtual) support multiple
> > > hardware queues.  Assigning separate CPU core for processing of each
> > > queue allows to scale throughput (in best case - linearly).
> > >
> >
> > Thanks for the contribution, we do need something like this, in
> > particular hash based distribution has been on the todo list for a
> > little while. I've not reviewed the implementation in detail but have
> > focused on the API changes specifically and have a few initial thoughts
> > below.
> >
> > One thing that is missing here is a method for the application to
> > configure how packets are distributed, i.e. which fields in the packet
> > are used when calculating the flow hash (not necessarily the hash
> > algorithm itself, that can be implementation defined).
> >
> > For example with the netmap implementation here, if you have a
> > compatible interface, I assume the user can control distribution to RX
> > queues using something like;
> >
> >   ethtool --config-ntuple rx-flow-hash esp4
> >
> > But it would be better if this were configurable via the ODP API. Petri
> > had a suggestion a while back related to this to add a structure to the

Re: [lng-odp] [RFC API-NEXT PATCH 0/6] Multiple queue packet IO API

[Nikita Kalyazin]

Hi Stuart,


Thanks for your feedback.

> One thing that is missing here is a method for the application to
> configure how packets are distributed, i.e. which fields in the packet
> are used when calculating the flow hash (not necessarily the hash
> algorithm itself, that can be implementation defined).
>
> For example with the netmap implementation here, if you have a
> compatible interface, I assume the user can control distribution to RX
> queues using something like;
> 
>   ethtool --config-ntuple rx-flow-hash esp4
> 
> But it would be better if this were configurable via the ODP API. Petri
> had a suggestion a while back related to this to add a structure to the
> odp_pktio_params_t;
> 
> enum odp_pktio_input_hash {
>   /** No specific fields defined */
>   ODP_PKTIN_HASH_NONE = 0,
>   /** IPv4/v6 addresses */
>   ODP_PKTIN_HASH_IP,
>   /** UDP ports and IPv4/v6 addresses */
>   ODP_PKTIN_HASH_UDP_IP,
>   /** TCP ports and IPv4/v6 addresses */
>   ODP_PKTIN_HASH_TCP_IP
> };
Thanks, I plan to think about it.
Btw, NICs provide wide variety of settings regarding distribution of
packets across the queues.  Is it supposed to extend the proposed
structure to support more flexible configuration?

> How about:
> 
> int odp_pktio_inq_create(odp_pktio_t pktio, const char *name,
>  const odp_queue_param_t *param,
>  odp_queue_t queues[], int num);
>
> Which would return the number of queues created, and ensure that all
> queues had the same parameters. Or, maybe it's time to start using
> odp_queue_group_t, which we have always intended to be a group of queues
> that all have the same properties.
Good point.  I'll consider the cleaner approach.

> I'm not keen on the odp_pktio_recv/send_queue() APIs as the queues
> referred to there are not ODP queues so it makes things a bit less clear,
> and in general we should be moving away from using the direct _send/_recv
> APIs. If we're explicitly using queues then we should use the queue and/or
> scheduler APIs.
Right, these API names look a bit confusing, because same word is used to
refer to both hardware NIC queues and ODP (software) queues.
However, I think it would be useful to preserve explicit send/recv APIs,
since in many cases they might appear more efficient (odp-ovs is an example
of such applications), and rename those to make them more obvious.

> Are you able to join the meeting on Tuesday to discuss?
> 
> http://www.opendataplane.org/meetings/
Thanks, I'm going to join the meeting.

-- 

Best regards,

Nikita Kalyazin,
n.kalya...@samsung.com

Software Engineer
Virtualization Group
Samsung R Institute Russia
Tel: +7 (495) 797-25-00 #3816
Tel: +7 (495) 797-25-03
Office #1501, 12-1, Dvintsev str.,
Moscow, 127018, Russia

On Fri, Oct 16, 2015 at 07:15:43PM +0100, Stuart Haslam wrote:
> On Wed, Oct 14, 2015 at 02:32:37PM +0300, Nikita Kalyazin wrote:
> > This series is a proposal of multiqueue API for ODP.
> > Modern network adapters (both physical and virtual) support multiple
> > hardware queues.  Assigning separate CPU core for processing of each
> > queue allows to scale throughput (in best case - linearly).
> > 
> 
> Thanks for the contribution, we do need something like this, in
> particular hash based distribution has been on the todo list for a
> little while. I've not reviewed the implementation in detail but have
> focused on the API changes specifically and have a few initial thoughts
> below.
> 
> One thing that is missing here is a method for the application to
> configure how packets are distributed, i.e. which fields in the packet
> are used when calculating the flow hash (not necessarily the hash
> algorithm itself, that can be implementation defined).
> 
> For example with the netmap implementation here, if you have a
> compatible interface, I assume the user can control distribution to RX
> queues using something like;
> 
>   ethtool --config-ntuple rx-flow-hash esp4
> 
> But it would be better if this were configurable via the ODP API. Petri
> had a suggestion a while back related to this to add a structure to the
> odp_pktio_params_t;
> 
> enum odp_pktio_input_hash {
>   /** No specific fields defined */
>   ODP_PKTIN_HASH_NONE = 0,
>   /** IPv4/v6 addresses */
>   ODP_PKTIN_HASH_IP,
>   /** UDP ports and IPv4/v6 addresses */
>   ODP_PKTIN_HASH_UDP_IP,
>   /** TCP ports and IPv4/v6 addresses */
>   ODP_PKTIN_HASH_TCP_IP
> };
> 
> As for RX queue creation, rather than proposed sequence that looks like
> this:
> 
> odp_pktio_max_num_queues()
> odp_pktio_configure()
> foreach inq
> odp_queue_create()
> 
> How about:
> 
> int odp_pktio_inq_create(odp_pktio_t pktio, const char *name,
>  const odp_queue_param_t *param,
>  odp_queue_t queues[], int num);
> 
> Which would return the number of queues created, and ensure that all
> queues had the same parameters. Or, maybe it's time to 

Re: [lng-odp] [RFC API-NEXT PATCH 0/6] Multiple queue packet IO API

[Stuart Haslam]

On Wed, Oct 14, 2015 at 02:32:37PM +0300, Nikita Kalyazin wrote:
> This series is a proposal of multiqueue API for ODP.
> Modern network adapters (both physical and virtual) support multiple
> hardware queues.  Assigning separate CPU core for processing of each
> queue allows to scale throughput (in best case - linearly).
> 

Thanks for the contribution, we do need something like this, in
particular hash based distribution has been on the todo list for a
little while. I've not reviewed the implementation in detail but have
focused on the API changes specifically and have a few initial thoughts
below.

One thing that is missing here is a method for the application to
configure how packets are distributed, i.e. which fields in the packet
are used when calculating the flow hash (not necessarily the hash
algorithm itself, that can be implementation defined).

For example with the netmap implementation here, if you have a
compatible interface, I assume the user can control distribution to RX
queues using something like;

  ethtool --config-ntuple rx-flow-hash esp4

But it would be better if this were configurable via the ODP API. Petri
had a suggestion a while back related to this to add a structure to the
odp_pktio_params_t;

enum odp_pktio_input_hash {
/** No specific fields defined */
ODP_PKTIN_HASH_NONE = 0,
/** IPv4/v6 addresses */
ODP_PKTIN_HASH_IP,
/** UDP ports and IPv4/v6 addresses */
ODP_PKTIN_HASH_UDP_IP,
/** TCP ports and IPv4/v6 addresses */
ODP_PKTIN_HASH_TCP_IP
};

As for RX queue creation, rather than proposed sequence that looks like
this:

odp_pktio_max_num_queues()
odp_pktio_configure()
foreach inq
odp_queue_create()

How about:

int odp_pktio_inq_create(odp_pktio_t pktio, const char *name,
 const odp_queue_param_t *param,
 odp_queue_t queues[], int num);

Which would return the number of queues created, and ensure that all
queues had the same parameters. Or, maybe it's time to start using
odp_queue_group_t, which we have always intended to be a group of queues
that all have the same properties.

I'm not keen on the odp_pktio_recv/send_queue() APIs as the queues
referred to there are not ODP queues so it makes things a bit less clear,
and in general we should be moving away from using the direct _send/_recv
APIs. If we're explicitly using queues then we should use the queue and/or
scheduler APIs.

Are you able to join the meeting on Tuesday to discuss?

http://www.opendataplane.org/meetings/

> This series should be considered a prototype.
> This implementation of multiqueue API is not complete (not all the
> code has been migrated to multiqueue API).  Existing packet IO API has
> been left untouched and reimplemented using mutliqueue API (queue 0 is
> always used in such cases).  The existing API is supposed to be
> removed in the future, as soon as all the remaining code migrates to
> the multiqueue API.
> Since single lock for all the queues makes all pktio Rx and Tx calls
> sequential, per queue locks have been added.
> For the purpose of evaluation, multiqueue support has been added to
> netmap pktio.
> 
> Patch 1 adds multiqueue prototypes of pktio API functions.
> Patches 2-4 contain common implementation of the multiqueue API.
> Patch 5 adds multiqueue support in netmap pktio.
> Patch 6 moves odp_generator to multiqueue API.
> 
> In our tests (odp_generator sends packets, odp_generator receives
> packets), the following results have been obtained:
> +--++---+
> | # queues | kpps (64-byte packets) | boost |
> +--++---+
> |1 |853 | 1.00x |
> |2 |   1404 | 1.64x |
> |4 |   2374 | 2.78x |
> |8 |   4345 | 5.09x |
> +--++---+
> 
> Nikita Kalyazin (6):
>   api: pktio: add multiqueue API
>   linux-generic: pktio: add per queue pktio locks
>   linux-generic: pktio: implement multiqueue API (part I)
>   linux-generic: pktio: imlement multiqueue API (part II)
>   linux-generic: pktio: multiqueue support in netmap
>   example: generator: use multiqueue API
> 
>  example/generator/odp_generator.c  |  63 +++-
>  include/odp/api/config.h   |   7 +-
>  include/odp/api/packet_io.h| 118 ++
>  .../linux-generic/include/odp_packet_io_internal.h |  19 +-
>  platform/linux-generic/include/odp_packet_netmap.h |   6 +-
>  .../linux-generic/include/odp_queue_internal.h |   2 +
>  .../linux-generic/include/odp_schedule_internal.h  |   1 +
>  platform/linux-generic/odp_packet_io.c | 405 
> +
>  platform/linux-generic/odp_schedule.c  |  26 +-
>  platform/linux-generic/pktio/netmap.c  | 218 +--
>  10 files changed, 725 insertions(+), 140 deletions(-)
> 
> -- 
> 

[lng-odp] [RFC API-NEXT PATCH 0/6] Multiple queue packet IO API

[Nikita Kalyazin]

This series is a proposal of multiqueue API for ODP.
Modern network adapters (both physical and virtual) support multiple
hardware queues.  Assigning separate CPU core for processing of each
queue allows to scale throughput (in best case - linearly).

This series should be considered a prototype.
This implementation of multiqueue API is not complete (not all the
code has been migrated to multiqueue API).  Existing packet IO API has
been left untouched and reimplemented using mutliqueue API (queue 0 is
always used in such cases).  The existing API is supposed to be
removed in the future, as soon as all the remaining code migrates to
the multiqueue API.
Since single lock for all the queues makes all pktio Rx and Tx calls
sequential, per queue locks have been added.
For the purpose of evaluation, multiqueue support has been added to
netmap pktio.

Patch 1 adds multiqueue prototypes of pktio API functions.
Patches 2-4 contain common implementation of the multiqueue API.
Patch 5 adds multiqueue support in netmap pktio.
Patch 6 moves odp_generator to multiqueue API.

In our tests (odp_generator sends packets, odp_generator receives
packets), the following results have been obtained:
+--++---+
| # queues | kpps (64-byte packets) | boost |
+--++---+
|1 |853 | 1.00x |
|2 |   1404 | 1.64x |
|4 |   2374 | 2.78x |
|8 |   4345 | 5.09x |
+--++---+

Nikita Kalyazin (6):
  api: pktio: add multiqueue API
  linux-generic: pktio: add per queue pktio locks
  linux-generic: pktio: implement multiqueue API (part I)
  linux-generic: pktio: imlement multiqueue API (part II)
  linux-generic: pktio: multiqueue support in netmap
  example: generator: use multiqueue API

 example/generator/odp_generator.c  |  63 +++-
 include/odp/api/config.h   |   7 +-
 include/odp/api/packet_io.h| 118 ++
 .../linux-generic/include/odp_packet_io_internal.h |  19 +-
 platform/linux-generic/include/odp_packet_netmap.h |   6 +-
 .../linux-generic/include/odp_queue_internal.h |   2 +
 .../linux-generic/include/odp_schedule_internal.h  |   1 +
 platform/linux-generic/odp_packet_io.c | 405 +
 platform/linux-generic/odp_schedule.c  |  26 +-
 platform/linux-generic/pktio/netmap.c  | 218 +--
 10 files changed, 725 insertions(+), 140 deletions(-)

-- 
2.5.3

___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp