Hi Alex, > I understand your reasoning but ODP Packet I/O (pktio) is _not_ (yet?) a NIC > abstraction. The fact is that in current ODP design ingress queues are > "connected" to the pktio device by the configuration of classifier or as > default/error CoS queues. Classifier can be changed dynamically, e.g, adding > or > removing CoSes or queues without stopping/re-configuring the pktio device. I > don't see in DPDK such features in PMD, all queues are configured and setup at > initialization time and they don't change thereafter. Sorry, could you clarify what do you mean by "ODP pktio is not a NIC abstraction"? I hear this second time, but not sure I fully understand the meaning. Does it mean, a pktio is not just a pipe to network with send/recv operations defined on it, but also a bunch of higher level mechanisms like cls/scheduling/shaping/dropping?
-- Best regards, Nikita Kalyazin, [email protected] Software Engineer Virtualization Group Samsung R&D Institute Russia Tel: +7 (495) 797-25-00 #3816 Tel: +7 (495) 797-25-03 Office #1501, 12-1, Dvintsev str., Moscow, 127018, Russia On Fri, Oct 23, 2015 at 10:26:02AM +0300, Alexandru Badicioiu wrote: > > > On 22 October 2015 at 18:06, Nikita Kalyazin <[1][email protected]> > wrote: > > Hi, > > > [Alex] ODP queues are neither software nor hardware by definition, each > > implementation is free to implement them as they see fit. Also PacketI/O > > abstraction is not an abstraction for a NIC device. The ODP way to use > multiple > > queues for ingress is through the classifier. There was a former > proposal > of > > enabling RSS at pktio level but it was not accepted. I also noticed that > this > > patch tries to introduce multiple default queues which is kind of > confusing - > > default traffic is all traffic which does not satisfy classifier > criteria. For > > output, multiple queues are used by the means of TM api. > As I understand, ODP implies three options of ingress processing: > - "recv" (poll pktio with raw recv() method); > - "poll" (poll poll pktio via default input queue associated with the > pktio); > - "sched" (call schedule() which makes use of classifier and queue > priorities). > Looks like each ODP implementation should support all the three options. > > For "sched" option, classification may be implemented in hardware, and ODP > queues, in their turn, can also be implemented in hardware. The hardware > would > distribute packets to its hardware queues according to PMRs or L2/L3 QoS. > > For "recv" option, the only way possible to access NIC's hardware queues > (if any), is to call recv() on a specified queue explicitly. So there > should be > an API for it. In my patch I proposed such APIs (odp_pktio_recv_queue()). > I admit, this might be not the best API name possible. However I don't > see > any > other way to access NIC's hardware queues at "recv" level. We can add > hw_queue_id parameter to existing odp_pktio_recv() alternatively (same way > as > DPDK does with rte_eth_rx_burst()). In case of using such an explicit > hardware > queue polling, RSS should be configured somehow to distribute packets > across them. > > For "poll" option, it's not completely clear to me how it should relate to > NIC's > hardware queues. It looks like classification is not involved here, and > this > option isn't really different from "recv". If this is correct, presence > of > multiple default input queues is reasonable, since they're directly mapped > to > NIC's hardware queues, and odp_queue_deq() simply extracts a packet from > the > appropriate HW queue. > [Alex] > > Hi Nikita, > I understand your reasoning but ODP Packet I/O (pktio) is _not_ (yet?) a NIC > abstraction. The fact is that in current ODP design ingress queues are > "connected" to the pktio device by the configuration of classifier or as > default/error CoS queues. Classifier can be changed dynamically, e.g, adding > or > removing CoSes or queues without stopping/re-configuring the pktio device. I > don't see in DPDK such features in PMD, all queues are configured and setup at > initialization time and they don't change thereafter. > > > -- > > Best regards, > > Nikita Kalyazin, > [2][email protected] > > Software Engineer > Virtualization Group > Samsung R&D Institute Russia > Tel: [3]+7 (495) 797-25-00 #3816 > Tel: [4]+7 (495) 797-25-03 > Office #1501, 12-1, Dvintsev str., > Moscow, 127018, Russia > > On Mon, Oct 19, 2015 at 08:17:17PM +0300, Alexandru Badicioiu wrote: > > > > > > On 19 October 2015 at 19:16, Nikita Kalyazin <[1][5] > [email protected]> wrote: > > > > Hi Stuart, > > > > > > Thanks for your feedback. > > > > > One thing that is missing here is a method for the application to > > > configure how packets are distributed, i.e. which fields in the > packet > > > are used when calculating the flow hash (not necessarily the hash > > > algorithm itself, that can be implementation defined). > > > > > > For example with the netmap implementation here, if you have a > > > compatible interface, I assume the user can control distribution > to > RX > > > queues using something like; > > > > > > ethtool --config-ntuple rx-flow-hash esp4 > > > > > > But it would be better if this were configurable via the ODP API. > Petri > > > had a suggestion a while back related to this to add a structure > to > the > > > odp_pktio_params_t; > > > > > > enum odp_pktio_input_hash { > > > /** No specific fields defined */ > > > ODP_PKTIN_HASH_NONE = 0, > > > /** IPv4/v6 addresses */ > > > ODP_PKTIN_HASH_IP, > > > /** UDP ports and IPv4/v6 addresses */ > > > ODP_PKTIN_HASH_UDP_IP, > > > /** TCP ports and IPv4/v6 addresses */ > > > ODP_PKTIN_HASH_TCP_IP > > > }; > > Thanks, I plan to think about it. > > Btw, NICs provide wide variety of settings regarding distribution of > > packets across the queues. Is it supposed to extend the proposed > > structure to support more flexible configuration? > > > > > How about: > > > > > > int odp_pktio_inq_create(odp_pktio_t pktio, const char *name, > > > const odp_queue_param_t *param, > > > odp_queue_t queues[], int num); > > > > > > Which would return the number of queues created, and ensure that > all > > > queues had the same parameters. Or, maybe it's time to start using > > > odp_queue_group_t, which we have always intended to be a group of > queues > > > that all have the same properties. > > Good point. I'll consider the cleaner approach. > > > > > I'm not keen on the odp_pktio_recv/send_queue() APIs as the queues > > > referred to there are not ODP queues so it makes things a bit less > clear, > > > and in general we should be moving away from using the direct > _send > /_recv > > > APIs. If we're explicitly using queues then we should use the > queue > and/ > > or > > > scheduler APIs. > > Right, these API names look a bit confusing, because same word is > used to > > refer to both hardware NIC queues and ODP (software) queues. > > > > [Alex] ODP queues are neither software nor hardware by definition, each > > implementation is free to implement them as they see fit. Also PacketI/O > > abstraction is not an abstraction for a NIC device. The ODP way to use > multiple > > queues for ingress is through the classifier. There was a former > proposal > of > > enabling RSS at pktio level but it was not accepted. I also noticed that > this > > patch tries to introduce multiple default queues which is kind of > confusing - > > default traffic is all traffic which does not satisfy classifier > criteria. For > > output, multiple queues are used by the means of TM api. > > > > However, I think it would be useful to preserve explicit send/recv > APIs, > > since in many cases they might appear more efficient (odp-ovs is an > example > > of such applications), and rename those to make them more obvious. > > > > > Are you able to join the meeting on Tuesday to discuss? > > > > > > [2][6]http://www.opendataplane.org/meetings/ > > Thanks, I'm going to join the meeting. > > > > -- > > > > Best regards, > > > > Nikita Kalyazin, > > [3][7][email protected] > > > > Software Engineer > > Virtualization Group > > Samsung R&D Institute Russia > > Tel: [4][8]+7 (495) 797-25-00 #3816 > > Tel: [5][9]+7 (495) 797-25-03 > > Office #1501, 12-1, Dvintsev str., > > Moscow, 127018, Russia > > > > On Fri, Oct 16, 2015 at 07:15:43PM +0100, Stuart Haslam wrote: > > > On Wed, Oct 14, 2015 at 02:32:37PM +0300, Nikita Kalyazin wrote: > > > > This series is a proposal of multiqueue API for ODP. > > > > Modern network adapters (both physical and virtual) support > multiple > > > > hardware queues. Assigning separate CPU core for processing of > each > > > > queue allows to scale throughput (in best case - linearly). > > > > > > > > > > Thanks for the contribution, we do need something like this, in > > > particular hash based distribution has been on the todo list for a > > > little while. I've not reviewed the implementation in detail but > have > > > focused on the API changes specifically and have a few initial > thoughts > > > below. > > > > > > One thing that is missing here is a method for the application to > > > configure how packets are distributed, i.e. which fields in the > packet > > > are used when calculating the flow hash (not necessarily the hash > > > algorithm itself, that can be implementation defined). > > > > > > For example with the netmap implementation here, if you have a > > > compatible interface, I assume the user can control distribution > to > RX > > > queues using something like; > > > > > > ethtool --config-ntuple rx-flow-hash esp4 > > > > > > But it would be better if this were configurable via the ODP API. > Petri > > > had a suggestion a while back related to this to add a structure > to > the > > > odp_pktio_params_t; > > > > > > enum odp_pktio_input_hash { > > > /** No specific fields defined */ > > > ODP_PKTIN_HASH_NONE = 0, > > > /** IPv4/v6 addresses */ > > > ODP_PKTIN_HASH_IP, > > > /** UDP ports and IPv4/v6 addresses */ > > > ODP_PKTIN_HASH_UDP_IP, > > > /** TCP ports and IPv4/v6 addresses */ > > > ODP_PKTIN_HASH_TCP_IP > > > }; > > > > > > As for RX queue creation, rather than proposed sequence that looks > like > > > this: > > > > > > odp_pktio_max_num_queues() > > > odp_pktio_configure() > > > foreach inq > > > odp_queue_create() > > > > > > How about: > > > > > > int odp_pktio_inq_create(odp_pktio_t pktio, const char *name, > > > const odp_queue_param_t *param, > > > odp_queue_t queues[], int num); > > > > > > Which would return the number of queues created, and ensure that > all > > > queues had the same parameters. Or, maybe it's time to start using > > > odp_queue_group_t, which we have always intended to be a group of > queues > > > that all have the same properties. > > > > > > I'm not keen on the odp_pktio_recv/send_queue() APIs as the queues > > > referred to there are not ODP queues so it makes things a bit less > clear, > > > and in general we should be moving away from using the direct > _send > /_recv > > > APIs. If we're explicitly using queues then we should use the > queue > and/ > > or > > > scheduler APIs. > > > > > > Are you able to join the meeting on Tuesday to discuss? > > > > > > [6][10]http://www.opendataplane.org/meetings/ > > > > > > > This series should be considered a prototype. > > > > This implementation of multiqueue API is not complete (not all > the > > > > code has been migrated to multiqueue API). Existing packet IO > API has > > > > been left untouched and reimplemented using mutliqueue API > (queue > 0 is > > > > always used in such cases). The existing API is supposed to be > > > > removed in the future, as soon as all the remaining code > migrates > to > > > > the multiqueue API. > > > > Since single lock for all the queues makes all pktio Rx and Tx > calls > > > > sequential, per queue locks have been added. > > > > For the purpose of evaluation, multiqueue support has been added > to > > > > netmap pktio. > > > > > > > > Patch 1 adds multiqueue prototypes of pktio API functions. > > > > Patches 2-4 contain common implementation of the multiqueue API. > > > > Patch 5 adds multiqueue support in netmap pktio. > > > > Patch 6 moves odp_generator to multiqueue API. > > > > > > > > In our tests (odp_generator sends packets, odp_generator > receives > > > > packets), the following results have been obtained: > > > > +----------+------------------------+-------+ > > > > | # queues | kpps (64-byte packets) | boost | > > > > +----------+------------------------+-------+ > > > > | 1 | 853 | 1.00x | > > > > | 2 | 1404 | 1.64x | > > > > | 4 | 2374 | 2.78x | > > > > | 8 | 4345 | 5.09x | > > > > +----------+------------------------+-------+ > > > > > > > > Nikita Kalyazin (6): > > > > api: pktio: add multiqueue API > > > > linux-generic: pktio: add per queue pktio locks > > > > linux-generic: pktio: implement multiqueue API (part I) > > > > linux-generic: pktio: imlement multiqueue API (part II) > > > > linux-generic: pktio: multiqueue support in netmap > > > > example: generator: use multiqueue API > > > > > > > > example/generator/odp_generator.c | 63 +++- > > > > include/odp/api/config.h | 7 +- > > > > include/odp/api/packet_io.h | 118 ++++++ > > > > .../linux-generic/include/odp_packet_io_internal.h | 19 +- > > > > platform/linux-generic/include/odp_packet_netmap.h | 6 +- > > > > .../linux-generic/include/odp_queue_internal.h | 2 + > > > > .../linux-generic/include/odp_schedule_internal.h | 1 + > > > > platform/linux-generic/odp_packet_io.c | 405 > > +++++++++++++++++---- > > > > platform/linux-generic/odp_schedule.c | 26 +- > > > > platform/linux-generic/pktio/netmap.c | 218 > +++++++++-- > > > > 10 files changed, 725 insertions(+), 140 deletions(-) > > > > > > > > -- > > > > 2.5.3 > > > > > > _______________________________________________ > > lng-odp mailing list > > [7][11][email protected] > > [8][12]https://lists.linaro.org/mailman/listinfo/lng-odp > > > > > > > > References: > > > > [1] mailto:[13][email protected] > > [2] [14]http://www.opendataplane.org/meetings/ > > [3] mailto:[15][email protected] > > [4] tel:%2B7%20%28495%29%20797-25-00%20%233816 > > [5] tel:%2B7%20%28495%29%20797-25-03 > > [6] [16]http://www.opendataplane.org/meetings/ > > [7] mailto:[17][email protected] > > [8] [18]https://lists.linaro.org/mailman/listinfo/lng-odp > > > > References: > > [1] mailto:[email protected] > [2] mailto:[email protected] > [3] tel:%2B7%20%28495%29%20797-25-00%20%233816 > [4] tel:%2B7%20%28495%29%20797-25-03 > [5] mailto:[email protected] > [6] http://www.opendataplane.org/meetings/ > [7] mailto:[email protected] > [8] tel:%2B7%20%28495%29%20797-25-00%20%233816 > [9] tel:%2B7%20%28495%29%20797-25-03 > [10] http://www.opendataplane.org/meetings/ > [11] mailto:[email protected] > [12] https://lists.linaro.org/mailman/listinfo/lng-odp > [13] mailto:[email protected] > [14] http://www.opendataplane.org/meetings/ > [15] mailto:[email protected] > [16] http://www.opendataplane.org/meetings/ > [17] mailto:[email protected] > [18] https://lists.linaro.org/mailman/listinfo/lng-odp _______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
