Hi, > [Alex] ODP queues are neither software nor hardware by definition, each > implementation is free to implement them as they see fit. Also PacketI/O > abstraction is not an abstraction for a NIC device. The ODP way to use > multiple > queues for ingress is through the classifier. There was a former proposal of > enabling RSS at pktio level but it was not accepted. I also noticed that this > patch tries to introduce multiple default queues which is kind of confusing - > default traffic is all traffic which does not satisfy classifier criteria. For > output, multiple queues are used by the means of TM api. As I understand, ODP implies three options of ingress processing: - "recv" (poll pktio with raw recv() method); - "poll" (poll poll pktio via default input queue associated with the pktio); - "sched" (call schedule() which makes use of classifier and queue priorities). Looks like each ODP implementation should support all the three options.
For "sched" option, classification may be implemented in hardware, and ODP queues, in their turn, can also be implemented in hardware. The hardware would distribute packets to its hardware queues according to PMRs or L2/L3 QoS. For "recv" option, the only way possible to access NIC's hardware queues (if any), is to call recv() on a specified queue explicitly. So there should be an API for it. In my patch I proposed such APIs (odp_pktio_recv_queue()). I admit, this might be not the best API name possible. However I don't see any other way to access NIC's hardware queues at "recv" level. We can add hw_queue_id parameter to existing odp_pktio_recv() alternatively (same way as DPDK does with rte_eth_rx_burst()). In case of using such an explicit hardware queue polling, RSS should be configured somehow to distribute packets across them. For "poll" option, it's not completely clear to me how it should relate to NIC's hardware queues. It looks like classification is not involved here, and this option isn't really different from "recv". If this is correct, presence of multiple default input queues is reasonable, since they're directly mapped to NIC's hardware queues, and odp_queue_deq() simply extracts a packet from the appropriate HW queue. -- Best regards, Nikita Kalyazin, [email protected] Software Engineer Virtualization Group Samsung R&D Institute Russia Tel: +7 (495) 797-25-00 #3816 Tel: +7 (495) 797-25-03 Office #1501, 12-1, Dvintsev str., Moscow, 127018, Russia On Mon, Oct 19, 2015 at 08:17:17PM +0300, Alexandru Badicioiu wrote: > > > On 19 October 2015 at 19:16, Nikita Kalyazin <[1][email protected]> > wrote: > > Hi Stuart, > > > Thanks for your feedback. > > > One thing that is missing here is a method for the application to > > configure how packets are distributed, i.e. which fields in the packet > > are used when calculating the flow hash (not necessarily the hash > > algorithm itself, that can be implementation defined). > > > > For example with the netmap implementation here, if you have a > > compatible interface, I assume the user can control distribution to RX > > queues using something like; > > > > ethtool --config-ntuple rx-flow-hash esp4 > > > > But it would be better if this were configurable via the ODP API. Petri > > had a suggestion a while back related to this to add a structure to the > > odp_pktio_params_t; > > > > enum odp_pktio_input_hash { > > /** No specific fields defined */ > > ODP_PKTIN_HASH_NONE = 0, > > /** IPv4/v6 addresses */ > > ODP_PKTIN_HASH_IP, > > /** UDP ports and IPv4/v6 addresses */ > > ODP_PKTIN_HASH_UDP_IP, > > /** TCP ports and IPv4/v6 addresses */ > > ODP_PKTIN_HASH_TCP_IP > > }; > Thanks, I plan to think about it. > Btw, NICs provide wide variety of settings regarding distribution of > packets across the queues. Is it supposed to extend the proposed > structure to support more flexible configuration? > > > How about: > > > > int odp_pktio_inq_create(odp_pktio_t pktio, const char *name, > > const odp_queue_param_t *param, > > odp_queue_t queues[], int num); > > > > Which would return the number of queues created, and ensure that all > > queues had the same parameters. Or, maybe it's time to start using > > odp_queue_group_t, which we have always intended to be a group of queues > > that all have the same properties. > Good point. I'll consider the cleaner approach. > > > I'm not keen on the odp_pktio_recv/send_queue() APIs as the queues > > referred to there are not ODP queues so it makes things a bit less > clear, > > and in general we should be moving away from using the direct > _send/_recv > > APIs. If we're explicitly using queues then we should use the queue and/ > or > > scheduler APIs. > Right, these API names look a bit confusing, because same word is used to > refer to both hardware NIC queues and ODP (software) queues. > > [Alex] ODP queues are neither software nor hardware by definition, each > implementation is free to implement them as they see fit. Also PacketI/O > abstraction is not an abstraction for a NIC device. The ODP way to use > multiple > queues for ingress is through the classifier. There was a former proposal of > enabling RSS at pktio level but it was not accepted. I also noticed that this > patch tries to introduce multiple default queues which is kind of confusing - > default traffic is all traffic which does not satisfy classifier criteria. For > output, multiple queues are used by the means of TM api. > > However, I think it would be useful to preserve explicit send/recv APIs, > since in many cases they might appear more efficient (odp-ovs is an > example > of such applications), and rename those to make them more obvious. > > > Are you able to join the meeting on Tuesday to discuss? > > > > [2]http://www.opendataplane.org/meetings/ > Thanks, I'm going to join the meeting. > > -- > > Best regards, > > Nikita Kalyazin, > [3][email protected] > > Software Engineer > Virtualization Group > Samsung R&D Institute Russia > Tel: [4]+7 (495) 797-25-00 #3816 > Tel: [5]+7 (495) 797-25-03 > Office #1501, 12-1, Dvintsev str., > Moscow, 127018, Russia > > On Fri, Oct 16, 2015 at 07:15:43PM +0100, Stuart Haslam wrote: > > On Wed, Oct 14, 2015 at 02:32:37PM +0300, Nikita Kalyazin wrote: > > > This series is a proposal of multiqueue API for ODP. > > > Modern network adapters (both physical and virtual) support multiple > > > hardware queues. Assigning separate CPU core for processing of each > > > queue allows to scale throughput (in best case - linearly). > > > > > > > Thanks for the contribution, we do need something like this, in > > particular hash based distribution has been on the todo list for a > > little while. I've not reviewed the implementation in detail but have > > focused on the API changes specifically and have a few initial thoughts > > below. > > > > One thing that is missing here is a method for the application to > > configure how packets are distributed, i.e. which fields in the packet > > are used when calculating the flow hash (not necessarily the hash > > algorithm itself, that can be implementation defined). > > > > For example with the netmap implementation here, if you have a > > compatible interface, I assume the user can control distribution to RX > > queues using something like; > > > > ethtool --config-ntuple rx-flow-hash esp4 > > > > But it would be better if this were configurable via the ODP API. Petri > > had a suggestion a while back related to this to add a structure to the > > odp_pktio_params_t; > > > > enum odp_pktio_input_hash { > > /** No specific fields defined */ > > ODP_PKTIN_HASH_NONE = 0, > > /** IPv4/v6 addresses */ > > ODP_PKTIN_HASH_IP, > > /** UDP ports and IPv4/v6 addresses */ > > ODP_PKTIN_HASH_UDP_IP, > > /** TCP ports and IPv4/v6 addresses */ > > ODP_PKTIN_HASH_TCP_IP > > }; > > > > As for RX queue creation, rather than proposed sequence that looks like > > this: > > > > odp_pktio_max_num_queues() > > odp_pktio_configure() > > foreach inq > > odp_queue_create() > > > > How about: > > > > int odp_pktio_inq_create(odp_pktio_t pktio, const char *name, > > const odp_queue_param_t *param, > > odp_queue_t queues[], int num); > > > > Which would return the number of queues created, and ensure that all > > queues had the same parameters. Or, maybe it's time to start using > > odp_queue_group_t, which we have always intended to be a group of queues > > that all have the same properties. > > > > I'm not keen on the odp_pktio_recv/send_queue() APIs as the queues > > referred to there are not ODP queues so it makes things a bit less > clear, > > and in general we should be moving away from using the direct > _send/_recv > > APIs. If we're explicitly using queues then we should use the queue and/ > or > > scheduler APIs. > > > > Are you able to join the meeting on Tuesday to discuss? > > > > [6]http://www.opendataplane.org/meetings/ > > > > > This series should be considered a prototype. > > > This implementation of multiqueue API is not complete (not all the > > > code has been migrated to multiqueue API). Existing packet IO API has > > > been left untouched and reimplemented using mutliqueue API (queue 0 is > > > always used in such cases). The existing API is supposed to be > > > removed in the future, as soon as all the remaining code migrates to > > > the multiqueue API. > > > Since single lock for all the queues makes all pktio Rx and Tx calls > > > sequential, per queue locks have been added. > > > For the purpose of evaluation, multiqueue support has been added to > > > netmap pktio. > > > > > > Patch 1 adds multiqueue prototypes of pktio API functions. > > > Patches 2-4 contain common implementation of the multiqueue API. > > > Patch 5 adds multiqueue support in netmap pktio. > > > Patch 6 moves odp_generator to multiqueue API. > > > > > > In our tests (odp_generator sends packets, odp_generator receives > > > packets), the following results have been obtained: > > > +----------+------------------------+-------+ > > > | # queues | kpps (64-byte packets) | boost | > > > +----------+------------------------+-------+ > > > | 1 | 853 | 1.00x | > > > | 2 | 1404 | 1.64x | > > > | 4 | 2374 | 2.78x | > > > | 8 | 4345 | 5.09x | > > > +----------+------------------------+-------+ > > > > > > Nikita Kalyazin (6): > > > api: pktio: add multiqueue API > > > linux-generic: pktio: add per queue pktio locks > > > linux-generic: pktio: implement multiqueue API (part I) > > > linux-generic: pktio: imlement multiqueue API (part II) > > > linux-generic: pktio: multiqueue support in netmap > > > example: generator: use multiqueue API > > > > > > example/generator/odp_generator.c | 63 +++- > > > include/odp/api/config.h | 7 +- > > > include/odp/api/packet_io.h | 118 ++++++ > > > .../linux-generic/include/odp_packet_io_internal.h | 19 +- > > > platform/linux-generic/include/odp_packet_netmap.h | 6 +- > > > .../linux-generic/include/odp_queue_internal.h | 2 + > > > .../linux-generic/include/odp_schedule_internal.h | 1 + > > > platform/linux-generic/odp_packet_io.c | 405 > +++++++++++++++++---- > > > platform/linux-generic/odp_schedule.c | 26 +- > > > platform/linux-generic/pktio/netmap.c | 218 +++++++++-- > > > 10 files changed, 725 insertions(+), 140 deletions(-) > > > > > > -- > > > 2.5.3 > > > > _______________________________________________ > lng-odp mailing list > [7][email protected] > [8]https://lists.linaro.org/mailman/listinfo/lng-odp > > > > References: > > [1] mailto:[email protected] > [2] http://www.opendataplane.org/meetings/ > [3] mailto:[email protected] > [4] tel:%2B7%20%28495%29%20797-25-00%20%233816 > [5] tel:%2B7%20%28495%29%20797-25-03 > [6] http://www.opendataplane.org/meetings/ > [7] mailto:[email protected] > [8] https://lists.linaro.org/mailman/listinfo/lng-odp _______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
