On 26 October 2015 at 16:28, Nikita Kalyazin <[email protected]> wrote:
> Hi Alex, > > > I understand your reasoning but ODP Packet I/O (pktio) is _not_ (yet?) a > NIC > > abstraction. The fact is that in current ODP design ingress queues are > > "connected" to the pktio device by the configuration of classifier or as > > default/error CoS queues. Classifier can be changed dynamically, e.g, > adding or > > removing CoSes or queues without stopping/re-configuring the pktio > device. I > > don't see in DPDK such features in PMD, all queues are configured and > setup at > > initialization time and they don't change thereafter. > Sorry, could you clarify what do you mean by "ODP pktio is not a NIC > abstraction"? I hear this second time, but not sure I fully understand the > meaning. [Alex] By NIC I understand, for example, an Intel e1000 PCI NIC or more generally, all devices supported by DPDK PMD . You already mentioned that your proposed changes are similar with DPDK functions to receive from a given NIC HW queue. > Does it mean, a pktio is not just a pipe to network with send/recv > operations defined on it, but also a bunch of higher level mechanisms like > cls/scheduling/shaping/dropping? > > -- > > Best regards, > > Nikita Kalyazin, > [email protected] > > Software Engineer > Virtualization Group > Samsung R&D Institute Russia > Tel: +7 (495) 797-25-00 #3816 > Tel: +7 (495) 797-25-03 > Office #1501, 12-1, Dvintsev str., > Moscow, 127018, Russia > > On Fri, Oct 23, 2015 at 10:26:02AM +0300, Alexandru Badicioiu wrote: > > > > > > On 22 October 2015 at 18:06, Nikita Kalyazin <[1][email protected]> > wrote: > > > > Hi, > > > > > [Alex] ODP queues are neither software nor hardware by definition, > each > > > implementation is free to implement them as they see fit. Also > PacketI/O > > > abstraction is not an abstraction for a NIC device. The ODP way to > use > > multiple > > > queues for ingress is through the classifier. There was a former > proposal > > of > > > enabling RSS at pktio level but it was not accepted. I also > noticed that > > this > > > patch tries to introduce multiple default queues which is kind of > > confusing - > > > default traffic is all traffic which does not satisfy classifier > > criteria. For > > > output, multiple queues are used by the means of TM api. > > As I understand, ODP implies three options of ingress processing: > > - "recv" (poll pktio with raw recv() method); > > - "poll" (poll poll pktio via default input queue associated with > the > > pktio); > > - "sched" (call schedule() which makes use of classifier and queue > > priorities). > > Looks like each ODP implementation should support all the three > options. > > > > For "sched" option, classification may be implemented in hardware, > and ODP > > queues, in their turn, can also be implemented in hardware. The > hardware > > would > > distribute packets to its hardware queues according to PMRs or L2/L3 > QoS. > > > > For "recv" option, the only way possible to access NIC's hardware > queues > > (if any), is to call recv() on a specified queue explicitly. So > there > > should be > > an API for it. In my patch I proposed such APIs > (odp_pktio_recv_queue()). > > I admit, this might be not the best API name possible. However I > don't see > > any > > other way to access NIC's hardware queues at "recv" level. We can > add > > hw_queue_id parameter to existing odp_pktio_recv() alternatively > (same way > > as > > DPDK does with rte_eth_rx_burst()). In case of using such an > explicit > > hardware > > queue polling, RSS should be configured somehow to distribute packets > > across them. > > > > For "poll" option, it's not completely clear to me how it should > relate to > > NIC's > > hardware queues. It looks like classification is not involved here, > and > > this > > option isn't really different from "recv". If this is correct, > presence of > > multiple default input queues is reasonable, since they're directly > mapped > > to > > NIC's hardware queues, and odp_queue_deq() simply extracts a packet > from > > the > > appropriate HW queue. > > [Alex] > > > > Hi Nikita, > > I understand your reasoning but ODP Packet I/O (pktio) is _not_ (yet?) a > NIC > > abstraction. The fact is that in current ODP design ingress queues are > > "connected" to the pktio device by the configuration of classifier or as > > default/error CoS queues. Classifier can be changed dynamically, e.g, > adding or > > removing CoSes or queues without stopping/re-configuring the pktio > device. I > > don't see in DPDK such features in PMD, all queues are configured and > setup at > > initialization time and they don't change thereafter. > > > > > > -- > > > > Best regards, > > > > Nikita Kalyazin, > > [2][email protected] > > > > Software Engineer > > Virtualization Group > > Samsung R&D Institute Russia > > Tel: [3]+7 (495) 797-25-00 #3816 > > Tel: [4]+7 (495) 797-25-03 > > Office #1501, 12-1, Dvintsev str., > > Moscow, 127018, Russia > > > > On Mon, Oct 19, 2015 at 08:17:17PM +0300, Alexandru Badicioiu wrote: > > > > > > > > > On 19 October 2015 at 19:16, Nikita Kalyazin <[1][5] > > [email protected]> wrote: > > > > > > Hi Stuart, > > > > > > > > > Thanks for your feedback. > > > > > > > One thing that is missing here is a method for the > application to > > > > configure how packets are distributed, i.e. which fields in > the > > packet > > > > are used when calculating the flow hash (not necessarily the > hash > > > > algorithm itself, that can be implementation defined). > > > > > > > > For example with the netmap implementation here, if you have > a > > > > compatible interface, I assume the user can control > distribution to > > RX > > > > queues using something like; > > > > > > > > ethtool --config-ntuple rx-flow-hash esp4 > > > > > > > > But it would be better if this were configurable via the ODP > API. > > Petri > > > > had a suggestion a while back related to this to add a > structure to > > the > > > > odp_pktio_params_t; > > > > > > > > enum odp_pktio_input_hash { > > > > /** No specific fields defined */ > > > > ODP_PKTIN_HASH_NONE = 0, > > > > /** IPv4/v6 addresses */ > > > > ODP_PKTIN_HASH_IP, > > > > /** UDP ports and IPv4/v6 addresses */ > > > > ODP_PKTIN_HASH_UDP_IP, > > > > /** TCP ports and IPv4/v6 addresses */ > > > > ODP_PKTIN_HASH_TCP_IP > > > > }; > > > Thanks, I plan to think about it. > > > Btw, NICs provide wide variety of settings regarding > distribution of > > > packets across the queues. Is it supposed to extend the > proposed > > > structure to support more flexible configuration? > > > > > > > How about: > > > > > > > > int odp_pktio_inq_create(odp_pktio_t pktio, const char *name, > > > > const odp_queue_param_t *param, > > > > odp_queue_t queues[], int num); > > > > > > > > Which would return the number of queues created, and ensure > that > > all > > > > queues had the same parameters. Or, maybe it's time to start > using > > > > odp_queue_group_t, which we have always intended to be a > group of > > queues > > > > that all have the same properties. > > > Good point. I'll consider the cleaner approach. > > > > > > > I'm not keen on the odp_pktio_recv/send_queue() APIs as the > queues > > > > referred to there are not ODP queues so it makes things a > bit less > > clear, > > > > and in general we should be moving away from using the > direct _send > > /_recv > > > > APIs. If we're explicitly using queues then we should use > the queue > > and/ > > > or > > > > scheduler APIs. > > > Right, these API names look a bit confusing, because same word > is > > used to > > > refer to both hardware NIC queues and ODP (software) queues. > > > > > > [Alex] ODP queues are neither software nor hardware by definition, > each > > > implementation is free to implement them as they see fit. Also > PacketI/O > > > abstraction is not an abstraction for a NIC device. The ODP way to > use > > multiple > > > queues for ingress is through the classifier. There was a former > proposal > > of > > > enabling RSS at pktio level but it was not accepted. I also > noticed that > > this > > > patch tries to introduce multiple default queues which is kind of > > confusing - > > > default traffic is all traffic which does not satisfy classifier > > criteria. For > > > output, multiple queues are used by the means of TM api. > > > > > > However, I think it would be useful to preserve explicit > send/recv > > APIs, > > > since in many cases they might appear more efficient (odp-ovs > is an > > example > > > of such applications), and rename those to make them more > obvious. > > > > > > > Are you able to join the meeting on Tuesday to discuss? > > > > > > > > [2][6]http://www.opendataplane.org/meetings/ > > > Thanks, I'm going to join the meeting. > > > > > > -- > > > > > > Best regards, > > > > > > Nikita Kalyazin, > > > [3][7][email protected] > > > > > > Software Engineer > > > Virtualization Group > > > Samsung R&D Institute Russia > > > Tel: [4][8]+7 (495) 797-25-00 #3816 > > > Tel: [5][9]+7 (495) 797-25-03 > > > Office #1501, 12-1, Dvintsev str., > > > Moscow, 127018, Russia > > > > > > On Fri, Oct 16, 2015 at 07:15:43PM +0100, Stuart Haslam wrote: > > > > On Wed, Oct 14, 2015 at 02:32:37PM +0300, Nikita Kalyazin > wrote: > > > > > This series is a proposal of multiqueue API for ODP. > > > > > Modern network adapters (both physical and virtual) support > > multiple > > > > > hardware queues. Assigning separate CPU core for > processing of > > each > > > > > queue allows to scale throughput (in best case - linearly). > > > > > > > > > > > > > Thanks for the contribution, we do need something like this, > in > > > > particular hash based distribution has been on the todo list > for a > > > > little while. I've not reviewed the implementation in detail > but > > have > > > > focused on the API changes specifically and have a few > initial > > thoughts > > > > below. > > > > > > > > One thing that is missing here is a method for the > application to > > > > configure how packets are distributed, i.e. which fields in > the > > packet > > > > are used when calculating the flow hash (not necessarily the > hash > > > > algorithm itself, that can be implementation defined). > > > > > > > > For example with the netmap implementation here, if you have > a > > > > compatible interface, I assume the user can control > distribution to > > RX > > > > queues using something like; > > > > > > > > ethtool --config-ntuple rx-flow-hash esp4 > > > > > > > > But it would be better if this were configurable via the ODP > API. > > Petri > > > > had a suggestion a while back related to this to add a > structure to > > the > > > > odp_pktio_params_t; > > > > > > > > enum odp_pktio_input_hash { > > > > /** No specific fields defined */ > > > > ODP_PKTIN_HASH_NONE = 0, > > > > /** IPv4/v6 addresses */ > > > > ODP_PKTIN_HASH_IP, > > > > /** UDP ports and IPv4/v6 addresses */ > > > > ODP_PKTIN_HASH_UDP_IP, > > > > /** TCP ports and IPv4/v6 addresses */ > > > > ODP_PKTIN_HASH_TCP_IP > > > > }; > > > > > > > > As for RX queue creation, rather than proposed sequence that > looks > > like > > > > this: > > > > > > > > odp_pktio_max_num_queues() > > > > odp_pktio_configure() > > > > foreach inq > > > > odp_queue_create() > > > > > > > > How about: > > > > > > > > int odp_pktio_inq_create(odp_pktio_t pktio, const char *name, > > > > const odp_queue_param_t *param, > > > > odp_queue_t queues[], int num); > > > > > > > > Which would return the number of queues created, and ensure > that > > all > > > > queues had the same parameters. Or, maybe it's time to start > using > > > > odp_queue_group_t, which we have always intended to be a > group of > > queues > > > > that all have the same properties. > > > > > > > > I'm not keen on the odp_pktio_recv/send_queue() APIs as the > queues > > > > referred to there are not ODP queues so it makes things a > bit less > > clear, > > > > and in general we should be moving away from using the > direct _send > > /_recv > > > > APIs. If we're explicitly using queues then we should use > the queue > > and/ > > > or > > > > scheduler APIs. > > > > > > > > Are you able to join the meeting on Tuesday to discuss? > > > > > > > > [6][10]http://www.opendataplane.org/meetings/ > > > > > > > > > This series should be considered a prototype. > > > > > This implementation of multiqueue API is not complete (not > all > > the > > > > > code has been migrated to multiqueue API). Existing > packet IO > > API has > > > > > been left untouched and reimplemented using mutliqueue API > (queue > > 0 is > > > > > always used in such cases). The existing API is supposed > to be > > > > > removed in the future, as soon as all the remaining code > migrates > > to > > > > > the multiqueue API. > > > > > Since single lock for all the queues makes all pktio Rx > and Tx > > calls > > > > > sequential, per queue locks have been added. > > > > > For the purpose of evaluation, multiqueue support has been > added > > to > > > > > netmap pktio. > > > > > > > > > > Patch 1 adds multiqueue prototypes of pktio API functions. > > > > > Patches 2-4 contain common implementation of the > multiqueue API. > > > > > Patch 5 adds multiqueue support in netmap pktio. > > > > > Patch 6 moves odp_generator to multiqueue API. > > > > > > > > > > In our tests (odp_generator sends packets, odp_generator > receives > > > > > packets), the following results have been obtained: > > > > > +----------+------------------------+-------+ > > > > > | # queues | kpps (64-byte packets) | boost | > > > > > +----------+------------------------+-------+ > > > > > | 1 | 853 | 1.00x | > > > > > | 2 | 1404 | 1.64x | > > > > > | 4 | 2374 | 2.78x | > > > > > | 8 | 4345 | 5.09x | > > > > > +----------+------------------------+-------+ > > > > > > > > > > Nikita Kalyazin (6): > > > > > api: pktio: add multiqueue API > > > > > linux-generic: pktio: add per queue pktio locks > > > > > linux-generic: pktio: implement multiqueue API (part I) > > > > > linux-generic: pktio: imlement multiqueue API (part II) > > > > > linux-generic: pktio: multiqueue support in netmap > > > > > example: generator: use multiqueue API > > > > > > > > > > example/generator/odp_generator.c | 63 > +++- > > > > > include/odp/api/config.h | 7 > +- > > > > > include/odp/api/packet_io.h | 118 > ++++++ > > > > > .../linux-generic/include/odp_packet_io_internal.h | 19 > +- > > > > > platform/linux-generic/include/odp_packet_netmap.h | 6 > +- > > > > > .../linux-generic/include/odp_queue_internal.h | 2 + > > > > > .../linux-generic/include/odp_schedule_internal.h | 1 + > > > > > platform/linux-generic/odp_packet_io.c | 405 > > > +++++++++++++++++---- > > > > > platform/linux-generic/odp_schedule.c | 26 > +- > > > > > platform/linux-generic/pktio/netmap.c | 218 > > +++++++++-- > > > > > 10 files changed, 725 insertions(+), 140 deletions(-) > > > > > > > > > > -- > > > > > 2.5.3 > > > > > > > > _______________________________________________ > > > lng-odp mailing list > > > [7][11][email protected] > > > [8][12]https://lists.linaro.org/mailman/listinfo/lng-odp > > > > > > > > > > > > References: > > > > > > [1] mailto:[13][email protected] > > > [2] [14]http://www.opendataplane.org/meetings/ > > > [3] mailto:[15][email protected] > > > [4] tel:%2B7%20%28495%29%20797-25-00%20%233816 > > > [5] tel:%2B7%20%28495%29%20797-25-03 > > > [6] [16]http://www.opendataplane.org/meetings/ > > > [7] mailto:[17][email protected] > > > [8] [18]https://lists.linaro.org/mailman/listinfo/lng-odp > > > > > > > > References: > > > > [1] mailto:[email protected] > > [2] mailto:[email protected] > > [3] tel:%2B7%20%28495%29%20797-25-00%20%233816 > > [4] tel:%2B7%20%28495%29%20797-25-03 > > [5] mailto:[email protected] > > [6] http://www.opendataplane.org/meetings/ > > [7] mailto:[email protected] > > [8] tel:%2B7%20%28495%29%20797-25-00%20%233816 > > [9] tel:%2B7%20%28495%29%20797-25-03 > > [10] http://www.opendataplane.org/meetings/ > > [11] mailto:[email protected] > > [12] https://lists.linaro.org/mailman/listinfo/lng-odp > > [13] mailto:[email protected] > > [14] http://www.opendataplane.org/meetings/ > > [15] mailto:[email protected] > > [16] http://www.opendataplane.org/meetings/ > > [17] mailto:[email protected] > > [18] https://lists.linaro.org/mailman/listinfo/lng-odp >
_______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
