Lock-free is a reference to how an implementation might choose to implement an ODP queue. Better to speak in terms of use cases and let the implementation decide how to most efficiently realize them. As currently specified, all ODP queues are by default designed to be multi-producer / multi-consumer, and that may not always be needed.
DPDK has various attributes like single/multiple consumer and single/multiple producer. Those are reasonable advisory attributes to pass to queue create to permit the underlying implementation to optimize queues based on their intended usage model. On Mon, Nov 9, 2015 at 5:14 AM, Savolainen, Petri (Nokia - FI/Espoo) < [email protected]> wrote: > We need a clean way to create lock-free N input and output queues for > packet IO. Lock-free here means that application use one rx/tx pair only > from one thread. This is what many server applications need (== those > currently using DPDK) – they don’t necessarily need classification, > scheduler, TM, timer or crypto (only packet IO). > > > > See under the entire setup needed for the use case with this API. > > > > -Petri > > > > > > odp_pktio_t pktio; > > odp_pktio_capability_t capa; > > odp_pktio_input_queue_param_t input_queue_param; > > odp_pktio_output_queue_param_t output_queue_param; > > odp_pktio_param_t pktio_param; > > unsigned num_worker = 2; > > > > // a worker per input/output queue > > odp_bool_t lock_free = 1; > > > > odp_pktio_param_init(&pktio_param); > > pktio_param.in_mode = ODP_PKTIN_MODE_RECV; > > pktio_param.out_mode = ODP_PKTIN_MODE_SEND; > > > > pktio = odp_pktio_open(“eth0”, pool, &pktio_param); > > > > odp_pktio_capability(pktio, &capa); > > > > if(capa.max_input_queues < num_worker || > > capa.max_output_queues < num_worker) { > > // use less workers … OR > > // share a queue between multiple workers, which is done here > > lock_free = 0; > > } > > > > odp_pktio_input_queue_param_init(&input_queue_param); > > input_queue_param.lock_free = lock_free; > > input_queue_param.hash_proto = ODP_PKTIN_HASH_IPV4_UDP; > > input_queue_param.num_queues = num_worker; > > > > odp_pktio_output_queue_param_init(&output_queue_param); > > output_queue_param.lock_free = lock_free; > > output_queue_param.num_queues = num_worker; > > > > odp_pktio_input_queues_config(pktio, &input_queue_param); > > odp_pktio_output_queues_config(pktio, &output_queue_param); > > > > odp_pktio_start(pktio); > > > > // Thread using queue index #0 > > odp_pktio_recv_queue(pktio, 0, packets, num); > > > > // Thread using queue index #1 > > odp_pktio_recv_queue(pktio, 1, packets, num); > > > > // Thread using queue index #0 > > odp_pktio_send_queue(pktio, 0, packets, num); > > > > // Thread using queue index #1 > > odp_pktio_send_queue(pktio, 1, packets, num); > > > > > > > > > > *From:* EXT Alexandru Badicioiu [mailto:[email protected]] > *Sent:* Monday, November 09, 2015 12:33 PM > *To:* Bala Manoharan > *Cc:* Bill Fischofer; LNG ODP Mailman List; Savolainen, Petri (Nokia - > FI/Espoo) > > *Subject:* Re: [lng-odp] [API-NEXT PATCH 2/5] api: pktio: added multiple > pktio input queues > > > > > > > > On 9 November 2015 at 12:26, Bala Manoharan <[email protected]> > wrote: > > The existing Classification infra structure supports this behaviour of > hashing after classification we have CoS API to allocate the packet to > a queue group. The only missing API definition is a mechanism to > configure header fields to be taken for hashing on CoS to distribute > the packets to the queues within a queue group. > > [Alex] No need for new definitions as we already have PMR terms. When > creating a PRM we may use the convention that val_sz 0 represents use for > hashing instead matching the val/mask pair. > > > Regards, > Bala > > > On 9 November 2015 at 15:17, Bill Fischofer <[email protected]> > wrote: > > Hashing was the idea behind queue groups. A "basic" PMR puts the output > on > > a designated queue. A "hashing" PMR distributes the results to the > > individual queues in a queue group. > > > > On Mon, Nov 9, 2015 at 3:43 AM, Alexandru Badicioiu > > <[email protected]> wrote: > >> > >> CoS concept can be easily extended to implement hashing instead of > >> classification. Instead of a single queue a CoS can have multiple > queues and > >> the packet fields used for hashing can be specified by the already > defined > >> PMR terms (enum odp_pmr_term) with a val_sz set to 0. Packets arriving > at > >> such a CoS will be spread on the CoS queues and the hash will be > computed on > >> the fields set by the PMRs/PMR set configured on the CoS. If the CoS is > the > >> default CoS and no other classification configuration exists this will > be > >> the default action executed on packets received from the pktio object. > >> I thought it was already decided that hashing functionality should be > >> integrated into classification API. > >> > >> Alex > >> > >> > >> On 9 November 2015 at 11:11, Savolainen, Petri (Nokia - FI/Espoo) > >> <[email protected]> wrote: > >>> > >>> > >>> > >>> > -----Original Message----- > >>> > From: EXT Bala Manoharan [mailto:[email protected]] > >>> > Sent: Friday, November 06, 2015 7:54 PM > >>> > To: Savolainen, Petri (Nokia - FI/Espoo) > >>> > Cc: LNG ODP Mailman List > >>> > Subject: Re: [lng-odp] [API-NEXT PATCH 2/5] api: pktio: added > multiple > >>> > pktio input queues > >>> > > >>> > Hi Petri, > >>> > > >>> > Why don't we add this hash parameter to CoS so that when the packet > >>> > arrives to a CoS it can be distributed based on the hash algorithm to > >>> > a number of queues. Also if a system does not support Classification > >>> > this hashing can be attached to the default CoS and in systems > >>> > supporting Classification this input hashing attached to the default > >>> > CoS will be applied when none of the PMR rules are matching the input > >>> > packet so the classification PMR rules will take a higher priority > >>> > than direct packet receive which I believe is the expected behavior. > >>> > >>> I'd like to keep the APIs modular (or a-la-carte). If user don't need > >>> classification (ability to pick a flow to a certain queue) and the > platform > >>> does not support it - it would be waste (and an entry barrier for ODP) > to > >>> force the user and the implementation to agree on hashing (load > spreading) > >>> via (full featured) classification API. > >>> > >>> You could think that this replaces the current single default queue > >>> calls: > >>> odp_pktio_inq_setdef(), odp_pktio_inq_getdef(), odp_pktio_inq_remdef(), > >>> odp_queue_t odp_pktio_outq_getdef() > >>> > >>> Those should be removed in next steps. Also I think the classification > >>> API needs also modification to notice multiple default input queues. > Or we > >>> can define that either hashing or classification can be used but not > both at > >>> the same time. The classification API can of course use the same hash > proto > >>> enum for its purposes. > >>> > >>> I actually had an "enable hashing vs. classification" boolean in > >>> odp_pktio_input_queue_param_t, but removed it before sending. > >>> > >>> Also one thing to consider is: how much classification is possible in > >>> direct packet receive mode. That mode uses indexes instead of > odp_queue_t, > >>> so that those are dedicated to incoming packets - instead of any event > type. > >>> > >>> > >>> > > >>> > Also regarding the APIs for attaching multiple output queues to pktio > >>> > interface I believe we should be able to get this behaviour using the > >>> > proposed TM APIs if not we should modify the TM API to support this > >>> > feature. > >>> > I think configuring a TM system with a single hierarchy level should > >>> > be enough to solve this issue. > >>> > >>> > >>> We have defined three pktio output modes: ODP_PKTOUT_MODE_SEND (direct > >>> output), ODP_PKTOUT_MODE_TM (output through TM), > ODP_PKTOUT_MODE_DISABLED > >>> (no output). The mode selection guides the implementation. E.g. if > >>> implementation does not have HW TM, it don't have to worry about it in > other > >>> two modes. Also TM output has different feature set to direct output > (e.g. > >>> queue fill rate feedback on output). > >>> > >>> Again, for performance multiple (potentially lock-free) queues are > >>> needed. TM functionality is not always needed. It would be waste to > force > >>> user and implementation to always use TM API, even when TM features > are not > >>> needed and are not provided (in HW). > >>> > >>> -Petri > >>> > >>> > >>> > > >>> > Regards, > >>> > Bala > >>> > > >>> > On 6 November 2015 at 18:03, Petri Savolainen > >>> > <[email protected]> wrote: > >>> > > Added input queue configuration parameters and functions > >>> > > to setup multiple input queue and hashing. Added also > >>> > > odp_pktio_input_queues to query the number of queues > >>> > > and queue handles. Direct receive does not use queue > >>> > > handles, but indexes. > >>> > > > >>> > > Signed-off-by: Petri Savolainen <[email protected]> > >>> > > --- > >>> > > include/odp/api/packet_io.h | 88 > >>> > +++++++++++++++++++++++++++++++++++++++++++++ > >>> > > 1 file changed, 88 insertions(+) > >>> > > > >>> > > diff --git a/include/odp/api/packet_io.h > >>> > b/include/odp/api/packet_io.h > >>> > > index 264fa75..bb0e67c 100644 > >>> > > --- a/include/odp/api/packet_io.h > >>> > > +++ b/include/odp/api/packet_io.h > >>> > > @@ -19,6 +19,7 @@ extern "C" { > >>> > > #endif > >>> > > > >>> > > #include <odp/api/packet_io_stats.h> > >>> > > +#include <odp/api/queue.h> > >>> > > > >>> > > /** @defgroup odp_packet_io ODP PACKET IO > >>> > > * Operations on a packet Input/Output interface. > >>> > > @@ -85,6 +86,43 @@ typedef enum odp_pktio_output_mode_t { > >>> > > } odp_pktio_output_mode_t; > >>> > > > >>> > > /** > >>> > > + * Packet input hash protocols > >>> > > + * > >>> > > + * The list of protocol header field combinations, which are > >>> > included into > >>> > > + * packet input hash calculation. > >>> > > + */ > >>> > > +typedef enum odp_pktin_hash_proto_t { > >>> > > + /** IPv4 addresses and UDP port numbers, non-fragmented > >>> > packets */ > >>> > > + ODP_PKTIN_HASH_IPV4_UDP = 0, > >>> > > + /** IPv4 addresses and TCP port numbers, non-fragmented > >>> > packets */ > >>> > > + ODP_PKTIN_HASH_IPV4_TCP > >>> > > +} odp_pktin_hash_proto_t; > >>> > > + > >>> > > +/** > >>> > > + * Packet input queue parameters > >>> > > + */ > >>> > > +typedef struct odp_pktio_input_queue_param_t { > >>> > > + /** Enable lock-free receive operation per queue > >>> > > + * 0: Receive is multi-thread safe, 1: Receive is > lock-free > >>> > */ > >>> > > + odp_bool_t lock_free; > >>> > > + > >>> > > + /** Select protocol fields used for hashing */ > >>> > > + odp_pktin_hash_proto_t hash_proto; > >>> > > + > >>> > > + /** Number of input queues to be created. More than one > input > >>> > queue > >>> > > + * require input hashing. Hash_proto is ignore when > >>> > num_queues is one. > >>> > > + * The value must be between 1 and interface capability. > >>> > Queue type is > >>> > > + * defined by the input mode. */ > >>> > > + unsigned num_queues; > >>> > > + > >>> > > + /** Queue parameters for creating input queues in > >>> > ODP_PKTIN_MODE_POLL > >>> > > + * or ODP_PKTIN_MODE_SCHED modes. Scheduler parameters are > >>> > considered > >>> > > + * only in ODP_PKTIN_MODE_SCHED mode. */ > >>> > > + odp_queue_param_t queue_param; > >>> > > + > >>> > > +} odp_pktio_input_queue_param_t; > >>> > > + > >>> > > +/** > >>> > > * Packet IO parameters > >>> > > * > >>> > > * In minimum, user must select input and output modes. Use 0 for > >>> > defaults. > >>> > > @@ -158,6 +196,47 @@ odp_pktio_t odp_pktio_open(const char *dev, > >>> > odp_pool_t pool, > >>> > > int odp_pktio_capability(odp_pktio_t pktio, odp_pktio_capability_t > >>> > *capa); > >>> > > > >>> > > /** > >>> > > + * Configure packet input queues > >>> > > + * > >>> > > + * Setup a number of packet input queues and configure those. The > >>> > maximum number > >>> > > + * of queues is platform dependent and can be queried with > >>> > > + * odp_pktio_capability(). Queue handles for input queues can be > >>> > requested with > >>> > > + * odp_pktio_input_queues() after this call. All requested queues > >>> > are setup on > >>> > > + * success, no queues are setup on failure. > >>> > > + * > >>> > > + * @param pktio Packet IO handle > >>> > > + * @param param Packet input queue configuration parameters > >>> > > + * > >>> > > + * @retval 0 on success > >>> > > + * @retval <0 on failure > >>> > > + * > >>> > > + * @see odp_pktio_capability(), odp_pktio_input_queues() > >>> > > + */ > >>> > > +int odp_pktio_input_queues_config(odp_pktio_t pktio, > >>> > > + const > odp_pktio_input_queue_param_t > >>> > *param); > >>> > > + > >>> > > +/** > >>> > > + * Packet input queues > >>> > > + * > >>> > > + * Returns the number of input queues configured for the > interface. > >>> > Outputs also > >>> > > + * up to 'num' queue handles created for ODP_PKTIN_MODE_POLL and > >>> > > + * ODP_PKTIN_MODE_SCHED modes. If return value is larger than > 'num', > >>> > there are > >>> > > + * more queues than the function was allowed to output handles. > >>> > > + * > >>> > > + * Depending on the input mode, packets (or events) from these > >>> > queues are > >>> > > + * received using odp_pktio_recv_queue(), odp_queue_deq(), > >>> > odp_schedule(), etc > >>> > > + * calls. > >>> > > + * > >>> > > + * @param pktio Packet IO handle > >>> > > + * @param[out] queues Points to an array of queue handles for > >>> > output > >>> > > + * @param num Maximum number of queue handles to output > >>> > > + * > >>> > > + * @return Number of packet input queues > >>> > > + * @retval <0 on failure > >>> > > + */ > >>> > > +int odp_pktio_input_queues(odp_pktio_t pktio, odp_queue_t > queues[], > >>> > int num); > >>> > > + > >>> > > +/** > >>> > > * Start packet receive and transmit > >>> > > * > >>> > > * Activate packet receive and transmit on a previously opened or > >>> > stopped > >>> > > @@ -411,6 +490,15 @@ uint64_t odp_pktio_to_u64(odp_pktio_t pktio); > >>> > > void odp_pktio_param_init(odp_pktio_param_t *param); > >>> > > > >>> > > /** > >>> > > + * Initialize packet input queue parameters > >>> > > + * > >>> > > + * Initialize an odp_pktio_input_queue_param_t to its default > >>> > values. > >>> > > + * > >>> > > + * @param param Input queue parameter structure to be > initialized > >>> > > + */ > >>> > > +void > odp_pktio_input_queue_param_init(odp_pktio_input_queue_param_t > >>> > *param); > >>> > > + > >>> > > +/** > >>> > > * Print pktio info to the console > >>> > > * > >>> > > * Print implementation-defined pktio debug information to the > >>> > console. > >>> > > -- > >>> > > 2.6.2 > >>> > > > >>> > > _______________________________________________ > >>> > > lng-odp mailing list > >>> > > [email protected] > >>> > > https://lists.linaro.org/mailman/listinfo/lng-odp > >>> _______________________________________________ > >>> lng-odp mailing list > >>> [email protected] > >>> https://lists.linaro.org/mailman/listinfo/lng-odp > >> > >> > >> > >> _______________________________________________ > >> lng-odp mailing list > >> [email protected] > >> https://lists.linaro.org/mailman/listinfo/lng-odp > >> > > > > > > _______________________________________________ > > lng-odp mailing list > > [email protected] > > https://lists.linaro.org/mailman/listinfo/lng-odp > > > > >
_______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
