We need a clean way to create lock-free N input and output queues for packet
IO. Lock-free here means that application use one rx/tx pair only from one
thread. This is what many server applications need (== those currently using
DPDK) – they don’t necessarily need classification, scheduler, TM, timer or
crypto (only packet IO).
See under the entire setup needed for the use case with this API.
-Petri
odp_pktio_t pktio;
odp_pktio_capability_t capa;
odp_pktio_input_queue_param_t input_queue_param;
odp_pktio_output_queue_param_t output_queue_param;
odp_pktio_param_t pktio_param;
unsigned num_worker = 2;
// a worker per input/output queue
odp_bool_t lock_free = 1;
odp_pktio_param_init(&pktio_param);
pktio_param.in_mode = ODP_PKTIN_MODE_RECV;
pktio_param.out_mode = ODP_PKTIN_MODE_SEND;
pktio = odp_pktio_open(“eth0”, pool, &pktio_param);
odp_pktio_capability(pktio, &capa);
if(capa.max_input_queues < num_worker ||
capa.max_output_queues < num_worker) {
// use less workers … OR
// share a queue between multiple workers, which is done here
lock_free = 0;
}
odp_pktio_input_queue_param_init(&input_queue_param);
input_queue_param.lock_free = lock_free;
input_queue_param.hash_proto = ODP_PKTIN_HASH_IPV4_UDP;
input_queue_param.num_queues = num_worker;
odp_pktio_output_queue_param_init(&output_queue_param);
output_queue_param.lock_free = lock_free;
output_queue_param.num_queues = num_worker;
odp_pktio_input_queues_config(pktio, &input_queue_param);
odp_pktio_output_queues_config(pktio, &output_queue_param);
odp_pktio_start(pktio);
// Thread using queue index #0
odp_pktio_recv_queue(pktio, 0, packets, num);
// Thread using queue index #1
odp_pktio_recv_queue(pktio, 1, packets, num);
// Thread using queue index #0
odp_pktio_send_queue(pktio, 0, packets, num);
// Thread using queue index #1
odp_pktio_send_queue(pktio, 1, packets, num);
From: EXT Alexandru Badicioiu [mailto:[email protected]]
Sent: Monday, November 09, 2015 12:33 PM
To: Bala Manoharan
Cc: Bill Fischofer; LNG ODP Mailman List; Savolainen, Petri (Nokia - FI/Espoo)
Subject: Re: [lng-odp] [API-NEXT PATCH 2/5] api: pktio: added multiple pktio
input queues
On 9 November 2015 at 12:26, Bala Manoharan
<[email protected]<mailto:[email protected]>> wrote:
The existing Classification infra structure supports this behaviour of
hashing after classification we have CoS API to allocate the packet to
a queue group. The only missing API definition is a mechanism to
configure header fields to be taken for hashing on CoS to distribute
the packets to the queues within a queue group.
[Alex] No need for new definitions as we already have PMR terms. When creating
a PRM we may use the convention that val_sz 0 represents use for hashing
instead matching the val/mask pair.
Regards,
Bala
On 9 November 2015 at 15:17, Bill Fischofer
<[email protected]<mailto:[email protected]>> wrote:
> Hashing was the idea behind queue groups. A "basic" PMR puts the output on
> a designated queue. A "hashing" PMR distributes the results to the
> individual queues in a queue group.
>
> On Mon, Nov 9, 2015 at 3:43 AM, Alexandru Badicioiu
> <[email protected]<mailto:[email protected]>> wrote:
>>
>> CoS concept can be easily extended to implement hashing instead of
>> classification. Instead of a single queue a CoS can have multiple queues and
>> the packet fields used for hashing can be specified by the already defined
>> PMR terms (enum odp_pmr_term) with a val_sz set to 0. Packets arriving at
>> such a CoS will be spread on the CoS queues and the hash will be computed on
>> the fields set by the PMRs/PMR set configured on the CoS. If the CoS is the
>> default CoS and no other classification configuration exists this will be
>> the default action executed on packets received from the pktio object.
>> I thought it was already decided that hashing functionality should be
>> integrated into classification API.
>>
>> Alex
>>
>>
>> On 9 November 2015 at 11:11, Savolainen, Petri (Nokia - FI/Espoo)
>> <[email protected]<mailto:[email protected]>> wrote:
>>>
>>>
>>>
>>> > -----Original Message-----
>>> > From: EXT Bala Manoharan
>>> > [mailto:[email protected]<mailto:[email protected]>]
>>> > Sent: Friday, November 06, 2015 7:54 PM
>>> > To: Savolainen, Petri (Nokia - FI/Espoo)
>>> > Cc: LNG ODP Mailman List
>>> > Subject: Re: [lng-odp] [API-NEXT PATCH 2/5] api: pktio: added multiple
>>> > pktio input queues
>>> >
>>> > Hi Petri,
>>> >
>>> > Why don't we add this hash parameter to CoS so that when the packet
>>> > arrives to a CoS it can be distributed based on the hash algorithm to
>>> > a number of queues. Also if a system does not support Classification
>>> > this hashing can be attached to the default CoS and in systems
>>> > supporting Classification this input hashing attached to the default
>>> > CoS will be applied when none of the PMR rules are matching the input
>>> > packet so the classification PMR rules will take a higher priority
>>> > than direct packet receive which I believe is the expected behavior.
>>>
>>> I'd like to keep the APIs modular (or a-la-carte). If user don't need
>>> classification (ability to pick a flow to a certain queue) and the platform
>>> does not support it - it would be waste (and an entry barrier for ODP) to
>>> force the user and the implementation to agree on hashing (load spreading)
>>> via (full featured) classification API.
>>>
>>> You could think that this replaces the current single default queue
>>> calls:
>>> odp_pktio_inq_setdef(), odp_pktio_inq_getdef(), odp_pktio_inq_remdef(),
>>> odp_queue_t odp_pktio_outq_getdef()
>>>
>>> Those should be removed in next steps. Also I think the classification
>>> API needs also modification to notice multiple default input queues. Or we
>>> can define that either hashing or classification can be used but not both at
>>> the same time. The classification API can of course use the same hash proto
>>> enum for its purposes.
>>>
>>> I actually had an "enable hashing vs. classification" boolean in
>>> odp_pktio_input_queue_param_t, but removed it before sending.
>>>
>>> Also one thing to consider is: how much classification is possible in
>>> direct packet receive mode. That mode uses indexes instead of odp_queue_t,
>>> so that those are dedicated to incoming packets - instead of any event type.
>>>
>>>
>>> >
>>> > Also regarding the APIs for attaching multiple output queues to pktio
>>> > interface I believe we should be able to get this behaviour using the
>>> > proposed TM APIs if not we should modify the TM API to support this
>>> > feature.
>>> > I think configuring a TM system with a single hierarchy level should
>>> > be enough to solve this issue.
>>>
>>>
>>> We have defined three pktio output modes: ODP_PKTOUT_MODE_SEND (direct
>>> output), ODP_PKTOUT_MODE_TM (output through TM), ODP_PKTOUT_MODE_DISABLED
>>> (no output). The mode selection guides the implementation. E.g. if
>>> implementation does not have HW TM, it don't have to worry about it in other
>>> two modes. Also TM output has different feature set to direct output (e.g.
>>> queue fill rate feedback on output).
>>>
>>> Again, for performance multiple (potentially lock-free) queues are
>>> needed. TM functionality is not always needed. It would be waste to force
>>> user and implementation to always use TM API, even when TM features are not
>>> needed and are not provided (in HW).
>>>
>>> -Petri
>>>
>>>
>>> >
>>> > Regards,
>>> > Bala
>>> >
>>> > On 6 November 2015 at 18:03, Petri Savolainen
>>> > <[email protected]<mailto:[email protected]>> wrote:
>>> > > Added input queue configuration parameters and functions
>>> > > to setup multiple input queue and hashing. Added also
>>> > > odp_pktio_input_queues to query the number of queues
>>> > > and queue handles. Direct receive does not use queue
>>> > > handles, but indexes.
>>> > >
>>> > > Signed-off-by: Petri Savolainen
>>> > > <[email protected]<mailto:[email protected]>>
>>> > > ---
>>> > > include/odp/api/packet_io.h | 88
>>> > +++++++++++++++++++++++++++++++++++++++++++++
>>> > > 1 file changed, 88 insertions(+)
>>> > >
>>> > > diff --git a/include/odp/api/packet_io.h
>>> > b/include/odp/api/packet_io.h
>>> > > index 264fa75..bb0e67c 100644
>>> > > --- a/include/odp/api/packet_io.h
>>> > > +++ b/include/odp/api/packet_io.h
>>> > > @@ -19,6 +19,7 @@ extern "C" {
>>> > > #endif
>>> > >
>>> > > #include <odp/api/packet_io_stats.h>
>>> > > +#include <odp/api/queue.h>
>>> > >
>>> > > /** @defgroup odp_packet_io ODP PACKET IO
>>> > > * Operations on a packet Input/Output interface.
>>> > > @@ -85,6 +86,43 @@ typedef enum odp_pktio_output_mode_t {
>>> > > } odp_pktio_output_mode_t;
>>> > >
>>> > > /**
>>> > > + * Packet input hash protocols
>>> > > + *
>>> > > + * The list of protocol header field combinations, which are
>>> > included into
>>> > > + * packet input hash calculation.
>>> > > + */
>>> > > +typedef enum odp_pktin_hash_proto_t {
>>> > > + /** IPv4 addresses and UDP port numbers, non-fragmented
>>> > packets */
>>> > > + ODP_PKTIN_HASH_IPV4_UDP = 0,
>>> > > + /** IPv4 addresses and TCP port numbers, non-fragmented
>>> > packets */
>>> > > + ODP_PKTIN_HASH_IPV4_TCP
>>> > > +} odp_pktin_hash_proto_t;
>>> > > +
>>> > > +/**
>>> > > + * Packet input queue parameters
>>> > > + */
>>> > > +typedef struct odp_pktio_input_queue_param_t {
>>> > > + /** Enable lock-free receive operation per queue
>>> > > + * 0: Receive is multi-thread safe, 1: Receive is lock-free
>>> > */
>>> > > + odp_bool_t lock_free;
>>> > > +
>>> > > + /** Select protocol fields used for hashing */
>>> > > + odp_pktin_hash_proto_t hash_proto;
>>> > > +
>>> > > + /** Number of input queues to be created. More than one input
>>> > queue
>>> > > + * require input hashing. Hash_proto is ignore when
>>> > num_queues is one.
>>> > > + * The value must be between 1 and interface capability.
>>> > Queue type is
>>> > > + * defined by the input mode. */
>>> > > + unsigned num_queues;
>>> > > +
>>> > > + /** Queue parameters for creating input queues in
>>> > ODP_PKTIN_MODE_POLL
>>> > > + * or ODP_PKTIN_MODE_SCHED modes. Scheduler parameters are
>>> > considered
>>> > > + * only in ODP_PKTIN_MODE_SCHED mode. */
>>> > > + odp_queue_param_t queue_param;
>>> > > +
>>> > > +} odp_pktio_input_queue_param_t;
>>> > > +
>>> > > +/**
>>> > > * Packet IO parameters
>>> > > *
>>> > > * In minimum, user must select input and output modes. Use 0 for
>>> > defaults.
>>> > > @@ -158,6 +196,47 @@ odp_pktio_t odp_pktio_open(const char *dev,
>>> > odp_pool_t pool,
>>> > > int odp_pktio_capability(odp_pktio_t pktio, odp_pktio_capability_t
>>> > *capa);
>>> > >
>>> > > /**
>>> > > + * Configure packet input queues
>>> > > + *
>>> > > + * Setup a number of packet input queues and configure those. The
>>> > maximum number
>>> > > + * of queues is platform dependent and can be queried with
>>> > > + * odp_pktio_capability(). Queue handles for input queues can be
>>> > requested with
>>> > > + * odp_pktio_input_queues() after this call. All requested queues
>>> > are setup on
>>> > > + * success, no queues are setup on failure.
>>> > > + *
>>> > > + * @param pktio Packet IO handle
>>> > > + * @param param Packet input queue configuration parameters
>>> > > + *
>>> > > + * @retval 0 on success
>>> > > + * @retval <0 on failure
>>> > > + *
>>> > > + * @see odp_pktio_capability(), odp_pktio_input_queues()
>>> > > + */
>>> > > +int odp_pktio_input_queues_config(odp_pktio_t pktio,
>>> > > + const odp_pktio_input_queue_param_t
>>> > *param);
>>> > > +
>>> > > +/**
>>> > > + * Packet input queues
>>> > > + *
>>> > > + * Returns the number of input queues configured for the interface.
>>> > Outputs also
>>> > > + * up to 'num' queue handles created for ODP_PKTIN_MODE_POLL and
>>> > > + * ODP_PKTIN_MODE_SCHED modes. If return value is larger than 'num',
>>> > there are
>>> > > + * more queues than the function was allowed to output handles.
>>> > > + *
>>> > > + * Depending on the input mode, packets (or events) from these
>>> > queues are
>>> > > + * received using odp_pktio_recv_queue(), odp_queue_deq(),
>>> > odp_schedule(), etc
>>> > > + * calls.
>>> > > + *
>>> > > + * @param pktio Packet IO handle
>>> > > + * @param[out] queues Points to an array of queue handles for
>>> > output
>>> > > + * @param num Maximum number of queue handles to output
>>> > > + *
>>> > > + * @return Number of packet input queues
>>> > > + * @retval <0 on failure
>>> > > + */
>>> > > +int odp_pktio_input_queues(odp_pktio_t pktio, odp_queue_t queues[],
>>> > int num);
>>> > > +
>>> > > +/**
>>> > > * Start packet receive and transmit
>>> > > *
>>> > > * Activate packet receive and transmit on a previously opened or
>>> > stopped
>>> > > @@ -411,6 +490,15 @@ uint64_t odp_pktio_to_u64(odp_pktio_t pktio);
>>> > > void odp_pktio_param_init(odp_pktio_param_t *param);
>>> > >
>>> > > /**
>>> > > + * Initialize packet input queue parameters
>>> > > + *
>>> > > + * Initialize an odp_pktio_input_queue_param_t to its default
>>> > values.
>>> > > + *
>>> > > + * @param param Input queue parameter structure to be initialized
>>> > > + */
>>> > > +void odp_pktio_input_queue_param_init(odp_pktio_input_queue_param_t
>>> > *param);
>>> > > +
>>> > > +/**
>>> > > * Print pktio info to the console
>>> > > *
>>> > > * Print implementation-defined pktio debug information to the
>>> > console.
>>> > > --
>>> > > 2.6.2
>>> > >
>>> > > _______________________________________________
>>> > > lng-odp mailing list
>>> > > [email protected]<mailto:[email protected]>
>>> > > https://lists.linaro.org/mailman/listinfo/lng-odp
>>> _______________________________________________
>>> lng-odp mailing list
>>> [email protected]<mailto:[email protected]>
>>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> [email protected]<mailto:[email protected]>
>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>
>
> _______________________________________________
> lng-odp mailing list
> [email protected]<mailto:[email protected]>
> https://lists.linaro.org/mailman/listinfo/lng-odp
>
_______________________________________________
lng-odp mailing list
[email protected]
https://lists.linaro.org/mailman/listinfo/lng-odp
