On Thu, Jun 15, 2017 at 9:21 AM, Petri Savolainen
<[email protected]> wrote:
> Added packet pool parameters for more fine grained pool
> configuration. The basic usage of the parameters is not changed,
> except that implementation may now round up 'num' by default.
> Application can limit the round up with new 'max_num' parameter.
> Another new parameter (opt) allows application give hints and
> requirements about e.g. memory to be used for a pool (or parts
> of the pool).
>
> Additionally, pool configuration may be extended with a table of
> num/len/opt values. This gives application more flexibility to
> specify requirements for various packet sizes.
>
> For example, the new specification allows to create this kind of
> pool:
>
> ^
> |
> max_num | - - - - - - - -
> |
> | *
> NUM | * *
> | * * *
> | o * * * *
> | o * * * * *
> +--------------->
> LEN
>
> o == small number of short packets into fast memory
> * == all other packets into DDR
>
> Signed-off-by: Petri Savolainen <[email protected]>
> ---
> include/odp/api/spec/pool.h | 132
> +++++++++++++++++++++++++++++++++++++++-----
> 1 file changed, 118 insertions(+), 14 deletions(-)
>
> diff --git a/include/odp/api/spec/pool.h b/include/odp/api/spec/pool.h
> index 6fc5b6b4..6b4e7495 100644
> --- a/include/odp/api/spec/pool.h
> +++ b/include/odp/api/spec/pool.h
> @@ -41,6 +41,9 @@ extern "C" {
> * Maximum pool name length in chars including null char
> */
>
> +/** Maximum number of packet pool subparameters */
> +#define ODP_POOL_PKT_SUB_MAX 7
Should this be defined in the spec or left to the implementation?
Other various MAX things are left to implementations and exposed via
odp_pool_capability(). That would seem to be more flexible.
> +
> /**
> * Pool capabilities
> */
> @@ -156,6 +159,30 @@ typedef struct odp_pool_capability_t {
> int odp_pool_capability(odp_pool_capability_t *capa);
>
> /**
> + * Pool memory type
> + */
> +typedef enum odp_pool_mem_type_t {
> + /** Default memory type */
> + ODP_POOL_MEM_DEFAULT = 0,
> +
> + /** Fast memory. Some implementations may have a restricted amount of
> + * (e.g. chip internal) pool memory that is faster access than the
> + * default memory type. */
> + ODP_POOL_MEM_FAST
Can we find a better name than "FAST" here? Why wouldn't every
application want to be fast? The idea here is that there is a memory
hierarchy that may be multi-level and present various tradeoffs in
terms of capacity and access speed. To the extent that such a
hierarchy is present on a given platform and visible to ODP
applications it may be desirable to allow the application to express
preferences in this regard. But the question is how can this be done
in a reasonably system-independent manner? Especially when we consider
operating in an NFV environment where there are hypervisors and/or
containers added into the mix.
So before we can say what needs to be input to odp_pool_param_t, don't
we need to say what is output from odp_pool_capability() to allow the
application to make informed decisions in this area? Information like:
Number of memory levels available, relative speeds and/or capacities,
etc.
Also, I continue to believe it may be premature to try to optimize in
this direction without addressing the long-deferred overall NUMA
questions and perhaps how this may also relate to header splitting. It
seems to me that lumping packets together simply by size is a very
coarse tradeoff, as if there are truly limited resources doing this on
a flow basis may make a lot more sense (e.g., high priority alerts
should get best service, etc.). Alternately, for general traffic,
being able to say that I want fast access to packet headers and are
less concerned with payload represents a large use case that's not
addressed here.
This whole area really falls into the QoS topic and so questions of
how this ties into RED and other QoS controls also comes into play. So
it may be best to omit mem_type at this stage and just focus on the
subpools. We can add mem_type info back later as part of a more
comprehensive set of strategies for advanced QoS management.
> +
> +} odp_pool_mem_type_t;
> +
> +/**
> + * Additional options for packet pool creation
> + */
> +typedef struct odp_pool_pkt_opt_t {
> + /** Pool memory type. Hint which memory type should be used for the
> + * pool or a subset of the pool. */
> + odp_pool_mem_type_t mem_type;
> +
> +} odp_pool_pkt_opt_t;
odp_pool_pkt_opt_t might be a good place to add info about header
splits, etc. Again it may be best to defer these sort of tuning
controls from this stage of the enhancement.
> +
> +/**
> * Pool parameters
> * Used to communicate pool creation options.
> * @note A single thread may not be able to allocate all 'num' elements
> @@ -185,35 +212,96 @@ typedef struct odp_pool_param_t {
>
> /** Parameters for packet pools */
> struct {
> - /** The number of packets that the pool must provide
> - that are packet length 'len' bytes or smaller.
> - The maximum value is defined by pool capability
> - pkt.max_num. */
> + /** The minimum number of packets that are packet
> length
> + * 'len' bytes or smaller. The maximum value is
> defined
> + * by pool capability pkt.max_num. An implementation
> + * may round up the value, as long as the 'max_num'
> + * parameter below is not violated.
> + */
> uint32_t num;
>
> - /** Minimum packet length that the pool must provide
> - 'num' packets. The number of packets may be less
> - than 'num' when packets are larger than 'len'.
> - The maximum value is defined by pool capability
> - pkt.max_len. Use 0 for default. */
> + /** The minimum packet length that at least 'num'
> + * packets are required. The maximum value is defined
> + * by pool capability pkt.max_len. Use 0 for default.
> + */
> uint32_t len;
>
> + /** Packet pool options
> + *
> + * Options contain additional hints and requirements,
> + * which quide implementation e.g. to select correct
> + * memory type for the pool.
> + */
> + odp_pool_pkt_opt_t opt;
Omit for now, per above comments.
> +
> + /** Number of subparameters
Do we want to call out subpools explicitly by name since that's what
we're now saying? Under this revised definition, an odp_pool_t
contains of one or more subpools (up to ODP_POOL_PKT_SUB_MAX) that
have various attributes and we're defining the requested attributes
for each of those subpools.
> + *
> + * The number of subparameter table (pkt.sub[])
> entries
> + * filled in. Subparameters continue pool
> configuration
> + * after the three parameters ('num', 'len' and
> 'opt').
> + * above. The value must not exceed
> + * ODP_POOL_PKT_SUB_MAX. The default value is 0.
> + */
> + uint8_t num_sub;
> +
> /** Maximum packet length that will be allocated from
> the pool. The maximum value is defined by pool
> capability pkt.max_len. Use 0 for default (the
> pool maximum). */
> uint32_t max_len;
>
> + /** Maximum number of packets
> + *
> + * This is the maximum number of packets of any
> length
> + * that can be allocated from the pool. The maximum
> + * value is defined by pool capability pkt.max_num.
> + * Use 0 for no requirement for maximum number.
> + * The default value is 0.
> + */
> + uint32_t max_num;
> +
> /** Minimum number of packet data bytes that are
> stored
> - in the first segment of a packet. The maximum
> value
> - is defined by pool capability pkt.max_seg_len.
> - Use 0 for default. */
> + * in the first segment of a packet. The maximum
> value
> + * is defined by pool capability pkt.max_seg_len.
> + * Use equal value with 'max_len' to ensure that all
> + * packets are single segment. Use 0 for default.
> + */
> uint32_t seg_len;
>
> /** User area size in bytes. The maximum value is
> defined by pool capability pkt.max_uarea_size.
> Specify as 0 if no user area is needed. */
> uint32_t uarea_size;
> +
> + /** Packet pool subparameters
subpool parameters?
> + *
> + * This table gives more fine grained requirements
> for
> + * pool configuration. The table continues from
> + * num/len/opt specification above. Therefore,
> + * pkt.sub[0].len must be greater than pkt.len, and
> + * pkt.sub[0].num refers to packet lengths between
> + * pkt.len + 1 and pkt.sub[0].len.
> + *
> + * Table enties must be ordered by the packet length.
> + * A number of packets figure (pkt.sub[N].num) refers
> + * to packet lengths between pkt.sub[N-1].len + 1 and
> + * pkt.sub[N].len. Each number of packets requirement
> + * may be rounded up, as long as the 'max_num'
> + * parameter is not violated. A pool fulfills each
> + * num/len requirement separately (when there are no
> + * other allocations from the pool).
> + */
> + struct {
> + /** Number of packets */
> + uint32_t num;
> +
> + /** Packet length in bytes */
> + uint32_t len;
> +
> + /** Packet pool options */
> + odp_pool_pkt_opt_t opt;
> +
> + } sub[ODP_POOL_PKT_SUB_MAX];
The syntax is certainly simpler done this way, but we can still have
ODP_POOL_PKT_SUB_MAX be defined by the implementation rather than the
spec. No different than having the typedefs need to be defined
externally.
> } pkt;
>
> /** Parameters for timeout pools */
> @@ -278,8 +366,24 @@ odp_pool_t odp_pool_lookup(const char *name);
> * Used to get information about a pool.
> */
> typedef struct odp_pool_info_t {
> - const char *name; /**< pool name */
> - odp_pool_param_t params; /**< pool parameters */
> + /** Pool name */
> + const char *name;
> +
> + /** Copy of the pool parameters */
> + odp_pool_param_t params;
> +
> + /** Packet pool info */
> + struct {
> + /** Maximum number of packets of any length
> + *
> + * This many packets in maximum can be allocated from the
> pool.
> + * Application can use this e.g. to prepare enough per packet
> + * contexts.
> + */
> + uint32_t max_num;
> +
> + } pkt;
> +
> } odp_pool_info_t;
>
> /**
> --
> 2.13.0
>
