Added a new "pile" mempool driver, based on the high-performance lock-free "pile" stack variant. With this driver added, the max number of mempool drivers (RTE_MEMPOOL_MAX_OPS_IDX) would be exceeded, so it was increased from 16 to 32.
Changed the other stack drivers to call their specific push/pop functions, instead of calling the generic stack push/pop API. Signed-off-by: Morten Brørup <[email protected]> --- v7: * Actually moved increase of max number of mempool drivers from stack patch to mempool driver patch, where it belongs. v6: * No changes. v5: * Added release note. (AI advanced model) * Moved increase of max number of mempool drivers from stack patch to mempool driver patch, where it belongs. * Mentioned increase of max number of mempool drivers in patch description. * Removed the rte_mempool_register_ops() and rte_mempool_set_ops_byname() changes. They are included in the "mempool: various cleanups" patch in the "mempool: cleanup, fixes, improvements and optimizations" series. v4: * No changes. v3: * Call other stack types' specific push/pop functions. v2: * There is no v2. --- doc/guides/mempool/stack.rst | 12 ++- doc/guides/rel_notes/release_26_11.rst | 1 + drivers/mempool/stack/rte_mempool_stack.c | 90 ++++++++++++++++++++--- lib/mempool/rte_mempool.h | 2 +- 4 files changed, 93 insertions(+), 12 deletions(-) diff --git a/doc/guides/mempool/stack.rst b/doc/guides/mempool/stack.rst index 80ea07e65d..c06ab2dc56 100644 --- a/doc/guides/mempool/stack.rst +++ b/doc/guides/mempool/stack.rst @@ -1,5 +1,6 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright(c) 2020 Intel Corporation. + Copyright(c) 2026 SmartShare Systems. Stack Mempool Driver ==================== @@ -28,6 +29,12 @@ can be selected as described in :ref:`Mempool_Handlers`: The underlying **rte_stack** operates in lock-free mode. For more information please refer to :ref:`Stack_Library_LF_Stack`. +- ``pile`` + + The underlying **rte_stack** operates in lock-free mode, + and is optimized for bulks of objects. + For more information please refer to :ref:`Stack_Library_Pile`. + The standard stack outperforms the lock-free stack on average, however the standard stack is non-preemptive: if a mempool user is preempted while holding the stack lock, that thread will block all other mempool accesses until it @@ -35,9 +42,12 @@ returns and releases the lock. As a result, an application using the standard stack whose threads can be preempted can suffer from brief, infrequent performance hiccups. -The lock-free stack, by design, is not susceptible to this problem; one thread can +The lock-free stack and the pile, by design, are not susceptible to this problem; one thread can be preempted at any point during a push or pop operation and will not impede the progress of any other thread. +The pile is not LIFO per object, but per bulk of objects. +Although the pile is optimized for bulks of objects, it can handle any request size. + For a more detailed description of the stack implementations, please refer to :doc:`/prog_guide/stack_lib`. diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst index 572e579ed5..17a1219222 100644 --- a/doc/guides/rel_notes/release_26_11.rst +++ b/doc/guides/rel_notes/release_26_11.rst @@ -61,6 +61,7 @@ New Features pushed and popped in LIFO manner, but objects within each bulk are not ordered as expected by a stack. Furthermore, it is not strictly bounded by its size, but might hold more objects. +* mempool: Added "pile" driver, using the lock-free "pile" stack-like implementation. Removed Items ------------- diff --git a/drivers/mempool/stack/rte_mempool_stack.c b/drivers/mempool/stack/rte_mempool_stack.c index 1476905227..11c1243bec 100644 --- a/drivers/mempool/stack/rte_mempool_stack.c +++ b/drivers/mempool/stack/rte_mempool_stack.c @@ -30,7 +30,7 @@ __stack_alloc(struct rte_mempool *mp, uint32_t flags) } static int -stack_alloc(struct rte_mempool *mp) +std_stack_alloc(struct rte_mempool *mp) { return __stack_alloc(mp, 0); } @@ -42,21 +42,81 @@ lf_stack_alloc(struct rte_mempool *mp) } static int -stack_enqueue(struct rte_mempool *mp, void * const *obj_table, +pile_alloc(struct rte_mempool *mp) +{ + return __stack_alloc(mp, RTE_STACK_F_PILE); +} + +static int +std_stack_enqueue(struct rte_mempool *mp, void * const *obj_table, + unsigned int n) +{ + struct rte_stack *s = mp->pool_data; + + RTE_ASSERT(s != NULL); + RTE_ASSERT(obj_table != NULL); + + return __rte_stack_std_push(s, obj_table, n) == 0 ? -ENOBUFS : 0; +} + +static int +std_stack_dequeue(struct rte_mempool *mp, void **obj_table, + unsigned int n) +{ + struct rte_stack *s = mp->pool_data; + + RTE_ASSERT(s != NULL); + RTE_ASSERT(obj_table != NULL); + + return __rte_stack_std_pop(s, obj_table, n) == 0 ? -ENOBUFS : 0; +} + +static int +lf_stack_enqueue(struct rte_mempool *mp, void * const *obj_table, + unsigned int n) +{ + struct rte_stack *s = mp->pool_data; + + RTE_ASSERT(s != NULL); + RTE_ASSERT(obj_table != NULL); + + return __rte_stack_lf_push(s, obj_table, n) == 0 ? -ENOBUFS : 0; +} + +static int +lf_stack_dequeue(struct rte_mempool *mp, void **obj_table, unsigned int n) { struct rte_stack *s = mp->pool_data; - return rte_stack_push(s, obj_table, n) == 0 ? -ENOBUFS : 0; + RTE_ASSERT(s != NULL); + RTE_ASSERT(obj_table != NULL); + + return __rte_stack_lf_pop(s, obj_table, n) == 0 ? -ENOBUFS : 0; } static int -stack_dequeue(struct rte_mempool *mp, void **obj_table, +pile_enqueue(struct rte_mempool *mp, void * const *obj_table, unsigned int n) { struct rte_stack *s = mp->pool_data; - return rte_stack_pop(s, obj_table, n) == 0 ? -ENOBUFS : 0; + RTE_ASSERT(s != NULL); + RTE_ASSERT(obj_table != NULL); + + return __rte_stack_pile_push(s, obj_table, n) == 0 ? -ENOBUFS : 0; +} + +static int +pile_dequeue(struct rte_mempool *mp, void **obj_table, + unsigned int n) +{ + struct rte_stack *s = mp->pool_data; + + RTE_ASSERT(s != NULL); + RTE_ASSERT(obj_table != NULL); + + return __rte_stack_pile_pop(s, obj_table, n) == 0 ? -ENOBUFS : 0; } static unsigned @@ -77,10 +137,10 @@ stack_free(struct rte_mempool *mp) static struct rte_mempool_ops ops_stack = { .name = "stack", - .alloc = stack_alloc, + .alloc = std_stack_alloc, .free = stack_free, - .enqueue = stack_enqueue, - .dequeue = stack_dequeue, + .enqueue = std_stack_enqueue, + .dequeue = std_stack_dequeue, .get_count = stack_get_count }; @@ -88,10 +148,20 @@ static struct rte_mempool_ops ops_lf_stack = { .name = "lf_stack", .alloc = lf_stack_alloc, .free = stack_free, - .enqueue = stack_enqueue, - .dequeue = stack_dequeue, + .enqueue = lf_stack_enqueue, + .dequeue = lf_stack_dequeue, + .get_count = stack_get_count +}; + +static struct rte_mempool_ops ops_pile = { + .name = "pile", + .alloc = pile_alloc, + .free = stack_free, + .enqueue = pile_enqueue, + .dequeue = pile_dequeue, .get_count = stack_get_count }; RTE_MEMPOOL_REGISTER_OPS(ops_stack); RTE_MEMPOOL_REGISTER_OPS(ops_lf_stack); +RTE_MEMPOOL_REGISTER_OPS(ops_pile); diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h index 50d958c7c6..3e161bfdb9 100644 --- a/lib/mempool/rte_mempool.h +++ b/lib/mempool/rte_mempool.h @@ -718,7 +718,7 @@ struct __rte_cache_aligned rte_mempool_ops { rte_mempool_dequeue_contig_blocks_t dequeue_contig_blocks; }; -#define RTE_MEMPOOL_MAX_OPS_IDX 16 /**< Max registered ops structs */ +#define RTE_MEMPOOL_MAX_OPS_IDX 32 /**< Max registered ops structs */ /** * Structure storing the table of registered ops structs, each of which contain -- 2.43.0
