There are two ways of allocating/freeing memory objects in DPDK:
1. From the memory heap, i.e. rte_malloc() etc.
2. From a memory pool, i.e. rte_mempool_get() etc.
The current memory heap in DPDK has two issues:
- It is too slow to be used in the fast path.
- There may be contention between control threads and
EAL threads for the same spinlocks in the heap.
The memory pools in the DPDK has multiple issues:
- It is designed for fixed size objects,
so individual mempools must be allocated for each type of object.
(In theory, mempools could be allocated per object size,
and shared across modules.
But that would require coordination across modules, including
the total number of objects required by these modules,
the choice of underlying mempool driver,
and the optimal mempool cache size.)
- The number of objects in each mempool is fixed,
so applications have to size their mempools to worst case usage.
- The memory for the objects in a mempool is pre-allocated.
In summary, mempools consume significantly more memory than effectively
being used.
We discussed the need for a new memory heap system,
and converged on the following features:
- Providing functions to allocate and free memory objects of varying
size, like the rte_malloc library, but usable in the fast path.
- Perhaps also a need for bulk alloc/free functions,
like the rte_mempool library.
- Building on top of memzones.
- No dependency on the current DPDK heap, so it can cleanly replace it.
This is a stretch requirement.
- NUMA aware.
- Using slabs for various block sizes like in the Linux Kernel,
possibly with object size 2^N.
- Using per-lcore caches to reduce contention,
resembling the mempool per-lcore caches.
Mattias Rönnblom has implemented a prototype with the above properties,
and this was a good starting point for the discussion.
Discussion:
- It does not seem to be a requirement to be able to free the memory
used by the heap back to the memzones.
- It is possible having header-less objects.
However, if the headers are relatively small compared to the size of
the objects themselves, having a header may offer some benefits.
- The mempool library has implementation details optimizing for
spreading usage across memory channels, cache alignment, etc..
Such performance optimization details might need consideration.
- There is a lower limit to how small objects will be allocated.
E.g. objects smaller than the size of a pointer is unlikely.
- It is acceptable to return an object larger than the size requested,
e.g. returning an object of 16 bytes when 12 bytes were requested.
- Preventing false sharing of allocated objects between CPU caches
may not be necessary, but must be considered.
- Optimally, the new heap should replace the existing heap.
As seen with the timer wheel library previously proposed, replacing
core libraries in DPDK is difficult, so adding the new heap library
in parallel with the existing heap, i.e. with separate APIs, might be
a path of less resistance.
It will then be an application choice to use this new memory heap.
And long term, it can replace the existing heap, if appropriate.
Mbuf discussion:
- As a stretch goal, mbufs should be dynamically allocated from
the new heap, instead of being pre-allocated in mempools.
- This might be achievable either
- by replacing the mempool library with a wrapper to the heap, or
- by providing a new mempool driver using the heap, as an
alternative to the existing ring and stack mempool drivers.
- In this context, it is important to note that mbuf objects have some
pre-initialized fields when held in their respective mempools.
If allocating an mbuf directly from the heap, these fields must
somehow be initialized.
We did not discuss solutions for this, but noted that using the new
heap for mbufs should be considered in the design choices.
Venlig hilsen / Kind regards,
-Morten Brørup
