Thanks for the follow-up question.
> I don't understand the build stage issue and why it needs a custom
allocator.
The fragmentation concern does not come from the amount of address space,
but from how the underlying heap allocator manages **large / mid-sized
temporary buffers** that are repeatedly allocated and freed during ACL
build.
ACL build allocates many temporary arrays, tables and sorted structures.
Some of them are several MB in size. When these allocations are done via
malloc/calloc, they typically end up in the general heap. Every build
iteration produces a different allocation pattern and size distribution.
Even if the allocations are freed at the end, the internal heap layout is
not restored to a “flat” state. Small holes remain, and future allocation of
large contiguous blocks may fail even if the total free memory is
sufficient.
This becomes a real operational issue in long-running processes.
> What exactly gets fragmented? It is the entire process address space
which is practically unlimited?
It is not the address space that is the limiting factor.
It is the **allocator's internal arena**.
Most allocators (glibc malloc, jemalloc, tcmalloc, etc) retain internal
metadata, bins, and split blocks. Their fragmentation behavior accumulates
over time. The process may still have hundreds of MB of “free memory”, but
not in **contiguous regions** that satisfy the next large request.
> How does malloc/free overhead compare to overall ACL build time?
The cost of malloc/free calls themselves is not the core problem.
The overhead is small relative to the total build time.
The risk is that allocator fragmentation increases unpredictably over a long
deployment, until a large block allocation fails in the data plane.
Our team has seen this exact behavior in production environments.
Because we cannot fully control the allocator state, we prefer a model
with zero dynamic allocation after init:
* persistent runtime structures → pre-allocated static region
* temporary build data → resettable memory pool
This avoids failure modes caused by allocator history and guarantees stable
latency regardless of system uptime or build frequency.
On 11/26/2025 3:57 PM, Dmitry Kozlyuk wrote:
On 11/26/25 05:44, mannywang(王永峰) wrote:
Thanks for sharing this suggestion.
We actually evaluated the heap-based approach before implementing this
patch.
It can help in some scenarios, but unfortunately it does not fully
solve our
use cases. Specifically:
1. **Heap count / scalability**
Our application maintains at least ~200 rte_acl_ctx instances (due
to the
total rule count and multi-tenant isolation). Allowing a dedicated
heap per
context would exceed the practical limits of the current rte_malloc
heap
model. The number of heaps that can be created is not unlimited, and
maintaining hundreds of separate heaps would introduce considerable
management overhead.
This is a valid point against heaps, thanks.
2. **Temporary allocations in build stage**
During `rte_acl_build`, a significant portion of memory is
allocated through
`calloc()` for internal temporary structures. These allocations are
freed
right after the build completes. Even if runtime memory could come
from a
custom heap, these temporary allocations would still need an
independent
allocator or callback mechanism to avoid fragmentation and repeated
malloc/free cycles.
I don't understand the build stage issue and why it needs a custom
allocator.
What exactly gets fragmented?
It is the entire process address space which is practically unlimited?
How does is malloc/free overhead compare to the overall ACL build time?
