On Thu, 17 Sep 2026 14:48:24 +0200
Morten Brørup <[email protected]> wrote:
> Bruce, David,
>
> I don't know if this would be useful or not, but here goes...
>
> Situation today:
>
> DPDK has 3 types of threads with a valid lcore_id:
> 1. EAL threads,
> 2. Service threads,
> 3. Non-EAL threads.
>
> EAL threads are controlled by DPDK, and associated with physical cores at
> startup using the --lcores=<lcore_id>@<cpu_ids> parameter, and their OS
> cpuset/affinity etc. is managed by DPDK.
>
> Service threads are also managed by DPDK.
>
> Non-EAL threads are application created threads, which are simply assigned a
> valid lcore_id, but their OS cpuset/affinity etc. remains managed by the
> application.
>
> Furthermore, macros such as RTE_LCORE_FOREACH_WORKER() only iterate over EAL
> threads, and rte_lcore_is_enabled() is only true for EAL threads.
> AFAIU, this means that DPDK only considers EAL threads as fastpath ("WORKER")
> threads.
> (The main thread is also an EAL thread, although it is usually not a fastpath
> thread. But its OS cpuset/affinity etc. is managed by DPDK, which is good.)
>
>
> Feature request:
>
> Functions to create and destroy EAL threads at runtime, so applications like
> Grout could dynamically create/destroy threads that:
> 1. DPDK considers EAL (i.e. WORKER) threads,
> 2. Have their OS cpuset/affinity etc. managed by DPDK.
>
> Such functions would probably need to take <lcore_id> and <cpu_ids>
> parameters, for dynamically updating the list of DPDK managed lcores and CPUs
> at runtime.
>
> Maybe the ability to dynamically manipulate the <lcore_id>@<cpu_ids> mapping
> could be a separate set of APIs, so rte_thread_create_eal() would not require
> those parameters. (Don't know, just brainstorming.)
>
>
> Reason for asking:
>
> Because DPDK doesn't provide this feature (hot-plug/remove of EAL threads),
> Grout has to take on the task of performing its own thread management (OS
> cpuset/affinity etc.), register its fastpath threads as Non-EAL threads, and
> cannot use RTE_LCORE_FOREACH_WORKER()/rte_lcore_is_enabled() for them, but
> considers Non-EAL as fastpath threads.
>
>
> Generally it would be logical for an application to register its control
> plane threads as Non-EAL threads to get the benefits that come with a valid
> lcore_id. But how then can DPDK discriminate between control plane and
> fastpath threads? Only the application knows.
>
> It would be nice if DPDK's perception of which threads are fastpath or
> control plane was aligned with the application's perception.
>
>
> NB:
> For completeness, I should mention threads with no lcore_id, created via
> rte_thread_create_control(), and the OS cpuset/affinity etc. managed by the
> application.
>
>
> Venlig hilsen / Kind regards,
> -Morten Brørup
>
After thinking about this a little more, it looks like the best design is:
- have fixed pool of cpu's reserved and isolated
- in that pool make sure and skip cpu 0 and its sibling shared core
- use a script to not only isolate cpu from scheduler but also IRQ mapping,
RCU, etc
- start DPDK with that full pool of isolated cpu's
- keep main lcore around for telemetry, management, alarms, etc.
- launch worker lcore's on demand; start with one and if load gets bigger
than use eal to launch more.
- if demand wains, the worker lcore should return from the thread routine
and go back to idle.
The key concepts:
- keep worker cpu's isolated
- idle workers sit in OS asleep on mutex so can reduce power/load
The hard parts:
- deciding when to ramp up/down active worker count is hard heuristic to get
right
- reprogramming RSS and flows when workers are added is non trivial
- stray packets during worker changes
- synchronization of worker addition and removal
