Here is my first cut at adding isolation support for ODP on Linux.
In addition to the new isolation helper API this patch includes
modifications to the pktio performance test. These changes not only provide
an example of how to apply the new API, but also show the benefits
of running in an isolated environment. Below are log excerpts from running the
pktio performance test on my intel Core I5 3.5 GHZ quad-core workstation
with a standard XUbuntu 14.04 environment. (This machine is also running
Apache2, Jenkins, and a KVM virtual machine with a LAVA server - so there is
some background loading in the non-isolated environment.)
(NOTE that this Ubuntu kernel has support for cpusets but does not support
NO_HZ_FULL tickless operation - which would give a marginal additional
performance boost.)
NON-ISOLATED test (prior to addition of this patch):
Starting test with params:
Transmit workers: 2
Receive workers: 2
Duration (seconds): 1
Transmit batch length: 8
Receive batch length: 8
Packet receive method: schedule
Interface(s): loop
PPS: 1000000 Succeeded: Yes TxPkts: 1000000 RxPkts: 1000000 DropPkts: 0
PPS: 2000000 Succeeded: No TxPkts: 1205824 RxPkts: 1205824 DropPkts: 0
PPS: 1500000 Succeeded: No TxPkts: 1202184 RxPkts: 1202184 DropPkts: 0
PPS: 1250000 Succeeded: No TxPkts: 1084935 RxPkts: 1084935 DropPkts: 0
PPS: 1125000 Succeeded: Yes TxPkts: 1125040 RxPkts: 1125040 DropPkts: 0
PPS: 1187500 Succeeded: Yes TxPkts: 1187370 RxPkts: 1187370 DropPkts: 0
Maximum packet rate: 1187500 PPS (416 Mbps)
ISOLATED test (after addition of this patch):
Starting test with params:
Transmit workers: 2
Receive workers: 1
Duration (seconds): 1
Transmit batch length: 8
Receive batch length: 8
Packet receive method: schedule
Interface(s): loop
PPS: 1000000 Succeeded: Yes TxPkts: 1000000 RxPkts: 1000000 DropPkts: 0
PPS: 2000000 Succeeded: Yes TxPkts: 2000000 RxPkts: 2000000 DropPkts: 0
PPS: 4000000 Succeeded: No TxPkts: 2915128 RxPkts: 2915128 DropPkts: 0
PPS: 3000000 Succeeded: No TxPkts: 2955336 RxPkts: 2955336 DropPkts: 0
PPS: 2500000 Succeeded: Yes TxPkts: 2500048 RxPkts: 2500048 DropPkts: 0
PPS: 2750000 Succeeded: Yes TxPkts: 2750128 RxPkts: 2750128 DropPkts: 0
PPS: 2875000 Succeeded: Yes TxPkts: 2875200 RxPkts: 2875200 DropPkts: 0
PPS: 2937500 Succeeded: Yes TxPkts: 2920125 RxPkts: 2920125 DropPkts: 0
Maximum packet rate: 2937500 PPS (1030 Mbps)
You may notice there is only one receive worker after the patch is added.
This is because isolation reserves CPU 0 for non-isolated operation and
only three isolated cores are available. Because the pktio test uses
pthreads, the 'main ODP process' also has to run in the first of these
isolated cores - AND one of the worker threads has to run alongside it
on that same core. Even with that sub-optimal configuration there is a
significant performance increase.
On an 8-core system the isolation setup API in this
patch would allocate 2 non-isolated cores and 6 isolated ones.
With a further modification to the pktio test so it would not generate a
worker thread for every isolated CPU, both the worker threads and the
'main ODP process' could run alone in isolated cores and - if the kernel
supports NO_HZ_FULL isolation - could all run in 'full tickless' mode
at least part of the time.
Gary S. Robertson (1):
Initial addition of isolation support
helper/Makefile.am | 2 +
helper/include/odp/helper/linux_isolation.h | 98 +
helper/linux_isolation.c | 2901 +++++++++++++++++++++++++++
test/performance/odp_pktio_perf.c | 21 +-
4 files changed, 3016 insertions(+), 6 deletions(-)
create mode 100644 helper/include/odp/helper/linux_isolation.h
create mode 100644 helper/linux_isolation.c
--
1.9.1
_______________________________________________
lng-odp mailing list
[email protected]
https://lists.linaro.org/mailman/listinfo/lng-odp
