On 31/08/2023 14:11, Robin Jarry wrote:
Since DPDK 23.03, it is possible to register a callback to report lcore
TSC cycles usage. Reuse the busy/idle cycles gathering in dpif-netdev
and expose them to the DPDK telemetry socket.
Hi Robin,
This came up in conversation with other maintainers as I mentioned I was
reviewing and the question raised was - Why add this ? if you want these
values exposed, wouldn't it be better to to add to ovsdb ?
Above is probably a more important starting point, but I have some
comments on the stats from this approach below too.
thanks,
Kevin.
Upon dpdk_attach_thread, record the mapping between the DPDK lcore_id
and the dpif-netdev core_id. Reuse that mapping in the lcore usage
callback to invoke dpif_netdev_get_pmd_cycles.
Here is an example output:
~# ovs-appctl dpif-netdev/pmd-stats-show | grep -e ^pmd -e cycles:
pmd thread numa_id 0 core_id 8:
idle cycles: 2720796781680 (100.00%)
processing cycles: 3566020 (0.00%)
pmd thread numa_id 0 core_id 9:
idle cycles: 2718974371440 (100.00%)
processing cycles: 3136840 (0.00%)
pmd thread numa_id 0 core_id 72:
pmd thread numa_id 0 core_id 73:
~# echo /eal/lcore/usage | dpdk-telemetry.py | jq
{
"/eal/lcore/usage": {
"lcore_ids": [
3,
5,
11,
15
],
Are you looking for individual lcore usage with identification of that
pmd? or overall aggregate usage ?
I ask because it will report lcore id's which would need to be mapped to
pmd core id's for anything regarding individual pmds.
That can be found in ovs-vswitchd.log or checked locally with
'ovs-appctl dpdk/lcore-list' but assuming if they were available, then
user would not be using dpdk telemetry anyway.
"total_cycles": [
2725722342740,
2725722347480,
2723899464040,
2725722354980
],
"busy_cycles": [
3566020,
3566020,
3136840,
3566020
]
}
}
These stats are cumulative so in the absence of 'ovs-appctl
dpif-netdev/pmd-stats-clear' that would need to be taken care of with
some post-processing by whatever is pulling these stats - otherwise
you'll get cumulative stats for an unknown time period and unknown
traffic profile (e.g. it would be counting before any traffic started).
These might also be reset with pmd-stats-clear independently, so that
would need to be accounted for too.
Another thing I noticed is that without the pmd-sleep info the stats in
isolation can be misleading. Example below:
With low rate traffic and clearing stats between 10 sec runs
2023-09-07T13:14:56Z|00158|dpif_netdev|INFO|PMD max sleep request is 0
usecs.
2023-09-07T13:14:56Z|00159|dpif_netdev|INFO|PMD load based sleeps are
disabled.
Time: 13:15:06.842
Measurement duration: 10.009 s
pmd thread numa_id 0 core_id 8:
Iterations: 51712564 (0.19 us/it)
- Used TSC cycles: 26021354654 (100.0 % of total cycles)
- idle iterations: 51710963 ( 99.9 % of used cycles)
- busy iterations: 1601 ( 0.1 % of used cycles)
- sleep iterations: 0 ( 0.0 % of iterations)
^^^ can see here that pmd does not sleep and is 0.1% busy
Sleep time (us): 0 ( 0 us/iteration avg.)
Rx packets: 37250 (4 Kpps, 866 cycles/pkt)
Datapath passes: 37250 (1.00 passes/pkt)
- PHWOL hits: 0 ( 0.0 %)
- MFEX Opt hits: 0 ( 0.0 %)
- Simple Match hits: 37250 (100.0 %)
- EMC hits: 0 ( 0.0 %)
- SMC hits: 0 ( 0.0 %)
- Megaflow hits: 0 ( 0.0 %, 0.00 subtbl lookups/hit)
- Upcalls: 0 ( 0.0 %, 0.0 us/upcall)
- Lost upcalls: 0 ( 0.0 %)
Tx packets: 0
{
"/eal/lcore/usage": {
"lcore_ids": [
1
],
"total_cycles": [
26127284389
],
"busy_cycles": [
32370313
]
}
}
^^^ This in isolation implies pmd is 32370313/26127284389 0.12% busy
which is true
2023-09-07T13:15:06Z|00160|dpif_netdev|INFO|PMD max sleep request is 500
usecs.
2023-09-07T13:15:06Z|00161|dpif_netdev|INFO|PMD load based sleeps are
enabled.
Time: 13:15:16.908
Measurement duration: 10.008 s
pmd thread numa_id 0 core_id 8:
Iterations: 75197 (133.09 us/it)
- Used TSC cycles: 237910969 ( 0.9 % of total cycles)
- idle iterations: 73782 ( 74.4 % of used cycles)
- busy iterations: 1415 ( 25.6 % of used cycles)
- sleep iterations: 74033 ( 98.5 % of iterations)
^^^ can see here that pmd spends most of the time sleeping and is 25%
busy when it is not sleeping
Sleep time (us): 9916314 (134 us/iteration avg.)
Rx packets: 37249 (4 Kpps, 1637 cycles/pkt)
Datapath passes: 37249 (1.00 passes/pkt)
- PHWOL hits: 0 ( 0.0 %)
- MFEX Opt hits: 0 ( 0.0 %)
- Simple Match hits: 37249 (100.0 %)
- EMC hits: 0 ( 0.0 %)
- SMC hits: 0 ( 0.0 %)
- Megaflow hits: 0 ( 0.0 %, 0.00 subtbl lookups/hit)
- Upcalls: 0 ( 0.0 %, 0.0 us/upcall)
- Lost upcalls: 0 ( 0.0 %)
Tx packets: 0
{
"/eal/lcore/usage": {
"lcore_ids": [
1
],
"total_cycles": [
238786638
],
"busy_cycles": [
61268951
]
}
}
^^^ this in isolation implies that pmd is 61268951/238786638 25% busy
but it's misleading because missing sleep info
Link: https://git.dpdk.org/dpdk/commit/?id=9ab1804922ba583b0b16
Cc: David Marchand <[email protected]>
Cc: Kevin Traynor <[email protected]>
Signed-off-by: Robin Jarry <[email protected]>
---
lib/dpdk-stub.c | 5 +++
lib/dpdk.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++-
lib/dpdk.h | 5 +++
lib/dpif-netdev.c | 38 +++++++++++++++++++
4 files changed, 142 insertions(+), 1 deletion(-)
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
