Hi Stefano,
On 18/10/2022 20:54, Stefano Stabellini wrote:
On Tue, 18 Oct 2022, Leo Yan wrote:
On Mon, Oct 17, 2022 at 04:50:05PM -0700, Stefano Stabellini wrote:
[...]
Which means it takes 543us to let Dom0 to receive the notification.
You could see DomU runs in CPU3 and Dom0 runs on CPU13, there should
not have contention for CPU resources. Seems to me, it's likely Xen
hypervisor takes long time to deliver the interrupt, note, it's not
take so long time for every skb transferring, sometimes the time for
response a notification is short (about ~10us).
Good find. I think this is worth investigating further. Do you have
vwfi=native in your Xen command line as well?
Yes, I have added "sched=null" and "vwfi=native" into Xen options:
options=noreboot dom0_mem=4096M bootscrub=0 iommu=on loglvl=error
guest_loglvl=error sched=null vwfi=native
After that, I would add printk also in Xen with the timestamp. The event
channel notification code path is the following:
# domU side
xen/arch/arm/vgic-v2.c:vgic_v2_to_sgi
xen/arch/arm/vgic.c:vgic_to_sgi
xen/arch/arm/vgic.c:vgic_inject_irq
xen/arch/arm/vgic.c:vcpu_kick
xen/arch/arm/gic-v2.c:gicv2_send_SGI
# dom0 side
xen/arch/arm/gic.c:do_sgi
xen/arch/arm/traps.c:leave_hypervisor_to_guest
It would be good to understand why sometimes it takes ~10us and some
other times it takes ~540us
Thanks a lot for detailed info.
Just note, in my platform DomU enables GICv3 driver rather than GICv2.
This would be a bit different in the Xen code. But it should be easy
for me to map to vgic-v3 files.
I have a question for how to trace Xen system. Outputting chars to UART
is time costy (usually it's millisecond level), it is not friendly to
use console for debugging performance issue. I searched a bit, either
"xl dmesg" or xentrace can be used for capturing trace logs, one thing
I am not certain is if we can save Xen logs only into log buffer and
doesn't output to UART, so that afterwards we can use "xl dmesg" to
capture the logs. Could anyone confirm for this is correct usage with
"xl dmesg" or I should use xentrace for this case?
I am not super familiar about this. But if you take away the console
entirely from Xen and give it to dom0 only (no dtuart command line
option to Xen), you should get the behavior you described:
- no console output because Xen has no console
- xl dmesg should still work
You are right, xl dmesg will still work. The console is built using a
ring so if you are using "xl dmesg" you will only get the latest log in
the ring.
xenconsoled has an option to save the hypervisor log in
/var/log/xen/hypervisor.log (default path) if you request to dump them.
This is an option at startup and can be set in the init.d/systemd
configuration for Xen (see XENCONSOLED_TRACE).
That said, xentrace is better suited if you are looking for low overhead
tracing. The console is not directly involved in the process and you can
already benefits from the trace point sprinkled in Xen code (such as the
scheduling).
Cheers,
--
Julien Grall
