Re: Fwd: Re: Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
> Date: Thu, 7 Jan 2021 22:03:15 +1000 > From: Jonathan Matthew > > On Wed, Jan 06, 2021 at 12:53:45PM +0100, Mark Kettenis wrote: > > > Date: Wed, 6 Jan 2021 21:29:52 +1000 > > > From: Jonathan Matthew > > > > > > On Wed, Jan 06, 2021 at 10:52:48AM +0100, Mark Kettenis wrote: > > > > > Date: Wed, 6 Jan 2021 20:29:09 +1100 > > > > > From: Jonathan Gray > > > > > > > > > > On Tue, Jan 05, 2021 at 10:28:20PM -1000, st...@wdwd.me wrote: > > > > > > I tested with a Protectli FW1 router (dmesg below) forwarding > > > > > > packets > > > > > > between two test machines. The latency spikes occur when running > > > > > > headless > > > > > > beginning with this commit: > > > > > > > > > > As the interrupt is handled via msi it wouldn't be a shared interrupt > > > > > related problem. > > > > > > > > > > Perhaps some drm kernel thread, but I can't think of anything that > > > > > would > > > > > be doing work with no display connected. > > > > > > > > Could be the kernel periodically polling whether a monitor is > > > > attached. Some generations of the Intel graphics hardware have broken > > > > hardware hotplug detection. And some rely on polling i2c code to > > > > detect a VGA monitor. > > > > > > > > Don't know this hardware. If it has a VGA port that's left > > > > unconnected it might help to actually connect it. Maybe one of those > > > > dongles that fake a VGA monitor would do. > > > > > > > > Disabling inteldrm(4) would also help. > > > > > > On my home router, which is a similar kind of machine, various drm work > > > queue > > > threads use a fair bit of cpu time. I normally have inteldrm disabled > > > just > > > for that - I hadn't noticed it causing latency problems, it just seemed > > > wrong > > > that my router spent more cpu time on drm stuff than on forwarding > > > packets. > > > > > > inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x35 > > > drm0 at inteldrm0 > > > inteldrm0: msi, CHERRYVIEW, gen 8 > > > inteldrm0: 1024x768, 32bpp > > > > > > It has one displayport and two hdmi, no vga, so hopefully analog hotplug > > > isn't involved. > > > > HDMI may be in the same boat. And I think there are cases where the > > VBIOS still advertises an (unconnected) VGA port even if there is no > > physical output. > > > > > $ vmstat -zi > > > interrupt total rate > > > irq0/clock 241657 396 > > > irq0/ipi17830 29 > > > irq96/acpi0 00 > > > irq144/inteldrm0 4690 > > > irq97/ahci0 50442 82 > > > irq98/xhci0250 > > > irq176/azalia0 10 > > > irq99/ppb0 00 > > > irq114/re0 12914 21 > > > irq100/ppb1 00 > > > irq115/re1 13261 21 > > > irq101/ppb2 00 > > > irq116/athn000 > > > irq102/ichiic0 00 > > > irq145/com0 1180 > > > irq146/pckbc0 00 > > > irq147/pckbc0 00 > > > Total 336717 551 > > > > > > This is what the drm workqueue threads have done in 15 minutes uptime: > > > > > > root 85080 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 > > > (drmlwq) > > > root 65272 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 > > > (drmtskl) > > > root 39235 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 > > > (drmlwq) > > > root 65215 0.0 0.0 0 0 ?? DK 9:06PM0:01.00 > > > (drmlwq) > > > root 62266 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 > > > (drmlwq) > > > root 20339 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 > > > (drmubwq) > > > root 62920 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 > > > (drmubwq) > > > root 58454 0.0 0.0 0 0 ?? DK 9:06PM0:01.00 > > > (drmubwq) > > > root 75983 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 > > > (drmubwq) > > > root 31352 0.0 0.0 0 0 ?? DK 9:06PM0:00.01 > > > (drmhpwq) > > > root 23634 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 > > > (drmhpwq) > > > root 95926 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 > > > (drmhpwq) > > > root 38038 0.0 0.0 0 0 ?? DK 9:06PM0:07.10 (drmwq) > > > root 10622 0.0 0.0 0 0 ?? DK 9:06PM0:06.50 (drmwq) > > > root 68591 0.0 0.0 0 0 ?? DK 9:06PM0:01.00 > > > (drmhpwq) > > > root 97843 0.0 0.0 0 0 ?? DK 9:06PM0:11.41 (drmwq) > > > root 36773 0.5 0.0 0 0 ?? DK 9:06PM0:10.98 (drmwq) > > > > > > I can work on gathering some profiling data with dt etc. if that would > >
Re: Fwd: Re: Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
On Wed, Jan 06, 2021 at 12:53:45PM +0100, Mark Kettenis wrote: > > Date: Wed, 6 Jan 2021 21:29:52 +1000 > > From: Jonathan Matthew > > > > On Wed, Jan 06, 2021 at 10:52:48AM +0100, Mark Kettenis wrote: > > > > Date: Wed, 6 Jan 2021 20:29:09 +1100 > > > > From: Jonathan Gray > > > > > > > > On Tue, Jan 05, 2021 at 10:28:20PM -1000, st...@wdwd.me wrote: > > > > > I tested with a Protectli FW1 router (dmesg below) forwarding packets > > > > > between two test machines. The latency spikes occur when running > > > > > headless > > > > > beginning with this commit: > > > > > > > > As the interrupt is handled via msi it wouldn't be a shared interrupt > > > > related problem. > > > > > > > > Perhaps some drm kernel thread, but I can't think of anything that would > > > > be doing work with no display connected. > > > > > > Could be the kernel periodically polling whether a monitor is > > > attached. Some generations of the Intel graphics hardware have broken > > > hardware hotplug detection. And some rely on polling i2c code to > > > detect a VGA monitor. > > > > > > Don't know this hardware. If it has a VGA port that's left > > > unconnected it might help to actually connect it. Maybe one of those > > > dongles that fake a VGA monitor would do. > > > > > > Disabling inteldrm(4) would also help. > > > > On my home router, which is a similar kind of machine, various drm work > > queue > > threads use a fair bit of cpu time. I normally have inteldrm disabled just > > for that - I hadn't noticed it causing latency problems, it just seemed > > wrong > > that my router spent more cpu time on drm stuff than on forwarding packets. > > > > inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x35 > > drm0 at inteldrm0 > > inteldrm0: msi, CHERRYVIEW, gen 8 > > inteldrm0: 1024x768, 32bpp > > > > It has one displayport and two hdmi, no vga, so hopefully analog hotplug > > isn't involved. > > HDMI may be in the same boat. And I think there are cases where the > VBIOS still advertises an (unconnected) VGA port even if there is no > physical output. > > > $ vmstat -zi > > interrupt total rate > > irq0/clock 241657 396 > > irq0/ipi17830 29 > > irq96/acpi0 00 > > irq144/inteldrm0 4690 > > irq97/ahci0 50442 82 > > irq98/xhci0250 > > irq176/azalia0 10 > > irq99/ppb0 00 > > irq114/re0 12914 21 > > irq100/ppb1 00 > > irq115/re1 13261 21 > > irq101/ppb2 00 > > irq116/athn000 > > irq102/ichiic0 00 > > irq145/com0 1180 > > irq146/pckbc0 00 > > irq147/pckbc0 00 > > Total 336717 551 > > > > This is what the drm workqueue threads have done in 15 minutes uptime: > > > > root 85080 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmlwq) > > root 65272 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmtskl) > > root 39235 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmlwq) > > root 65215 0.0 0.0 0 0 ?? DK 9:06PM0:01.00 (drmlwq) > > root 62266 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmlwq) > > root 20339 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmubwq) > > root 62920 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmubwq) > > root 58454 0.0 0.0 0 0 ?? DK 9:06PM0:01.00 (drmubwq) > > root 75983 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmubwq) > > root 31352 0.0 0.0 0 0 ?? DK 9:06PM0:00.01 (drmhpwq) > > root 23634 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmhpwq) > > root 95926 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmhpwq) > > root 38038 0.0 0.0 0 0 ?? DK 9:06PM0:07.10 (drmwq) > > root 10622 0.0 0.0 0 0 ?? DK 9:06PM0:06.50 (drmwq) > > root 68591 0.0 0.0 0 0 ?? DK 9:06PM0:01.00 (drmhpwq) > > root 97843 0.0 0.0 0 0 ?? DK 9:06PM0:11.41 (drmwq) > > root 36773 0.5 0.0 0 0 ?? DK 9:06PM0:10.98 (drmwq) > > > > I can work on gathering some profiling data with dt etc. if that would help. > > Yes, that may be helpful. > > If you're not running X, there really shouldn't be much activity. > Really just the software hotplug detection and maybe some power > management. But the monitor detection code uses delay(9) in some of > its code paths, so that may result in more accumulated CPU cycles than > one would naively expect. With no X running and not much else
Re: Fwd: Re: Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
On 2021-01-05 23:52, Mark Kettenis wrote:
Date: Wed, 6 Jan 2021 20:29:09 +1100
From: Jonathan Gray
On Tue, Jan 05, 2021 at 10:28:20PM -1000, st...@wdwd.me wrote:
> I tested with a Protectli FW1 router (dmesg below) forwarding packets
> between two test machines. The latency spikes occur when running headless
> beginning with this commit:
As the interrupt is handled via msi it wouldn't be a shared interrupt
related problem.
Perhaps some drm kernel thread, but I can't think of anything that
would
be doing work with no display connected.
Could be the kernel periodically polling whether a monitor is
attached. Some generations of the Intel graphics hardware have broken
hardware hotplug detection. And some rely on polling i2c code to
detect a VGA monitor.
That sounds consistent with what I see. Without a monitor attached there
are small CPU spikes every few seconds that do not occur once a monitor
is connected.
load averages: 0.23, 0.07, 0.02 test2.lan.local
23:41:52
73 processes: 69 idle, 4 on processor
up 1:30
CPU0: 0.0% user, 0.0% nice, 3.6% sys, 0.0% spin, 0.0% intr, 96.4%
idle
CPU1: 0.0% user, 0.0% nice, 0.2% sys, 0.0% spin, 0.0% intr, 99.8%
idle
CPU2: 0.0% user, 0.0% nice, 0.2% sys, 0.0% spin, 0.0% intr, 99.8%
idle
CPU3: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
Memory: Real: 22M/1116M act/tot Free: 2729M Cache: 618M Swap: 0K/102M
PID USERNAME PRI NICE SIZE RES STATE WAIT TIMECPU
COMMAND
26377 root 1000K 9908K sleep/0 bored 0:43 0.68%
drmwq
84910 root 1000K 9908K sleep/2 bored 0:39 0.29%
drmwq
41303 root 2800K 9908K onproc/2 -89:27 0.00%
idle2
72463 root -2200K 9908K sleep/3 -89:25 0.00%
idle3
38068 root 2800K 9908K onproc/1 -89:17 0.00%
idle1
36923 root 2800K 9908K onproc/0 -88:48 0.00%
idle0
Don't know this hardware. If it has a VGA port that's left
unconnected it might help to actually connect it. Maybe one of those
dongles that fake a VGA monitor would do.
Disabling inteldrm(4) would also help.
The system does have a VGA port. I'm good with disabling drm. Is the
best way to do that to comment it out in
/usr/src/sys/arch/amd64/conf/GENERIC then rebuild the kernel?
Can you show 'vmstat -iz' output?
>
> commit 78cd60329a9b42e2a8e91bb88c3d556b4e420e89
> Author: jsg
> Date: Mon Sep 9 01:35:43 2019 +
>
> When no display outputs are connected on boot linux 4.19 drm relies on
> deferred setup to handle the console framebuffer where as linux 4.4 drm
> created a 1024x768 console framebuffer in this situation.
>
> As we only handle setting up rasops and wsdisplay on attach go back to
> the old behaviour for now so a display can be connected after booting
> with none attached to interact with the console.
>
> This partly reverts linux commit
> drm/fb-helper: Support deferred setup
> ca91a2758fcef6635626993557dd51cfbb6dd134
>
> Reported and tested by Marcus MERIGHI.
> Tested by and ok kettenis@
>
> diff --git sys/dev/pci/drm/drm_fb_helper.c sys/dev/pci/drm/drm_fb_helper.c
> index 8e134187f6c..d95b8acbebd 100644
> --- sys/dev/pci/drm/drm_fb_helper.c
> +++ sys/dev/pci/drm/drm_fb_helper.c
> @@ -2002,15 +2002,23 @@ static int drm_fb_helper_single_fb_probe(struct
> drm_fb_helper *fb_helper,
> }
>
> if (crtc_count == 0 || sizes.fb_width == -1 || sizes.fb_height ==
> -1) {
> +#ifdef __linux__
> DRM_INFO("Cannot find any crtc or sizes\n");
>
> /* First time: disable all crtc's.. */
> -#ifdef notyet
> /* XXX calling this hangs boot with no connected outputs */
> if (!fb_helper->deferred_setup /* &&
> SPLAY_EMPTY(fb_helper->dev->files) */)
> restore_fbdev_mode(fb_helper);
> -#endif
> return -EAGAIN;
> +#else
> + /*
> +* hmm everyone went away - assume VGA cable just fell out
> +* and will come back later.
> +*/
> + DRM_INFO("Cannot find any crtc or sizes - going
> 1024x768\n");
> + sizes.fb_width = sizes.surface_width = 1024;
> + sizes.fb_height = sizes.surface_height = 768;
> +#endif
> }
>
> /* Handle our overallocation */
>
>
> Here are the test results:
>
> OpenBSD 6.5
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.444/0.892/2.103/0.200 ms
>
> OpenBSD 6.6
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.371/2.625/173.884/13.270 ms
>
> OpenBSD 6.6 with commit reversed
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets
Re: Fwd: Re: Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
The drmwq processes are more active without a monitor connected: load averages: 0.09, 0.05, 0.01 test2.lan.local 22:50:50 73 processes: 69 idle, 4 on processor up 0:39 CPU0: 0.0% user, 0.0% nice, 3.8% sys, 0.0% spin, 0.2% intr, 96.0% idle CPU1: 0.0% user, 0.0% nice, 0.4% sys, 0.0% spin, 0.0% intr, 99.6% idle CPU2: 0.0% user, 0.0% nice, 0.2% sys, 0.0% spin, 0.0% intr, 99.8% idle CPU3: 0.0% user, 0.0% nice, 0.2% sys, 0.0% spin, 0.0% intr, 99.8% idle Memory: Real: 22M/1087M act/tot Free: 2758M Cache: 601M Swap: 0K/102M PID USERNAME PRI NICE SIZE RES STATE WAIT TIMECPU COMMAND 87692 root 1000K 9908K sleep/2 bored 0:23 0.78% drmwq 84910 root 1000K 9908K sleep/1 bored 0:22 0.34% drmwq 72463 root -2200K 9908K sleep/3 -38:45 0.00% idle3 41303 root 2800K 9908K onproc/2 -38:42 0.00% idle2 38068 root 2800K 9908K onproc/1 -38:40 0.00% idle1 36923 root 2800K 9908K onproc/0 -38:24 0.00% idle0 than with a monitor connected: load averages: 0.02, 0.04, 0.01 test2.lan.local 23:15:02 73 processes: 69 idle, 4 on processor up 1:04 CPU0: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle CPU1: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle CPU2: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle CPU3: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100% idle Memory: Real: 22M/1116M act/tot Free: 2729M Cache: 618M Swap: 0K/102M PID USERNAME PRI NICE SIZE RES STATE WAIT TIMECPU COMMAND 38068 root 2800K 9908K onproc/1 -62:40 0.00% idle1 41303 root 2800K 9908K onproc/2 -62:33 0.00% idle2 36923 root -2200K 9908K sleep/0 -62:11 0.00% idle0 72463 root 2800K 9908K onproc/3 -61:30 0.00% idle3 26377 root 1000K 9908K idle bored 0:38 0.00% drmwq 87692 root 1000K 9908K idle bored 0:36 0.00% drmwq 84910 root 1000K 9908K idle bored 0:34 0.00% drmwq 53797 root 1000K 9908K idle bored 0:34 0.00% drmwq Here is vmstat -iz with the system up for 1 hour: interrupt total rate irq0/clock1425711 397 irq0/ipi164814 irq96/acpi0 20 irq144/inteldrm058241 16 irq97/ahci0 38881 10 irq98/xhci0 00 irq99/ppb0 00 irq114/em0 00 irq100/ppb1 00 irq115/em1 40 irq101/ppb2 00 irq116/em2 00 irq102/ppb3 00 irq117/em3 00 irq103/ichiic0 00 irq145/com0 66591 irq146/pckbc0 10 irq147/pckbc0 00 Total 1545980 431 On 2021-01-05 23:29, Jonathan Gray wrote: On Tue, Jan 05, 2021 at 10:28:20PM -1000, st...@wdwd.me wrote: I tested with a Protectli FW1 router (dmesg below) forwarding packets between two test machines. The latency spikes occur when running headless beginning with this commit: As the interrupt is handled via msi it wouldn't be a shared interrupt related problem. Perhaps some drm kernel thread, but I can't think of anything that would be doing work with no display connected. Can you show 'vmstat -iz' output? commit 78cd60329a9b42e2a8e91bb88c3d556b4e420e89 Author: jsg Date: Mon Sep 9 01:35:43 2019 + When no display outputs are connected on boot linux 4.19 drm relies on deferred setup to handle the console framebuffer where as linux 4.4 drm created a 1024x768 console framebuffer in this situation. As we only handle setting up rasops and wsdisplay on attach go back to the old behaviour for now so a display can be connected after booting with none attached to interact with the console. This partly reverts linux commit drm/fb-helper: Support deferred setup ca91a2758fcef6635626993557dd51cfbb6dd134 Reported and tested by Marcus MERIGHI. Tested by and ok kettenis@ diff --git sys/dev/pci/drm/drm_fb_helper.c sys/dev/pci/drm/drm_fb_helper.c index 8e134187f6c..d95b8acbebd 100644 --- sys/dev/pci/drm/drm_fb_helper.c +++ sys/dev/pci/drm/drm_fb_helper.c @@ -2002,15 +2002,23 @@ static int drm_fb_helper_single_fb_probe(struct drm_fb_helper *fb_helper, } if (crtc_count == 0 || sizes.fb_width == -1 ||
Re: Fwd: Re: Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
> Date: Wed, 6 Jan 2021 21:29:52 +1000 > From: Jonathan Matthew > > On Wed, Jan 06, 2021 at 10:52:48AM +0100, Mark Kettenis wrote: > > > Date: Wed, 6 Jan 2021 20:29:09 +1100 > > > From: Jonathan Gray > > > > > > On Tue, Jan 05, 2021 at 10:28:20PM -1000, st...@wdwd.me wrote: > > > > I tested with a Protectli FW1 router (dmesg below) forwarding packets > > > > between two test machines. The latency spikes occur when running > > > > headless > > > > beginning with this commit: > > > > > > As the interrupt is handled via msi it wouldn't be a shared interrupt > > > related problem. > > > > > > Perhaps some drm kernel thread, but I can't think of anything that would > > > be doing work with no display connected. > > > > Could be the kernel periodically polling whether a monitor is > > attached. Some generations of the Intel graphics hardware have broken > > hardware hotplug detection. And some rely on polling i2c code to > > detect a VGA monitor. > > > > Don't know this hardware. If it has a VGA port that's left > > unconnected it might help to actually connect it. Maybe one of those > > dongles that fake a VGA monitor would do. > > > > Disabling inteldrm(4) would also help. > > On my home router, which is a similar kind of machine, various drm work queue > threads use a fair bit of cpu time. I normally have inteldrm disabled just > for that - I hadn't noticed it causing latency problems, it just seemed wrong > that my router spent more cpu time on drm stuff than on forwarding packets. > > inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x35 > drm0 at inteldrm0 > inteldrm0: msi, CHERRYVIEW, gen 8 > inteldrm0: 1024x768, 32bpp > > It has one displayport and two hdmi, no vga, so hopefully analog hotplug > isn't involved. HDMI may be in the same boat. And I think there are cases where the VBIOS still advertises an (unconnected) VGA port even if there is no physical output. > $ vmstat -zi > interrupt total rate > irq0/clock 241657 396 > irq0/ipi17830 29 > irq96/acpi0 00 > irq144/inteldrm0 4690 > irq97/ahci0 50442 82 > irq98/xhci0250 > irq176/azalia0 10 > irq99/ppb0 00 > irq114/re0 12914 21 > irq100/ppb1 00 > irq115/re1 13261 21 > irq101/ppb2 00 > irq116/athn000 > irq102/ichiic0 00 > irq145/com0 1180 > irq146/pckbc0 00 > irq147/pckbc0 00 > Total 336717 551 > > This is what the drm workqueue threads have done in 15 minutes uptime: > > root 85080 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmlwq) > root 65272 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmtskl) > root 39235 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmlwq) > root 65215 0.0 0.0 0 0 ?? DK 9:06PM0:01.00 (drmlwq) > root 62266 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmlwq) > root 20339 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmubwq) > root 62920 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmubwq) > root 58454 0.0 0.0 0 0 ?? DK 9:06PM0:01.00 (drmubwq) > root 75983 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmubwq) > root 31352 0.0 0.0 0 0 ?? DK 9:06PM0:00.01 (drmhpwq) > root 23634 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmhpwq) > root 95926 0.0 0.0 0 0 ?? DK 9:06PM0:00.00 (drmhpwq) > root 38038 0.0 0.0 0 0 ?? DK 9:06PM0:07.10 (drmwq) > root 10622 0.0 0.0 0 0 ?? DK 9:06PM0:06.50 (drmwq) > root 68591 0.0 0.0 0 0 ?? DK 9:06PM0:01.00 (drmhpwq) > root 97843 0.0 0.0 0 0 ?? DK 9:06PM0:11.41 (drmwq) > root 36773 0.5 0.0 0 0 ?? DK 9:06PM0:10.98 (drmwq) > > I can work on gathering some profiling data with dt etc. if that would help. Yes, that may be helpful. If you're not running X, there really shouldn't be much activity. Really just the software hotplug detection and maybe some power management. But the monitor detection code uses delay(9) in some of its code paths, so that may result in more accumulated CPU cycles than one would naively expect.
Re: Fwd: Re: Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
> Date: Wed, 6 Jan 2021 20:29:09 +1100
> From: Jonathan Gray
>
> On Tue, Jan 05, 2021 at 10:28:20PM -1000, st...@wdwd.me wrote:
> > I tested with a Protectli FW1 router (dmesg below) forwarding packets
> > between two test machines. The latency spikes occur when running headless
> > beginning with this commit:
>
> As the interrupt is handled via msi it wouldn't be a shared interrupt
> related problem.
>
> Perhaps some drm kernel thread, but I can't think of anything that would
> be doing work with no display connected.
Could be the kernel periodically polling whether a monitor is
attached. Some generations of the Intel graphics hardware have broken
hardware hotplug detection. And some rely on polling i2c code to
detect a VGA monitor.
Don't know this hardware. If it has a VGA port that's left
unconnected it might help to actually connect it. Maybe one of those
dongles that fake a VGA monitor would do.
Disabling inteldrm(4) would also help.
> Can you show 'vmstat -iz' output?
>
> >
> > commit 78cd60329a9b42e2a8e91bb88c3d556b4e420e89
> > Author: jsg
> > Date: Mon Sep 9 01:35:43 2019 +
> >
> > When no display outputs are connected on boot linux 4.19 drm relies on
> > deferred setup to handle the console framebuffer where as linux 4.4 drm
> > created a 1024x768 console framebuffer in this situation.
> >
> > As we only handle setting up rasops and wsdisplay on attach go back to
> > the old behaviour for now so a display can be connected after booting
> > with none attached to interact with the console.
> >
> > This partly reverts linux commit
> > drm/fb-helper: Support deferred setup
> > ca91a2758fcef6635626993557dd51cfbb6dd134
> >
> > Reported and tested by Marcus MERIGHI.
> > Tested by and ok kettenis@
> >
> > diff --git sys/dev/pci/drm/drm_fb_helper.c sys/dev/pci/drm/drm_fb_helper.c
> > index 8e134187f6c..d95b8acbebd 100644
> > --- sys/dev/pci/drm/drm_fb_helper.c
> > +++ sys/dev/pci/drm/drm_fb_helper.c
> > @@ -2002,15 +2002,23 @@ static int drm_fb_helper_single_fb_probe(struct
> > drm_fb_helper *fb_helper,
> > }
> >
> > if (crtc_count == 0 || sizes.fb_width == -1 || sizes.fb_height ==
> > -1) {
> > +#ifdef __linux__
> > DRM_INFO("Cannot find any crtc or sizes\n");
> >
> > /* First time: disable all crtc's.. */
> > -#ifdef notyet
> > /* XXX calling this hangs boot with no connected outputs */
> > if (!fb_helper->deferred_setup /* &&
> > SPLAY_EMPTY(fb_helper->dev->files) */)
> > restore_fbdev_mode(fb_helper);
> > -#endif
> > return -EAGAIN;
> > +#else
> > + /*
> > +* hmm everyone went away - assume VGA cable just fell out
> > +* and will come back later.
> > +*/
> > + DRM_INFO("Cannot find any crtc or sizes - going
> > 1024x768\n");
> > + sizes.fb_width = sizes.surface_width = 1024;
> > + sizes.fb_height = sizes.surface_height = 768;
> > +#endif
> > }
> >
> > /* Handle our overallocation */
> >
> >
> > Here are the test results:
> >
> > OpenBSD 6.5
> > --- 192.168.2.100 ping statistics ---
> > 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> > round-trip min/avg/max/stddev = 0.444/0.892/2.103/0.200 ms
> >
> > OpenBSD 6.6
> > --- 192.168.2.100 ping statistics ---
> > 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> > round-trip min/avg/max/stddev = 0.371/2.625/173.884/13.270 ms
> >
> > OpenBSD 6.6 with commit reversed
> > --- 192.168.2.100 ping statistics ---
> > 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> > round-trip min/avg/max/stddev = 0.418/0.567/1.303/0.093 ms
> >
> > OpenBSD 6.7
> > --- 192.168.2.100 ping statistics ---
> > 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> > round-trip min/avg/max/stddev = 0.392/1.312/170.699/4.993 ms
> >
> > OpenBSD 6.7 with commit reversed
> > --- 192.168.2.100 ping statistics ---
> > 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> > round-trip min/avg/max/stddev = 0.397/0.545/1.263/0.071 ms
> >
> > OpenBSD 6.8
> > --- 192.168.2.100 ping statistics ---
> > 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> > round-trip min/avg/max/stddev = 0.402/1.144/97.022/2.396 ms
> >
> > OpenBSD 6.8 with commit reversed
> > --- 192.168.2.100 ping statistics ---
> > 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> > round-trip min/avg/max/stddev = 0.466/0.935/2.116/0.239 ms
> >
> > OpenBSD 2021-01-03 snapshot
> > --- 192.168.2.100 ping statistics ---
> > 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> > round-trip min/avg/max/stddev = 0.405/1.192/79.021/2.304 ms
> >
> >
> > OpenBSD 6.8 (GENERIC.MP) #98: Sun Oct 4 18:13:26 MDT 2020
> > dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
Re: Fwd: Re: Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
On Tue, Jan 05, 2021 at 10:28:20PM -1000, st...@wdwd.me wrote:
> I tested with a Protectli FW1 router (dmesg below) forwarding packets
> between two test machines. The latency spikes occur when running headless
> beginning with this commit:
As the interrupt is handled via msi it wouldn't be a shared interrupt
related problem.
Perhaps some drm kernel thread, but I can't think of anything that would
be doing work with no display connected.
Can you show 'vmstat -iz' output?
>
> commit 78cd60329a9b42e2a8e91bb88c3d556b4e420e89
> Author: jsg
> Date: Mon Sep 9 01:35:43 2019 +
>
> When no display outputs are connected on boot linux 4.19 drm relies on
> deferred setup to handle the console framebuffer where as linux 4.4 drm
> created a 1024x768 console framebuffer in this situation.
>
> As we only handle setting up rasops and wsdisplay on attach go back to
> the old behaviour for now so a display can be connected after booting
> with none attached to interact with the console.
>
> This partly reverts linux commit
> drm/fb-helper: Support deferred setup
> ca91a2758fcef6635626993557dd51cfbb6dd134
>
> Reported and tested by Marcus MERIGHI.
> Tested by and ok kettenis@
>
> diff --git sys/dev/pci/drm/drm_fb_helper.c sys/dev/pci/drm/drm_fb_helper.c
> index 8e134187f6c..d95b8acbebd 100644
> --- sys/dev/pci/drm/drm_fb_helper.c
> +++ sys/dev/pci/drm/drm_fb_helper.c
> @@ -2002,15 +2002,23 @@ static int drm_fb_helper_single_fb_probe(struct
> drm_fb_helper *fb_helper,
> }
>
> if (crtc_count == 0 || sizes.fb_width == -1 || sizes.fb_height ==
> -1) {
> +#ifdef __linux__
> DRM_INFO("Cannot find any crtc or sizes\n");
>
> /* First time: disable all crtc's.. */
> -#ifdef notyet
> /* XXX calling this hangs boot with no connected outputs */
> if (!fb_helper->deferred_setup /* &&
> SPLAY_EMPTY(fb_helper->dev->files) */)
> restore_fbdev_mode(fb_helper);
> -#endif
> return -EAGAIN;
> +#else
> + /*
> +* hmm everyone went away - assume VGA cable just fell out
> +* and will come back later.
> +*/
> + DRM_INFO("Cannot find any crtc or sizes - going
> 1024x768\n");
> + sizes.fb_width = sizes.surface_width = 1024;
> + sizes.fb_height = sizes.surface_height = 768;
> +#endif
> }
>
> /* Handle our overallocation */
>
>
> Here are the test results:
>
> OpenBSD 6.5
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.444/0.892/2.103/0.200 ms
>
> OpenBSD 6.6
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.371/2.625/173.884/13.270 ms
>
> OpenBSD 6.6 with commit reversed
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.418/0.567/1.303/0.093 ms
>
> OpenBSD 6.7
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.392/1.312/170.699/4.993 ms
>
> OpenBSD 6.7 with commit reversed
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.397/0.545/1.263/0.071 ms
>
> OpenBSD 6.8
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.402/1.144/97.022/2.396 ms
>
> OpenBSD 6.8 with commit reversed
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.466/0.935/2.116/0.239 ms
>
> OpenBSD 2021-01-03 snapshot
> --- 192.168.2.100 ping statistics ---
> 3600 packets transmitted, 3600 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.405/1.192/79.021/2.304 ms
>
>
> OpenBSD 6.8 (GENERIC.MP) #98: Sun Oct 4 18:13:26 MDT 2020
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4164894720 (3971MB)
> avail mem = 4023623680 (3837MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 3.0 @ 0xebfd0 (51 entries)
> bios0: vendor American Megatrends Inc. version "5.6.5" date 05/14/2019
> bios0: Protectli FW1
> acpi0 at bios0: ACPI 5.0
> acpi0: sleep states S0 S5
> acpi0: tables DSDT FACP APIC FPDT FIDT MCFG LPIT HPET SSDT SSDT SSDT UEFI
> acpi0: wakeup devices PS2K(S0) PS2M(S0) XHC1(S4) RP01(S4) PXSX(S4) RP02(S4)
> PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
>
