Re: textdumps are too slow
On Fri, Apr 7, 2023, 4:07 PM Rodney W. Grimes wrote: > > On Fri, Apr 7, 2023 at 12:08?PM Warner Losh wrote: > > > > > > > > > > > > On Fri, Apr 7, 2023 at 6:20?AM Poul-Henning Kamp > wrote: > > >> > > >> > > >> Alan Somers writes: > > >> > On Fri, Apr 7, 2023 at 2:22=E2=80=AFAM Poul-Henning Kamp < > p...@phk.freebsd.dk> wrote: > > >> > > >> > > I would expect them to be limited by the serial console speed > before > > >> > > the video system speed ? > > >> > > > >> > That was my first guess, too. But my serial console is 115200 baud, > > >> > which is faster than the low performance server grade video card. > > >> > > >> Ok, that must truly be sucky hardware... > > > > > > > > > That's 10x slower than 1990s era VGA cards then... 115200 is 11k > characters a second, > > > and FreeBSD on 1990s era VGA cards was in excess of 1MB/s, faster if > not a pure ISA > > > card... Video hardware that's slower than *THAT* likely indicates a > big big problem in > > > our video stack. > > > > > > Warner > > > > While I'm logged into the video terminal in a normal login session, I > > measure about 28M characters/second. So maybe the slow speed is due > > to something inefficient in ddb(4). > > Do you mean to a physically connected vga display and usb/ps2 connected > keyboard, or do you mean a serially connected video terminal, or something > else like a console created by a BMC? > > One other thing that could be at issue is a console redirection via > serial port, that can make "VGA device" performance appear abismal. > > > -- > Rod Grimes > rgri...@freebsd.org It's a BMC video console. I'm also using a BMC serial port, but during panic I measured the serial port as faster than the video port. >
Re: textdumps are too slow
> On Fri, Apr 7, 2023 at 12:08?PM Warner Losh wrote: > > > > > > > > On Fri, Apr 7, 2023 at 6:20?AM Poul-Henning Kamp > > wrote: > >> > >> > >> Alan Somers writes: > >> > On Fri, Apr 7, 2023 at 2:22=E2=80=AFAM Poul-Henning Kamp > >> > wrote: > >> > >> > > I would expect them to be limited by the serial console speed before > >> > > the video system speed ? > >> > > >> > That was my first guess, too. But my serial console is 115200 baud, > >> > which is faster than the low performance server grade video card. > >> > >> Ok, that must truly be sucky hardware... > > > > > > That's 10x slower than 1990s era VGA cards then... 115200 is 11k characters > > a second, > > and FreeBSD on 1990s era VGA cards was in excess of 1MB/s, faster if not a > > pure ISA > > card... Video hardware that's slower than *THAT* likely indicates a big big > > problem in > > our video stack. > > > > Warner > > While I'm logged into the video terminal in a normal login session, I > measure about 28M characters/second. So maybe the slow speed is due > to something inefficient in ddb(4). Do you mean to a physically connected vga display and usb/ps2 connected keyboard, or do you mean a serially connected video terminal, or something else like a console created by a BMC? One other thing that could be at issue is a console redirection via serial port, that can make "VGA device" performance appear abismal. -- Rod Grimes rgri...@freebsd.org
Re: textdumps are too slow
On Fri, Apr 7, 2023 at 12:08 PM Warner Losh wrote: > > > > On Fri, Apr 7, 2023 at 6:20 AM Poul-Henning Kamp wrote: >> >> >> Alan Somers writes: >> > On Fri, Apr 7, 2023 at 2:22=E2=80=AFAM Poul-Henning Kamp >> > wrote: >> >> > > I would expect them to be limited by the serial console speed before >> > > the video system speed ? >> > >> > That was my first guess, too. But my serial console is 115200 baud, >> > which is faster than the low performance server grade video card. >> >> Ok, that must truly be sucky hardware... > > > That's 10x slower than 1990s era VGA cards then... 115200 is 11k characters a > second, > and FreeBSD on 1990s era VGA cards was in excess of 1MB/s, faster if not a > pure ISA > card... Video hardware that's slower than *THAT* likely indicates a big big > problem in > our video stack. > > Warner While I'm logged into the video terminal in a normal login session, I measure about 28M characters/second. So maybe the slow speed is due to something inefficient in ddb(4).
Re: textdumps are too slow
On Fri, Apr 7, 2023 at 6:20 AM Poul-Henning Kamp wrote: > > Alan Somers writes: > > On Fri, Apr 7, 2023 at 2:22=E2=80=AFAM Poul-Henning Kamp < > p...@phk.freebsd.dk> wrote: > > > > I would expect them to be limited by the serial console speed before > > > the video system speed ? > > > > That was my first guess, too. But my serial console is 115200 baud, > > which is faster than the low performance server grade video card. > > Ok, that must truly be sucky hardware... > That's 10x slower than 1990s era VGA cards then... 115200 is 11k characters a second, and FreeBSD on 1990s era VGA cards was in excess of 1MB/s, faster if not a pure ISA card... Video hardware that's slower than *THAT* likely indicates a big big problem in our video stack. Warner
Re: textdumps are too slow
Alan Somers writes: > On Fri, Apr 7, 2023 at 2:22=E2=80=AFAM Poul-Henning Kamp > wrote: > > I would expect them to be limited by the serial console speed before > > the video system speed ? > > That was my first guess, too. But my serial console is 115200 baud, > which is faster than the low performance server grade video card. Ok, that must truly be sucky hardware... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.
Re: textdumps are too slow
On Fri, Apr 7, 2023 at 2:22 AM Poul-Henning Kamp wrote: > > > Alan Somers writes: > > > However, these servers also have about 10,000 threads > > apiece, so textdumps are also slow. They take tens of minutes. > > I have dual console setup: both serial and video. It seems to me that > > the speed of the textdump might be limited by the video system. > > I would expect them to be limited by the serial console speed before > the video system speed ? That was my first guess, too. But my serial console is 115200 baud, which is faster than the low performance server grade video card.
Re: textdumps are too slow
Alan Somers writes: > However, these servers also have about 10,000 threads > apiece, so textdumps are also slow. They take tens of minutes. > I have dual console setup: both serial and video. It seems to me that > the speed of the textdump might be limited by the video system. I would expect them to be limited by the serial console speed before the video system speed ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.
Re: textdumps are too slow
On Thu, Apr 6, 2023 at 9:08 AM Alan Somers wrote: > > I have servers with lots of RAM yet slow swap disks. So full crash > dumps are too slow. Instead, I use textdumps, which are theoretically > much faster. However, these servers also have about 10,000 threads > apiece, so textdumps are also slow. They take tens of minutes. I > have dual console setup: both serial and video. It seems to me that > the speed of the textdump might be limited by the video system. So is > there anyway to capture a textdump without printing everything to the > console? textdump(4) implies that there isn't; you must print stuff > to console in order to capture it, and you must capture in order to > dump. Would somebody please tell me that I'm wrong? > > -Alan I'll answer my own question. It turns out that ddb can mute the console, just like "conscontrol mute on" would. That makes textdumps much much faster. The command is "write cn_mute 1". So to use this trick, write the following to /etc/ddb.conf: script kdb.enter.panic=textdump set; capture on; write cn_mute 1; run lockinfo; show pcpu; bt; ps; alltrace; capture off; textdump dump; reset
