Re: disk lights on but top showed nothing!
I’ve seen “security” running before which was pretty busy and made it not just flash, it was solid on like the time a couple days ago. Also with a 6 disk array in a maximum 8 disk card, it has two data cables plugged into the raid card. Is it likely that all 6 disks are not only flashing lights, but solid on with one bad cable? Perhaps if it’s the raid card or maybe the data plane (maybe I could try plugging it into one of the two other 8 disk sections in the data plane it can handle 24 disks) that they’re plugged into. I’m guessing it was a consistency check by the raid card like Letcher Ross said which I presume wouldn’t show up on top. Also, with a 3 way stripe raid 10 and 15000 rpm disks, theyre not SSDs but it’s pretty responsive, there’s not a lot of latency for a process to sit on its laurels. Unless the process is sleeping, it’s gonna be difficult to not put up some big cpu percentage numbers. On Sun, Jan 2, 2022 at 11:43 AM Crystal Kolipe wrote: > On Sun, Jan 02, 2022 at 11:09:40AM -0600, Luke Small wrote: > > And if there was a super busy disk program running which would make a 2x3 > > RAID10 array of 15000 RPM disks busy running on OpenBSD, I presume it > would > > be without fail taking up more than 1 percent cpu time, which nothing > other > > than top was. > > Then you 'presume' incorrectly. > > Try running two dd processes concurrently, one that reads data from the > start > of a disk, (or in your case disk array), and one that reads data from near > the end of the disk, (or array), using the raw character devices. > > I see <1% cpu usage for both dd processes testing on my current > workstation. > > Think about why this might be happening. > > Sure, if you have a single process reading data from disk such that the > disk > is hitting it's maximum data transfer rate, it's probably reasonable to > expect > that the host CPU is going to be doing something with that data and the > reading > process will show more than 1% CPU usage in 'top'. > > However, your 'server' may well have more than one process accessing disk > at > the same time, reading from all over the media. > > To put it in very simple terms: the led that flashes on and off to show > disk > activity, will also be on whilst the heads are seeking across the disk. So > if it's going from the first cylinder to the last and back again > repeatedly, > most of that time will not be spent actually writing data to the SATA bus > or > wherever else the disk is connected, so the flow of data to the CPU will be > minimal, but the led will still be on. > > Quite possibly a particuar pattern of reads and writes caused the disk to > thrash and therefore data throughput to fall. > > Or maybe one of the disks in the array was thermal re-calibrating itself. > > Or maybe your 'server' has an unreliable data cable somewhere. > > I'm not seeing anything that suggests that it's a software issue with > OpenBSD. > -- -Luke
Re: disk lights on but top showed nothing!
On 1/1/22 3:08 PM, Luke Small wrote: 6 disk raid 10 15000 rpm array (2 mirroring x 3 striping) with a raid card which can handle a maximum of 8 disks. It didn’t last forever. mfii0 at pci12 dev 0 function 0 "Symbios Logic MegaRAID SAS2208" rev 0x05: msi mfii0: "LSI MegaRAID SAS 9271-8i", firmware 23.28.0-0010, 1024MB cache scsibus2 at mfii0: 64 targets sd0 at scsibus2 targ 0 lun 0: naa.600605b00902fb50217c572a11e3e137 sd0: 418464MB, 512 bytes/sector, 857014272 sectors [...snip...] On Sat, Jan 1, 2022 at 4:22 AM Crystal Kolipe wrote: On Sat, Jan 01, 2022 at 12:01:28AM -0600, Luke Small wrote: > The lights on my server which shows that the disks are busy were on and not > just flashing and I looked at top and usually it???s because security is > running, but this time NOTHING! I even killed Firefox and by far the > busiest thing on there was top! pftop didn???t seem especially busy either! From the extremely limited information you've given, it's hard to diagnose the problem. In future, please include the output of dmesg and the output of relevant commands in problem reports. This answers questions such as: how many disks are in this server? Are they part of an array? ... Definitely see your problem...most of your dmesg is missing! (when people ask for a dmesg, it's the entire dmesg, not the chunk YOU think is interesting) However, seeing that you have a HW RAID card and an array on it, I'm pretty sure you WILL be seeing disk activity as the HW RAID card does its things on the disks with nothing going on at the OS or application level. Is that ALL you are seeing? Can't say. But disk activity on a HW RAID card is just normal. Disk activity from processes using very little processor is also not unexpected, but I'd expect the disk lights on the HW RAID attached drives to periodically flicker even if sitting at the boot> prompt. And...if you are running firefox or any other browser, you have a LOT of packages loaded on your system. Good luck trying to quiet all of them. (better idea: don't fight it). Nick.
Re: disk lights on but top showed nothing!
On Sun, Jan 02, 2022 at 11:09:40AM -0600, Luke Small wrote: > And if there was a super busy disk program running which would make a 2x3 > RAID10 array of 15000 RPM disks busy running on OpenBSD, I presume it would > be without fail taking up more than 1 percent cpu time, which nothing other > than top was. Then you 'presume' incorrectly. Try running two dd processes concurrently, one that reads data from the start of a disk, (or in your case disk array), and one that reads data from near the end of the disk, (or array), using the raw character devices. I see <1% cpu usage for both dd processes testing on my current workstation. Think about why this might be happening. Sure, if you have a single process reading data from disk such that the disk is hitting it's maximum data transfer rate, it's probably reasonable to expect that the host CPU is going to be doing something with that data and the reading process will show more than 1% CPU usage in 'top'. However, your 'server' may well have more than one process accessing disk at the same time, reading from all over the media. To put it in very simple terms: the led that flashes on and off to show disk activity, will also be on whilst the heads are seeking across the disk. So if it's going from the first cylinder to the last and back again repeatedly, most of that time will not be spent actually writing data to the SATA bus or wherever else the disk is connected, so the flow of data to the CPU will be minimal, but the led will still be on. Quite possibly a particuar pattern of reads and writes caused the disk to thrash and therefore data throughput to fall. Or maybe one of the disks in the array was thermal re-calibrating itself. Or maybe your 'server' has an unreliable data cable somewhere. I'm not seeing anything that suggests that it's a software issue with OpenBSD.
Re: disk lights on but top showed nothing!
It’s a 2U Supermicro pizza box server with a low profile Radeon graphics card. It’s always attached to a 32 inch monitor, a keyboard, mouse, KVM and I program on it, but it does a lot of server stuff. I doubt most people would call a pizza box a workstation no matter what you put on it. And if there was a super busy disk program running which would make a 2x3 RAID10 array of 15000 RPM disks busy running on OpenBSD, I presume it would be without fail taking up more than 1 percent cpu time, which nothing other than top was. On Sun, Jan 2, 2022 at 7:32 AM Jan Stary wrote: > On Jan 01 00:01:28, lukensm...@gmail.com wrote: > > The lights on my server which shows that the disks are busy were on and > not > > just flashing and I looked at top and usually it’s because security is > > running, but this time NOTHING! > > top shows you the top cpu usage. > That's not the same as disk activity. > > man ps > ps | grep ' D' > > > I even killed Firefox > > You are running firefox on a 'server'? > -- -Luke
Re: disk lights on but top showed nothing!
On Jan 01 00:01:28, lukensm...@gmail.com wrote: > The lights on my server which shows that the disks are busy were on and not > just flashing and I looked at top and usually it’s because security is > running, but this time NOTHING! top shows you the top cpu usage. That's not the same as disk activity. man ps ps | grep ' D' > I even killed Firefox You are running firefox on a 'server'?
Re: disk lights on but top showed nothing!
Perhaps it was your Megaraid controller doing a patrol read or consistency check? -Taru On Sat, Jan 1, 2022 at 12:13 PM Luke Small wrote: > 6 disk raid 10 15000 rpm array (2 mirroring x 3 striping) with a raid card > which can handle a maximum of 8 disks. It didn’t last forever. > > mfii0 at pci12 dev 0 function 0 "Symbios Logic MegaRAID SAS2208" rev 0x05: > msi > mfii0: "LSI MegaRAID SAS 9271-8i", firmware 23.28.0-0010, 1024MB cache > scsibus2 at mfii0: 64 targets > sd0 at scsibus2 targ 0 lun 0: > naa.600605b00902fb50217c572a11e3e137 > sd0: 418464MB, 512 bytes/sector, 857014272 sectors > scsibus3 at mfii0: 256 targets > ppb11 at pci7 dev 9 function 0 "Intel X58 PCIE" rev 0x22: msi > pci13 at ppb11 bus 135 > "Intel X58 IOxAPIC" rev 0x22 at pci7 dev 19 function 0 not configured > "Intel X58 Misc" rev 0x22 at pci7 dev 20 function 0 not configured > "Intel X58 GPIO" rev 0x22 at pci7 dev 20 function 1 not configured > "Intel X58 RAS" rev 0x22 at pci7 dev 20 function 2 not configured > "Intel X58 Throttle" rev 0x22 at pci7 dev 20 function 3 not configured > "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 0 not configured > "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 1 not configured > "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 2 not configured > "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 3 not configured > "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 4 not configured > "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 5 not configured > "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 6 not configured > "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 7 not configured > vmm0 at mainbus0: VMX/EPT > dt: 445 probes > vscsi0 at root > scsibus4 at vscsi0: 256 targets > softraid0 at root > scsibus5 at softraid0: 256 targets > sd1 at scsibus5 targ 1 lun 0: > sd1: 418458MB, 512 bytes/sector, 857002898 sectors > sd2 at scsibus1 targ 5 lun 0: > naa.5001173100147bc4 > sd2: 95367MB, 512 bytes/sector, 195312501 sectors > sd3 at scsibus1 targ 6 lun 0: > naa.5001173100146d54 > sd3: 95367MB, 512 bytes/sector, 195312501 sectors > sd4 at scsibus1 targ 7 lun 0: > naa.50011731001432cc > sd4: 95367MB, 512 bytes/sector, 195312501 sectors > sd5 at scsibus1 targ 8 lun 0: > naa.500117310014346c > sd5: 95367MB, 512 bytes/sector, 195312501 sectors > disklabels not read: > root on sd1a (7f4c2a49b672064f.a) swap on sd1b dump on sd1b > radeondrm0: CAICOS > radeondrm0: 1920x1080, 32bpp > wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using > wskbd0 > wsdisplay0: screen 1-5 added (std, vt100 emulation) > > On Sat, Jan 1, 2022 at 4:22 AM Crystal Kolipe > wrote: > > > On Sat, Jan 01, 2022 at 12:01:28AM -0600, Luke Small wrote: > > > The lights on my server which shows that the disks are busy were on and > > not > > > just flashing and I looked at top and usually it???s because security > is > > > running, but this time NOTHING! I even killed Firefox and by far the > > > busiest thing on there was top! pftop didn???t seem especially busy > > either! > > > > From the extremely limited information you've given, it's hard to > diagnose > > the problem. > > > > In future, please include the output of dmesg and the output of relevant > > commands in problem reports. This answers questions such as: how many > > disks are in this server? Are they part of an array? > > > > Did the output from top show a process with low or zero cpu usage, but > > stuck in the biowait state? > > > > This can happen, for example, if you have a bad or failing data cable > from > > the disk to the motherboard. It can also happen with some SSDs, likely > due > > to the firmware doing some kind of internal management of the flash > memory. > > > -- > -Luke >
Re: disk lights on but top showed nothing!
6 disk raid 10 15000 rpm array (2 mirroring x 3 striping) with a raid card which can handle a maximum of 8 disks. It didn’t last forever. mfii0 at pci12 dev 0 function 0 "Symbios Logic MegaRAID SAS2208" rev 0x05: msi mfii0: "LSI MegaRAID SAS 9271-8i", firmware 23.28.0-0010, 1024MB cache scsibus2 at mfii0: 64 targets sd0 at scsibus2 targ 0 lun 0: naa.600605b00902fb50217c572a11e3e137 sd0: 418464MB, 512 bytes/sector, 857014272 sectors scsibus3 at mfii0: 256 targets ppb11 at pci7 dev 9 function 0 "Intel X58 PCIE" rev 0x22: msi pci13 at ppb11 bus 135 "Intel X58 IOxAPIC" rev 0x22 at pci7 dev 19 function 0 not configured "Intel X58 Misc" rev 0x22 at pci7 dev 20 function 0 not configured "Intel X58 GPIO" rev 0x22 at pci7 dev 20 function 1 not configured "Intel X58 RAS" rev 0x22 at pci7 dev 20 function 2 not configured "Intel X58 Throttle" rev 0x22 at pci7 dev 20 function 3 not configured "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 0 not configured "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 1 not configured "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 2 not configured "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 3 not configured "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 4 not configured "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 5 not configured "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 6 not configured "Intel X58 QuickData" rev 0x22 at pci7 dev 22 function 7 not configured vmm0 at mainbus0: VMX/EPT dt: 445 probes vscsi0 at root scsibus4 at vscsi0: 256 targets softraid0 at root scsibus5 at softraid0: 256 targets sd1 at scsibus5 targ 1 lun 0: sd1: 418458MB, 512 bytes/sector, 857002898 sectors sd2 at scsibus1 targ 5 lun 0: naa.5001173100147bc4 sd2: 95367MB, 512 bytes/sector, 195312501 sectors sd3 at scsibus1 targ 6 lun 0: naa.5001173100146d54 sd3: 95367MB, 512 bytes/sector, 195312501 sectors sd4 at scsibus1 targ 7 lun 0: naa.50011731001432cc sd4: 95367MB, 512 bytes/sector, 195312501 sectors sd5 at scsibus1 targ 8 lun 0: naa.500117310014346c sd5: 95367MB, 512 bytes/sector, 195312501 sectors disklabels not read: root on sd1a (7f4c2a49b672064f.a) swap on sd1b dump on sd1b radeondrm0: CAICOS radeondrm0: 1920x1080, 32bpp wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using wskbd0 wsdisplay0: screen 1-5 added (std, vt100 emulation) On Sat, Jan 1, 2022 at 4:22 AM Crystal Kolipe wrote: > On Sat, Jan 01, 2022 at 12:01:28AM -0600, Luke Small wrote: > > The lights on my server which shows that the disks are busy were on and > not > > just flashing and I looked at top and usually it???s because security is > > running, but this time NOTHING! I even killed Firefox and by far the > > busiest thing on there was top! pftop didn???t seem especially busy > either! > > From the extremely limited information you've given, it's hard to diagnose > the problem. > > In future, please include the output of dmesg and the output of relevant > commands in problem reports. This answers questions such as: how many > disks are in this server? Are they part of an array? > > Did the output from top show a process with low or zero cpu usage, but > stuck in the biowait state? > > This can happen, for example, if you have a bad or failing data cable from > the disk to the motherboard. It can also happen with some SSDs, likely due > to the firmware doing some kind of internal management of the flash memory. > -- -Luke
Re: disk lights on but top showed nothing!
On Sat, Jan 01, 2022 at 12:01:28AM -0600, Luke Small wrote: > The lights on my server which shows that the disks are busy were on and not > just flashing and I looked at top and usually it???s because security is > running, but this time NOTHING! I even killed Firefox and by far the > busiest thing on there was top! pftop didn???t seem especially busy either! >From the extremely limited information you've given, it's hard to diagnose the >problem. In future, please include the output of dmesg and the output of relevant commands in problem reports. This answers questions such as: how many disks are in this server? Are they part of an array? Did the output from top show a process with low or zero cpu usage, but stuck in the biowait state? This can happen, for example, if you have a bad or failing data cable from the disk to the motherboard. It can also happen with some SSDs, likely due to the firmware doing some kind of internal management of the flash memory.
