Re: Unconsistent two-level write speed bouncing on softraid RAID1 SSD's
I decided talking about my performance issue to the manufacturer's support (Crucial by Micron). I convinced them that the disks had a problem so they proposed me RMA for my two disks and initiated the procedure from their side. I hope this would help someone getting a similar issue. Hopping this would help someone facing a similar situation. Thanks all for your replies. Cheers PS: I was pleasently surprised Crucial's support did not forced me installing windows to run their diag tool and told they "Understood" I was running OpenBSD On Wed, 2021-06-09 at 03:45 +0200, xavie...@mailoo.org wrote: > Hello, There's a strange write speed bounce behavior on my SATA > softraid > RAID1 SSD (Crucial BX500 480GB 3D NAND). Sequential writes starts > high > (~450MB/s with dd and a bs of 1M) then after about 30s to 1:30 minute > it > falls to a low ~7MB/s for one minute, then bounce back to the high > speed > of 450MB/s and so forth. > > Maybe the problem come from my Crucial BX500 480GB 3D NAND SATA 2.5- > inch > SSD which are new. But I'm not 100% sure what's happening really. > Maybe > this would help someone facing a similar situation with this > particular > high / low write speed bounces. I also tried with a second softraid > on > the same machine but with spinning USB disks. No problems so far, the > write speed is constant. Read speed are fine and constant on SSD as > well. > > Please let me know if there something I should try to workaroud or > identify this > problem. > > Reproduction scenario: > > note: The test I made to show you used the default 512B block size > with dd (so > the high speed is limited to ~130MB/s and the low speed remains > around 7MB/s) > > - disabled pf and system logs > - dd if=/dev/zero of=testfile # on /home > - iostat -w1 sd0 sd1 sd6 # chunk0 chunk1 softraid_volume > > See iostat: for results > > mount: > /dev/sd6a on / type ffs (local, softdep) > /dev/sd6h on /home type ffs (local, nodev, nosuid, softdep) > /dev/sd6e on /tmp type ffs (local, nodev, nosuid, softdep) > /dev/sd6f on /usr type ffs (local, nodev, softdep) > /dev/sd6g on /var type ffs (local, nodev, nosuid, softdep) > > disklabel: > # /dev/rsd0c: > type: SCSI > disk: SCSI disk > label: CT480BX500SSD1 > duid: 808fe38d1751a671 > flags: > bytes/sector: 512 > sectors/track: 63 > tracks/cylinder: 255 > sectors/cylinder: 16065 > cylinders: 58369 > total sectors: 937703088 > boundstart: 64noatimenoatime > boundend: 937697985 > drivedata: 0 > > 16 partitions: > # size offset fstype [fsize bsize cpg] > a: 937697921 64 RAID > c: 937703088 0 unused > # /dev/rsd1c: > type: SCSI > disk: SCSI disk > label: CT480BX500SSD1 > duid: 33c950831897af57 > flags: > bytes/sector: 512 > sectors/track: 63 > tracks/cylinder: 255 > sectors/cylinder: 16065 > cylinders: 58369 > total sectors: 937703088 > boundstart: 64 > boundend: 937697985 > drivedata: 0 > > 16 partitions: > # size offset fstype [fsize bsize cpg] > a: 937697921 64 RAID > c: 937703088 0 unused > # /dev/rsd6c: > type: SCSI > disk: SCSI disk > label: SR RAID 1 > duid: 1266e4d9a58f149d > flags: > bytes/sector: 512 > sectors/track: 63 > tracks/cylinder: 255 > sectors/cylinder: 16065 > cylinders: 58368 > total sectors: 937697393 > boundstart: 64 > boundend: 937681920 > drivedata: 0 > > 16 partitions: > # size offset fstype [fsize bsize cpg] > a: 2104448 64 4.2BSD 2048 16384 12960 # / > b: 33768633 2104512 swap # none > c: 937697393 0 unused > d: 2104480 35873152 4.2BSD 2048 16384 12960 > e: 8402016 37977632 4.2BSD 2048 16384 12960 # /tmp > f: 62926592 46379648 4.2BSD 2048 16384 12960 # /usr > g: 62926624 109306240 4.2BSD 2048 16384 12960 # /var > h: 765449024 172232896 4.2BSD 4096 32768 26062 # /home > > bioctl: > Volume Status Size Device > softraid0 1 Online 1000170315776 sd7 RAID1 > 0 Online 1000170315776 1:0.0 noencl > 1 Online 1000170315776 1:1.0 noencl > > dd: > 23679552+0 records in > 679551+0 records out > 123930112 bytes transferred in 177.691 secs (68230103 bytes/sec) > > corresponding iostat: > sd0 sd1 sd6 > KB/t t/s MB/s KB/t t/s MB/s KB/t t/s MB/s > 30.06 31 0.92 3023679552+0 records in > 679551+0 records out > 123930112 bytes transferred in 177.691 secs (68230103 bytes/sec) .12 > 31 0.92 29.81 32 0.95 > 14.47 17 0.24 14.47 17 0.24 14.47 17 0.24 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 > 2.00 2 0.00 2.00 2 0.00 2.00 2 0.00 > 16.00 1 0.02 16.00 1 0.02 16.00 1 0.02 > 16.00 1 0.02 16.00 1 0.02 16.00 1 0.02 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 > 32.00 250 7.80 32.00 250 7.80 32.00 250 7.80 DD START > 32.00 5116 159.88 32.00 5116 159.88 32.00 5116 159.88 > 31.95 4656 145.30 31.95 4655 145.27 31.95 4655 145.27 > 31.99 4501 140.60 31.99 4502 140.63 31.99 4502 140.63 > 32.00 4446 138.94 32.00 4446 138.94 32.00 4446 138.94 > 32.00 4303 134.47 32.00 4302 134.44 32.00 4303 134.47 > 32.00 4313 134.77 32.00 4313 134.77 32.00 4313 134.77 > 32.00 4380 136.88
Re: Unconsistent two-level write speed bouncing on softraid RAID1 SSD's
I don't see how an SSD can be SMR or CMR as it's not spinning plates. But I can understand those SSD's quality can be part of the problem. On Thu, 2021-06-10 at 12:50 +, Kent Watsen wrote: > The Crucial BX500 SSD uses SMR technology, which is best used for > infrequent-write applications. > > For general-purpose, and especially NAS, applications, CMR technology > should be used. > > K. > > > On Jun 10, 2021, at 6:20 AM, Xavier Sanchez > > wrote: > > > > Hi ! not so surprising news: hardware is the problem > > > > I managed to get one of the two disks apart yesterday and I figured > > out > > that those disks was in cause. (both of them) > > > > Written from my laptop directly to the device and > > - good and constant read speed > > - bouncing 7MB/s to high write speed > > > > I did looked at the serial number, they're the same. > > > > Manufacturer's support suggests that if there's no trim, write > > speed > > may be impacted ( but so much ? ) and told to let the disk idle for > > 6 > > to 8 hours so the internal garbage collector could clean it. > > > > I tried that with no luck as well. > > > > Read somewhere that issuing a security erase could also help. So I > > tried issuing the following: > > > > # atactl sd0c secsetpass user high > > User password: > > Retype user password: > > atactl: ATA device returned error register 0 > > > > But any sec* command returned: > > atactl: ATA device returned error register 0 > > > > even after a coldboot ( non-frozen ), despite the devices supports > > the > > Security Mode feature set > > > > - Am I attempting to issue the security erase the wrong way ? > > > > To me it was 0) check if not frozen 2) set user pass 3) issue > > security > > erase command with password. > > > > # atactl sd0c > > Model: CT480BX500SSD1, Rev: M6CR022, Serial #: 2030E408CA88 > > Device type: ATA, fixed > > Cylinders: 16383, heads: 16, sec/track: 63, total sectors: > > 937703088 > > Device capabilities: > > ATA standby timer values > > IORDY operation > > IORDY disabling > > Device supports the following standards: > > ATA-3 ATA-4 ATA-5 ATA-6 ATA-7 ATA-8 ATA-9 ATA-10 > > Master password revision code 0xfffe > > Device supports the following command sets: > > NOP command > > READ BUFFER command > > WRITE BUFFER command > > Host Protected Area feature set > > Read look-ahead > > Write cache > > Power Management feature set > > Security Mode feature set > > SMART feature set > > Flush Cache Ext command > > Flush Cache command > > 48bit address feature set > > Advanced Power Management feature set > > DOWNLOAD MICROCODE command > > Device has enabled the following command sets/features: > > NOP command > > READ BUFFER command > > WRITE BUFFER command > > Host Protected Area feature set > > Read look-ahead > > Write cache > > Power Management feature set > > SMART feature set > > Flush Cache Ext command > > Flush Cache command > > 48bit address feature set > > DOWNLOAD MICROCODE command > > > > > > > On Wed, 2021-06-09 at 03:45 +0200, xavie...@mailoo.org wrote: > > > Hello, There's a strange write speed bounce behavior on my SATA > > > softraid > > > RAID1 SSD (Crucial BX500 480GB 3D NAND). Sequential writes starts > > > high > > > (~450MB/s with dd and a bs of 1M) then after about 30s to 1:30 > > > minute > > > it > > > falls to a low ~7MB/s for one minute, then bounce back to the > > > high > > > speed > > > of 450MB/s and so forth. > > > > > > Maybe the problem come from my Crucial BX500 480GB 3D NAND SATA > > > 2.5- > > > inch > > > SSD which are new. But I'm not 100% sure what's happening really. > > > Maybe > > > this would help someone facing a similar situation with this > > > particular > > > high / low write speed bounces. I also tried with a second > > > softraid > > > on > > > the same machine but with spinning USB disks. No problems so far, > > > the > > > write speed is con
Re: Unconsistent two-level write speed bouncing on softraid RAID1 SSD's
All right, thanks for pointing out the details and the procedure, seems legit secfreeze is issued by default. On Thu, 2021-06-10 at 07:08 -0700, Bryan Linton wrote: > On 2021-06-10 11:49:59, Xavier Sanchez wrote: > > > > Read somewhere that issuing a security erase could also help. So I > > tried issuing the following: > > > > # atactl sd0c secsetpass user high > > User password: > > Retype user password: > > atactl: ATA device returned error register 0 > > > > But any sec* command returned: > > atactl: ATA device returned error register 0 > > > > even after a coldboot ( non-frozen ), despite the devices supports > > the > > Security Mode feature set > > > > - Am I attempting to issue the security erase the wrong way ? > > > > This is not possible on OpenBSD. It's actually a feature, not a > bug. OpenBSD issues the secfreeze command at the driver level > when disks attach. > > From atactl(8): > > secfreeze > Prevents changes to passwords until a following power > cycle. > The purpose of this command is to prevent password > setting > attacks on the security system. After command > completion any > other commands that update the device lock mode will be > aborted. > > > You can see in src/sys/dev/ata/atascsi.c:408 and > src/sys/dev/ata/wd.c:305 that the same command is issued to all > sd(4) and wd(4) drives as a security measure. > > You're going to need to boot from a live CD/USB in order to set a > password on the drive. > > You should also double-check that your BIOS doesn't have a setting > to disable this too. I've heard that some BIOSes have a toggle > for this to help mitigate the above-mentioned password setting > attacks. > > Also, another poster mentioned that these are SMR drives. If > that's the case, then the "stuttering" speeds you described is > normal for them. SMR drives are good for storing infrequently > accessed files. They're big and they're cheap, but they're not > always very fast. > > Like the old saying goes when it comes to hard drives, "Pick any > two: cheap, fast, big". SMR drives write data in "stripes". If > you change even one bit of one byte anywhere in that stripe, the > drive has to read the entire stripe into memory, change what was > changed, then re-write the entire stripe. > > This is a limitation of the technology they use. It allows very > high density drives, but has the drawback of slowing things down a > lot whenever the drive has to re-write a stripe of data. > > > I've personally found that SMR drives are good enough for my use > case, but I wouldn't recommend them for a live database where > latency is much more critical. > > It seems like the new hierarchy is now: > > SSD >> PMR > SMR > > when it comes to speed. The inverse is true when it comes to > capacity. > > So to summarize, your drive may be working exactly as intended. >
Re: Unconsistent two-level write speed bouncing on softraid RAID1 SSD's
Hi ! not so surprising news: hardware is the problem I managed to get one of the two disks apart yesterday and I figured out that those disks was in cause. (both of them) Written from my laptop directly to the device and - good and constant read speed - bouncing 7MB/s to high write speed I did looked at the serial number, they're the same. Manufacturer's support suggests that if there's no trim, write speed may be impacted ( but so much ? ) and told to let the disk idle for 6 to 8 hours so the internal garbage collector could clean it. I tried that with no luck as well. Read somewhere that issuing a security erase could also help. So I tried issuing the following: # atactl sd0c secsetpass user high User password: Retype user password: atactl: ATA device returned error register 0 But any sec* command returned: atactl: ATA device returned error register 0 even after a coldboot ( non-frozen ), despite the devices supports the Security Mode feature set - Am I attempting to issue the security erase the wrong way ? To me it was 0) check if not frozen 2) set user pass 3) issue security erase command with password. # atactl sd0c Model: CT480BX500SSD1, Rev: M6CR022, Serial #: 2030E408CA88 Device type: ATA, fixed Cylinders: 16383, heads: 16, sec/track: 63, total sectors: 937703088 Device capabilities: ATA standby timer values IORDY operation IORDY disabling Device supports the following standards: ATA-3 ATA-4 ATA-5 ATA-6 ATA-7 ATA-8 ATA-9 ATA-10 Master password revision code 0xfffe Device supports the following command sets: NOP command READ BUFFER command WRITE BUFFER command Host Protected Area feature set Read look-ahead Write cache Power Management feature set Security Mode feature set SMART feature set Flush Cache Ext command Flush Cache command 48bit address feature set Advanced Power Management feature set DOWNLOAD MICROCODE command Device has enabled the following command sets/features: NOP command READ BUFFER command WRITE BUFFER command Host Protected Area feature set Read look-ahead Write cache Power Management feature set SMART feature set Flush Cache Ext command Flush Cache command 48bit address feature set DOWNLOAD MICROCODE command On Wed, 2021-06-09 at 03:45 +0200, xavie...@mailoo.org wrote: > Hello, There's a strange write speed bounce behavior on my SATA > softraid > RAID1 SSD (Crucial BX500 480GB 3D NAND). Sequential writes starts > high > (~450MB/s with dd and a bs of 1M) then after about 30s to 1:30 minute > it > falls to a low ~7MB/s for one minute, then bounce back to the high > speed > of 450MB/s and so forth. > > Maybe the problem come from my Crucial BX500 480GB 3D NAND SATA 2.5- > inch > SSD which are new. But I'm not 100% sure what's happening really. > Maybe > this would help someone facing a similar situation with this > particular > high / low write speed bounces. I also tried with a second softraid > on > the same machine but with spinning USB disks. No problems so far, the > write speed is constant. Read speed are fine and constant on SSD as > well. > > Please let me know if there something I should try to workaroud or > identify this > problem. > > Reproduction scenario: > > note: The test I made to show you used the default 512B block size > with dd (so > the high speed is limited to ~130MB/s and the low speed remains > around 7MB/s) > > - disabled pf and system logs > - dd if=/dev/zero of=testfile # on /home > - iostat -w1 sd0 sd1 sd6 # chunk0 chunk1 softraid_volume > > See iostat: for results > > mount: > /dev/sd6a on / type ffs (local, softdep) > /dev/sd6h on /home type ffs (local, nodev, nosuid, softdep) > /dev/sd6e on /tmp type ffs (local, nodev, nosuid, softdep) > /dev/sd6f on /usr type ffs (local, nodev, softdep) > /dev/sd6g on /var type ffs (local, nodev, nosuid, softdep) > > disklabel: > # /dev/rsd0c: > type: SCSI > disk: SCSI disk > label: CT480BX500SSD1 > duid: 808fe38d1751a671 > flags: > bytes/sector: 512 > sectors/track: 63 > tracks/cylinder: 255 > sectors/cylinder: 16065 > cylinders: 58369 > total sectors: 937703088 > boundstart: 64noatimenoatime > boundend: 937697985 > drivedata: 0 > > 16 partitions: > # size offset fstype [fsize bsize cpg] > a: 937697921 64 RAID > c: 937703088 0 unused > # /dev/rsd1c: > type: SCSI > disk: SCSI disk > label: CT480BX500SSD1 > duid: 33c950831897af57 > flags: > bytes/sector: 512 > sectors/track: 63 > tracks/cylinder: 255 > sectors/cylinder: 16065 > cylinders: 58369 > total sectors: 937703088 > boundstart: 64 > boundend: 937697985 > drivedata: 0 > > 16 partitions: > # size offset fstype [fsize bsize cpg] > a: 937697921 64 RAID > c: 937703088 0 unused > # /dev/rsd6c: >
