Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Thor Lancelot Simon]

On Wed, Oct 11, 2017 at 07:56:04PM -, Michael van Elst wrote:
> t...@panix.com (Thor Lancelot Simon) writes:
> 
> >It probably has to do with our small maximum transfer size.  The disk is
> >probably trying to be safer and *not* caching tagged writes as aggressively,
> >but with only 32 commands in-flight (SCSI/SAS allow 256) and a maximum
> >transfer size of 64K (our MAXPHYS), that's only 2MB of in-flight data;
> >maybe not enough to keep up with the actual bandwidth of the media at
> >its real latency.
> 
> With a larger MAXPHYS there wouldn't be as many transfers in flight.

That depends on the workload and the clustering behavior of the filesystem.

Remember, in the relevant sense, data the disk has already accepted is
"in flight" from the point of view of absorbing latency, in the non-NCQ,
write-cache-enabled case.  The host can't tell it's "in flight", but it
still is -- so the host will continue to write aggressively even if the
filesystem's clustering behavior would otherwise throttle it -- because
to the host, these writes appear to be "complete".

If the drive is not aggressively caching writes in the same way when NCQ
is in use, this won't happen, since less writes will appear, to the host,
to be completed.  With only 32 tags available, we may need a larger
transfer per tag to allow the same amount of in-flight data (if the disk,
with NCQ enabled, has stopped lying about whether the transfers are
really complete).

-- 
  Thor Lancelot Simont...@panix.com
 "The two most common variations translate as follows:
illegitimi non carborundum = the unlawful are not silicon carbide
illegitimis non carborundum = the unlawful don't have silicon carbide."

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Chavdar Ivanov]

It supports AHCI mode only on the regular internal disk, not on a disk
connected via a tray to the CDROM.

On Thu, 12 Oct 2017, 00:11 Michael van Elst,  wrote:

> On Wed, Oct 11, 2017 at 08:19:13PM +, Chavdar Ivanov wrote:
> > I see you are on a ThinkPad as well. I have mentioned it elsewhere, on my
> > ThinkPad T61p I am getting reliably since the NCQ update the following:
> > ...
> > wd0 at atabus0 drive 0
> > wd0: 
> > wd0: drive supports 16-sector PIO transfers, LBA48 addressing
> > wd0: 298 GB, 620181 cyl, 16 head, 63 sec, 512 bytes/sect x 625142448
> sectors
> > piixide0:0:0: bad state 0 in wdc_ata_bio_intr
> > panic: wdc_ata_bio_intr: bad state
> > fatal breakpoint trap in supervisor mode
> > trap type 1 code 0 rip 0x8021c0c5 cs 0x8 rflags 0x246 cr2 0
> ilevel
> > 0x8 rsp 0xe40040003c38
> > curlwp 0xe4013bb27840 pid 0.2 lowest kstack 0xe40042c0
> > rebooting...
>
> Looks like bad handling of the drives RESET state.
>
> N.B. the T61p supports AHCI mode too, any reason why this is not
> selected in the BIOS? AHCI mode so far didn't show problems.
>
>
> Greetings,
> --
> Michael van Elst
> Internet: mlel...@serpens.de
> "A potential Snark may lurk in every tree."
>

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Chavdar Ivanov]

I see you are on a ThinkPad as well. I have mentioned it elsewhere, on my
ThinkPad T61p I am getting reliably since the NCQ update the following:
...
wd0 at atabus0 drive 0
wd0: 
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 298 GB, 620181 cyl, 16 head, 63 sec, 512 bytes/sect x 625142448 sectors
piixide0:0:0: bad state 0 in wdc_ata_bio_intr
panic: wdc_ata_bio_intr: bad state
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip 0x8021c0c5 cs 0x8 rflags 0x246 cr2 0 ilevel
0x8 rsp 0xe40040003c38
curlwp 0xe4013bb27840 pid 0.2 lowest kstack 0xe40042c0
rebooting...
...

The disk is on the place of the CD with an appropriate tray, this happens
with kernels from releng, not only with mine. There is another disk, an SSD
in the internal bay, with a Windows 10 and some Linux installation, the
default boot is NetBSD from the tray.

Chavdar Ivanov


On Wed, 11 Oct 2017 at 20:57 Michael van Elst  wrote:

> t...@panix.com (Thor Lancelot Simon) writes:
>
> >It probably has to do with our small maximum transfer size.  The disk is
> >probably trying to be safer and *not* caching tagged writes as
> aggressively,
> >but with only 32 commands in-flight (SCSI/SAS allow 256) and a maximum
> >transfer size of 64K (our MAXPHYS), that's only 2MB of in-flight data;
> >maybe not enough to keep up with the actual bandwidth of the media at
> >its real latency.
>
> With a larger MAXPHYS there wouldn't be as many transfers in flight.
>
> The disk here in the Thinkpad behaves differently. With up to 5 concurrent
> transfers (320k buffer size for dd reading the raw partition) everything
> is fine. With 6 concurrent transfers the speed goes down again, with
> 30 transfers we are at half speed.
>
> --
> --
> Michael van Elst
> Internet: mlel...@serpens.de
> "A potential Snark may lurk in every tree."
>

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Michael van Elst]

On Wed, Oct 11, 2017 at 08:19:13PM +, Chavdar Ivanov wrote:
> I see you are on a ThinkPad as well. I have mentioned it elsewhere, on my
> ThinkPad T61p I am getting reliably since the NCQ update the following:
> ...
> wd0 at atabus0 drive 0
> wd0: 
> wd0: drive supports 16-sector PIO transfers, LBA48 addressing
> wd0: 298 GB, 620181 cyl, 16 head, 63 sec, 512 bytes/sect x 625142448 sectors
> piixide0:0:0: bad state 0 in wdc_ata_bio_intr
> panic: wdc_ata_bio_intr: bad state
> fatal breakpoint trap in supervisor mode
> trap type 1 code 0 rip 0x8021c0c5 cs 0x8 rflags 0x246 cr2 0 ilevel
> 0x8 rsp 0xe40040003c38
> curlwp 0xe4013bb27840 pid 0.2 lowest kstack 0xe40042c0
> rebooting...

Looks like bad handling of the drives RESET state.

N.B. the T61p supports AHCI mode too, any reason why this is not
selected in the BIOS? AHCI mode so far didn't show problems.


Greetings,
-- 
Michael van Elst
Internet: mlel...@serpens.de
"A potential Snark may lurk in every tree."

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Michael van Elst]

macal...@netbsd.org (Michael) writes:

>/home/ml# sysctl -w hw.wd1.use_ncq=0
>hw.wd1.use_ncq: 1 -> 0
>/home/ml# dd if=/dev/rwd1c of=/dev/null bs=1m count=2048
>2048+0 records in
>2048+0 records out
>2147483648 bytes transferred in 21.747 secs (98748500 bytes/sec)

Please try to use different buffer sizes. Maybe there is a
pattern too.


-- 
-- 
Michael van Elst
Internet: mlel...@serpens.de
"A potential Snark may lurk in every tree."

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Michael van Elst]

t...@panix.com (Thor Lancelot Simon) writes:

>It probably has to do with our small maximum transfer size.  The disk is
>probably trying to be safer and *not* caching tagged writes as aggressively,
>but with only 32 commands in-flight (SCSI/SAS allow 256) and a maximum
>transfer size of 64K (our MAXPHYS), that's only 2MB of in-flight data;
>maybe not enough to keep up with the actual bandwidth of the media at
>its real latency.

With a larger MAXPHYS there wouldn't be as many transfers in flight.

The disk here in the Thinkpad behaves differently. With up to 5 concurrent
transfers (320k buffer size for dd reading the raw partition) everything
is fine. With 6 concurrent transfers the speed goes down again, with
30 transfers we are at half speed.

-- 
-- 
Michael van Elst
Internet: mlel...@serpens.de
"A potential Snark may lurk in every tree."

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Michael]

Hello,

On Tue, 10 Oct 2017 23:11:54 +0200
Jaromír Doleček  wrote:

> I've seen this on one of my disks, too. It seems it's much slower in NCQ
> mode. I think the firmware might not utilise the disk cache properly when
> in NCQ mode.
> 
> You can try switching it off via sysctl, hw.wdX.use_ncq. You can also try
> to turn off use_ncq_prio if that makes any difference.

/home/ml# sysctl -w hw.wd1.use_ncq=0
hw.wd1.use_ncq: 1 -> 0
/home/ml# dd if=/dev/rwd1c of=/dev/null bs=1m count=2048
2048+0 records in
2048+0 records out
2147483648 bytes transferred in 21.747 secs (98748500 bytes/sec)

/home/ml# sysctl -w hw.wd1.use_ncq=1
hw.wd1.use_ncq: 0 -> 1
/home/ml# dd if=/dev/rwd1c of=/dev/null bs=1m count=2048
2048+0 records in
2048+0 records out
2147483648 bytes transferred in 26.125 secs (82200331 bytes/sec)

/home/ml# sysctl -w hw.wd1.use_ncq_prio=0
hw.wd1.use_ncq_prio: 1 -> 0
/home/ml# dd if=/dev/rwd1c of=/dev/null bs=1m count=2048
2048+0 records in
2048+0 records out
2147483648 bytes transferred in 24.187 secs (88786689 bytes/sec)

... that's the siisata:
siisata0 at pci0 dev 2 function 0: vendor 1095 product 3124 (rev. 0x02)
siisata0: interrupting at ivec 700
siisata0: SiI3124, 3.0Gb/s
siisata0: 64-bit 66MHz PCI

... with one of these:

siisata0 port 0: device present, speed: 3.0Gb/s
wd1(siisata0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100) 
(using DMA), NCQ (30 tags) w/PRIO

Model: WDC WD5000AAKX-083CA1, Rev: 17.01H17, Serial #:  WD-WCAYUCL72135
World Wide Name: 50014EE104595309
Device type: ATA, fixed
Capacity 500 Gbytes, 976773168 sectors, 512 bytes/sector
Cylinders: 16383, heads: 16, sec/track: 63
...
Serial ATA capabilities:
1.5Gb/s signaling
3.0Gb/s signaling
6.0Gb/s signaling
Native Command Queuing
PHY Event Counters

have fun
Michael

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Thor Lancelot Simon]

On Tue, Oct 10, 2017 at 11:11:54PM +0200, Jarom??r Dole??ek wrote:
> I've fixed the compilation for ALL kernels.
> 
> 2017-10-10 17:34 GMT+02:00 Michael :
> > I tried sequential reads ( dd if=/dev/rwd0c ... ) and throughput took a
> > significant hit. I used to get about 120MB/s with the siisata, now it
> > fluctuates between 80 and 90MB/s, ahcisata dropped from about 80MB/s to
> > 70MB/s. Both spinning rust of varying vintage.
> > I should probably do a bonnie run on either one before & after to see
> > if there's any change in random access.
> 
> I've seen this on one of my disks, too. It seems it's much slower in NCQ
> mode. I think the firmware might not utilise the disk cache properly when
> in NCQ mode.

It probably has to do with our small maximum transfer size.  The disk is
probably trying to be safer and *not* caching tagged writes as aggressively,
but with only 32 commands in-flight (SCSI/SAS allow 256) and a maximum
transfer size of 64K (our MAXPHYS), that's only 2MB of in-flight data;
maybe not enough to keep up with the actual bandwidth of the media at
its real latency.

Most other operating systems will hand the drive 32 x 128K transfers, or
even larger.  If you want to attack the -- fairly minor at this point,
I think -- problems of rebasing and merging the tls-maxphys branch,
I bet it would help here.

I have the interest, but not the time, I'm afraid.  Real life interferes
again.

Thor

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Jaromír Doleček]

I've fixed the compilation for ALL kernels.

2017-10-10 17:34 GMT+02:00 Michael :
> I tried sequential reads ( dd if=/dev/rwd0c ... ) and throughput took a
> significant hit. I used to get about 120MB/s with the siisata, now it
> fluctuates between 80 and 90MB/s, ahcisata dropped from about 80MB/s to
> 70MB/s. Both spinning rust of varying vintage.
> I should probably do a bonnie run on either one before & after to see
> if there's any change in random access.

I've seen this on one of my disks, too. It seems it's much slower in NCQ
mode. I think the firmware might not utilise the disk cache properly when
in NCQ mode.

You can try switching it off via sysctl, hw.wdX.use_ncq. You can also try
to turn off use_ncq_prio if that makes any difference.

We might need to introduce some heuristics for this.

Jaromir

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Michael]

Hello,

On Sat, 7 Oct 2017 18:34:04 +0200
Jaromír Doleček  wrote:

> I've merged the NCQ branch to HEAD.

Nice, thanks!

> The code was quite extensively tested on that harware on amd64. Other archs
> and drivers compile, but I had no way to test them. Particularily, I had no
> chance to really test any real IDE disks, neither any non-PCI controller
> attachments. If you are in position to confirm them working, I'd appreciate
> it.

I tried siisata on sparc64 ( in a 64bit/66MHz slot ) and ahcisata on a
Cubietruck, both seem to work as intended.

> Please report any problems, I'll fix any potential fallout ASAP.
> 
> Performance wise, I've tested so far only sequential I/O via fio and dd
> with cache enabled. I observed very mild (>2%, 72.7->74.8 MB/s) performance
> increase for spinning rust HDD, but quite big increase for SSD (350->450
> MB/s). With disabled disk cache, or random I/O, NCQ will probably have more
> effect.

I tried sequential reads ( dd if=/dev/rwd0c ... ) and throughput took a
significant hit. I used to get about 120MB/s with the siisata, now it
fluctuates between 80 and 90MB/s, ahcisata dropped from about 80MB/s to
70MB/s. Both spinning rust of varying vintage.
I should probably do a bonnie run on either one before & after to see
if there's any change in random access.

have fun
Michael