Re: [gentoo-user] Slightly OT - My new SSD.
> I think q0 is the "administrative" queue, and the other 8 are ordinary > queues. (Sorry, I read that somewhere, but can't remember where). > Good to know. > > > but on your system i'd say there should be some queues on other > > interrupts to they can be serviced by other cores, so that doesnt look > > right. > > It wasn't right. > > > Do you have MSI enabled? Bus options -> PCI Support -> Message Signaled > > Interrupts (MSI and MSI-X) > > I didn't have MSI enabled, but do now. I now get 5 queues (nvmeq[0-4]) > on four interrups, and the interrupts are spread over the four cores > (I've got a 4 core Athlon). > > > If your system is not too old you may get more interrupts or a better > > spread with that enabled. > > Yes, that is the case. From a "warm start", I was able to copy my 1.4GB > file in 2.76s. This is a bit more like it! > Nice! > > > You look like you're getting SATA speeds, but since you have the nvme > > device, i guess that implies you havent fallen back to SATA. > > I don't think the SSD has a SATA interface. > That was a guess on my behalf - i know that M2 disks can come in SATA or NVMe, but I dont know if NVMe drives can fallback to SATA. > > > Could well be older hardware or less PCIe slots/lanes. > > My $ hdparm -tT /dev/nvme0n1 speeds haven't improved since enabling MSI > in the kernel. > Ah well, its only a simple synthetic benchmark. Adam
Re: [gentoo-user] Slightly OT - My new SSD.
On 26/03/17 03:52, Alan Mackenzie wrote: Hello, Adam. On Mon, Mar 20, 2017 at 17:09:15 +1100, Adam Carter wrote: That, indeed, seems to to be the case. When I do cat /proc/interrupts | egrep '(CPU/nvm)', I get just the header line with one data line: CPU0 CPU1 CPU2 CPU3 17: 0 0 15 14605 IO-APIC 17-fasteoi ehci_hcd:usb1, nvme0q0, nvme0q1 I'm kind of feeling a bit out of my depth here. What are the nvme0q0, etc.? "Queues" of some kind? You appear to have nine of these things, I've just got two. I'm sure there's a fine manual I ought to be I have been looking at this as my MS surface (built in nvme) has been showing two queue's similar to the above. Turns out that in order to successfully hibernate to disk (and put extra delays in the initrd) on a surface you have to boot on a single cpu (maxcpus=1) and then turn on the rest once past the resume or before the suspend stages. If booting with all cpu's enabled the queues are as expected. Is there a way to manipulate the queues themselves in a similar way to individual cpu's? BillK
Re: [gentoo-user] Slightly OT - My new SSD.
Hello, Adam. On Mon, Mar 20, 2017 at 17:09:15 +1100, Adam Carter wrote: > > That, indeed, seems to to be the case. When I do cat /proc/interrupts | > > egrep '(CPU/nvm)', I get just the header line with one data line: > >CPU0 CPU1 CPU2 CPU3 > > 17: 0 0 15 14605 IO-APIC 17-fasteoi > > ehci_hcd:usb1, nvme0q0, nvme0q1 > > I'm kind of feeling a bit out of my depth here. What are the nvme0q0, > > etc.? "Queues" of some kind? You appear to have nine of these things, > > I've just got two. I'm sure there's a fine manual I ought to be > > reading. Do you know where I might find this manual? > Can't remember where i read up on this. Might have been troubleshooting > poor small packet performance on a firewall (some network drivers can have > multiqueue too). Maybe start with this; > https://www.thomas-krenn.com/en/wiki/Linux_Multi-Queue_Block_IO_Queueing_Mechanism_(blk-mq) Interesting article. > It looks like the nvme driver was made "mutliqueue" in kernel 3.19. > FWIW my system is 8 core (AMD 8350). Its odd having two queues on the same > interrupt, but I have the same for q0 and q1, I think q0 is the "administrative" queue, and the other 8 are ordinary queues. (Sorry, I read that somewhere, but can't remember where). > but on your system i'd say there should be some queues on other > interrupts to they can be serviced by other cores, so that doesnt look > right. It wasn't right. > Do you have MSI enabled? Bus options -> PCI Support -> Message Signaled > Interrupts (MSI and MSI-X) I didn't have MSI enabled, but do now. I now get 5 queues (nvmeq[0-4]) on four interrups, and the interrupts are spread over the four cores (I've got a 4 core Athlon). > If your system is not too old you may get more interrupts or a better > spread with that enabled. Yes, that is the case. From a "warm start", I was able to copy my 1.4GB file in 2.76s. This is a bit more like it! > When I look at the entire /proc/interrupts, there are just 30 lines > > listed, and I suspect there are no more than 32 interrupt numbers > > available. Is there any way I can configure Linux to give my SSD more > > than one interrupt line to work with? > > > FWIW > > > # hdparm -tT /dev/nvme0n1 > > > /dev/nvme0n1: > > > Timing cached reads: 9884 MB in 2.00 seconds = 4945.35 MB/sec > > > Timing buffered disk reads: 4506 MB in 3.00 seconds = 1501.84 MB/sec > > I get: > > /dev/nvme0n1: > > Timing cached reads: 4248 MB in 2.00 seconds = 2124.01 MB/sec > > Timing buffered disk reads: 1214 MB in 3.00 seconds = 404.51 MB/sec > > So my "cached reads" speed is (a little under) half of yours. This is > > to be expected, since my PCIe lanes are only version 2 (and yours are > > probably version 3). > FWIW the motherboard manual says it has PCIe 2.0 x16 slots. Agree that > cache speed is likely a hardware issue. > > But the "buffered disk read" are much slower. Is > > this just the age of my PC, or might I have something suboptimally > > configured? > You look like you're getting SATA speeds, but since you have the nvme > device, i guess that implies you havent fallen back to SATA. I don't think the SSD has a SATA interface. > Could well be older hardware or less PCIe slots/lanes. My $ hdparm -tT /dev/nvme0n1 speeds haven't improved since enabling MSI in the kernel. But I'm intending to build a new machine "soon" (depending on when I can get a suitable Ryzen motherboard), and will put this NVMe SSD into the new box. I have to decide whether to get another one of these NVMe SSDs to run as RAID-1, or whether to put swap, /usr, etc. on it, and build a RAID-1 from two cheaper SATA SSDs. -- Alan Mackenzie (Nuremberg, Germany).
Re: [gentoo-user] Slightly OT - My new SSD.
> That, indeed, seems to to be the case. When I do cat /proc/interrupts | > egrep '(CPU/nvm)', I get just the header line with one data line: > >CPU0 CPU1 CPU2 CPU3 > 17: 0 0 15 14605 IO-APIC 17-fasteoi > ehci_hcd:usb1, nvme0q0, nvme0q1 > > I'm kind of feeling a bit out of my depth here. What are the nvme0q0, > etc.? "Queues" of some kind? You appear to have nine of these things, > I've just got two. I'm sure there's a fine manual I ought to be > reading. Do you know where I might find this manual? > Can't remember where i read up on this. Might have been troubleshooting poor small packet performance on a firewall (some network drivers can have multiqueue too). Maybe start with this; https://www.thomas-krenn.com/en/wiki/Linux_Multi-Queue_Block_IO_Queueing_Mechanism_(blk-mq) It looks like the nvme driver was made "mutliqueue" in kernel 3.19. FWIW my system is 8 core (AMD 8350). Its odd having two queues on the same interrupt, but I have the same for q0 and q1, but on your system i'd say there should be some queues on other interrupts to they can be serviced by other cores, so that doesnt look right. Do you have MSI enabled? Bus options -> PCI Support -> Message Signaled Interrupts (MSI and MSI-X) If your system is not too old you may get more interrupts or a better spread with that enabled. When I look at the entire /proc/interrupts, there are just 30 lines > listed, and I suspect there are no more than 32 interrupt numbers > available. Is there any way I can configure Linux to give my SSD more > than one interrupt line to work with? > > > FWIW > > # hdparm -tT /dev/nvme0n1 > > > /dev/nvme0n1: > > Timing cached reads: 9884 MB in 2.00 seconds = 4945.35 MB/sec > > Timing buffered disk reads: 4506 MB in 3.00 seconds = 1501.84 MB/sec > > I get: > > /dev/nvme0n1: > Timing cached reads: 4248 MB in 2.00 seconds = 2124.01 MB/sec > Timing buffered disk reads: 1214 MB in 3.00 seconds = 404.51 MB/sec > > So my "cached reads" speed is (a little under) half of yours. This is > to be expected, since my PCIe lanes are only version 2 (and yours are > probably version 3). FWIW the motherboard manual says it has PCIe 2.0 x16 slots. Agree that cache speed is likely a hardware issue. > But the "buffered disk read" are much slower. Is > this just the age of my PC, or might I have something suboptimally > configured? > You look like you're getting SATA speeds, but since you have the nvme device, i guess that implies you havent fallen back to SATA. Could well be older hardware or less PCIe slots/lanes.
Re: [gentoo-user] Slightly OT - My new SSD.
Hello, Adam. Thanks for the reply. On Sun, Mar 19, 2017 at 11:45:34 +1100, Adam Carter wrote: > On Sun, Mar 19, 2017 at 3:36 AM, Alan Mackenzie wrote: > > I've just bought myself a Samsung NVMe 960 EVO M.2 SSD. > > > > Some timings: > > > > An emerge -puND @world (when there's nothing to merge) took 38.5s. With > > my mirrored HDDs, this took 45.6s. (Though usually it takes nearer a > > minute.) > > > > An emerge of Firefox took 34m23s, compared with 37m34s with the HDDs. > > The lack of the sound of the HDD heads moving was either disconcerting > > or a bit of a relief, I'm not sure which. > > > > Copying my email spool file (~110,000 entries, ~1.4 GB) from SSD -> SSD > > took 6.1s. From HDD RAID -> HDD RAID it took 30.0s. > > > > > > > I was also hoping for more speed up when i got mine, but of course it only > helps with the system is IO bound. Its great for loading VMs. > It may be mandatory with NVM, but you can check multiqueue is setup/working > with; > # cat /proc/interrupts | egrep '(CPU|nvm)' > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 > CPU6 CPU7 > 30: 21596 0 0 0 0 0 >0 0 PCI-MSI 1572864-edge nvme0q0, nvme0q1 > 40: 0 12195 0 0 0 0 >0 0 PCI-MSI 1572865-edge nvme0q2 > 41: 0 0 12188 0 0 0 >0 0 PCI-MSI 1572866-edge nvme0q3 > 42: 0 0 0 13696 0 0 >0 0 PCI-MSI 1572867-edge nvme0q4 > 43: 0 0 0 0 11698 0 >0 0 PCI-MSI 1572868-edge nvme0q5 > 44: 0 0 0 0 0 45820 >0 0 PCI-MSI 1572869-edge nvme0q6 > 45: 0 0 0 0 0 0 > 10917 0 PCI-MSI 1572870-edge nvme0q7 > 46: 0 0 0 0 0 0 >0 12865 PCI-MSI 1572871-edge nvme0q8 > If its not setup there'll be just a single IRQ/core handling all the IO. That, indeed, seems to to be the case. When I do cat /proc/interrupts | egrep '(CPU/nvm)', I get just the header line with one data line: CPU0 CPU1 CPU2 CPU3 17: 0 0 15 14605 IO-APIC 17-fasteoi ehci_hcd:usb1, nvme0q0, nvme0q1 I'm kind of feeling a bit out of my depth here. What are the nvme0q0, etc.? "Queues" of some kind? You appear to have nine of these things, I've just got two. I'm sure there's a fine manual I ought to be reading. Do you know where I might find this manual? When I look at the entire /proc/interrupts, there are just 30 lines listed, and I suspect there are no more than 32 interrupt numbers available. Is there any way I can configure Linux to give my SSD more than one interrupt line to work with? > FWIW > # hdparm -tT /dev/nvme0n1 > /dev/nvme0n1: > Timing cached reads: 9884 MB in 2.00 seconds = 4945.35 MB/sec > Timing buffered disk reads: 4506 MB in 3.00 seconds = 1501.84 MB/sec I get: /dev/nvme0n1: Timing cached reads: 4248 MB in 2.00 seconds = 2124.01 MB/sec Timing buffered disk reads: 1214 MB in 3.00 seconds = 404.51 MB/sec So my "cached reads" speed is (a little under) half of yours. This is to be expected, since my PCIe lanes are only version 2 (and yours are probably version 3). But the "buffered disk read" are much slower. Is this just the age of my PC, or might I have something suboptimally configured? -- Alan Mackenzie (Nuremberg, Germany).
Re: [gentoo-user] Slightly OT - My new SSD.
On 03/18/2017 09:36 AM, Alan Mackenzie wrote: > Hello, Gentoo. > > I've just bought myself a Samsung NVMe 960 EVO M.2 SSD. (Do I get a > prize for the number of inscruitable abbreviations in a row? ;-) The > idea is, this will form a core part of my new machine, just as soon as > AMD Ryzen motherboards start being reasonably available. > I recently bought a ThinkPad P70 (it has two m.2 nvme ports.) I have a 960 EVO, decided to order the WD Black 250G nvme for Windoze so they have their separate space. The WD Black is supposed to be slower than the EVO but I don't use Windows all that often anymore. The price was quite a bit cheaper than the EVO. I did install the EVO and the kernel I compiled already had support, so I didn't have to do anything special. It did see it no problem, I think I went to partition it but stopped - I am still waiting for the other one to arrive so I can reconfigure my laptop and fiddle with them a bit. Dan
Re: [gentoo-user] Slightly OT - My new SSD.
On Sun, Mar 19, 2017 at 3:36 AM, Alan Mackenzie wrote: > Hello, Gentoo. > > I've just bought myself a Samsung NVMe 960 EVO M.2 SSD. > > Some timings: > > An emerge -puND @world (when there's nothing to merge) took 38.5s. With > my mirrored HDDs, this took 45.6s. (Though usually it takes nearer a > minute.) > > An emerge of Firefox took 34m23s, compared with 37m34s with the HDDs. > The lack of the sound of the HDD heads moving was either disconcerting > or a bit of a relief, I'm not sure which. > > Copying my email spool file (~110,000 entries, ~1.4 GB) from SSD -> SSD > took 6.1s. From HDD RAID -> HDD RAID it took 30.0s. > > > I was also hoping for more speed up when i got mine, but of course it only helps with the system is IO bound. Its great for loading VMs. It may be mandatory with NVM, but you can check multiqueue is setup/working with; # cat /proc/interrupts | egrep '(CPU|nvm)' CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 30: 21596 0 0 0 0 0 0 0 PCI-MSI 1572864-edge nvme0q0, nvme0q1 40: 0 12195 0 0 0 0 0 0 PCI-MSI 1572865-edge nvme0q2 41: 0 0 12188 0 0 0 0 0 PCI-MSI 1572866-edge nvme0q3 42: 0 0 0 13696 0 0 0 0 PCI-MSI 1572867-edge nvme0q4 43: 0 0 0 0 11698 0 0 0 PCI-MSI 1572868-edge nvme0q5 44: 0 0 0 0 0 45820 0 0 PCI-MSI 1572869-edge nvme0q6 45: 0 0 0 0 0 0 10917 0 PCI-MSI 1572870-edge nvme0q7 46: 0 0 0 0 0 0 0 12865 PCI-MSI 1572871-edge nvme0q8 If its not setup there'll be just a single IRQ/core handling all the IO. FWIW # hdparm -tT /dev/nvme0n1 /dev/nvme0n1: Timing cached reads: 9884 MB in 2.00 seconds = 4945.35 MB/sec Timing buffered disk reads: 4506 MB in 3.00 seconds = 1501.84 MB/sec
