Re: nvme SSD and poor performance

[piorunz]

On 23/08/2021 13:09, Alexander V. Makartsev wrote:

If you don't have a NVMe-capable Windows PC at hand, you can download an
evaluation version of Windows 10 from Microsoft¹ and install it on
separate storage device, just to update SSD firmware.
Fill the form with gibberish. Microsoft Product Key or Activation are
not required. Evaluation copy will work for 90 days.


It can be even easier:

Download Windows 10 ISO from here
https://www.microsoft.com/en-gb/software-download/windows10iso

put on USB stick, do 10 minutes install, do you thing and nuke it. No
need to register anything. Do it while internet is disconnected, if will
install even faster.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄

Re: nvme SSD and poor performance

[Alexander V. Makartsev]

On 23.08.2021 13:46, Pierre Willaime wrote:

Le 20/08/2021 à 19:45, Alexander V. Makartsev a écrit :

That looks abnormal to me. Have you tried to update SSD firmware¹ and 
BIOS for your Dell PC?


[1] 
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=0n4c4

Thanks all again for your precious answers.

I will try first to update Bios and SSD firmwares.

For the ssd, instructions (see link below) are for windows only.

Is there a way to update dell ssd firmware without taking the ssd out 
and plug it in into a windows machine?


In this particular case, I don't thinks so, because it looks like SSD 
firmware and flasher are fused together into a single PE file and I 
don't think WINE will be able to handle it properly.
There is a possibility that extracted "23201112_WIN.EXE" file will work 
under WinPE / WinRE environments booted from a USB flash drive, but I 
wouldn't count on it.


If you don't have a NVMe-capable Windows PC at hand, you can download an 
evaluation version of Windows 10 from Microsoft¹ and install it on 
separate storage device, just to update SSD firmware.
Fill the form with gibberish. Microsoft Product Key or Activation are 
not required. Evaluation copy will work for 90 days.



[1] 
https://www.microsoft.com/en-us/evalcenter/evaluate-windows-10-enterprise


--
With kindest regards, Alexander.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄

Re: nvme SSD and poor performance

[Pierre Willaime]

Le 20/08/2021 à 19:45, Alexander V. Makartsev a écrit :

That looks abnormal to me. Have you tried to update SSD firmware¹ and 
BIOS for your Dell PC?


[1] 
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=0n4c4

Thanks all again for your precious answers.

I will try first to update Bios and SSD firmwares.

For the ssd, instructions (see link below) are for windows only.

Is there a way to update dell ssd firmware without taking the ssd out 
and plug it in into a windows machine?

Re: nvme SSD and poor performance

[Sven Hartge]

Anssi Saari  wrote:
> Jochen Spieker  writes:

>> Stay away from the "discard" option and do not worry about SSD life.

> What's the issue with the discard option? AFAIK, there may have been
> issues with it in the decade before last but again AFAIK, today some
> distros enable discard and some run fstrim on timer, both work.

> But please share.

This may be part cargo culting and part real problem.

In the past it was shown that many SSD needed to flush their whole queue
whenever a TRIM command was received and the Kernel was really zealos of
sending those for every block freed. 

This basically neutered many SSD and even caused corruption issues on
some especially shoddy ones.

Some firmwares got fixed and newer drives don't suffer from the same
problems, but even now end then you can find some devices on the lower
end of the consumer spectrum behaving poorly when bombarded with TRIM
commands.

So in the end the consensous was to just play it safe and batch-TRIM
once a week or day, because TRIMming on demand hasn't been shown to be a
big win.

It is far easier to fall back on that option than trying to
good/bad-list specific drive or firmware versions and select the
appropriate TRIM method.

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.

Re: nvme SSD and poor performance

[David Christensen]

On 8/17/21 2:54 AM, Pierre Willaime wrote:
> I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
> (bulleye now).
>
> For a long time, I suffer poor I/O performances which slow down a lot of
> tasks (apt upgrade when unpacking for example).


On 8/20/21 1:50 AM, Pierre Willaime wrote:

Thanks all.

I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).

But I do not think my issue is trim related after all. 



I agree.


I have always a 
lot of I/O activities from jdb2 even just after booting and even when 
the computer is doing nothing for hours.


Here is an extended  log of iotop where you can see jdb2 anormal 
activities: https://pastebin.com/eyGcGdUz



Was your command something like this?

# iotop -o -k -t -q -q -q > iotop.out


I cannot (yet) find what process is generating this activities. I tried 
to kill a lot of jobs seing in atop output with no results.



Analyzing the first ten minutes worth of data with an improvised Perl 
script:


index of field 'read_bw'
1256840.19 firefox-esr
  77926.07 apt
316.08 perl
 22.74 dpkg
 15.27 [kworker/6:0-events]
index of field 'write_bw'
 220512.79 thunderbird
  29613.87 firefox-esr
  23221.20 dpkg
  15211.66 [jbd2/nvme0n1p2-]
   5529.57 [dpkg]
   4148.09 systemd-journald
   1699.13 perl
533.28 mandb
507.15 apt
145.61 rsyslogd
131.59 atop
115.77 syncthing
 46.35 xfce4-terminal
 38.60 smartd
 15.48 Xorg
 15.44 NetworkManager
  7.64 bash
index of field 'swap_percent'
index of field 'io_wait_percent'
  12427.58 [jbd2/nvme0n1p2-]
   1334.15 firefox-esr
568.91 dpkg
385.31 thunderbird
293.57 mandb
 99.99 syncthing
 64.82 [kworker/13:3-events_freezable_power_]
 63.86 smartd
 55.12 [kworker/u32:3+flush-259:0]
 38.64 [kworker/u32:2-flush-259:0]
 25.27 [kworker/u32:3-events_unbound]
 23.13 [kworker/4:0-events_freezable_power_]
 22.68 [kworker/u32:2-events_unbound]
 21.55 apt
 12.51 [kworker/u32:1-ext4-rsv-conversion]
  9.87 [kworker/u32:2-ext4-rsv-conversion]
  9.63 [kworker/9:1-events]
  8.90 [kworker/u32:1-flush-259:0]
  8.58 perl
  8.11 [kworker/9:1-events_freezable_power_]
  5.85 [dpkg]
  4.33 [kworker/u32:3-ext4-rsv-conversion]
  4.26 NetworkManager
  3.57 [kworker/13:3-mm_percpu_wq]
  2.85 [kworker/9:1-mm_percpu_wq]
  2.71 [kworker/4:0-mm_percpu_wq]
  0.40 [kworker/13:3-events] [kworker/4:0-events]
  0.38 [kworker/6:1-events]
  0.36 [kworker/9:1-rcu_gp]
  0.30 [kworker/u32:3-flush-259:0]
  0.26 [kworker/6:0-events_freezable_power_]
  0.16 systemd-journald
  0.03 [kworker/6:0-events]


I appears:

- firefox-esr used the most read bandwidth -- 1256840.19 K/s total

- thunderbird used the most write bandwdith -- 220512.79 K/s total

- No processes were swapping.

- jbd2/nvme0n1p2- waited the longest for I/O -- 12427.58 % total


Both apt(8) and dpkg(1) were also running and using a small amount of 
I/O.  While I may leave Firefox and Thunderbird running when installing 
a package or two, I shut them down for updates and upgrades.  Was the 
iotop data collected during a long-running upgrade?



AIUI the jbd2/nvme0n1p2 I/O corresponds to the bottom half of the kernel 
(e.g. device driver interface, DDI) in response to I/O via the top half 
of the kernel (e.g. application programming interface, API).  The way to 
reduce jdb2 I/O is to reduce application I/O.



> On 8/17/21 7:07 AM, Dan Ritter wrote:
>> I don't think you have a significant performance problem, but
>> you are definitely feeling some pain -- so can you tell us more
>> about what feels slow? Does it happen during the ordinary course
>> of the day?

Program are slow to start. Sometimes there is a delay when I type 
(letters are displayed few second after typing). Apt unpack take forewer 
(5 hours to unpack packages when upgrading to bulleye).


The computer is a recent Dell precision desktop with i9-9900 as CPU, an 
NVIDIA GP107GL [Quadro P400] (and the GPU integrated to the CPU). The 
nvme SSD is supposed to be a decent one. This desktop is yet a lot 
slower than my (more basic) laptop.


Complete system info: https://pastebin.com/zaGVEpae



That's a good workstation.  :-)


Firefox and Thunderbird are habitual trouble-makers on my Debian 
desktops.  I run Xfce with the CPU Graph panel applet.  If I leave 
Firefox or Thunderbird open long enough, eventually I will see a core 
pegged at 100% and the CPU fan will spin up.  Both applications keep 
working in this state; but a swirling toilet bowl mouse pointer in 
Thunderbird is a danger sign -- I have lost e-mail messages when moving 
a message produced that symptom.  The only cure is to close the 
offending program(s) and implement 

Re: nvme SSD and poor performance

[piorunz]

On 20/08/2021 18:11, Marco Möller wrote:

browser.sessionstore.interval
Storing the session is not the only data written to the disk. If it
would only be this data, then indeed setting a higher interval would be
sufficient. But there is much more going on. Especially the cache seems
to be a cause for the extreme high I/O of Firefox. That's why I finally
decided to go for the psd tool, and this tool made it for me.


That's why you can tune disk cache in Firefox, disable it, make is
smaller and so on. By definition disk cache for a browser is a hot cache
where everything lands and it's being taken from again instead of being
downloaded from internet. On my PC I have /home on RAID10 made of 4
spinning drives, so all programs can cache and thrash on the storage all
day. If you, or anyone else in this topic, have /home on SSD and/or
Firefox/TB data churn is a concern, you could disable Fx disk cache
altogether.
I am glad that solution you have chosen, that psd tool (I don't know it)
work for you.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄

Re: nvme SSD and poor performance

[Alexander V. Makartsev]

On 20.08.2021 13:50, Pierre Willaime wrote:


Program are slow to start. Sometimes there is a delay when I type 
(letters are displayed few second after typing). Apt unpack take 
forewer (5 hours to unpack packages when upgrading to bulleye).


That looks abnormal to me. Have you tried to update SSD firmware¹ and 
BIOS for your Dell PC?
LiteOn Storage branch was bought by KIOXIA (Toshiba), so I can't find 
any official information about technical specifications of SSD you have, 
but just by looking at its photos, I assume it is an average one, with 
external cache and TLC NAND chips.
The performance of a NVMe looks like it working at slower x2 speed, 
instead of utilizing full x4 lanes. But it could be also its normal 
working state and delays\slowdowns could come from parts unrelated to 
NVMe, like SATA HDDs.

A "smartctl" output for all storage devices could be useful.

The computer is a recent Dell precision desktop with i9-9900 as CPU, 
an NVIDIA GP107GL [Quadro P400] (and the GPU integrated to the CPU). 
The nvme SSD is supposed to be a decent one. This desktop is yet a lot 
slower than my (more basic) laptop.


Complete system info: https://pastebin.com/zaGVEpae




[1] 
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=0n4c4


--
With kindest regards, Alexander.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄

Re: nvme SSD and poor performance

[Tixy]

On Fri, 2021-08-20 at 17:48 +0200, Christian Britz wrote:
> 
> Am 20.08.21 um 10:50 schrieb Pierre Willaime:
> > 
> > I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).
> > 
> Interesting. I am almost 100% certain that it was enabled for me by 
> default on bullseye. Maybe that behaviour changed during the release 
> process.

It's enabled on my PC which was a fresh install in December using a
Bullseye release candidate installer.

-- 
Tixy

Re: nvme SSD and poor performance

[Linux-Fan]

Pierre Willaime writes:


Thanks all.

I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).


You're welcome :)

But I do not think my issue is trim related after all. I have always a lot  
of I/O activities from jdb2 even just after booting and even when the  
computer is doing nothing for hours.


Here is an extended  log of iotop where you can see jdb2 anormal activities:  
https://pastebin.com/eyGcGdUz


According to that, a lot of firefox-esr and dpkg and some thunderbird  
processes are active. Is there a high intensity of I/O operations when all  
Firefox, Thunderbird instances and system upgrades are closed?


When testing with iotop here, options `-d 10 -P` seemed to help getting a  
steadier and less cluttered view. Still, filtering your iotop output for  
Firefox, Thunderbird and DPKG respectively seems to be quite revealing:


| $ grep firefox-esr eyGcGdUz | grep -E '[0-9]{4,}.[0-9]{2} K/s'
| 10:38:513363 be/4 pierre  0.00 K/s 1811.89 K/s  0.00 % 17.64 % 
firefox-esr [mozStorage #3]
| 10:39:585117 be/4 pierre  0.00 K/s 1112.59 K/s  0.00 %  0.37 % 
firefox-esr [IndexedDB #14]
| 10:41:553363 be/4 pierre  0.00 K/s 6823.06 K/s  0.00 %  0.00 % 
firefox-esr [mozStorage #3]
| 10:41:553305 be/4 pierre   1469.88 K/s0.00 K/s  0.00 % 60.57 % 
firefox-esr [QuotaManager IO]
| 10:41:553363 be/4 pierre   6869.74 K/s 6684.07 K/s  0.00 % 31.96 % 
firefox-esr [mozStorage #3]
| 10:41:566752 be/4 pierre   2517.19 K/s0.00 K/s  0.00 % 99.99 % 
firefox-esr [Indexed~Mnt #13]
| 10:41:566755 be/4 pierre   31114.18 K/s0.00 K/s  0.00 % 99.58 % 
firefox-esr [Indexed~Mnt #16]
| 10:41:563363 be/4 pierre   9153.40 K/s0.00 K/s  0.00 % 87.06 % 
firefox-esr [mozStorage #3]
| 10:41:576755 be/4 pierre   249206.18 K/s0.00 K/s  0.00 % 59.01 % 
firefox-esr [Indexed~Mnt #16]
| 10:41:576755 be/4 pierre   251353.11 K/s0.00 K/s  0.00 % 66.02 % 
firefox-esr [Indexed~Mnt #16]
| 10:41:586755 be/4 pierre   273621.58 K/s0.00 K/s  0.00 % 59.51 % 
firefox-esr [Indexed~Mnt #16]
| 10:41:586755 be/4 pierre   51639.70 K/s0.00 K/s  0.00 % 94.90 % 
firefox-esr [Indexed~Mnt #16]
| 10:41:596755 be/4 pierre   113869.64 K/s0.00 K/s  0.00 % 79.03 % 
firefox-esr [Indexed~Mnt #16]
| 10:41:596755 be/4 pierre   259549.09 K/s0.00 K/s  0.00 % 56.99 % 
firefox-esr [Indexed~Mnt #16]
| 10:44:413265 be/4 pierre   1196.21 K/s0.00 K/s  0.00 % 20.89 % 
firefox-esr
| 10:44:413289 be/4 pierre   3813.36 K/s  935.22 K/s  0.00 %  4.59 % 
firefox-esr [Cache2 I/O]
| 10:44:533363 be/4 pierre  0.00 K/s 1176.90 K/s  0.00 %  0.00 % 
firefox-esr [mozStorage #3]
| 10:49:283363 be/4 pierre  0.00 K/s 1403.16 K/s  0.00 %  0.43 % 
firefox-esr [mozStorage #3]

So there are incredible amounts of data being read by Firefox (Gigabytes in  
a few minutes)? Does this load reflect in atop or iotop's summarizing lines  
at the begin of the respective screens?


| $ grep thunderbird eyGcGdUz | grep -E '[0-9]{4,}.[0-9]{2} K/s'
| 10:38:432846 be/4 pierre  0.00 K/s 1360.19 K/s  0.00 % 15.51 % 
thunderbird [mozStorage #1]
| 10:39:492873 be/4 pierre  0.00 K/s 4753.74 K/s  0.00 %  0.00 % 
thunderbird [mozStorage #6]
| 10:39:492875 be/4 pierre  0.00 K/s 19217.56 K/s  0.00 %  0.00 % 
thunderbird [mozStorage #7]
| 10:39:502883 be/4 pierre  0.00 K/s 18014.56 K/s  0.00 % 29.39 % 
thunderbird [mozStorage #8]
| 10:39:502883 be/4 pierre  0.00 K/s 3305.94 K/s  0.00 % 27.28 % 
thunderbird [mozStorage #8]
| 10:39:512883 be/4 pierre  0.00 K/s 61950.19 K/s  0.00 % 63.11 % 
thunderbird [mozStorage #8]
| 10:39:512883 be/4 pierre  0.00 K/s 41572.77 K/s  0.00 % 27.19 % 
thunderbird [mozStorage #8]
| 10:39:522883 be/4 pierre  0.00 K/s 20961.20 K/s  0.00 % 65.02 % 
thunderbird [mozStorage #8]
| 10:39:522883 be/4 pierre  0.00 K/s 43345.16 K/s  0.00 %  0.19 % 
thunderbird [mozStorage #8]
| 10:42:272846 be/4 pierre  0.00 K/s 1189.63 K/s  0.00 %  0.45 % 
thunderbird [mozStorage #1]
| 10:42:332846 be/4 pierre  0.00 K/s 1058.52 K/s  0.00 %  0.31 % 
thunderbird [mozStorage #1]
| 10:47:272846 be/4 pierre  0.00 K/s 2113.53 K/s  0.00 %  0.66 % 
thunderbird [mozStorage #1]

Thunderbird seems to write a lot here. This would average at ~18 MiB/s of writing  
and hence explain why the SSD is loaded continuously. Again: Does it match the  
data reported by atop? [I am not experienced in reading iotop output, hence  
I might interpret the data wrongly].


By comparison, dpkg looks rather harmless:

| $ grep dpkg eyGcGdUz | grep -E '[0-9]{4,}.[0-9]{2} K/s'
| 10:38:254506 be/4 root0.00 K/s 4553.67 K/s  0.00 %  0.26 % dpkg 
--status-fd 23 --no-triggers --unpack --auto-deconfigure 
--force-remove-protected --recursive /tmp/apt-dpkg-install-E69bfZ
| 10:38:334506 be/4 root7.73 K/s 4173.77 K/s  0.00 %  1.52 % dpkg 
--status-fd 23 --no-triggers --unpack --auto-deconfigure 

Re: nvme SSD and poor performance

[Marco Möller]

On 20.08.21 18:22, piorunz wrote:

On 17/08/2021 15:03, Marco Möller wrote:

I have no experience with SSD, but running my Debian Desktop from a USB
Memory Stick since years, please allow me to share information which
supports the suggestion of Linux-Fan to also investigate if there is
extraordinary I/O taking place and maybe could be avoided:
In the past I found extreme(!) I/O to be produced by Firefox, when it is
writing its cache, and when it is writing its session restore
information. These writes to my observation occure all the time, kind of
nonstop! I could get it very satisfactorily reduced by applying a tool
called "Profile Sync Daemon" (psd) from package "profile-sync-daemon".
While it is packaged for Debian, as a starting point to study its
documentation I suggest to better look it up in the Arch Linux Wiki.


Firefox setting which determines how often write session data to disk is:
browser.sessionstore.interval

Default setting is value of 15000 = 15 seconds, not that bad. But still
I changed that to 10 minutes (value of 60). If I lose an open tab
once in a year when Firefox crashes, so be it. To be honest, I haven't
seen Fx crash in a year, or more.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄



browser.sessionstore.interval
Storing the session is not the only data written to the disk. If it 
would only be this data, then indeed setting a higher interval would be 
sufficient. But there is much more going on. Especially the cache seems 
to be a cause for the extreme high I/O of Firefox. That's why I finally 
decided to go for the psd tool, and this tool made it for me. Well, I 
assume it has good reasons that someone programmed psd, probably the 
author also noticed that with Firefox parameters alone it often is not 
enough to get the I/O to the HDD caused by Firefox significantly reduced?


I am curious to see what's producing the high jdb2 traffic on the system 
of Pierre, and if this can be stopped and if this finally can be 
attributed to have been the cause for the trouble.


Unfortunately I did not make good notes on how I searched for what was 
causing the traffic on my system, I only remember to have monitored my 
system with iotop and inotifywait.


---
Always stay in good spirits!
Marco

Re: nvme SSD and poor performance

[piorunz]

On 17/08/2021 15:03, Marco Möller wrote:

I have no experience with SSD, but running my Debian Desktop from a USB
Memory Stick since years, please allow me to share information which
supports the suggestion of Linux-Fan to also investigate if there is
extraordinary I/O taking place and maybe could be avoided:
In the past I found extreme(!) I/O to be produced by Firefox, when it is
writing its cache, and when it is writing its session restore
information. These writes to my observation occure all the time, kind of
nonstop! I could get it very satisfactorily reduced by applying a tool
called "Profile Sync Daemon" (psd) from package "profile-sync-daemon".
While it is packaged for Debian, as a starting point to study its
documentation I suggest to better look it up in the Arch Linux Wiki.


Firefox setting which determines how often write session data to disk is:
browser.sessionstore.interval

Default setting is value of 15000 = 15 seconds, not that bad. But still
I changed that to 10 minutes (value of 60). If I lose an open tab
once in a year when Firefox crashes, so be it. To be honest, I haven't
seen Fx crash in a year, or more.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄

Re: nvme SSD and poor performance

[Christian Britz]

Am 20.08.21 um 10:50 schrieb Pierre Willaime:


I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).

Interesting. I am almost 100% certain that it was enabled for me by 
default on bullseye. Maybe that behaviour changed during the release 
process.


Best Regards,
Christian

Re: nvme SSD and poor performance

[Pierre Willaime]

Thanks all.

I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).

But I do not think my issue is trim related after all. I have always a 
lot of I/O activities from jdb2 even just after booting and even when 
the computer is doing nothing for hours.


Here is an extended  log of iotop where you can see jdb2 anormal 
activities: https://pastebin.com/eyGcGdUz


I cannot (yet) find what process is generating this activities. I tried 
to kill a lot of jobs seing in atop output with no results.



I don't think you have a significant performance problem, but
you are definitely feeling some pain -- so can you tell us more
about what feels slow? Does it happen during the ordinary course
of the day?


Program are slow to start. Sometimes there is a delay when I type 
(letters are displayed few second after typing). Apt unpack take forewer 
(5 hours to unpack packages when upgrading to bulleye).


The computer is a recent Dell precision desktop with i9-9900 as CPU, an 
NVIDIA GP107GL [Quadro P400] (and the GPU integrated to the CPU). The 
nvme SSD is supposed to be a decent one. This desktop is yet a lot 
slower than my (more basic) laptop.


Complete system info: https://pastebin.com/zaGVEpae




Le 18/08/2021 à 00:24, David Christensen a écrit :

On 8/17/21 2:54 AM, Pierre Willaime wrote:
 > Hi,
 >
 > I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
 > (bulleye now).
 >
 > For a long time, I suffer poor I/O performances which slow down a lot of
 > tasks (apt upgrade when unpacking for example).
 >
 > I am now trying to fix this issue.
 >
 > Using fstrim seems to restore speed. There are always many GiB which are
 > reduced :
 >
 >  #  fstrim -v /
 >  / : 236,7 GiB (254122389504 octets) réduits
 >
 > then, directly after :
 >
 >  #  fstrim -v /
 >  / : 0 B (0 octets) réduits
 >
 > but few minutes later, there are already 1.2 Gib to trim again :
 >
 >  #  fstrim -v /
 >  / : 1,2 GiB (1235369984 octets) réduits
 >
 >
 > /Is it a good idea to trim, if yes how (and how often)?/
 >
 > Some people use fstrim as a cron job, some other add "discard" option to
 > the /etc/fstab / line. I do not know what is the best if any. I also
 > read triming frequently could reduce the ssd life.
 >
 >
 >
 > I also noticed many I/O access from jbd2 and kworker such as :
 >
 >  # iotop -bktoqqq -d .5
 >  11:11:16 364 be/3 root    0.00 K/s    7.69 K/s  0.00 %
 > 23.64 % [jbd2/nvme0n1p2-]
 >  11:11:16   8 be/4 root    0.00 K/s    0.00 K/s  0.00 %
 > 25.52 % [kworker/u32:0-flush-259:0]
 >
 > The percentage given by iotop (time the thread/process spent while
 > swapping in and while waiting on I/O) is often high.
 >
 > I do not know what to do for kworker and if it is a normal behavior. For
 > jdb2, I have read it is filesystem (ext4 here) journal.
 >
 > I added the "noatime" option to /etc/fstab / line but it does not seem
 > to reduce the number of access.
 >
 > Regards,
 > Pierre
 >
 >
 > P-S: If triming it is needed for ssd, why debian do not trim by default?



On 8/17/21 6:14 AM, Pierre Willaime wrote:

Le 17/08/2021 à 14:02, Dan Ritter a écrit :

The first question is, how slow is this storage?


Here is a good article on using fio:
https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/ 



Thanks for the help.

Here are the output of fio tests.


Single 4KiB random write process test:

WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), 
io=12.0GiB (12.9GB), run=62271-62271msec


https://pastebin.com/5Cyg9Xvt


16 parallel 64KiB random write processes (two different results, 
further tests are closer to the second than the first):


WRITE: bw=523MiB/s (548MB/s), 31.8MiB/s-33.0MiB/s (33.4MB/s-35.6MB/s), 
io=35.5GiB (38.1GB), run=63568-69533msec


WRITE: bw=201MiB/s (211MB/s), 11.9MiB/s-14.8MiB/s (12.5MB/s-15.5MB/s), 
io=14.3GiB (15.3GB), run=60871-72618msec



https://pastebin.com/XVpPpqsC
https://pastebin.com/HEx8VvhS


Single 1MiB random write process:

   WRITE: bw=270MiB/s (283MB/s), 270MiB/s-270MiB/s (283MB/s-283MB/s), 
io=16.0GiB (17.2GB), run=60722-60722msec


https://pastebin.com/skk6mi7M




Thank you for posting your fio(1) runs on pastebin -- it is far easier 
to comment on real data.  :-)



It would help if you told us:

1.  Make and model of computer (or motherboard and chassis, if DIY).

2.  Make and model of CPU.

3.  Quantity, make, and model of memory modules, and how your memory 
slots are populated.


4.  NVMe drive partitioning, formatting, space usage, etc..


STFW "CAZ-82512-Q11 NVMe LITEON 512GB", that looks like a decent desktop 
NVMe drive.



Looking at your fio(1) runs:

1. It appears that you held 6 parameters constant:

 --name=random-write
 --ioengine=posixaio
 --rw=randwrite
 --runtime=60
 --time_based
 --end_fsync=1

2.  It appears that you varied 4 parameters:

 run    bs    size    numjobs    iodepth
 1    4k    4g    1 

Re: nvme SSD and poor performance

[Anssi Saari]

Jochen Spieker  writes:

> Stay away from the "discard" option and do not worry about SSD life.

What's the issue with the discard option? AFAIK, there may have been
issues with it in the decade before last but again AFAIK, today some
distros enable discard and some run fstrim on timer, both work.

But please share.

Re: nvme SSD and poor performance

[Jochen Spieker]

Pierre Willaime:
> 
> Using fstrim seems to restore speed. There are always many GiB which are
> reduced :
> 
>   #  fstrim -v /
>   / : 236,7 GiB (254122389504 octets) réduits

This is probably the total amount of unused space on that SSD. The first
fstrim run after a reboot always trims all free space, as far as I
understand. After that, only space that was freed since the last fstrim
run will be trimmed.

> then, directly after :
> 
>   #  fstrim -v /
>   / : 0 B (0 octets) réduits

See above, this is expected.

> but few minutes later, there are already 1.2 Gib to trim again :
> 
>   #  fstrim -v /
>   / : 1,2 GiB (1235369984 octets) réduits

It is odd that something wrote and deleted 1.2 GB in "a few minutes". I
would run iotop to see what process is responsible.

> /Is it a good idea to trim, if yes how (and how often)?/

See the other answers. Usually once per week is enough, but that assumes
that the free space is not overwritten and freed again several times
each day.

On the other hand, personally I have never felt a difference in
performance before and after an fstrim run.

> Some people use fstrim as a cron job, some other add "discard" option to the
> /etc/fstab / line. I do not know what is the best if any. I also read
> triming frequently could reduce the ssd life.

Stay away from the "discard" option and do not worry about SSD life.
Even with heavy usage you should not exceed the expected lifetime
writes. Do worry about data loss in general and create a viable backup
strategy, independent of the storage medium (or service) that you are
using.

> I also noticed many I/O access from jbd2 and kworker such as :
> 
>   # iotop -bktoqqq -d .5
>   11:11:16 364 be/3 root0.00 K/s7.69 K/s  0.00 % 23.64 %
> [jbd2/nvme0n1p2-]
>   11:11:16   8 be/4 root0.00 K/s0.00 K/s  0.00 % 25.52 %
> [kworker/u32:0-flush-259:0]

jbd2 is the journalling service of the filesystem and makes sure that
your filesystem does not become corrupt in case of power failures. 

kworkers are general purpose kernel threads doing kernel stuff, for
example disk I/O. Seeing these is not a problem in itself.

> 

-- 
I spend money without thinking on products and clothes that I believe
will enhance my social standing.
[Agree]   [Disagree]
 


signature.asc
Description: PGP signature

Re: nvme SSD and poor performance

[David Christensen]

On 8/17/21 2:54 AM, Pierre Willaime wrote:
> Hi,
>
> I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
> (bulleye now).
>
> For a long time, I suffer poor I/O performances which slow down a lot of
> tasks (apt upgrade when unpacking for example).
>
> I am now trying to fix this issue.
>
> Using fstrim seems to restore speed. There are always many GiB which are
> reduced :
>
>  #  fstrim -v /
>  / : 236,7 GiB (254122389504 octets) réduits
>
> then, directly after :
>
>  #  fstrim -v /
>  / : 0 B (0 octets) réduits
>
> but few minutes later, there are already 1.2 Gib to trim again :
>
>  #  fstrim -v /
>  / : 1,2 GiB (1235369984 octets) réduits
>
>
> /Is it a good idea to trim, if yes how (and how often)?/
>
> Some people use fstrim as a cron job, some other add "discard" option to
> the /etc/fstab / line. I do not know what is the best if any. I also
> read triming frequently could reduce the ssd life.
>
>
>
> I also noticed many I/O access from jbd2 and kworker such as :
>
>  # iotop -bktoqqq -d .5
>  11:11:16 364 be/3 root0.00 K/s7.69 K/s  0.00 %
> 23.64 % [jbd2/nvme0n1p2-]
>  11:11:16   8 be/4 root0.00 K/s0.00 K/s  0.00 %
> 25.52 % [kworker/u32:0-flush-259:0]
>
> The percentage given by iotop (time the thread/process spent while
> swapping in and while waiting on I/O) is often high.
>
> I do not know what to do for kworker and if it is a normal behavior. For
> jdb2, I have read it is filesystem (ext4 here) journal.
>
> I added the "noatime" option to /etc/fstab / line but it does not seem
> to reduce the number of access.
>
> Regards,
> Pierre
>
>
> P-S: If triming it is needed for ssd, why debian do not trim by default?



On 8/17/21 6:14 AM, Pierre Willaime wrote:

Le 17/08/2021 à 14:02, Dan Ritter a écrit :

The first question is, how slow is this storage?


Here is a good article on using fio:
https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/ 



Thanks for the help.

Here are the output of fio tests.


Single 4KiB random write process test:

WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), 
io=12.0GiB (12.9GB), run=62271-62271msec


https://pastebin.com/5Cyg9Xvt


16 parallel 64KiB random write processes (two different results, further 
tests are closer to the second than the first):


WRITE: bw=523MiB/s (548MB/s), 31.8MiB/s-33.0MiB/s (33.4MB/s-35.6MB/s), 
io=35.5GiB (38.1GB), run=63568-69533msec


WRITE: bw=201MiB/s (211MB/s), 11.9MiB/s-14.8MiB/s (12.5MB/s-15.5MB/s), 
io=14.3GiB (15.3GB), run=60871-72618msec



https://pastebin.com/XVpPpqsC
https://pastebin.com/HEx8VvhS


Single 1MiB random write process:

   WRITE: bw=270MiB/s (283MB/s), 270MiB/s-270MiB/s (283MB/s-283MB/s), 
io=16.0GiB (17.2GB), run=60722-60722msec


https://pastebin.com/skk6mi7M




Thank you for posting your fio(1) runs on pastebin -- it is far easier 
to comment on real data.  :-)



It would help if you told us:

1.  Make and model of computer (or motherboard and chassis, if DIY).

2.  Make and model of CPU.

3.  Quantity, make, and model of memory modules, and how your memory 
slots are populated.


4.  NVMe drive partitioning, formatting, space usage, etc..


STFW "CAZ-82512-Q11 NVMe LITEON 512GB", that looks like a decent desktop 
NVMe drive.



Looking at your fio(1) runs:

1. It appears that you held 6 parameters constant:

--name=random-write
--ioengine=posixaio
--rw=randwrite
--runtime=60
--time_based
--end_fsync=1

2.  It appears that you varied 4 parameters:

run bs  sizenumjobs iodepth
1   4k  4g  1   1
2   64k 256m16  16
3   64k 256m16  16
4   1m  16g 1   1

3.  Runs #2 and #3 had the same parameters, but had very different 
operation and results.  Why did run #3 layout 16 IO files, but run #2 
did not?  I suspect that experimental prodedure and/or system loading 
were not the same during the two runs (?).


4.  The best way to eliminate system loading variances is to create your 
own Debian USB live stick.  At the "Choose software to install" screen, 
choose "SSH server" and "standard system utilities".  Once the system is 
running, install the tools you want.  Use script(1) to capture console 
sessions.


5.  Pick one set of benchmark tool parameters and do several runs.  You 
want to gain enough skill running the tool so that the runs have little 
variance.


6.  In addition to running the benchmark tool the same way every time, 
you need to do all the other steps the same way every time -- rebooting, 
clearing kernel buffers, erasing files, trimming SSD's, enabling/ 
disabling processes, whatever.


7.  Document your experimental procedure such that you and others can 
duplicate your work and compare results.


8.  For each set of runs, throw out the top third of results, throw out 
the bottom 

Re: nvme SSD and poor performance

[Stefan Monnier]

> P-S: If triming it is needed for ssd, why debian do not trim by default?

AFAIK trimming is not needed.  It can be beneficial in some cases, but
as a general rule, the SSDs should be able to provide great performance
without it.


Stefan

Re: nvme SSD and poor performance

[Linux-Fan]

Christian Britz writes:


On 17.08.21 at 15:30 Linux-Fan wrote:

Pierre Willaime writes:


P-S: If triming it is needed for ssd, why debian do not trim by default?


Detecting reliably if the current system has SSDs that would benefit from  
trimming AND that the user has not taken their own measures is difficult. I  
guess this might be the reason for there not being an automatism, but you  
can enable the systemd timer suggested above with a single command.


I am pretty sure that I have never played with fstrim.timer and this is the  
output of "systemctl status fstrim.timer" on my bullseye system:


● fstrim.timer - Discard unused blocks once a week
 Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor  
preset: enabled)

 Active: active (waiting) since Tue 2021-08-17 14:10:17 CEST; 3h 5min ago
    Trigger: Mon 2021-08-23 01:01:39 CEST; 5 days left
   Triggers: ● fstrim.service
   Docs: man:fstrim

So it seems this weekly schedule is enabled by default on bullseye.


Nice, thanks for sharing :)

BTW, if my system is not online at that time, will it be triggered on next  
boot?


Yes, I would think so.

A systemd timer can be configured to run if its schedule is missed by  
`[Timer] Persistent=true` which on my machine is configured for  
`fstrim.timer` (check `systemctl cat fstrim.timer`).


See also: https://jeetblogs.org/post/scheduling-jobs-cron-anacron-systemd/

Of course, it is also possible to find out experimentally. Check
`systemctl list-timers` to find out about all registered timers and when  
they ran last.


HTH
Linux-Fan

öö


pgpb4GJLB0KsQ.pgp
Description: PGP signature

Re: nvme SSD and poor performance

[Christian Britz]

On 17.08.21 at 15:30 Linux-Fan wrote:

Pierre Willaime writes:


P-S: If triming it is needed for ssd, why debian do not trim by default?


Detecting reliably if the current system has SSDs that would benefit 
from trimming AND that the user has not taken their own measures is 
difficult. I guess this might be the reason for there not being an 
automatism, but you can enable the systemd timer suggested above with 
a single command.


I am pretty sure that I have never played with fstrim.timer and this is 
the output of "systemctl status fstrim.timer" on my bullseye system:


● fstrim.timer - Discard unused blocks once a week
 Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor 
preset: enabled)
 Active: active (waiting) since Tue 2021-08-17 14:10:17 CEST; 3h 
5min ago

    Trigger: Mon 2021-08-23 01:01:39 CEST; 5 days left
   Triggers: ● fstrim.service
   Docs: man:fstrim

So it seems this weekly schedule is enabled by default on bullseye.
BTW, if my system is not online at that time, will it be triggered on 
next boot?

Re: nvme SSD and poor performance

[Dan Ritter]

Pierre Willaime wrote: 
> Le 17/08/2021 à 14:02, Dan Ritter a écrit :
> > The first question is, how slow is this storage?
> > 
> > 
> > Here is a good article on using fio:
> > https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/
> 
> Thanks for the help.
> 
> Here are the output of fio tests.
> 
> Single 4KiB random write process test:
> 
> WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s),
> io=12.0GiB (12.9GB), run=62271-62271msec
> 
> 16 parallel 64KiB random write processes (two different results, further
> tests are closer to the second than the first):
> 
> WRITE: bw=523MiB/s (548MB/s), 31.8MiB/s-33.0MiB/s (33.4MB/s-35.6MB/s),
> io=35.5GiB (38.1GB), run=63568-69533msec
> 
> WRITE: bw=201MiB/s (211MB/s), 11.9MiB/s-14.8MiB/s (12.5MB/s-15.5MB/s),
> io=14.3GiB (15.3GB), run=60871-72618msec
> 
> Single 1MiB random write process:
> 
>   WRITE: bw=270MiB/s (283MB/s), 270MiB/s-270MiB/s (283MB/s-283MB/s),
> io=16.0GiB (17.2GB), run=60722-60722msec


These numbers look reasonable for a low-end SSD that
manufacturers throw into their laptops -- which is what your
drive is, as far as I can tell. I can't find specs, but I
suspect it is a QLC set with a small DRAM cache.

I don't think you have a significant performance problem, but
you are definitely feeling some pain -- so can you tell us more
about what feels slow? Does it happen during the ordinary course
of the day?


-dsr-

Re: nvme SSD and poor performance

[Marco Möller]

On 17.08.21 15:30, Linux-Fan wrote:

Pierre Willaime writes:

I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable 
(bulleye now).


For a long time, I suffer poor I/O performances which slow down a lot 
of tasks (apt upgrade when unpacking for example).


I am now trying to fix this issue.

Using fstrim seems to restore speed. There are always many GiB which 
are reduced :


[...]


but few minutes later, there are already 1.2 Gib to trim again :

#  fstrim -v /
/ : 1,2 GiB (1235369984 octets) réduits

Is it a good idea to trim, if yes how (and how often)?

Some people use fstrim as a cron job, some other add "discard" option 
to the /etc/fstab / line. I do not know what is the best if any. I 
also read triming frequently could reduce the ssd life.


I do `fstrim` once per week by a minimalistic and custom script as cron 
job:

https://github.com/m7a/lp-ssd-optimization

There is no need for custom scripts anymore, you can nowdays enable the 
timer from `util-linux` without hassle:


 # systemctl enable fstrim.timer

This will perform the trim once per week by default.

When the use of SSDs increased people tried out the `discard` options 
and found them to have strange performance characteristics, potential 
negative effects on SSD life and in some cases even data corruption (?) 
Back then it was recommended to use the periodic `fstrim` instead. I do 
not know if any of the issues with discard are still there today.



I also noticed many I/O access from jbd2 and kworker such as :

# iotop -bktoqqq -d .5
11:11:16 364 be/3 root    0.00 K/s    7.69 K/s  0.00 % 
23.64 % [jbd2/nvme0n1p2-]
11:11:16   8 be/4 root    0.00 K/s    0.00 K/s  0.00 % 
25.52 % [kworker/u32:0-flush-259:0]


[...]

I do not know what to do for kworker and if it is a normal behavior. 
For jdb2, I have read it is filesystem (ext4 here) journal.


I highly recommend you to find out what exactly is causing the high 
number of I/O operations. Usually there is a userspace process 
responsible (or a RAID resync operation) for all the I/O which is then 
processed by the kernel threads you see in iotop.


I usually look at `atop -a 4` (package `atop`) for half a minute or so 
to find out what processes are active on the system. It is possible that 
something is amiss and causing an exceedingly high I/O load leading to 
the performance degradation you observe.


[...]


P-S: If triming it is needed for ssd, why debian do not trim by default?


Detecting reliably if the current system has SSDs that would benefit 
from trimming AND that the user has not taken their own measures is 
difficult. I guess this might be the reason for there not being an 
automatism, but you can enable the systemd timer suggested above with a 
single command.


HTH
Linux-Fan

öö



I have no experience with SSD, but running my Debian Desktop from a USB 
Memory Stick since years, please allow me to share information which 
supports the suggestion of Linux-Fan to also investigate if there is 
extraordinary I/O taking place and maybe could be avoided:
In the past I found extreme(!) I/O to be produced by Firefox, when it is 
writing its cache, and when it is writing its session restore 
information. These writes to my observation occure all the time, kind of 
nonstop! I could get it very satisfactorily reduced by applying a tool 
called "Profile Sync Daemon" (psd) from package "profile-sync-daemon". 
While it is packaged for Debian, as a starting point to study its 
documentation I suggest to better look it up in the Arch Linux Wiki.


Good Luck!
Marco

Re: nvme SSD and poor performance

[Linux-Fan]

Pierre Willaime writes:

I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable  
(bulleye now).


For a long time, I suffer poor I/O performances which slow down a lot of  
tasks (apt upgrade when unpacking for example).


I am now trying to fix this issue.

Using fstrim seems to restore speed. There are always many GiB which are  
reduced :


[...]


but few minutes later, there are already 1.2 Gib to trim again :

#  fstrim -v /
/ : 1,2 GiB (1235369984 octets) réduits

Is it a good idea to trim, if yes how (and how often)?

Some people use fstrim as a cron job, some other add "discard" option to the  
/etc/fstab / line. I do not know what is the best if any. I also read  
triming frequently could reduce the ssd life.


I do `fstrim` once per week by a minimalistic and custom script as cron job:
https://github.com/m7a/lp-ssd-optimization

There is no need for custom scripts anymore, you can nowdays enable the  
timer from `util-linux` without hassle:


# systemctl enable fstrim.timer

This will perform the trim once per week by default.

When the use of SSDs increased people tried out the `discard` options and  
found them to have strange performance characteristics, potential negative  
effects on SSD life and in some cases even data corruption (?) Back then it  
was recommended to use the periodic `fstrim` instead. I do not know if any  
of the issues with discard are still there today.



I also noticed many I/O access from jbd2 and kworker such as :

# iotop -bktoqqq -d .5
	11:11:16 364 be/3 root0.00 K/s7.69 K/s  0.00 % 23.64  
% [jbd2/nvme0n1p2-]
	11:11:16   8 be/4 root0.00 K/s0.00 K/s  0.00 % 25.52  
% [kworker/u32:0-flush-259:0]


[...]

I do not know what to do for kworker and if it is a normal behavior. For  
jdb2, I have read it is filesystem (ext4 here) journal.


I highly recommend you to find out what exactly is causing the high number  
of I/O operations. Usually there is a userspace process responsible (or a  
RAID resync operation) for all the I/O which is then processed by the kernel  
threads you see in iotop.


I usually look at `atop -a 4` (package `atop`) for half a minute or so to  
find out what processes are active on the system. It is possible that  
something is amiss and causing an exceedingly high I/O load leading to the  
performance degradation you observe.


[...]


P-S: If triming it is needed for ssd, why debian do not trim by default?


Detecting reliably if the current system has SSDs that would benefit from  
trimming AND that the user has not taken their own measures is difficult. I  
guess this might be the reason for there not being an automatism, but you  
can enable the systemd timer suggested above with a single command.


HTH
Linux-Fan

öö


pgpMM3E95N78t.pgp
Description: PGP signature

Re: nvme SSD and poor performance

[Pierre Willaime]

Le 17/08/2021 à 14:02, Dan Ritter a écrit :

The first question is, how slow is this storage?


Here is a good article on using fio:
https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/


Thanks for the help.

Here are the output of fio tests.


Single 4KiB random write process test:

WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), 
io=12.0GiB (12.9GB), run=62271-62271msec


https://pastebin.com/5Cyg9Xvt


16 parallel 64KiB random write processes (two different results, further 
tests are closer to the second than the first):


WRITE: bw=523MiB/s (548MB/s), 31.8MiB/s-33.0MiB/s (33.4MB/s-35.6MB/s), 
io=35.5GiB (38.1GB), run=63568-69533msec


WRITE: bw=201MiB/s (211MB/s), 11.9MiB/s-14.8MiB/s (12.5MB/s-15.5MB/s), 
io=14.3GiB (15.3GB), run=60871-72618msec



https://pastebin.com/XVpPpqsC
https://pastebin.com/HEx8VvhS


Single 1MiB random write process:

  WRITE: bw=270MiB/s (283MB/s), 270MiB/s-270MiB/s (283MB/s-283MB/s), 
io=16.0GiB (17.2GB), run=60722-60722msec


https://pastebin.com/skk6mi7M

Re: nvme SSD and poor performance

[Dan Ritter]

Pierre Willaime wrote: 
> Hi,
> 
> I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
> (bulleye now).
> 
> For a long time, I suffer poor I/O performances which slow down a lot of
> tasks (apt upgrade when unpacking for example).
> 
> I am now trying to fix this issue.

The first question is, how slow is this storage?


Here is a good article on using fio:
https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/


> Using fstrim seems to restore speed. There are always many GiB which are
> reduced :
> 
>   #  fstrim -v /
>   / : 236,7 GiB (254122389504 octets) réduits
> 
> then, directly after :
> 
>   #  fstrim -v /
>   / : 0 B (0 octets) réduits

That's normal. 

> but few minutes later, there are already 1.2 Gib to trim again :
> 
>   #  fstrim -v /
>   / : 1,2 GiB (1235369984 octets) réduits

That's high but not impossible.

> /Is it a good idea to trim, if yes how (and how often)?/

I trim in a cron job once per week on high activity systems and
once per month on lower activity systems.


> I also noticed many I/O access from jbd2 and kworker such as :
> 
>   # iotop -bktoqqq -d .5
>   11:11:16 364 be/3 root0.00 K/s7.69 K/s  0.00 % 23.64 %
> [jbd2/nvme0n1p2-]
>   11:11:16   8 be/4 root0.00 K/s0.00 K/s  0.00 % 25.52 %
> [kworker/u32:0-flush-259:0]
> 
> The percentage given by iotop (time the thread/process spent while swapping
> in and while waiting on I/O) is often high.

These are the kernel processes responsible for writing to disk. 

> I added the "noatime" option to /etc/fstab / line but it does not seem to
> reduce the number of access.

It does, but not much more than the Debian default.

Test your drive's speed, then come back with the results.

-dsr-

nvme SSD and poor performance

[Pierre Willaime]

Hi,

I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable 
(bulleye now).


For a long time, I suffer poor I/O performances which slow down a lot of 
tasks (apt upgrade when unpacking for example).


I am now trying to fix this issue.

Using fstrim seems to restore speed. There are always many GiB which are 
reduced :


#  fstrim -v /
/ : 236,7 GiB (254122389504 octets) réduits

then, directly after :

#  fstrim -v /
/ : 0 B (0 octets) réduits

but few minutes later, there are already 1.2 Gib to trim again :

#  fstrim -v /
/ : 1,2 GiB (1235369984 octets) réduits


/Is it a good idea to trim, if yes how (and how often)?/

Some people use fstrim as a cron job, some other add "discard" option to 
the /etc/fstab / line. I do not know what is the best if any. I also 
read triming frequently could reduce the ssd life.




I also noticed many I/O access from jbd2 and kworker such as :

# iotop -bktoqqq -d .5
	11:11:16 364 be/3 root0.00 K/s7.69 K/s  0.00 % 23.64 % 
[jbd2/nvme0n1p2-]
	11:11:16   8 be/4 root0.00 K/s0.00 K/s  0.00 % 25.52 % 
[kworker/u32:0-flush-259:0]


The percentage given by iotop (time the thread/process spent while 
swapping in and while waiting on I/O) is often high.


I do not know what to do for kworker and if it is a normal behavior. For 
jdb2, I have read it is filesystem (ext4 here) journal.


I added the "noatime" option to /etc/fstab / line but it does not seem 
to reduce the number of access.


Regards,
Pierre


P-S: If triming it is needed for ssd, why debian do not trim by default?