On top of this (and I don't know why, maybe because of softraid FS
encryption?) I haven't seen any effect of the FS cache for files of any
size (not even 128Mb) that is supposed to be using at least the 32-bit
mem (some percent of the first 4Gb,
https://unix.stackexchange.com/questions/61459/does-sysctl-kern-bufcachepercent-not-work-in-openbsd-5-2-above-1-7gb/62184#62184).
In the presence of FS/hardware management inefficiencies, things could
be dramatically improved with an efficient FS cache if one has enough
RAM as reading from RAM should be in the range of dozens of GB/s with
nanoseconds latency, but that's not the case unfortunately (at least in
my setup).
*From:* Joseph Mayer <joseph.ma...@protonmail.com>
*Sent:* Monday, April 08, 2019 22:52
*To:* Chris Cappuccio <ch...@nmedia.net>
*Cc:* Anatoli <m...@anatoli.ws>, Misc <misc@openbsd.org>
*Subject:* Re: 10GBit network performance on OpenBSD 6.4
On Tuesday, April 9, 2019 3:28 AM, Chris Cappuccio <ch...@nmedia.net> wrote:
Anatoli [m...@anatoli.ws] wrote:
I've seen extremely slow HDD performance in OpenBSD, like 12x slower than on
Linux, also no filesystem cache, so depending on your HDD with scp you may
be hitting the max throughput for the FS, not the network.
12x slower? That's insane. What are you talking about? USB HDD? USB Flash?
SATA? Driver? You should submit a bug report with lots of details.
Chris
Chris,
Isn't the filesystem layer in OpenBSD altogether serial-processing, all
the way pretty much from userland fwrite() down to hardware access (as
in no use of hardware multiqueueing).
The non-use of multiqueueing is problematic for random reads from SSD:s
as they have extremely high latency within the individual read op e.g.
~~1 millisecond.
On the hardware where I tested, OpenBSD will give ~120MB/sec
system-wide filesystem IO on any number of disks, also using an NVMe
SSD which has ~500-900MB/sec random access performance. I took this as
confirmation of the filesystem layer itself being the primary
bottleneck.
Also is the filesystem's internal sector size which it then accesses
underlying hardware with, 4KB, 16KB or 512B? I always suspected the
lastmentioned.
One thing that will be very interesting to see in OpenBSD is how serial
and random accesses perform on Intel Optane NVMe disks, with their
incredibly low latency. These could offset OpenBSD filesystem
limitations in not parallellizing IO.
Also the filesystem logics can be sidestepped by doing 16KB aligned
accesses to /dev/rsd* .
Joseph
