GCC only on OpenBSD adds -L/usr/lib as prefix, why? Re: OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?

2021-03-02 Thread Bob
Hi Stuart and Marc,

Thanks a lot for responding.

Debugging the problem brought me to realize that GCC (both bundled
gcc/g++ and the egcc/eg++ port) adds a "-L/usr/lib" argument *in the
leading position* to LD.

Example:

$ echo "int main(){}">t.c; gcc -c -o t.o t.c; gcc -o t t.o -LMYDIRTEST -Wl,-v
collect2 version 4.2.1 20070719  (OpenBSD/x86-64 ELF)
/usr/bin/ld --eh-frame-hdr -e __start -Bdynamic -dynamic-linker 
/usr/libexec/ld.so -L/usr/lib -o t /usr/lib/crt0.o /usr/lib/crtbegin.o 
-LMYDIRTEST -L/usr/lib/gcc-lib/amd64-unknown-openbsd6.7/4.2.1 t.o -v -lgcc -lc 
-lgcc /usr/lib/crtend.o
LLD 8.0.1 (compatible with GNU linkers)

As you see here, my "-LMYDIRTEST" argument ends up AFTER the -L/usr/lib
argument which gcc/egcc adds by itself.

This then has the effect that "-L/path/to/my/zlib/build/lib -lz" will
NOT attempt to pick my custom build EVER, but instead ALWAYS pick the
/usr/lib/libz.so .

This is a great headache for any attempt to make a local build of
software where zlib is a component!

And makes me curious, is this deliberate?


I tested clang on OpenBSD and on Linux and it does not have this
behavior. Also tested GCC on Linux, and it does not have this behavior.

I could imagine that the LEADING instead of TRAILING -L/usr/lib could
be some kind of security measure, however if that was the case, I'm
sure OpenBSD clang would have had the same, and it does not.

What is the motivation?


The likely follow-up question will then be, how do I disable this
behavior (and get -L/usr/lib as last LD argument as is the case on
clang everywhere and GCC on other platforms).

Below here for completeness response to Stuart's and Marc's previously
shared thoughts on the topic.

Overall after having given this question a couple of days of work, I
feel there is a certain asymmetry to this -L/usr/lib prefix behavior
and it could need a bit of discussion or at the very least
clarification.

Either OpenBSD's port changes this, or I need to build a local
patched port myself.


Also, and this is a bit beyond my GCC skills - the OpenBSD GCC port
contains patches that introduces this behavior in the first place,
does it not - if anyone has in depth understanding of GCC, please
feel free to point out which OpenBSD port patch file(s) it is that
causes this behavior, if so I could just "rm" that one in my local
ports repo and voillas my eg++ and egcc will work the way I want
out of the box.


Before ending this email I should also mention a partial mitigation
of the -L/usr/lib prefix problem: Namely, link in the zlib .so file
by mentioning it by .so filename on the GCC (LD) command line, instead
of via the "-lz" argument.

This works, however it will only work within a fully home-built custom
codebase, as any other code project that depends on zlib out there,
guaranteedly is hardcoded to link to zlib by "-lz". One would need to
patch all such projects' makefiles to link to it by "libz.so" instead
of "-lz", and that would be tedious and error-prone, good only as a
hack.


Thanks!
Bob


(I wrote:)
> > > I'll start with the detail problem, and discuss the reproduction at the
> > > bottom:
> > > This has brought me to the bizarre issue that the libpng build plainly
> > > refuses to link to my custom libz file /home/myuser/lib/libz.so.1 .
> > > Instead, it insists with linking to the OS' global
> > > /usr/lib/libz.so.5.0 .

(Stuart wrote:)
> > This is as expected really, it looks for the higher library version
> > number.

I evaluated the version number hypothesis, for instance by bumping
my zlib build's SONAME version number to 999.0 , and my build
would not get preference. I also talked to a GCC guy and he said
there is no such thing as version preference in respect of gcc -lNAME .
Have you actually seen this "pick highest version number" behavior on
any platform?


(Marc wrote:)
> Not really, it's more that you have to make sure to put your own directory
> before the system directory, which requires a bit more magic than just -L.
>
> (remember that linking will stop at the first directory with a satisfying
> library, and link with the highest version number found in there, which
> is fortunate for stuff like libtool!)

Marc, what you suggest here is correct and not correct - because
GCC on OpenBSD *always* adds -L/usr/lib in the leading position to LD's
arguments, then it does not matter in what order you as user add any
-L yourself.

(Re. libtool I don't know what you had on your mind.)


(I wrote:)
> > Hi,
> > I am trying to make a custom build of libpng in my home directory,
> > using a libz build that I made in my home directory also.
> > Both are latest version, libpng 1.6.37 same as OpenBSD's port
> > https://cvsweb.openbsd.org/ports/graphics/png/Makefile?rev=1.125=text/x-cvsweb-markup
> > and libz the latest one they published. For zlib it's 1.2.11
> > from http://zlib.net/ from 2017, OpenBSD does not have a port but
> > base bundles a 2009 version. Since 2009, significantly an export
> > "inflateReset2" has 

Re: OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?

2021-03-02 Thread Bob
Hi Stuart and Marc,

Thank you very much for responding.

I have given some more days to debugging this issue and narrowed it
down very much now.

I'll changing name on this thread to "GCC only on OpenBSD adds
-L/usr/lib as prefix, why" and follow up there.

Bob

On Wednesday, 24 February 2021 15:40, Marc Espie  wrote:
> On Wed, Feb 24, 2021 at 02:17:14PM -, Stuart Henderson wrote:
>
> > On 2021-02-23, Bob obs...@protonmail.com wrote:
> >
> > > Hi,
> > > I am trying to make a custom build of libpng in my home directory,
> > > using a libz build that I made in my home directory also.
> > > Both are latest version, libpng 1.6.37 same as OpenBSD's port
> > > https://cvsweb.openbsd.org/ports/graphics/png/Makefile?rev=1.125=text/x-cvsweb-markup
> > > and libz the latest one they published. For zlib it's 1.2.11
> > > from http://zlib.net/ from 2017, OpenBSD does not have a port but
> > > base bundles a 2009 version. Since 2009, significantly an export
> > > "inflateReset2" has been added.
> >
> > As you have seen it is difficult to have library functions in one
> > version in base and in another version built elsewhere (whether that's
> > in $HOME or in ports). Ports only does this for libraries where there's
> > really no other choice and where that has been done they're used very
> > rarely (the port can then not depend on any libraries which pull in the
> > library from base). Currently that is libbind (used only by asdig and
> > zeek) and openssl (used as a static library by sslscan, and dynamic
> > for nrpe and nsca-ng).
> >
> > > I'll start with the detail problem, and discuss the reproduction at the
> > > bottom:
> > > This has brought me to the bizarre issue that the libpng build plainly
> > > refuses to link to my custom libz file /home/myuser/lib/libz.so.1 .
> > > Instead, it insists with linking to the OS' global
> > > /usr/lib/libz.so.5.0 .
> >
> > This is as expected really, it looks for the higher library version
> > number.
>
> Not really, it's more that you have to make sure to put your own directory
> before the system directory, which requires a bit more magic than just -L.
>
> (remember that linking will stop at the first directory with a satisfying
> library, and link with the highest version number found in there, which
> is fortunate for stuff like libtool!)




Re: 4k sector disk on APU2 problems

2021-03-02 Thread Raimo Niskanen
On Mon, Mar 01, 2021 at 09:41:31PM +, Stuart Henderson wrote:
> On 2021-03-01, Raimo Niskanen  wrote:
> > Hi Misc!
> >
> > Unfortunately I do not have one clear question here, but I wonder if 
> > somebody
> > could shed som light on some problems I have encountered on my PC Engines 
> > APU2.
> >
> > It runs OpenBSD 6.7 from a 32 GB mSATA SSD disk, and I would like to change
> > the disk since it is a few years old now, so I buyed a 128 GB Kingston
> > mSATA SSD, and an mSATA -> SATA adapter and put that combo in an USB 2
> > external disk adapter.
> >
> > The disk showed up as a 4k sector disk, and after installing OpenBSD 6.7
> > over USB over the mSATA-SATA adapter I plugged it in the internal mSATA
> > connector, and it did not boot.
> 
> This is a problem with some USB-SATA adapters. See e.g.
> https://www.klennet.com/notes/2018-04-14-usb-and-sector-size.aspx
> 

Just my bad luck, then...

> > Much fumbling later it seems that when the disk is connected to the
> > internal mSATA slot it is seen as a 512 bytes per sector disk.  I do not
> > know what the BIOS thinks of it (factory SeaBIOS 1.10.something).  When I
> > re-installed with the disk in the mSATA slot I got a bootable installation.
> > Both fdisk and disklabel now says the disk has got 512 bytes per sector.
> > (fdisk says nothing but for a 4k disk it should say it is a 4k disk)
> >
> > My old 32 GB mSATA disk is readable over the mSATA-SATA adapter USB adapter
> > as a 512 bytes per sector disk.
> 
> You could try looking for a different adapter but at this point 
> I would probably install on the new drive (PXE boot or use another USB
> drive to boot the installer), then copy files back from the old drive.
> 
> > So I am just curious about how to handle this disk.  I can install to it
> > in the internal mSATA connector and read the old installation over the
> > mSATA-SATA-USB-adapter.  But one day when I want to install to a new disk
> > again, I will not be able to read from the disk in the 
> > mSATA-SATA-USB-adapter,
> > so the next re-installation looks unpromising.
> 
> backup/restore over the network via another machine perhaps?
> 
> > Some more specific questions:
> > * Would upgrading the BIOS be a good idea
> 
> yes but it won't help with this problem.
> (https://github.com/pcengines/apu2-documentation/blob/master/docs/apu_CPU_boost.md)
> 
> > * Sould upgrading to OpenBSD 6.8 improve the situation
> 
> it won't.
> 
> > * How is the disk sector size determined, and can I affect that?
> 
> by the manufacturer.
> 

Thank you for the information!  Enlighting!

Since I can boot from the internal SD card as well, I can use a different
USB drive as dump/restore storage instead of an external machine.  This
USB encosure sector size pecularity only blocks me from direct copy from
old installation to new, for future re-installations.

But for current re-installation I can read the old disk from the USB
enclosure, since it apparently does not alter the sector size for the old
32 GB disk.

Cheers
-- 

/ Raimo Niskanen, Erlang/OTP, Ericsson AB



Re: OpenBSD 6.8 - softraid issue: "uvm_fault(0xffffffff821f5490, 0x40, 0, 1) -> e"

2021-03-02 Thread Mark Schneider

On 02.03.21 10:39, Stuart Henderson wrote:

On 2021/03/02 00:09, Mark Schneider wrote:

Hi,

Thank you for your feeeback.

Also OpenBSD 6.9beta snapshot is crashing when I setup RAID5 with three
"Samsung PRO 860 1TB" SSDs.
OpenBSD obsd69b.it-infra.org 6.9 GENERIC.MP#368 amd64

obsd69b# dmesg | grep  -i bios
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xdc312018 (61 entries)
bios0: vendor American Megatrends Inc. version "2201" date 03/23/2015
bios0: ASUSTeK COMPUTER INC. CROSSHAIR V FORMULA-Z
acpi0 at bios0: ACPI 5.0

Can you isolate softraid from the equation? Are the drives reliable with
this hardware configuration when not using softraid? I guess it would
need testing with simultaneous writes to the 3 drives to give a closer
match to the situation with softraid.


Thanks a lot for all hints Stuart.

The isolated 1TB SSD Samsung PRO 860 drives have some AHCI errors 
(OpenBSD_6.9beta-RAID5-3x1TB-SSD-isolated.txt in the attachment).



Writing to an "isolated" drive does not crash OpenBSD even there are 
AHCI errors and sometimes an I/O error from dd (see directly below).


# ---
obsd69b# dd if=/dev/urandom of=/ssd1T-sd1a/1GB-urandom.bin bs=1M count=1024
dd: /ssd1T-sd1a/1GB-urandom.bin: Input/output error
1+0 records in
0+0 records out
0 bytes transferred in 0.014 secs (0 bytes/sec)

obsd69b# dd if=/dev/urandom of=/ssd1T-sd1a/1GB-urandom.bin bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 5.156 secs (208228191 bytes/sec)

# ---

ahci2: NCQ errored slot 3 is idle (0400 active)
ahci2: NCQ errored slot 13 is idle (7c0f01eb active)
ahci2: NCQ errored slot 26 is idle (03fe1e07 active)
ahci2: NCQ errored slot 30 is idle (03e1e38f active)
ahci2: NCQ errored slot 28 is idle (03e1fc71 active)
ahci2: NCQ errored slot 30 is idle (03fc9f81 active)
ahci2: NCQ errored slot 9 is idle (0f0ee03f active)
ahci2: NCQ errored slot 16 is idle (70f400ff active)
ahci2: NCQ errored slot 28 is idle (0f3c407f active)
ahci2: NCQ errored slot 13 is idle (70dc41fc active)
ahci2: NCQ errored slot 17 is idle (0f3c1fe0 active)
ahci2: NCQ errored slot 30 is idle (0f7c181f active)


Writing to all "isolated" drives simultanously does not crash OpenBSD 
even there are AHCI errors


# 
--
obsd69b# dd if=/dev/urandom of=/ssd1T-sd1a/10GB-urandom.bin bs=10M 
count=1024

1024+0 records in
1024+0 records out
10737418240 bytes transferred in 160.129 secs (67054710 bytes/sec)

obsd69b# dd if=/dev/urandom of=/ssd1T-sd2a/10GB-urandom.bin bs=10M 
count=1024

1024+0 records in
1024+0 records out
10737418240 bytes transferred in 158.783 secs (67623059 bytes/sec

obsd69b# dd if=/dev/urandom of=/ssd1T-sd3a/10GB-urandom.bin bs=10M 
count=1024

1024+0 records in
1024+0 records out
10737418240 bytes transferred in 160.085 secs (67072961 bytes/sec

# ---

ahci2: NCQ errored slot 25 is idle (00ff active)
ahci2: NCQ errored slot 21 is idle (7fff active)
ahci2: NCQ errored slot 6 is idle (0001 active)
ahci2: NCQ errored slot 21 is idle (03ff active)
ahci2: NCQ errored slot 1 is idle (0380 active)
ahci2: NCQ errored slot 27 is idle (01ff active)
ahci2: NCQ errored slot 23 is idle (0030 active)
ahci2: NCQ errored slot 25 is idle (0fff active)


# OpenBSD 6.9beta is crashing after a dd command writing to the RAID5 
softraid volume (sd4a) and the access to the ddb{4}> prompt is not 
possible to run trace, ps or sh commands (the root console is dead).




"trace" and "sh reg" from ddb would give more clues.


I am not able to run the commands above as the root ddb{4} console is 
dead (I can see only the last error message but I am not able to type in 
using the keyboard)



I will connect those Samsung PRO 860 1TB SSDs to a Xeon based system 
(another SATA-controller) and check there for AHCI errors.


Maybe it is worth to mention, that the original RAID tests on Debian 
buster with six of 512GB Samsung PRO 860 (the same drives andf RAID6 set 
with mdadm) worked without crashing the OS.



Kind regards

Mark



bs=10M count=1024

# Error messages

uvm_fault(0x821f5490, 0x40, 0, 1) -> e
kernel: page fault trap, code=0
Stopped at  sr_validate_io+0x44:    cmpl $0,0x40(%r9)
ddb{2}>

$ objdump -dlr softraid.o | less
...skipping...
9cc0 :
sr_validate_io():
/usr/src/sys/dev/softraid.c:4569
 9cc0:   4c 8b 1d 00 00 00 00mov0(%rip),%r11# 9cc7 

 9cc3: R_X86_64_PC32 
__retguard_3962+0xfffc
 9cc7:   4c 33 1c 24 xor(%rsp),%r11
 9ccb:   55  push   %rbp
 9ccc:   48 89 e5mov%rsp,%rbp
 9ccf:   57  push   %rdi
 9cd0:   56  push   %rsi
 9cd1:   52  push   %rdx
 9cd2:   57  push   %rdi
 9cd3:   41 53   push   %r11
 9cd5:   50   

Re: relayd error: socket_rlimit: max open files 1024

2021-03-02 Thread Stuart Henderson
On 2021-03-02, Jean-Pierre de Villiers  
wrote:
> The entry openfiles-cur=1024 is overriding the entry openfiles=1024.
> Note that openfiles=value sets both openfiles-max=value and
> openfiles-cur=value.
>
> The setting openfiles-max setting is the upper limit which can only be
> changed by root while any user can change their own openfiles-cur up the
> maximum value set previously.  You will either need to increase
> openfiles-cur or remove it completely and only have openfiles=4096.
>
> All this info and more is contained in login.conf(5) and getrlimit(2).

yep.

> However, I believe the recommended practice would be to create a new
> login class, called 'relayd' say, that inherits from the daemon class.
> This is specified using the entry: tc=daemon.  Otherwise, every process
> running as a user in the daemon class will have these heightened
> privileges - and there's a lot of them.

For daemons started by rc.d, the class is set by the rc script, by default
to a class named after the daemon if it exists (i.e. "relayd"), if not
then fallback to "daemon" (unless you set "relayd_class" in rc.conf.local).

The class set in the passwd file isn't used.

> > $ doas cap_mkdb /etc/login.conf

I would just rm the db file, it isn't present by default. The
optimization isn't so useful these days, and it one more thing to
(remember|forget) when you change the file..




Re: relayd error: socket_rlimit: max open files 1024

2021-03-02 Thread Jean-Pierre de Villiers
The entry openfiles-cur=1024 is overriding the entry openfiles=1024.
Note that openfiles=value sets both openfiles-max=value and
openfiles-cur=value.

The setting openfiles-max setting is the upper limit which can only be
changed by root while any user can change their own openfiles-cur up the
maximum value set previously.  You will either need to increase
openfiles-cur or remove it completely and only have openfiles=4096.

All this info and more is contained in login.conf(5) and getrlimit(2).

However, I believe the recommended practice would be to create a new
login class, called 'relayd' say, that inherits from the daemon class.
This is specified using the entry: tc=daemon.  Otherwise, every process
running as a user in the daemon class will have these heightened
privileges - and there's a lot of them.

Regards,
JP



Re: OpenBSD 6.8 - softraid issue: "uvm_fault(0xffffffff821f5490, 0x40, 0, 1) -> e"

2021-03-02 Thread Stefan Sperling
On Tue, Mar 02, 2021 at 09:39:15AM +, Stuart Henderson wrote:
> putting sr_validate_io+0x44 at the xs->datalen dereference,
> 
> 4580 if (sd->sd_vol_status == BIOC_SVOFFLINE) {
> 4581 DNPRINTF(SR_D_DIS, "%s: %s device offline\n",
> 4582 DEVNAME(sd->sd_sc), func);
> 4583 goto bad;
> 4584 }
> 4585
> 4586 if (xs->datalen == 0) {
> 4587 printf("%s: %s: illegal block count for %s\n",
> 4588 DEVNAME(sd->sd_sc), func, sd->sd_meta->ssd_devname)  
>;
> 4589 goto bad;
> 4590 }
> 
> ...so null/invalid xs?

Yes, I've looked at this function already and I think a bad deref of xs
is the only reasonable explanation. But we don't know how that happens.



relayd error: socket_rlimit: max open files 1024

2021-03-02 Thread jrmu
I am trying to use relayd to provide TLS acceleration for 20+ user services.

Here is my /etc/relayd.conf (with ip4 and ip6 redacted):

ip4="192.168.0.1"
ip6="2001:db8::"
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }   
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
table  { 127.0.0.1 }
log connection

http protocol https {
match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
match request header append "X-Forwarded-By" \
match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
match request header append "X-Forwarded-By" \
value "$SERVER_ADDR:$SERVER_PORT"
match request header set "Connection" value "close"
tcp { sack, backlog 128 }  
tls { keypair username01.example.ircnow.org
  keypair username02.example.ircnow.org
  keypair username03.example.ircnow.org
  keypair username04.example.ircnow.org
  keypair username05.example.ircnow.org
  keypair username06.example.ircnow.org
  keypair username07.example.ircnow.org
  keypair username08.example.ircnow.org
  keypair username09.example.ircnow.org
  keypair username10.example.ircnow.org
  keypair username11.example.ircnow.org
  keypair username12.example.ircnow.org
  keypair username13.example.ircnow.org
  keypair username14.example.ircnow.org
  keypair username15.example.ircnow.org
  keypair username16.example.ircnow.org
  keypair username17.example.ircnow.org
  keypair username18.example.ircnow.org }
match request header "Host" value "username01.example.ircnow.org" 
forward to 
match request header "Host" value "username01.example.ircnow.org" 
forward to 
match request header "Host" value "username02.example.ircnow.org" 
forward to 
match request header "Host" value "username03.example.ircnow.org" 
forward to 
match request header "Host" value "username04.example.ircnow.org" 
forward to 
match request header "Host" value "username05.example.ircnow.org" 
forward to 
match request header "Host" value "username06.example.ircnow.org" 
forward to 
match request header "Host" value "username07.example.ircnow.org" 
forward to 
match request header "Host" value "username08.example.ircnow.org" 
forward to 
match request header "Host" value "username09.example.ircnow.org" 
forward to 
match request header "Host" value "username10.example.ircnow.org" 
forward to 
match request header "Host" value "username11.example.ircnow.org" 
forward to 
match request header "Host" value "username12.example.ircnow.org" 
forward to 
match request header "Host" value "username13.example.ircnow.org" 
forward to 
match request header "Host" value "username14.example.ircnow.org" 
forward to 
match request header "Host" value "username15.example.ircnow.org" 
forward to 
match request header "Host" value "username16.example.ircnow.org" 
forward to 
match request header "Host" value "username17.example.ircnow.org" 
forward to 
match request header "Host" value "username18.example.ircnow.org" 
forward to 
}
relay wwwtls {
listen on $ip4 port 443 tls
protocol https
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
}
relay www6tls {
listen on $ip6 port 443 tls
protocol https
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 check icmp
forward to  port 8001 

Re: OpenBSD 6.8 - softraid issue: "uvm_fault(0xffffffff821f5490, 0x40, 0, 1) -> e"

2021-03-02 Thread Stuart Henderson
On 2021/03/02 00:09, Mark Schneider wrote:
> Hi,
> 
> Thank you for your feeeback.
> 
> Also OpenBSD 6.9beta snapshot is crashing when I setup RAID5 with three
> "Samsung PRO 860 1TB" SSDs.
> OpenBSD obsd69b.it-infra.org 6.9 GENERIC.MP#368 amd64
> 
> obsd69b# dmesg | grep  -i bios
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xdc312018 (61 entries)
> bios0: vendor American Megatrends Inc. version "2201" date 03/23/2015
> bios0: ASUSTeK COMPUTER INC. CROSSHAIR V FORMULA-Z
> acpi0 at bios0: ACPI 5.0

Can you isolate softraid from the equation? Are the drives reliable with
this hardware configuration when not using softraid? I guess it would
need testing with simultaneous writes to the 3 drives to give a closer
match to the situation with softraid.

> > > bs=10M count=1024
> > > 
> > > # Error messages
> > > 
> > > uvm_fault(0x821f5490, 0x40, 0, 1) -> e
> > > kernel: page fault trap, code=0
> > > Stopped at  sr_validate_io+0x44:    cmpl $0,0x40(%r9)
> > > ddb{2}>

$ objdump -dlr softraid.o | less
...skipping...
9cc0 :
sr_validate_io():
/usr/src/sys/dev/softraid.c:4569
9cc0:   4c 8b 1d 00 00 00 00mov0(%rip),%r11# 9cc7 

9cc3: R_X86_64_PC32 
__retguard_3962+0xfffc
9cc7:   4c 33 1c 24 xor(%rsp),%r11
9ccb:   55  push   %rbp
9ccc:   48 89 e5mov%rsp,%rbp
9ccf:   57  push   %rdi
9cd0:   56  push   %rsi
9cd1:   52  push   %rdx
9cd2:   57  push   %rdi
9cd3:   41 53   push   %r11
9cd5:   50  push   %rax
/usr/src/sys/dev/softraid.c:4570
9cd6:   4c 8b 47 08 mov0x8(%rdi),%r8
/usr/src/sys/dev/softraid.c:4577
9cda:   49 8b 88 70 09 00 00mov0x970(%r8),%rcx
9ce1:   83 b9 94 00 00 00 00cmpl   $0x0,0x94(%rcx)
9ce8:   0f 84 a2 01 00 00   je 9e90 
9cee:   b8 01 00 00 00  mov$0x1,%eax
/usr/src/sys/dev/softraid.c:4580
9cf3:   41 83 b8 20 0a 00 00cmpl   $0x1,0xa20(%r8)
9cfa:   01
9cfb:   0f 84 69 01 00 00   je 9e6a 
9d01:   4c 8b 0fmov(%rdi),%r9
/usr/src/sys/dev/softraid.c:4586
9d04:   41 83 79 40 00  cmpl   $0x0,0x40(%r9)
9d09:   74 47   je 9d52 
/usr/src/sys/dev/softraid.c:4592

putting sr_validate_io+0x44 at the xs->datalen dereference,

4580 if (sd->sd_vol_status == BIOC_SVOFFLINE) {
4581 DNPRINTF(SR_D_DIS, "%s: %s device offline\n",
4582 DEVNAME(sd->sd_sc), func);
4583 goto bad;
4584 }
4585
4586 if (xs->datalen == 0) {
4587 printf("%s: %s: illegal block count for %s\n",
4588 DEVNAME(sd->sd_sc), func, sd->sd_meta->ssd_devname)
 ;
4589 goto bad;
4590 }

...so null/invalid xs?

"trace" and "sh reg" from ddb would give more clues.