On Wed, Jan 13, 2021 at 09:46:27PM +0100, Paul de Weerd wrote:
> Hi all,
>
> While doing some clean-up on my backup filesystem (which extensively
> uses hardlinks), I came across the error in Subject:
>
> rm: fts_read: No such file or directory
>
> Traversing the hierarchy I was trying to remove, I get similar
> fts_read errors when I `ls` in certain places, but a repeated rm runs
> to completion fine (the tree is gone afterwards).
>
> There's nothing in dmesg suggesting filesystem corruption, the
> filesystem unmounts and remounts cleanly, I'm running a forced fsck
> now which says "** File system is already clean". It's a rather large
> filesystem with many inodes in use, so it'll take some time to
> complete. Also, it's on a softraid crypto device, if that matters:
>
> sd2: 5231654MB, 512 bytes/sector, 10714427745 sectors
>
> Reading fts_read(3) wasn't really enlightening as to why a directory
> that's supposedly there, wouldn't be there anymore. (note that I
> wasn't running another rm in the same tree in parallel when I got
> these errors - I did try to force the error by doing just that, but
> that went through without a single error).
>
> Could there be some TOCTOU issue here somewhere? Or some cache
> misbehaviour? Or is it really dying hardware?
My first bet would be some form of corruption. FLipped bits in e..g
directories while operating normally cannot be seen by the
clean/unclean flag in the superblock. That one only records if the
filesystem was unmounted before reboot, shutdown or crash.
The forced fsck might reveal more.
-Otto
>
> Paul 'WEiRD' de Weerd
>
> OpenBSD 6.8-current (GENERIC.MP) #267: Sat Jan 9 19:23:55 MST 2021
> [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 34311208960 (32721MB)
> avail mem = 33256046592 (31715MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe6690 (57 entries)
> bios0: vendor Dell Inc. version "2.10.0" date 05/24/2018
> bios0: Dell Inc. PowerEdge R210 II
> acpi0 at bios0: ACPI 4.0
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP SPMI DMAR ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT
> SSDT SPCR HEST ERST BERT EINJ
> acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) XHC_(S4) RP01(S5)
> PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) RP05(S5)
> PXSX(S4) RP06(S5) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpihpet0 at acpi0: 14318179 Hz
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2394.91 MHz, 06-2a-07
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 99MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2394.58 MHz, 06-2a-07
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 1, core 0, package 0
> cpu2 at mainbus0: apid 2 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2394.58 MHz, 06-2a-07
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 1, package 0
> cpu3 at mainbus0: apid 3 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2394.58 MHz, 06-2a-07
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 1, core 1, package 0
> cpu4 at mainbus0: apid 4 (application processor)
> cpu4: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2394.58 MHz, 06-2a-07
> cpu4:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu4: 256KB 64b/line 8-way L2 cache
> cpu4: smt 0, core 2, package 0
> cpu5 at mainbus0: apid 5 (application processor)
> cpu5: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2394.58 MHz, 06-2a-07
> cpu5:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu5: 256KB 64b/line 8-way L2 cache
> cpu5: smt 1, core 2, package 0
> cpu6 at mainbus0: apid 6 (application processor)
> cpu6: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2394.59 MHz, 06-2a-07
> cpu6:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu6: 256KB 64b/line 8-way L2 cache
> cpu6: smt 0, core 3, package 0
> cpu7 at mainbus0: apid 7 (application processor)
> cpu7: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2394.58 MHz, 06-2a-07
> cpu7:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu7: 256KB 64b/line 8-way L2 cache
> cpu7: smt 1, core 3, package 0
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xe0000000, bus 0-255
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 2 (P0P1)
> acpiprt2 at acpi0: bus 1 (RP01)
> acpiprt3 at acpi0: bus -1 (RP02)
> acpiprt4 at acpi0: bus -1 (RP03)
> acpiprt5 at acpi0: bus -1 (RP04)
> acpiprt6 at acpi0: bus -1 (RP05)
> acpiprt7 at acpi0: bus -1 (RP06)
> acpiprt8 at acpi0: bus -1 (RP07)
> acpiprt9 at acpi0: bus -1 (RP08)
> acpiprt10 at acpi0: bus -1 (PEG0)
> acpiprt11 at acpi0: bus -1 (PEG1)
> acpiprt12 at acpi0: bus -1 (PEG2)
> acpiprt13 at acpi0: bus -1 (PEG3)
> acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
> "IPI0001" at acpi0 not configured
> acpicmos0 at acpi0
> "PNP0A05" at acpi0 not configured
> "PNP0A05" at acpi0 not configured
> "PNP0C14" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> acpicpu0 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10),
> C1(1000@1 mwait.1), PSS
> acpicpu1 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10),
> C1(1000@1 mwait.1), PSS
> acpicpu2 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10),
> C1(1000@1 mwait.1), PSS
> acpicpu3 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10),
> C1(1000@1 mwait.1), PSS
> acpicpu4 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10),
> C1(1000@1 mwait.1), PSS
> acpicpu5 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10),
> C1(1000@1 mwait.1), PSS
> acpicpu6 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10),
> C1(1000@1 mwait.1), PSS
> acpicpu7 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10),
> C1(1000@1 mwait.1), PSS
> acpipwrres0 at acpi0: FN00, resource for FAN0
> acpipwrres1 at acpi0: FN01, resource for FAN1
> acpipwrres2 at acpi0: FN02, resource for FAN2
> acpipwrres3 at acpi0: FN03, resource for FAN3
> acpipwrres4 at acpi0: FN04, resource for FAN4
> acpitz0 at acpi0: critical temperature is 74 degC
> ipmi at mainbus0 not configured
> cpu0: using VERW MDS workaround (except on vmm entry)
> cpu0: Enhanced SpeedStep 2394 MHz: speeds: 2401, 2400, 2300, 2200, 2100,
> 2000, 1900, 1800, 1700, 1600 MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 Host" rev 0x09
> ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 20
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
> addr 1
> ppb0 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb4: msi
> pci1 at ppb0 bus 1
> bnx0 at pci1 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16
> bnx1 at pci1 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17
> ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 23
> usb1 at ehci1: USB revision 2.0
> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
> addr 1
> ppb1 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4
> pci2 at ppb1 bus 2
> 2:3:0: rom address conflict 0xffff0000/0x10000
> vga1 at pci2 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
> wsdisplay at vga1 not configured
> pcib0 at pci0 dev 31 function 0 "Intel C202 LPC" rev 0x04
> ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI" rev 0x04: msi, AHCI 1.3
> ahci0: port 0: 3.0Gb/s
> ahci0: port 1: 3.0Gb/s
> ahci0: port 4: 1.5Gb/s
> scsibus1 at ahci0: 32 targets
> sd0 at scsibus1 targ 0 lun 0: <ATA, INTEL SSDSC2BW48, DC32>
> naa.55cd2e414c807f33
> sd0: 457862MB, 512 bytes/sector, 937703088 sectors, thin
> sd1 at scsibus1 targ 1 lun 0: <ATA, WDC WD60EFRX-68L, 82.0>
> naa.50014ee20dd29e68
> sd1: 5723166MB, 512 bytes/sector, 11721045168 sectors
> cd0 at scsibus1 targ 4 lun 0: <TSSTcorp, DVD+-RW TS-L633J, D150> removable
> ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 0 int
> 19
> iic0 at ichiic0
> sdtemp0 at iic0 addr 0x18: mcp98243
> sdtemp1 at iic0 addr 0x19: mcp98243
> sdtemp2 at iic0 addr 0x1a: mcp98243
> sdtemp3 at iic0 addr 0x1b: mcp98243
> spdmem0 at iic0 addr 0x50: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> spdmem1 at iic0 addr 0x51: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> spdmem2 at iic0 addr 0x52: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> spdmem3 at iic0 addr 0x53: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: console
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> vmm0 at mainbus0: VMX/EPT
> uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub"
> rev 2.00/0.00 addr 2
> uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub"
> rev 2.00/0.00 addr 2
> uhidev0 at uhub3 port 1 configuration 1 interface 0 "RDing TEMPERHUM1V1.2"
> rev 2.00/0.01 addr 3
> uhidev0: iclass 3/1, 1 report id
> ukbd0 at uhidev0 reportid 1: 8 variable keys, 5 key codes
> wskbd0 at ukbd0 mux 1
> uhidev1 at uhub3 port 1 configuration 1 interface 1 "RDing TEMPERHUM1V1.2"
> rev 2.00/0.01 addr 3
> uhidev1: iclass 3/1
> ugold0 at uhidev1
> uhub4 at uhub3 port 5 configuration 1 interface 0 "Standard Microsystems
> product 0x2514" rev 2.00/0.00 addr 4
> uhidev2 at uhub4 port 1 configuration 1 interface 0 "USB usb keyboard" rev
> 1.10/1.10 addr 5
> uhidev2: iclass 3/1
> ukbd1 at uhidev2: 8 variable keys, 6 key codes
> wskbd1 at ukbd1 mux 1
> uhidev3 at uhub4 port 1 configuration 1 interface 1 "USB usb keyboard" rev
> 1.10/1.10 addr 5
> uhidev3: iclass 3/0, 2 report ids
> uhid0 at uhidev3 reportid 1: input=2, output=0, feature=0
> uhid1 at uhidev3 reportid 2: input=1, output=0, feature=0
> umass0 at uhub4 port 2 configuration 1 interface 0 "Kingston DataTraveler
> 2.0" rev 2.00/1.00 addr 6
> umass0: using SCSI over Bulk-Only
> scsibus2 at umass0: 2 targets, initiator 0
> sd2 at scsibus2 targ 1 lun 0: <Kingston, DataTraveler 2.0, PMAP> removable
> serial.09306545EF90EA307FE6
> sd2: 14891MB, 512 bytes/sector, 30497664 sectors
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> sd3 at scsibus4 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006>
> sd3: 409602MB, 512 bytes/sector, 838865577 sectors
> root on sd3a (5a5905f3a7206f17.a) swap on sd3b dump on sd3b
> bnx0: address bc:30:5b:e8:d6:48
> brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
> bnx1: address bc:30:5b:e8:d6:49
> brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
> ugold0: 2 sensors type si7006 (temperature and humidity)
> sd2 detached
> scsibus2 detached
> umass0 detached
> wskbd1 detached
> ukbd1 detached
> uhidev2 detached
> uhid0 detached
> uhid1 detached
> uhidev3 detached
> sd2 at scsibus4 targ 2 lun 0: <OPENBSD, SR CRYPTO, 006>
> sd2: 5231654MB, 512 bytes/sector, 10714427745 sectors
> softraid0: volume sd2 is roaming, it used to be sd3, updating metadata
>
> --
> >++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
> +++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
> http://www.weirdnet.nl/
>
