Re: softdep issue in 5.3-current ?
The reported problems are gone in CURRENT: # dmesg|head -n2 OpenBSD 5.4 (GENERIC.MP) #0: Sat Jul 20 17:56:10 CEST 2013 root@test:/usr/src/sys/arch/amd64/compile/GENERIC.MP time buildsrc.sh takes 31 minutes. measured directly after building src (which was slow before): # time tar -xzpf /usr/releasedir/comp54.tgz 0m5.67s real 0m2.07s user 0m0.89s system # time tar -xzpf /usr/releasedir/man54.tgz 0m1.26s real 0m0.45s user 0m0.30s system Best Regards Andreas
Re: softdep issue in 5.3-current ?
On 07/03/13 05:45, Andreas Bartelt wrote: I made a new build of current and the problem with tar performance seems to be resolved now. before: # time tar -xzpf /usr/releasedir/comp53.tgz 3m17.81s real 0m2.14s user 0m2.22s system # time tar -xzpf /usr/releasedir/base53.tgz 3m39.33s real 0m2.23s user 0m2.23s system after: # dmesg|head -n2 OpenBSD 5.3-current (GENERIC.MP) #0: Tue Jul 2 22:44:07 CEST 2013 root@test:/usr/src/sys/arch/amd64/compile/GENERIC.MP # time tar -xzpf /usr/releasedir/comp53.tgz 0m8.92s real 0m1.80s user 0m1.07s system # time tar -xzpf /usr/releasedir/base53.tgz 0m11.29s real 0m2.21s user 0m1.17s system I was wrong -- the problem persists! Directly after booting into a system built with the current source, tar extraction performance is OK (like in my second example from above), but after 'make build make release' of current source on the same system, tar extraction performance is horrible (like in the first example from above). So tar extraction performance seems to get much worse after the system was under heavy I/O for a while (i.e., after make build make release). Can anyone reproduce this? Best Regards Andreas
Re: softdep issue in 5.3-current ?
I made a new build of current and the problem with tar performance seems to be resolved now. before: # time tar -xzpf /usr/releasedir/comp53.tgz 3m17.81s real 0m2.14s user 0m2.22s system # time tar -xzpf /usr/releasedir/base53.tgz 3m39.33s real 0m2.23s user 0m2.23s system after: # dmesg|head -n2 OpenBSD 5.3-current (GENERIC.MP) #0: Tue Jul 2 22:44:07 CEST 2013 root@test:/usr/src/sys/arch/amd64/compile/GENERIC.MP # time tar -xzpf /usr/releasedir/comp53.tgz 0m8.92s real 0m1.80s user 0m1.07s system # time tar -xzpf /usr/releasedir/base53.tgz 0m11.29s real 0m2.21s user 0m1.17s system Best Regards Andreas
Re: softdep issue in 5.3-current ?
On Fri, Jun 28, 2013 at 10:25 PM, Andreas Bartelt o...@bartula.de wrote: I also noticed that tar performance got much worse on current, and time for building release doubled somewhere around the first half of June. Hmm, please excuse my frustration, but I'm going to have to rant a moment. Gah! Build performance *was cut in half* and you didn't say anything until *at least two weeks later*!?!? No last known good date given for when things were fast. No date given for when you first noticed that it had slowed down. No definitive time measurements. No dmesg or information about the system involved (softdeps? NFS? hell, *architecture*?!?!) additional text deleted Folks, if you're following -current and see a support or performance regression, SAY SOMETHING. We test as much as we believe necessary and reasonable, which means that sometimes we miss cases that are actually show-stoppers and other times we're simply wrong. Delays in reporting just make it harder. Philip Guenther
Re: softdep issue in 5.3-current ?
On 06/29/13 08:15, Philip Guenther wrote: On Fri, Jun 28, 2013 at 10:25 PM, Andreas Bartelt o...@bartula.de wrote: I also noticed that tar performance got much worse on current, and time for building release doubled somewhere around the first half of June. Hmm, please excuse my frustration, but I'm going to have to rant a moment. Gah! Build performance *was cut in half* and you didn't say anything until *at least two weeks later*!?!? No last known good date given for when things were fast. No date given for when you first noticed that it had slowed down. No definitive time measurements. No dmesg or information about the system involved (softdeps? NFS? hell, *architecture*?!?!) additional text deleted Folks, if you're following -current and see a support or performance regression, SAY SOMETHING. We test as much as we believe necessary and reasonable, which means that sometimes we miss cases that are actually show-stoppers and other times we're simply wrong. Delays in reporting just make it harder. sorry, I'm quite busy atm. My last known good /bsd.mp kernel is from June 7: OpenBSD 5.3-current (GENERIC.MP) #0: Fri Jun 7 07:15:17 CEST 2013 I first noticed this problem a couple of days later. I usually build release via the following script (12 cores, hyperthreading deactivated): # cat buildsrc.sh #!/bin/sh cd /usr/src \ make obj \ cd etc \ env DESTDIR=/ make distrib-dirs \ cd /usr/src \ make -j12 build \ export DESTDIR=/usr/destdir; export RELEASEDIR=/usr/releasedir \ cd /usr/src/etc \ make -j12 release \ cd /usr/src/distrib/sets \ sh checkflist \ unset RELEASEDIR DESTDIR time ./buildsrc.sh took about 41 minutes at 5.3 release, then went down to 32 minutes at some point afterwards. At some point after June 7th, build time doubled to 64 minutes. As I already mentioned, tar got very slow. softdep is enabled on this box, kern.bufcachepercent=40, system is on a raid0 drive which spans over 2 ciss(4) HW raid1 drives. ciss(4) performance generally isn't optimal but I suppose this isn't directly related to the problem described above. # mount /dev/sd2a on / type ffs (local, noatime, softdep) /dev/sd2e on /usr type ffs (local, noatime, nodev, softdep) /dev/sd2d on /var type ffs (local, noatime, nodev, nosuid, softdep) # bioctl sd2 Volume Status Size Device softraid0 0 Online 599704600576 sd2 RAID0 0 Online 299852318720 0:0.0 noencl sd0d 1 Online 299852318720 0:1.0 noencl sd1d Best Regards Andreas OpenBSD 5.3-current (GENERIC.MP) #0: Thu Jun 27 22:50:00 CEST 2013 root@test:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 17152794624 (16358MB) avail mem = 16688459776 (15915MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xdf7fe000 (127 entries) bios0: vendor HP version P68 date 01/28/2011 bios0: HP ProLiant DL360 G7 acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP SPCR MCFG HPET SPMI ERST APIC SRAT BERT HEST DMAR SSDT SSDT SSDT SSDT SSDT acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimcfg0 at acpi0 addr 0xe000, bus 0-63 acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz, 2667.16 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG,LAHF,PERF,ITSC cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 cpu0: apic clock running at 133MHz cpu1 at mainbus0: apid 32 (application processor) cpu1: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz, 2666.77 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG,LAHF,PERF,ITSC cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 0, package 1 cpu2 at mainbus0: apid 16 (application processor) cpu2: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz, 2666.77 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG,LAHF,PERF,ITSC cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 8, package 0 cpu3 at mainbus0: apid 48 (application processor) cpu3: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz, 2666.76 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG,LAHF,PERF,ITSC cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 8, package 1 cpu4 at mainbus0: apid
Re: softdep issue in 5.3-current ?
On 29 June 2013 09:51, Andreas Bartelt o...@bartula.de wrote: snip time ./buildsrc.sh took about 41 minutes at 5.3 release, then went down to 32 minutes at some point afterwards. At some point after June 7th, build time doubled to 64 minutes. /snip Hi Andreas, story doesn't tell whether you have sysctl kern.pool_debug set to 0. Is it? In release it is, in current it is not. -- Sincerely, Ville Valkonen
Re: softdep issue in 5.3-current ?
On 06/29/13 11:18, Ville Valkonen wrote: On 29 June 2013 09:51, Andreas Bartelt o...@bartula.de wrote: snip time ./buildsrc.sh took about 41 minutes at 5.3 release, then went down to 32 minutes at some point afterwards. At some point after June 7th, build time doubled to 64 minutes. /snip Hi Andreas, story doesn't tell whether you have sysctl kern.pool_debug set to 0. Is it? In release it is, in current it is not. yep, forgot to mention this... kern.pool_debug was set to 0 during all these measurements. All measurements were done on current -- with 41 minutes at 5.3 release I actually meant the approximate time when 5.3 was released. The build time improvement to 32 minutes happened about mid-May, if I remember correctly. # cat /etc/sysctl.conf |grep -v ^# kern.pool_debug=0 kern.bufcachepercent=40 Best Regards Andreas
Re: softdep issue in 5.3-current ?
On 06/26/13 12:35, Tori Mus wrote: Hi, I'm running current snapshot of OpenBSD on amd64 architecture, MP kernel (Lenovo Thinkpad to be concrete). Based on the official docs tried to tune disk performance by adding `softdep' mounting option for ffs slices. After updating of /etc/fstab and clean reboot, checked all particular slices like /home, /usr etc. are really mounted with softdep. The issue is about much worse performance then with the default nosoftdep. Now, for example, when extracting ports.tar.gz snapshot in /usr, other process cann't open even small files without very long delays like vi $HOME/.profile takes about 2 minutes whereas cpu usage shown with top is about 5% only ! Turning off softdep redeems the access time of the previous example to about 4 seconds. I've searched mailing lists and read about softdep regression on OpenBSD 4.8 that was later fixed. Is this regression back. Does anybody else experiences similar behaviour ? I also noticed that tar performance got much worse on current, and time for building release doubled somewhere around the first half of June. Best Regards Andreas
Re: softdep issue in 5.3-current ?
Unfortunately, even after new kernel build with version 1.27 of sys/kern/vfs_biomem.c issue with `softdep' persists. dmesg output: OpenBSD 5.3-current (GENERIC.MP) #0: Fri Jun 28 00:20:42 ... bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xe0010 (44 entries) bios0: vendor LENOVO version 6JET93WW (1.51 ) date 03/26/2012 bios0: LENOVO 28477TG ... cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM)2 Duo CPU T6570 @ 2.10GHz, 2095.09 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF cpu0: 2MB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 cpu0: apic clock running at 199MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Core(TM)2 Duo CPU T6570 @ 2.10GHz, 2094.76 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF cpu1: 2MB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins ... cpu0: Enhanced SpeedStep 2095 MHz: speeds: 2101, 2100, 1600, 1200 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 Intel GM45 Host rev 0x07 ... azalia0 at pci0 dev 27 function 0 Intel 82801I HD Audio rev 0x03: msi azalia0: codecs: Realtek ALC269, Intel/0x2802, using Realtek ALC269 audio0 at azalia0 ppb0 at pci0 dev 28 function 0 Intel 82801I PCIE rev 0x03: msi pci1 at ppb0 bus 2 JMicron SD/MMC rev 0x00 at pci1 dev 0 function 0 not configured sdhc0 at pci1 dev 0 function 2 JMicron SD Host Controller rev 0x00: apic 2 int 16 sdmmc0 at sdhc0 JMicron Memory Stick rev 0x00 at pci1 dev 0 function 3 not configured JMicron xD rev 0x00 at pci1 dev 0 function 4 not configured ppb1 at pci0 dev 28 function 1 Intel 82801I PCIE rev 0x03: msi pci2 at ppb1 bus 3 ppb2 at pci0 dev 28 function 2 Intel 82801I PCIE rev 0x03: msi pci3 at ppb2 bus 4 ppb3 at pci0 dev 28 function 3 Intel 82801I PCIE rev 0x03: msi pci4 at ppb3 bus 5 Realtek 8192SE rev 0x10 at pci4 dev 0 function 0 not configured ppb4 at pci0 dev 28 function 4 Intel 82801I PCIE rev 0x03: msi pci5 at ppb4 bus 6 ppb5 at pci0 dev 28 function 5 Intel 82801I PCIE rev 0x03: msi pci6 at ppb5 bus 8 ... ppb6 at pci0 dev 30 function 0 Intel 82801BAM Hub-to-PCI rev 0x93 pci7 at ppb6 bus 9 pcib0 at pci0 dev 31 function 0 Intel 82801IBM LPC rev 0x03 ahci0 at pci0 dev 31 function 2 Intel 82801I AHCI rev 0x03: msi, AHCI 1.2 scsibus0 at ahci0: 32 targets sd0 at scsibus0 targ 0 lun 0: ATA, Hitachi HTS54757, JE4O SCSI3 0/direct fixed naa.5000cca63fc2c8ee sd0: 715404MB, 512 bytes/sector, 1465149168 sectors cd0 at scsibus0 targ 1 lun 0: HL-DT-ST, DVDRAM GT30N, LG09 ATAPI 5/cdrom removable ichiic0 at pci0 dev 31 function 3 Intel 82801I SMBus rev 0x03: apic 2 int 19 iic0 at ichiic0 spdmem0 at iic0 addr 0x50: 2GB DDR3 SDRAM PC3-8500 SO-DIMM ... isa0 at pcib0 isadma0 at isa0 ... mtrr: Pentium Pro MTRR support vscsi0 at root scsibus1 at vscsi0: 256 targets softraid0 at root scsibus2 at softraid0: 256 targets root on sd0a (0441f60ac76c6254.a) swap on sd0b dump on sd0b - snip --- /etc/fstab 0441f60ac76c6254.b none swap sw 0441f60ac76c6254.a / ffs rw,softdep 1 1 0441f60ac76c6254.m /home ffs rw,nodev,nosuid,softdep 1 2 0441f60ac76c6254.d /tmp ffs rw,nodev,nosuid,softdep 1 2 0441f60ac76c6254.f /usr ffs rw,nodev,softdep 1 2 0441f60ac76c6254.g /usr/X11R6 ffs rw,nodev,softdep 1 2 0441f60ac76c6254.h /usr/local ffs rw,nodev,softdep 1 2 0441f60ac76c6254.l /usr/obj ffs rw,nodev,nosuid,softdep 1 2 0441f60ac76c6254.k /usr/src ffs rw,nodev,softdep 1 2 0441f60ac76c6254.e /var ffs rw,nodev,softdep 1 2 Any idea why softdep hogs / limits simultaneous didk access ? 2013/6/27 Bob Beck b...@openbsd.org Update to something that has version 1.27 of sys/kern/vfs_biomem.c and tell me if you still have the issue. On Wed, Jun 26, 2013 at 4:35 AM, Tori Mus torimus...@gmail.com wrote: Hi, I'm running current snapshot of OpenBSD on amd64 architecture, MP kernel (Lenovo Thinkpad to be concrete). Based on the official docs tried to tune disk performance by adding `softdep' mounting option for ffs slices. After updating of /etc/fstab and clean reboot, checked all particular slices like /home, /usr etc. are really mounted with softdep. The issue is about much worse performance then with the default nosoftdep. Now, for example, when extracting ports.tar.gz snapshot in /usr, other process cann't open even small files without very long delays like vi $HOME/.profile takes about 2 minutes whereas cpu usage shown with top is about 5% only ! Turning off softdep redeems the access time of the previous example to about 4 seconds. I've searched mailing lists and read about softdep regression on OpenBSD 4.8 that was later fixed. Is this regression back. Does anybody else
softdep issue in 5.3-current ?
Hi, I'm running current snapshot of OpenBSD on amd64 architecture, MP kernel (Lenovo Thinkpad to be concrete). Based on the official docs tried to tune disk performance by adding `softdep' mounting option for ffs slices. After updating of /etc/fstab and clean reboot, checked all particular slices like /home, /usr etc. are really mounted with softdep. The issue is about much worse performance then with the default nosoftdep. Now, for example, when extracting ports.tar.gz snapshot in /usr, other process cann't open even small files without very long delays like vi $HOME/.profile takes about 2 minutes whereas cpu usage shown with top is about 5% only ! Turning off softdep redeems the access time of the previous example to about 4 seconds. I've searched mailing lists and read about softdep regression on OpenBSD 4.8 that was later fixed. Is this regression back. Does anybody else experiences similar behaviour ?
Re: softdep issue in 5.3-current ?
Update to something that has version 1.27 of sys/kern/vfs_biomem.c and tell me if you still have the issue. On Wed, Jun 26, 2013 at 4:35 AM, Tori Mus torimus...@gmail.com wrote: Hi, I'm running current snapshot of OpenBSD on amd64 architecture, MP kernel (Lenovo Thinkpad to be concrete). Based on the official docs tried to tune disk performance by adding `softdep' mounting option for ffs slices. After updating of /etc/fstab and clean reboot, checked all particular slices like /home, /usr etc. are really mounted with softdep. The issue is about much worse performance then with the default nosoftdep. Now, for example, when extracting ports.tar.gz snapshot in /usr, other process cann't open even small files without very long delays like vi $HOME/.profile takes about 2 minutes whereas cpu usage shown with top is about 5% only ! Turning off softdep redeems the access time of the previous example to about 4 seconds. I've searched mailing lists and read about softdep regression on OpenBSD 4.8 that was later fixed. Is this regression back. Does anybody else experiences similar behaviour ?
