Re: Strange ARC/Swap/CPU on yesterday's -CURRENT
On 2018-Mar-17, at 11:26 AM, Andriy Gapon wrote: > On 17/03/2018 18:51, Mark Millard wrote: >> I'll note that top was a -w that reports: >> >> -w Display approximate swap usage for each process. > > As far as I can tell, this option is quite broken. > The "approximate swap usage" it reports is nowhere like it. I have a hypothesis for part of what top is counting in the process/thread SWAP column that might not be what one would expect. It appears to me that vnode-backed pages are being re-classfied sometimes for inactive processes, and this classification leads to top classifying the pages as not-resident but swapped (in that a "VN PAGER in" would be required, in systat -vmstat terms). Supporting details, if you care, otherwise skip the below: The hypothesis is from observing various hours of over 20 hours of poudriere-devel "bulk -a" activity, for which, at that point: vm.stats.vm.v_swappgsout: 0 vm.stats.vm.v_swappgsin: 0 vm.stats.vm.v_swapout: 0 vm.stats.vm.v_swapin: 0 vm.stats.vm.v_vnodepgsout: 6996 vm.stats.vm.v_vnodepgsin: 32641833 vm.stats.vm.v_vnodeout: 1030 vm.stats.vm.v_vnodein: 4305027 Sometimes top showed lots of wait/select/pause and such with positive SWAP and (mostly) 0K RES. Most of the processes were in the "wait" STATE. At other times, more like between 1 and 2 dozen had positive SWAP. There would be sudden large jumps in the number of such processes. Then over time it would decrease as the processes quit waiting (children process trees finished). The large jumps were not tied to Free becoming small or anything else obvious from what I was looking at. But the Free figure would increase at that time. For example, I recently saw such a large jump that was associated with Free increasing from "90G" as shown in top. (Much of the time there were between, say, 170 and 300 sleeping processes.) The context was under Hyper-V with 29 logical processors assigned to FreeBSD on a machine with 16 cores/32 threads and 114 GiBytes of RAM assigned to FreeBSD (of 128 GiBytes) and 256 GiBytes of swap-partition set up. PARALLEL_JOBS allowed the 29 and ALLOW_MAKE_JOBS was in use (allowing a potential for 29*29=841 or so running processes via poudriere bulk). For reference: at 25 hours-in [idle] had 148.3H (around 20% of the 29 threads * 25 H/thread) and [bufdaemon] had 48.9H (around 6.7%). [kernel] showed around 13.6H (817 min converted) and [pagedaemon] showed around 1.7H (101 min converted). Other processes had less TIME than any of these. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: i386 4/4 change
On Sun, Apr 01, 2018 at 01:05:57AM +0200, Dimitry Andric wrote: > I haven't yet run any performance tests, I'll try building world and a > few large ports tomorrow. General operation from the command line does > not feel "sluggish" in any way, however. I just updated the review with some changes which should have effect on the copyout performance. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: i386 4/4 change
On 31 Mar 2018, at 17:57, Bruce Evans wrote: > > On Sat, 31 Mar 2018, Konstantin Belousov wrote: > >> the change to provide full 4G of address space for both kernel and >> user on i386 is ready to land. The motivation for the work was to both >> mitigate Meltdown on i386, and to give more breazing space for still >> used 32bit architecture. The patch was tested by Peter Holm, and I am >> satisfied with the code. >> >> If you use i386 with HEAD, I recommend you to apply the patch from >> https://reviews.freebsd.org/D14633 >> and report any regressions before the commit, not after. Unless >> a significant issue is reported, I plan to commit the change somewhere >> at Wed/Thu next week. >> >> Also I welcome patch comments and reviews. > > It crashes at boot time in getmemsize() unless booted with loader which > I don't want to use. For me, it at least compiles and boots OK, but I'm one of those crazy people who use the default boot loader. ;) I haven't yet run any performance tests, I'll try building world and a few large ports tomorrow. General operation from the command line does not feel "sluggish" in any way, however. -Dimitry signature.asc Description: Message signed with OpenPGP
Re: i386 4/4 change
On Sat, 31 Mar 2018, Konstantin Belousov wrote: the change to provide full 4G of address space for both kernel and user on i386 is ready to land. The motivation for the work was to both mitigate Meltdown on i386, and to give more breazing space for still used 32bit architecture. The patch was tested by Peter Holm, and I am satisfied with the code. If you use i386 with HEAD, I recommend you to apply the patch from https://reviews.freebsd.org/D14633 and report any regressions before the commit, not after. Unless a significant issue is reported, I plan to commit the change somewhere at Wed/Thu next week. Also I welcome patch comments and reviews. It crashes at boot time in getmemsize() unless booted with loader which I don't want to use. It is much slower, and I couldn't find an option to turn it off. For makeworld, the system time is slightly more than doubled, the user time is increased by 16%, and the real time is increased by 21%. On amd64, turning off pti and not having ibrs gives almost no increase in makeworld times relative to old versions, and pti only costs about 5% IIRC. Makeworld is not very syscall-intensive. netblast is very syscall-intensive, and its throughput is down by a factor of 5 (660/136 = 4.9, 1331/242 = 5.5). netblast 127.0.0.1 5001 5 10 (localhost, port 5001, 5-byte tinygrams for 10 s): 537 kpps sent, 0 kpps dropped # before this patch (CPU use 1.3) 136 kpps sent, 0 kpps dropped # after (CPU use 2.1) (Pure software overheads. It uses 1.6 times as much CPU to go 4 times slower). netblast 192.168.2.8 (low end PCI33 lem on low latency 1 Gbps LAN) 275 kpps sent, 1045 kpps dropped # before (CPU use 1.3) 245 kpps sent, 0kpps dropped # after (CPU use 1.3) (The hardware can't do anywhere near line rate of ~1500 kpps, so this becomes a benchmark of syscalls and dropping packets. The change makes FreeBSD so slow that 8 CPUs at 4.08 can't saturate a low end PCI33 NIC (the hardware saturates at about 282 kpps for tx and about 400 kpps for rx)). netblast 192.168.2.8 (low end PCIe em on low latency 1 Gbps LAN) 1316 kpps sent, 3 kpps dropped # before (CPU use 1.6) 243 kpps sent, 0 kpps dropped # after (CPU use 1.2) This is seriously slower for the most useful case. It reduces a system that could almost reach line rate using about 2 of 8 CPUs at 4 GHz to one that that is slower than with 1 CPU at 2 GHz (the latter saturates in software at about 640 kpps in old versions of FreeBSD at at about 400 kpps in -current). Initial debugging of the crash: it crashes on the first pmap_kenter() in getmemsize(). I configure debug.late_console to 0. That works, and without it getmemsize() can't even be debugged since it is after console initialization and ddb entry with -d. In getmemsize(), of course all the preload calls return 0 and smapbase is NULL. Then vm86 bios calls work and give basemem = 0x276. Then basemem_setup() is called and it returns. Then pmap_kenter() is called and it crashes: Stopped at getmemsize+0xb3:pushl $0x1000 Stopped at getmemsize+0xb8:pushl $0x1000 Stopped at getmemsize+0xbd:callpmap_kenter Stopped at pmap_kenter:pushl %ebp Stopped at pmap_kenter+0x1:movl%esp,%ebp Stopped at pmap_kenter+0x3:movl0x8(%ebp),%eax Stopped at pmap_kenter+0x6:shrl$0xc,%eax Stopped at pmap_kenter+0x9:movl0xc(%ebp),%edx Stopped at pmap_kenter+0xc:orl $0x3,%edx Stopped at pmap_kenter+0xf:movl%edx,PTmap(,%eax,4) The last instruction crashes because PTmap is not mapped at this point: db> p/x $edx 1003 db> p/x PTmap ff80 db> p/x $eax 1 db> x/x PTmap PTmap:KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(cec5cb,1420a04,c6de83,1420978,1,...) at db_trace_self_wrapper+0x24/frame 0x142095c kdb_reenter(1420978,1,ff80003a,1420998,8f1419,...) at kdb_reenter+0x24/frame 0x1420968 trap(1420a10) at trap+0xa0/frame 0x1420a04 calltrap() at calltrap+0x8/frame 0x1420a04 --- trap 0xc, eip = 0xc5c394, esp = 0x1420a50, ebp = 0x1420a88 --- db_read_bytes(ff81,3,1420aa0) at db_read_bytes+0x29/frame 0x1420a88 db_get_value(ff80,4,0,0,d2d304,...) at db_get_value+0x20/frame 0x1420ab4 db_examine(ff80,1,,1420b00) at db_examine+0x144/frame 0x1420ae4 db_command(cb1d99,1420be4,8f0f01,d1d28a,0,...) at db_command+0x20a/frame 0x1420b90 db_command_loop(d1d28a,0,1420bac,1420b9c,1420be4,...) at db_command_loop+0x55/frame 0x1420b9c db_trap(a,4ff0,1,1,80046,...) at db_trap+0xe1/frame 0x1420be4 kdb_trap(a,4ff0,1420cc4) at kdb_trap+0xb1/frame 0x1420c10 trap(1420cc4) at trap+0x523/frame 0x1420cb8 calltrap() at calltrap+0x8/frame 0x1420cb8 --- trap 0xa, eip = 0xc65a4a, esp = 0x1420d04, ebp = 0x1420d04 --- pmap_kenter(1000,1000,1429000,8efe13,0,...) at pmap_kenter+0xf/frame 0x1420d04 getmemsize(1,5a8807ff,ee,59a80097,ee,...) at getmemsize+0xc2/frame 0x1420fc4 init386(1
Re: 12-Current panics on boot (didn't a week ago.)
The drm-next-kmod, and drm-stable-kmod modules panic for me. I will attach logs when I can. On Friday, March 30, 2018, Andrew Reilly wrote: > Hi Jonathan, all, > > I've just compiled and booted a kernel derived from current-GENERIC > but with nooptions TCP_BLACKBOX, and much to my surprise it boots. > Possible link to network-related activities is that the next line > of boot output that was not being displayed during the crash is: > > [ath_hal] loaded > > That's vaguely network-shaped: could it be an issue? > > Please let me know if there's anything else that I could test or > poke, in order to find the real culprit. > > My make.conf says: > > KERNCONF=ZEN > WRKDIRPREFIX=/usr/obj/ports > MALLOC_PRODUCTION=yes > > My /usr/src/sys/amd64/conf/ZEN says: > > include GENERIC > nooptions TCP_BLACKBOX > > Uname -a says: > FreeBSD Zen.ac-r.nu 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r331768M: Sat > Mar 31 10:47:52 AEDT 2018 root@Zen:/usr/obj/usr/src/amd64.amd64/sys/ZEN > amd64 > > Cheers, > > Andrew > > > Here's the top part of the new dmesg.boot, FYI: > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 12.0-CURRENT #0 r331768M: Sat Mar 31 10:47:52 AEDT 2018 > root@Zen:/usr/obj/usr/src/amd64.amd64/sys/ZEN amd64 > FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM > 6.0.0) > WARNING: WITNESS option enabled, expect reduced performance. > VT(vga): resolution 640x480 > CPU: AMD Ryzen 7 1700 Eight-Core Processor (2994.45-MHz K8-class > CPU) > Origin="AuthenticAMD" Id=0x800f11 Family=0x17 Model=0x1 Stepping=1 > Features=0x178bfbff APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> > Features2=0x7ed8320b SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> > AMD Features=0x2e500800 > AMD Features2=0x35c233ff Prefetch,OSVW,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX> > Structured Extended Features=0x209c01a9 BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA> > XSAVE Features=0xf > AMD Extended Feature Extensions ID EBX=0x7 > SVM: (disabled in BIOS) NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 > TSC: P-state invariant, performance statistics > real memory = 34359738368 (32768 MB) > avail memory = 33271214080 (31729 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs > FreeBSD/SMP: 1 package(s) x 2 cache groups x 4 core(s) > random: unblocking device. > Firmware Warning (ACPI): Optional FADT field Pm2ControlBlock has valid > Length but zero Address: 0x/0x1 (20180313/tbfadt-796) > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 24-55 on motherboard > SMP: AP CPU #7 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #2 Launched! > SMP: AP CPU #6 Launched! > SMP: AP CPU #5 Launched! > SMP: AP CPU #4 Launched! > SMP: AP CPU #1 Launched! > Timecounter "TSC-low" frequency 1497224985 Hz quality 1000 > random: entropy device external interface > [ath_hal] loaded > module_register_init: MOD_LOAD (vesa, 0x8109f600, 0) error 19 > random: registering fast source Intel Secure Key RNG > random: fast provider: "Intel Secure Key RNG" > kbd1 at kbdmux0 > netmap: loaded module > nexus0 > vtvga0: on motherboard > cryptosoft0: on motherboard > aesni0: on motherboard > acpi0: on motherboard > acpi0: Power Button (fixed) > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > attimer0: port 0x40-0x43 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atrtc0: port 0x70-0x71 on acpi0 > atrtc0: registered as a time-of-day clock, resolution 1.00s > Event timer "RTC" frequency 32768 Hz quality 0 > hpet0: iomem 0xfed0-0xfed003ff irq 0,8 on > acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 350 > Event timer "HPET1" frequency 14318180 Hz quality 350 > Event timer "HPET2" frequency 14318180 Hz quality 350 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > amdsmn0: on hostb0 > amdtemp0: on hostb0 > > > On Sun, Mar 25, 2018 at 04:35:31AM +, Jonathan Looney wrote: > > For now, you can update through r331485 and then take TCP_BLACKBOX out of > > your kernel config file. That won’t really “fix” anything, but should at > > least get you a booting system (assuming the new code from r331347 is > > really triggering a problem). > > > > > > I’ll take another look to see if I missed something in the commit. But, > at > > the moment, I’m hard-pressed to see how r331347 would cause the problem
i386 4/4 change
Hi, the change to provide full 4G of address space for both kernel and user on i386 is ready to land. The motivation for the work was to both mitigate Meltdown on i386, and to give more breazing space for still used 32bit architecture. The patch was tested by Peter Holm, and I am satisfied with the code. If you use i386 with HEAD, I recommend you to apply the patch from https://reviews.freebsd.org/D14633 and report any regressions before the commit, not after. Unless a significant issue is reported, I plan to commit the change somewhere at Wed/Thu next week. Also I welcome patch comments and reviews. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
