Hi,
I am working on new lang/rust version (next stable in 1 week), and I
have problem building it under i386 (full dmesg below).
I suspect memory pressure in some way, but I dunno options I have to
workaround (if possible).
It is possible that occasionnal failures sthen@ saw in bulk with current
rustc version (1.21) in ports to be related. The version 1.22 hits it
almost at every build try.
In short:
- the build aborts due to ENOMEM (mmap(2) calls return ENOMEM. the program
asked for 4096 bytes)
- kdump(8) reports lot of
mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0) calls
- at ENOMEM time, the rustc process has SIZE and RES (from top(1)) to almost
1.7 Go
- the ulimit data was 3145728 (3Go)
- the host has 4Go RAM installed and dmesg reports avail mem = 3149758464
(3003MB)
- the host has swap, but system doesn't seems to use it at any time
- rustc is threaded (but only one thread actively works)
the top(1) output is:
load averages: 1.13, 0.81, 0.41
rust-i386.proxmox2 14:20:12
49 threads: 1 running, 46 idle, 2 on processor
up 1:47
CPU0 states: 7.7% user, 0.0% nice, 86.5% system, 0.0% interrupt, 5.8% idle
CPU1 states: 1.9% user, 0.0% nice, 7.7% system, 0.0% interrupt, 90.4% idle
Memory: Real: 1680M/2021M act/tot Free: 979M Cache: 242M Swap: 0K/2946M
Process ID to show:
PID TID PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
83860 403198 57 0 1652M 1623M run/0 - 4:28 99.02% rustc
83860 240149 36 0 1652M 1623M idle thrslee 0:00 0.00% rustc
83860 412091 53 0 1652M 1623M idle fsleep 0:00 0.00% rustc
83860 106973 53 0 1652M 1623M idle fsleep 0:00 0.00% rustc
The process died at bit later.
I instrumented the kernel to understand the origin of ENOMEM from
mmap(2). It comes from uvm_map_findspace().
sys_mmap(size=4096)
uvm_mmapanon()
uvm_mapanon()
uvm_map_findspace()
During the build, vmstat -m outputs looks like:
# vmstat -m
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 6645 11 24940 1280 280
32 3179 21 14376 640 1081
64 3856 48 26549 320 324
128 4515 445 13628 160 614
256 444 4 5922 80 932
512 559 17 1239 40 0
1024 170 10 6852 20 0
2048 23 3 81 10 0
4096 41 1 1430 5 0
8192 197 1 202 5 0
16384 6 0 2269 5 0
32768 2 0 3 5 0
65536 1 0 10370 5 0
131072 2 0 2 5 0
524288 1 0 1 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, rtable, ifaddr, UFS mount, dirhash, ACPI, in_multi, exec,
VM swap, UVM amap, UVM aobj, USB, USB device, temp
32 devbuf, pcb, rtable, ifaddr, vnodes, UFS mount, sem, dirhash, ACPI,
proc, ether_multi, VM swap, UVM amap, USB, USB device, NDP, temp
64 devbuf, pcb, rtable, ifaddr, sysctl, counters, sem, dirhash, ACPI,
NFS srvsock, in_multi, UVM amap, USB, USB device, temp
128 devbuf, pcb, rtable, ifaddr, counters, vnodes, UFS mount, VM map,
dirhash, ACPI, UVM amap, USB, USB device, temp
256 devbuf, rtable, ifaddr, sysctl, counters, ioctlops, iov, UFS mount,
shm, file desc, exec, newblk, UVM amap, temp
512 devbuf, pcb, counters, ioctlops, shm, dirhash, proc, ttys, UVM amap,
temp
1024 devbuf, counters, ioctlops, mount, ACPI, ttys, exec, UVM amap,
UVM aobj, USB, crypto data, temp
2048 devbuf, ioctlops, UFS mount, proc, VM swap, temp
4096 devbuf, ifaddr, counters, ioctlops, ttys, pagedep, memdesc, temp,
SYN cache
8192 devbuf, counters, NFS daemon, MSDOSFS mount, ttys, temp
16384 devbuf, UFS quota, UFS mount, ISOFS mount, inodedep, temp
32768 devbuf
65536 devbuf, temp
131072 VM swap
524288 temp
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
devbuf 1345 500K 501K 39322K 2294 0 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
pcb 77 12K 13K 39322K 635 0 0 16,32,64,128,512
rtable 69 2K 3K 39322K 141 0 0 16,32,64,128,256
ifaddr 25 8K 8K 39322K 25 0 0
16,32,64,128,256,4096
sysctl 2 1K 1K 39322K 2 0 0 64,256
counters 37 34K 34K 39322K 37 0 0
64,128,256,512,1024,4096,8192
ioctlops 0 0K 4K 39322K 1475 0 0
256,512,1024,2048,4096
iov 0 0K 1K 39322K 1 0 0 256
mount 6 6K 6K 39322K 6 0 0 1024
vnodes 1160 37K 37K 39322K 1167 0 0 32,128
UFS quota 1 16K 16K 39322K 1 0 0 16384
UFS mount 25 42K 42K 39322K 25 0 0
16,32,128,256,2048,16384
shm 2 1K 1K 39322K 2 0 0 256,512
VM map 2 1K 1K 39322K 2 0 0 128
sem 2 1K 1K 39322K 2 0 0 32,64
dirhash 60 12K 12K 39322K 84 0 0 16,32,64,128,512
ACPI 1359 81K 171K 39322K 20319 0 0
16,32,64,128,1024
file desc 1 1K 1K 39322K 1 0 0 256
proc 16 7K 7K 39322K 16 0 0 32,512,2048
NFS srvsock 1 1K 1K 39322K 1 0 0 64
NFS daemon 1 8K 8K 39322K 1 0 0 8192
in_multi 14 1K 1K 39322K 14 0 0 16,64
ether_multi 1 1K 1K 39322K 1 0 0 32
ISOFS mount 1 16K 16K 39322K 1 0 0 16384
MSDOSFS mount 1 8K 8K 39322K 1 0 0 8192
ttys 414 1751K 1751K 39322K 414 0 0
512,1024,4096,8192
exec 0 0K 2K 39322K 1058 0 0 16,256,1024
pagedep 1 4K 4K 39322K 1 0 0 4096
inodedep 1 16K 16K 39322K 1 0 0 16384
newblk 1 1K 1K 39322K 1 0 0 256
VM swap 7 215K 215K 39322K 7 0 0
16,32,2048,131072
UVM amap 14896 938K 941K 39322K 50764 0 0
16,32,64,128,256,512,1024
UVM aobj 2 2K 2K 39322K 2 0 0 16,1024
USB 26 3K 3K 39322K 29 0 0
16,32,64,128,1024
USB device 5 1K 1K 39322K 5 0 0 16,32,64,128
memdesc 1 4K 4K 39322K 1 0 0 4096
crypto data 1 1K 1K 39322K 1 0 0 1024
NDP 4 1K 1K 39322K 4 0 0 32
temp 73 2121K 2250K 39322K 29321 0 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,65536,524288
SYN cache 2 8K 8K 39322K 2 0 0 4096
Memory Totals: In Use Free Requests
5848K 97K 107865
Memory resource pool statistics
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
phpool 56 1083 0 593 9 0 9 9 0 8 0
extentpl 20 247 0 38 1 0 1 1 0 8 0
pmappl 160 1004 0 46 2 0 2 2 0 8 0
pvpl 16 2700998 0 430401 3698 1990 1708 1775 1 3 0
vmsppl 160 1004 0 46 2 0 2 2 0 8 0
vmmpepl 100 864738 0 225557 11650 6011 5639 6017 0 205 0
vmmpekpl 100 38658 0 187 39 2 37 39 0 8 0
uaddr 12 1005 0 47 1 0 1 1 0 8 0
uaddrbest 16 2 0 2 1 0 1 1 0 8 0
uaddrrnd 12 1005 0 47 1 0 1 1 0 8 0
aobjpl 40 1 0 1 1 0 1 1 0 8 0
dma32 32 7 0 0 1 1 0 1 0 8 0
dma64 64 2 0 0 1 1 0 1 0 8 0
dma256 256 8 0 0 1 1 0 1 0 8 0
dma1024 1024 1 0 1 1 0 1 1 0 8 0
dma4096 4096 1 0 0 1 1 0 1 0 8 0
amappl 48 18616 0 5169 123 61 62 62 0 49 0
amappl1 48 543363 0 193214 4854 2553 2301 2478 0 8 0
amappl2 52 15583 0 4858 127 63 64 65 0 8 0
amappl3 56 6765 0 1850 55 29 26 26 0 8 0
amappl4 60 8832 0 1708 63 37 26 26 0 8 0
amappl5 64 2667 0 401 15 8 7 7 0 8 0
amappl6 68 3736 0 1069 41 22 19 19 0 8 0
amappl7 72 3754 0 985 35 17 18 18 0 8 0
amappl8 76 3343 0 554 22 11 11 12 0 8 0
amappl9 80 535 0 141 5 2 3 3 0 8 0
amappl10 84 1684 0 171 8 4 4 5 0 8 0
amappl11 88 830 0 144 8 4 4 4 0 8 0
amappl12 92 1544 0 558 26 12 14 14 0 8 0
amappl13 96 3324 0 764 36 17 19 19 0 8 0
amappl14 100 549 0 135 6 2 4 4 0 8 0
amappl15 104 535 0 59 2 0 2 2 0 8 0
amappl16 108 27994 0 6892 415 218 197 199 0 8 5
amapchunkpl 80 56012 0 8256 329 163 166 167 0 82 0
anonpl 12 1934635 0 402972 2468 1257 1211 1247 0 143 11
bufpl 152 21365 0 15148 624 41 583 583 0 8 0
mbufpl 256 23798 0 146 15 1 14 14 0 8 0
mcl2k 2048 6020 0 24 6 0 6 6 0 8 0
mcl2k2 2112 3 0 0 1 0 1 1 0 8 0
mcl4k 4096 42 0 2 1 0 1 1 0 8 0
mcl8k 8192 28 0 0 1 0 1 1 0 8 0
mcl9k 9216 5 0 0 1 0 1 1 0 8 0
sockpl 228 704 0 77 5 0 5 5 0 8 0
procpl 356 1070 0 72 8 1 7 7 0 8 0
processpl 456 1026 0 67 33 24 9 9 0 8 0
zombiepl 80 960 0 1 189 188 1 1 0 8 0
ucredpl 96 160 0 28 1 0 1 1 0 8 0
pgrppl 24 93 0 29 1 0 1 1 0 8 0
sessionpl 80 26 0 24 1 0 1 1 0 8 0
lockfpl 56 122 0 2 2 1 1 1 0 8 0
filepl 80 53368 0 164 4 0 4 4 0 8 0
fdescpl 304 1005 0 47 5 1 4 4 0 8 0
pipepl 84 504 0 10 1 0 1 1 0 8 0
kqueuepl 36 13 0 13 1 0 1 1 0 8 0
knotepl 72 12267 0 83 2 0 2 2 0 8 0
futexpl 28 41 0 4 2 1 1 1 0 8 0
sigapl 296 1004 0 46 4 0 4 4 0 8 0
scxspl 120 20331 0 0 214 213 1 3 0 8 1
uhcixfer 160 39 0 1 1 0 1 1 0 8 0
percpumem 8 29 0 29 1 0 1 1 0 8 0
namei 1024 104291 0 0 210 209 1 2 0 8 1
vnodes 116 6412 0 6412 184 0 184 184 0 8 0
uvmvnodes 48 6412 0 6412 77 0 77 77 0 8 0
nchpl 88 11335 0 5926 155 23 132 132 0 8 0
ffsino 184 6397 0 4761 255 38 217 217 0 8 0
dino1pl 128 6397 0 4761 176 27 149 149 0 8 0
dirhash 1024 123 0 89 12 0 12 12 0 8 0
art_node 8 26 0 26 1 0 1 1 0 8 0
art_table 24 98 0 98 1 0 1 1 0 8 0
art_heap4 128 97 0 97 4 0 4 4 0 8 0
art_heap8 2048 1 0 1 1 0 1 1 0 8 0
pfosfpen 108 1428 0 714 20 0 20 20 0 8 0
pfosfp 28 846 0 423 3 0 3 3 0 8 0
tcpcb 396 17 0 6 1 0 1 1 0 8 0
sackhl 20 3 0 0 3 3 0 1 0 8 0
syncache 196 1 0 0 1 1 0 1 0 8 0
rtentry 76 26 0 26 1 0 1 1 0 8 0
plcache 128 20 0 20 1 0 1 1 0 8 0
plimitpl 148 37 0 25 1 0 1 1 0 8 0
inpcbpl 196 83 0 14 1 0 1 1 0 8 0
arp 28 2 0 2 1 0 1 1 0 8 0
In use 50704K, total allocated 52456K; utilization 96.7%
I assume "utilization 96.7%" isn't a good sign.
Any pointer would be welcome.
Thanks.
--
Sebastien Marie
OpenBSD 6.2-current (GENERIC.MP) #7: Thu Nov 16 12:32:21 CET 2017
[email protected]:/home/semarie/repos/openbsd/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Common KVM processor ("GenuineIntel" 686-class) 2.94 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,NXE,LONG,SSE3,CX16,x2APIC,HV,LAHF
real mem = 3220570112 (3071MB)
avail mem = 3144904704 (2999MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: date 06/23/99, BIOS32 rev. 0 @ 0xff046, SMBIOS rev. 2.8 @
0xf69f0 (10 entries)
bios0: vendor SeaBIOS version "rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org"
date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 999MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Common KVM processor ("GenuineIntel" 686-class) 2.94 GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,NXE,LONG,SSE3,CX16,x2APIC,HV,LAHF
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
bios0: ROM list: 0xc0000/0x9600 0xc9800/0xe00 0xca800/0x2400 0xed800/0x2800!
pvbus0 at mainbus0: KVM
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0
wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Memory" rev 0x00
viomb0 at virtio0
virtio0: apic 0 int 11
virtio1 at pci0 dev 10 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus1 at vioblk0: 2 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd0: 35840MB, 512 bytes/sector, 73400320 sectors
virtio1: apic 0 int 10
virtio2 at pci0 dev 18 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio2: address a2:6e:16:9f:42:74
virtio2: apic 0 int 10
ppb0 at pci0 dev 30 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci1 at ppb0 bus 1
ppb1 at pci0 dev 31 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci2 at ppb1 bus 2
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00
addr 1
nvram: invalid checksum
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev
2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (c763ffdb54130f4b.a) swap on sd0b dump on sd0b
clock: unknown CMOS layout
fd0 at fdc0 drive 1: density unknown
uvm_mapanon:1037: ENOMEM from uvm_map_findspace()
uvm_mmapanon:1062: ENOMEM from uvm_mapanon(): size=4096
sys_mmap:558: ENOMEM from uvm_mmapanon(): size=4096
