Re: Memory blackhole in 11. Possibly libc.so.7?
I'm replying late, but last night I almost had a complete meltdown of the system. I did a partial "pkg upgrade" for the packages that managed to get built by poudriere, then all hell broke loose. Screen lock-ups, random reboots, and several hard reboots later I decided to do a fresh buildworld/buildkernel and we seem to be back at some level of normality. It makes no sense, I know; and I would not be able to tell you what it was that went wrong. What I can tell you is that * While compiling, code would compile for a while, then freeze, then continue where it left off. "# top -P" was showing that during these freeze-ups, of the 4 cores on my system, 3 were mostly idle (80% to 95%) while one core would be at or near 0% idle. Not always the same core, and when one core got freed-up, you could have another core drop to 0%-8% idle. I thought maybe radeon was using cpu instead of gpu but that was debunked when same behavior was observed while compiling my regular kernel without radeon / xorg loaded. * Initially a debug-kernel was built with below, but screen goes black when radeon / xorg is loaded. (include GENERIC \ ident KERNDEBUG \ nooptions INET6 \ options KDB \ options DDB \ options GDB \ options INVARIANTS \ options INVARIANT_SUPPORT \ options WITNESS \ options DEBUG_LOCKS \ options DEBUG_VFS_LOCKS \ options DIAGNOSTIC) * I have modified my zfs-related entries in loader.conf, and latest is: vfs.zfs.trim.enabled=1 #Enable_Trim vfs.zfs.prefetch_disable=0 #I have 4G of Ram #vfs.zfs.arc_min="512M" #Ram 4GB => 512, Ram 8GB => 1024 #vfs.zfs.arc_max="1536M" #TotalRam x 0.5 - 512 MB #vm.kmem_size="6G" #Ram x 1.5 #vfs.zfs.vdev_max_pending="1" Currently, kstat.zfs.misc.arcstats.size: 2012423600 and the number seems to hang around in that vicinity. Separate ZFS question then: I have 2 sata-III HDD's (64G SSD + 7200rpm spindle) and 3 zpools. System is my personal desktop running sql, http server etc but no commercial load. tank-b: root, usr, var on SSD tank-d: home, all data, sql-db, NFS exported PXE root tank-a: all code compile (poudriere, world), located nearest to the center of spindle HDD + has 3GB ZIL from SSD drive. Q1: Is it ok to assume that arcstat.size will not change much regardless of the number of zpools? Q2: I have 3GB free space on the SSD reserved for an L2ARC, but decided it was not necessary after reading that this would be mostly useful for a commercial web server for example. Was my assessment incorrect and will the system benefit from a 3GB (or larger?) L2ARC on SSD? If so, which pool (not sure tank-b makes sense since it's already fully on the same SSD): Regards. - FreeBSD-11-current_amd64_root-on-zfs_RadeonKMS -- View this message in context: http://freebsd.1045724.n5.nabble.com/Memory-blackhole-in-11-Possibly-libc-so-7-tp5916161p5916406.html Sent from the freebsd-current mailing list archive at Nabble.com. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Memory blackhole in 11. Possibly libc.so.7?
On Thursday, May 29, 2014 10:42:18 am Beeblebrox wrote: > >> Why do you think libc.so.7 has anything to do with this? > > Well, because there are two instances of it running in the lsof dump, with > several possible "child process" candidates. Why would they be hanging > around when practically everything has been killed? Everything that is dynamically pulls in libc.so.7. This includes /bin/sh, etc. However, those mappings are not _wired_ memory. Those would just be active or inactive pages. > Radeon*.ko modules are a very strong possibility, as I knew before I posted. > Unfortunately I cannot kldunload those modules once laded because then tty's > go dark and I must reboot. I send the Radeon developer some of my crash > reports once and a while, so nothing to add there. Have not heard from him > in a while, and I d't want to disturb him unnecessarily. > > And yes, as Alexander points out, zfs is also a possibility. ZFS is almost certainly the cause of the wired memory usage. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Memory blackhole in 11. Possibly libc.so.7?
On 2014-05-29 12:04, John Baldwin wrote: > On Thursday, May 29, 2014 9:57:22 am Alexander Kabaev wrote: >> On Thu, 29 May 2014 09:08:10 -0400 >> John Baldwin wrote: >> >>> On Thursday, May 29, 2014 5:41:03 am Beeblebrox wrote: uname: FreeBSD 11.0-CURRENT #0 r266393M: Sun May 18 13:04:00 2014 amd64 I'm also loading the Radeon_kms modules >> I don't know if the lsof dump in single user mode will be of any help, but it seems like lib/libc.so.7 has something to do with it: >>> >>> >>> Why do you think libc.so.7 has anything to do with this? wired >>> memory is usually only allocated in the kernel, so if anything you >>> are looking at some sort of memory leak in a driver (maybe the radeon >>> driver?) >>> >>> -- >>> John Baldwin >> >> Aren't ZFS buffers accounted as kernel wired memory? > > Ah, yes, using ZFS this might be "normal". > sysctl kstat.zfs.misc.arcstats.size will tell you how much memory the ARC is using, it will likely be 'most' of the wired memory. 'top' gives a decent display as well: Mem: 8896K Active, 205M Inact, 5764M Wired, 1926M Free ARC: 2939M Total, 39M MFU, 2182M MRU, 1936K Anon, 136M Header, 581M Other -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: Memory blackhole in 11. Possibly libc.so.7?
On Thursday, May 29, 2014 9:57:22 am Alexander Kabaev wrote: > On Thu, 29 May 2014 09:08:10 -0400 > John Baldwin wrote: > > > On Thursday, May 29, 2014 5:41:03 am Beeblebrox wrote: > > > uname: FreeBSD 11.0-CURRENT #0 r266393M: Sun May 18 13:04:00 2014 > > > amd64 I'm also loading the Radeon_kms modules > > > > > > > > > > I don't know if the lsof dump in single user mode will be of any > > > help, but it seems like lib/libc.so.7 has something to do with it: > > > > > > Why do you think libc.so.7 has anything to do with this? wired > > memory is usually only allocated in the kernel, so if anything you > > are looking at some sort of memory leak in a driver (maybe the radeon > > driver?) > > > > -- > > John Baldwin > > Aren't ZFS buffers accounted as kernel wired memory? Ah, yes, using ZFS this might be "normal". -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Memory blackhole in 11. Possibly libc.so.7?
>> Why do you think libc.so.7 has anything to do with this? Well, because there are two instances of it running in the lsof dump, with several possible "child process" candidates. Why would they be hanging around when practically everything has been killed? Radeon*.ko modules are a very strong possibility, as I knew before I posted. Unfortunately I cannot kldunload those modules once laded because then tty's go dark and I must reboot. I send the Radeon developer some of my crash reports once and a while, so nothing to add there. Have not heard from him in a while, and I d't want to disturb him unnecessarily. And yes, as Alexander points out, zfs is also a possibility. - FreeBSD-11-current_amd64_root-on-zfs_RadeonKMS -- View this message in context: http://freebsd.1045724.n5.nabble.com/Memory-blackhole-in-11-Possibly-libc-so-7-tp5916161p5916224.html Sent from the freebsd-current mailing list archive at Nabble.com. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Memory blackhole in 11. Possibly libc.so.7?
On 2014-05-29 09:57, Alexander Kabaev wrote: > On Thu, 29 May 2014 09:08:10 -0400 > John Baldwin wrote: > >> On Thursday, May 29, 2014 5:41:03 am Beeblebrox wrote: >>> uname: FreeBSD 11.0-CURRENT #0 r266393M: Sun May 18 13:04:00 2014 >>> amd64 I'm also loading the Radeon_kms modules >>> > >>> >>> I don't know if the lsof dump in single user mode will be of any >>> help, but it seems like lib/libc.so.7 has something to do with it: >> >> >> Why do you think libc.so.7 has anything to do with this? wired >> memory is usually only allocated in the kernel, so if anything you >> are looking at some sort of memory leak in a driver (maybe the radeon >> driver?) >> >> -- >> John Baldwin > > Aren't ZFS buffers accounted as kernel wired memory? > > -- > Alexander Kabaev > Yes, the ZFS ARC cache is counted as wired memory. You might want to consider setting a lower vfs.zfs.arc_max in /boot/loader.conf The default is all memory less 1GB, or 95% of memory, whichever is more, and this is likely too high for your usage case. It is infact too high for every use case except a server that does nothing but act as a file server. -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: Memory blackhole in 11. Possibly libc.so.7?
On Thu, 29 May 2014 09:08:10 -0400 John Baldwin wrote: > On Thursday, May 29, 2014 5:41:03 am Beeblebrox wrote: > > uname: FreeBSD 11.0-CURRENT #0 r266393M: Sun May 18 13:04:00 2014 > > amd64 I'm also loading the Radeon_kms modules > > > > > > I don't know if the lsof dump in single user mode will be of any > > help, but it seems like lib/libc.so.7 has something to do with it: > > > Why do you think libc.so.7 has anything to do with this? wired > memory is usually only allocated in the kernel, so if anything you > are looking at some sort of memory leak in a driver (maybe the radeon > driver?) > > -- > John Baldwin Aren't ZFS buffers accounted as kernel wired memory? -- Alexander Kabaev signature.asc Description: PGP signature
Re: Memory blackhole in 11. Possibly libc.so.7?
On Thursday, May 29, 2014 5:41:03 am Beeblebrox wrote: > uname: FreeBSD 11.0-CURRENT #0 r266393M: Sun May 18 13:04:00 2014 amd64 > I'm also loading the Radeon_kms modules > > Upon system startup, memory profile is clean. I get locked memory (mem_wire) > usage as: > 9%before Radeon*.ko modules loaded > 12% when slim is started (loads Radeon*.ko modules) > 14% after I login through slim > These are quite normal. However, after some usage (I don't know what > exactly) mem_free drops significantly and causes display problems (Radeon > has issues when memory falls very low, if I remember correctly). > > Upon observing low level of mem_free, I went to single user (shutdown now), > also kldunload any modules not compiled into kernel (using custom built > kernel). Final kldstat: > Id Refs AddressSize Name > 1 99 0x8020 941c18 kernel > 21 0x80b42000 b0f0 linprocfs.ko > 32 0x80b4e000 45ba0linux.ko > 41 0x80b94000 2a38 linsysfs.ko > 51 0x80b97000 243fc8 zfs.ko > 61 0x80ddb000 2940 acl_nfs4.ko > 72 0x80dde000 5d10 opensolaris.ko > 81 0x80de4000 40a50snd_hda.ko > 92 0x80e25000 756a0sound.ko > 101 0x80e9b000 7550 umass.ko > 115 0x80ea3000 3f7b0usb.ko > 131 0x80f32000 19758ext2fs.ko > 141 0x80f4c000 d050 ehci.ko > 151 0x80f5a000 a3f0 ohci.ko > 161 0x81011000 1adb ums.ko > 171 0x81013000 951 pflog.ko > 181 0x81014000 27641pf.ko > 191 0x8103c000 108ffc radeonkms.ko > 201 0x81145000 3fd39drm2.ko > 212 0x81185000 ae68 agp.ko > 224 0x8119 1776 iicbus.ko > 231 0x81192000 d1a iic.ko > 241 0x81193000 167d iicbb.ko > 254 0x81195000 1bde firmware.ko > 261 0x81197000 ac6 radeonkmsfw_RS780_pfp.ko > 271 0x81198000 55c6 radeonkmsfw_RS780_me.ko > 281 0x8119e000 dc6 radeonkmsfw_R600_rlc.ko > 311 0x811d6000 9118 netgraph.ko > 321 0x811e 160a ng_ether.ko > > But look at memory (70%), even though there are absolutely no processes > running. SYSTEM MEMORY INFORMATION: > mem_wire:2735349760 ( 2608MB) [ 70%] Wired: disabled for paging > out > mem_active: + 8638464 ( 8MB) [ 0%] Active: recently referenced > mem_inactive:+137596928 (131MB) [ 3%] Inactive: recently not > referenced > mem_cache: + 34885632 ( 33MB) [ 0%] Cached: almost avail. for > allocation > mem_free:+970948608 (925MB) [ 24%] Free: fully available for > allocation > mem_gap_vm: + 413696 ( 0MB) [ 0%] Memory gap: UNKNOWN > -- --- -- > mem_all: = 3887833088 ( 3707MB) [100%] Total real memory managed > mem_gap_sys: +123482112 (117MB)Memory gap: Kernel?! > -- --- > mem_phys:= 4011315200 ( 3825MB)Total real memory available > mem_gap_hw: +283652096 (270MB)Memory gap: Segment > Mappings?! > -- --- > mem_hw: = 4294967296 ( 4096MB)Total real memory installed > > SYSTEM MEMORY SUMMARY: > mem_used:3151536128 ( 3005MB) [ 73%] Logically used memory > mem_avail: + 1143431168 ( 1090MB) [ 26%] Logically available memory > -- --- -- > mem_total: = 4294967296 ( 4096MB) [100%] Logically total memory > > I don't know if the lsof dump in single user mode will be of any help, but > it seems like lib/libc.so.7 has something to do with it: Why do you think libc.so.7 has anything to do with this? wired memory is usually only allocated in the kernel, so if anything you are looking at some sort of memory leak in a driver (maybe the radeon driver?) -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
