On 04/04/11 21:01, Jeremy Chadwick wrote:
On Mon, Apr 04, 2011 at 08:56:10PM -0400, Boris Kochergin wrote:
On 04/04/11 18:43, Jeremy Chadwick wrote:
On Mon, Apr 04, 2011 at 04:56:31PM -0400, Boris Kochergin wrote:
On 04/02/11 11:41, Boris Kochergin wrote:
On 04/02/11 11:33, Kostik Belousov wrote:
On Sat, Apr 02, 2011 at 10:17:27AM -0400, Boris Kochergin wrote:
Ahoy. This morning, I awoke to the following on one of my servers:
pid 59630 (httpd), uid 80, was killed: out of swap space
pid 59341 (find), uid 0, was killed: out of swap space
pid 23134 (irssi), uid 1001, was killed: out of swap space
pid 49332 (sshd), uid 1001, was killed: out of swap space
pid 69074 (httpd), uid 0, was killed: out of swap space
pid 11879 (eggdrop-1.6.19), uid 1001, was killed: out of swap space
...
And so on.
The machine is:
FreeBSD exodus.poly.edu 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #2: Thu
Dec 2 11:39:21 EST 2010
[email protected]:/usr/obj/usr/src/sys/EXODUS amd64
10:13AM up 120 days, 20:06, 2 users, load averages: 0.00, 0.01, 0.00
The memory line from top intrigued me:
Mem: 16M Active, 48M Inact, 6996M Wired, 229M Cache, 828M Buf,
605M Free
The machine has 8 gigs of memory, and I don't know what all that wired
memory is being used for. There is a large-ish (6 x 1.5-TB) ZFS RAID-Z2
on it which has had a disk in the UNAVAIL state for a few months:
# zpool status
pool: home
state: DEGRADED
status: One or more devices could not be used because the label is
missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
home DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
ada0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
ada4 ONLINE 0 0 0
ada5 UNAVAIL 0 85 11 experienced
I/O failures
errors: No known data errors
"vmstat -m" and "vmstat -z" output:
http://acm.poly.edu/~spawk/vmstat-m.txt
http://acm.poly.edu/~spawk/vmstat-z.txt
Anyone have a clue? I know it's just going to happen again if I reboot
the machine. It is still up in case there are diagnostics for
me to run.
Try r218795. Most likely, your issue is not leak.
Thanks. Will update to today's 8-STABLE and report back.
-Boris
The problem persists, I'm afraid, and seems to have crept up a lot
more quickly than before:
# uname -a
FreeBSD exodus.poly.edu 8.2-STABLE FreeBSD 8.2-STABLE #3: Sat Apr 2
11:48:43 EDT 2011
[email protected]:/usr/obj/usr/src/sys/EXODUS amd64
Mem: 314M Active, 955M Inact, 6356M Wired, 267M Cache, 828M Buf, 18M Free
Any ideas for a diagnostic recourse?
Can you please provide the details I requested here? Thanks.
http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/062147.html
No swap, blank /boot/loader.conf, default /etc/sysctl.conf. I'm
going to try this ARC tuning thing. I vaguely recall several claims
that tuning wasn't necessary anymore on amd64 systems with the
amount of memory mine has, but that's obviously not the case.
Given that you don't have swap (again: very, very bad idea), your
applications crashing due to there not being any swap space is expected:
no place to swap them out to.
All you should need to set, in /boot/loader.conf, is:
vfs.zfs.arc_max
For example, if you want to limit the ARC to only use up to 2GB of RAM:
vfs.zfs.arc_max="2048M"
Thanks. I will attempt just this and report back.
-Boris
This would reserve (on an 8GB machine) approximately ~6GB of RAM for
userland applications, the kernel, network buffers/mbufs, etc..
Finally, please note that most of the stuff you'll read online for ZFS
tuning on FreeBSD is outdated with 8.2. E.g. you should not need to set
vm.kmem_size and you should never need to adjust vm.kmem_size_max.
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[email protected]"
