Re: geli(4) memory leak
On Mon, 4 Apr 2011 01:51:24 +0200 Victor Balada Diaz wrote: VBD On Sun, Apr 03, 2011 at 08:43:45PM +0300, Mikolaj Golub wrote: On Sat, 2 Apr 2011 12:17:50 +0200 Pawel Jakub Dawidek wrote: PJD On Sat, Apr 02, 2011 at 12:04:09AM +0300, Mikolaj Golub wrote: For me your patch look correct. But the same issue is for read :-). Also, to avoid the leak I think we can just do g_destroy_bio() before all sectors check. See the attached patch (had some testing). PJD The patch looks good. Please commit. Commited, thanks. VBD I've been out all the weekend, so i've been unable to answer before. I'm glad VBD it got commited and it's great you discovered and fixed the same problem on the VBD read path. VBD Are there any plans to MFC this? Approximately after one week. -- Mikolaj Golub ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
drives 2TB on mpt device
Hi all, I have a freshly installed 8.2-REL with a SuperMicro AOC-USASLP-L8i controller (LSI/MPT 1068E chipset). I have several of these controllers working nicely in other systems. However, this time I tried drives 2TB for the first time (Hitachi Deskstar 3TB). It appears that the mpt device reports only 2TB in this case. I have already flashed the controller's firmware to the latest available version (from 2009), but that did not change anything. The drive is working fine on the standard SATA connectors on the mainboard (Supermicro H8DME-2) and reports 2.8TB there. Are there any hints how to access the full drive? Am I seeing a limitation of the controller/firmware or rather of the driver (mpt)? cu Gerrit ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: drives 2TB on mpt device
On Mon, 4 Apr 2011 15:07:01 +0200 Gerrit Kühn ger...@pmp.uni-hannover.de wrote: Are there any hints how to access the full drive? Am I seeing a limitation of the controller/firmware or rather of the driver (mpt)? It looks like a known issue: http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/147572 -- Bruce Cran ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: drives 2TB on mpt device
On Mon, 4 Apr 2011 14:36:25 +0100 Bruce Cran br...@cran.org.uk wrote about Re: drives 2TB on mpt device: Hi Bruce, BC It looks like a known issue: BC http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/147572 Hm, I don't know if this is exactly what I'm seeing here (although the cause may be the same): I do not use mptutil. The controller is dumb (without actual raid processor), and I intend to use it with zfs. However, I cannot even get gpart to create a partition larger than 2TB, because mpt comes back with only 2TB after probing the drive. As this is a problem that already exists with 1 drive, I cannot use gstripe or zfs to get around this. But the PR above states that this limitation is already built into mpt, so my only chance is probably to try a different controller/driver (any suggestions for a cheap 8port controller to use with zfs?), or to wait until mpt is updated to support larger drives. Does anyone know if there is already ongoing effort to do this? cu Gerrit ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: drives 2TB on mpt device
2011/4/4 Gerrit Kühn ger...@pmp.uni-hannover.de: On Mon, 4 Apr 2011 14:36:25 +0100 Bruce Cran br...@cran.org.uk wrote about Re: drives 2TB on mpt device: Hi Bruce, BC It looks like a known issue: BC http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/147572 Hm, I don't know if this is exactly what I'm seeing here (although the cause may be the same): I do not use mptutil. The controller is dumb (without actual raid processor), and I intend to use it with zfs. However, I cannot even get gpart to create a partition larger than 2TB, because mpt comes back with only 2TB after probing the drive. As this is a problem that already exists with 1 drive, I cannot use gstripe or zfs to get around this. But the PR above states that this limitation is already built into mpt, so my only chance is probably to try a different controller/driver (any suggestions for a cheap 8port controller to use with zfs?), or to wait until mpt is updated to support larger drives. Does anyone know if there is already ongoing effort to do this? You're probably out of luck as far as 2Tb+ support for 1068-based HBAs: http://kb.lsi.com/KnowledgebaseArticle16399.aspx Newer controllers based on LSI2008 (mps driver?) should not have that limit. --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
8.2: ISCSI: ISTGT a bit slow, I think
hi, I testing the maximum throughput from ISCSI, but I've reached only ~50MB/s (dd if=/dev/zero of=/dev/da13 bs=1M count=2048) with crossover 1Gb/s cabel and raw disk. Both machines are FreeBSD 8.2-stable with istgt and the Onboard ISCSI initiator With ZFS as target we loose round about 8-10MB/s. istgt.conf == [global] Timeout 30 NopInInterval20 DiscoveryAuthMethod Auto MaxSessions 32 MaxConnections 8 #FirstBurstLength 65536 MaxBurstLength 1048576 MaxRecvDataSegmentLength 262144 # maximum number of sending R2T in each connection # actual number is limited to QueueDepth and MaxCmdSN and ExpCmdSN # 0=disabled, 1-256=improves large writing MaxR2T 32 # iSCSI initial parameters negotiate with initiators # NOTE: incorrect values might crash MaxOutstandingR2T 16 DefaultTime2Wait 2 DefaultTime2Retain 60 MaxBurstLength 1048576 [] [LogicalUnit4] Comment 40GB Disk (iqn.san.foo:40gb) TargetName 40gb TargetAlias Data 40GB Mapping PortalGroup1 InitiatorGroup1 #AuthMethod Auto #AuthGroupAuthGroup2 UnitType Disk UnitInquiry FreeBSD iSCSI Disk 01234 1004 QueueDepth 32 LUN0Storage /failover/bigPool/disk40gb 40960MB [LogicalUnit5] Comment 2TB Disk (iqn.san.foo:2tb) TargetName 2tb TargetAlias Data 2TB Mapping PortalGroup1 InitiatorGroup1 #AuthMethod Auto #AuthGroupAuthGroup2 UnitType Disk UnitInquiry FreeBSD iSCSI Disk 01235 1005 QueueDepth 32 LUN0Storage /dev/da12 200480MB = The raw disks, itself reaches over 150-200MB/s with or without ZFS (raidz2) We have 4GB Ram and 4 x 3Ghz Xeon CPUs on board. I thought, we should reach over 80-100MB/s, so, ISTGT or the Initiator is a bis slow, I think. I've tested just in the moment with Ubuntu 10.10 Initiator and I've got round about 70MB/s - or without ZFS - constant 80MB/s, over a regular switched network. Is this the end what we could reach? 'Cause of TCP and ISCSI overhead? What we can't: enable Jumbo frames. Our switches (Cisco catalyst WS-X4515) doesn't support jumbo frames. I've tested Jumbo Frames (9k) over the crossover, but the performance was worse. Round about 20MB/s So, does anyone has some hints for me? :-) cu denny signature.asc Description: This is a digitally signed message part
Re: 8.2: ISCSI: ISTGT a bit slow, I think
I testing the maximum throughput from ISCSI, but I've reached only ~50MB/s (dd if=/dev/zero of=/dev/da13 bs=1M count=2048) with crossover 1Gb/s cabel and raw disk. Both machines are FreeBSD 8.2-stable with istgt and the Onboard ISCSI initiator I've reached almost 118 MB/s but I don't have access to the configuration atm. This was from a windows 7 client. From vmware I've gotten 107 MB/s during a debian 6 server installation. I'll post the settings when I get back to work. You could verify that there are no mismatch between the nics. Have you tried a plain scp of a large file and some rsync of ie. the ports-tree (with distfiles)? -- regards Claus When lenity and cruelty play for a kingdom, the gentler gamester is the soonest winner. Shakespeare twitter.com/kometen ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Any success stories for HAST + ZFS?
On Sat, Apr 2, 2011 at 1:44 AM, Pawel Jakub Dawidek p...@freebsd.org wrote: On Thu, Mar 24, 2011 at 01:36:32PM -0700, Freddie Cash wrote: [Not sure which list is most appropriate since it's using HAST + ZFS on -RELEASE, -STABLE, and -CURRENT. Feel free to trim the CC: on replies.] I'm having a hell of a time making this work on real hardware, and am not ruling out hardware issues as yet, but wanted to get some reassurance that someone out there is using this combination (FreeBSD + HAST + ZFS) successfully, without kernel panics, without core dumps, without deadlocks, without issues, etc. I need to know I'm not chasing a dead rabbit. I just committed a fix for a problem that might look like a deadlock. With trociny@ patch and my last fix (to GEOM GATE and hastd) do you still have any issues? Just to confirm, this is commit r220264, 220265, 220266 to -CURRENT? Looking through the commit logs, I don't see any of these MFC'd to -STABLE yet, so I can't test them directly. The storage box that was having the issues is running 8-STABLE r219754 at the moment (with ZFSv28 and Mikolag's ggate patches). I see there have been a lot of hast/ggate-related MFCs in the past week, but they don't include the deadlock patches. Once the deadlock patches above are MFC'd to -STABLE, I can do an upgrade cycle and test them. I do have the previous 9-CURRENT install saved, just nothing to run it on atm. -- Freddie Cash fjwc...@gmail.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: 8.2: ISCSI: ISTGT a bit slow, I think
hi, Am 04.04.2011 um 18:04 schrieb Claus Guttesen: I've reached almost 118 MB/s but I don't have access to the configuration atm. This was from a windows 7 client. From vmware I've gotten 107 MB/s during a debian 6 server installation. I'll post the settings when I get back to work. that would be nice. I will test also a Windows7 client, maybe the initiator aren't the best, from Ubuntu and FreeBSD. You could verify that there are no mismatch between the nics. both are the same hardware, so two Intel e1000 ... Have you tried a plain scp of a large file and some rsync of ie. the ports-tree (with distfiles)? nope, nothing should be faster than dd :-) So there is no more protocol overhead. cu denny ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Kernel memory leak in 8.2-PRERELEASE?
On 04/02/11 11:41, Boris Kochergin wrote: On 04/02/11 11:33, Kostik Belousov wrote: On Sat, Apr 02, 2011 at 10:17:27AM -0400, Boris Kochergin wrote: Ahoy. This morning, I awoke to the following on one of my servers: pid 59630 (httpd), uid 80, was killed: out of swap space pid 59341 (find), uid 0, was killed: out of swap space pid 23134 (irssi), uid 1001, was killed: out of swap space pid 49332 (sshd), uid 1001, was killed: out of swap space pid 69074 (httpd), uid 0, was killed: out of swap space pid 11879 (eggdrop-1.6.19), uid 1001, was killed: out of swap space ... And so on. The machine is: FreeBSD exodus.poly.edu 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #2: Thu Dec 2 11:39:21 EST 2010 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 10:13AM up 120 days, 20:06, 2 users, load averages: 0.00, 0.01, 0.00 The memory line from top intrigued me: Mem: 16M Active, 48M Inact, 6996M Wired, 229M Cache, 828M Buf, 605M Free The machine has 8 gigs of memory, and I don't know what all that wired memory is being used for. There is a large-ish (6 x 1.5-TB) ZFS RAID-Z2 on it which has had a disk in the UNAVAIL state for a few months: # zpool status pool: home state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: none requested config: NAMESTATE READ WRITE CKSUM homeDEGRADED 0 0 0 raidz2DEGRADED 0 0 0 ada0ONLINE 0 0 0 ada1ONLINE 0 0 0 ada2ONLINE 0 0 0 ada3ONLINE 0 0 0 ada4ONLINE 0 0 0 ada5UNAVAIL 08511 experienced I/O failures errors: No known data errors vmstat -m and vmstat -z output: http://acm.poly.edu/~spawk/vmstat-m.txt http://acm.poly.edu/~spawk/vmstat-z.txt Anyone have a clue? I know it's just going to happen again if I reboot the machine. It is still up in case there are diagnostics for me to run. Try r218795. Most likely, your issue is not leak. Thanks. Will update to today's 8-STABLE and report back. -Boris The problem persists, I'm afraid, and seems to have crept up a lot more quickly than before: # uname -a FreeBSD exodus.poly.edu 8.2-STABLE FreeBSD 8.2-STABLE #3: Sat Apr 2 11:48:43 EDT 2011 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 Mem: 314M Active, 955M Inact, 6356M Wired, 267M Cache, 828M Buf, 18M Free Any ideas for a diagnostic recourse? -Boris ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Kernel memory leak in 8.2-PRERELEASE?
On Mon, Apr 4, 2011 at 1:56 PM, Boris Kochergin sp...@acm.poly.edu wrote: The problem persists, I'm afraid, and seems to have crept up a lot more quickly than before: # uname -a FreeBSD exodus.poly.edu 8.2-STABLE FreeBSD 8.2-STABLE #3: Sat Apr 2 11:48:43 EDT 2011 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 Mem: 314M Active, 955M Inact, 6356M Wired, 267M Cache, 828M Buf, 18M Free Any ideas for a diagnostic recourse? My bet would be that the wired memory is used by ZFS ARC. In your vmstat-m output you can see that ~2.2G were allocated for 'solaris' subsystem. Due to the fact that ARC allocations tend to be random, we do waste a lot of memory on that. There were few patches floating on stable@ adn fs@ that were supposed to mitigate the issue, but so far there's no good out of the box solution. General advice is to tune ARC so that it works in your case. Key loader tunables are vfs.zfs.arc_min and vfs.zfs.arc_max. Don't set min too high and experimentally set max to the maximum value that does not cause problems. One of the factors that makes things noticeably worse is presence of i/o actifity on non-ZFS filesystems. Regular filesystems cache competes with ZFS for RAM. In the past ZFS used to give up memory way too easily. Currently it's a bit more balanced, but is still far from perfect. By the way, if you don't have on-disk swap configured, I'd recommend adding some. It may help avoiding processes being killed during intermittent memory shortages. If would also help if you could post your /boot/loader.conf and output of zfs-stats -a (available in ports). --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Kernel memory leak in 8.2-PRERELEASE?
On Mon, Apr 04, 2011 at 04:56:31PM -0400, Boris Kochergin wrote: On 04/02/11 11:41, Boris Kochergin wrote: On 04/02/11 11:33, Kostik Belousov wrote: On Sat, Apr 02, 2011 at 10:17:27AM -0400, Boris Kochergin wrote: Ahoy. This morning, I awoke to the following on one of my servers: pid 59630 (httpd), uid 80, was killed: out of swap space pid 59341 (find), uid 0, was killed: out of swap space pid 23134 (irssi), uid 1001, was killed: out of swap space pid 49332 (sshd), uid 1001, was killed: out of swap space pid 69074 (httpd), uid 0, was killed: out of swap space pid 11879 (eggdrop-1.6.19), uid 1001, was killed: out of swap space ... And so on. The machine is: FreeBSD exodus.poly.edu 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #2: Thu Dec 2 11:39:21 EST 2010 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 10:13AM up 120 days, 20:06, 2 users, load averages: 0.00, 0.01, 0.00 The memory line from top intrigued me: Mem: 16M Active, 48M Inact, 6996M Wired, 229M Cache, 828M Buf, 605M Free The machine has 8 gigs of memory, and I don't know what all that wired memory is being used for. There is a large-ish (6 x 1.5-TB) ZFS RAID-Z2 on it which has had a disk in the UNAVAIL state for a few months: # zpool status pool: home state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: none requested config: NAMESTATE READ WRITE CKSUM homeDEGRADED 0 0 0 raidz2DEGRADED 0 0 0 ada0ONLINE 0 0 0 ada1ONLINE 0 0 0 ada2ONLINE 0 0 0 ada3ONLINE 0 0 0 ada4ONLINE 0 0 0 ada5UNAVAIL 08511 experienced I/O failures errors: No known data errors vmstat -m and vmstat -z output: http://acm.poly.edu/~spawk/vmstat-m.txt http://acm.poly.edu/~spawk/vmstat-z.txt Anyone have a clue? I know it's just going to happen again if I reboot the machine. It is still up in case there are diagnostics for me to run. Try r218795. Most likely, your issue is not leak. Thanks. Will update to today's 8-STABLE and report back. -Boris The problem persists, I'm afraid, and seems to have crept up a lot more quickly than before: # uname -a FreeBSD exodus.poly.edu 8.2-STABLE FreeBSD 8.2-STABLE #3: Sat Apr 2 11:48:43 EDT 2011 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 Mem: 314M Active, 955M Inact, 6356M Wired, 267M Cache, 828M Buf, 18M Free Any ideas for a diagnostic recourse? Can you please provide the details I requested here? Thanks. http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/062147.html -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Kernel memory leak in 8.2-PRERELEASE?
On 04/04/11 18:43, Jeremy Chadwick wrote: On Mon, Apr 04, 2011 at 04:56:31PM -0400, Boris Kochergin wrote: On 04/02/11 11:41, Boris Kochergin wrote: On 04/02/11 11:33, Kostik Belousov wrote: On Sat, Apr 02, 2011 at 10:17:27AM -0400, Boris Kochergin wrote: Ahoy. This morning, I awoke to the following on one of my servers: pid 59630 (httpd), uid 80, was killed: out of swap space pid 59341 (find), uid 0, was killed: out of swap space pid 23134 (irssi), uid 1001, was killed: out of swap space pid 49332 (sshd), uid 1001, was killed: out of swap space pid 69074 (httpd), uid 0, was killed: out of swap space pid 11879 (eggdrop-1.6.19), uid 1001, was killed: out of swap space ... And so on. The machine is: FreeBSD exodus.poly.edu 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #2: Thu Dec 2 11:39:21 EST 2010 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 10:13AM up 120 days, 20:06, 2 users, load averages: 0.00, 0.01, 0.00 The memory line from top intrigued me: Mem: 16M Active, 48M Inact, 6996M Wired, 229M Cache, 828M Buf, 605M Free The machine has 8 gigs of memory, and I don't know what all that wired memory is being used for. There is a large-ish (6 x 1.5-TB) ZFS RAID-Z2 on it which has had a disk in the UNAVAIL state for a few months: # zpool status pool: home state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: none requested config: NAMESTATE READ WRITE CKSUM homeDEGRADED 0 0 0 raidz2DEGRADED 0 0 0 ada0ONLINE 0 0 0 ada1ONLINE 0 0 0 ada2ONLINE 0 0 0 ada3ONLINE 0 0 0 ada4ONLINE 0 0 0 ada5UNAVAIL 08511 experienced I/O failures errors: No known data errors vmstat -m and vmstat -z output: http://acm.poly.edu/~spawk/vmstat-m.txt http://acm.poly.edu/~spawk/vmstat-z.txt Anyone have a clue? I know it's just going to happen again if I reboot the machine. It is still up in case there are diagnostics for me to run. Try r218795. Most likely, your issue is not leak. Thanks. Will update to today's 8-STABLE and report back. -Boris The problem persists, I'm afraid, and seems to have crept up a lot more quickly than before: # uname -a FreeBSD exodus.poly.edu 8.2-STABLE FreeBSD 8.2-STABLE #3: Sat Apr 2 11:48:43 EDT 2011 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 Mem: 314M Active, 955M Inact, 6356M Wired, 267M Cache, 828M Buf, 18M Free Any ideas for a diagnostic recourse? Can you please provide the details I requested here? Thanks. http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/062147.html No swap, blank /boot/loader.conf, default /etc/sysctl.conf. I'm going to try this ARC tuning thing. I vaguely recall several claims that tuning wasn't necessary anymore on amd64 systems with the amount of memory mine has, but that's obviously not the case. -Boris ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Kernel memory leak in 8.2-PRERELEASE?
On Mon, Apr 04, 2011 at 08:56:10PM -0400, Boris Kochergin wrote: On 04/04/11 18:43, Jeremy Chadwick wrote: On Mon, Apr 04, 2011 at 04:56:31PM -0400, Boris Kochergin wrote: On 04/02/11 11:41, Boris Kochergin wrote: On 04/02/11 11:33, Kostik Belousov wrote: On Sat, Apr 02, 2011 at 10:17:27AM -0400, Boris Kochergin wrote: Ahoy. This morning, I awoke to the following on one of my servers: pid 59630 (httpd), uid 80, was killed: out of swap space pid 59341 (find), uid 0, was killed: out of swap space pid 23134 (irssi), uid 1001, was killed: out of swap space pid 49332 (sshd), uid 1001, was killed: out of swap space pid 69074 (httpd), uid 0, was killed: out of swap space pid 11879 (eggdrop-1.6.19), uid 1001, was killed: out of swap space ... And so on. The machine is: FreeBSD exodus.poly.edu 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #2: Thu Dec 2 11:39:21 EST 2010 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 10:13AM up 120 days, 20:06, 2 users, load averages: 0.00, 0.01, 0.00 The memory line from top intrigued me: Mem: 16M Active, 48M Inact, 6996M Wired, 229M Cache, 828M Buf, 605M Free The machine has 8 gigs of memory, and I don't know what all that wired memory is being used for. There is a large-ish (6 x 1.5-TB) ZFS RAID-Z2 on it which has had a disk in the UNAVAIL state for a few months: # zpool status pool: home state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: none requested config: NAMESTATE READ WRITE CKSUM homeDEGRADED 0 0 0 raidz2DEGRADED 0 0 0 ada0ONLINE 0 0 0 ada1ONLINE 0 0 0 ada2ONLINE 0 0 0 ada3ONLINE 0 0 0 ada4ONLINE 0 0 0 ada5UNAVAIL 08511 experienced I/O failures errors: No known data errors vmstat -m and vmstat -z output: http://acm.poly.edu/~spawk/vmstat-m.txt http://acm.poly.edu/~spawk/vmstat-z.txt Anyone have a clue? I know it's just going to happen again if I reboot the machine. It is still up in case there are diagnostics for me to run. Try r218795. Most likely, your issue is not leak. Thanks. Will update to today's 8-STABLE and report back. -Boris The problem persists, I'm afraid, and seems to have crept up a lot more quickly than before: # uname -a FreeBSD exodus.poly.edu 8.2-STABLE FreeBSD 8.2-STABLE #3: Sat Apr 2 11:48:43 EDT 2011 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 Mem: 314M Active, 955M Inact, 6356M Wired, 267M Cache, 828M Buf, 18M Free Any ideas for a diagnostic recourse? Can you please provide the details I requested here? Thanks. http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/062147.html No swap, blank /boot/loader.conf, default /etc/sysctl.conf. I'm going to try this ARC tuning thing. I vaguely recall several claims that tuning wasn't necessary anymore on amd64 systems with the amount of memory mine has, but that's obviously not the case. Given that you don't have swap (again: very, very bad idea), your applications crashing due to there not being any swap space is expected: no place to swap them out to. All you should need to set, in /boot/loader.conf, is: vfs.zfs.arc_max For example, if you want to limit the ARC to only use up to 2GB of RAM: vfs.zfs.arc_max=2048M This would reserve (on an 8GB machine) approximately ~6GB of RAM for userland applications, the kernel, network buffers/mbufs, etc.. Finally, please note that most of the stuff you'll read online for ZFS tuning on FreeBSD is outdated with 8.2. E.g. you should not need to set vm.kmem_size and you should never need to adjust vm.kmem_size_max. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Kernel memory leak in 8.2-PRERELEASE?
On 04/04/11 21:01, Jeremy Chadwick wrote: On Mon, Apr 04, 2011 at 08:56:10PM -0400, Boris Kochergin wrote: On 04/04/11 18:43, Jeremy Chadwick wrote: On Mon, Apr 04, 2011 at 04:56:31PM -0400, Boris Kochergin wrote: On 04/02/11 11:41, Boris Kochergin wrote: On 04/02/11 11:33, Kostik Belousov wrote: On Sat, Apr 02, 2011 at 10:17:27AM -0400, Boris Kochergin wrote: Ahoy. This morning, I awoke to the following on one of my servers: pid 59630 (httpd), uid 80, was killed: out of swap space pid 59341 (find), uid 0, was killed: out of swap space pid 23134 (irssi), uid 1001, was killed: out of swap space pid 49332 (sshd), uid 1001, was killed: out of swap space pid 69074 (httpd), uid 0, was killed: out of swap space pid 11879 (eggdrop-1.6.19), uid 1001, was killed: out of swap space ... And so on. The machine is: FreeBSD exodus.poly.edu 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #2: Thu Dec 2 11:39:21 EST 2010 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 10:13AM up 120 days, 20:06, 2 users, load averages: 0.00, 0.01, 0.00 The memory line from top intrigued me: Mem: 16M Active, 48M Inact, 6996M Wired, 229M Cache, 828M Buf, 605M Free The machine has 8 gigs of memory, and I don't know what all that wired memory is being used for. There is a large-ish (6 x 1.5-TB) ZFS RAID-Z2 on it which has had a disk in the UNAVAIL state for a few months: # zpool status pool: home state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: none requested config: NAMESTATE READ WRITE CKSUM homeDEGRADED 0 0 0 raidz2DEGRADED 0 0 0 ada0ONLINE 0 0 0 ada1ONLINE 0 0 0 ada2ONLINE 0 0 0 ada3ONLINE 0 0 0 ada4ONLINE 0 0 0 ada5UNAVAIL 08511 experienced I/O failures errors: No known data errors vmstat -m and vmstat -z output: http://acm.poly.edu/~spawk/vmstat-m.txt http://acm.poly.edu/~spawk/vmstat-z.txt Anyone have a clue? I know it's just going to happen again if I reboot the machine. It is still up in case there are diagnostics for me to run. Try r218795. Most likely, your issue is not leak. Thanks. Will update to today's 8-STABLE and report back. -Boris The problem persists, I'm afraid, and seems to have crept up a lot more quickly than before: # uname -a FreeBSD exodus.poly.edu 8.2-STABLE FreeBSD 8.2-STABLE #3: Sat Apr 2 11:48:43 EDT 2011 sp...@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 Mem: 314M Active, 955M Inact, 6356M Wired, 267M Cache, 828M Buf, 18M Free Any ideas for a diagnostic recourse? Can you please provide the details I requested here? Thanks. http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/062147.html No swap, blank /boot/loader.conf, default /etc/sysctl.conf. I'm going to try this ARC tuning thing. I vaguely recall several claims that tuning wasn't necessary anymore on amd64 systems with the amount of memory mine has, but that's obviously not the case. Given that you don't have swap (again: very, very bad idea), your applications crashing due to there not being any swap space is expected: no place to swap them out to. All you should need to set, in /boot/loader.conf, is: vfs.zfs.arc_max For example, if you want to limit the ARC to only use up to 2GB of RAM: vfs.zfs.arc_max=2048M Thanks. I will attempt just this and report back. -Boris This would reserve (on an 8GB machine) approximately ~6GB of RAM for userland applications, the kernel, network buffers/mbufs, etc.. Finally, please note that most of the stuff you'll read online for ZFS tuning on FreeBSD is outdated with 8.2. E.g. you should not need to set vm.kmem_size and you should never need to adjust vm.kmem_size_max. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Kernel memory leak in 8.2-PRERELEASE?
On Mon, 4 Apr 2011, Jeremy Chadwick wrote: Finally, please note that most of the stuff you'll read online for ZFS tuning on FreeBSD is outdated with 8.2. E.g. you should not need to set vm.kmem_size and you should never need to adjust vm.kmem_size_max. Slight tangent, does this apply to i386 as well or just amd64? Someone should open the zfs wiki page to a broader range of editors I think - it seems like this mailing list is the only decent reference for tuning these days. Charles -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Kernel memory leak in 8.2-PRERELEASE?
On Mon, Apr 04, 2011 at 09:48:17PM -0400, Charles Sprickman wrote: On Mon, 4 Apr 2011, Jeremy Chadwick wrote: Finally, please note that most of the stuff you'll read online for ZFS tuning on FreeBSD is outdated with 8.2. E.g. you should not need to set vm.kmem_size and you should never need to adjust vm.kmem_size_max. Slight tangent, does this apply to i386 as well or just amd64? Someone should open the zfs wiki page to a broader range of editors I think - it seems like this mailing list is the only decent reference for tuning these days. Off the top of my head: wouldn't know, I tend not to run i386 anywhere anymore. I'm not sure what all i386 requires, other than adjusting KVA_PAGES in one's kernel config. Sadly I'd probably refer to the Wiki oops. ;-) -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Kernel memory leak in 8.2-PRERELEASE?
So, vfs.zfs.arc_max=2048M in /boot/loader.conf was indeed apparently all that was necessary to bring the situation under control. I remember it being a lot more nightmarish, so it's nice to see that it's improved. Thanks for everyone's advice. Per an earlier request, here is the output of zfs-stats -a right now (not when the system is running out of memory, but perhaps still interesting): # zfs-stats -a ZFS Subsystem ReportMon Apr 4 22:43:43 2011 System Information: Kernel Version: 802502 (osreldate) Hardware Platform: amd64 Processor Architecture: amd64 FreeBSD 8.2-STABLE #3: Sat Apr 2 11:48:43 EDT 2011 spawk 10:43PM up 1:32, 3 users, load averages: 0.13, 0.09, 0.07 System Memory Statistics: Physical Memory:8181.32M Kernel Memory: 2134.72M DATA: 99.65% 2127.20M TEXT: 0.35% 7.52M ZFS pool information: Storage pool Version (spa): 15 Filesystem Version (zpl): 4 ARC Misc: Deleted:583540 Recycle Misses: 355 Mutex Misses: 11 Evict Skips:11 ARC Size: Current Size (arcsize): 100.00% 2048.07M Target Size (Adaptive, c): 100.00% 2048.00M Min Size (Hard Limit, c_min): 12.50% 256.00M Max Size (High Water, c_max): ~8:12048.00M ARC Size Breakdown: Recently Used Cache Size (p): 93.32% 1911.27M Frequent;y Used Cache Size (arcsize-p): 6.68% 136.80M ARC Hash Breakdown: Elements Max: 38168 Elements Current: 99.18% 37856 Collisions: 127822 Chain Max: 5 Chains: 4567 ARC Eviction Statistics: Evicts Total: 80383607808 Evicts Eligible for L2: 9.95% 8001851392 Evicts Ineligible for L2: 90.05% 72381756416 Evicts Cached to L2:0 ARC Efficiency: Cache Access Total: 1439376 Cache Hit Ratio:55.64% 800797 Cache Miss Ratio: 44.36% 638579 Actual Hit Ratio: 51.12% 735809 Data Demand Efficiency: 97.21% Data Prefetch Efficiency: 8.78% CACHE HITS BY CACHE LIST: Anonymously Used: 5.96% 47738 Most Recently Used (mru): 28.97% 232003 Most Frequently Used (mfu): 62.91% 503806 MRU Ghost (mru_ghost):0.74% 5933 MFU Ghost (mfu_ghost):1.41% 11317 CACHE HITS BY DATA TYPE: Demand Data: 31.17% 249578 Prefetch Data:7.47% 59804 Demand Metadata: 60.33% 483145 Prefetch Metadata:1.03% 8270 CACHE MISSES BY DATA TYPE: Demand Data: 1.12% 7162 Prefetch Data:97.31% 621373 Demand Metadata: 0.91% 5805 Prefetch Metadata:0.66% 4239 VDEV Cache Summary: Access Total: 32382 Hits Ratio: 21.62% 7001 Miss Ratio: 78.38% 25381 Delegations:11361 File-Level Prefetch Stats (DMU): DMU Efficiency: Access Total: 640507 Hit Ratio: 92.40% 591801 Miss Ratio: 7.60% 48706 Colinear Access Total: 48706 Colinear Hit Ratio: 0.20% 99 Colinear Miss Ratio:99.80% 48607 Stride Access Total:533456 Stride Hit Ratio: 99.97% 533296 Stride Miss Ratio: 0.03% 160 DMU misc: Reclaim successes: 8857 Reclaim failures: 39750 Stream resets: 126 Stream noresets:58504 Bogus streams:
