Re: [zfs-discuss] very slow write performance on 151a
Hi Milosz, As far as I know, fix for space map thrashing is in the works. As of now, you still have to do with the above mentioned tunables. Thanks, Deepak. On 02/ 1/12 11:20 PM, milosz wrote: hi guys, does anyone know if a fix for this (space map thrashing) is in the works? i've been running into this on and off on a number of systems i manage. sometimes i can delete snapshots and things go back to normal, sometimes the only thing that works is enabling metaslab_debug. obviously the latter is only really an option for systems with a huge amount of ram. or: am i doing something wrong? milosz On Mon, Dec 19, 2011 at 8:02 AM, Jim Klimov wrote: 2011-12-15 22:44, milosz цкщеу: There are a few metaslab-related tunables that can be tweaked as well. - Bill For the sake of completeness, here are the relevant lines I have in /etc/system: ** * fix up metaslab min size (recent default ~10Mb seems bad, * recommended return to 4Kb, we'll do 4*8K) * greatly increases write speed in filled-up pools set zfs:metaslab_min_alloc_size = 0x8000 set zfs:metaslab_smo_bonus_pct = 0xc8 ** These values were described in greater detail on the list this summer, I think. HTH, //Jim ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] very slow write performance on 151a
hi guys, does anyone know if a fix for this (space map thrashing) is in the works? i've been running into this on and off on a number of systems i manage. sometimes i can delete snapshots and things go back to normal, sometimes the only thing that works is enabling metaslab_debug. obviously the latter is only really an option for systems with a huge amount of ram. or: am i doing something wrong? milosz On Mon, Dec 19, 2011 at 8:02 AM, Jim Klimov wrote: > 2011-12-15 22:44, milosz цкщеу: > >>> There are a few metaslab-related tunables that can be tweaked as well. >>> - Bill > > > For the sake of completeness, here are the relevant lines > I have in /etc/system: > > ** > * fix up metaslab min size (recent default ~10Mb seems bad, > * recommended return to 4Kb, we'll do 4*8K) > * greatly increases write speed in filled-up pools > set zfs:metaslab_min_alloc_size = 0x8000 > set zfs:metaslab_smo_bonus_pct = 0xc8 > ** > > These values were described in greater detail on the list > this summer, I think. > > HTH, > //Jim ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] very slow write performance on 151a
2011-12-15 22:44, milosz цкщеу: There are a few metaslab-related tunables that can be tweaked as well. - Bill For the sake of completeness, here are the relevant lines I have in /etc/system: ** * fix up metaslab min size (recent default ~10Mb seems bad, * recommended return to 4Kb, we'll do 4*8K) * greatly increases write speed in filled-up pools set zfs:metaslab_min_alloc_size = 0x8000 set zfs:metaslab_smo_bonus_pct = 0xc8 ** These values were described in greater detail on the list this summer, I think. HTH, //Jim ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] very slow write performance on 151a
thanks, bill. i killed an old filesystem. also forgot about arc_meta_limit. kicked it up to 4gb from 2gb. things are back to normal. On Thu, Dec 15, 2011 at 1:06 PM, Bill Sommerfeld wrote: > On 12/15/11 09:35, milosz wrote: >> >> hi all, >> >> suddenly ran into a very odd issue with a 151a server used primarily >> for cifs... out of (seemingly) nowhere, writes are incredibly slow, >> often<10kb/s. this is what zpool iostat 1 looks like when i copy a >> big file: >> >> storepool 13.4T 1.07T 57 0 6.13M 0 >> storepool 13.4T 1.07T 216 91 740K 5.58M > > ... > >> any ideas? pretty stumped. > > > Behavior I've observed with multiple pools is that you will sometimes hit a > performance wall when the pool gets too full; the system spends lots of time > reading in metaslab metadata looking for a place to put newly-allocated > blocks. If you're in this mode, kernel profiling will show a lot of time > spent in metaslab-related code. > > Exactly where you hit the wall seems to depend on the history of what went > into the pool; I've seen the problem kick in with only 69%-70% usage in one > pool that was used primarily for solaris development. > > The workaround turned out to be simple: delete stuff you don't need to keep. > Once there was enough free space, write performance returned to normal. > > There are a few metaslab-related tunables that can be tweaked as well. > > - Bill ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] very slow write performance on 151a
hi all, suddenly ran into a very odd issue with a 151a server used primarily for cifs... out of (seemingly) nowhere, writes are incredibly slow, often <10kb/s. this is what zpool iostat 1 looks like when i copy a big file: storepool 13.4T 1.07T 57 0 6.13M 0 storepool 13.4T 1.07T216 91 740K 5.58M storepool 13.4T 1.07T127182 232K 1004K storepool 13.4T 1.07T189 99 361K 5.47M storepool 13.4T 1.07T357172 910K 949K storepool 13.4T 1.07T454222 1.42M 2.14M storepool 13.4T 1.07T 55209 711K 1.05M basically instead of the usual txg 5-second write pattern zfs writes to the zpool every second. this is certainly not an issue with the disks... iostat -En shows no errors and -Xn shows that the disks are barely being used (<20%). the only situation in which i've seen this before was a multi-terabyte pool with dedup=on and constant writes (goes away once you turn off dedup). no dedup anywhere on this zpool, though. arc usage is normal (total ram is 12gb, max is set to 11gb, current usage is 8gb). pool is an 8-disk raidz2. any ideas? pretty stumped. milosz ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
