Re: [zfs-discuss] very slow write performance on 151a
Hi Milosz, As far as I know, fix for space map thrashing is in the works. As of now, you still have to do with the above mentioned tunables. Thanks, Deepak. On 02/ 1/12 11:20 PM, milosz wrote: hi guys, does anyone know if a fix for this (space map thrashing) is in the works? i've been running into this on and off on a number of systems i manage. sometimes i can delete snapshots and things go back to normal, sometimes the only thing that works is enabling metaslab_debug. obviously the latter is only really an option for systems with a huge amount of ram. or: am i doing something wrong? milosz On Mon, Dec 19, 2011 at 8:02 AM, Jim Klimov wrote: 2011-12-15 22:44, milosz цкщеу: There are a few metaslab-related tunables that can be tweaked as well. - Bill For the sake of completeness, here are the relevant lines I have in /etc/system: ** * fix up metaslab min size (recent default ~10Mb seems bad, * recommended return to 4Kb, we'll do 4*8K) * greatly increases write speed in filled-up pools set zfs:metaslab_min_alloc_size = 0x8000 set zfs:metaslab_smo_bonus_pct = 0xc8 ** These values were described in greater detail on the list this summer, I think. HTH, //Jim ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] very slow write performance on 151a
hi guys, does anyone know if a fix for this (space map thrashing) is in the works? i've been running into this on and off on a number of systems i manage. sometimes i can delete snapshots and things go back to normal, sometimes the only thing that works is enabling metaslab_debug. obviously the latter is only really an option for systems with a huge amount of ram. or: am i doing something wrong? milosz On Mon, Dec 19, 2011 at 8:02 AM, Jim Klimov wrote: > 2011-12-15 22:44, milosz цкщеу: > >>> There are a few metaslab-related tunables that can be tweaked as well. >>> - Bill > > > For the sake of completeness, here are the relevant lines > I have in /etc/system: > > ** > * fix up metaslab min size (recent default ~10Mb seems bad, > * recommended return to 4Kb, we'll do 4*8K) > * greatly increases write speed in filled-up pools > set zfs:metaslab_min_alloc_size = 0x8000 > set zfs:metaslab_smo_bonus_pct = 0xc8 > ** > > These values were described in greater detail on the list > this summer, I think. > > HTH, > //Jim ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] very slow write performance on 151a
2011-12-15 22:44, milosz цкщеу: There are a few metaslab-related tunables that can be tweaked as well. - Bill For the sake of completeness, here are the relevant lines I have in /etc/system: ** * fix up metaslab min size (recent default ~10Mb seems bad, * recommended return to 4Kb, we'll do 4*8K) * greatly increases write speed in filled-up pools set zfs:metaslab_min_alloc_size = 0x8000 set zfs:metaslab_smo_bonus_pct = 0xc8 ** These values were described in greater detail on the list this summer, I think. HTH, //Jim ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] very slow write performance on 151a
thanks, bill. i killed an old filesystem. also forgot about arc_meta_limit. kicked it up to 4gb from 2gb. things are back to normal. On Thu, Dec 15, 2011 at 1:06 PM, Bill Sommerfeld wrote: > On 12/15/11 09:35, milosz wrote: >> >> hi all, >> >> suddenly ran into a very odd issue with a 151a server used primarily >> for cifs... out of (seemingly) nowhere, writes are incredibly slow, >> often<10kb/s. this is what zpool iostat 1 looks like when i copy a >> big file: >> >> storepool 13.4T 1.07T 57 0 6.13M 0 >> storepool 13.4T 1.07T 216 91 740K 5.58M > > ... > >> any ideas? pretty stumped. > > > Behavior I've observed with multiple pools is that you will sometimes hit a > performance wall when the pool gets too full; the system spends lots of time > reading in metaslab metadata looking for a place to put newly-allocated > blocks. If you're in this mode, kernel profiling will show a lot of time > spent in metaslab-related code. > > Exactly where you hit the wall seems to depend on the history of what went > into the pool; I've seen the problem kick in with only 69%-70% usage in one > pool that was used primarily for solaris development. > > The workaround turned out to be simple: delete stuff you don't need to keep. > Once there was enough free space, write performance returned to normal. > > There are a few metaslab-related tunables that can be tweaked as well. > > - Bill ___ zfs-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
