On May 4, 2011, at 4:16 PM, Victor Latushkin wrote: > Try > > echo metaslab_debug/W1 | mdb -kw > > If it does not help, reset it back to zero > > echo metaslab_debug/W0 | mdb -kw
That appears to have resolved the issue! Within seconds of making the change performance has increased by an order of magnitude. I was typing the reply below when your message came in. Is this bug 7000208? On May 4, 2011, at 4:01 PM, Garrett D'Amore wrote: > Sounds like a nasty bug, and not one I've seen in illumos or > NexentaStor. What build are you running? running snv_151a Running some synthetic tests right now and comparing the various stats, one thing that stands out as very different on this system compared to our others is that writes seem to be going to ~5 mirror sets at a time (of 22 configured.) The next batch of writes will move on to the next ~5 mirror sets, and so forth cycling around. The other systems will write to many more mirror sets simultaneously. This particular machine does not appear to be buffering writes and appears to be doing everything sync to disk despite having sync/zil disabled. I'm trying to do a little more introspection into the zpool thread that is using cpu but not having much luck finding anything meaningful. Occasionally the cpu usage for that thread will drop, and when it does performance of the filesystem increases. > On Wed, 2011-05-04 at 15:40 -0700, Adam Serediuk wrote: >> Dedup is disabled (confirmed to be.) Doing some digging it looks like >> this is a very similar issue >> to http://forums.oracle.com/forums/thread.jspa?threadID=2200577&tstart=0. >> >> >> >> On May 4, 2011, at 2:26 PM, Garrett D'Amore wrote: >> >>> My first thought is dedup... perhaps you've got dedup enabled and >>> the DDT no longer fits in RAM? That would create a huge performance >>> cliff. >>> >>> -----Original Message----- >>> From: zfs-discuss-boun...@opensolaris.org on behalf of Eric D. >>> Mudama >>> Sent: Wed 5/4/2011 12:55 PM >>> To: Adam Serediuk >>> Cc: zfs-discuss@opensolaris.org >>> Subject: Re: [zfs-discuss] Extremely Slow ZFS Performance >>> >>> On Wed, May 4 at 12:21, Adam Serediuk wrote: >>>> Both iostat and zpool iostat show very little to zero load on the >>> devices even while blocking. >>>> >>>> Any suggestions on avenues of approach for troubleshooting? >>> >>> is 'iostat -en' error free? >>> >>> >>> -- >>> Eric D. Mudama >>> edmud...@bounceswoosh.org >>> >>> _______________________________________________ >>> zfs-discuss mailing list >>> zfs-discuss@opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>> >>> >>> >>> >> >> > > _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
