Re: [zfs-discuss] Extremely Slow ZFS Performance
Sounds like a nasty bug, and not one I've seen in illumos or NexentaStor. What build are you running? - Garrett On Wed, 2011-05-04 at 15:40 -0700, Adam Serediuk wrote: Dedup is disabled (confirmed to be.) Doing some digging it looks like this is a very similar issue to http://forums.oracle.com/forums/thread.jspa?threadID=2200577tstart=0. On May 4, 2011, at 2:26 PM, Garrett D'Amore wrote: My first thought is dedup... perhaps you've got dedup enabled and the DDT no longer fits in RAM? That would create a huge performance cliff. -Original Message- From: zfs-discuss-boun...@opensolaris.org on behalf of Eric D. Mudama Sent: Wed 5/4/2011 12:55 PM To: Adam Serediuk Cc: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] Extremely Slow ZFS Performance On Wed, May 4 at 12:21, Adam Serediuk wrote: Both iostat and zpool iostat show very little to zero load on the devices even while blocking. Any suggestions on avenues of approach for troubleshooting? is 'iostat -en' error free? -- Eric D. Mudama edmud...@bounceswoosh.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Extremely Slow ZFS Performance
On Wed, May 4 at 12:21, Adam Serediuk wrote: Both iostat and zpool iostat show very little to zero load on the devices even while blocking. Any suggestions on avenues of approach for troubleshooting? is 'iostat -en' error free? -- Eric D. Mudama edmud...@bounceswoosh.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Extremely Slow ZFS Performance
iostat doesn't show any high service times and fsstat also shows low throughput. Occasionally I can generate enough load that you do see some very high asvc_t but when that occurs the pool is performing as expected. As a precaution I just added two extra drives to the zpool incase zfs was having difficulty finding a location to allocate new blocks and it has made no difference. You can still see still see block allocation cycling evenly across the available mirrors. iostat -xnz 3 before, during, and after the blocking extended device statistics r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.0 33.30.0 184.1 0.0 0.10.01.5 0 1 c12t0d0 0.0 15.30.0 98.7 0.0 0.00.01.0 0 0 c12t1d0 0.0 15.30.0 98.7 0.0 0.00.01.1 0 0 c12t2d0 0.0 14.30.0 98.7 0.0 0.00.01.0 0 0 c12t6d0 0.0 20.70.0 256.6 0.0 0.00.01.4 0 0 c11t5d0 0.0 48.00.0 273.4 0.0 0.10.01.3 0 1 c11t6d0 0.0 34.00.0 199.9 0.0 0.00.01.5 0 1 c11t7d0 0.0 20.30.0 256.6 0.0 0.00.01.4 0 0 c10t5d0 0.0 47.70.0 273.4 0.0 0.10.01.2 0 1 c10t6d0 0.0 34.00.0 199.9 0.0 0.00.01.5 0 1 c10t7d0 0.09.70.0 1237.9 0.0 0.00.04.7 0 1 c14t0d0 0.0 33.30.0 184.1 0.0 0.10.01.8 0 1 c13t0d0 0.0 15.70.0 98.7 0.0 0.00.01.2 0 0 c13t1d0 0.0 15.70.0 98.7 0.0 0.00.01.0 0 0 c13t2d0 0.0 13.70.0 98.7 0.0 0.00.01.1 0 0 c13t6d0 extended device statistics r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.00.30.00.0 0.0 0.00.00.0 0 0 c8t1d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c8t2d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c8t3d0 0.30.31.70.0 0.0 0.00.06.0 0 0 c8t4d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c8t5d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c8t6d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c8t7d0 0.02.00.01.3 0.0 0.00.00.2 0 0 c12t0d0 0.0 15.70.0 235.9 0.0 0.00.00.8 0 0 c12t1d0 0.3 16.31.8 235.9 0.0 0.00.01.2 0 1 c12t2d0 0.3 15.02.8 130.6 0.0 0.00.01.0 0 1 c12t3d0 0.0 11.70.0 127.9 0.0 0.00.00.8 0 0 c12t4d0 0.30.30.20.0 0.0 0.00.09.0 0 1 c12t5d0 0.0 40.30.0 599.9 0.0 0.00.00.8 0 1 c12t6d0 0.30.32.80.0 0.0 0.00.03.2 0 0 c11t0d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c11t1d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c11t2d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c11t3d0 0.70.33.00.0 0.0 0.00.06.5 0 1 c11t4d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c11t5d0 0.32.01.51.3 0.0 0.00.02.8 0 1 c11t6d0 0.02.00.01.3 0.0 0.00.00.2 0 0 c11t7d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c10t0d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c10t1d0 0.30.30.20.0 0.0 0.00.06.7 0 0 c10t2d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c10t3d0 0.30.31.50.0 0.0 0.00.09.1 0 1 c10t4d0 0.30.31.50.0 0.0 0.00.07.7 0 1 c10t5d0 0.32.01.51.3 0.0 0.00.02.2 0 0 c10t6d0 0.02.00.01.3 0.0 0.00.00.2 0 0 c10t7d0 0.00.30.0 42.6 0.0 0.00.00.7 0 0 c14t1d0 0.30.31.50.0 0.0 0.00.05.9 0 0 c9t1d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c9t2d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c9t3d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c9t4d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c9t5d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c9t6d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c9t7d0 0.02.00.01.3 0.0 0.00.00.4 0 0 c13t0d0 0.0 15.70.0 235.9 0.0 0.00.00.8 0 0 c13t1d0 0.0 16.30.0 235.9 0.0 0.00.00.9 0 0 c13t2d0 0.0 14.70.0 130.6 0.0 0.00.00.7 0 0 c13t3d0 0.0 12.00.0 127.9 0.0 0.00.00.9 0 0 c13t4d0 0.00.30.00.0 0.0 0.00.00.0 0 0 c13t5d0 0.0 39.60.0 599.9 0.0 0.00.00.8 0 1 c13t6d0 extended device statistics r/sw/s
Re: [zfs-discuss] Extremely Slow ZFS Performance
Dedup is disabled (confirmed to be.) Doing some digging it looks like this is a very similar issue to http://forums.oracle.com/forums/thread.jspa?threadID=2200577tstart=0. On May 4, 2011, at 2:26 PM, Garrett D'Amore wrote: My first thought is dedup... perhaps you've got dedup enabled and the DDT no longer fits in RAM? That would create a huge performance cliff. -Original Message- From: zfs-discuss-boun...@opensolaris.org on behalf of Eric D. Mudama Sent: Wed 5/4/2011 12:55 PM To: Adam Serediuk Cc: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] Extremely Slow ZFS Performance On Wed, May 4 at 12:21, Adam Serediuk wrote: Both iostat and zpool iostat show very little to zero load on the devices even while blocking. Any suggestions on avenues of approach for troubleshooting? is 'iostat -en' error free? -- Eric D. Mudama edmud...@bounceswoosh.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Extremely Slow ZFS Performance
On May 4, 2011, at 4:16 PM, Victor Latushkin wrote: Try echo metaslab_debug/W1 | mdb -kw If it does not help, reset it back to zero echo metaslab_debug/W0 | mdb -kw That appears to have resolved the issue! Within seconds of making the change performance has increased by an order of magnitude. I was typing the reply below when your message came in. Is this bug 7000208? On May 4, 2011, at 4:01 PM, Garrett D'Amore wrote: Sounds like a nasty bug, and not one I've seen in illumos or NexentaStor. What build are you running? running snv_151a Running some synthetic tests right now and comparing the various stats, one thing that stands out as very different on this system compared to our others is that writes seem to be going to ~5 mirror sets at a time (of 22 configured.) The next batch of writes will move on to the next ~5 mirror sets, and so forth cycling around. The other systems will write to many more mirror sets simultaneously. This particular machine does not appear to be buffering writes and appears to be doing everything sync to disk despite having sync/zil disabled. I'm trying to do a little more introspection into the zpool thread that is using cpu but not having much luck finding anything meaningful. Occasionally the cpu usage for that thread will drop, and when it does performance of the filesystem increases. On Wed, 2011-05-04 at 15:40 -0700, Adam Serediuk wrote: Dedup is disabled (confirmed to be.) Doing some digging it looks like this is a very similar issue to http://forums.oracle.com/forums/thread.jspa?threadID=2200577tstart=0. On May 4, 2011, at 2:26 PM, Garrett D'Amore wrote: My first thought is dedup... perhaps you've got dedup enabled and the DDT no longer fits in RAM? That would create a huge performance cliff. -Original Message- From: zfs-discuss-boun...@opensolaris.org on behalf of Eric D. Mudama Sent: Wed 5/4/2011 12:55 PM To: Adam Serediuk Cc: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] Extremely Slow ZFS Performance On Wed, May 4 at 12:21, Adam Serediuk wrote: Both iostat and zpool iostat show very little to zero load on the devices even while blocking. Any suggestions on avenues of approach for troubleshooting? is 'iostat -en' error free? -- Eric D. Mudama edmud...@bounceswoosh.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
