Howdy, We're having a ZFS performance issue over here that I was hoping you guys could help me troubleshoot. We have a ZFS pool made up of 24 disks, arranged into 7 raid-z devices of 4 disks each. We're using it as an iSCSI back-end for VMWare and some Oracle RAC clusters.
Under normal circumstances performance is very good both in benchmarks and under real-world use. Every couple days, however, I/O seems to hang for anywhere between several seconds and several minutes. The hang seems to be a complete stop of all write I/O. The following zpool iostat illustrates: pool0 2.47T 5.13T 120 0 293K 0 pool0 2.47T 5.13T 127 0 308K 0 pool0 2.47T 5.13T 131 0 322K 0 pool0 2.47T 5.13T 144 0 347K 0 pool0 2.47T 5.13T 135 0 331K 0 pool0 2.47T 5.13T 122 0 295K 0 pool0 2.47T 5.13T 135 0 330K 0 While this is going on our VMs all hang, as do any "zfs create" commands or attempts to touch/create files in the zfs pool from the local system. After several minutes the system "un-hangs" and we see very high write rates before things return to normal across the board. Some more information about our configuration: We're running OpenSolaris svn-134. ZFS is at version 22. Our disks are 15kRPM 300gb Seagate Cheetahs, mounted in Promise J610S Dual enclosures, hanging off a Dell SAS 5/e controller. We'd tried out most of this configuration previously on OpenSolaris 2009.06 without running into this problem. The only thing that's new, aside from the newer OpenSolaris/ZFS is a set of four SSDs configured as log disks. At first we blamed de-dupe, but we've disabled that. Next we suspected the SSD log disks, but we've seen the problem with those removed, as well. Has anyone seen anything like this before? Are there any tools we can use to gather information during the hang which might be useful in determining what's going wrong? Thanks for any insights you may have. -Charles -- This message posted from opensolaris.org _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
