On Sat, Apr 18, 2009 at 04:27:55PM -0500, Gary Mills wrote:
> We have an IMAP server with ZFS for mailbox storage that has recently
> become extremely slow on most weekday mornings and afternoons. When
> one of these incidents happens, the number of processes increases, the
> load average increases, but ZFS I/O bandwidth decreases. Users notice
> very slow response to IMAP requests. On the server, even `ps' becomes
> slow.
After I moved a couple of Cyrus databases from ZFS to UFS on Sunday
morning, the server seemed to run quite nicely. One of these
databases is memory-mapped by all of the lmtpd and pop3d processes.
The other is opened by all the lmtpd processes. Both were quite
active, with many small writes, so I assumed they'd be better on UFS.
All of the IMAP mailboxes were still on ZFS.
However, this morning, things went from bad to worse. All writes to
the ZFS filesystems stopped completely. Look at this:
$ zpool iostat 5 5
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
space 1.04T 975G 86 67 4.53M 2.57M
space 1.04T 975G 5 0 159K 0
space 1.04T 975G 7 0 337K 0
space 1.04T 975G 3 0 179K 0
space 1.04T 975G 4 0 167K 0
`fsstat' told me that there was both writes and memory-mapped I/O
to UFS, but nothing to ZFS. At the same time, the `ps' command
would hang and could not be interrupted. `truss' on `ps' looked
like this, but it eventually also stopped and not be interrupted.
open("/proc/6359/psinfo", O_RDONLY) = 4
read(4, "02\0\0\0\0\0\001\0\018D7".., 416) = 416
close(4) = 0
open("/proc/12782/psinfo", O_RDONLY) = 4
read(4, "02\0\0\0\0\0\001\0\0 1EE".., 416) = 416
close(4) = 0
What could cause this sort of behavior? It happened three times today!
--
-Gary Mills- -Unix Support- -U of M Academic Computing and Networking-
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
