Re: [Dovecot] Dovecot write activity (mostly 1.1.x)
On Sun, November 4, 2007 4:32 pm, Timo Sirainen wrote: I didn't know that mail_nfs_index=yes resulted in a forced chown. How come that's necessary with NFS but not on local disks? It's used to flush NFS attribute cache. Enabling it allows you to use multiple servers to access the same maildir at the same time while still having attribute cache enabled (v1.0 required actime=0). If you don't need this (and it's better if you don't), then just set the mail_nfs_* to no and it works faster. By the way I misinformed you about fsync_disable=yes. It was like that before i upgraded to v1.1, but v1.1 requires fsync_disable=no when mail_nfs_index=true so I had to disable it. So you use ZFS on the NFS server, but Dovecot is using NFS on client side? The fsyncs then just mean that the data is sent to NFS server, not a real fsync on ZFS. Thanks a lot for the help - this changed a lot! Dist writes fell to about 1/3 of before. I guess the reason is that ZFS can now make use if it's caching capabilities. Delivers activity is completely random since it's impossible to load balance a connection based on the e-mail recipient, since only the ip is known at the load balancing point. Therefore I have fsync_disable=no for deliver. It's easy to force the clients using imap/pop3 to the same server since it can be based on the ip only. Therefore I have fsync_disable=yes for imap/pop3. This changed everything. Now there's a real performance gain upgrading from 1.0.x to 1.1.x. About two or three times less disk activity overall (reads were already improved) for both reads and writes. Thats pretty neat!
Re: [Dovecot] Dovecot write activity (mostly 1.1.x)
On Sun, 2007-11-04 at 13:02 +0100, [EMAIL PROTECTED] wrote: Write activity is about half that of read activity when measuring throughput. But when measuring operations it’s about 5-7 times as high (measured with zpool iostat on ZFS). Have you tried with fsync_disable=yes? ZFS's fsyncing performance apparently isn't all that great. I'm guessing this is also the reason for the I/O stalls in your other mail. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Dovecot write activity (mostly 1.1.x)
On Sun, November 4, 2007 1:51 pm, Timo Sirainen wrote: On Sun, 2007-11-04 at 13:02 +0100, [EMAIL PROTECTED] wrote: Write activity is about half that of read activity when measuring throughput. But when measuring operations itâs about 5-7 times as high (measured with zpool iostat on ZFS). Have you tried with fsync_disable=yes? ZFS's fsyncing performance apparently isn't all that great. I'm guessing this is also the reason for the I/O stalls in your other mail. I'm using fsync_disable=yes already. I know ZFS has issues. In my opinion it was never ready when it was released but it has such nice features Im trying to cope with its specialities. I've also disabled flush cache requests in ZFS since it made the performance horrible. If Im the only one experiencing this then I guess I'll just have to accept it as yet another ZFS curiosity :| (Possibly this is also the answer to my other post regarding stalled/delayed I/O) - Mikkel
Re: [Dovecot] Dovecot write activity (mostly 1.1.x)
On Sun, 2007-11-04 at 14:10 +0100, [EMAIL PROTECTED] wrote: Write activity is about half that of read activity when measuring throughput. But when measuring operations it’s about 5-7 times as high (measured with zpool iostat on ZFS). Have you tried with fsync_disable=yes? ZFS's fsyncing performance apparently isn't all that great. I'm guessing this is also the reason for the I/O stalls in your other mail. I'm using fsync_disable=yes already. Well, if you use only clients that don't really need indexes they could just slow things down. You could try disabling indexes to see how it works then (:INDEX=MEMORY to mail_location). Although v1.1 should be writing less to disk than v1.0, because v1.1 doesn't anymore update dovecot.index as often. v1.1 versions before beta6 had some problems updating cache file though. They could have updated it more often than necessary, and beta5 didn't really even update it at all. (Possibly this is also the answer to my other post regarding stalled/delayed I/O) You could truss the hanging process to see what it's doing. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Dovecot write activity (mostly 1.1.x)
On Sun, November 4, 2007 2:20 pm, Timo Sirainen wrote: Well, if you use only clients that don't really need indexes they could just slow things down. You could try disabling indexes to see how it works then (:INDEX=MEMORY to mail_location). I tried that earlier and it did result in less writes. It would be a nice-to-have option to be able to individually tell deliver, imapd and popd whether they should update indexes and cache. Although v1.1 should be writing less to disk than v1.0, because v1.1 doesn't anymore update dovecot.index as often. v1.1 versions before beta6 had some problems updating cache file though. They could have updated it more often than necessary, and beta5 didn't really even update it at all. Okay, then I really need to wait and see if things change (it'll probably take a few days before the majority of e-mail accounts are re-indexed and cached). By the way writes increased noticeably when i upgraded from 1.0 to 1.1. On the other hand reads decreased a lot as well. I guess the fail-to-update-cache bug you mentioned could have a lot to do with that. (Possibly this is also the answer to my other post regarding stalled/delayed I/O) You could truss the hanging process to see what it's doing. It's not an easy task since the delay is sometimes just a few (5-10) seconds. And when there is a complete stall the client aborts before I can find the process. But I'll give it a go. Thanks for your input.
Re: [Dovecot] Dovecot write activity (mostly 1.1.x)
On Sun, November 4, 2007 2:54 pm, [EMAIL PROTECTED] wrote: You could truss the hanging process to see what it's doing. It's not an easy task since the delay is sometimes just a few (5-10) seconds. And when there is a complete stall the client aborts before I can find the process. But I'll give it a go. I tried trussing a normal running process. Here's what I see all the time: stat64([path]/Maildir/new, 0xFFBFF470) = 0 stat64([path]/Maildir/cur, 0xFFBFF4E0) = 0 stat64([path]/Maildir/new, 0xFFBFF2F0) = 0 stat64([path]/Maildir/cur, 0xFFBFF258) = 0 stat64([path]/Maildir/dovecot-uidlist, 0xFFBFF1D0) = 0 chown([path]/Maildir/dovecot-uidlist, 105, -1) = 0 stat64([path]/Maildir/dovecot-uidlist, 0xFFBFF2F0) = 0 stat64([path]/Maildir/dovecot.index.log, 0xFFBFDAE0) = 0 chown([path]/Maildir/dovecot.index.log, 105, -1) = 0 stat64([path]/Maildir/dovecot.index.log, 0xFFBFDBF0) = 0 What i notice is that stat64 is very often called twice in a row on the same file. Also I notice that chown() is always run before a file is accessed. Regarding chown it looks like dovecot either thinks that the file hasn't got the rights it should have, or it just calls chown anyway to be sure. I'm not a C-programmer so I have no idea whether its supposed to be like that. But if it isnt then perhaps it could explain the many writes (chowning constantly)? What do you think?
Re: [Dovecot] Dovecot write activity (mostly 1.1.x)
On Sun, 2007-11-04 at 15:27 +0100, [EMAIL PROTECTED] wrote: chown([path]/Maildir/dovecot-uidlist, 105, -1) = 0 stat64([path]/Maildir/dovecot-uidlist, 0xFFBFF2F0) = 0 stat64([path]/Maildir/dovecot.index.log, 0xFFBFDAE0) = 0 chown([path]/Maildir/dovecot.index.log, 105, -1) = 0 stat64([path]/Maildir/dovecot.index.log, 0xFFBFDBF0) = 0 What i notice is that stat64 is very often called twice in a row on the same file. Also I notice that chown() is always run before a file is accessed. Regarding chown it looks like dovecot either thinks that the file hasn't got the rights it should have, or it just calls chown anyway to be sure. I'm not sure about the double stat(), but chown() most means you've set mail_nfs_index=yes. I'll see if I can do something about the second stat, but it's unlikely that it makes any difference to disk I/O. signature.asc Description: This is a digitally signed message part
