Re: Recent (probably MacOS) mail app provoking endless cyrus.index writes on 2.3 server. [WARNING: DKIM validation failed]
On Fri, Oct 23, 2015 at 09:37:28AM +1100, Bron Gondwana wrote:
> >
> > If they are thinking that the example in the RFC is the specification, they
> > are
> > not correct. The IMAP server responses to determine completion of the
> > EXPUNGE
> > command are "OK" for a completed EXPUNGE, "NO" for a failure, and "BAD for
> > unknown
> > command or invalid arguments. Everything after those words are not germane
> > to
> > whether or not the command completed. The tag is what links the OK to the
> > EXPUNGE
> > command and not the "EXPUNGE completed". Sigh, they had it right and now it
> > is
> > broken.
>
> Actually, the ImapTest command complained about this too:
>
> http://imapwiki.org/ImapTest
>
> >From RFC3501:
>
> response-tagged = tag SP resp-cond-state CRLF
>
>
> resp-cond-state = ("OK" / "NO" / "BAD") SP resp-text
> ; Status condition
>
> resp-text = ["[" resp-text-code "]" SP] text
>
> text= 1*TEXT-CHAR
>
> So the exact text after the OK response doesn't matter, but it MUST be SP
> followed by at least 1 TEXT-CHAR.
>
> This is pretty easy to patch in 2.3.x if you're forced to remain there for
> reasons. It is, of course, fixed in later versions.
>
> Bron.
>
Hi Bron,
I don't suppose you have a patch? If not, I will work one up.
Regards,
Ken
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Recent (probably MacOS) mail app provoking endless cyrus.index writes on 2.3 server. [WARNING: DKIM validation failed]
Sorry, in phone so not able to look at code today. So from this it appears that calling expunge frequently will still happen. Recent 2.4 and later will be the same, but expunge with no changes to make is cheap on more recent Cyrus and won't even cause a sync log. On Sat, Oct 24, 2015, at 10:38, Matt Elson wrote: > > So the exact text after the OK response doesn't matter, but it MUST > > be SP followed by at least 1 TEXT-CHAR. > > > > This is pretty easy to patch in 2.3.x if you're forced to remain > > there for reasons. It is, of course, fixed in later versions. > > > > Bron. > > > > > > I'm seeing this issue in my environment.. I think. I'm only *fairly* > sure - I see clients that are identifying themselves as 10.11 causing > lots of entries showing up in the sync log and a whole lot of index > churning on.. usually smaller folders, luckily. I've been trying to > track down a test machine to validate, but resources are quite limited > where I am. > > But I'm a bit confused about the current working theory as to what's > causing Mail.app to go into its strange loop. > > My Cyrus (RedHat provided 2.3.16-13.el6_6 with a very minor local patch) > responds to EXPUNGE with " OK COMPLETED" as near as I can tell.. > which seems spec compliant if I understand the above. > > * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID > MUPDATE=mupdate://redacted.example.com/ AUTH=GSSAPI AUTH=PLAIN > AUTH=LOGIN SASL-IR COMPRESS=DEFLATE] redacted.example.com Cyrus IMAP > Murder v2.3.16-Fedora-RPM-2.3.16-13.el6_6 server ready > A0001 login REDACTED PASSWORD > A0001 OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID > MUPDATE=mupdate://redacted.example.com/ LOGINDISABLED AUTH=GSSAPI > AUTH=PLAIN AUTH=LOGIN COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA > MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN > MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT > THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN IDLE LISTEXT > LIST-SUBSCRIBED X-NETSCAPE URLAUTH] User logged in > A0002 SELECT Trash > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen) > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] > * 4560 EXISTS > * 0 RECENT > * OK [UNSEEN 1] > * OK [UIDVALIDITY 1347582296] > * OK [UIDNEXT 6437] > * OK [NOMODSEQ] Sorry, modsequences have not been enabled on this mailbox > * OK [URLMECH INTERNAL] > A0002 OK [READ-WRITE] Completed > A0003 idle > + idling > done > A0003 OK Completed > A0004 EXPUNGE > * 4560 EXISTS > * 0 RECENT > A0004 OK Completed > > > In fact a 2.4.18 install I'm building (I'm using this issue as an excuse > to prioritize a long time coming upgrade) seems to respond to expunge in > the same way. > > > * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE > MUPDATE=mupdate://redacted.example.com/ STARTTLS AUTH=LOGIN AUTH=PLAIN > AUTH=DIGEST-MD5 SASL-IR] redacted.example.com Cyrus IMAP Murder > v2.4.18-Invoca-RPM-2.4.18-1.el7.centos server ready > A0001 login redacted redacted > A0001 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA > MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN > MULTIAPPEND BINARY CATENATE CONDSTORE SORT SORT=MODSEQ SORT=DISPLAY > THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE WITHIN SCAN XLIST > URLAUTH X-NETSCAPE MUPDATE=mupdate://redacted.example.com/ LOGINDISABLED > COMPRESS=DEFLATE IDLE] User logged in > SESSIONID= > A0002 SELECT Trash > * 0 EXISTS > * 0 RECENT > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen) > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok > * OK [UIDVALIDITY 144503] Ok > * OK [UIDNEXT 1] Ok > * OK [HIGHESTMODSEQ 1] Ok > * OK [URLMECH INTERNAL] Ok > A0002 OK [READ-WRITE] Completed > A0003 idle > + idling > done > A0003 OK Completed > A0004 EXPUNGE > A0004 OK Completed > > (And fastmail's IMAP responds with a OK Completed, and I assume it > represents the most up to date Cyrus :P) > > Am I misunderstanding the theoretical issue? My IMAP-fu is a little > rusty these days and I'm a bit bleary eyed so apologies in advance if > I'm missing something obvious - I'm just testing by typing the commands > in raw since imaptest is panicking on me and I've not had time to sort > that out (though my Thunderbird debug log also shows an OK Completed as > a repsonse). > > It does seem from one of the trace logs from the Apple discussion that > there is a version of Cyrus that responds with just OK, but.. it's not > all 2.3.x 's at the very least :P. > > Sorry again if this noise and thanks in advance for any clarification > someone can give. > > Matt > > > > > > > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To
Re: Recent (probably MacOS) mail app provoking endless cyrus.index writes on 2.3 server. [WARNING: DKIM validation failed]
> So the exact text after the OK response doesn't matter, but it MUST > be SP followed by at least 1 TEXT-CHAR. > > This is pretty easy to patch in 2.3.x if you're forced to remain > there for reasons. It is, of course, fixed in later versions. > > Bron. > > I'm seeing this issue in my environment.. I think. I'm only *fairly* sure - I see clients that are identifying themselves as 10.11 causing lots of entries showing up in the sync log and a whole lot of index churning on.. usually smaller folders, luckily. I've been trying to track down a test machine to validate, but resources are quite limited where I am. But I'm a bit confused about the current working theory as to what's causing Mail.app to go into its strange loop. My Cyrus (RedHat provided 2.3.16-13.el6_6 with a very minor local patch) responds to EXPUNGE with " OK COMPLETED" as near as I can tell.. which seems spec compliant if I understand the above. * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID MUPDATE=mupdate://redacted.example.com/ AUTH=GSSAPI AUTH=PLAIN AUTH=LOGIN SASL-IR COMPRESS=DEFLATE] redacted.example.com Cyrus IMAP Murder v2.3.16-Fedora-RPM-2.3.16-13.el6_6 server ready A0001 login REDACTED PASSWORD A0001 OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID MUPDATE=mupdate://redacted.example.com/ LOGINDISABLED AUTH=GSSAPI AUTH=PLAIN AUTH=LOGIN COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH] User logged in A0002 SELECT Trash * FLAGS (\Answered \Flagged \Draft \Deleted \Seen) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] * 4560 EXISTS * 0 RECENT * OK [UNSEEN 1] * OK [UIDVALIDITY 1347582296] * OK [UIDNEXT 6437] * OK [NOMODSEQ] Sorry, modsequences have not been enabled on this mailbox * OK [URLMECH INTERNAL] A0002 OK [READ-WRITE] Completed A0003 idle + idling done A0003 OK Completed A0004 EXPUNGE * 4560 EXISTS * 0 RECENT A0004 OK Completed In fact a 2.4.18 install I'm building (I'm using this issue as an excuse to prioritize a long time coming upgrade) seems to respond to expunge in the same way. * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE MUPDATE=mupdate://redacted.example.com/ STARTTLS AUTH=LOGIN AUTH=PLAIN AUTH=DIGEST-MD5 SASL-IR] redacted.example.com Cyrus IMAP Murder v2.4.18-Invoca-RPM-2.4.18-1.el7.centos server ready A0001 login redacted redacted A0001 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE SORT SORT=MODSEQ SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE WITHIN SCAN XLIST URLAUTH X-NETSCAPE MUPDATE=mupdate://redacted.example.com/ LOGINDISABLED COMPRESS=DEFLATE IDLE] User logged in SESSIONID= A0002 SELECT Trash * 0 EXISTS * 0 RECENT * FLAGS (\Answered \Flagged \Draft \Deleted \Seen) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok * OK [UIDVALIDITY 144503] Ok * OK [UIDNEXT 1] Ok * OK [HIGHESTMODSEQ 1] Ok * OK [URLMECH INTERNAL] Ok A0002 OK [READ-WRITE] Completed A0003 idle + idling done A0003 OK Completed A0004 EXPUNGE A0004 OK Completed (And fastmail's IMAP responds with a OK Completed, and I assume it represents the most up to date Cyrus :P) Am I misunderstanding the theoretical issue? My IMAP-fu is a little rusty these days and I'm a bit bleary eyed so apologies in advance if I'm missing something obvious - I'm just testing by typing the commands in raw since imaptest is panicking on me and I've not had time to sort that out (though my Thunderbird debug log also shows an OK Completed as a repsonse). It does seem from one of the trace logs from the Apple discussion that there is a version of Cyrus that responds with just OK, but.. it's not all 2.3.x 's at the very least :P. Sorry again if this noise and thanks in advance for any clarification someone can give. Matt Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Recent (probably MacOS) mail app provoking endless cyrus.index writes on 2.3 server.
On Tue, October 13, 2015 3:26 pm, k...@rice.edu wrote: > The El Capitan Mac Mail was recently released. It definitely has a problem. > We saw our typical sync log size go from 10k when busy to 100k+. We are going > to recommend that people wait to upgrade to that release until the problem is > addressed. > > Regards, > Ken Earlier today I used DTrace to capture I/O operations on our Solaris 10 mail server. The imapd processes hammered by those El Capitan Mail.app clients perform roughly 500 (five hundred) cyrus.index + cyrus.cache rewrites per second. Multiply by the number of affected users, now numbering beyond one hundred. Cool. Eric. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Recent (probably MacOS) mail app provoking endless cyrus.index writes on 2.3 server.
On Oct 23, 2015, at 2:03 PM, k...@rice.edu wrote: > > On Fri, Oct 23, 2015 at 10:59:08PM +0200, Eric Luyten wrote: >> On Tue, October 13, 2015 3:26 pm, k...@rice.edu wrote: >> >>> The El Capitan Mac Mail was recently released. It definitely has a problem. >>> We saw our typical sync log size go from 10k when busy to 100k+. We are >>> going >>> to recommend that people wait to upgrade to that release until the problem >>> is >>> addressed. >>> >>> Regards, >>> Ken >> >> >> Earlier today I used DTrace to capture I/O operations on our Solaris 10 mail >> server. >> The imapd processes hammered by those El Capitan Mail.app clients perform >> roughly 500 (five hundred) cyrus.index + cyrus.cache rewrites per second. >> Multiply by the number of affected users, now numbering beyond one hundred. >> Cool. >> >> >> Eric. >> > > Hi Eric, > > A bug in cyrus-imapd version 2.3.x. I am looking into generating an rpm > with a patch. In the meantime, we are disabling the automatic folder > compaction > option to keep from hammering the server. > > Regards, > Ken > Glad I found this thread. :) I’m now seeing this on our 2.3.x server. We had maybe 3-5 people upgrade to El Capitan so far, and I’m seeing the described behavior. Would greatly appreciate an RPM with a patch! Thanks, Bryan --- Bryan Hill Lead System Administrator UCSD Physics Computing Facility 9500 Gilman Dr. # 0319 La Jolla, CA 92093 +1-858-534-5538 bh...@ucsd.edu Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Recent (probably MacOS) mail app provoking endless cyrus.index writes on 2.3 server.
On Fri, Oct 23, 2015 at 10:59:08PM +0200, Eric Luyten wrote: > On Tue, October 13, 2015 3:26 pm, k...@rice.edu wrote: > > > The El Capitan Mac Mail was recently released. It definitely has a problem. > > We saw our typical sync log size go from 10k when busy to 100k+. We are > > going > > to recommend that people wait to upgrade to that release until the problem > > is > > addressed. > > > > Regards, > > Ken > > > Earlier today I used DTrace to capture I/O operations on our Solaris 10 mail > server. > The imapd processes hammered by those El Capitan Mail.app clients perform > roughly 500 (five hundred) cyrus.index + cyrus.cache rewrites per second. > Multiply by the number of affected users, now numbering beyond one hundred. > Cool. > > > Eric. > Hi Eric, A bug in cyrus-imapd version 2.3.x. I am looking into generating an rpm with a patch. In the meantime, we are disabling the automatic folder compaction option to keep from hammering the server. Regards, Ken Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Recent (probably MacOS) mail app provoking endless cyrus.index writes on 2.3 server.
On Fri, October 23, 2015 11:03 pm, k...@rice.edu wrote: > I am looking into generating an rpm > with a patch. Ken, Feel free to throw a 'diff' file in my general direction, even when generated against another 2.3 version. I compile from source because of two small site specific patches. Could be tested on our system shortly after this weekend. Eric. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
