Re: Mailshare conversations
Sorry for the delay. Yes, this does seem like the right thing to do. The conversation root being the user is a convenience for a ton of things to the point that we're basically using "fake users" for storing related things for a business. But doing the same thing for mailshare would be quite easy I think. There are only a few places where we convert mailbox name to conversations_db path. Bron. On Tue, Sep 22, 2020, at 01:00, Thomas Fricker wrote: > Hello, > > I noticed that Cyrus does not keep track of conversations in mailshare > folders. > There is a code comment addressing this issue in conversations.c: > > /* only users have conversations. Later we may introduce the idea of > * "conversationroot" in the same way we have quotaroot, but for now > * it's hard-coded as the user */ > > Is it still imaginable adding this feature in a future Cyrus version? > Can you elaborate on a reasonable approach of implementing this on the > current codebase? > My goal is to have 1 conversation.db per mailshare subtree (not per single > folder). > > Any ideas? > > Thanks, > -- > Thomas Fricker > > > BlueMind LogoLinkedin > <https://www.linkedin.com/company/blue-mind/?originalSubdomain=fr> Facebook > <https://www.facebook.com/BlueMind.Mail/> Twitter > <https://twitter.com/_bluemind?lang=fr> > <https://www.youtube.com/channel/UCd12EAS5e3U8M0f2IRkisqw> > *Thomas Fricker * > DEVELOPPEUR > +33 (0)5 81 91 55 60 > www.bluemind.net / Blog <https://blog.bluemind.net/fr/> > [*WEBINAR*] > *Migrer sa messagerie vers BlueMind depuis Exchange* > *24/09/20 - 11h* > *INSCRIPTION* > <https://content.bluemind.net/migrer-vers-bluemind-depuis-exchange> > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > *Attachments:* > * x-disclaimer1569892563-0.png > * x-disclaimer1569892563-1.png > * x-disclaimer1569892563-2.png > * x-disclaimer1569892563-3.png > * x-disclaimer1569892563-4.png -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Replication - current status and how to do failover
On Sun, Apr 5, 2020, at 00:45, Olaf Frączyk wrote: > Hello, > > 1. Is currently master-master replication possible (maybe 3.2) Is it OK > to sync them two-way? No, not really. It'll mostly be fine, but it doesn't (yet) handle folder create/rename/delete safely. > If yes - how to set up such config? > > 2. If master-master is impossible, is there any guide how to setup > failover from master to slave and possibly back? If split-brain happens > - is there an easy recovery from such state? The way we do it at Fastmail is with an nginx proxy in front which knows which one is the master. For a clean shutdown, we shut down the master, then run sync_client -r -f with the log file (if anything was unreplicated) to make sure it's up to date, then shut down both and bring them up with the config pointing the replication the other way. For a case where the master crashed hard, we switch the replica to be master by changing the config (with a restart again) then bring the old master back up and for sync_client everything again like above to switch back, so all new changes from the regular replica are back on the regular master. Then we bring up the regular master as master again, and run sync_client -A from there to replicate all remaining changes. That mostly works. The plan in 3.4+ is to use the mailbox tombstone records to get the create/rename/delete to the same level of split-brain safety as the UIDs inside the mailbox have. Cheers, Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Suggested feature and contribution
On Thu, Jul 18, 2019, at 19:09, Egoitz Aurrekoetxea wrote:
> Good morning,
>
> When using delete_delayed if someone removes a big folder (that one with more
> than 20 subfolders anywhere below it) in mboxlist_delayed_deletemailbox()
> only last 20 are preserved. We think it could be a good idea to preserve all
> and to have a parameter for configuring it. The reason for that, is that we
> use delete_delayed for storing the removed content remotely with the customer
> hired retention period in slow disk space. Perhaps could be a good idea
> something like :
>
> In mboxlist_delayed_deletemailbox() :
>
> *If (!preserve_delete_delayed_folders_always)*
> *{*
> /* keep the last 19, so the new one is the 20th */
> for (i = 0; i < (int)existing.count - 19; i++) {
> const char *subname = strarray_nth(, i);
> syslog(LOG_NOTICE, "too many subfolders for %s, deleting %s (%d / %d)",
> newname, subname, i+1, (int)existing.count);
> r = mboxlist_deletemailbox(subname, 1, userid, auth_state, NULL, 0, 1, 1,
> keep_intermediaries);
> if (r) goto done;
> }
> *}*
Hmm yeah, OK. This is actually buggy in that case! The intended behaviour
was to avoid a Denial of Service attack where you would create and delete the
same mailbox name millions of times - however, the whole concept is bogus
because there's nothing stopping somebody creating and deleting folder01
through folderFF and creating the same attack.
I suggest that we just remove this whole silly check entirely, and if we want a
similar level of attack protection we do something smarter like a quota for
total folders+deleted folders that haven't been cleaned up yet - set it high
enough that anybody hitting that is clearly doing something wrong, and require
the administrator semi-manually clean up the deleted folders in order to
re-allow folder creation.
Cheers,
Bron.
--
br...@fastmailteam.com
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Suggested feature and contribution
On Fri, Jul 19, 2019, at 10:09, Bron Gondwana wrote:
> On Thu, Jul 18, 2019, at 19:09, Egoitz Aurrekoetxea wrote:
>> Good morning,
>>
>> When using delete_delayed if someone removes a big folder (that one with
>> more than 20 subfolders anywhere below it) in
>> mboxlist_delayed_deletemailbox() only last 20 are preserved. We think it
>> could be a good idea to preserve all and to have a parameter for configuring
>> it. The reason for that, is that we use delete_delayed for storing the
>> removed content remotely with the customer hired retention period in slow
>> disk space. Perhaps could be a good idea something like :
>>
>> In mboxlist_delayed_deletemailbox() :
>>
>> *If (!preserve_delete_delayed_folders_always)*
>> *{*
>> /* keep the last 19, so the new one is the 20th */
>> for (i = 0; i < (int)existing.count - 19; i++) {
>> const char *subname = strarray_nth(, i);
>> syslog(LOG_NOTICE, "too many subfolders for %s, deleting %s (%d / %d)",
>> newname, subname, i+1, (int)existing.count);
>> r = mboxlist_deletemailbox(subname, 1, userid, auth_state, NULL, 0, 1, 1,
>> keep_intermediaries);
>> if (r) goto done;
>> }
>> *}*
>
> Hmm yeah, OK. This is actually buggy in that case! The intended behaviour
> was to avoid a Denial of Service attack where you would create and delete the
> same mailbox name millions of times - however, the whole concept is bogus
> because there's nothing stopping somebody creating and deleting folder01
> through folderFF and creating the same attack.
>
> I suggest that we just remove this whole silly check entirely, and if we want
> a similar level of attack protection we do something smarter like a quota for
> total folders+deleted folders that haven't been cleaned up yet - set it high
> enough that anybody hitting that is clearly doing something wrong, and
> require the administrator semi-manually clean up the deleted folders in order
> to re-allow folder creation.
>
> Cheers,
>
> Bron.
I've pushed commits to master and 3.0 to remove this check.
--
Bron Gondwana, CEO, Fastmail Pty Ltd
br...@fastmailteam.com
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Folder subscription issue
Awesome, I'll take a look. We're just talking about having some level of
self-service restore in JMAP as well, so it'll be great to look at what you've
done.
Cheers,
Bron.
On Thu, Jul 18, 2019, at 20:30, Egoitz Aurrekoetxea wrote:
> Thanks :) :)
>
> Take a look please at what I told you too in a new thread about the
> contribution with our deleted mail restoring system… perhaps it’s interesting
> for Cyrus?. For us it’s pretty useful at least… it allows users to restore
> deleted mail by their own…
>
> Cheers!!
>
>
> sarenet
> *Egoitz Aurrekoetxea*
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es
> www.sarenet.es
>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>
>> El 18 jul 2019, a las 12:22, Bron Gondwana escribió:
>>
>> Hmmm so sync_client isn't validating the user match properly. Fun.
>> That's definitely a bug that we should fix.
>>
>> I'll ask ellie (CC'd) to play with the test on that, or guide you through
>> the process of adding one to Cassandane!
>>
>> Cheers,
>>
>> Bron.
>>
>> On Thu, Jul 18, 2019, at 17:22, Egoitz Aurrekoetxea wrote:
>>> Hi!,
>>>
>>> Yes, I’m so sorry for the noise… but you know…. It’s an un modified script
>>> in more than 8 years, not done by me… which you assume it works… because
>>> indeed, it copies mailbox content (sync_client -u aaa^a...@bbb.es
>>> <mailto:aaa%5ea...@bbb.es> seems to copy mailbox content as aaa.a...@bbb.es
>>> but not subs) but not “properly” the subs, sieve… Basically taking a look
>>> at some stored consoles (ssh sessions) of the migration…. I noted that were
>>> in fact those USER ___^@ so I did… I’m doing it by me!! Then looked
>>> at that script and saw….
>>>
>>> Sorry for the time wasted Ellie.. same Bron,
>>>
>>> Cheers!
>>>
>>> sarenet
>>> *Egoitz Aurrekoetxea*
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnológico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> ego...@sarenet.es
>>> www.sarenet.es
>>>
>>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>>>
>>>> El 17 jul 2019, a las 23:58, Bron Gondwana
>>>> escribió:
>>>>
>>>> Oh awesome - thanks for letting us know :) This will save ellie spending
>>>> more time on it, though it's good to have tests for it anyway.
>>>>
>>>> Bron.
>>>>
>>>> On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
>>>>> Hi mates,
>>>>>
>>>>> Reproduced and properly located (*_not a Cyrus bug_*). We have to
>>>>> apologize for the generated noise. *_Cyrus replication works just fine_*.
>>>>>
>>>>> As I explained before, we fetch from Mysql the accounts each server has
>>>>> (instead of using -A) for no reason… just for historical reason…
>>>>>
>>>>> So, when we fetched the emails, as in very old (now) versions of Cyrus a
>>>>> conversion from . to ^ seem to be needed (the first Cyrus version was 2.0
>>>>> o even older), we were doing the following in the Perl script :
>>>>>
>>>>> while (@fila = $query->fetchrow()) {
>>>>> $fila[0] =~ s/\./^/g;
>>>>> $email = $fila[0]."@".$fila[1];
>>>>> system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client
>>>>> -S xx.sarenet.es -v -u ".$email);
>>>>>
>>>>> This explained the existence of .seen and .sub files with “^” and Sieve
>>>>> dirs with ‘^’ too. That didn’t in fact explain, why they had non zero
>>>>> size. The reason of the non zero size (of files with ^ and ending in .sub
>>>>> and .seen and Sieve dirs), is that as we come from Cyrus versions 2.3
>>>>> (and probably 2.4 perhaps do it) used the ‘^’ in the seen and sub
>>>>> filenames and Sieve names. So if you do a sync_client of a user with ‘^’
>>>>> , even in 3.0 it copies you that file content of .sub and .seen from the
>>>>> master to the slave (because it exists! because it has copied with
>>>>> sync_client too but from a 2.3/2.4 version), even when they have not been
>>>>> used in a 3.0 version. Apart the own mail account content is always
>>>>> copied and
Re: Folder subscription issue
Hmmm so sync_client isn't validating the user match properly. Fun. That's
definitely a bug that we should fix.
I'll ask ellie (CC'd) to play with the test on that, or guide you through the
process of adding one to Cassandane!
Cheers,
Bron.
On Thu, Jul 18, 2019, at 17:22, Egoitz Aurrekoetxea wrote:
> Hi!,
>
> Yes, I’m so sorry for the noise… but you know…. It’s an un modified script in
> more than 8 years, not done by me… which you assume it works… because indeed,
> it copies mailbox content (sync_client -u aaa^a...@bbb.es
> <mailto:aaa%5ea...@bbb.es> seems to copy mailbox content as aaa.a...@bbb.es
> but not subs) but not “properly” the subs, sieve… Basically taking a look at
> some stored consoles (ssh sessions) of the migration…. I noted that were in
> fact those USER ___^@ so I did… I’m doing it by me!! Then looked at
> that script and saw….
>
> Sorry for the time wasted Ellie.. same Bron,
>
> Cheers!
>
> sarenet
> *Egoitz Aurrekoetxea*
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es
> www.sarenet.es
>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>
>> El 17 jul 2019, a las 23:58, Bron Gondwana escribió:
>>
>> Oh awesome - thanks for letting us know :) This will save ellie spending
>> more time on it, though it's good to have tests for it anyway.
>>
>> Bron.
>>
>> On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
>>> Hi mates,
>>>
>>> Reproduced and properly located (*_not a Cyrus bug_*). We have to apologize
>>> for the generated noise. *_Cyrus replication works just fine_*.
>>>
>>> As I explained before, we fetch from Mysql the accounts each server has
>>> (instead of using -A) for no reason… just for historical reason…
>>>
>>> So, when we fetched the emails, as in very old (now) versions of Cyrus a
>>> conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o
>>> even older), we were doing the following in the Perl script :
>>>
>>> while (@fila = $query->fetchrow()) {
>>> $fila[0] =~ s/\./^/g;
>>> $email = $fila[0]."@".$fila[1];
>>> system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client -S
>>> xx.sarenet.es -v -u ".$email);
>>>
>>> This explained the existence of .seen and .sub files with “^” and Sieve
>>> dirs with ‘^’ too. That didn’t in fact explain, why they had non zero size.
>>> The reason of the non zero size (of files with ^ and ending in .sub and
>>> .seen and Sieve dirs), is that as we come from Cyrus versions 2.3 (and
>>> probably 2.4 perhaps do it) used the ‘^’ in the seen and sub filenames and
>>> Sieve names. So if you do a sync_client of a user with ‘^’ , even in 3.0 it
>>> copies you that file content of .sub and .seen from the master to the slave
>>> (because it exists! because it has copied with sync_client too but from a
>>> 2.3/2.4 version), even when they have not been used in a 3.0 version. Apart
>>> the own mail account content is always copied and sync_client doesn’t
>>> complain about it (whether you use ^ or a dot for the user).
>>>
>>> So mates, thanks a lot again for your time and I honestly think I’ll sleep
>>> better today :)
>>>
>>> We have just discovered these, so we wanted to clarify it.
>>>
>>> Thanks again,
>>> Regards,
>>>
>>>
>>> sarenet
>>> *Egoitz Aurrekoetxea*
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnológico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> ego...@sarenet.es
>>> www.sarenet.es
>>>
>>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>>>
>>>> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea
>>>> escribió:
>>>>
>>>> Anyway, not being able to reproduce it… working on reproducing….
>>>>
>>>>
>>>> sarenet
>>>> *Egoitz Aurrekoetxea*
>>>> Dpto. de sistemas
>>>> 944 209 470
>>>> Parque Tecnológico. Edificio 103
>>>> 48170 Zamudio (Bizkaia)
>>>> ego...@sarenet.es
>>>> www.sarenet.es
>>>>
>>>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>>>>
>>>>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea
>>>>> escribió:
>>>>>
>>>>
Re: Folder subscription issue
Oh awesome - thanks for letting us know :) This will save ellie spending more
time on it, though it's good to have tests for it anyway.
Bron.
On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
> Hi mates,
>
> Reproduced and properly located (*_not a Cyrus bug_*). We have to apologize
> for the generated noise. *_Cyrus replication works just fine_*.
>
> As I explained before, we fetch from Mysql the accounts each server has
> (instead of using -A) for no reason… just for historical reason…
>
> So, when we fetched the emails, as in very old (now) versions of Cyrus a
> conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o
> even older), we were doing the following in the Perl script :
>
> while (@fila = $query->fetchrow()) {
> $fila[0] =~ s/\./^/g;
> $email = $fila[0]."@".$fila[1];
> system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client -S
> xx.sarenet.es -v -u ".$email);
>
> This explained the existence of .seen and .sub files with “^” and Sieve dirs
> with ‘^’ too. That didn’t in fact explain, why they had non zero size. The
> reason of the non zero size (of files with ^ and ending in .sub and .seen and
> Sieve dirs), is that as we come from Cyrus versions 2.3 (and probably 2.4
> perhaps do it) used the ‘^’ in the seen and sub filenames and Sieve names. So
> if you do a sync_client of a user with ‘^’ , even in 3.0 it copies you that
> file content of .sub and .seen from the master to the slave (because it
> exists! because it has copied with sync_client too but from a 2.3/2.4
> version), even when they have not been used in a 3.0 version. Apart the own
> mail account content is always copied and sync_client doesn’t complain about
> it (whether you use ^ or a dot for the user).
>
> So mates, thanks a lot again for your time and I honestly think I’ll sleep
> better today :)
>
> We have just discovered these, so we wanted to clarify it.
>
> Thanks again,
> Regards,
>
>
> sarenet
> *Egoitz Aurrekoetxea*
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es
> www.sarenet.es
>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>
>> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea
>> escribió:
>>
>> Anyway, not being able to reproduce it… working on reproducing….
>>
>>
>> sarenet
>> *Egoitz Aurrekoetxea*
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnológico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> ego...@sarenet.es
>> www.sarenet.es
>>
>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>>
>>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea
>>> escribió:
>>>
>>> Perhaps mbname_userid it’s not the exact function… perhaps we could just
>>> call something like _append_extbuf() in this case but…. That’s what I was
>>> trying to explain… could that be?
>>>
>>>
>>> sarenet
>>> *Egoitz Aurrekoetxea*
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnológico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> ego...@sarenet.es
>>> www.sarenet.es
>>>
>>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>>>
>>>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea
>>>> escribió:
>>>>
>>>> mbname_userid
>>>
>>>
>>> Cyrus Home Page: http://www.cyrusimap.org/
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>
>>
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Bron Gondwana, CEO, Fastmail Pty Ltd
br...@fastmailteam.com
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Folder subscription issue
Sorry for not getting back to you yesterday. I was on my way back from vacation and had family commitments all last night. Regarding contribution - the very best thing to do for a case like this is to build Cassandane tests which isolate the issue :) I'll see if I can get that done today. Cheers, Bron. On Tue, Jul 16, 2019, at 02:34, Egoitz Aurrekoetxea wrote: > Hi Bron! > > Sorry for answering so late I had the thought I answered you before…. I’m > slightly stressed these days... > > I answer below in green bold for instance….. > > > sarenet > *Egoitz Aurrekoetxea* > Dpto. de sistemas > 944 209 470 > Parque Tecnológico. Edificio 103 > 48170 Zamudio (Bizkaia) > ego...@sarenet.es > www.sarenet.es > > Antes de imprimir este correo electrónico piense si es necesario hacerlo. > >> El 14 jul 2019, a las 14:36, Bron Gondwana escribió: >> >> On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote: >>> Good morning, >>> >>> In previous thread (with the title slightly incorrect) I talk about an >>> issue suffered some day with Sieve script and folder subscriptions. The >>> Sieve part, was related to the migration, so for the moment let’s forget >>> about it (for me at least) as it has never reproduced again since then. >> >> Sorry, I missed the previous thread. Did you mention which version of Cyrus >> you are running? There's clearly a bug and it would be good to know which >> version(s) it affects. > > > *I think I have specified all these details (including what the previous > thread was saying) in the answered mail this morning… I think it's all there… > the version was 3.0.8…. If is not all or you need something, please tell me > and I’ll try my bests for providing you all the info you need..* > > >> >>> About the folder subscription issue, I think I got something, at least a >>> close approximation... When a user causes in mua a rename mailbox (a rename >>> for a folder caused by a folder move in hierarchy), after the own rename, >>> if folders were subscribed “should” (for the "plain user" at least) become >>> subscribed in the new path. It seems that after a user rename in Cyrus the >>> new folder is automatically subscribed (even if no subscribe command is >>> sent by the mua). But this, does not cause in the replica (in the slave, if >>> SUB is not sent by the client) a sync_apply_changesub() or something like >>> entering in the “ move_subscription” condition in mboxlist_renamemailbox(), >>> and then, the folder is properly renamed but not subscribed in the slave. I >>> think this is what I’m suffering. Obviously, if after a rename the mua >>> sends a subscribe too, no issue is seen. I think the problem happens when a >>> mailbox rename happens and a SUB is not send later. >> >> Subscriptions aren't automatically renamed when a folder is renamed, but >> they should be automatically renamed for a user rename. Intermediate >> replicated states can be bit messy due to race conditions with replication, >> but I believe it should always wind up in the final correct state. If not, >> that's a bug for sure! > > *This tests are some days ago… I’m slightly confused now because I don’t > remember it exactly… * > >> >>> An example : >>> >>> The folder domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG >>> is moved (renamed) to trash. >>> >>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed >>> domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to >>> domain.com!user.parta^partb.Trash.XINGANG >>> May 16 16:16:50 mx6c imap[83976]: Rename: >>> domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> >>> domain.com!user.parta^partb.Trash.XINGANG >>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox >>> domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG >>> >>> In the master (sync), the folder in Trash is subscribed but in the slave it >>> is not, and remains subscribed the folder in the original location in the >>> “.sub” file. >> >> This might just be a matter of failing to sync_log the subscription in that >> codepath. Hmm... >> >> But there is a question of whether we should even be renaming the >> subscription - RFC3501 is a little silent on this issue, but it does say in >> a couple of places that the server MUST NOT remove a subscription even if >> the mailbox with that name doesn't exist, which makes rena
Re: Folder subscription issue
.XINGANG >>>> to domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG >>>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com >>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG >>>> -> domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG >>>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com >>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG >>>> >>>> In the master (sync), the folder in Trash is subscribed but in the slave >>>> it is not, and remains subscribed the folder in the original location in >>>> the “.sub” file. >>>> >>>> A diff between the master (replication client) .sub file and the slave’s >>>> one is (mx5 is the master, the client and 6 the slave): >>>> >>>> --- subscripciones-mx5c/parta.partb.sub2019-07-10 08:47:29.0 +0200 >>>> +++ subscripciones-mx6c/parta.partb.sub2019-07-10 08:48:08.0 +0200 >>>> >>>> +domain.com >>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG >>>> -domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG >>>> >>>> Perhaps a move_subscription to one or sync_apply_changesub should be >>>> forced in order to fix it?. I have seen issues with Outlook 2016 and >>>> Thunderbird… both mua… perhaps by RFC they should send the SUB command >>>> but… it’s the only theory I could arrive to… I have a ton more cases >>>> like this one.. Data gets properly handled but subscriptions have this >>>> issue (perhaps we could say is a mua issue but….).. >>> <0x197B06994D105B45.asc> >> >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Folder subscription issue
On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote: > Good morning, > > In previous thread (with the title slightly incorrect) I talk about an issue > suffered some day with Sieve script and folder subscriptions. The Sieve part, > was related to the migration, so for the moment let’s forget about it (for me > at least) as it has never reproduced again since then. Sorry, I missed the previous thread. Did you mention which version of Cyrus you are running? There's clearly a bug and it would be good to know which version(s) it affects. > About the folder subscription issue, I think I got something, at least a > close approximation... When a user causes in mua a rename mailbox (a rename > for a folder caused by a folder move in hierarchy), after the own rename, if > folders were subscribed “should” (for the "plain user" at least) become > subscribed in the new path. It seems that after a user rename in Cyrus the > new folder is automatically subscribed (even if no subscribe command is sent > by the mua). But this, does not cause in the replica (in the slave, if SUB is > not sent by the client) a sync_apply_changesub() or something like entering > in the “ move_subscription” condition in mboxlist_renamemailbox(), and then, > the folder is properly renamed but not subscribed in the slave. I think this > is what I’m suffering. Obviously, if after a rename the mua sends a subscribe > too, no issue is seen. I think the problem happens when a mailbox rename > happens and a SUB is not send later. Subscriptions aren't automatically renamed when a folder is renamed, but they should be automatically renamed for a user rename. Intermediate replicated states can be bit messy due to race conditions with replication, but I believe it should always wind up in the final correct state. If not, that's a bug for sure! > An example : > > The folder domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is > moved (renamed) to trash. > > May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed > domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to > domain.com!user.parta^partb.Trash.XINGANG > May 16 16:16:50 mx6c imap[83976]: Rename: > domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> > domain.com!user.parta^partb.Trash.XINGANG > May 16 16:16:50 mx6c imap[83976]: Deleted mailbox > domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG > > In the master (sync), the folder in Trash is subscribed but in the slave it > is not, and remains subscribed the folder in the original location in the > “.sub” file. This might just be a matter of failing to sync_log the subscription in that codepath. Hmm... But there is a question of whether we should even be renaming the subscription - RFC3501 is a little silent on this issue, but it does say in a couple of places that the server MUST NOT remove a subscription even if the mailbox with that name doesn't exist, which makes renaming subscriptions a bit unclear. I'll check in with the authors of draft-ietf-extra-imap4rev2 and ask for this to be clarified next time around! > A diff between the master (replication client) .sub file and the slave’s one > is (mx5 is the master, the client and 6 the slave): > > --- subscripciones-mx5c/parta.partb.sub 2019-07-10 08:47:29.0 +0200 > +++ subscripciones-mx6c/parta.partb.sub 2019-07-10 08:48:08.0 +0200 > > +domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG > -domain.com!user.parta^partb.Trash.XINGANG > > Perhaps a move_subscription to one or sync_apply_changesub should be forced > in order to fix it?. I have seen issues with Outlook 2016 and Thunderbird… > both mua… perhaps by RFC they should send the SUB command but… it’s the only > theory I could arrive to… I have a ton more cases like this one.. Data gets > properly handled but subscriptions have this issue (perhaps we could say is a > mua issue but….).. Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: thoughts about conversation db and cyrus [faild to rename deleted mailbox, or how conversation db sucks again]
ill supported? > > 2. What are to pros and cons for the supported search engines? > > 3. Should, or to which extend should, the search engines work without > conversations db? > Or is enabling the conversations db a new requirement for some/all > search engines? > > 4. Is the conversations db murder aware? And how do shared folders > (one user shared one > of his folders with other users) on the same server/cross server > affect search results and performances > > 5. What is stored in the conversations db? > > https://www.cyrusimap.org/dev/imap/concepts/deployment/databases.html#conversations-userid-conversations > is incomplete as conversations db also contains hashes of mime parts. > > 6. Which Information is affected by conversations_expire_days > > TLDR; I like cyrus, but there is some work to do regarding to > conversation db and search engines, > in the field of documentation, code testing and feature interaction > > > ---- > M.Menge Tel.: (49) 7071/29-70316 > Universität Tübingen Fax.: (49) 7071/29-5912 > Zentrum für Datenverarbeitung mail: > michael.me...@zdv.uni-tuebingen.de > Wächterstraße 76 > 72074 Tübingen > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: faild to rename deleted mailbox, or how conversation db sucks again
This isn't quite renaming the folder, and you'll lose per-user seen state, but you can create a new folder and, using the admin login, copy the messages: . RENAME deleted.user.brongfoo.xx.5c7c8...@fastmailtest.com user.brongfoo...@fastmailtest.com . NO Invalid mailbox name . SELECT deleted.user.brongfoo.xx.5c7c8...@fastmailtest.com * OK [CLOSED] Ok * 2 EXISTS [...] . CREATE user.brongfoo...@fastmailtest.com . OK [MAILBOXID (648d3c90-6c68-49cf-ac2b-49c2cfb9ee8b)] Completed . COPY 1:* user.brongfoo...@fastmailtest.com . OK [COPYUID 1551666806 1:2 1:2] Completed . SELECT user.brongfoo...@fastmailtest.com * OK [CLOSED] Ok * 2 EXISTS * 2 RECENT On Mon, Mar 4, 2019, at 13:25, ellie timoney wrote: > On Sat, Mar 2, 2019, at 2:54 AM, Michael Menge wrote: > > Is there an other way than disabling > > - conversation db, > > - restoring the folder, > > - enabeling conversation db again (so that squat files are used for > > header searches), > > - rebuilding conversation db for all users (because there will be some > > new mails between > > disabling and re-enabling conversation db). > > If you can arrange for your MX to hold the incoming mail during the period > where conversations is disabled, that would at least avoid needing to rebuild > conversationsdb for everyone again afterwards? You might be able to do this > by temporarily disabling the lmtpd service, but I'm not sure (it would depend > on how your MX handles that) > > Cheers, > > ellie > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: another problem with conversations db
On Wed, Feb 13, 2019, at 00:39, Michael Menge wrote:
> Hi Bron,
>
> sorry, i had to rearrange some quotes to put them my answers in a more
> meaningful order.
>
>
> Quoting Bron Gondwana :
>
> >> >> The file was already at
> >> /srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.
>
> I was able to fix these problems with reconstruct, and the didn't
> reappear till now.
> Also there where other accounts which had IOERRORS regarding the
> conversation db,
> with no cyr_expire archive errors, so i believe that these problems
> are not related.
>
> I tried rebuilding the conversation db for the accounts with errors,
> but some other
> accounts will show up with errors some time later. I counldn't find a
> some thing in
> common jet.
OK. That's hard to disagnore from remotely :(
>
> >> >> > Anyway, I don't think that would break anything.
> >> >> >
> >> >> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
> >> >> > metapartition_files: header index cache expunge squat annotations
> >> >> > lock dav archivecache
> >> >> >
> >> >> > Ooh, I haven't tested having cache and archivecache on the same
> >> >> > location. That's really interesting. Again, I'd be in favour of
> >> >> > separation here, give them different paths. That might be tricky
> >> >> > with ssd though, the way this is laid out. I assume you have some
> >> >> > kind of symlink farm going on?
> >> >> >
> >> >>
> >> >> I didn't know that there could be a problem with cache and archivecache.
> >> >> At the time we decided on the configuration for cyrus 3.0 I looked at
> >> >> the
> >> >> imapd.conf man page and for metapartition_files decided that I want all
> >> >> meta files on the ssd storage. There was no indication in the man page
> >> >> that there could be a problem.
> >> >
> >> > Fair. I'd have to test that to see if it works correctly. I would
> >> > hope so, but I haven't tested that configuration. This is the
> >> > downside with having lots of different ways to do things!
> >> >
> >> >> How do I separate location of archivecache from the other
> >> >> metapartition path?
> >> >> And fix the cache and archivecache files?
> >> >
> >> > This I don't know a good answer for. I will test if having the same
> >> > path for cache and archivecache could fail. I THINK that I made the
> >> > code safe for it, but I'm not sure that it's been tested.
> >> >
> >> >> No there is no sysmlink farm. We have mounted different iSCSI volumes to
> >> >> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be
> >> >
> >> > Right. That makes sense.
>
> Did you have time to look into the cache/archivecache situation jet?
Yes, I've looked at the code!
in mailbox_archive():
/* got a new cache record to write */
if (differentcache)
{
dirtycache = 1;
copyrecord.cache_offset = 0;
if (mailbox_append_cache(mailbox, ))
continue;
}
And the code for differentcache is:
char *spoolcache = xstrdup(mailbox_meta_fname(mailbox, META_CACHE));
char *archivecache = xstrdup(mailbox_meta_fname(mailbox, META_ARCHIVECACHE));
int differentcache = strcmp(spoolcache, archivecache);
So it looks like the answer is cache/archivecache is fine. It is not your
problem.
>
> >> > Right! I do wonder if there are some bugs in 3.0.x which are fixed
> >> > on master around delivery to archive partition. We definitely had
> >> > bugs on master, but I thought they were newly introduced on master
> >> > as well, which is why the fixes weren't backported. But if you're
> >> > having files be in the wrong location, maybe there are bugs on 3.0.x
> >> > as well.
>
> Are all fixes from master backported to 3.0?
Unfortunately it's hard to tell, because many of the fixes on master are fixes
to bugs that were only introduced on master, and some bugs on 3.0 we just say
"the fix is so invasive that it's basically just backporting 90% of master,
which is pointless for a stable release".
> Is the new Commit "I will try your new commits regarding CID" related to the
> "IOERROR: conversations_audit on load:" and "IOERROR:
> conversations_audit on store"?
Shouldn't be. It just means we store the G keys regardless of whether the
record has a CID.
> I will try your new commits in the next days on my test s
Re: Removing email from Xapian tier databases
Excellent - I hope you grabbed both that commit and the one afterwards where I fixed the order of CID parsing. Actually, it might not be as big a deal on 3.0, but not calculating the CID first did break one JMAP case on future. Cheers, Bron. On Tue, Feb 12, 2019, at 02:22, Sebastian Hagedorn wrote: > Thanks! I rolled my own RPM with that patch, and I can confirm that it > works. > > --On 11. Februar 2019 um 09:12:14 -0500 Bron Gondwana > wrote: > > > Yep, it's fixed in git now, so the next release will automatically create > > G keys for messages, even if they don't have a threadid! > > > > Bron. > > > > On Mon, Feb 11, 2019, at 21:30, Sebastian Hagedorn wrote: > >> So running ctl_conversationsdb -z followed by -b would assign thread ids > >> to those messages? Because it works when I do that. Clearly this is an > >> edge case, but IMO it should be handled somehow other than silently > >> failing ;-) > >> > >> --On 11. Februar 2019 um 05:16:47 -0500 Bron Gondwana > >> wrote: > >> > >> > That sounds like the source messages have no thread id, and hence they > >> > aren't being stored. > >> > > >> > This is an interesting question actually, should we still store G keys > >> > for messages without thread identifier (CID)? > >> > > >> > Bron. > >> > > >> > On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote: > >> >> Hi Bron, > >> >> > >> >> --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana > >> >> wrote: > >> >> > >> >> > The data in conversations.db is added and removed in real time as > >> >> > messages are appended and updated in the cyrus.index. > >> >> > >> >> do you know why that does not seem to happen when using the "old" sync > >> >> protocol for replication? > >> >> > >> >> <https://github.com/cyrusimap/cyrus-imapd/issues/2376> > >> -- > -- > .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:. > .:.Regionales Rechenzentrum (RRZK).:. > .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Removing email from Xapian tier databases
It's definitely safe to have one rolling mode writing and one repacking. I
wouldn't run multiple repacks in parallel, as they can wind up doing duplicate
work (though the end result should always be correct and safe).
Here's what we run:
# Any time the disk gets over 50%, compress -o single down to data
13 * * * * [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a
-o -d 50 temp data' %]
# Copy the temporary search databases down to data during the week
43 1 * * 1,2,3,4,5,6 [% INCLUDE cronjob
c='/home/mod_perl/hm/scripts/xapian_compact.pl -a temp,meta data' %]
# Sundays repack the entire data directory
43 1 * * 0 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a
temp,meta,data data' %]
# Late on Sundays, pack any oversized data directories down to archive
0 15 * * 0 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_archive.pl
-a' %]
And here's the interesting logic. In xapian_compact.pl:
if ($Opts{d}) {
my $Path = $Slot->SearchPath();
my $Usage = df($Path);
my $RunUsage = df("/run/cyrus");
return Process::Status->new(0) if ($Usage->{per} < $Opts{d} and
$RunUsage->{per} < $Opts{d});
}
my @args = (-z => $dest, -t => $src);
push @args, '-v' if $Opts{v};
push @args, '-o' if $Opts{o};
push @args, '-F' if $Opts{F};
push @args, '-X' if $Opts{X};
push @args, ('-T' => $Opts{T}) if $Opts{T};
push @args, ('-u' => $Opts{u}) if $Opts{u};
my %RunOpts = (
PrintOutput => 1,
);
$RunOpts{Nice} = 1 unless $Opts{N};
$RunOpts{Daemon} = 1 if $Opts{D};
$0 = "xapian_compact: $SN";
$Slot->RunCommand(\%RunOpts, 'squatter', @args);
And in xapian_archive.pl:
my $Percent = $Opts{P} || 20;
[...]
foreach my $user (sort keys %$DataUsage) {
my $au = $ArchiveUsage->{$user} || 1;
my $du = $DataUsage->{$user} || 1;
if ($du < 5000) {
print "Too small $user ($du)\n";
next;
}
my $This = int($du * 100 / $au);
if ($This < $Percent) {
print "Not enough dirty $user: ($du, $au)\n";
next;
}
print "Recompacting $user: ($du, $au)\n";
my @args = (-z => 'archive', -t => 'data,archive');
[...]
In summary, repack data down to archive if data is more than 1/5 size of
existing archive. So each of these scripts is a wrapper around squatter to help
it run automatically.
Bron.
On Mon, Feb 11, 2019, at 21:55, Egoitz Aurrekoetxea wrote:
> Now I'm noticing for instance, for moving data between Xapian databases.. you
> need to launch something like :
>
> sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -v -z
> archive -t temp,meta,data,archive -u user/ego...@sarenet.es
>
>
> perhaps would be better to do :
> sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf _*-F*_ -v
> -z archive -t temp,meta,data,archive -u user/ego...@sarenet.es
> But then, having two Squatter processes running at same time, one for rolling
> mode and one for moving/repacking data, should not be an issue?.
>
>
> Thanks mates!!
>
> ---
>
> sarenet
> *Egoitz Aurrekoetxea*
> Departamento de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es
> www.sarenet.es
>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>
> El 11-02-2019 11:22, Egoitz Aurrekoetxea escribió:
>> Hi Bron,
>>
>> So, it would be interesting to run once a day... for instance in cyrus.conf
>> in events section :
>> repack_xapian cmd="squatter -F" at=0200
>> Is it needed top stop the other rolling Squatter we run, in same cyrus.conf
>> as :
>> START {
>> # do not delete this entry!
>> recover cmd="ctl_cyrusdb -r"
>>
>> squatter cmd="squatter -R"
>> }
>>
>> Thank you so much for all the clarifications mate :) really :)
>>
>> Cheers!
>> ---
>>
>> sarenet
>> *Egoitz Aurrekoetxea*
>> Departamento de sistemas
>> 944 209 470
>> Parque Tecnológico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> ego...@sarenet.es
>> www.sarenet.es
>>
>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>>
>> El 11-02-2019 10:23, Bron Gondwana escribió:
>>> Conversations.db is an index over lots of interesting bits of the message,
>>> but the key part that's used by Xapian is the mapping from G key (aka:
>>> GUID, aka: sha1 of the message RFC822 data) to individual email. It's used
>>> for deduplication and for mapping from results to messages.
>>>
>>> The data in conversations.db is added and removed in real time as messages
>>> are appended and updated in the cyrus.index.
>>>
>>>
Re: Removing email from Xapian tier databases
Yep, it's fixed in git now, so the next release will automatically create G keys for messages, even if they don't have a threadid! Bron. On Mon, Feb 11, 2019, at 21:30, Sebastian Hagedorn wrote: > So running ctl_conversationsdb -z followed by -b would assign thread ids to > those messages? Because it works when I do that. Clearly this is an edge > case, but IMO it should be handled somehow other than silently failing ;-) > > --On 11. Februar 2019 um 05:16:47 -0500 Bron Gondwana > wrote: > > > That sounds like the source messages have no thread id, and hence they > > aren't being stored. > > > > This is an interesting question actually, should we still store G keys > > for messages without thread identifier (CID)? > > > > Bron. > > > > On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote: > >> Hi Bron, > >> > >> --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana > >> wrote: > >> > >> > The data in conversations.db is added and removed in real time as > >> > messages are appended and updated in the cyrus.index. > >> > >> do you know why that does not seem to happen when using the "old" sync > >> protocol for replication? > >> > >> <https://github.com/cyrusimap/cyrus-imapd/issues/2376> > -- > .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:. > .:.Regionales Rechenzentrum (RRZK).:. > .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Removing email from Xapian tier databases
That sounds like the source messages have no thread id, and hence they aren't being stored. This is an interesting question actually, should we still store G keys for messages without thread identifier (CID)? Bron. On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote: > Hi Bron, > > --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana > wrote: > > > The data in conversations.db is added and removed in real time as > > messages are appended and updated in the cyrus.index. > > do you know why that does not seem to happen when using the "old" sync > protocol for replication? > > <https://github.com/cyrusimap/cyrus-imapd/issues/2376> > > Cheers, > Sebastian > -- > .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:. > .:.Regionales Rechenzentrum (RRZK).:. > .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Removing email from Xapian tier databases
Conversations.db is an index over lots of interesting bits of the message, but the key part that's used by Xapian is the mapping from G key (aka: GUID, aka: sha1 of the message RFC822 data) to individual email. It's used for deduplication and for mapping from results to messages. The data in conversations.db is added and removed in real time as messages are appended and updated in the cyrus.index. The data in the xapian databases on the other hand is append only - so you can wind up with hits that no longer map to existing emails. The way to solve that is with a xapian repack that filters messages - which can be done using the -F flag to squatter. Cheers, Bron. On Sat, Feb 9, 2019, at 23:04, Egoitz Aurrekoetxea wrote: > Good morning, > > As far as I understood, for Xapian you first create it's conversation > database in order to work. Later you create database(s) for each mailbox > where Xapian can search in. You can move data between them, new mails become > indexed for instance Squatter in rolling mode... that's ok... and understood > I think. I was wondering, what happens when mail indexed in the archive > database in removed and then does not exist any more in the database... does > Squatter rolling log manage that too?. > > By the way. I was wondering if mail gets indexed in the tier databases (for > instance in Fastmail in temp, meta, data, archine...) what's the role or > function of conversations databases you create with ctl_conversationsdb -b -r > ?. > > Cheers! > -- > > sarenet > *Egoitz Aurrekoetxea* > Departamento de sistemas > 944 209 470 > Parque Tecnológico. Edificio 103 > 48170 Zamudio (Bizkaia) > ego...@sarenet.es > www.sarenet.es > > Antes de imprimir este correo electrónico piense si es necesario hacerlo. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Sieve script not working
On Tue, Feb 5, 2019, at 06:20, Andrea Venturoli wrote:
> On 2/2/19 8:25 PM, Bron Gondwana wrote:
> > What is being written to syslog by your lmtpd?
>
> Absolutely nothing (apart from transient errors on the rare occasions
> when I reboot the server and sendmail comes up before imapd).
>
> Is this log to be enabled somehow?
>
No, there should be something in the syslog if sieve errors:
2019-02-06T12:31:26-05:00 imap38 sloti38d1t08/lmtp[2916700]: sieve runtime
error for f...@fastmail.com id : Fileinto
(Testing): Mailbox does not exist
> > There's also a sieve-test binary that you can build in the Cyrus source
> > code. I don't think it's built by default on most platforms though, we
> > build it specially at FM. It takes a script and a raw email and spits
> > out a list of actions.
>
> I don't have this (FreeBSD 11.2).
> AFAICT the port doesn't disable this (in configure) or leave it behind
> when installing.
> I tried searching for this binary between "make" and "make install"
> phases (*), but I didn't find it (or any source named anything like it).
> What's the procedure to compile it?
>
> (*)
> > # cd work/cyrus-imapd-2.5.12/
> > # find .|grep -i "sieve-test"
> > # find . -type f -exec grep -i "sieve-test" "{}" ";"
>
Yeah, it's just called test.c in the sieve directory. Here's our buildscript
code:
make install DESTDIR=\$(CURDIR)/debian/$basename
make install-binsymlinks DESTDIR=\$(CURDIR)/debian/$basename
/bin/bash ./libtool --mode=install install -o root -m 755 sieve/test
\$(PWD)/debian/$basename/$basedir/bin/sieve-test
install -o root -m 755 tools/rehash debian/$basename/$basedir/bin/rehash
install -o root -m 755 tools/mkimap debian/$basename/$basedir/bin/mkimap
So the binary will probably be sieve/test from a standard build
Oh yeah, you might be interested in the config options we build with too :)
./configure --without-krb --with-perl=/usr/bin/perl --enable-silent-rules
--enable-http --enable-calalarmd --enable-idled --with-extraident=$tag
--prefix=/$basedir --with-lmdb --with-zlib --without-snmp --enable-replication
--enable-xapian --enable-jmap --enable-backup
XAPIAN_CONFIG=/usr/local/$CYRUSLIBS/bin/xapian-config-1.5
$LEXFIX
make -j 8 all CFLAGS="-g -fPIC -W -Wall -Werror -fstack-protector-all"
Cheers,
Bron.
--
Bron Gondwana, CEO, FastMail Pty Ltd
br...@fastmailteam.com
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Fatal error: Exists wrong 513 512 483 17057
Awesome - yes, the IOERROR messages are because files it expected to find weren't there. It's frustrating that reconstruct isn't robust enough to bring a slightly bogus cyrus.index back from the dead, but the end result is a working mailbox with a fully correct cyrus.index file :) The only thing that may have happened is that some emails which were previously deleted in the TareasCron folder came back from the dead - and of course all the flags on those messages will have gone away too. But the mailbox will work correctly now. Cheers, Bron. On Tue, Feb 5, 2019, at 03:26, Daniel Bareiro wrote: > Hi, Bron. Thanks for your reply. > > On 4/2/19 05:45, Bron Gondwana wrote: > > > Oh how frustrating... > > >> I tried using 'reconstruct' as you suggested and this was the result: > >> > >> -- > >> # /usr/lib/cyrus/bin/reconstruct -r user.admin > >> user.admin > >> user.admin.Backups > >> user.admin.Drafts > >> user.admin.Sent > >> user.admin.TareasCron: record corrupted 13 (maybe uid 9836) > >> user.admin.TareasCron: Mailbox format corruption detected Failed to > >> reconstruct mailbox > >> user.admin.Trash > >> -- > > > This clearly isn't syslogging enough! Can you check the permissions on > > the underlying filesystem! Also, try running reconstruct with -G, > > though it shouldn't make a difference. > > The previous thing was the output that I was obtaining in the console > when executing 'reconstruct'. This is what I get in the syslog when > executing the command with the '-G' flag: > > -- > Feb 4 11:48:29 mail cyrus/reconstruct[22727]: reconstructing user.admin > Feb 4 11:48:47 mail cyrus/reconstruct[22727]: mailbox: longlock > user.admin for 17.8 seconds > Feb 4 11:48:47 mail cyrus/reconstruct[22727]: Repacking mailbox > user.admin version 13 > Feb 4 11:48:48 mail cyrus/reconstruct[22727]: reconstructing > user.admin.Backups > Feb 4 11:48:50 mail cyrus/reconstruct[22727]: mailbox: longlock > user.admin.Backups for 2.5 seconds > Feb 4 11:48:50 mail cyrus/reconstruct[22727]: Repacking mailbox > user.admin.Backups version 13 > Feb 4 11:48:50 mail cyrus/reconstruct[22727]: reconstructing > user.admin.Drafts > Feb 4 11:48:51 mail cyrus/reconstruct[22727]: reconstructing > user.admin.Sent > Feb 4 11:48:51 mail cyrus/reconstruct[22727]: reconstructing > user.admin.TareasCron > -- > > But in essence the console shows the same output as without adding '-G': > > -- > # /usr/lib/cyrus/bin/reconstruct -G -r user.admin > user.admin > user.admin.Backups > user.admin.Drafts > user.admin.Sent > user.admin.TareasCron: record corrupted 13 (maybe uid 9836) > user.admin.TareasCron: Mailbox format corruption detected Failed to > reconstruct mailbox > user.admin.Trash > -- > > >> Just 'TasksCron' is the folder that I can not open. What do you suggest > >> to do in this case? > > > I'd take a backup, and then delete the cyrus.index file in that folder > > and run reconstruct again... That's a pretty extreme approach, but it > > should work. > > Let's try the pretty extreme approach... :-D > > -- > Feb 4 12:04:43 mail cyrus/reconstruct[22990]: reconstructing user.admin > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing > user.admin.Backups > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing > user.admin.Drafts > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing > user.admin.Sent > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing > user.admin.TareasCron > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: IOERROR: opening index > user.admin.TareasCron: System I/O error > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: create new mailbox > user.admin.TareasCron > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: failed to read index > header for user.admin.TareasCron > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid > 9837 found - adding > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid > 9838 found - adding > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid > 9839 found - adding > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid > 9840 found - adding > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid > 9841 found - adding > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid > 9842 found - adding > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid > 9843 found - adding > Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid > 9844 found
Re: another problem with conversations db
On Mon, Feb 4, 2019, at 22:00, Michael Menge wrote:
>
> Quoting Bron Gondwana :
>
> > On Mon, Feb 4, 2019, at 20:21, Michael Menge wrote:
> >> Hi,
> >>
> >> Quoting Bron Gondwana :
> >>
> >> > Hi Michael,
> >> >
> >> > Sorry about the delay in looking at this - I was mad crazy busy
> >> > getting ready to go overseas. At Fosdem now, about to give a talk
> >> > about JMAP!
> >> >
> >> > OK, let's start with the things that give me a little bit of hives...
> >> >
> >> > configdirectory: /srv/cyrus-be
> >> > partition-default: /srv/cyrus-be
> >> > partition-ssd: /srv/cyrus-be/ssd-part
> >> >
> >> > Ouch. There's a couple of things I wouldn't do there - having the
> >> > partition be the same as the config directory, and having a separate
> >> > partition be a subdirectory of a partition. They're both asking for
> >> > trouble. I would probably lay my system out like:
> >> >
> >> > configdirectory: /srv/cyrus-be/conf
> >> > partition-default: /srv/cyrus-be/default-part
> >> > partition-ssd: /srv/cyrus-be/ssd-part
> >> >
> >>
> >> partition-default isn't used any more. To use the metapartition we moved
> >> all accounts form the default partition to the ssd partition which is the
> >> the new defaultpartition ("defaultpartition: ssd")
> >
> > Right - that makes sense.
> >
> >> > And then each tree would only have one type of thing in it.
> >> >
> >> > Anyway, I don't think that would break anything.
> >> >
> >> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
> >> > metapartition_files: header index cache expunge squat annotations
> >> > lock dav archivecache
> >> >
> >> > Ooh, I haven't tested having cache and archivecache on the same
> >> > location. That's really interesting. Again, I'd be in favour of
> >> > separation here, give them different paths. That might be tricky
> >> > with ssd though, the way this is laid out. I assume you have some
> >> > kind of symlink farm going on?
> >> >
> >>
> >> I didn't know that there could be a problem with cache and archivecache.
> >> At the time we decided on the configuration for cyrus 3.0 I looked at the
> >> imapd.conf man page and for metapartition_files decided that I want all
> >> meta files on the ssd storage. There was no indication in the man page
> >> that there could be a problem.
> >
> > Fair. I'd have to test that to see if it works correctly. I would
> > hope so, but I haven't tested that configuration. This is the
> > downside with having lots of different ways to do things!
> >
> >> How do I separate location of archivecache from the other
> >> metapartition path?
> >> And fix the cache and archivecache files?
> >
> > This I don't know a good answer for. I will test if having the same
> > path for cache and archivecache could fail. I THINK that I made the
> > code safe for it, but I'm not sure that it's been tested.
> >
> >> No there is no sysmlink farm. We have mounted different iSCSI volumes to
> >> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be
> >
> > Right. That makes sense.
> >
> >> > Otherwise it all looks OK. Are you getting other IOERRORs in your
> >> > log files which could show things aborting? It really looks like
> >> > your conversations DB is getting out of sync due to other failures.
> >>
> >> I found a few instances of 3 related errors.
> >>
> >> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
> >> /srv/cyrus-be/ssd-part/L/user//2185.: No such file or directory
> >> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
> >> /srv/cyrus-be/ssd-part/L/user//2185.: No such file or directory
> >> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive
> >> user. 2185 failed to copyfile
> >> (/srv/cyrus-be/ssd-part/L/user//2185. =>
> >> /srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.): Unknown code
> >> 255
> >
> >
> > Ouch. Yeah, that could have been caused by a bug in delivery, and
> > would definitely cause conversations DB corruption if the index file
> > was updated but the conversations DB wasn't or vice versa.
> >
> >> The file was already at
> >> /srv
Re: another problem with conversations db
On Mon, Feb 4, 2019, at 20:21, Michael Menge wrote:
> Hi,
>
> Quoting Bron Gondwana :
>
> > Hi Michael,
> >
> > Sorry about the delay in looking at this - I was mad crazy busy
> > getting ready to go overseas. At Fosdem now, about to give a talk
> > about JMAP!
> >
> > OK, let's start with the things that give me a little bit of hives...
> >
> > configdirectory: /srv/cyrus-be
> > partition-default: /srv/cyrus-be
> > partition-ssd: /srv/cyrus-be/ssd-part
> >
> > Ouch. There's a couple of things I wouldn't do there - having the
> > partition be the same as the config directory, and having a separate
> > partition be a subdirectory of a partition. They're both asking for
> > trouble. I would probably lay my system out like:
> >
> > configdirectory: /srv/cyrus-be/conf
> > partition-default: /srv/cyrus-be/default-part
> > partition-ssd: /srv/cyrus-be/ssd-part
> >
>
> partition-default isn't used any more. To use the metapartition we moved
> all accounts form the default partition to the ssd partition which is the
> the new defaultpartition ("defaultpartition: ssd")
Right - that makes sense.
> > And then each tree would only have one type of thing in it.
> >
> > Anyway, I don't think that would break anything.
> >
> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
> > metapartition_files: header index cache expunge squat annotations
> > lock dav archivecache
> >
> > Ooh, I haven't tested having cache and archivecache on the same
> > location. That's really interesting. Again, I'd be in favour of
> > separation here, give them different paths. That might be tricky
> > with ssd though, the way this is laid out. I assume you have some
> > kind of symlink farm going on?
> >
>
> I didn't know that there could be a problem with cache and archivecache.
> At the time we decided on the configuration for cyrus 3.0 I looked at the
> imapd.conf man page and for metapartition_files decided that I want all
> meta files on the ssd storage. There was no indication in the man page
> that there could be a problem.
Fair. I'd have to test that to see if it works correctly. I would hope so, but
I haven't tested that configuration. This is the downside with having lots of
different ways to do things!
> How do I separate location of archivecache from the other metapartition path?
> And fix the cache and archivecache files?
This I don't know a good answer for. I will test if having the same path for
cache and archivecache could fail. I THINK that I made the code safe for it,
but I'm not sure that it's been tested.
> No there is no sysmlink farm. We have mounted different iSCSI volumes to
> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be
Right. That makes sense.
> > Otherwise it all looks OK. Are you getting other IOERRORs in your
> > log files which could show things aborting? It really looks like
> > your conversations DB is getting out of sync due to other failures.
>
> I found a few instances of 3 related errors.
>
> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
> /srv/cyrus-be/ssd-part/L/user//2185.: No such file or directory
> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
> /srv/cyrus-be/ssd-part/L/user//2185.: No such file or directory
> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive
> user. 2185 failed to copyfile
> (/srv/cyrus-be/ssd-part/L/user//2185. =>
> /srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.): Unknown code
> 255
Ouch. Yeah, that could have been caused by a bug in delivery, and would
definitely cause conversations DB corruption if the index file was updated but
the conversations DB wasn't or vice versa.
> The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.
Right! I do wonder if there are some bugs in 3.0.x which are fixed on master
around delivery to archive partition. We definitely had bugs on master, but I
thought they were newly introduced on master as well, which is why the fixes
weren't backported. But if you're having files be in the wrong location, maybe
there are bugs on 3.0.x as well.
Do you have the syslog lines at the time that email was delivered?
Bron.
--
Bron Gondwana
br...@fastmail.fm
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Fatal error: Exists wrong 513 512 483 17057
Oh how frustrating... On Mon, Feb 4, 2019, at 08:48, Daniel Bareiro wrote: > Hi, Bron. Thanks for your reply. > > On 3/2/19 08:33, Bron Gondwana wrote: > > > Wow. A reconstruct should fix that. What version are you running? > > I tried using 'reconstruct' as you suggested and this was the result: > > -- > # /usr/lib/cyrus/bin/reconstruct -r user.admin > user.admin > user.admin.Backups > user.admin.Drafts > user.admin.Sent > user.admin.TareasCron: record corrupted 13 (maybe uid 9836) > user.admin.TareasCron: Mailbox format corruption detected Failed to > reconstruct mailbox This clearly isn't syslogging enough! Can you check the permissions on the underlying filesystem! Also, try running reconstruct with -G, though it shouldn't make a difference. > user.admin.Trash > -- > > Just 'TasksCron' is the folder that I can not open. What do you suggest > to do in this case? I'd take a backup, and then delete the cyrus.index file in that folder and run reconstruct again... That's a pretty extreme approach, but it should work. > I'm using Cyrus 2.5.10-3 on Debian Stretch. A few days ago I upgraded > the entire operating system with Debian Jessie and Cyrus 2.4.17. That upgrade should be fine, and 2.5.10 should be fine as well. I just poked around in the code there to see how the reconstruct might be failing, and there's not enough information to know what's happening. Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Fatal error: Exists wrong 513 512 483 17057
Wow. A reconstruct should fix that. What version are you running?
On Sun, Feb 3, 2019, at 02:11, Daniel Bareiro wrote:
> Hi!
>
> Several days ago I am trying to solve this issue but without success.
> When opening a folder inside a mailbox through Thunderbird, I get this
> message: "Server [servername] has disconnected. The server may have gone
> down or there may be a network problem".
>
> In that access attempt the log shows an entry like this:
>
> --
> Feb 2 21:55:38 mail cyrus/imaps[6510]: Fatal error: Exists wrong 513
> 512 483 17057
> --
>
> Maybe the index is corrupt and has some duplicate message?
>
> Investigating on the web I found this reference [1]:
>
> --
> /* make sure we don't overflow the memory we mapped */
> if (msgno >= state->mapsize) {
> char buf[2048];
> sprintf(buf, "Exists wrong %u %u %u %u", msgno,
> state->mapsize, mailbox->i.exists, mailbox->i.num_records);
> fatal(buf, EC_IOERR);
> }
> --
>
> I would like to know what you think. Is it possible that this is a
> problem with the index for this mailbox? If so, do you recommend some
> way to regenerate it?
>
>
> Thanks in advance for your reply.
>
> Kind regards,
> Daniel
>
> [1]
> https://forge.bluemind.net/stash/projects/BM/repos/bm-cyrus-imapd/browse/imap/index.c?at=c196236d937608270283c681ae27f1bb00265b62
>
>
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
> *Attachments:*
> * signature.asc
--
Bron Gondwana
br...@fastmail.fm
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: another problem with conversations db
Hi Michael, Sorry about the delay in looking at this - I was mad crazy busy getting ready to go overseas. At Fosdem now, about to give a talk about JMAP! OK, let's start with the things that give me a little bit of hives... configdirectory: /srv/cyrus-be partition-default: /srv/cyrus-be partition-ssd: /srv/cyrus-be/ssd-part Ouch. There's a couple of things I wouldn't do there - having the partition be the same as the config directory, and having a separate partition be a subdirectory of a partition. They're both asking for trouble. I would probably lay my system out like: configdirectory: /srv/cyrus-be/conf partition-default: /srv/cyrus-be/default-part partition-ssd: /srv/cyrus-be/ssd-part And then each tree would only have one type of thing in it. Anyway, I don't think that would break anything. metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part metapartition_files: header index cache expunge squat annotations lock dav archivecache Ooh, I haven't tested having cache and archivecache on the same location. That's really interesting. Again, I'd be in favour of separation here, give them different paths. That might be tricky with ssd though, the way this is laid out. I assume you have some kind of symlink farm going on? Otherwise it all looks OK. Are you getting other IOERRORs in your log files which could show things aborting? It really looks like your conversations DB is getting out of sync due to other failures. Cheers, Bron. On Thu, Jan 31, 2019, at 19:36, Michael Menge wrote: > > Quoting Bron Gondwana : > > > On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: > >> Hi Bron > >> > >> > >> Quoting Bron Gondwana : > >> > >> > Sorry I haven't been following along with this earlier. Can you post > >> > your imapd.conf and cyrus.conf as well as let her know if you run > >> > anything which directly messes with files on disk. > >> > >> Thanks for looking into it. I have attached the backend and replica > >> configs. There should be nothing messing with the files on disk > >> directly. Some times we need to restore mails from file based > >> backup (bacula) followed by a reconstruct but this was not the > >> case this time. > > > > Do you always rebuild the conversations db after doing the > > reconstruct? That will be necessary now. We switched to doing all > > our restores using IMAP append a while back so we're never fiddling > > the file system under Cyrus. > > > >> > > >> > Also, what operating system is this on and what Cyrus version? > >> > > >> > >> We are running a cyrus 3.0.8 compiled with the following Options > >> (./configure --enable-murder --enable-http --enable-calalarmd > >> --enable-replication --enable-backup --enable-idled > >> --enable-autocreate CFLAGS="-fPIC -g") > >> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. > > > > They should be fine. I'll have a read of the config when I'm at a > > real computer. > > > > did you find anything in the config that would explain the problems? > > >> > >> > Bron. > >> > > >> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: > >> >> Hi, > >> >> > >> >> I have discovered an other problem with the conversations db: > >> >> > >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and > >> >> "IOERROR: conversations_audit on store:" > >> >> A look at the source code shows that these errors are logged after > >> >> "_sanity_check_counts" is called. > >> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a > >> >> serious problem. Do I? > >> >> > >> >> This problem occurred for accounts where the rebuild of the > >> >> conversations db was successful. > >> >> > >> >> I don't want to rebuild the conversations db every few days. > >> >> > >> >> Any help is appreciated. > >> >> > >> >> Kind regards > >> >> > >> >> > >> >> Michael Menge > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> >> M.Menge Tel.: (49) 7071/29-70316 > >> >> Universität Tübingen Fax.: (49) 7071/29-5912 > >> >> Zentrum für Datenverarbeitung mail: > &
Re: another problem with conversations db
Yes, updating reconstruct is on our roadmap. The biggest problem is that there are multiple file types across mailboxes and conversations and the changes can't be atomic, so if reconstruct discovers something partially done, it can't tell if the conversations were correct. The longer term grand plan is to reduce the number of db types and make conversations updates able to be atomic. Cheers, Bron. On Fri, Jan 25, 2019, at 22:30, Michael Menge wrote: > Hi, > > > Quoting Bron Gondwana : > > > On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: > >> Hi Bron > >> > >> > >> Quoting Bron Gondwana : > >> > >> > Sorry I haven't been following along with this earlier. Can you post > >> > your imapd.conf and cyrus.conf as well as let her know if you run > >> > anything which directly messes with files on disk. > >> > >> Thanks for looking into it. I have attached the backend and replica > >> configs. There should be nothing messing with the files on disk > >> directly. Some times we need to restore mails from file based > >> backup (bacula) followed by a reconstruct but this was not the > >> case this time. > > > > Do you always rebuild the conversations db after doing the > > reconstruct? That will be necessary now. We switched to doing all > > our restores using IMAP append a while back so we're never fiddling > > the file system under Cyrus. > > > > We didn't have to restore any mails from backup since we enabled > conversation db > a few weeks ago. It is good to know that the rebuild is necessary, but > shouldn't > reconstruct also update the conversation db if it re-appends the message? > At least a hint should be put in the reconstruct man page, like the > one for "quota -f" > > > >> > > >> > Also, what operating system is this on and what Cyrus version? > >> > > >> > >> We are running a cyrus 3.0.8 compiled with the following Options > >> (./configure --enable-murder --enable-http --enable-calalarmd > >> --enable-replication --enable-backup --enable-idled > >> --enable-autocreate CFLAGS="-fPIC -g") > >> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. > > > > They should be fine. I'll have a read of the config when I'm at a > > real computer. > > > >> > >> > Bron. > >> > > >> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: > >> >> Hi, > >> >> > >> >> I have discovered an other problem with the conversations db: > >> >> > >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and > >> >> "IOERROR: conversations_audit on store:" > >> >> A look at the source code shows that these errors are logged after > >> >> "_sanity_check_counts" is called. > >> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a > >> >> serious problem. Do I? > >> >> > >> >> This problem occurred for accounts where the rebuild of the > >> >> conversations db was successful. > >> >> > >> >> I don't want to rebuild the conversations db every few days. > >> >> > >> >> Any help is appreciated. > >> >> > >> >> Kind regards > >> >> > >> >> > >> >> Michael Menge > > > > > M.Menge Tel.: (49) 7071/29-70316 > Universität Tübingen Fax.: (49) 7071/29-5912 > Zentrum für Datenverarbeitung mail: > michael.me...@zdv.uni-tuebingen.de > Wächterstraße 76 > 72074 Tübingen > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: another problem with conversations db
On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: > Hi Bron > > > Quoting Bron Gondwana : > > > Sorry I haven't been following along with this earlier. Can you post > > your imapd.conf and cyrus.conf as well as let her know if you run > > anything which directly messes with files on disk. > > Thanks for looking into it. I have attached the backend and replica > configs. There should be nothing messing with the files on disk > directly. Some times we need to restore mails from file based > backup (bacula) followed by a reconstruct but this was not the > case this time. Do you always rebuild the conversations db after doing the reconstruct? That will be necessary now. We switched to doing all our restores using IMAP append a while back so we're never fiddling the file system under Cyrus. > > > > Also, what operating system is this on and what Cyrus version? > > > > We are running a cyrus 3.0.8 compiled with the following Options > (./configure --enable-murder --enable-http --enable-calalarmd > --enable-replication --enable-backup --enable-idled > --enable-autocreate CFLAGS="-fPIC -g") > in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. They should be fine. I'll have a read of the config when I'm at a real computer. > > > Bron. > > > > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: > >> Hi, > >> > >> I have discovered an other problem with the conversations db: > >> > >> Thousends of lines with "IOERROR: conversations_audit on load:" and > >> "IOERROR: conversations_audit on store:" > >> A look at the source code shows that these errors are logged after > >> "_sanity_check_counts" is called. > >> The log level LOG_ERR and the prefix IOERROR indicate that I have a > >> serious problem. Do I? > >> > >> This problem occurred for accounts where the rebuild of the > >> conversations db was successful. > >> > >> I don't want to rebuild the conversations db every few days. > >> > >> Any help is appreciated. > >> > >> Kind regards > >> > >> > >> Michael Menge > >> > >> > >> > >> > >> > >> > >> M.Menge Tel.: (49) 7071/29-70316 > >> Universität Tübingen Fax.: (49) 7071/29-5912 > >> Zentrum für Datenverarbeitung mail: > >> michael.me...@zdv.uni-tuebingen.de > >> Wächterstraße 76 > >> 72074 Tübingen > >> > >> > >> Cyrus Home Page: http://www.cyrusimap.org/ > >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > >> To Unsubscribe: > >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > > -- > > Bron Gondwana > > br...@fastmail.fm > > > > ---- > M.Menge Tel.: (49) 7071/29-70316 > Universität Tübingen Fax.: (49) 7071/29-5912 > Zentrum für Datenverarbeitung mail: > michael.me...@zdv.uni-tuebingen.de > Wächterstraße 76 > 72074 Tübingen > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > *Attachments:* > * imapd_be.conf > * cyrus_be.conf > * cyrus_re.conf > * imapd_re.conf -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Question about manual replication (-u )
Yep, that's totally safe. Even doing the same user twice at the same time should be safe, though it may do extra work. Bron. On Tue, Jan 8, 2019, at 05:05, Egoitz Aurrekoetxea wrote: > Good afternoon, > > I know it seems a pretty stupid question, but some time ago, you cannot have > a Cyrus server acting for instance as a master and as slave... it was not > supported... worked... but not supported... so having multiple sync_client > instances... perhaps could damage something (although I suppose not) I'll > probably have the same dount for ctl_conversations -z and -b multiple > parallel commands... > > I know it seems ridiculous... but I ask it because I prefer to try get some > knowledge from Cyrus gurus... :) :) > > Cheers! > --- > > sarenet > *Egoitz Aurrekoetxea* > Departamento de sistemas > 944 209 470 > Parque Tecnológico. Edificio 103 > 48170 Zamudio (Bizkaia) > ego...@sarenet.es > www.sarenet.es > > Antes de imprimir este correo electrónico piense si es necesario hacerlo. > > El 03-01-2019 16:32, Egoitz Aurrekoetxea escribió: >> Good afternoon, >> >> Is it possible to launch several instances of >> "/usr/local/cyrus/bin/sync_client -S DEST-HOST -v -u EMAIL" in parallel?. >> Doing it just one mailbox at a time takes ages It would help me a lot, >> the fact of parallelizing and have no disk bottleneck issues >> >> I think it should be possible... isn't it?. Perhaps it just allowed between >> same version in source and dest?. Or can be done for instance too, with a >> 2.4 as master to 3.0 slave?. >> >> Cheers!! >> -- >> >> sarenet >> *Egoitz Aurrekoetxea* >> Departamento de sistemas >> 944 209 470 >> Parque Tecnológico. Edificio 103 >> 48170 Zamudio (Bizkaia) >> ego...@sarenet.es >> www.sarenet.es >> >> Antes de imprimir este correo electrónico piense si es necesario hacerlo. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Thunderbird's color labels
Thunderbird is using the $label flags for storing colours. I don't believe the colour itself is stored on the server. On Sat, 1 Dec 2018, at 07:07, Joseph Brennan wrote: > > I am trying to find where cyrus imap stores the colors Thunderbird uses in > the message index display. I thought they were probably custom flags and I > could use mbexamine to find them. > > I have a mailbox that has this in the first stanza from mbexamine, which > looked promising-- > User Flags: $label1 $label2 $label3 $label4 $label5 $Forwarded Redirected > > Each message has a line with "> USERFLAGS" but those are mostly four groups > of zeros, and the values do not correspond to the Tbird colors. > > What else can I try? > > > -- > Joseph Brennan > Lead, Email and Systems Applications > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: inaccessible mailbox for one user
"ctl_conversationsdb -R -r" should rebuild all conversations databases for all users. Bron. On Sat, Oct 13, 2018, at 22:35, Lists Nethead wrote: > > Quoting Albert Shih : > > > Le 10/10/2018 à 07:41:52+0200, Lists Nethead a écrit > >> > >> Quoting Albert Shih : > >> > >> > Le 09/10/2018 à 21:26:28+0200, Lists Nethead a écrit > >> >> > >> >> Quoting Lists Nethead : > >> >> > >> >> > > Hi, > >> >> > > > >> >> > > A user clicks on a mail in Horde IMP and next the inbox appears > >> >> > > completely empty. This message files all still there of course, but > >> >> > > the problem is that I cannot run any cyrus command on that mailbox, > >> >> > > reconstruct, chk_cyrus or anything. > >> >> > > > >> >> > > Is there a lock possibly sitting somewhere that prohibits running > >> >> > > any commands? Any other hints as to where I should start? > >> >> > > > >> >> > >> >> Restored users seen and sub files from backup, ran reconstruct again > >> >> and > >> >> then I had to run unexpunge. Now user files are back but I see > >> the following > >> >> error for this user in the logs > >> >> > >> >> IOERROR: conversations_audit on store > >> >> > >> >> Should I try sbin/ctl_conversationsdb -b ? > >> > > >> > I'm not a cyrus guru but, are you sure your FS are clean ? I'm running > >> > FreeBSD and long time ago I got a FS error without crash. It was > >> a standard > >> > server (apache) and I got very strange error, like unable to remove > >> > file. > >> > I eventually restart the server and run a fsck -f manually to fix de > >> > problem. I never find out where and how I got that problem > >> > > >> > The FS was UFS2. > >> > > >> > I'm absolutly not sure it's the problem, just try to help. > >> > >> Thank you for your response. > > > > You'r welcome. And seem...my answer was not very usefull ;-) > > > >> Yes, I think the initial fault was not a Cyrus issue and the locking > >> problem vanished after the jail was restarted. Mail works ok now > >> except the error above. > > > > Did you find a solution ? > > > >> We run zfs so fsck is not of much help, but none of the pools report a > >> problem. > > > > We too, run FreeBSD + jail + ZFS (two, one for the mailbox on mechanical > > disk and another on ssd). > > > > No solution found so far, still get "IOERROR: conversations_audit on > store" on this account, tried all switches for ctl_conversationsdb. > Currently out of ideas. > > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Consersations, threaded view with sent messages in sent folder
Yes, 3.0 release supports the XCONV commands if you put "conversations: yes" in your imapd.conf. They're quite undocumented though, so you'll have to pretty much figure them out yourself or read through the archives for where there have been examples pasted in emails. Bron. On Wed, Sep 12, 2018, at 22:14, Egoitz Aurrekoetxea wrote: > Hi Bron! > > So, you mean there's no way of achieving it even with one of the very last > releases of Cyrus?. Does Cyrus support XCONV for instance?. Or in some > manner, this non-standard threading variant?. > > Best regards, > --- > > sarenet > *Egoitz Aurrekoetxea* > Departamento de sistemas > 944 209 470 > Parque Tecnológico. Edificio 103 > 48170 Zamudio (Bizkaia) > ego...@sarenet.es > www.sarenet.es > > Antes de imprimir este correo electrónico piense si es necesario hacerlo. > > El 12-09-2018 08:50, Bron Gondwana escribió: >> In JMAP and the XCONV commands (which are deprecated and will be removed) >> the THREAD concept is cross folder, so it absolutely includes the messages >> in the sent items (or archive folders, or wherever). >> >> Standard IMAP threading is single-folder-only. >> >> Regards, >> >> Bron. >> >> On Tue, Sep 11, 2018, at 20:27, Egoitz Aurrekoetxea wrote: >>> Good morning, >>> >>> How is this topic in recent versions of Cyrus?. Could exist some manner in >>> which a Thread view could include the sent items (the mail replied to the >>> thread) to that thread?. Or should it be done client side with databases?. >>> >>> Best regards, >>> -- >>> >>> sarenet >>> *Egoitz Aurrekoetxea* >>> Departamento de sistemas >>> 944 209 470 >>> Parque Tecnológico. Edificio 103 >>> 48170 Zamudio (Bizkaia) >>> ego...@sarenet.es >>> www.sarenet.es >>> >>> Antes de imprimir este correo electrónico piense si es necesario hacerlo. >>> >>> Cyrus Home Page: http://www.cyrusimap.org/ >>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >>> To Unsubscribe: >>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> >> -- >> Bron Gondwana, CEO, FastMail Pty Ltd >> br...@fastmailteam.com >> >> >> >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Consersations, threaded view with sent messages in sent folder
In JMAP and the XCONV commands (which are deprecated and will be removed) the THREAD concept is cross folder, so it absolutely includes the messages in the sent items (or archive folders, or wherever). Standard IMAP threading is single-folder-only. Regards, Bron. On Tue, Sep 11, 2018, at 20:27, Egoitz Aurrekoetxea wrote: > Good morning, > > How is this topic in recent versions of Cyrus?. Could exist some manner in > which a Thread view could include the sent items (the mail replied to the > thread) to that thread?. Or should it be done client side with databases?. > > Best regards, > -- > > sarenet > *Egoitz Aurrekoetxea* > Departamento de sistemas > 944 209 470 > Parque Tecnológico. Edificio 103 > 48170 Zamudio (Bizkaia) > ego...@sarenet.es > www.sarenet.es > > Antes de imprimir este correo electrónico piense si es necesario hacerlo. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Conversations and search
On Sat, Aug 18, 2018, at 00:08, Albert Shih wrote: > Le 16/08/2018 à 14:04:22+0200, Sebastian Hagedorn a écrit > Hi > >> >> I have a question regarding the conversations db and how it affects >> (Xapian)>> search. In this GitHub issue >> (<https://github.com/cyrusimap/cyrus-imapd/issues/2376>) I was >> dealing with>> Xapian search always failing. The underlying reason turned out >> to be an>> empty conversations db. When I regenerate that user's >> conversations db,>> Xapian search works fine. So far, so good. But then I >> noticed this >> piece of>> documentation: >> >> <https://www.cyrusimap.org/imap/concepts/deployment/databases.html?highlight=conversations#conversations-userid-conversations>>> >> >> Quote: "This file contains all the message-id fields from every >> email that>> has been seen in the ***past three months***, mapping to the >> conversation>> IDs in which this message ID has been seen, and the timestamp >> when it was>> last seen." >> >> This raises a number of questions. What does "has been seen" mean >> in this>> context? And does it mean that Xapian search will always fail to >> find emails>> that arrived more than three months ago?? That doesn't sound >> very >> useful ...> > If I'm correct (I'm new with cyrus too), the conversations DB as > nothing to> do with the database of xapian. > > The xapian database is some file with .glass extension and > contain all the> index of all your mail. Those files stand inside > > t1searchpartition-default/FIRST_LETTER_OF_LOGIN/user/LOGIN/ > > The conversation database is some cyrus internal database use by > cyrus to> **create** the xapian index. > > In other word the three months means, if you stop the xapian index > now, you> have three months to restart it or you're going to need to > regenerated the> xapian database from the beginning (and that take very > lng time).> > Please if it's incorrect please someone correct me. That's incorrect - the messageids are actually only used for thread calculations - so if you get a new messages more than 3 months later, it won't be threaded with related messages (conversations.db threads only, aka: JMAP and XCONV commands). The G keys used for xapian are kept forever (at least: until the message is deleted from the index - which is not when it's expunged, but when cyr_expire cleans up the expunged record). Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: archive and normal partition mixing question
No, don't do that. You can put them on the same filesystem, but don't put different-named partitions on the same disk path. That will confuse Cyrus no end. Bron. On Tue, Jul 24, 2018, at 00:10, Marco wrote: > Hello, > > I would like to create a quota root mailbox with a normal partition> which > is the same of the archive partition. Does Cyrus allow this? > > For instance, I would know if this configuration is valid: > > partition-maildata1: /maildata/maildata1 > partition-maildata2: /maildata/maildata2 > partition-arcmaildata1: /archivio/maildata1 > partition-arcmaildata2: /archivio/maildata2 > archivepartition-maildata1: /archivio/maildata1 > archivepartition-maildata2: /archivio/maildata2 > > metapartition-maildata1: /metamaildata/maildata1 > metapartition-maildata2: /metamaildata/maildata2 > metapartition-arcmaildata1: /metarchivio/maildata1 > metapartition-arcmaildata2: /metarchivio/maildata2 > > archive_enabled: 1 > archive_days: 30 > > > > In this way Cyrus could archive old mails in /archivio/maildata* > But these partitions are used for other mailboxes too. For > instance I can> > createmailbox user/john@example.com maildata1 > createmailbox user/john.doe/arch...@example.com arcmaildata1 > > In this way all messages moved or sent to "Archive" folder will stay > under arcmaildata1. > All mailboxes archived by "cyr_expire -A" will go to the same path of> > arcmaildata1 too. > > Thank you very much > Regards > Marco > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Donate to cyrus team
Hi Hans, At the moment the donations to the project are in the form of people's time or services that the project needs, but we don't have any facility for handling cash donations. We've talked about moving the project out from CMU into an organisation which specialises in supporting open source projects, and could arrange for donations to handled by said organisation. But for now - there's no way to donate money. Just to donate bug reports or code :) Cheers, Bron. On Fri, Jul 6, 2018, at 00:26, Johann Hörmann wrote: > Hi, > > couldn't find a donate-link at cyrusimap.org, and searching the > net with> 'cyrus imap donat' returns no results on donating. > Is there no need for donations? > > Hans > -- > Hoermann Solutions > Johann Hörmann > Dipl.Ing.(FH) > - Software Analyse, Design, Entwicklung - > - Netzwerk Administration - > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: switch master/slve replication
Yep, that will be enough. The only thing it might not catch is if there are users on the replica which aren't present on the master (for whatever reason)... in that case, they will remain on the replica still. Bron. On Wed, Jun 27, 2018, at 13:26, Antonio Conte wrote: > hi all, > > I have to switch a master/replica in rolling replication. > > How I can be sure the replication is totally done ? it's > enough to run > > /usr/lib/cyrus/sync_client -v -A > > from master ? and then, which operation I have to do ? > > thanks in advance > > -- > Never try to teach a pig to sing. > It wastes your time and annoys the pig. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Prefork or not prefork
We have a handful with "prefork=1 maxchild=1" (one for each service type) listening on separate ports that can be connected for debugging. The advantage here being that the process is already waiting for you to connect a debugger before using it. But yeah, otherwise we have prefork=0 at FastMail too. Fork is pretty fast compared to everything else in an IMAP session. Bron. On Mon, Jun 25, 2018, at 22:45, Eric Luyten wrote: > > > On 25/06/2018 14:39, Albert Shih wrote: >> Hi everyone, >> >> I would like to know when that's become usefull to use prefork >> (for imap)>> and how much you set the prefork (vs nb of users). > > > We have prefork=0 set on all Cyrus services for many years now. > > > > Eric. > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: XLIST, special-use mailboxes
Hi Paul, In newer versions the support is more accurately to spec, where you create with (USE \Foo) and then it's stored alongside that mailbox, even if it's renamed. The old 2.4 version was just a hack for XLIST before the standards support was added. Cheers, Bron. On Fri, Jun 15, 2018, at 01:03, Paul van der Vlis wrote: > Hello, > > Does Cyrus IMAP support RFC 6154 about special-use mailboxes? > https://tools.ietf.org/html/rfc6154 > > I read that it only wass supported in Cyrus version 2.4 ? > I am using version 2.5.10 (from Debian 9). > > Can I put this in imapd.conf for all users? > xlist-archive: Archives > xlist-drafts: Drafts > xlist-sent: Sent > xlist-spam: Spam > xlist-trash: Trash > > Is it only used on new created folders or always? > > Do I need "specialusealways: 1" or something like that? > > What about client-support? Is it much used in clients? > > With regards, > Paul van der Vlis > > > > > -- > Paul van der Vlis Linux systeembeheer Groningen > https://www.vandervlis.nl/ > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: SOLVED: IDLE not working for one account only
Ah, well done! Sorry I didn't get back to the attached, been a busy few days of conference here On Fri, Jun 8, 2018, at 01:06, Neil Price wrote: > cyrus.index needed to be updated to latest version. This affected > accounts upgraded from 2.2.4 > > reconstruct -rf -V max user/badaccount > > HIGHESTMODSEQ now looks more reasonable on telnet 143 login > > > On 05/06/2018 09:17, Neil Price wrote: >> I'm using 2.5.10-3 (the one that comes with debian stretch) >> >> I have one user that I cannot get IDLE to work for using either poll>> or >> idled. >> >> Other accounts work normally. No errors in the log. >> >> Have set him up with a number of email clients and they all do the >> same thing. >> >> I can not see anything the account setup that can cause this, >> suggestions? > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Can't view (other) users subfolders with imapd 3.0.6
All the namespace conversion is supposed to be encapsulated in a couple
of places. This breaks that, which means that something earlier of later
is doing its own version of pattern to mailbox conversion. Ideally this
would be captured in the abstraction layer in mbname. I haven't looked
deeply though.
It's correct, but it likely points to an abstraction leakage elsewhere.
Bron.
On Wed, Jun 6, 2018, at 13:47, ellie timoney wrote:
> On Tue, Jun 5, 2018, at 7:27 PM, Edda wrote:
>> thank you very much for the handling of my patch! I don't have an
>> account on github. So, it's easiest for me to just dump it on the
>> users list. Please let me know if that bothers you.>>
>
> It's fine for small patches like this, especially when they work right
> first time. ;)>
> For larger pieces of work, a pull request certainly makes it easier
> to collaboratively iterate towards a good solution! But if you'd
> prefer to not create a GitHub account for whatever reason, that's
> fair enough.>
>> As an author, "Edda" would be fine :) Thank you.
>>
>
> Great! This is now on the 3.0 and master branches. Thanks very
> much! :)>
> On Tue, Jun 5, 2018, at 8:24 PM, Bron Gondwana wrote:
>> The patch looks correct at a reading. It also looks like it's
>> masking a deeper bug, but that's fine - we can fix that later :)>>
>
> What are you seeing that I'm missing?
>
> I can see that this bug was not just "other users", but any
> common/solo list pattern with a hiersep in it, but this patch is
> general enough to fix both.>
>
>> Best regards,
>> Edda
>>
>> Am 05.06.18 um 04:50 schrieb ellie timoney:
>>> Hi Edda,
>>>
>>> Thanks for the patch! I've added your email below to the GitHub
>>> issue for this: https://github.com/cyrusimap/cyrus-imapd/issues/2362>>>
>>> I'll review the patch and if it looks good, will include it in the
>>> next 3.0 release. I'll also forward-port it to master if necessary.>>>
>>> Let me know if you'd like to submit this as a Pull Request, so your
>>> GitHub identity can be credited for it, otherwise I'll just set your
>>> email address as the author on the commit. :)>>>
>>> Cheers,
>>>
>>> ellie
>>>
>>>
>>> On Mon, Jun 4, 2018, at 9:51 PM, Edda wrote:
>>>> Hi,
>>>>
>>>> I was able to reproduce the issue with 3.0.5 with these settings:
>>>>
>>>> unixhierarchysep: yes
>>>> altnamespace: no
>>>> crossdomains: no
>>>>
>>>> Key factor is unixhierarchysep on. (crossdomains is a special case.
>>>> altnamespace on/off only looks different.)>>>>
>>>> I did a lot of debugging and found out the following:
>>>>
>>>> Compared to version 2.4.x and 2.5.x 3.0.x function list_data in
>>>> ../imap/imapd.c does not transform the hierarchy separator.>>>>
>>>> In function mboxlist_do_find (../imap/mboxlist.c) below "Other
>>>> Users namespace" we have a prefix search. With "unixhierarchysep
>>>> on" cyrus uses the wrong prefix (domainpat) for that search.>>>>
>>>> Example:
>>>>
>>>> . LIST "" "user/%"
>>>> * LIST (\HasChildren) "/" user/bbb
>>>> . OK Completed (0.000 secs 4 calls)
>>>> . LIST "" "user/bbb/sub01"
>>>> . OK Completed (0.000 secs)
>>>>
>>>> In this case cyrus searches with domainpat: user.bbb/sub01/subsub01>>>>
>>>> The prefix is previously generated from a variable called
>>>> commonpat, which is used only for this specific search.>>>>
>>>> So I fixed it there with this patch:
>>>>
>>>> --- cyrus-imapd-3.0.5/imap/mboxlist.c.orig2018-04-12
>>>> 13:12:47.633956338 +0200>>>> +++ cyrus-imapd-3.0.5/imap/mboxlist.c
>>>> 2018-05-30
>>>> 10:56:44.255724710 +0200>>>> @@ -2774,6 +2774,10 @@
>>>> const char *pat = strarray_nth(patterns, i);
>>>> if (pat[prefixlen] != c) break;
>>>> }
>>>> +if (rock->namespace->hier_sep == '/') {
>>>> +if (c == '/') c = '.';
>>>> +else if (c == '.') c = DOTCHAR;
>>>> +}
>>>> if (i < patterns->count) break;
>>>> if (c == '*' || c == '%' || c == '?') break;
>>&g
Re: IDLE not working for one account only
The timeouts are fine - that's just a select timeout so it can do
cleanup.
recvfrom(14, "\2\0\0\0\0\0\0\0user.elangenhoven\0", 1032, 0,
{sa_family=AF_UNIX, sun_path="/var/lib/cyrus/socket/idle.31167"}, [110-
>35]) = 26 sendto(14, "\2\0\0\0\0\0\0\0user.elangenhoven\0", 26,
MSG_DONTWAIT, {sa_family=AF_UNIX,
sun_path="/var/lib/cyrus/socket/idle.30947"}, 110) = 26
That's a push from the lmtpd (PID 31167) for delivery, followed by a
push to the imapd (PID 30947) saying that the mailbox has been touched!
Sorry to be a pain, but can you do it again and get an strace of the
imapd as well - the one which is in idle.
Thanks,
Bron.
On Wed, Jun 6, 2018, at 00:53, Neil Price wrote:
> I've attached an strace of idled. It shows the bad user client
> (elangenhoven) going into idle and a message being sent to it.You can
> see the message being sent but there seems to be no reaction to it
> arriving.> There are a bunch of timeouts there?
>
> On 05/06/2018 15:42, Bron Gondwana wrote:
>> Damn. I guess I'm going to have to ask for an strace next! Of the
>> idled process probably.>>
>> Idle changed a bunch between 2.5 and 3.0, so I don't know how much
>> I'll be able to help :( Definitely an strace of the idled during
>> both the start of the IDLE command and the delivery will help see
>> what's happening.>>
>> Bron.
>>
>>
>
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To
> Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> Email had 1 attachment:
> * strace.txt
> 12k (text/plain)
--
Bron Gondwana, CEO, FastMail Pty Ltd
br...@fastmailteam.com
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: IDLE not working for one account only
Damn. I guess I'm going to have to ask for an strace next! Of the idled process probably. Idle changed a bunch between 2.5 and 3.0, so I don't know how much I'll be able to help :( Definitely an strace of the idled during both the start of the IDLE command and the delivery will help see what's happening. Bron. On Tue, Jun 5, 2018, at 23:11, Neil Price wrote: > Nothing, only mention of idle is on shutdown > > On 05/06/2018 14:35, Bron Gondwana wrote: >> What's showing up in syslog? >> > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: IDLE not working for one account only
What's showing up in syslog? Bron. On Tue, Jun 5, 2018, at 22:19, Neil Price wrote: > On 05/06/2018 09:17, Neil Price wrote: >> I'm using 2.5.10-3 (the one that comes with debian stretch) >> >> I have one user that I cannot get IDLE to work for using either poll>> or >> idled. >> >> Other accounts work normally. No errors in the log. >> >> Have set him up with a number of email clients and they all do the >> same thing. >> > I now have a second user who was working fine until we added a second> imap > client (Android) for her. Now the primary imap client does work > with idle (but the Android client does). > :-\ > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Can't view (other) users subfolders with imapd 3.0.6
folders. Distro:Gentoo Cyrus version: >>>> cyrus-imapd-3.0.6 imapd-3.0.5.conf attached In the test >>>> environment we created a user named "aaa" and "bbb", then we >>>> created the folder "bbb/mytest/mysubtest". We set the acl for the >>>> user aaa to share the mailbox bbb/mytest/mysubtest user.aaa >>>> 0 default aaa lrswipkxtecdan anonymous p user.bbb0 >>>> default bbb lrswipkxtecdan anonymous p aaa >>>> lrswipkxtecdan user.bbb.mytest 0 default bbb lrswipkxtecdan >>>> anonymous p aaa lrswipkxtecdan user.bbb.mytest.mysubtest >>>> 0 default bbb lrswipkxtecdan anonymous p aaa >>>> lrswipkxtecdan From whatever imap client, when we search for new >>>> folders, user.bbb appears, but neither user.bbb.mytest nor >>>> user.bbb.mytest.mysubtest. This is the capture from imapd 3.0.6: >>>> IMAP> 62 LIST "" "user/%" IMAP< * LIST (\HasChildren) "/" user/bbb >>>> IMAP< 62 OK Completed (0.001 secs 3 calls) IMAP> 63 LIST "" >>>> "user/bbb/%" IMAP< 63 OK Completed (0.000 secs) IMAP> 64 LSUB "" >>>> "user/bbb/%" IMAP< 64 OK Completed (0.000 secs) IMAP> 65 LSUB "" >>>> "user/%" IMAP< 65 OK Completed (0.000 secs) IMAP> 66 LSUB "" "%" >>>> IMAP< * LSUB (\Noselect \HasChildren) "/" INBOX IMAP< 66 OK >>>> Completed (0.000 secs 4 calls) IMAP> 16 NOOP IMAP< 16 OK Success >>>> IMAP> 67 SUBSCRIBE "user/bbb" IMAP< 67 OK Completed IMAP> 68 LSUB >>>> "" "%" IMAP< * LSUB (\Noselect \HasChildren) "/" INBOX IMAP< * LSUB >>>> (\Noselect \HasChildren) "/" user IMAP< 68 OK Completed (0.001 secs >>>> 5 calls) IMAP> 69 LSUB "" "INBOX/%" IMAP< * LSUB () "/" >>>> INBOX/Drafts ... IMAP> 74 LSUB "" "user/%" IMAP< * LSUB () "/" >>>> user/bbb IMAP< 74 OK Completed (0.001 secs 1 calls) IMAP> 75 LSUB >>>> "" "user/bbb/%" IMAP< 75 OK Completed (0.001 secs) Refreshing only >>>> the user.bbb folders nothing changed IMAP> 83 LIST "" "user/%" >>>> IMAP< * LIST (\HasChildren) "/" user/bbb IMAP< 83 OK Completed >>>> (0.001 secs 3 calls) IMAP> 84 LIST "" "user/bbb/%" IMAP< 84 OK >>>> Completed (0.001 secs) IMAP> 85 LSUB "" "user/bbb/%" IMAP< 85 OK >>>> Completed (0.001 secs) IMAP> 86 LSUB "" "user/%" IMAP< * LSUB () >>>> "/" user/bbb IMAP< 86 OK Completed (0.001 secs 1 calls) With >>>> net-mail/cyrus-imapd- >>>> 2.4.17 all is working fine imapd-2.4.17.conf attached user.aaa >>>> 0 default aaa lrswipkxtecda anonymous p user.bbb0 >>>> default bbb lrswipkxtecda anonymous p aaa >>>> lrswipkxtecda user.bbb.mytest 0 default bbb lrswipkxtecda >>>> anonymous p aaa lrswipkxtecda user.bbb.mytest.mysubtest >>>> 0 default bbb lrswipkxtecda anonymous p aaa >>>> lrswipkxtecda if we search for new folders to subscribe user/bbb >>>> and all the subfolders appeared IMAP> 57 LIST "" "user/%" IMAP< * >>>> LIST (\HasChildren) "/" user/bbb IMAP< 57 OK Completed (0.000 secs >>>> 3 calls) IMAP> 58 LIST "" "user/bbb/%" IMAP< * LIST (\HasChildren) >>>> "/" user/bbb/mytest IMAP< 58 OK Completed (0.000 secs 2 calls) >>>> IMAP> 59 LIST "" "user/bbb/mytest/%" IMAP< * LIST (\HasNoChildren) >>>> "/" user/bbb/mytest/mysubtest IMAP< 59 OK Completed (0.000 secs 2 >>>> calls) IMAP> 60 LIST "" "user/bbb/mytest/mysubtest/%" IMAP< 60 OK >>>> Completed (0.000 secs) IMAP> 61 LSUB "" >>>> "user/bbb/mytest/mysubtest/%" IMAP< 61 OK Completed (0.000 secs) >>>> IMAP> 62 LSUB "" "user/bbb/mytest/%" IMAP< 62 OK Completed (0.000 >>>> secs) IMAP> 63 LSUB "" "user/bbb/%" IMAP< 63 OK Completed (0.000 >>>> secs) IMAP> 64 LSUB "" "user/%" IMAP< 64 OK Completed (0.000 secs) >>>> IMAP> 65 LSUB "" "%" IMAP< * LSUB (\Noselect \HasChildren) "/" >>>> INBOX IMAP< 65 OK Completed (0.000 secs 4 calls) IMAP> 66 NOOP >>>> IMAP< 66 OK Completed IMAP> 67 SUBSCRIBE "user/bbb" IMAP< 67 OK >>>> Completed IMAP> 68 SUBSCRIBE "user/bbb/mytest/mysubtest" IMAP< 68 >>>> OK Completed IMAP> 69 SUBSCRIBE "user/bbb/mytest" IMAP< 69 OK >>>> Completed Any suggestion? Thanks for the support Marco >>>>>>>> >>>> >>>> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: >>>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: >>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >>>>>>> >>> >>> Cyrus Home Page: http://www.cyrusimap.org/ >>> List Archives/Info: >>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/>>> To Unsubscribe: >>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> >> >> >> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: >> http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus> > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Mailbox inaccessible after setaclmailbox with invalid identifier
https://github.com/cyrusimap/cyrus-imapd/issues/2378 to track the issue. Sorry about the short comment earlier on my phone as I woke up - meant to save it as a draft and add some detail! # /usr/cyrus/bin/cyr_dbtool Usage: /usr/cyrus/bin/cyr_dbtool [-C altconfig] [] []Usable Backends: flat, skiplist, quotalegacy, twoskip Actions: * show [] * get * set * delete * dump - internal format dump * consistent - check consistency * repack - repack/checkpoint the DB (if supported) * damage - start a commit then die during * batch - read from stdin and execute commands You may omit key or key/value and specify one per line on stdin keys are terminated by tab or newline, values are terminated by newline = will need to be your mailboxes.db (e.g. /var/imap/mailboxes.db on many systems). You'll have to run it as the cyrus user. You can use "show" to dump the whole file and grep for the individual mailbox, or "get" on the mailbox key directly to get the one line. Then "set" to set it with a value with that group name removed. You'll probably want to do another setacl on that mailbox with something safe afterwards, just so the cyrus.header file gets cleaned up as well. It contains a value which is a cache of the value in mailboxes.db. Cheers, Bron. On Tue, May 29, 2018, at 07:55, Bron Gondwana wrote: > If you use cyr_dbtool you can edit the offending line in mailboxes db.> > Clearly there's a bug here too, we should be sanitising group names! > Or at least storing them in a reversible format> > > On Tue, May 29, 2018, at 00:33, Sven Schwedas wrote: >> After running `setaclmailbox some/mailbox "group:name with >> spaces" all`,>> all attempts to access the mailbox in any way either result >> in an >> "Invalid identifier" error message, or can't even find the mailbox. >> >> deleteaclmailbox with exactly the same parameters as set doesn't help>> >> (same "invalid identifier" error); cyrreconstruct doesn't even >> find the>> mailbox. Deliveries are deferred as well. >> >> Any ideas how I can recover the mailbox without rolling back >> the whole>> server to an older backup? >> >> (Cyrus 2.5.10 on Debian stretch, if that matters.) >> >> -- >> Mit freundlichen Grüßen, / Best Regards, >> Sven Schwedas, Systemadministrator >> ✉ sven.schwe...@tao.at | ☎ +43 680 301 7167 >> TAO Digital | Teil der TAO Beratungs- & Management GmbH >> Lendplatz 45 | FN 213999f/Klagenfurt, FB-Gericht Villach >> A8020 Graz| https://www.tao-digital.at >> >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/>> To >> Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> Email had 1 attachment: >> * signature.asc 1k (application/pgp-signature)> > -- > Bron Gondwana, CEO, FastMail Pty Ltd > br...@fastmailteam.com > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Mailbox inaccessible after setaclmailbox with invalid identifier
If you use cyr_dbtool you can edit the offending line in mailboxes db. Clearly there's a bug here too, we should be sanitising group names! Or at least storing them in a reversible format On Tue, May 29, 2018, at 00:33, Sven Schwedas wrote: > After running `setaclmailbox some/mailbox "group:name with > spaces" all`,> all attempts to access the mailbox in any way either result in > an > "Invalid identifier" error message, or can't even find the mailbox. > > deleteaclmailbox with exactly the same parameters as set doesn't help> (same > "invalid identifier" error); cyrreconstruct doesn't even > find the> mailbox. Deliveries are deferred as well. > > Any ideas how I can recover the mailbox without rolling back the whole> > server to an older backup? > > (Cyrus 2.5.10 on Debian stretch, if that matters.) > > -- > Mit freundlichen Grüßen, / Best Regards, > Sven Schwedas, Systemadministrator > ✉ sven.schwe...@tao.at | ☎ +43 680 301 7167 > TAO Digital | Teil der TAO Beratungs- & Management GmbH > Lendplatz 45 | FN 213999f/Klagenfurt, FB-Gericht Villach > A8020 Graz| https://www.tao-digital.at > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > Email had 1 attachment: > * signature.asc 1k (application/pgp-signature) -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: sync_client crash with sieve
It sounds like the sieve file might be incorrect in some way? Either it doesn't parse with the current version of sievec, or it's got the wrong permissions, or even the wrong naming. On our servers we have: websieve.script websievebc defaultbc -> websievebc As managed by having timsieved save a script called "webscript". If you have sieve scripts which aren't correctly laid out to how Cyrus expects, that could maybe cause sync to abort because it doesn't know how to replicate that layout. Bron. On Fri, May 18, 2018, at 01:57, Albert Shih wrote: > Hi, > > I've got some issue with sync_client (first run, before rolling sync).> > When I launch > > sync_client -A > > the synchronization start and after some time crash. I activate > the verbose> mode, and It seem sync_client crash over some sieve file. > > So I not sure, but I pretty sure, everytime I got a sieve file for > the user> (in /var/imap/sieve) the sync crash. > > Any idea ? > > Regards > -- > Albert SHIH > DIO bâtiment 15 > Observatoire de Paris > xmpp: j...@obspm.fr > Heure local/Local time: > Thu May 17 17:51:04 CEST 2018 > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: INBOX.INBOX.
No, it's deliberately allowing that case because in altnamespace, those are the Inbox subfolders, so they have to be allowed. It still stops INBOX.INBOX.INBOX.INBOX.INBOX.INBOX which was the pathological case. Bron. On Wed, May 16, 2018, at 02:57, Wolfgang Breyha wrote: > Hi! > > I'm not sure if this is a bug or intended... > > cyrus checks for mailbox names including "INBOX.INBOX." since 2.5 and> > prevents creation to stop buggy clients like Apple Mail from > recursing.> > But it still allows eg. > INBOX.INBOX.Trash > since the check is done after conversion to the internal > user format.> > INBOX.INBOX.(Trash|Sent|Drafts) is widely used by K9 (and > others) as it> seems. K9 fixed it "recently" ... > https://github.com/k9mail/k-9/issues/701 > ... but it seems existing configs still use this pattern. > > Does cyrus intentionally allow user..INBOX. and only prevents > user..INBOX.INBOX or is this a bug and will be changed > in future> versions? > > With kind regards, > Wolfgang Breyha > -- > Wolfgang Breyha <wbre...@gmx.net> | https://www.blafasel.at/ > Vienna University Computer Center | Austria > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Size of partition.
On Tue, 1 May 2018, at 04:12, Albert Shih wrote: > Le 30/04/2018 à 08:30:44+0200, Robert Stepanek a écrit >> On Fri, Apr 27, 2018, at 15:48, Albert Shih wrote: >>> I activated squatter but put the index not on ssd. The index >>> created by squatter are (in my case) about 5-8% of the size of >>> the mailbox. >>> >>> But I got lot of « conversation » index. What's the purpose >>> of those « conversations » ? >> >> conversations.db initially was used to keep an index of email >> conversations (e.g. the threaded views typically found in most email>> >> readers nowadays). But today it also helps for most use-cases where >> we need to keep track of messages across mailboxes (e.g. if you have>> two >> copies of an email in two mailboxes, you'll see that reflected in>> >> conversations.db). It confusingly is still named conversations.db for>> >> historic reasons. But for any recent (v3) Cyrus IMAP installation, >> including Xapian and JMAP, it's a very critical piece in the puzzle.> > Ok. Thanks you. > > Still one question, I see in the configuration file > > conversations_expire_days : 90 > > why do I need to keep those information during 90 days ? For > example what> would be the impact if I say > > conversations_expire_days : 7 > The bit which is expired is the mapping from message-id(s) to conversation id. If you set it to 7 days, then if you receive a reply to an existing message more than a week later, it won't detect that they are related. Cheers, Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: archive and downgrade
Interestingly enough, Cyrus 3.0 could downgrade the indexes for you. Otherwise you'll pretty much have to "find . -name cyrus.index | xargs rm" or something. Backup first of course. Our just upgrade the machine! On Sat, 24 Mar 2018, at 04:41, Gabriele Bulfon wrote: > Hi, > > for some reasons, I had to move old folders onto an older system for > archiving.> The source system has cyrus 2.5.11, while the old archive one > has 2.4.12.> > If I reconstruct folders on the target system (reconstruct -rf > user.mailbox) the folder cause error because of new cyrus.index > version files contained in all the tree.> Is there any way I can run the > 2.4.12 reconstruct and tell it to not > consider any cyrus index and create it from scratch with its own > version?> The archived folders have many subfolders. > > Thanx for any idea > Gabriele > > > *Sonicle S.r.l. *: http://www.sonicle.com[1] > *Music: *http://www.gabrielebulfon.com[2] > *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Links: 1. http://www.sonicle.com/ 2. http://www.gabrielebulfon.com/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: 2.5.11 shared seen
Shared seen in 2.4 and below was done by storing the seen information in a global database. In 2.5 and above, it's stored in the cyrus.index (if shared seen is enabled, it duplicates the owner's seen information). It may be that the upgrade path is broken for shared seen, I'm not sure how well it was tested - we didn't have a testing framework back then. Regards, Bron. On Wed, 14 Mar 2018, at 15:20, ellie timoney wrote: > Hi info-cyrus, > > If anyone with experience using shared seen could jump in and help > out, that'd be great. I'm out of my depth here and don't know what > else to ask!> > Cheers, > > ellie > > > On Fri, Mar 9, 2018, at 7:11 PM, Gabriele Bulfon wrote: >> Hello, upgraded from 2.4.12, and yes, everyone has ACLs with "all" >> flags, including 's'.>> >> >> *Sonicle S.r.l. *: http://www.sonicle.com[1] >> *Music: *http://www.gabrielebulfon.com[2] >> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon >> >> >> >> *Da:* ellie timoney <el...@fastmail.com> *A:* info- >> cy...@lists.andrew.cmu.edu *Data:* 9 marzo 2018 2.53.06 CET >> *Oggetto:* Re: 2.5.11 shared seen>> >> >>> Hi Gabriele, >>> >>> Which version have you upgraded from? (I'm not sure if this makes a >>> difference, but it might)>>> >>> The annotation you're using looks correct to me. Do the users have >>> the 's' ACL permission on the mailboxes?>>> >>> Cheers, >>> >>> ellie >>> >>> On Wed, Mar 7, 2018, at 2:14 AM, Gabriele Bulfon wrote: >>>> Hi, >>>> >>>> I've noticed that on upgraded systems to 2.5.11 shared seen is no >>>> more working.>>>> Our software used to set shared seen state via set >>>> annotation /vendor/cmu/cyrus- >>>> imapd/sharedseen true>>>> >>>> Is this still ok or is there a different way to do it? >>>> >>>> Thanks, >>>> Gabriele >>>> >>>> >>>> *Sonicle S.r.l. *: http://www.sonicle.com[3] >>>> *Music: *http://www.gabrielebulfon.com[4] >>>> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon >>>> >>>> Cyrus Home Page: http://www.cyrusimap.org/ >>>> List Archives/Info: >>>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/>>>> To Unsubscribe: >>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >>> >>> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: >>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: >>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >>>> > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Links: 1. http://www.sonicle.com/ 2. http://www.gabrielebulfon.com/ 3. http://www.sonicle.com/ 4. http://www.gabrielebulfon.com/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: message "unable to setsocketopt(IP_TOS)" in logs
On Fri, 16 Feb 2018, at 22:55, Vladislav Kurz wrote:
> On 02/16/18 12:42, Adam Tauno Williams wrote:
>> On Fri, 2018-02-16 at 08:55 +0100, Chentao Credungtao via Info-cyrus>> wrote:
>>> cyrus/master[7082]: unable to setsocketopt(IP_TOS) service
>>> lmtpunix/unix: Operation not supported
>>> cyrus/master[7082]: unable to setsocketopt(IP_TOS) service
>>> notify/unix: Operation not supported
>
> Hello all,
>
> I see this too.
>
>> If you want to dig further you will have to try to grant the process>> the
>> CAP_NET_ADMIN capability, which might make it work.
>>
>> Capabilities are what allow you to do things like run ping as
>> non-root.>>
>> # sudo getcap /usr/bin/ping
>> /usr/bin/ping = cap_net_raw+ep
>>
>> So you can add the capability to the Cyrus binaries if it is
>> important>> to you; provided the feature is supported in the underlying OS.
>
> Is there a way to tell cyrus not to try to setsocketopt ? (Instead of>
> allowing it to do that?) I don't need the ToS bits to be set in any
> way
It looks like: only by patching the source.
/* set IP ToS if supported */
#if defined(SOL_IP) && defined(IP_TOS)
if (s->family == AF_INET || s->family == AF_INET6) {
{ "qosmarking", "cs0", ENUM("cs0", "cs1", "cs2", "cs3", "cs4", "cs5",
"cs6", "cs7", "af11", "af12", "af13", "af21", "af22", "af23", "af31",
"af32", "af33", "af41", "af42", "af43", "ef") }/* This specifies the Class
Selector or Differentiated Services
Code Point designation on IP headers (in the ToS field). */
Goodness! It's quite a lot of options! I guess the way to do it would
be to add a "none" option there and have that mean it doesn't try to
setsockopt.
Bron.
--
Bron Gondwana, CEO, FastMail Pty Ltd
br...@fastmailteam.com
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Missing .seen db files
Hi Eric, For the owner of the mailbox (or all users if shared seen is enabled), the seen state is stored as a system flags in the cyrus.index file. As for dealing with seen issues - you need to authenticate as that user or use a tool to introspect the index file (like mbdump or my index_uids.pl tool) to see the flags. Bron. On Sat, 3 Feb 2018, at 08:21, Eric Cunningham wrote: > Hi list, I had a user report that their read message status was > lost. I> recall having dealt with seen message db files in the past, but on > my> current implementation (2.5 on FreeBSD 10.2), I'm not able to > locate any> in or within my configdirectory, > /var/spool/CyrusDB-Sieve/cyrusdb. I> have > /var/spool/CyrusDB-Sieve/cyrusdb/user/u/username.sub but > not .seen.> Would they be located anywhere else or perhaps the seen > functionality> is contained in another database file? Without per-user > .seen > files, how can 'seen' issues be dealt with for individual > users? Thanks!> > -Eric > > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Adding archiving to an existing Cyrus installation
On Fri, 10 Nov 2017, at 00:02, Nic Bernstein wrote:
> Bron,
> Thanks for the response. Your solution has pointed us towards the
> proper approach. We are using ZFS, so would be performing ZFS
> send/receive replication rather than "mv", to move the filesystems.
> Our typical approach for this sort of thing, then, would be:>
> * Create a filesystem snapshot
>* zfs snapshot $ropt ${SOURCE_FS}@$newsnap
> * Perform ZFS send/receive, something like this:
>* zfs send -p $SENDVERBOSE $cloneorigin@$sourcestartsnap | zfs recv
> -u $RECVVERBOSE -F "$destfs"
> * Then, once we've completed a replication, we need to quiesce the
>filesystem and do the same, again, to catch-up to current state
That's basically exactly how FastMail's user moves work, and the sync-
protocol based XFER that Ken built based on it. Except inside Cyrus
rather than at the FS level.
> * Finally, replace the existing filesystem with the new replica, and
>discard the original> To quiesce the filesystem, we would normally like to
> tell whatever
> applications are using it to temporarily freeze operations, so the
> underlying filesystem is in a consistent state. When I was in
> Melbourne, back in April, we (you, Ellie & I) discussed what would be
> needed to introduce such a feature to Cyrus. I'm curious if there's
> been any further discussion or work on this? Should I open a feature
> request?
There has been discussion, not yet work.
https://github.com/cyrusimap/cyrus-imapd/issues/1763
> It would be nice to be able to complete consistent snapshots for
> filesystem operations like replication or backup, and this is a
> feature of many large applications, such as mail and DB servers.
Yep :) It's on our radar. There's a whole lot of work planned - Ken is
going to write up the notes on that at some point. We had a planning
meeting while he was in Australia.
> Thanks again for shining a welcome light on how to achieve the goal
> of adding archive to an existing system.>
> Cheers,
> -nic
>
>
> On 11/04/2017 10:59 PM, Bron Gondwana wrote:
>> Hi Nic,
>>
>> Sorry I didn't get back to answering you on this the other day!
>>
>> So... this one is kinda tricky, because everything is going to be on
>> "spool", but here's how I would do it.>>
>> Before:
>>
>> /mnt/smalldisk/conf -> meta files only
>> /mnt/bigdisk/spool -> all email right now
>>
>> Stage 1: splitting:
>>
>> /mnt/smalldisk/conf
>> /mnt/bigdisk/spool
>> /mnt/bigdisk/spool-archive -> empty
>>
>> And set archivepartition-default to /mnt/bigdisk/spoolarchive
>>
>> Now you need to run an initial cyr_expire. This will take a long
>> time, but it should be able to use hardlinks to move the files - it's
>> using cyrus_copyfile.>>
>> Once cyr_expire has finished an most of your email is moved into spool-
>> archive, shut down cyrus.>>
>> mv /mnt/bigdisk/spool /mnt/smalldisk/spool
>>
>> And set partition-default to /mnt/smalldisk/spool
>>
>> That way your downtime is only while the small remaining spool gets
>> moved to the other disk.>>
>> Bron.
>>
>>
>> On Sun, 5 Nov 2017, at 02:58, Nic Bernstein wrote:
>>> Thanks much to you both for your comments and suggestions. We had>>>
>>> already considered creating a temporary "staging" partition and
>>> shuffling mailboxes around, as Michael discussed, but have the same>>>
>>> reservations about it. Since we're dealing with nearly 6TB of data,>>>
>>> most of it old, this scheme would introduce considerable
>>> disruption to a>>> very active mail system. We have a hard time getting a
>>> two hour
>>> maintenance window, and this would take days!
>>>
>>> Bron, other Fastmailers, any thoughts??
>>> -nic
>>>
>>> On 11/03/2017 11:20 AM, Michael Menge wrote:
>>>> Hi,
>>>>
>>>> Quoting Reinaldo Gil Lima de Carvalho <reinal...@gmail.com>:
>>>>
>>>>> I think that singleinstancestore (message hard links) will not
>>>>> survive when
>>>>> moving from one partition to the other and storage total size will>>>>>
>>>>> increase
>>>>> significantly.
>>>>>
>>>>
>>>> thanks for the hint. This was not a problem while migration to the>>>>
>>>> meta-data partition,
>>>> as the mails stayed on the same partition (as in file-system
>>>> and
Re: Adding archiving to an existing Cyrus installation
put "archive" operation in effect. >>>>> >>>>> I have read the documentation (hell, I wrote most of it) and >>>>> understand >>>>> the settings, but what I cannot quite wrap my brain around is >>>>> this:>>>>> There >>>>> is already all of this data sitting in all of these data >>>>> partitions>>>>> (we use >>>>> a total of 34 separate partitions each for data & metadata) so how>>>>> >>>>> do I >>>>> make the transition to separate archive partitions, since all that>>>>> >>>>> data is >>>>> on the "slow" drives? Can I just reassign all of the current data>>>>> >>>>> partitions to archivedata partitions, define the new set of "Hot" >>>>> data>>>>> partitions on the Fast pool, and let 'er rip, or what? >>>>> >>>>> I promise, if you tell me, I'll write it up as real >>>>> documentation. :-)>>>>> >>>>> >>>> We are interested in such a migration too. Our fallback plan, if we>>>> >>>> don't >>>> find a >>>> better way to do it is, do use the same method as we introduced >>>> the ssd>>>> meta-data >>>> partition. >>>> >>>> >>>> 1. We created a new partition in our cyrus configuration, >>>> 2. we moved moved the accounts from one partition to the other one>>>> by >>>> one. >>>> 3. (this will be new for the archive partition) run cyrus expire to>>>> >>>> move >>>> the old mails back to the slow disks. >>>> >>>> This method will have two downsides. >>>> 1. we have to copy all mails to the fast storage, and move the old>>>> >>>> mails >>>>back to the slow storage. So we have to move most of the mails >>>> twice. >>>> 2. the path of the old mail will change so they will be stored >>>>again in>>>>our file based backup >>>> >>>> so a method without these downsides will be appreciated >>>> >>>> Regards >>>> >>>>Michael >>>> >>>> >>>> >>>> M.MengeTel.: (49) 7071/29-70316 >>>> Universität Tübingen Fax.: (49) 7071/29-5912 >>>> Zentrum für Datenverarbeitung mail: >>>> michael.me...@zdv.uni-tuebingen.de >>>> Wächterstraße 76 >>>> 72074 Tübingen >>>> >>>> >>>> Cyrus Home Page: http://www.cyrusimap.org/ >>>> List Archives/Info: >>>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/>>>> To Unsubscribe: >>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> >> >> >> - >> --->> >> M.MengeTel.: (49) 7071/29-70316 >> Universität Tübingen Fax.: (49) 7071/29-5912 >> Zentrum für Datenverarbeitung mail: >> michael.me...@zdv.uni-tuebingen.de >> Wächterstraße 76 >> 72074 Tübingen >> >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/>> To >> Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > -- > Nic Bernstein n...@onlight.com > Onlight Inc. www.onlight.com > 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 > Milwaukee, Wisconsin 53213-4073 f. 414.290.0335 > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus 3.0.4 - Over quota returns 550-Mailbox unknown
I have created https://github.com/cyrusimap/cyrus-imapd/issues/2191 to
track this. Thanks for the patch!
Cheers,
Bron.
On Fri, 3 Nov 2017, at 00:38, Edda wrote:
> Hi,
>
> that's surprising and the same here.
>
> I think there is a bug in imap/lmtpengine.c, function
> process_recipient:>
> verify_user returns the correct error code as we can see with
> LOG_DEBUG> "on":
>
> Nov 2 14:10:50 popc lmtp[2092]: verify_user(t...@sendmaid.org)
> failed:> Over quota
>
> but this return code is not used in process_recipient
>
> There we have:
> [...]
> if (sl) {
> char *rcpt = xstrndup(addr, sl);
> mbname = mbname_from_recipient(rcpt, msg->ns);
> free(rcpt);
>
> int forcedowncase =
> config_getswitch(IMAPOPT_LMTP_DOWNCASE_RCPT);> if
> (forcedowncase) mbname_downcaseuser(mbname);
>
> /* strip username if postuser */
> if (!strcmpsafe(mbname_localpart(mbname),
> config_getstring(IMAPOPT_POSTUSER))) {
> mbname_set_localpart(mbname, NULL);
> if (!config_virtdomains ||
> !strcmpsafe(mbname_domain(mbname), config_defdomain))
> mbname_set_domain(mbname, NULL);
> }
>
> if (verify_user(mbname,
> (quota_t) (ignorequota ? -1 : msg->size),
> ignorequota ? -1 : 1, msg->authstate)) {
> mbname_free();
> }
> }
>
> if (!mbname) {
> const char *catchall =
> config_getstring(IMAPOPT_LMTP_CATCHALL_MAILBOX);
> if (catchall) {
> mbname = mbname_from_userid(catchall);
> if (verify_user(mbname,
> ignorequota ? -1 : msg->size,
> ignorequota ? -1 : 1, msg->authstate)) {
> mbname_free();
> }
> }
> }
>
> if (!mbname) {
> /* we lost */
> return IMAP_MAILBOX_NONEXISTENT;
> }
> [...]
>
> means as far as I understand: if verify_user returns its error (for
> example IMAP_QUOTA_EXCEEDED), mbname is freed and process_recipient
> always returns IMAP_MAILBOX_NONEXISTENT.
>
> Below is a patch that works for me. But I don't know if this is a good> way
> to fix it.
> Hopefully one of the developers helps :)
>
> Regards, Edda
>
>
> diff -Naur cyrus-imapd-3.0.4.orig/imap/lmtpengine.c
> cyrus-imapd-3.0.4/imap/lmtpengine.c
> --- cyrus-imapd-3.0.4.orig/imap/lmtpengine.c2017-09-04
> 02:09:46.0 +0200
> +++ cyrus-imapd-3.0.4/imap/lmtpengine.c2017-11-02
> 13:59:56.764175245> +0100
> @@ -830,6 +830,7 @@
> }
>
> mbname_t *mbname = NULL;
> +int r = 0;
>
> size_t sl = strlen(addr);
> if (addr[sl-1] == '>') sl--;
> @@ -849,7 +850,7 @@
> mbname_set_domain(mbname, NULL);
> }
>
> -if (verify_user(mbname,
> +if (r = verify_user(mbname,
> (quota_t) (ignorequota ? -1 : msg->size),
> ignorequota ? -1 : 1, msg->authstate)) {
> mbname_free();
> @@ -860,7 +861,7 @@
> const char *catchall =
> config_getstring(IMAPOPT_LMTP_CATCHALL_MAILBOX);
> if (catchall) {
> mbname = mbname_from_userid(catchall);
> -if (verify_user(mbname,
> +if (r = verify_user(mbname,
> ignorequota ? -1 : msg->size,
> ignorequota ? -1 : 1, msg->authstate)) {>
> mbname_free();
> @@ -870,6 +871,9 @@
>
> if (!mbname) {
> /* we lost */
> +if (r) {
> +return r;
> +}
> return IMAP_MAILBOX_NONEXISTENT;
> }
>
>
>
> Am 02.11.17 um 09:27 schrieb Maros Vegh:
>> Hello,
>>
>> i'm using compiled cyrus version 3.0.4 on Debian 9.2.
>>
>> When the Postfix server is trying to deliver a message via lmtp to
>> cyrus mailbox which is over quota, it receives the 550-Mailbox
>> unknown>> return code and not the 452 or 552 Over quota.
>>
>> In the previous version 2.5.10 it worked fine with default values in>>
>> imapd.conf
>>
>> Is it a bug or my fault?
>>
>> Thanks
>>
>> Maros Vegh
>>
>>
>
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To
> Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Bron Gondwana, CEO, FastMail Pty Ltd
br...@fastmailteam.com
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Default value changes in Cyrus 3
Yes, if you change either it will force a re-download of messages and resync of mailbox listing. I would recommend setting it to the old values if you upgrade an existing server. You can also do something like we did at FastMail - set up a new hostname with the new defaults using per-service imapd.conf options, and recommend that users use the new hostname when setting up new clients. That way you don't break things for existing users. The reason for new defaults in a major version change is to allow new sites to have sane defaults for the future. Bron. On Fri, 20 Oct 2017, at 12:15, Stephen Ingram wrote: > I'm not sure I really know what that means. Would that change the > hierarchy? Force a download of all messages again on the client side?> > Steve > > On Thu, Oct 19, 2017 at 6:04 PM, Patrick Boutilier > <bouti...@ednet.ns.ca> wrote:>> On 10/19/2017 08:56 PM, Janos Dohanics wrote: >>> On Thu, 19 Oct 2017 16:43:48 -0700 Stephen Ingram >>> <sbing...@gmail.com> wrote: >>> >>>> While we are talking about it, can this just be switched on the fly >>>> if someone is using the "." namespace?>>> >>> Never tried it, but I read this page that the answer is yes. >>> >>> https://www.cyrusimap.org/imap/reference/admin/sop/altnamespace.html>> >> Yes, seems possible but there is this warning on >> https://www.cyrusimap.org/imap/concepts/features/namespaces.html#imap-admin-namespaces-mode>> >> >> >> >> Warning >> >> Changing altnamespace in an active operating environment will cause >> all IMAP clients to need to resync the entire hierarchy.>> >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: >> http://lists.andrew.cmu.edu/pipermail/info-cyrus/>> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: How to remove attachments from many mails on Cyrus-IMAP Store?
Hi Walter, The only way to remove attachments is to replace the message with an edited copy of the message. You can't alter emails in an IMAP mailstore once they are delivered. At FastMail we're using the attached Perl module to strip attachments from emails for our "remove attachments" feature. It's not pretty, but it gets the job done. It needs to be hooked up to some code that actually connects via IMAP and does the work. Cheers, Bron. On Thu, 19 Oct 2017, at 22:10, Walter H. via Info-cyrus wrote: > Hello, > > my energy supplier sends a daily mail about the electricity power > consumption of the previous day; > these mails have two attachment - one .csv and one .xml > > I'd like to remove the .xml attachments from the mails already > stored in> the cyrus database, as these are bigger and really not needed; > how would I achieve this? > (how to delete these from each file in the database is not problem) > > why I would like to do this: saving storage ... > > the directory has these files: > > 1. > 2. > 3. > ... > 10007. > 10008. > cyrus.cache > cyrus.header > cyrus.index > cyrus.squat > > my system: CentOS 6, Cyrus v2.3.16-Fedora-RPM-2.3.16-15.el6 > > Thanks, > Walter > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com StripAttachments.pm Description: Perl program Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Default value changes in Cyrus 3
Basically: 1) most of the rest of the world is using '/', so people expect it. Also, it means you can use '.' in mailbox names and in usernames, both of which people expect to be able to do due to other systems allowing it. 2) various clients expect to be able to create top level names and get sad if they don't. The reason not to use altnamespace in the past was that it didn't allow subfolders of INBOX, and there were a few other broken things about sort. They have now been fixed, so it's safe to make it the default. Both changes are to make the "out of the box" behaviour closer to what most people expect. Cheers, Bron. On Fri, 20 Oct 2017, at 00:44, Janos Dohanics wrote: > As I'm preparing to move to Cyrus 3, I have noticed that > unixhierarchysep (/) is now "on" as well as altnamespace is now "on".> > I have mostly been using default settings in previous versions, and I> guess > one shouldn't change sensible defaults willy-nilly, so I assume> developers > had good reasons to change these defaults? > > -- > Janos Dohanics > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyr_dbtool repack
Yes, cyr_dbtool repack is safe to use while Cyrus is running in all versions of Cyrus. Bron. On Tue, 17 Oct 2017, at 21:32, Wolfgang Breyha wrote: > Wolfgang Breyha wrote on 17/10/17 12:22: >> Hi! >> >> Is it safe to use >> cyr_dbtool (skiplist|twoskip) repack >> while cyrus is running? Or do I have to shut it down while >> repacking the>> databases? mycheckpoint() in both cyrus_skiplist.c and >> cyrus_twoskip.c seems>> to do locking and transaction checking as it seems. >> Still I'm not >> sure if it's>> safe to use while active. > > Sorry, forgot to mention... I'm running 2.5.x. > > Greetings, > Wolfgang > -- > Wolfgang Breyha <wbre...@gmx.net> | http://www.blafasel.at/ > Vienna University Computer Center | Austria > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: SASL login as another user - unexpected behavior on cyrus 2.4.17
That's going to be the case on all versions of Cyrus - admins have quite a lot of "root"-like power, and the @domain limitation is quite lightweight as you can see! I would recommend running a completely separate Cyrus instance per zone of control if this is a concern - while it would be nice to lock down all the ways in which admins are powerful, realistically it's a ton of work. (the usual "pull requests welcome" of course - I wouldn't object to making boundaries around domain admins solid, but I don't have the dev cycles to throw at it) Bron. On Mon, 25 Sep 2017, at 18:19, Marco wrote: > Hello, > > I run Cyrus-IMAPD 2.4.17 with many virtual domains: > > virtdomains: userid > > I configured a domain administrator: > > admins: ad...@example.com > > With this account I can LIST all accounts in example.com domain > only, as> expected. > > Let suppose the Cyrus-IMAPD server stores also accounts for other > domains, such as example2.com domain. > > Well, I see that I can SASL PLAIN login using ad...@example.com on > example2.com accounts too, if I know their names. I can't > understand why> this could happen. It seems a security issue. > > > Is there a way to prevent this issue without modifying ACL on all > accounts? > > Thank you > Marco > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
CVE reported for Cyrus 3.0.0 - 3.0.3
Hi All, I have obtained CVE-2017-14230 for the crasher in Cyrus up to 3.0.3 where: tag FIND "" "Other Users" Would cause uninitialised memory to be written to a buffer which was then interpreted as an unbounded C string. This bug is fixed in 3.0.4, and we recommend everybody upgrade. Regards, Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: update to 3.0.3 even more problematic than 3.0.2
https://github.com/cyrusimap/cyrus-imapd/issues/2132 We're pretty sure we've found the cause. Sorry about not being active in the thread earlier, it's been a busy couple of weeks at FastMail and we haven't had time to deal with everything! Bron. On Sat, 26 Aug 2017, at 23:57, Gabriele Bulfon wrote: > Tried a trick, but it it did not work... > > Just used cyradm to : > > > cm "Other Users" > > and then I have that folder really existing! > > It works just the first time I try A01 LIST "" "Other Users" > But then it fails later... > > looks like some kind of cache trying to work out this phantom > folder> > > *Sonicle S.r.l. *: http://www.sonicle.com[1] > *Music: *http://www.gabrielebulfon.com[2] > *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon > > > > *From:* Gabriele Bulfon <gbul...@sonicle.com> *To:* Arnaldo Viegas de > Lima <arna...@viegasdelima.com> *Cc:* info-cyrus@lists.andrew.cmu.edu > *Date:* August 25, 2017 5:24:13 PM CEST *Subject:* Re: update to 3.0.3 > even more problematic than 3.0.2> > >> FYI, upgrading from 2.4.12 to 2.5.11, and then to 3.0.3 (so, after >> that 2.5.11 converted the db files), I got a clean startup from the >> beginning, with no hassle on conversions.>> >> Still, I have the problem with LIST "" "Other Users" returning >> nothing :(>> As I mentioned, if I run LIST "" "Other Users/%" to get its >> subfolders, then previous commands return the correct answer all >> the times.>> But I also found that if I select a folder (e.g. SELECT INBOX), >> then >> again that command returns nothing again.>> Maybe this can suggest what's >> happening? >> >> >> *Sonicle S.r.l. *: http://www.sonicle.com[3] >> *Music: *http://www.gabrielebulfon.com[4] >> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon >> >> >> >> *From:* Gabriele Bulfon <gbul...@sonicle.com> *To:* Arnaldo Viegas de >> Lima <arna...@viegasdelima.com> *Cc:* info-cyrus@lists.andrew.cmu.edu >> *Date:* August 25, 2017 4:54:20 PM CEST *Subject:* Re: update to >> 3.0.3 even more problematic than 3.0.2>> >> >>> That may be a good try :) >>> Anyway I tried building / upgrading to 2.5.11, it converted my files >>> in twoskip with no problems, and the shared folders work fine.>>> >>> Pity, I see it doesn't implement "THREAD=REFS" nor xapian ... :( >>> >>> I will try this version a little, then will try upgrading from this >>> to 3.0.x.>>> >>> >>> *Sonicle S.r.l. *: http://www.sonicle.com[5] >>> *Music: *http://www.gabrielebulfon.com[6] >>> *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon >>> >>> >>> >>> >>> -- >>> >>> From: Arnaldo Viegas de Lima <arna...@viegasdelima.com> To: Gabriele >>> Bulfon <gbul...@sonicle.com> Cc: info-cyrus@lists.andrew.cmu.edu >>> Date: August 25, 2017 4:32:17 PM CEST Subject: Re: update to 3.0.3 >>> even more problematic than 3.0.2 >>>>>>> Have you tried 3.0.3 (or 3.0.2) on a clean system just to be sure >>>> that your compile/install is working? >>>> >>> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: >>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: >>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >>> >> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: >> http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >>> > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Links: 1. http://www.sonicle.com/ 2. http://www.gabrielebulfon.com/ 3. http://www.sonicle.com/ 4. http://www.gabrielebulfon.com/ 5. http://www.sonicle.com/ 6. http://www.gabrielebulfon.com/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: list of IMAP extensions
. OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA MAILBOX- REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS ANNOTATE-EXPERIMENT-1 METADATA LIST- EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1 X- REPLICATION XAPPLEPUSHSERVICE LOGINDISABLED XCONVERSATIONS COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION- STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] That's a login against current Cyrus master - but 3.0 supports all them too. Note that these are custom/non-standard: * X-REPLICATION (Cyrus only) * XAPPLEPUSHSERVICE (Apple non-standard, requires signed cert and NDA to implement)* XCONVERSATIONS (Cyrus/FastMail only) * X-QUOTA =STORAGE =MESSAGE =X-ANNOTATION-STORAGE =X-NUM-FOLDERS (only STORAGE and MESSAGE are defined by standard) * XMOVE (replaced by MOVE) * DIGEST=SHA1 (Cyrus/FastMail only) * LIST-MYRIGHTS (draft) Off the top of my head that's the stuff that's custom. Cyrus definitely supports (since 2.4 at least) all of those. There was some rudimentary QRESYNC support in 2.3.x, but it was buggy. Bron. On Fri, 23 Jun 2017, at 08:34, Stephen Ingram wrote: > On Thu, Jun 22, 2017 at 1:08 PM, Stephen Ingram > <sbing...@gmail.com> wrote:>> On Thu, Jun 22, 2017 at 12:55 PM, Adam Tauno > Williams >> <awill...@whitemice.org> wrote:>> >>> Quoting Stephen Ingram <sbing...@gmail.com>: >>>> Is there a comprehensive list of IMAP extensions supported by >>>> Cyrus-IMAP >>>> 2.4.x and 3.x? Not the RFCs, but the actual extensions like >>>> QRESYNC and THREAD=REFS.>>> >>> So nothttps://cyrusimap.org/imap/rfc-support.html ? >> >> >> No. I had already found that, but would love it if the extensions >> were listed directly. I'm trying to verify product support for Cyrus- >> IMAP and would love to have a quick checklist. I could sworn I've >> seen something like this before, but can't seem to locate it now.> > I forgot to include exactly the extensions I'm looking for: > > IDLE > QRESYNC > CONDSTORE > CHILDREN > THREAD=REFS > UIDPLUS > SORT > UID SORT > LIST-STATUS > SPECIAL-USE > MOVE > > I know that IDLE is supported and it looks like CONDSTORE can be > turned on using cyradm on an account-by-account basis, but I'm not > totally sure about the rest. Does anyone have an idea?> > Steve > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/> To > Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: master & won't start
If you look in syslog, it should tell you what the problem was. On Thu, 22 Jun 2017, at 15:32, haider al-shook via Info-cyrus wrote: > I cannot start master process beacause it exit with code 71 > Cyradim also does not work giving the error that it couldnot connect to server > Imtest also does not work mentioning that network initializatio failed > What is wrong is it imapd.conf or cyrus.conf or something else > Thanks > Sent from my iPhone > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: sync_client problems
Just read the 148 file and see if there is corruption. I suspect that's the cause. Bron On Fri, 12 May 2017, at 23:09, Eric Cunningham wrote: > I have snapshot backups of that account, that particular message and the > various cyrus db files. What can I provide for review? > > > On 5/11/17 7:38 PM, Bron Gondwana wrote: > > It looked like the '148.' file was corrupted in some way. I assume you've > > deleted all the evidence now, so we can't see how! Oh well :( > > > > Bron. > > > > On Fri, 12 May 2017, at 04:55, Eric Cunningham wrote: > >> Just to close the loop on this, once the corrupted cdm-lit account on my > >> copy server successfully reconstructed, the backlog of replication has > >> completed successfully and is now caught up. > >> > >> -Eric > >> > >> On 05/11/2017 02:20 PM, Eric Cunningham wrote: > >>> -G didn't seem to help, but I tried a "reconstruct -f -r" command > >>> without having previously deleted all occurrences of the cyrus.* files > >>> and it was then successful. > >>> > >>> > >>> > >>> On 05/11/2017 01:44 PM, Patrick Boutilier wrote: > >>>> Try a -G with reconstruct? > >>>> > >>>> -G Force re-parsing of the underlying message (checks GUID > >>>> correctness). Reconstruct with -G should fix all possible individual > >>>> message issues, including corrupted data files. > >>>> > >>>> > >>>> On 05/11/2017 02:37 PM, Eric Cunningham wrote: > >>>>> I have to walk this back. In looking slightly further back in my > >>>>> logfiles, before every instance of a failure to sync some folder, I > >>>>> see a common error reported prior to every "bailing out! Bad > >>>>> protocol" error - "IOERROR: GUID mismatch > >>>>> /var/spool/cyrus/mail/c/user/cdm-lit/Sent/148." > >>>>> > >>>>> May 11 12:44:09 imap1 sync_client[48590]: IOERROR: GUID mismatch > >>>>> /var/spool/cyrus/mail/c/user/cdm-lit/Sent/148. > >>>>> May 11 12:44:22 imap1 sync_client[48590]: IOERROR: zero length > >>>>> response to MAILBOX (end of file reached) > >>>>> May 11 12:44:22 imap1 sync_client[48590]: do_folders(): update > >>>>> failed: user.cdm-lit.Sent 'Bad protocol' > >>>>> May 11 12:44:22 imap1 sync_client[48590]: sync_mailboxes: doing 1000 > >>>>> May 11 12:44:22 imap1 sync_client[48590]: IOERROR: zero length > >>>>> response to MAILBOXES (end of file reached) > >>>>> May 11 12:44:22 imap1 sync_client[48590]: sync_mailboxes: doing 1000 > >>>>> May 11 12:44:22 imap1 sync_client[48590]: IOERROR: zero length > >>>>> response to MAILBOXES (end of file reached) > >>>>> May 11 12:44:22 imap1 sync_client[48590]: sync_mailboxes: doing 1000 > >>>>> May 11 12:44:22 imap1 sync_client[48590]: IOERROR: zero length > >>>>> response to MAILBOXES (end of file reached) > >>>>> May 11 12:44:22 imap1 sync_client[48590]: sync_mailboxes: doing 1000 > >>>>> May 11 12:44:22 imap1 sync_client[48590]: IOERROR: zero length > >>>>> response to MAILBOXES (end of file reached) > >>>>> May 11 12:44:22 imap1 sync_client[48590]: sync_mailboxes: doing 1000 > >>>>> May 11 12:44:22 imap1 sync_client[48590]: IOERROR: zero length > >>>>> response to MAILBOXES (end of file reached) > >>>>> May 11 12:44:22 imap1 sync_client[48590]: sync_mailboxes: doing 1000 > >>>>> May 11 12:44:22 imap1 sync_client[48590]: IOERROR: zero length > >>>>> response to MAILBOXES (end of file reached) > >>>>> May 11 12:44:22 imap1 sync_client[48590]: IOERROR: zero length > >>>>> response to MAILBOXES (end of file reached) > >>>>> May 11 12:44:22 imap1 sync_client[48590]: IOERROR: zero length > >>>>> response to UNMAILBOX (end of file reached) > >>>>> May 11 12:44:22 imap1 sync_client[48590]: folder_delete(): failed: > >>>>> user.guest-student-coordinator.Trash.Aarflot 'Bad protocol' > >>>>> May 11 12:44:22 imap1 sync_client[48590]: Error in do_sync(): bailing > >>>>> out! Bad protocol > >>>>> May 11 12:44:22 imap1 sync_client[48590]: Processing sync log file > >>>>> /var/spool/CyrusDB-Sieve/cyrusdb/sync/log-run failed:
Re: sync_client problems
> record->size > >>> > >>> Since I couldn't get a successful reconstruct on this account, I > >>> deleted it and recreated it from my master host. However, I'm still > >>> unable to get a successful reconstruct with "failed to read index > >>> header" for every subfolder and "fatal error: Internal error: > >>> assertion failed: imap/mailbox.c: 2847: record->size" > >>> > >>> Any ideas on how to correct this so I can see if I can then get past > >>> the replication error? > >>> > >>> Thanks! > >>> > >>> -Eric > >>> > >>> > >>> On 05/11/2017 11:46 AM, Eric Cunningham wrote: > >>>> Thanks Bron, but that doesn't seem to work for me, unless I'm > >>>> missing something. > >>>> > >>>> I ran reconstructs for this account on both my master and copy hosts: > >>>> > >>>> [cyrus@imap1 ~]$ reconstruct -f -r user.scramer > >>>> user.scramer > >>>> user.scramer.Archive > >>>> user.scramer.Archives > >>>> user.scramer.Archives.2004 > >>>> ... > >>>> user.scramer.Trash.IS Networking and Operations > >>>> user.scramer.Trash.IS Security > >>>> user.scramer.Trash.IS Servers > >>>> user.scramer.Trash.IS Staff > >>>> user.scramer.Trash.IS Surveys > >>>> ... > >>>> > >>>> > >>>> [cyrus@imap2 ~]$ reconstruct -f -r user.scramer > >>>> user.scramer > >>>> user.scramer.Archive > >>>> user.scramer.Archives > >>>> user.scramer.Archives.2004 > >>>> ... > >>>> user.scramer.Trash.IS Networking and Operations > >>>> user.scramer.Trash.IS Security > >>>> user.scramer.Trash.IS Servers > >>>> user.scramer.Trash.IS Staff > >>>> user.scramer.Trash.IS Surveys > >>>> ... > >>>> > >>>> > >>>> I then restart the replication against the log-run file and it again > >>>> fails on a folder that no longer exists in that account: > >>>> > >>>> ... > >>>> May 11 11:40:51 imap1 sync_client[60696]: sync_mailboxes: doing 1000 > >>>> May 11 11:40:51 imap1 sync_client[60696]: IOERROR: zero length > >>>> response to MAILBOXES (end of file reached) > >>>> May 11 11:40:51 imap1 sync_client[60696]: IOERROR: zero length > >>>> response to MAILBOXES (end of file reached) > >>>> May 11 11:40:51 imap1 sync_client[60696]: IOERROR: zero length > >>>> response to UNMAILBOX (end of file reached) > >>>> May 11 11:40:51 imap1 sync_client[60696]: folder_delete(): failed: > >>>> user.scramer.Trash.IS Software Management 'Bad protocol' > >>>> May 11 11:40:51 imap1 sync_client[60696]: Error in do_sync(): > >>>> bailing out! Bad protocol > >>>> May 11 11:40:51 imap1 sync_client[60696]: Processing sync log file > >>>> /var/spool/CyrusDB-Sieve/cyrusdb/sync/log-run failed: Bad protocol > >>>> > >>>> > >>>> -Eric > >>>> > >>>> > >>>> On 05/11/2017 02:14 AM, Bron Gondwana wrote: > >>>>> Looks like you have a corrupted mailboxes database - if you run a > >>>>> reconstruct at each end then it should be fine. Replication won't > >>>>> bail for an actually-doesn't-exist mailbox, but it can't fix broken > >>>>> mailboxes, that's what reconstruct is for. > >>>>> > >>>>> Bron. > >>>>> > >>>>> > >>>>> On Thu, 11 May 2017, at 01:01, Eric Cunningham wrote: > >>>>>> Hi list, I'm running cyrus replication on cyrus 2.5.10. It seems > >>>>>> whenever a folder is encountered in log-run that doesn't exist on the > >>>>>> client and/or the server, the replication process logs "Error in > >>>>>> do_sync(): bailing out! Bad protocol". Sometimes, it dies > >>>>>> completely, > >>>>>> other times it restarts at the beginning of the log-run file. But > >>>>>> when > >>>>>> it re-encounters the folder in question, the same failure occurs > >>>>>> and the > >>>>>> problem co
Re: sync_client problems
Looks like you have a corrupted mailboxes database - if you run a reconstruct at each end then it should be fine. Replication won't bail for an actually-doesn't-exist mailbox, but it can't fix broken mailboxes, that's what reconstruct is for. Bron. On Thu, 11 May 2017, at 01:01, Eric Cunningham wrote: > Hi list, I'm running cyrus replication on cyrus 2.5.10. It seems > whenever a folder is encountered in log-run that doesn't exist on the > client and/or the server, the replication process logs "Error in > do_sync(): bailing out! Bad protocol". Sometimes, it dies completely, > other times it restarts at the beginning of the log-run file. But when > it re-encounters the folder in question, the same failure occurs and the > problem continuously loops until some manual action is taken. I have a > checker script that attempts to restart a stopped sync but I haven't > tried to code it to get past these types of folder errors. In the > meantime, replication effectively stops and the replication log > continuously builds in size. You might imagine my horror at discovering > yesterday that replication of my production server with 1200 accounts > had been failing for over 7 weeks unnoticed, generating 9.5 million > lines in the replication log files. I had gotten out of the habit of > daily monitoring of replication as it had been running fine and I > eventually moved on to other things. I'm trying to get replication to > run through the enormous backlog but am running into more similar folder > issues that cause the process to fail, I make manual fixes, and start > the process again. In my experience with this, reconstruction of the > affected folders on the master host and copy host has no effect. > > Can this otherwise excellent feature be made more robust to work past > these folder issues? > > Here are a couple of examples I've observed thus far of how the process > is failing for me. > > -- > > In this example, the subfolder "Fly America Foreign Travel" no longer > existed on the master host but did exist on the copy host. To resolve > this, I had to manually delete that folder from the copy host. > > May 9 15:17:48 imap1 sync_client[34614]: inefficient replication (8 > > 5) user.jtaft.PO.Fly America Foreign Travel > May 9 15:17:48 imap1 sync_client[34614]: FULLMAILBOX received NO > response: IMAP_MAILBOX_NONEXISTENT No Such Mailbox > May 9 15:17:48 imap1 sync_client[34614]: do_folders(): update failed: > user.jtaft.PO.Fly America Foreign Travel 'Mailbox does not exist' > May 9 15:17:48 imap1 sync_client[34614]: IOERROR: Mailbox does not exist > May 9 15:17:48 imap1 sync_client[34614]: Error in do_sync(): bailing > out! Mailbox does not exist > May 9 15:17:48 imap1 sync_client[34614]: Processing sync log file > /var/spool/CyrusDB-Sieve/cyrusdb/sync/log-run failed: Mailbox does not exist > May 9 15:17:48 imap1 sync_client[34614]: Reprocessing sync log file > /var/spool/CyrusDB-Sieve/cyrusdb/sync/log-run > > -- > > In this example, the subfolder "Sohn" did not exist on the master host > nor on the copy host. To resolve, I had to edit log-run and completely > remove all entries for this "Sohn" subfolder. > > May 10 10:13:26 imap1 sync_client[47011]: IOERROR: zero length response > to MAILBOXES (end of file reached) > May 10 10:13:26 imap1 sync_client[47011]: IOERROR: zero length response > to MAILBOXES (end of file reached) > May 10 10:13:26 imap1 sync_client[47011]: IOERROR: zero length response > to UNMAILBOX (end of file reached) > May 10 10:13:26 imap1 sync_client[47011]: folder_delete(): failed: > user.gg-chair.Trash.Sohn 'Bad protocol' > May 10 10:13:26 imap1 sync_client[47011]: Error in do_sync(): bailing > out! Bad protocol > May 10 10:13:26 imap1 sync_client[47011]: Processing sync log file > /var/spool/CyrusDB-Sieve/cyrusdb/sync/log-run failed: Bad protocol > > -- > > Often, errors are logged without "bailing out!" and replication > continues to run: > > May 10 10:45:00 imap1 sync_client[87438]: sync_mailboxes: doing 1000 > May 10 10:45:01 imap1 sync_client[87438]: sync_mailboxes: doing 1000 > May 10 10:45:24 imap1 sync_client[87438]: IOERROR: GUID mismatch > /var/spool/cyrus/mail/c/user/cdm-lit/Sent/148. > May 10 10:45:37 imap1 sync_client[87438]: IOERROR: zero length response > to MAILBOX (end of file reached) > May 10 10:45:37 imap1 sync_client[87438]: do_folders(): update failed: > user.cdm-lit.Sent 'Bad protocol' > May 10 10:45:37 imap1 sync_client[87438]: sync_mailboxes: doing 1000 > > -- > > Thank you, list! > > -Eric > > > Cyrus Home Page: http://
Re: Migrating from Dovecot to Cyrus
Oooh... I have a bunch of code which speaks replication protocol that does that which we used for the Pobox => FastMail migration. I'll see if I can dig it out and publish it. There's a setting to make Cyrus use Dovecot-compatible UIDLs. Bron. On Fri, 28 Apr 2017, at 00:01, Matt Campbell wrote: > Hello: > > What is the recommended way to migrate mailboxes from Dovecot to Cyrus, > ideally preserving IMAP UIDs and POP3 UIDLs? This seems like it should > be a FAQ, but I haven't been able to find a good answer. > > Thanks, > Matt > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Users cannot access to their imap-folders
What is Cyrus syslogging?
On Tue, 11 Apr 2017, at 15:16, Anton Shilov via Info-cyrus wrote:
> Users cannot access to their imap-folders.
> Sometimes (once or twice a week) users complains that they cannot
> connect to their mailboxes and read emails. At the same time I see
> such records in maillog:
> "Apr 10 15:01:01 mailserv postfix/lmtp[22843]: D64651001D37:
> to=<u...@mailserv.example.com[1]>, orig_to=<u...@example.com[2]>,
> relay=mailserv.example.com[/var/lib/imap/socket/lmtp], delay=300,
> delays=0.13/0/0/300, dsn=4.4.2, status=deferred (conversation with
> mailserv.example.com[/var/lib/imap/socket/lmtp] timed out while
> sending MAIL FROM)». It means mail queue increases
> Also I find out lots of "PID-files" in /dev/shm/cyrus-imapd/proc which
> don't contain info of linked imap-folder. For example: "pc-
> user.example.com [192.168.0.11] user"
> It's unhealthy situation and I don't understannd what reason is. In
> order to help users I just kill their IMAP-process and make
> reconstruct mailbox. Please help me with investigation of the
> problems.
> My setup is:
>
> Static hostname: mailserv Icon name: computer-vm Chassis: vm Machine
> ID: cbb779ad0b103a10ec75c10c452e0f71 Boot ID:
> c31a27599dc84971902e6c70dca26c18 Virtualization: kvm Operating System:
> CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel:
> Linux 3.10.0-514.6.1.el7.x86_64 Architecture: x86-64
>
> name : Cyrus IMAPD version : v2.4.17-Fedora-RPM-2.4.17-8.el7_1
> d1df8aff 2012-12-01 vendor : Project Cyrus support-url:
> http://www.cyrusimap.org[3] os : Linux os-version : 3.10.0-
> 514.6.1.el7.x86_64 environment: Built w/Cyrus SASL 2.1.26 Running
> w/Cyrus SASL 2.1.26 Built w/Berkeley DB 5.3.21: (May 11, 2012) Running
> w/Berkeley DB 5.3.21: (May 11, 2012) Built w/OpenSSL 1.0.1e-fips 11
> Feb 2013 Running w/OpenSSL 1.0.1e-fips 11 Feb 2013 Built w/zlib 1.2.7
> Running w/zlib 1.2.7 CMU Sieve 2.4 TCP Wrappers NET-SNMP mmap = shared
> lock = fcntl nonblock = fcntl idle = idled
> Server Wide: freespace: 1820105352
>
> imapd.conf
>
> debug: no commandmintimer: 2 configdirectory: /var/lib/imap partition-
> default: /var/spool/mail proc_path: /dev/shm/cyrus-imapd/proc
> mboxname_lockpath: /dev/shm/cyrus-imapd/mboxname_lock
> duplicate_db_path: /dev/shm/cyrus-imapd/duplicate_db
> statuscache_db_path: /dev/shm/cyrus-imapd/statuscache_db
> allowplaintext: yes allowallsubscribe: yes allowusermoves: yes
> imapidlepoll: 10 idlemethod: idled imapidresponse: yes admins: cyrus
> lmtp_admins: cyrus singleinstancestore: yes duplicatesuppression: yes
> reject8bit: no pass8bit: yes munge8bit: no lmtp_downcase_rcpt: no
> sieveusehomedir: false sievedir: /var/lib/imap/sieve duplicate_db:
> skiplist mboxlist_db: skiplist postmaster: postmaster postuser: shared
> lmtp_over_quota_perm_failure: yes lmtp_strict_quota: yes quotawarn: 98
> sendmail: /usr/sbin/sendmail hashimapspool: 0 sasl_pwcheck_method:
> saslauthd sasl_mech_list: PLAIN LOGIN tls_cert_file:
> /var/lib/imap/SSL.crt tls_key_file: /var/lib/imap/SSL.key tls_ca_file:
> /var/lib/imap/CA_SSL.crt tls_ca_path: /var/lib/imap tcp_keepalive: yes
> tcp_keepalive_cnt: 3 tcp_keepalive_intvl: 10 tcp_keepalive_idle: 180
> temp_path: /tmp telemetry_bysessionid: yes timeout: 40
> imapidletimeout: 0 expunge_mode: delayed expunge_days: 30
> deletedprefix: DELETED delete_mode: immediate flushseenstate: 1
> ---
> cyrus.conf
> ---
> START { recover<->cmd="ctl_cyrusdb -r"
> idled><-->cmd="idled"
> }
> SERVICES { imap cmd="imapd" listen="imap" proto=tcp4 prefork=5 imaps
> cmd="imapd -s" listen="imaps" proto=tcp4 prefork=10 pop3 cmd="pop3d"
> listen="pop3" proto=tcp4 prefork=3 pop3s cmd="pop3d -s" listen="pop3s"
> proto=tcp4 prefork=3 sieve cmd="timsieved" listen="sieve" proto=tcp4
> prefork=0 lmtpunix<>cmd="lmtpd" listen="/var/lib/imap/socket/lmtp"
> prefork=1 .. }
> EVENTS { checkpoint<-->cmd="ctl_cyrusdb -c" period=30 delprune<
> >cmd="cyr_expire -E 3" at=0400 tlsprune<>cmd="tls_prune" at=0400
> squatter cmd="squatter -i" at=0600
> }
>
>
> Best regards, Anton
> 2017-Apr-11
>
>
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Bron Gondwana
br...@fastmail.fm
Links:
1. https://e.mail.ru/compose?To=u...@mailserv.example.com
2. https://e.mail.ru/compose?To=u...@example.com
3. http://www.cyrusimap.org/
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyradm not showing metadata if logged as admin
Not really, no. It's the same problem we have with the \Seen flag, which is
private to each user.
Bron.
On Fri, 7 Apr 2017, at 12:32, ellie timoney wrote:
> Hi Olaf,
>
> I think this is expected behaviour. Mailbox annotations in the
> "/private/" namespace are per user -- such that if multiple users have
> access to the same mailbox, they can each have their own annotations on
> it.
>
> So, when you log into this account with IMAP and set /private/specialuse
> annotations, you're setting those annotations for the user you logged in
> as.
>
> When you log in as an admin and look at the same mailbox, you're seeing
> the admin's annotations (which you haven't set any), not the user's.
>
> I don't know if there's an effective way to do what you want. Can
> anyone else chime in?
>
> Cheers,
>
> ellie
>
> On Fri, Apr 7, 2017, at 12:29 AM, Olaf Frączyk wrote:
> > Hello,
> >
> > cyrus-imapd-2.5.10-2.3.el7.x86_64 from open build system
> >
> > I have set metadata for mailboxes:
> >
> > t3 SETMETADATA "INBOX/Sent" (/private/specialuse "\\Sent")
> >
> > I can see the metadata from imap connection and from cyradm but only
> > when logged as user:
> >
> > t3 LIST (SPECIAL-USE) "" "*"
> > * LIST (\HasNoChildren \Drafts) "/" INBOX/Drafts
> > * LIST (\HasNoChildren \Sent) "/" INBOX/Sent
> > * LIST (\HasNoChildren \Trash) "/" INBOX/Trash
> > * LIST (\HasNoChildren \Junk) "/" INBOX/spam
> >
> > 192.168.1.8> getmd INBOX/Sent
> > {INBOX/Sent}:
> >private:
> > check: NIL
> > checkperiod: NIL
> > comment: NIL
> > sort: NIL
> > specialuse: \Sent
> > thread: NIL
> > expire: NIL
> > news2mail: NIL
> > sieve: NIL
> > squat: NIL
> >
> > When logged as cyrus admin I get:
> >
> > 192.168.1.8> getmd user/info/s...@navi.pl
> > {user/info/s...@navi.pl}:
> >private:
> > check: NIL
> > checkperiod: NIL
> > comment: NIL
> > sort: NIL
> > specialuse: NIL
> > thread: NIL
> > expire: NIL
> > news2mail: NIL
> > sieve: NIL
> > squat: NIL
> >
> > I want to be able to set the metadata for users' mailboxes, so the
> > Outlook and Thunderbird use correct folders.
> >
> > I tried to give the admin full ACL rights for this mailbox but it didn't
> > help.
> >
> > Is there any configuration option to change this behaviour?
> >
> > Best regards,
> >
> > Olaf Frączyk
> >
> > --
> > NAVI Sp. z o.o.
> > Promienista 5/1
> > 60-288 Poznań
> >
> > mobile: +48609769035
> > phone: +48616622881
> > fax: +48616622882
> > http://www.navi.pl
> >
> >
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Bron Gondwana
br...@fastmail.fm
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: How to retrieve unread status from backup ?
Nicola, this might be a nice thing to have in docs somewhere. it's non-obvious just from reading specs, but it is a very nice side effect of esearch. Bron On Fri, 31 Mar 2017, at 08:43, Xavier Bestel wrote: > ... aaand it worked like a charm ! > I spent way more time setting up a new serveur from the backup than > doing the flags restore. Thanks again ! > > Xav > > Le jeudi 30 mars 2017 à 09:49 +0200, Xavier Bestel a écrit : > > Thanks a lot Bron, precisely the kind of thing I was looking for :) > > > > Regards, > > Xav > > > > Le mercredi 29 mars 2017 à 12:12 -0500, Bron Gondwana a écrit : > > > Actually, if you restore from backup to another Cyrus instance and > > > can log in it's very easy using search-extended: > > > > > > telnet restoreserver 143 > > > . login username password > > > . select INBOX > > > . UID SEARCH RETURN () UNSEEN > > > * ESEARCH (TAG ".") UID ALL > > > 1015337,1015340:1015366,1015378:1015379,1015381,1015384 > > > . logout > > > > > > telnet productionserver 143 > > > . login username password > > > . select INBOX > > > . uid store 1015337,1015340:1015366,1015378:1015379,1015381,1015384 > > > -flags \Seen > > > . logout > > > > > > And you're done :) Repeat the search for any other mailbox. > > > > > > Bron. > > > > > > On Wed, 29 Mar 2017, at 10:34, James Cassell wrote: > > > > On Wed, Mar 29, 2017, at 04:18 AM, Xavier Bestel wrote: > > > > > Hi, > > > > > > > > > > I have done an unfortunate "mark all as read" on my mailbox, > > > > > and > > > > > I'm > > > > > not one of those "mailbox zero" guys. So I have lost some > > > > > valuable > > > > > information. But I have some (old) backups. > > > > > > > > > > So my question is: how do I retrieve the "read/unread" status > > > > > from the > > > > > (not so recent) backup, and how can I reapply it to my running > > > > > server > > > > > (which continues to receive mails regularly) ? > > > > > > > > > > > > > > > > > You could do it with some invocations of imapsync. The quickest > > > > way would also restore any messages you may have deleted since > > > > your > > > > backup. Do an imapsync run with your backup server as "host1" > > > > and > > > > your live server as "host2". > > > > > > > > How I'd do it would be to take several steps: > > > > 1. Sync from your live server to a temporary server (or temp > > > > folder > > > > on the same server). > > > > 2. Apply a unique imap flag to all messages in the temporary > > > > server > > > > 3. sync from your backup to the temporary server. This will > > > > clear > > > > the unique flag on sync'ed messages. > > > > 4. delete from the temporary server all messages that still have > > > > the unique flag. > > > > 5. sync from the temporary server back to the live server. > > > > 6. delete the temporary server > > > > > > > > Hope that helps! > > > > > > > > V/r, > > > > James Cassell > > > > > > > > Cyrus Home Page: http://www.cyrusimap.org/ > > > > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cy > > > > ru > > > > s/ > > > > To Unsubscribe: > > > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > > > > > > > > > > Cyrus Home Page: http://www.cyrusimap.org/ > > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > > To Unsubscribe: > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: How to retrieve unread status from backup ?
Actually, if you restore from backup to another Cyrus instance and can log in it's very easy using search-extended: telnet restoreserver 143 . login username password . select INBOX . UID SEARCH RETURN () UNSEEN * ESEARCH (TAG ".") UID ALL 1015337,1015340:1015366,1015378:1015379,1015381,1015384 . logout telnet productionserver 143 . login username password . select INBOX . uid store 1015337,1015340:1015366,1015378:1015379,1015381,1015384 -flags \Seen . logout And you're done :) Repeat the search for any other mailbox. Bron. On Wed, 29 Mar 2017, at 10:34, James Cassell wrote: > On Wed, Mar 29, 2017, at 04:18 AM, Xavier Bestel wrote: > > Hi, > > > > I have done an unfortunate "mark all as read" on my mailbox, and I'm > > not one of those "mailbox zero" guys. So I have lost some valuable > > information. But I have some (old) backups. > > > > So my question is: how do I retrieve the "read/unread" status from the > > (not so recent) backup, and how can I reapply it to my running server > > (which continues to receive mails regularly) ? > > > > > You could do it with some invocations of imapsync. The quickest way would > also restore any messages you may have deleted since your backup. Do an > imapsync run with your backup server as "host1" and your live server as > "host2". > > How I'd do it would be to take several steps: > 1. Sync from your live server to a temporary server (or temp folder on the > same server). > 2. Apply a unique imap flag to all messages in the temporary server > 3. sync from your backup to the temporary server. This will clear the unique > flag on sync'ed messages. > 4. delete from the temporary server all messages that still have the unique > flag. > 5. sync from the temporary server back to the live server. > 6. delete the temporary server > > Hope that helps! > > V/r, > James Cassell > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyr_expire suddenly segfaulting
You can wind up with a broken Berkeley environment without even any databases
defined! I'm glad we've removed it entirely now :)
Bron.
On Mon, 13 Mar 2017, at 23:36, Kenneth Marshall wrote:
> On Mon, Mar 13, 2017 at 11:32:38AM +1100, Bron Gondwana wrote:
> > So everything is 2.4 defaults, which are:
> >
> > brong@wot:~/src/cyrus-imapd$ grep '_db"' lib/imapoptions
> > { "annotation_db", "skiplist", STRINGLIST("berkeley", "berkeley-hash",
> > "skiplist")}
> > { "duplicate_db", "skiplist", STRINGLIST("berkeley", "berkeley-nosync",
> > "berkeley-hash", "berkeley-hash-nosync", "skiplist", "sql")}
> > { "mboxkey_db", "skiplist", STRINGLIST("berkeley", "skiplist") }
> > { "mboxlist_db", "skiplist", STRINGLIST("flat", "berkeley",
> > "berkeley-hash", "skiplist")}
> > { "ptscache_db", "skiplist", STRINGLIST("berkeley", "berkeley-hash",
> > "skiplist")}
> > { "quota_db", "quotalegacy", STRINGLIST("flat", "berkeley",
> > "berkeley-hash", "skiplist", "sql", "quotalegacy")}
> > { "seenstate_db", "skiplist", STRINGLIST("flat", "berkeley",
> > "berkeley-hash", "skiplist")}
> > { "subscription_db", "flat", STRINGLIST("flat", "berkeley",
> > "berkeley-hash", "skiplist")}
> > { "statuscache_db", "skiplist", STRINGLIST("berkeley", "berkeley-nosync",
> > "berkeley-hash", "berkeley-hash-nosync", "skiplist") }
> > { "tlscache_db", "skiplist", STRINGLIST("berkeley", "berkeley-nosync",
> > "berkeley-hash", "berkeley-hash-nosync", "skiplist", "sql")}
> > { "userdeny_db", "flat", STRINGLIST("flat", "berkeley", "berkeley-hash",
> > "skiplist", "sql")}
> >
> > So you shouldn't have any berkeley databases at all. It should be
> > perfectly safe to shut down Cyrus, delete the $confdir/db folder, and
> > restart Cyrus.
> >
> > that's:
> >
> > rm -rf /var/lib/imap/db/
> >
> > While Cyrus is shut down :)
> >
> > (Normal warnings about backing up your stuff and making sure that Centos
> > packages aren't doing something different with defaults than upstream)
> >
> > Cheers,
> >
> > Bron.
> >
> > On Mon, 13 Mar 2017, at 08:28, Nick Howitt wrote:
> > >
> > >
> > > On 12/03/2017 21:09, Patrick Boutilier wrote:
> > > > On 03/12/2017 02:24 PM, Nick Howitt wrote:
> > > >> Hi,
> > > >> I've been running cyrus-imap for a few years not. I am using 2.4.17-8
> > > >> on
> > > >> ClearOS (a Centos derivative). I've been running with "delprune
> > > >> cmd="cyr_expire -E 1 -X 7 -D 7" at=0001" in cyrus.conf, but in the last
> > > >> week or so it has started segfaulting. I've tried moving the command to
> > > >> cron.daily "/usr/lib/cyrus-imapd/cyr_expire -E 1 -X 7 -D 7" but it
> > > >> still
> > > >> segfaulting.
> > > >>
> > > >> In /var/log/maillog I get:
> > > >> Mar 12 03:15:53 server cyr_expire[15065]: Repacking mailbox
> > > >> user.mail-ebay
> > > >> Mar 12 03:15:53 server master[15064]: setrlimit: Unable to set file
> > > >> descriptors limit to -1: Operation not permitted
> > > >> Mar 12 03:15:53 server master[15064]: retrying with 4096 (current max)
> > > >> Mar 12 03:15:53 server cyr_expire[15065]: Repacking mailbox
> > > >> user.mail-ebay.Drafts
> > > >> Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox user.nick
> > > >> Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox
> > > >> user.ourfamily
> > > >> Mar 12 03:15:54 server cyr_expire[15065]: Repacking mailbox
> > > >> user.ourfamily.Drafts
> > > >> Mar 12 03:15:54 server cyr_expire[15065]: Expunged 0 out of 9150
> > > >> messages from 57 mailboxes
> > > >> Mar 12 03:15:54 server cyr_expire[15065]: duplicate_prune: pruning back
> >
Re: cyr_expire suddenly segfaulting
1 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file > >> handle: /var/lib/imap/db/__db.001 > >> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file > >> handle: /var/lib/imap/db/__db.002 > >> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB1582 Open file > >> handle: /var/lib/imap/db/__db.003 > >> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR db5: BDB0060 PANIC: > >> fatal region error detected; run recovery > >> Mar 11 03:51:04 server cyr_expire[27625]: DBERROR: critical database > >> situation > >> > >> I'm afraid I've absolutely no idea how to troubleshoot this. Can anyone > >> please help? > >> > >> Thanks, > >> > >> Nick > > > > > > Looks like you might have a corrupted BerlkeyDB . What is the output of ? > > > > grep _db /etc/imapd.conf > Nothing? I can't see any alternative file that may hold this info. > imapd.conf is: > > configdirectory: /var/lib/imap > partition-default: /var/spool/imap > admins: root > sievedir: /var/lib/imap/sieve > sendmail: /usr/sbin/sendmail > hashimapspool: true > sasl_pwcheck_method: saslauthd > sasl_mech_list: PLAIN > tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem > tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem > tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt > > flushseenstate: 1 > allowplaintext: yes > reject8bit: no > munge8bit:no > lmtp_over_quota_perm_failure: 1 > timeout: 30 > imapidlepoll: 60 > idlesocket: /var/lib/imap/socket/idle > lmtpsocket: /var/lib/imap/socket/lmtp > allowapop:no > altnamespace: 0 > unixhierarchysep: yes > lmtp_downcase_rcpt: yes > username_tolower: 1 > autocreatequota: -1 > createonpost: 1 > virtdomains: off > expire-days:1 > expunge-days:1 > delete-days:7 > > > > > > > > >> > >> > >> Cyrus Home Page: http://www.cyrusimap.org/ > >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > >> To Unsubscribe: > >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > > > > > > > > Cyrus Home Page: http://www.cyrusimap.org/ > > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > > To Unsubscribe: > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Excisting idle connections and changing password
Newer Cyrus can kill all connections for a single user with one command, cyr_deny. It adds an entry to the deny database and kills current connections. Bron. On Wed, 22 Feb 2017, at 05:04, Marcus Schopen wrote: > Hi, > > Am Mittwoch, den 22.02.2017, 12:45 +0100 schrieb Paul van der Vlis: > > Hello, > > > > When I change the password of a user what's logged in using an IMAP > > idle-connection, how long will the connection "stay"? > > > > My goal is that the user cannot access the mailbox anymore and does not > > get "new mail". > > > > What I did now is restarting Cyrus and I think all excisting idle > > connections will be lost and all users have to login again. > > > > So my question is "for the next time". > > I would change the password and just kill the process the user is > hanging on. > > Ciao! > > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: bugzilla.cyrusimap.org is down
Instructions need updating, please use https:://github.com/cyrusimap/cyrus- imapd/ Regards, Bron. On Thu, 9 Feb 2017, at 04:59, Mikhail T. via Info-cyrus wrote: > The Cyrus Bugzilla is not responding. Firefox claims: >> The server at bugzilla.cyrusimap.org is taking too long to respond. > I got the reference from https://cyrusimap.org/feedback-bugs.html > Is the service coming back up shortly, or do the bug-submission > instructions need updating? Thanks! >> -mi > > This e-mail, including attachments, may include confidential and/or > proprietary information, and may be used only by the person or entity > to which it is addressed. If the reader of this e-mail is not the > intended > recipient or his or her authorized agent, the reader is hereby > notified > that any dissemination, distribution or copying of this e-mail is > prohibited. If you have received this e-mail in error, please > notify the > sender by replying to this message and delete this e-mail immediately. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: What happened to normalizeuid?
We force it to lowercase in Nginx. Bron. On Fri, 20 Jan 2017, at 21:24, Sebastian Hagedorn wrote: > > --On 20. Januar 2017 um 08:04:25 +1100 Bron Gondwana via Info-cyrus > <info-cyrus@lists.andrew.cmu.edu> wrote: > > > On Fri, 20 Jan 2017, at 03:31, Sebastian Hagedorn via Info-cyrus wrote: > >> --On 19. Januar 2017 um 17:18:06 +0100 Simon Matter > >> <simon.mat...@invoca.ch> wrote: > >> > >> > We and others had this as a patch in our RPMs but I think it has never > >> > been part of vanilla cyrus-imapd. > >> > >> Oops. Should I open an issue for a feature request? I'm surprised that's > >> not something many sites want ... > > > > OK, I've never heard of this thing. What is it? > > > > .. lmgtfy .. > > > > Right, so it's something to normalise the userid when you log in. > > > > It will definitely have to be rewritten for Cyrus 3+, because all that > > stuff got moved into mbname_t and friends. > > Perhaps my assumption that the option is necessary is wrong? But I know for > certain that our webmail users use varied case-spellings of their user > names, because in earlier versions of our webmail system they would get > different user profiles depending on how they had entered their user names > ;-) > > Bron, how does Fastmail deal with that? Do you simply force users to use > the canonic spelling? I guess we could do that, but I'd rather not. > > Cheers > Sebastian > -- > .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:. > .:.Regionales Rechenzentrum (RRZK).:. >.:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:. > Email had 1 attachment: > + Attachment2 > 1k (application/pgp-signature) -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: What happened to normalizeuid?
On Fri, 20 Jan 2017, at 03:31, Sebastian Hagedorn via Info-cyrus wrote: > --On 19. Januar 2017 um 17:18:06 +0100 Simon Matter > <simon.mat...@invoca.ch> wrote: > > > We and others had this as a patch in our RPMs but I think it has never > > been part of vanilla cyrus-imapd. > > Oops. Should I open an issue for a feature request? I'm surprised that's > not something many sites want ... OK, I've never heard of this thing. What is it? .. lmgtfy .. Right, so it's something to normalise the userid when you log in. It will definitely have to be rewritten for Cyrus 3+, because all that stuff got moved into mbname_t and friends. Regards, Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus IMAP 3.0.0-rc1 Config Lint Surprise
Thanks Nic, yeah - that page was back to front. Good find Andy. Bron. On Tue, 17 Jan 2017, at 04:01, Nic Bernstein via Info-cyrus wrote: > Thanks for the heads-up. We'll get this fixed, it is the man page which > is correct. > -nic > > On 01/16/2017 10:19 AM, Andy Dorman via Info-cyrus wrote: > > On 01/13/2017 12:07 AM, ellie timoney via Info-cyrus wrote: > >> The Cyrus team is proud to announce the immediate availability of the > >> first release candidate from the Cyrus IMAP 3.0 series: 3.0.0-rc1. > >> > >> As a release candidate, it is considered near-stable for production > >> usage. Interfaces, APIs, features, etc are not likely to change > >> between now and the full release. > >> > >> If you plan to upgrade to 3.0, we recommend starting to test your > >> upgrade process now. Feedback and bug reports will be greatly > >> appreciated! > >> > > > > Hi everyone. We started the process of checking our Cyrus configs in > > preparation for testing our dev/beta Debian Cyrus install (2.5.10-3) > > and ran into an issue right away. > > > > I believe the instruction line at this URL: > > > > http://www.cyrusimap.org/dev/imap/download/upgrade.html#copy-config-files-and-update > > > > > > > > for lint checking the cyrus & imapd configs have the file types > > backwards. The instruction shows: > > > > cyr_info conf-lint -C -M > > > > But according to the man page the command should look like this: > > > > cyr_info [ -C alt imapd.conf ] [ -M alt cyrus.conf ] command > > > > And indeed, if you run the command as shown on the web page you get an > > error caused by the first line of the START section. > > > > /usr/lib/cyrus/bin/cyr_info conf-lint -C /etc/cyrus.conf -M > > /etc/imapd.conf > > fatal error: invalid option name on line 6 of configuration file > > /etc/cyrus.conf > > > > If I run it as the man page suggests, the output looks a little more > > reasonable. (and I have a little bit of work to do :-) > > > > Hope this helps. > > > > -- > Nic Bernstein n...@onlight.com > Onlight Inc. www.onlight.com > 6525 W Bluemound Rd., Ste 24v. 414.272.4477 > Milwaukee, Wisconsin 53213-4073f. 414.290.0335 > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Is guid_mode: sha1 really gone from imapd.conf?
Yep, it's gone away - sha1 is the only mode right now. We're going to want to look at blake2 soon though I reckon. Bron. On Tue, 17 Jan 2017, at 03:48, Andy Dorman via Info-cyrus wrote: > So the subject is the question... > > Running cyr_info conf-lint against our imapd.conf indicates "guid_mode" > is gone and indeed, the imapd.conf man page doesn't mention it now. > > But many older versions of the man page do mention it and I can not find > any mention in past release notes at > > http://www.cyrusimap.org/dev/imap/download/release-notes/index.html > > of it being deprecated or removed. > > So is it really gone or could it have been accidentally left out of the > man page and conf-lint check code? I agree that is very unlikely, but > with all the massive amount of excellent work that has been done by the > Cyrus team in the last few years, sometimes a small thing like this can > fall through a crack. > > Hopefully it will be OK just to remove the "guid_mode" sha1" line and > restart and nothing will explode? > > Thanks and my apologies if the above is a really dumb question. > > -- > Andy Dorman > Ironic Design, Inc. > AnteSpam.com > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: R: Re: R: Re: DBERROR: critical database situation
Just removing the config options that say berkeley for those two databases in
imapd.conf should do the trick (followed by nuking them again, and while Cyrus
is shut down)
Bron.
On Wed, 4 Jan 2017, at 22:08, absolutely_free--- via Info-cyrus wrote:
> After deleting /var/imap/db/*, /var/imap/mailboxes.db and
> /var/imap/deliver.db,
> it rans fine for about 20 minutes, and after:
>
> Jan 4 11:58:41 mail lmtpunix[60214]: DBERROR db5: pthread suspend failed:
> Invalid argument
> Jan 4 11:58:41 mail lmtpunix[60214]: DBERROR db5: BDB0061 PANIC: Invalid
> argument
> Jan 4 11:58:41 mail lmtpunix[60214]: DBERROR: critical database situation
> Jan 4 11:58:41 mail lmtpunix[60215]: DBERROR db5: BDB0060 PANIC: fatal
> region
> error detected; run recovery
> Jan 4 11:58:41 mail lmtpunix[60215]: DBERROR: critical database situation
> Jan 4 11:58:41 mail master[60046]: service lmtpunix pid 60215 in READY
> state:
> terminated abnormally
> Jan 4 11:58:41 mail lmtpunix[60217]: DBERROR db5: BDB0060 PANIC: fatal
> region
> error detected; run recovery
> Jan 4 11:58:41 mail lmtpunix[60217]: DBERROR: critical database situation
> Jan 4 11:58:41 mail master[60046]: service lmtpunix pid 60217 in READY
> state:
> terminated abnormally
> Jan 4 11:58:41 mail lmtpunix[60218]: DBERROR db5: BDB0060 PANIC: fatal
> region
> error detected; run recovery
> Jan 4 11:58:41 mail lmtpunix[60218]: DBERROR: critical database situation
> Jan 4 11:58:41 mail master[60046]: service lmtpunix pid 60218 in READY
> state:
> terminated abnormally
> Jan 4 11:58:41 mail lmtpunix[60219]: DBERROR db5: BDB0060 PANIC: fatal
> region
> error detected; run recovery
> Jan 4 11:58:41 mail lmtpunix[60219]: DBERROR: critical database situation
> Jan 4 11:58:41 mail master[60046]: service lmtpunix pid 60219 in READY
> state:
> terminated abnormally
> Jan 4 11:58:41 mail lmtpunix[60220]: DBERROR db5: BDB0060 PANIC: fatal
> region
> error detected; run recovery
> Jan 4 11:58:41 mail lmtpunix[60220]: DBERROR: critical database situation
> Jan 4 11:58:41 mail master[60046]: service lmtpunix pid 60220 in READY
> state:
> terminated abnormally
> Jan 4 11:58:41 mail lmtpunix[60221]: DBERROR db5: BDB0060 PANIC: fatal
> region
> error detected; run recovery
> Jan 4 11:58:41 mail lmtpunix[60221]: DBERROR: critical database situation
> Jan 4 11:58:41 mail master[60046]: service lmtpunix pid 60221 in READY
> state:
> terminated abnormally
>
>
> I am wondering what is my best option?
> Upgrade to 2.4 (at least)?
>
> Thank you
>
>
> >Messaggio originale
> >Da: "Niels Dettenbach" <n...@syndicat.com>
> >Data: 04/01/2017 11.28
> >A: <info-cyrus@lists.andrew.cmu.edu>, "absolutely_f...@libero.it"
> <absolutely_f...@libero.it>
> >Ogg: Re: R: Re: DBERROR: critical database situation
> >
> >Am Mittwoch, 4. Januar 2017, 11:18:42 CET schrieb absolutely_free--- via
> Info-
> >cyrus:
> >> ./tls_sessions.db: Berkeley DB (Btree, version 9, little-endian)
> >> ./deliver.db: Berkeley DB (Btree, version 9, little-endian)
> >>
> >> So, I can convert tls_sessions and deliver db to skiplist format, right?
> >> How can I determine which database is giving me "DBERROR: critical database
> >> situation"?
> >
> >As far as i remember, it is uncritical to just delete both files, without
> >loosing "practical" data. tls_sessions just held details to ("running") TLS
> >Sessions and deliver.db avoids duplicates if i remember correctly.
> >
> >Stop cyrus - do a backup of delliver.db and tls_sessions.db (or just "Move
> it
> >out of the way) for security (but i'm pretty shure that you can delete it)
> or
> >try to just delete it and restart.
> >
> >After that try to run reconstruct -f to re-generate any indize files etc.
> >within the mailboxes itself.
> >
> >If that not helps, come back again. ß)
> >
> >hth a bit,
> >best regards and good luck,
> >
> >Niels.
> >
> >
> >--
> > ---
> > Niels Dettenbach
> > Syndicat IT & Internet
> > http://www.syndicat.com
> > PGP: https://syndicat.com/pub_key.asc
> > ---
> >
> >
> >
> >
> >
>
>
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Bron Gondwana
br...@fastmail.fm
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: R: Re: DBERROR: critical database situation
Oh good, you can just delete those two BDB files with Cyrus shut down (and
change them to skiplist while you're at it!)
Once you've done that you can delete everything in the $confdir/db folder too.
(delete might mean take a copy somewhere else until you're happy everything is
working of course)
They're both things that don't matter across restarts.
Bron.
On Wed, 4 Jan 2017, at 21:18, absolutely_free--- via Info-cyrus wrote:
> Hi Niels,
>
> thank you for your reply.
> I am using cyrus from sources (ports).
>
> root@mail:/var/imap# find . -name "*.db" -exec file '{}' \;
> ./mailboxes.db: Cyrus skiplist DB
> ./annotations.db: Cyrus skiplist DB
> ./db.backup1/annotations.db: Cyrus skiplist DB
> ./db.backup1/mailboxes.db: Cyrus skiplist DB
> ./tls_sessions.db: Berkeley DB (Btree, version 9, little-endian)
> ./db.backup2/annotations.db: Cyrus skiplist DB
> ./db.backup2/mailboxes.db: Cyrus skiplist DB
> ./deliver.db: Berkeley DB (Btree, version 9, little-endian)
>
> So, I can convert tls_sessions and deliver db to skiplist format, right?
> How can I determine which database is giving me "DBERROR: critical database
> situation"?
>
> Thank you
>
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Bron Gondwana
br...@fastmail.fm
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: R: Re: DBERROR: critical database situation
Indeed, with all the right libraries. You can't upgrade libdb without everything going to shit, which is why we have ditched BDB - the upgrade path is bogus. Bron. On Wed, 4 Jan 2017, at 21:20, absolutely_free--- via Info-cyrus wrote: > > Hi Bron, > thank you for you reply. > What do you mean with "get some version 10 binaries"? > Do you mean "binaries from FreeBSD 10"? > > Thank you > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: DBERROR: critical database situation
2.4 was releasted in 2010, it's 2017. We advise not to run Berkeley DB
(even on 2.3.19), because it doesn't upgrade very nicely.
I would advise that you get some version 10 binaries and use cvt_cyrusdb
to convert all your berkeley databases to skiplist. Skiplist in 2.3.19
is rock solid. You'll need to update the config as well, but the
default for every database type in 2.3.19 is skiplist, so it's just a
matter of removing some lines from imapd.conf.
Regards,
Bron.
On Wed, 4 Jan 2017, at 19:45, absolutely_free--- via Info-cyrus wrote:
>
> Hi,
> I am using cyrus-imapd23-2.3.19_2 on FreeBSD.
> After BSD upgrade (from 10 to 11) I get problems with cyrus.
> I get this kind of errors on:
>
> Jan 4 09:27:31 mail imaps[65141]: DBERROR db5: pthread suspend
> failed: Invalid argument
> Jan 4 09:27:31 mail imaps[65141]: DBERROR db5: BDB0061 PANIC: Invalid
> argument
> Jan 4 09:27:31 mail imaps[65141]: DBERROR: critical database
> situation
> Jan 4 09:27:31 mail imaps[65140]: DBERROR db5: BDB0060 PANIC: fatal
> region error detected; run recovery
> Jan 4 09:27:31 mail imaps[65140]: DBERROR: critical database
> situation
> Jan 4 09:27:38 mail imaps[65142]: DBERROR db5: BDB0060 PANIC: fatal
> region error detected; run recovery
> Jan 4 09:27:38 mail imaps[65142]: DBERROR: critical database
> situation
> Jan 4 09:27:38 mail master[65081]: service imaps pid 65142 in READY
> state: terminated abnormally
> Jan 4 09:27:38 mail imaps[65143]: DBERROR db5: BDB0060 PANIC: fatal
> region error detected; run recovery
> Jan 4 09:27:38 mail imaps[65143]: DBERROR: critical database
> situation
> Jan 4 09:27:38 mail master[65081]: service imaps pid 65143 in READY
> state: terminated abnormally
>
> So I stopped imapd service, and ran /usr/local/cyrus/bin/ctl_cyrusdb
> -r as cyrus user.
>
> Rebuild went fine:
>
> Jan 4 09:36:17 mail ctl_cyrusdb[62332]: recovering cyrus databases
> Jan 4 09:36:17 mail ctl_cyrusdb[62332]: done recovering cyrus
> databases
>
> and I also checked that just after ctl_cyrusdb command, I was able to
> dump mailboxes.db content with ctl_mboxlist -d
>
> After few minutes, I get same errors, and ctl_mboxlist -d returns
> nothing:
>
> cyrus@mail:/root$ /usr/local/cyrus/bin/ctl_cyrusdb -r
> cyrus@mail:/root$
>
> Yesterday I tried to:
>
> stop services
> remove and recreate /var/imap folder
> run reconstruct
>
> but it seems I got no stable solution
>
> Can you help me to solve this problem?
> Thank you very much
>
> This is my imapd.conf:
>
> root@mail:/usr/local/etc# grep -v ^# imapd.conf |grep -v ^$
> configdirectory: /var/imap
> partition-default: /var/spool/imap
> allowapop: 0
> unixhierarchysep: no
> altnamespace: no
> allowanonymouslogin: no
> allowplaintext: yes
> quotawarn: 90
> timeout: 30
> imapidlepoll: 60
> poptimeout: 10
> popminpoll: 0
> admins: root cyrus
> defaultacl: anyone lrs
> duplicatesuppression: no
> sievedir: /var/imap/sieve
> postmaster: postmaster
> sieve_maxscriptsize: 32
> sieve_maxscripts: 5
> duplicate_db: berkeley
> mboxlist_db: skiplist
> ptscache_db: berkeley
> seenstate_db: skiplist
> sasl_pwcheck_method: saslauthd
> sasl_mech_list: plain
> tls_cert_file: /etc/certs/ssl.crt/server.crt
> tls_key_file: /etc/certs/ssl.key/server.key
> tls_ca_file: /etc/certs/ssl.crt/gd_bundle-g2-g1.crt
> tls_ca_path: /etc/certs/ssl.crt
> notifysocket: /var/imap/socket/notify
>
> And cyrus.conf
>
> root@mail:/usr/local/etc# grep -v ^# cyrus.conf |grep -v ^$
> START {
> # do not delete this entry!
> recover cmd="ctl_cyrusdb -r"
> # this is only necessary if using idled for IMAP IDLE
> }
> SERVICES {
> # add or remove based on preferences
> imap cmd="imapd" listen="127.0.0.1:imap" prefork=1
> imaps cmd="imapd -s" listen="imaps" prefork=0
> #pop3 cmd="pop3d" listen="pop3" prefork=0
> pop3s cmd="pop3d -s" listen="pop3s" prefork=0
> sieve cmd="timsieved" listen="sieve" prefork=0
> # these are only necessary if receiving/exporting usenet via NNTP
> # at least one LMTP is required for delivery
> lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
> # this is required if using notifications
> }
> EVENTS {
> # this is required
> checkpointcmd="ctl_cyrusdb -c" period=30
> # this is only necessary if using duplicate delivery suppression,
> # Sieve or NNTP
> delprune cmd="cyr_expire
Re: Did calculating the quota change from 2.3 to 2.5?
On Sat, 31 Dec 2016, at 10:57, Wolfgang Breyha via Info-cyrus wrote: > On 29/11/16 22:37, Jason L Tibbitts III via Info-cyrus wrote: > > Fun random question: Does anything blow up if you run hardlink on your > > mail spool? (The hardlink program finds identical files and hardlinks > > them.) > > Using "hardlink" is IMO not save on imap spools since it also links cyrus.* > files what's definitely not what you want. If your cyrus.* files are identical then you have pretty weird mailboxes, but yeah - I guess it could happen if you had two folders with identical messages in identical order and all the timestamps identical. > I recommend using freedup instead. Something like > freedup -n -v -a -T -o '-name "*."' -l /var/spool/.. But yeah, that looks safe :) It shouldn't blow anything up so long as it renames the file into place atomically. Also be a little careful about file ownership, you'll want to run this as the cyrus user, so it retains ownership and access (unless freedup fixes up ownership, I've never tried it myself) Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Release plan blog post
I posted on the FastMail advent about our plans for releasing Cyrus 3.0 - it's a bit roundabout doing it this way rather than here first, but hey - we talked about it on Monday night's regular meeting. Here's the blog post: https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/ tl;dr, Ellie recently released 3.0beta6. We're going to do a release candidate on Jan 13th and then release for real soon afterwards, so get testing! There are no major changes expected before release. I'll be doing a couple of small JMAP changes to align with the latest spec and possibly to add getMessageListUpdates if I can manage it in time. Other than that, I'm looking a reverse UniqueId indexing similar to the RACL support - it's already in testing and might get added behind a default-off config switch. We'll be assessing all the defaults. I'm really tempted to turn RACL on, but it needs group support if your site uses groups, and that's not done yet, so I'd need someone willing to test it! Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: IOERROR: mapping /srv/imap/config/annotations.db.NEW file: Cannot allocate memory
How big is your annotations.db ? What architecture are you running on? On Sat, 17 Dec 2016, at 04:34, Jan Kowalsky via Info-cyrus wrote: > Hi all, > > for a few days all time creating or deleting folders we get an error in > mail.log: > > Dec 16 18:28:17 mail imap[10078]: IOERROR: mapping > /srv/imap/config/annotations.db.NEW file: Cannot allocate memory > Dec 16 18:28:17 mail imap[10078]: Fatal error: failed to mmap > /srv/imap/config/annotations.db.NEW file > Dec 16 18:28:17 mail imap[10078]: twoskip: > /srv/imap/config/annotations.db closed while still locked > > We can create mailboxes - but cannot delete them anymore. > > The annotations.db is twoskip format and we are running cyrus 2.5 (with > kolab). > > Any Idea? > > Thanks and best regards > Jan > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus Project bug tracking moved to Github
Effective immediately, we are no longer using bugzilla.cyrusimap.org for bug tracking, and anything filed there since last Friday (December 9th) will be lost. I have asked CMU to shut the old bugzilla down. All the bugs have been migrated to Issues section of the github repositories for the three projects: https://github.com/cyrusimap/cyrus-imapd/ https://github.com/cyrusimap/cyrus-sasl/ https://github.com/cyrusimap/cassandane/ This includes already closed bugs. Each bug has a "Bugzilla-Id" item in the long description, allowing you to quickly see what it used to be called. Attachments have been extracted into a separate git repo and linked from the comments on each issue. I for one welcome our new octocat overlords. Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: 2.5.10 autocreate on login not working?
Is there anything in syslog? Bron On Tue, 6 Dec 2016, at 05:40, Nels Lindquist via Info-cyrus wrote: > I'm experimenting with a build of 2.5.10 on CentOS 7 in preparation for > upgrading from our installed 2.3.16 (built from Simon Mattar's RPMs). > > We rely on mailbox auto-creation functionality tied to LDAP > authentication with virtual domain support; as long as the LDAP account > exists the mailbox may be autocreated. > > As the autocreate patch has now been incorporated in 2.5, I was hoping > it would work fairly seamlessly, but even after updating all the > deprecated imapd.conf directives, I'm having trouble. > > I'm able to log in to IMAP successfully with an existing LDAP account, > but a LIST command produces no output, and if I log in with cyradm the > expected mailboxes are not present. I'm able to create mailboxes > manually with cyradm and everything works as expected, but I really need > autocreate to work. > > Here's the relevant section of my imapd.conf: > > virtdomains: yes > defaultdomain: example.com > username_tolower: yes > lmtp_downcase_rcpt: yes > autocreate_quota: 0 > autocreate_quota_messages: 0 > autocreate_inbox_folders: Drafts|Sent|Trash > autocreate_subscribe_folders: Drafts|Sent|Trash > autocreate_post: yes > > Anything I've done obviously wrong? > > > Nels Lindquist > <nli...@maei.ca> > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Did calculating the quota change from 2.3 to 2.5?
On Wed, 30 Nov 2016, at 08:37, Jason L Tibbitts III via Info-cyrus wrote:
> >>>>> "BG" == Bron Gondwana via Info-cyrus <info-cyrus@lists.andrew.cmu.edu>
> >>>>> writes:
>
> BG> If you use imapsync, it doesn't know about that, and will upload the
> BG> same message twice. 2.5 doesn't have the smarts to recognise that
> BG> it's the same message.
>
> Fun random question: Does anything blow up if you run hardlink on your
> mail spool? (The hardlink program finds identical files and hardlinks
> them.)
No, that is fine.
> Given an index of message-id/filenames it should be possible to write a
> deduplicator that's orders of magnitude faster than hardlink, but I have
> a sneaking suspicion that someone's already done that.
Yep, I wrote something which can read 2.5 cyrus.index files and hardlink
matching files. It depends on a ton of FastMail internals though.
3.0 will have much better support for deduplication when you upload via
IMAP, because it will know where all the other copies in the same user are
(there's no support for cross-user deduplication because we don't use it at
all, every user gets their own sieve script and their own lmtp pre-processing
at FastMail, so every message will have different headers and hence be a
different GUID. I have to prioritise designs that I actually use)
#!/usr/bin/perl -w
# SETUP {{{
use strict;
use warnings;
use ME;
use Date::Manip;
use IO::File;
use ME::Machine;
use Cyrus::HeaderFile;
use Data::Dumper;
use Cyrus::IndexFile;
use Getopt::Std;
use Digest::SHA;
use ME::CyrusBackup;
use ME::User;
use Data::Dumper;
# }}}
my $sn = shift;
my (undef,undef,$uid,$gid) = getpwnam('cyrus');
foreach my $Slot (ME::Machine->ImapSlots()) {
next if ($sn and $sn ne $Slot->Name());
my $users = $Slot->AllMailboxes();
my $conf = $Slot->ImapdConf();
foreach my $user (sort keys %$users) {
process($conf, $user, $users->{$user});
}
}
sub process {
my ($conf, $user, $folders) = @_;
print "$user\n";
my %ihave;
foreach my $folder (@$folders) {
my $meta = $conf->GetUserLocation('meta', $user, 'default', $folder);
my $index = Cyrus::IndexFile->new_file("$meta/cyrus.index") || die "Failed
to open $meta/cyrus.index";
while (my $record = $index->next_record()) {
push @{$ihave{$record->{MessageGuid}}}, [$folder, $record->{Uid}];
}
}
foreach my $guid (keys %ihave) {
next if @{$ihave{$guid}} <= 1;
my ($inode, $srcname);
my @others;
foreach my $item (@{$ihave{$guid}}) {
my $spool = $conf->GetUserLocation('spool', $user, 'default', $item->[0]);
$spool =~ s{/$}{};
my $file = "$spool/$item->[1].";
my (@sd) = stat($file);
if ($inode) {
next if $sd[1] == $inode;
push @others, $file;
}
else {
$inode = $sd[1];
$srcname = $file;
}
}
next unless @others;
print "fixing up files for $guid ($srcname)\n";
foreach my $file (@others) {
my $tmpfile = $file . "tmp";
print "link error $tmpfile\n" unless link($srcname, $tmpfile);
chown($uid, $gid, $tmpfile);
chmod(0600, $tmpfile);
print "rename error $file\n" unless rename($tmpfile, $file);
}
}
}
--
Bron Gondwana
br...@fastmail.fm
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Did calculating the quota change from 2.3 to 2.5?
Quota is a sum of byte sizes of raw unexpunged messages. It doesn't deduplicate. Likely issue is incorrect quota_mailbox_used in the cyrus.index header on 2.3. a reconstruct will fix those, then quota -f again. It's not related to du. The problem with imapsync is that it doesn't handle single instance store. If you have copied messages or delivered then into multiple mailboxes with sieve, they will have hard links on disk. If you use imapsync, it doesn't know about that, and will upload the same message twice. 2.5 doesn't have the smarts to recognise that it's the same message. Bron. On Wed, 30 Nov 2016, at 01:24, Marc Patermann via Info-cyrus wrote: > Bron, > > Am 29.11.2016 um 13:26 Uhr schrieb Bron Gondwana via Info-cyrus: > > No, the quota calculations are identical. It's possible that your > > quota was incorrectly calculated on the source server though. A > > quota -f there should correct the calculations. > unluckily it does not. > > quota -f on seems not to be related to the du counter on the old server > in any way for some mailboxes. > > First we create the mailbox on the new server and sync the quota. > Then imapsync syncs the messages. > Till the quota is exceeded … > > oldserver> lq user.xxx > STORAGE 658949/125 (52.71592%) > > # du -sh /var/lib/imap/meta/user/xxx/ > 105M/var/lib/imap/meta/user/xxx/ > # du -sh /var/spool/imap/user/xxx/ > 1,2G/var/spool/imap/user/xxx/ > > > newserver> lq user.xxx > STORAGE 1098788/125 (87.90304%) > > # du -sh /var/spool/imap/user/xxx/ > 1,7G/var/spool/imap/user/xxx/ > > There is no separate meta partition on the new server. > Meta data is about 500 MB now on the new server, this is about 5x the space. > > I think quota is just plain wrong on the old server. > > squatter file are huge in comparison now. > Is this right? > > > Marc > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Did calculating the quota change from 2.3 to 2.5?
No, the quota calculations are identical. It's possible that your quota was incorrectly calculated on the source server though. A quota -f there should correct the calculations. Regards, Bron. On Tue, 29 Nov 2016, at 01:36, Marc Patermann via Info-cyrus wrote: > Hi, > > while migrating from 2.3 to 2.5 (see my last post here), mailboxes can > not be synced, because the quota is exceeded on the new server. > > A mailbox which has a du of about 800 MB in a 900 MB quota mailbox fills > the new mailbox by over 100%. > > Are meta files now calculated into the quota? > > > Marc > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: mailboxes.db locking problem after updating from 2.4 to 2.5.9
On Fri, 18 Nov 2016, at 10:51, Wolfgang Breyha via Info-cyrus wrote: > On 17/11/16 14:00, Deniss via Info-cyrus wrote: > > Any ideas or suggestion for investigation ? > > I already filed a bug > https://github.com/cyrusimap/cyrus-imapd/issues/43 > but no response so far. I directly asked Bron, but no response as well. Sorry, I really don't have a clue. 2.5 does have a different mailboxes.db format, so it's a bit more CPU intensive. The real massive win for CPU usage is going to come with reverse ACLs: https://blog.fastmail.com/2015/12/05/reverse-acls-making-imap-list-fast/ But to get there, we need to solve reverse ACLs for groups. I did ask about it here: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2015-November/038628.html But then didn't follow up to add group reverse ACL support in Cyrus, so reverse ACLs are broken if you're using groups. Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cannot unlink .... mboxkey: No such file or directory
If you're running rolling replication, it SHOULD have logged both the old mailbox name and the DELETED mailbox name, and calculate the move when syncing. If you're running sync_client -m on just the mailbox, sure - it will get deleted, because sync_client can't tell what name it was renamed to (there are plans afoot to make rename detection more clever with some mailboxes.db changes, but that's still a while away as part of a fully unified murder environment) Anyway, you didn't say which Cyrus version you're using there. If it's pre 2.4 all bets are off! Pretty sure this was solid in 2.4.0 and above though, with rolling replication. On Tue, 8 Nov 2016, at 23:36, Marcus Schopen via Info-cyrus wrote: > Hi, > > I'm running cyrus in master and slave. When deleting a user's mailbox I > see the following error on master and slave log: > > Nov 8 12:00:44 master cyrus/imap[26104]: cannot > unlink /var/lib/cyrus/user/t/test20161108.mboxkey: No such file or > directory > > Another strange thing is, that the deleted mailbox is completely removed > and not moved to DELETED on slave side whereas on master it's moved to > DELETED directory. > > delete_mode and expunge_mode on both sides: > > delete_mode: delayed > expunge_mode: delayed > > Ciao > Marcus > > > > > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyrus master fails with status 71
On Tue, 8 Nov 2016, at 05:01, Eric Cunningham via Info-cyrus wrote: > Are such numbers of imapd processes to be expected? Depends on your usage patterns of course. > Why is lmptunix complaining about options passed to imapd? Because the names of the config files are bogus for historical reasons. imapd.conf should really be called settings.conf and cyrus.conf should really be called master.conf. It's definitely not a config file for just the imap daemon. Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian search databases
On Mon, 7 Nov 2016, at 23:23, Adam Tauno Williams via Info-cyrus wrote: > On Thu, 2016-11-03 at 10:47 +1100, Bron Gondwana via Info-cyrus wrote: > > Just out of interest - is anyone other than Fastmail currently using > > the Cyrus Xapian-based search system? > > Not using Xapian. Cool. It turns out I didn't have to make it all backwards incompatible just yet, because instead I just detect if it's using the old Chert backend and force a reindex of all email to upgrade to Glass. (but can still read from a mix of both, so you can phase the upgrade) This is going to the rest of FastMail production today after running happily for the past 4 days for ~5% of our users. Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Xapian search databases
Just out of interest - is anyone other than Fastmail currently using the Cyrus Xapian-based search system? We're making a bunch of changes: * transition to Glass backend in Xapian 1.4 * move cyrus.indexed information into Xapian metadata * index by GUID rather than folder/uid pair These changes aren't entirely backwards compatible, so we'll have to do a conversion process at FastMail - but keeping code in upstream Cyrus to do the conversion will make everything more complex and brittle forever, so I'd like to throw it away ASAP. Obviously if there are other sites which are running Xapian already, then you'll have to go through the same pain, so I'd try to make it all more generic! Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: limit max child per user or IP
No idea with perdition either.
There is support directly in Cyrus though, at least new cyrus:
{ "maxlogins_per_host", 0, INT }
/* Maximum number of logged in sessions allowed per host,
zero means no limit */
{ "maxlogins_per_user", 0, INT }
/* Maximum number of logged in sessions allowed per user,
zero means no limit */
Bron.
On Fri, 28 Oct 2016, at 18:23, Marcus Schopen via Info-cyrus wrote:
> Hi,
>
> I'm running cyrus behind a perdition proxy and from time to time one of
> my users is eating up "maxchild" which is set to 500 (it's just a small
> server for friends and family) and no further connections are accepted
> for other users. I checked /var/run/cyrus/proc and mostly "Archives"
> folders are opened. I talked to the user to check his client settings,
> but it's not clear why his client behaves like this.
> Is there a way to limit max connections per user or IP? Perdition
> itself doesn't support this (only limit on the number of simultaneous
> connections).
>
> Ciao
> Marcus
>
>
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Bron Gondwana
br...@fastmail.fm
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyrus master fails with status 71
On Tue, 25 Oct 2016, at 02:45, Eric Cunningham via Info-cyrus wrote: > Hi list, we're running cyrus imap 2.5.9 built from the FreeBSD 10-2 > (release-p7) ports tree. > > The cyrus master process is failing periodically (every 1-2 weeks) as > follows: > > Oct 22 07:38:48 imap1 master[7767]: process type:SERVICE name:imaps > path:/usr/local/cyrus/bin/imapd age:305.215s pid:32760 exited, status 71 > Oct 22 07:38:48 imap1 master[7767]: service imaps/ipv4 pid 32760 in > READY state: terminated abnormally > Oct 22 07:38:48 imap1 master[7767]: too many failures for service > imaps/ipv4, disabling until next SIGHUP > > This prevents new connections by clients until cyrus is restarted. I've > looked around the web but have not seen this issue reported. > > A little background: > > Our initial thought on this was that we were running out of listen > queues so have upped that incrementally from the default of 32 to a > current setting of 32768 via /usr/local/etc/rc.d/imapd using the -l > option, with increased kern.ipc.soacceptqueue set to 32768, but that > hasn't helped. Sometimes the "status 71" occurs during periods of light > use during off hours, like on Saturday mornings. > > We have ~1400 imap accounts, though the number of impad processes hovers > around 3,000-4,000. There have been spikes observed as high as 12,000 > imapd processes. In that particular case, 1 user had 2 imap clients > accounting for near 6,000 of those connections. We've attempted to > limit these high numbers using the following imapd.conf values: > > maxlogins_per_host: 50 > maxlogins_per_user: 30 > tcp_keepalive: 1 > tcp_keepalive_cnt: 1 > tcp_keepalive_idle: 30 > tcp_keepalive_intvl: 900 > > However, it seems that once these were reached, no new connections were > permitted and resulted in all manner of user complaints about not being > able to get at their email. > > Any ideas on this "status 71" issue? Could an upgrade to 2.5.10 > possibly address this? Thanks! https://www.freebsd.org/cgi/man.cgi?query=sysexits EX_OSERR (71) An operating system error has been detected. This is intended to be used for such things as ``cannot fork'', ``cannot create pipe'', or the like. It includes things like getuid returning a user that does not exist in the passwd file. So the question is: what failed? Is there anything earlier in the log to suggest what the imapd was doing when it died? Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
