Re: bugzilla.cyrusimap.org is down

[Bron Gondwana via Info-cyrus]

Instructions need updating, please use https:://github.com/cyrusimap/cyrus-
imapd/


Regards,


Bron.





On Thu, 9 Feb 2017, at 04:59, Mikhail T. via Info-cyrus wrote:

> The Cyrus Bugzilla is not responding. Firefox claims:



>> The server at bugzilla.cyrusimap.org is taking too long to respond.
> I got the reference from https://cyrusimap.org/feedback-bugs.html



> Is the service coming back up shortly, or do the bug-submission
> instructions need updating? Thanks!
>> -mi



> 

> This e-mail, including attachments, may include confidential and/or

> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the
> intended
> recipient or his or her authorized agent, the reader is hereby
> notified
> that any dissemination, distribution or copying of this e-mail is

> prohibited. If you have received this e-mail in error, please
> notify the
> sender by replying to this message and delete this e-mail immediately.
> 

> Cyrus Home Page: http://www.cyrusimap.org/

> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:

> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus



--

  Bron Gondwana

  br...@fastmail.fm





Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: What happened to normalizeuid?

[Bron Gondwana via Info-cyrus]

We force it to lowercase in Nginx.

Bron.

On Fri, 20 Jan 2017, at 21:24, Sebastian Hagedorn wrote:
> 
> --On 20. Januar 2017 um 08:04:25 +1100 Bron Gondwana via Info-cyrus 
> <info-cyrus@lists.andrew.cmu.edu> wrote:
> 
> > On Fri, 20 Jan 2017, at 03:31, Sebastian Hagedorn via Info-cyrus wrote:
> >> --On 19. Januar 2017 um 17:18:06 +0100 Simon Matter
> >> <simon.mat...@invoca.ch> wrote:
> >>
> >> > We and others had this as a patch in our RPMs but I think it has never
> >> > been part of vanilla cyrus-imapd.
> >>
> >> Oops. Should I open an issue for a feature request? I'm surprised that's
> >> not something many sites want ...
> >
> > OK, I've never heard of this thing. What is it?
> >
> > .. lmgtfy ..
> >
> > Right, so it's something to normalise the userid when you log in.
> >
> > It will definitely have to be rewritten for Cyrus 3+, because all that
> > stuff got moved into mbname_t and friends.
> 
> Perhaps my assumption that the option is necessary is wrong? But I know for 
> certain that our webmail users use varied case-spellings of their user 
> names, because in earlier versions of our webmail system they would get 
> different user profiles depending on how they had entered their user names 
> ;-)
> 
> Bron, how does Fastmail deal with that? Do you simply force users to use 
> the canonic spelling? I guess we could do that, but I'd rather not.
> 
> Cheers
> Sebastian
> -- 
> .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
>  .:.Regionales Rechenzentrum (RRZK).:.
>.:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.
> Email had 1 attachment:
> + Attachment2
>   1k (application/pgp-signature)


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: What happened to normalizeuid?

[Bron Gondwana via Info-cyrus]

On Fri, 20 Jan 2017, at 03:31, Sebastian Hagedorn via Info-cyrus wrote:
> --On 19. Januar 2017 um 17:18:06 +0100 Simon Matter 
>  wrote:
> 
> > We and others had this as a patch in our RPMs but I think it has never
> > been part of vanilla cyrus-imapd.
> 
> Oops. Should I open an issue for a feature request? I'm surprised that's 
> not something many sites want ...

OK, I've never heard of this thing. What is it?

.. lmgtfy ..

Right, so it's something to normalise the userid when you log in.

It will definitely have to be rewritten for Cyrus 3+, because all that stuff got
moved into mbname_t and friends.

Regards,

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.0.0-rc1 Config Lint Surprise

[Bron Gondwana via Info-cyrus]

Thanks Nic, yeah - that page was back to front.  Good find Andy.

Bron.

On Tue, 17 Jan 2017, at 04:01, Nic Bernstein via Info-cyrus wrote:
> Thanks for the heads-up.  We'll get this fixed, it is the man page which 
> is correct.
>  -nic
> 
> On 01/16/2017 10:19 AM, Andy Dorman via Info-cyrus wrote:
> > On 01/13/2017 12:07 AM, ellie timoney via Info-cyrus wrote:
> >> The Cyrus team is proud to announce the immediate availability of the
> >> first release candidate from the Cyrus IMAP 3.0 series: 3.0.0-rc1.
> >>
> >> As a release candidate, it is considered near-stable for production
> >> usage.   Interfaces, APIs, features, etc are not likely to change
> >> between now and the full release.
> >>
> >> If you plan to upgrade to 3.0, we recommend starting to test your
> >> upgrade process now.  Feedback and bug reports will be greatly
> >> appreciated!
> >>
> >
> > Hi everyone.  We started the process of checking our Cyrus configs in 
> > preparation for testing our dev/beta Debian Cyrus install (2.5.10-3) 
> > and ran into an issue right away.
> >
> > I believe the instruction line at this URL:
> >
> > http://www.cyrusimap.org/dev/imap/download/upgrade.html#copy-config-files-and-update
> >  
> >
> >
> > for lint checking the cyrus & imapd configs have the file types 
> > backwards.  The instruction shows:
> >
> > cyr_info conf-lint -C  -M 
> >
> > But according to the man page the command should look like this:
> >
> > cyr_info [ -C alt imapd.conf ] [ -M alt cyrus.conf ] command
> >
> > And indeed, if you run the command as shown on the web page you get an 
> > error caused by the first line of the START section.
> >
> > /usr/lib/cyrus/bin/cyr_info conf-lint -C /etc/cyrus.conf -M 
> > /etc/imapd.conf
> > fatal error: invalid option name on line 6 of configuration file 
> > /etc/cyrus.conf
> >
> > If I run it as the man page suggests, the output looks a little more 
> > reasonable. (and I have a little bit of work to do :-)
> >
> > Hope this helps.
> >
> 
> -- 
> Nic Bernstein n...@onlight.com
> Onlight Inc.  www.onlight.com
> 6525 W Bluemound Rd., Ste 24v. 414.272.4477
> Milwaukee, Wisconsin  53213-4073f. 414.290.0335
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Is guid_mode: sha1 really gone from imapd.conf?

[Bron Gondwana via Info-cyrus]

Yep, it's gone away - sha1 is the only mode right now.  We're going to want to 
look at blake2 soon though I reckon.

Bron.

On Tue, 17 Jan 2017, at 03:48, Andy Dorman via Info-cyrus wrote:
> So the subject is the question...
> 
> Running cyr_info conf-lint against our imapd.conf indicates "guid_mode" 
> is gone and indeed, the imapd.conf man page doesn't mention it now.
> 
> But many older versions of the man page do mention it and I can not find 
> any mention in past release notes at
> 
> http://www.cyrusimap.org/dev/imap/download/release-notes/index.html
> 
> of it being deprecated or removed.
> 
> So is it really gone or could it have been accidentally left out of the 
> man page and conf-lint check code? I agree that is very unlikely, but 
> with all the massive amount of excellent work that has been done by the 
> Cyrus team in the last few years, sometimes a small thing like this can 
> fall through a crack.
> 
> Hopefully it will be OK just to remove the "guid_mode" sha1" line and 
> restart and nothing will explode?
> 
> Thanks and my apologies if the above is a really dumb question.
> 
> -- 
> Andy Dorman
> Ironic Design, Inc.
> AnteSpam.com
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: R: Re: R: Re: DBERROR: critical database situation

[Bron Gondwana via Info-cyrus]

Just removing the config options that say berkeley for those two databases in 
imapd.conf should do the trick (followed by nuking them again, and while Cyrus 
is shut down)

Bron.

On Wed, 4 Jan 2017, at 22:08, absolutely_free--- via Info-cyrus wrote:
> After deleting /var/imap/db/*, /var/imap/mailboxes.db and 
> /var/imap/deliver.db, 
> it rans fine for about 20 minutes, and after:
> 
> Jan  4 11:58:41 mail lmtpunix[60214]: DBERROR db5: pthread suspend failed: 
> Invalid argument
> Jan  4 11:58:41 mail lmtpunix[60214]: DBERROR db5: BDB0061 PANIC: Invalid 
> argument
> Jan  4 11:58:41 mail lmtpunix[60214]: DBERROR: critical database situation
> Jan  4 11:58:41 mail lmtpunix[60215]: DBERROR db5: BDB0060 PANIC: fatal 
> region 
> error detected; run recovery
> Jan  4 11:58:41 mail lmtpunix[60215]: DBERROR: critical database situation
> Jan  4 11:58:41 mail master[60046]: service lmtpunix pid 60215 in READY 
> state: 
> terminated abnormally
> Jan  4 11:58:41 mail lmtpunix[60217]: DBERROR db5: BDB0060 PANIC: fatal 
> region 
> error detected; run recovery
> Jan  4 11:58:41 mail lmtpunix[60217]: DBERROR: critical database situation
> Jan  4 11:58:41 mail master[60046]: service lmtpunix pid 60217 in READY 
> state: 
> terminated abnormally
> Jan  4 11:58:41 mail lmtpunix[60218]: DBERROR db5: BDB0060 PANIC: fatal 
> region 
> error detected; run recovery
> Jan  4 11:58:41 mail lmtpunix[60218]: DBERROR: critical database situation
> Jan  4 11:58:41 mail master[60046]: service lmtpunix pid 60218 in READY 
> state: 
> terminated abnormally
> Jan  4 11:58:41 mail lmtpunix[60219]: DBERROR db5: BDB0060 PANIC: fatal 
> region 
> error detected; run recovery
> Jan  4 11:58:41 mail lmtpunix[60219]: DBERROR: critical database situation
> Jan  4 11:58:41 mail master[60046]: service lmtpunix pid 60219 in READY 
> state: 
> terminated abnormally
> Jan  4 11:58:41 mail lmtpunix[60220]: DBERROR db5: BDB0060 PANIC: fatal 
> region 
> error detected; run recovery
> Jan  4 11:58:41 mail lmtpunix[60220]: DBERROR: critical database situation
> Jan  4 11:58:41 mail master[60046]: service lmtpunix pid 60220 in READY 
> state: 
> terminated abnormally
> Jan  4 11:58:41 mail lmtpunix[60221]: DBERROR db5: BDB0060 PANIC: fatal 
> region 
> error detected; run recovery
> Jan  4 11:58:41 mail lmtpunix[60221]: DBERROR: critical database situation
> Jan  4 11:58:41 mail master[60046]: service lmtpunix pid 60221 in READY 
> state: 
> terminated abnormally
> 
> 
> I am wondering what is my best option?
> Upgrade to 2.4 (at least)?
> 
> Thank you
> 
> 
> >Messaggio originale
> >Da: "Niels Dettenbach" 
> >Data: 04/01/2017 11.28
> >A: , "absolutely_f...@libero.it"
> 
> >Ogg: Re: R: Re: DBERROR: critical database situation
> >
> >Am Mittwoch, 4. Januar 2017, 11:18:42 CET schrieb absolutely_free--- via 
> Info-
> >cyrus:
> >> ./tls_sessions.db: Berkeley DB (Btree, version 9, little-endian)
> >> ./deliver.db: Berkeley DB (Btree, version 9, little-endian)
> >> 
> >> So, I can convert tls_sessions and deliver db to skiplist format, right?
> >> How can I determine which database is giving me "DBERROR: critical database
> >> situation"?
> >
> >As far as i remember, it is uncritical to just delete both files, without 
> >loosing "practical" data. tls_sessions just held details to ("running") TLS 
> >Sessions and deliver.db avoids duplicates if i remember correctly.
> >
> >Stop cyrus - do a backup of delliver.db and tls_sessions.db (or just "Move 
> it 
> >out of the way) for security (but i'm pretty shure that you can delete it) 
> or 
> >try to just delete it and restart. 
> >
> >After that try to run reconstruct -f to re-generate any indize files etc. 
> >within the mailboxes itself.
> >
> >If that not helps, come back again. ß)
> >
> >hth a bit,
> >best regards and good luck,
> >
> >Niels.
> >
> >
> >-- 
> > ---
> > Niels Dettenbach
> > Syndicat IT & Internet
> > http://www.syndicat.com
> > PGP: https://syndicat.com/pub_key.asc
> > ---
> > 
> >
> >
> >
> >
> 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: R: Re: DBERROR: critical database situation

[Bron Gondwana via Info-cyrus]

Oh good, you can just delete those two BDB files with Cyrus shut down (and 
change them to skiplist while you're at it!)

Once you've done that you can delete everything in the $confdir/db folder too.

(delete might mean take a copy somewhere else until you're happy everything is 
working of course)

They're both things that don't matter across restarts.

Bron.

On Wed, 4 Jan 2017, at 21:18, absolutely_free--- via Info-cyrus wrote:
> Hi Niels,
> 
> thank you for your reply.
> I am using cyrus from sources (ports).
> 
> root@mail:/var/imap# find . -name "*.db" -exec file '{}' \;
> ./mailboxes.db: Cyrus skiplist DB
> ./annotations.db: Cyrus skiplist DB
> ./db.backup1/annotations.db: Cyrus skiplist DB
> ./db.backup1/mailboxes.db: Cyrus skiplist DB
> ./tls_sessions.db: Berkeley DB (Btree, version 9, little-endian)
> ./db.backup2/annotations.db: Cyrus skiplist DB
> ./db.backup2/mailboxes.db: Cyrus skiplist DB
> ./deliver.db: Berkeley DB (Btree, version 9, little-endian)
> 
> So, I can convert tls_sessions and deliver db to skiplist format, right?
> How can I determine which database is giving me "DBERROR: critical database 
> situation"?
> 
> Thank you
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: R: Re: DBERROR: critical database situation

[Bron Gondwana via Info-cyrus]

Indeed, with all the right libraries.  You can't upgrade libdb without
everything going to shit, which is why we have ditched BDB - the upgrade
path is bogus.


Bron.





On Wed, 4 Jan 2017, at 21:20, absolutely_free--- via Info-cyrus wrote:

> 

> Hi Bron,

> thank you for you reply.

> What do you mean with "get some version 10 binaries"?

> Do you mean "binaries from FreeBSD 10"?

> 
> Thank you

> 

> 

> Cyrus Home Page: http://www.cyrusimap.org/

> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:

> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus



--

  Bron Gondwana

  br...@fastmail.fm





Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: DBERROR: critical database situation

[Bron Gondwana via Info-cyrus]

2.4 was releasted in 2010, it's 2017.  We advise not to run Berkeley DB
  (even on 2.3.19), because it doesn't upgrade very nicely.


I would advise that you get some version 10 binaries and use cvt_cyrusdb
to convert all your berkeley databases to skiplist. Skiplist in 2.3.19
is rock solid.  You'll need to update the config as well, but the
default for every database type in 2.3.19 is skiplist, so it's just a
matter of removing some lines from imapd.conf.


Regards,



Bron.



On Wed, 4 Jan 2017, at 19:45, absolutely_free--- via Info-cyrus wrote:

> 

> Hi,

> I am using cyrus-imapd23-2.3.19_2 on FreeBSD.

> After BSD upgrade (from 10 to 11) I get problems with cyrus.

> I get this kind of errors on:

> 

> Jan  4 09:27:31 mail imaps[65141]: DBERROR db5: pthread suspend
> failed: Invalid argument
> Jan  4 09:27:31 mail imaps[65141]: DBERROR db5: BDB0061 PANIC: Invalid
> argument
> Jan  4 09:27:31 mail imaps[65141]: DBERROR: critical database
> situation
> Jan  4 09:27:31 mail imaps[65140]: DBERROR db5: BDB0060 PANIC: fatal
> region error detected; run recovery
> Jan  4 09:27:31 mail imaps[65140]: DBERROR: critical database
> situation
> Jan  4 09:27:38 mail imaps[65142]: DBERROR db5: BDB0060 PANIC: fatal
> region error detected; run recovery
> Jan  4 09:27:38 mail imaps[65142]: DBERROR: critical database
> situation
> Jan  4 09:27:38 mail master[65081]: service imaps pid 65142 in READY
> state: terminated abnormally
> Jan  4 09:27:38 mail imaps[65143]: DBERROR db5: BDB0060 PANIC: fatal
> region error detected; run recovery
> Jan  4 09:27:38 mail imaps[65143]: DBERROR: critical database
> situation
> Jan  4 09:27:38 mail master[65081]: service imaps pid 65143 in READY
> state: terminated abnormally
> 

> So I stopped imapd service, and ran /usr/local/cyrus/bin/ctl_cyrusdb
> -r as cyrus user.
> 

> Rebuild went fine: 

> 

> Jan  4 09:36:17 mail ctl_cyrusdb[62332]: recovering cyrus databases

> Jan  4 09:36:17 mail ctl_cyrusdb[62332]: done recovering cyrus
> databases
> 

> and I also checked that just after ctl_cyrusdb command, I was able to
> dump mailboxes.db content with ctl_mboxlist -d
> 

> After few minutes, I get same errors, and ctl_mboxlist -d returns
> nothing:
> 

> cyrus@mail:/root$ /usr/local/cyrus/bin/ctl_cyrusdb -r

> cyrus@mail:/root$ 

> 

> Yesterday I tried to:

> 

> stop services

> remove and recreate /var/imap folder

> run reconstruct

> 

> but it seems I got no stable solution

> 

> Can you help me to solve this problem?

> Thank you very much

> 

> This is my imapd.conf:

> 

> root@mail:/usr/local/etc# grep -v ^# imapd.conf |grep -v ^$

> configdirectory: /var/imap

> partition-default: /var/spool/imap

> allowapop: 0

> unixhierarchysep: no

> altnamespace: no

> allowanonymouslogin: no

> allowplaintext: yes

> quotawarn: 90

> timeout: 30

> imapidlepoll: 60

> poptimeout: 10

> popminpoll: 0

> admins: root cyrus

> defaultacl: anyone lrs

> duplicatesuppression: no

> sievedir: /var/imap/sieve

> postmaster: postmaster

> sieve_maxscriptsize: 32

> sieve_maxscripts: 5

> duplicate_db: berkeley

> mboxlist_db: skiplist

> ptscache_db: berkeley

> seenstate_db: skiplist

> sasl_pwcheck_method: saslauthd

> sasl_mech_list: plain

> tls_cert_file: /etc/certs/ssl.crt/server.crt

> tls_key_file: /etc/certs/ssl.key/server.key

> tls_ca_file:  /etc/certs/ssl.crt/gd_bundle-g2-g1.crt

> tls_ca_path: /etc/certs/ssl.crt

> notifysocket: /var/imap/socket/notify

> 

> And cyrus.conf

> 

> root@mail:/usr/local/etc# grep -v ^# cyrus.conf |grep -v ^$
> START {

>   # do not delete this entry!

>   recover   cmd="ctl_cyrusdb -r"

>   # this is only necessary if using idled for IMAP IDLE

> }

> SERVICES {

>   # add or remove based on preferences

>   imap  cmd="imapd" listen="127.0.0.1:imap" prefork=1

>   imaps cmd="imapd -s" listen="imaps" prefork=0

>   #pop3 cmd="pop3d" listen="pop3" prefork=0

>   pop3s cmd="pop3d -s" listen="pop3s" prefork=0

>   sieve cmd="timsieved" listen="sieve" prefork=0

>   # these are only necessary if receiving/exporting usenet via NNTP

>   # at least one LMTP is required for delivery

>   lmtpunix  cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0

>   # this is required if using notifications

> }

> EVENTS {

>   # this is required

>   checkpointcmd="ctl_cyrusdb -c" period=30

>   # this is only necessary if using duplicate delivery suppression,

>   # Sieve or NNTP

>   delprune  cmd="cyr_expire -E 3" at=0400

>   # this is only necessary if caching TLS sessions

>   tlsprune  cmd="tls_prune" at=0400

> }

> 

> 

> 

> 

> 

> 

> Cyrus Home Page: http://www.cyrusimap.org/

> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:

> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus



--

  Bron Gondwana

  br...@fastmail.fm





Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: 

Re: Did calculating the quota change from 2.3 to 2.5?

[Bron Gondwana via Info-cyrus]

On Sat, 31 Dec 2016, at 10:57, Wolfgang Breyha via Info-cyrus wrote:
> On 29/11/16 22:37, Jason L Tibbitts III via Info-cyrus wrote:
> > Fun random question: Does anything blow up if you run hardlink on your
> > mail spool?  (The hardlink program finds identical files and hardlinks
> > them.)
> 
> Using "hardlink" is IMO not save on imap spools since it also links cyrus.*
> files what's definitely not what you want.

If your cyrus.* files are identical then you have pretty weird mailboxes, but 
yeah - I guess it could happen if you had two folders with identical messages 
in identical order and all the timestamps identical.

> I recommend using freedup instead. Something like
> freedup -n -v -a -T -o '-name "*."' -l /var/spool/..

But yeah, that looks safe :)

It shouldn't blow anything up so long as it renames the file into place 
atomically.  Also be a little careful about file ownership, you'll want to run 
this as the cyrus user, so it retains ownership and access (unless freedup 
fixes up ownership, I've never tried it myself)

Bron.



-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Release plan blog post

[Bron Gondwana via Info-cyrus]

I posted on the FastMail advent about our plans for releasing Cyrus 3.0 - it's 
a bit roundabout doing it this way rather than here first, but hey - we talked 
about it on Monday night's regular meeting.

Here's the blog post:

https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/

tl;dr, Ellie recently released 3.0beta6.  We're going to do a release candidate 
on Jan 13th and then release for real soon afterwards, so get testing!

There are no major changes expected before release.  I'll be doing a couple of 
small JMAP changes to align with the latest spec and possibly to add 
getMessageListUpdates if I can manage it in time.

Other than that, I'm looking a reverse UniqueId indexing similar to the RACL 
support - it's already in testing and might get added behind a default-off 
config switch.

We'll be assessing all the defaults.  I'm really tempted to turn RACL on, but 
it needs group support if your site uses groups, and that's not done yet, so 
I'd need someone willing to test it!

Bron.



-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: IOERROR: mapping /srv/imap/config/annotations.db.NEW file: Cannot allocate memory

[Bron Gondwana via Info-cyrus]

How big is your annotations.db ?  What architecture are you running on?

On Sat, 17 Dec 2016, at 04:34, Jan Kowalsky via Info-cyrus wrote:
> Hi all,
> 
> for a few days all time creating or deleting folders we get an error in
> mail.log:
> 
> Dec 16 18:28:17 mail imap[10078]: IOERROR: mapping
> /srv/imap/config/annotations.db.NEW file: Cannot allocate memory
> Dec 16 18:28:17 mail imap[10078]: Fatal error: failed to mmap
> /srv/imap/config/annotations.db.NEW file
> Dec 16 18:28:17 mail imap[10078]: twoskip:
> /srv/imap/config/annotations.db closed while still locked
> 
> We can create mailboxes - but cannot delete them anymore.
> 
> The annotations.db is twoskip format and we are running cyrus 2.5 (with
> kolab).
> 
> Any Idea?
> 
> Thanks and best regards
> Jan
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Project bug tracking moved to Github

[Bron Gondwana via Info-cyrus]

Effective immediately, we are no longer using bugzilla.cyrusimap.org for bug 
tracking, and anything filed there since last Friday (December 9th) will be 
lost.  I have asked CMU to shut the old bugzilla down.

All the bugs have been migrated to Issues section of the github repositories 
for the three projects:

https://github.com/cyrusimap/cyrus-imapd/
https://github.com/cyrusimap/cyrus-sasl/
https://github.com/cyrusimap/cassandane/

This includes already closed bugs.

Each bug has a "Bugzilla-Id" item in the long description, allowing you to 
quickly see what it used to be called.  Attachments have been extracted into a 
separate git repo and linked from the comments on each issue.

I for one welcome our new octocat overlords.

Bron.


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.5.10 autocreate on login not working?

[Bron Gondwana via Info-cyrus]

Is there anything in syslog?

Bron

On Tue, 6 Dec 2016, at 05:40, Nels Lindquist via Info-cyrus wrote:
> I'm experimenting with a build of 2.5.10 on CentOS 7 in preparation for 
> upgrading from our installed 2.3.16 (built from Simon Mattar's RPMs).
> 
> We rely on mailbox auto-creation functionality tied to LDAP 
> authentication with virtual domain support; as long as the LDAP account 
> exists the mailbox may be autocreated.
> 
> As the autocreate patch has now been incorporated in 2.5, I was hoping 
> it would work fairly seamlessly, but even after updating all the 
> deprecated imapd.conf directives, I'm having trouble.
> 
> I'm able to log in to IMAP successfully with an existing LDAP account, 
> but a LIST command produces no output, and if I log in with cyradm the 
> expected mailboxes are not present.  I'm able to create mailboxes 
> manually with cyradm and everything works as expected, but I really need 
> autocreate to work.
> 
> Here's the relevant section of my imapd.conf:
> 
> virtdomains: yes
> defaultdomain: example.com
> username_tolower: yes
> lmtp_downcase_rcpt: yes
> autocreate_quota: 0
> autocreate_quota_messages: 0
> autocreate_inbox_folders: Drafts|Sent|Trash
> autocreate_subscribe_folders: Drafts|Sent|Trash
> autocreate_post: yes
> 
> Anything I've done obviously wrong?
> 
> 
> Nels Lindquist
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Did calculating the quota change from 2.3 to 2.5?

[Bron Gondwana via Info-cyrus]

On Wed, 30 Nov 2016, at 08:37, Jason L Tibbitts III via Info-cyrus wrote:
> >>>>> "BG" == Bron Gondwana via Info-cyrus <info-cyrus@lists.andrew.cmu.edu> 
> >>>>> writes:
> 
> BG> If you use imapsync, it doesn't know about that, and will upload the
> BG> same message twice. 2.5 doesn't have the smarts to recognise that
> BG> it's the same message.
> 
> Fun random question: Does anything blow up if you run hardlink on your
> mail spool?  (The hardlink program finds identical files and hardlinks
> them.)

No, that is fine.

> Given an index of message-id/filenames it should be possible to write a
> deduplicator that's orders of magnitude faster than hardlink, but I have
> a sneaking suspicion that someone's already done that.

Yep, I wrote something which can read 2.5 cyrus.index files and hardlink
matching files.  It depends on a ton of FastMail internals though.

3.0 will have much better support for deduplication when you upload via
IMAP, because it will know where all the other copies in the same user are
(there's no support for cross-user deduplication because we don't use it at
all, every user gets their own sieve script and their own lmtp pre-processing
at FastMail, so every message will have different headers and hence be a
different GUID.  I have to prioritise designs that I actually use)


#!/usr/bin/perl -w

# SETUP {{{
use strict;
use warnings;
use ME;
use Date::Manip;
use IO::File;
use ME::Machine;
use Cyrus::HeaderFile;
use Data::Dumper;
use Cyrus::IndexFile;
use Getopt::Std;
use Digest::SHA;
use ME::CyrusBackup;
use ME::User;
use Data::Dumper;
# }}}

my $sn = shift;

my (undef,undef,$uid,$gid) = getpwnam('cyrus');

foreach my $Slot (ME::Machine->ImapSlots()) {
  next if ($sn and $sn ne $Slot->Name());
  my $users = $Slot->AllMailboxes();
  my $conf = $Slot->ImapdConf();
  foreach my $user (sort keys %$users) {
process($conf, $user, $users->{$user});
  }
}

sub process {
  my ($conf, $user, $folders) = @_;
  print "$user\n";
  my %ihave;
  foreach my $folder (@$folders) {
my $meta = $conf->GetUserLocation('meta', $user, 'default', $folder);
my $index = Cyrus::IndexFile->new_file("$meta/cyrus.index") || die "Failed 
to open $meta/cyrus.index";
while (my $record = $index->next_record()) {
  push @{$ihave{$record->{MessageGuid}}}, [$folder, $record->{Uid}];
}
  }

  foreach my $guid (keys %ihave) {
next if @{$ihave{$guid}} <= 1;
my ($inode, $srcname);
my @others;
foreach my $item (@{$ihave{$guid}}) {
  my $spool = $conf->GetUserLocation('spool', $user, 'default', $item->[0]);
  $spool =~ s{/$}{};
  my $file = "$spool/$item->[1].";
  my (@sd) = stat($file);
  if ($inode) {
next if $sd[1] == $inode;
push @others, $file;
  }
  else {
$inode = $sd[1];
$srcname = $file;
  }
}
next unless @others;
print "fixing up files for $guid ($srcname)\n";
foreach my $file (@others) {
  my $tmpfile = $file . "tmp";
  print "link error $tmpfile\n" unless link($srcname, $tmpfile);
  chown($uid, $gid, $tmpfile);
  chmod(0600, $tmpfile);
  print "rename error $file\n" unless rename($tmpfile, $file);
}
  }
}





-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Did calculating the quota change from 2.3 to 2.5?

[Bron Gondwana via Info-cyrus]

Quota is a sum of byte sizes of raw unexpunged messages. It doesn't 
deduplicate. Likely issue is incorrect quota_mailbox_used in the cyrus.index 
header on 2.3. a reconstruct will fix those, then quota -f again.

It's not related to du.

The problem with imapsync is that it doesn't handle single instance store. If 
you have copied messages or delivered then into multiple mailboxes with sieve, 
they will have hard links on disk.

If you use imapsync, it doesn't know about that, and will upload the same 
message twice. 2.5 doesn't have the smarts to recognise that it's the same 
message.

Bron.

On Wed, 30 Nov 2016, at 01:24, Marc Patermann via Info-cyrus wrote:
> Bron,
> 
> Am 29.11.2016 um 13:26 Uhr schrieb Bron Gondwana via Info-cyrus:
> > No, the quota calculations are identical.  It's possible that your
> > quota was incorrectly calculated on the source server though.  A
> > quota -f there should correct the calculations.
> unluckily it does not.
> 
> quota -f on seems not to be related to the du counter on the old server 
> in any way for some mailboxes.
> 
> First we create the mailbox on the new server and sync the quota.
> Then imapsync syncs the messages.
> Till the quota is exceeded …
> 
> oldserver> lq user.xxx
>   STORAGE 658949/125 (52.71592%)
> 
> # du -sh /var/lib/imap/meta/user/xxx/
> 105M/var/lib/imap/meta/user/xxx/
> # du -sh /var/spool/imap/user/xxx/
> 1,2G/var/spool/imap/user/xxx/
> 
> 
> newserver> lq user.xxx
>   STORAGE 1098788/125 (87.90304%)
> 
> # du -sh /var/spool/imap/user/xxx/
> 1,7G/var/spool/imap/user/xxx/
> 
> There is no separate meta partition on the new server.
> Meta data is about 500 MB now on the new server, this is about 5x the space.
> 
> I think quota is just plain wrong on the old server.
> 
> squatter file are huge in comparison now.
> Is this right?
> 
> 
> Marc
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Did calculating the quota change from 2.3 to 2.5?

[Bron Gondwana via Info-cyrus]

No, the quota calculations are identical.  It's possible that your quota was 
incorrectly calculated on the source server though.  A quota -f there should 
correct the calculations.

Regards,

Bron.

On Tue, 29 Nov 2016, at 01:36, Marc Patermann via Info-cyrus wrote:
> Hi,
> 
> while migrating from 2.3 to 2.5 (see my last post here), mailboxes can 
> not be synced, because the quota is exceeded on the new server.
> 
> A mailbox which has a du of about 800 MB in a 900 MB quota mailbox fills 
> the new mailbox by over 100%.
> 
> Are meta files now calculated into the quota?
> 
> 
> Marc
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mailboxes.db locking problem after updating from 2.4 to 2.5.9

[Bron Gondwana via Info-cyrus]

On Fri, 18 Nov 2016, at 10:51, Wolfgang Breyha via Info-cyrus wrote:
> On 17/11/16 14:00, Deniss via Info-cyrus wrote:
> > Any ideas or suggestion for investigation ?
> 
> I already filed a bug
> https://github.com/cyrusimap/cyrus-imapd/issues/43
> but no response so far. I directly asked Bron, but no response as well.

Sorry, I really don't have a clue.  2.5 does have a different mailboxes.db 
format, so it's a bit more CPU intensive.  The real massive win for CPU usage 
is going to come with reverse ACLs:

https://blog.fastmail.com/2015/12/05/reverse-acls-making-imap-list-fast/

But to get there, we need to solve reverse ACLs for groups.  I did ask about it 
here:

https://lists.andrew.cmu.edu/pipermail/info-cyrus/2015-November/038628.html

But then didn't follow up to add group reverse ACL support in Cyrus, so reverse 
ACLs are broken if you're using groups.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cannot unlink .... mboxkey: No such file or directory

[Bron Gondwana via Info-cyrus]

If you're running rolling replication, it SHOULD have logged both the old 
mailbox name and the DELETED mailbox name, and calculate the move when syncing.

If you're running sync_client -m on just the mailbox, sure - it will get 
deleted, because sync_client can't tell what name it was renamed to (there are 
plans afoot to make rename detection more clever with some mailboxes.db 
changes, but that's still a while away as part of a fully unified murder 
environment)

Anyway, you didn't say which Cyrus version you're using there.  If it's pre 2.4 
all bets are off!  Pretty sure this was solid in 2.4.0 and above though, with 
rolling replication.

On Tue, 8 Nov 2016, at 23:36, Marcus Schopen via Info-cyrus wrote:
> Hi,
> 
> I'm running cyrus in master and slave. When deleting a user's mailbox I
> see the following error on master and slave log:
> 
> Nov  8 12:00:44 master cyrus/imap[26104]: cannot
> unlink /var/lib/cyrus/user/t/test20161108.mboxkey: No such file or
> directory
> 
> Another strange thing is, that the deleted mailbox is completely removed
> and not moved to DELETED on slave side whereas on master it's moved to
> DELETED directory.
> 
> delete_mode and expunge_mode on both sides:
> 
>   delete_mode: delayed
>   expunge_mode: delayed
> 
> Ciao
> Marcus
> 
> 
> 
> 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus master fails with status 71

[Bron Gondwana via Info-cyrus]

On Tue, 8 Nov 2016, at 05:01, Eric Cunningham via Info-cyrus wrote:
> Are such numbers of imapd processes to be expected?

Depends on your usage patterns of course.

> Why is lmptunix complaining about options passed to imapd?

Because the names of the config files are bogus for historical reasons.  
imapd.conf should really be called settings.conf and cyrus.conf should really 
be called master.conf.  It's definitely not a config file for just the imap 
daemon.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian search databases

[Bron Gondwana via Info-cyrus]

On Mon, 7 Nov 2016, at 23:23, Adam Tauno Williams via Info-cyrus wrote:
> On Thu, 2016-11-03 at 10:47 +1100, Bron Gondwana via Info-cyrus wrote:
> > Just out of interest - is anyone other than Fastmail currently using
> > the Cyrus Xapian-based search system?
> 
> Not using Xapian.

Cool.  It turns out I didn't have to make it all backwards incompatible just 
yet, because instead I just detect if it's using the old Chert backend and 
force a reindex of all email to upgrade to Glass.

(but can still read from a mix of both, so you can phase the upgrade)

This is going to the rest of FastMail production today after running happily 
for the past 4 days for ~5% of our users.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Xapian search databases

[Bron Gondwana via Info-cyrus]

Just out of interest - is anyone other than Fastmail currently using the Cyrus 
Xapian-based search system?

We're making a bunch of changes:
* transition to Glass backend in Xapian 1.4
* move cyrus.indexed information into Xapian metadata
* index by GUID rather than folder/uid pair

These changes aren't entirely backwards compatible, so we'll have to do a 
conversion process at FastMail - but keeping code in upstream Cyrus to do the 
conversion will make everything more complex and brittle forever, so I'd like 
to throw it away ASAP.

Obviously if there are other sites which are running Xapian already, then 
you'll have to go through the same pain, so I'd try to make it all more generic!

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: limit max child per user or IP

[Bron Gondwana via Info-cyrus]

No idea with perdition either.

There is support directly in Cyrus though, at least new cyrus:

{ "maxlogins_per_host", 0, INT }
/* Maximum number of logged in sessions allowed per host,
   zero means no limit */

{ "maxlogins_per_user", 0, INT }
/* Maximum number of logged in sessions allowed per user,
   zero means no limit */

Bron.

On Fri, 28 Oct 2016, at 18:23, Marcus Schopen via Info-cyrus wrote:
> Hi,
> 
> I'm running cyrus behind a perdition proxy and from time to time one of
> my users is eating up "maxchild" which is set to 500 (it's just a small
> server for friends and family) and no further connections are accepted
> for other users. I checked /var/run/cyrus/proc and mostly "Archives"
> folders are opened. I talked to the user to check his client settings,
> but it's not clear why his client behaves like this.
>  Is there a way to limit max connections per user or IP? Perdition
> itself doesn't support this (only limit on the number of simultaneous
> connections).
> 
> Ciao
> Marcus
> 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus master fails with status 71

[Bron Gondwana via Info-cyrus]

On Tue, 25 Oct 2016, at 02:45, Eric Cunningham via Info-cyrus wrote:
> Hi list, we're running cyrus imap 2.5.9 built from the FreeBSD 10-2 
> (release-p7) ports tree.
> 
> The cyrus master process is failing periodically (every 1-2 weeks) as 
> follows:
> 
> Oct 22 07:38:48 imap1 master[7767]: process type:SERVICE name:imaps 
> path:/usr/local/cyrus/bin/imapd age:305.215s pid:32760 exited, status 71
> Oct 22 07:38:48 imap1 master[7767]: service imaps/ipv4 pid 32760 in 
> READY state: terminated abnormally
> Oct 22 07:38:48 imap1 master[7767]: too many failures for service 
> imaps/ipv4, disabling until next SIGHUP
> 
> This prevents new connections by clients until cyrus is restarted.  I've 
> looked around the web but have not seen this issue reported.
> 
> A little background:
> 
> Our initial thought on this was that we were running out of listen 
> queues so have upped that incrementally from the default of 32 to a 
> current setting of 32768 via /usr/local/etc/rc.d/imapd using the -l 
> option, with increased kern.ipc.soacceptqueue set to 32768, but that 
> hasn't helped.  Sometimes the "status 71" occurs during periods of light 
> use during off hours, like on Saturday mornings.
> 
> We have ~1400 imap accounts, though the number of impad processes hovers 
> around 3,000-4,000.  There have been spikes observed as high as 12,000 
> imapd processes.  In that particular case, 1 user had 2 imap clients 
> accounting for near 6,000 of those connections.  We've attempted to 
> limit these high numbers using the following imapd.conf values:
> 
> maxlogins_per_host: 50
> maxlogins_per_user: 30
> tcp_keepalive: 1
> tcp_keepalive_cnt: 1
> tcp_keepalive_idle: 30
> tcp_keepalive_intvl: 900
> 
> However, it seems that once these were reached, no new connections were 
> permitted and resulted in all manner of user complaints about not being 
> able to get at their email.
> 
> Any ideas on this "status 71" issue?  Could an upgrade to 2.5.10 
> possibly address this?  Thanks!

https://www.freebsd.org/cgi/man.cgi?query=sysexits

 EX_OSERR (71) An operating system error has been detected.  This
   is intended to be used for such things as ``cannot
   fork'', ``cannot create pipe'', or the like.  It
   includes things like getuid returning a user that
   does not exist in the passwd file.

So the question is: what failed?  Is there anything earlier in the log to 
suggest
what the imapd was doing when it died?

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: command line deletion of files

[Bron Gondwana via Info-cyrus]

Ahh, and reading on I see that this exists already :)

On Fri, 30 Sep 2016, at 08:02, Bron Gondwana wrote:
> You're the reason we can't have nice things :( rm + reconstruct will bite you 
> one upgrade for sure.
> 
> A dry run option to ipurge sounds like a great idea. We just always use IMAP 
> to do admin on Cyrus, but I can see a case for improving ipurge.
> 
> On Fri, 30 Sep 2016, at 01:04, Vladislav Kurz via Info-cyrus wrote:
> > On 09/29/16 16:32, Patrick Boutilier via Info-cyrus wrote:
> > > On 09/29/2016 11:27 AM, Shawn Bakhtiar via Info-cyrus wrote:
> > >> Good morning,
> > >>
> > >> trying to get rid of some emails that have large attachments (i.e.
> > >> videos sent over email, or cd images, etc...)
> > >>
> > >> Would it be proper to
> > >>
> > >> rm -rf /var/spool/imap/u/username/mailbox/4321.
> > >>
> > >> then
> > >>
> > >> reconstruct -rf user.username
> > >>
> > >> Or is there a more "proper" way using cyrus?
> > > 
> > > Not sure about deleting a single message but you can use ipurge to
> > > delete messages based on size. Good to use in a script to parses the
> > > mail spool.
> > > 
> > 
> > rm + reconstruct is IMHO ok, we use that for trash/spam cleanup,
> > antivirus checks, and similar things.
> > 
> > ipurge is nice but I would really appreciate if it had a --dry-run
> > option. I'm never sure what it will really delete.
> > 
> > 
> > -- 
> > Best Regards
> > Vladislav Kurz
> > 
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> 
> -- 
>   Bron Gondwana
>   br...@fastmail.fm


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: command line deletion of files

[Bron Gondwana via Info-cyrus]

You're the reason we can't have nice things :( rm + reconstruct will bite you 
one upgrade for sure.

A dry run option to ipurge sounds like a great idea. We just always use IMAP to 
do admin on Cyrus, but I can see a case for improving ipurge.

On Fri, 30 Sep 2016, at 01:04, Vladislav Kurz via Info-cyrus wrote:
> On 09/29/16 16:32, Patrick Boutilier via Info-cyrus wrote:
> > On 09/29/2016 11:27 AM, Shawn Bakhtiar via Info-cyrus wrote:
> >> Good morning,
> >>
> >> trying to get rid of some emails that have large attachments (i.e.
> >> videos sent over email, or cd images, etc...)
> >>
> >> Would it be proper to
> >>
> >> rm -rf /var/spool/imap/u/username/mailbox/4321.
> >>
> >> then
> >>
> >> reconstruct -rf user.username
> >>
> >> Or is there a more "proper" way using cyrus?
> > 
> > Not sure about deleting a single message but you can use ipurge to
> > delete messages based on size. Good to use in a script to parses the
> > mail spool.
> > 
> 
> rm + reconstruct is IMHO ok, we use that for trash/spam cleanup,
> antivirus checks, and similar things.
> 
> ipurge is nice but I would really appreciate if it had a --dry-run
> option. I'm never sure what it will really delete.
> 
> 
> -- 
> Best Regards
> Vladislav Kurz
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Twoskip DB files broken

[Bron Gondwana via Info-cyrus]

A couple of things.

1. Why are you doing this? What do you hope to achieve?

2. Possibly kolab's Cyrus configuration stores files in other paths (tmpfs, 
data dirs) which are Berkeley dbs and don't expect their environment to be 
trashed under them.

On Thu, 29 Sep 2016, at 23:47, Tobias Brunner via Info-cyrus wrote:
> Hi,
> 
> I've discovered an odd behaviour which I don't understand: After a
> completely fresh Kolab 16 install on CentOS 7
> (cyrus-imapd-2.5.9.27-5.1.el7.kolab_wf.x86_64) everything looks fine. I
> can create mailboxes, stop/start Cyrus, all works as it should do. The
> contents of /var/lib/imap looks like this:
> 
> -rw--- 1 cyrus mail 3.8K Sep 29 14:48 annotations.db
> drwxr-x--- 2 cyrus mail6 Sep 22 17:25 backup
> drwxr-x--- 2 cyrus mail   22 Sep 29 14:48 db
> drwx-- 2 cyrus mail   46 Sep 29 14:48 db.backup1
> -rw--- 1 cyrus mail  336 Sep 29 14:48 deliver.db
> drwx-- 3 cyrus mail   14 Sep 29 14:48 domain
> drwx-- 5 cyrus mail   35 Sep 29 14:48 lock
> drwxr-x--- 2 cyrus mail6 Sep 22 17:25 log
> -rw--- 1 cyrus mail 3.7K Sep 29 14:48 mailboxes.db
> drwxr-x--- 2 cyrus mail6 Sep 22 17:25 md5
> drwxr-x--- 2 cyrus mail6 Sep 22 17:25 meta
> drwxr-x--- 2 cyrus mail6 Sep 22 17:25 msg
> drwxr-x--- 2 cyrus mail   18 Sep 29 14:56 proc
> drwxr-x--- 2 cyrus mail   37 Sep 29 14:48 ptclient
> drwxr-x--- 2 cyrus mail6 Sep 22 17:25 quota
> drwxr-x--- 2 cyrus mail   51 Sep 29 14:48 rpm
> drwxr-x--- 2 cyrus mail6 Sep 22 17:25 sieve
> drwxr-x--- 2 cyrus mail 4.0K Sep 29 14:56 socket
> -rw--- 1 cyrus mail  336 Sep 29 14:48 statuscache.db
> drwxr-x--- 2 cyrus mail6 Sep 22 17:25 sync
> -rw--- 1 cyrus mail  768 Sep 29 14:48 tls_sessions.db
> drwxr-x--- 2 cyrus mail6 Sep 22 17:25 user
> 
> Then I do:
> 
> 1. Stop Cyrus: systemctl stop cyrus-imapd. Log says:
> 
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: attempting clean
> shutdown on signal
> Sep 29 14:57:59 kolabv.vagrant.dev pop3[16965]: graceful shutdown
> initiated by unexpected process 1 (/usr/lib/systemd/systemd
> --switched-root --system --deserialize 21)
> Sep 29 14:57:59 kolabv.vagrant.dev pop3[16971]: graceful shutdown
> initiated by unexpected process 1 (/usr/lib/systemd/systemd
> --switched-root --system --deserialize 21)
> Sep 29 14:57:59 kolabv.vagrant.dev lmtpunix[16967]: graceful shutdown
> initiated by unexpected process 1 (/usr/lib/systemd/systemd
> --switched-root --system --deserialize 21)
> Sep 29 14:57:59 kolabv.vagrant.dev systemd[1]: Stopping Cyrus-imapd
> IMAP/POP3 email server...
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:imap path:/usr/lib/cyrus-imapd/imapd age:561.385s pid:16959 exited,
> status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:imap path:/usr/lib/cyrus-imapd/imapd age:561.385s pid:16961 exited,
> status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.385s pid:16963 exited,
> status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.385s pid:16964 exited,
> status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:pop3s path:/usr/lib/cyrus-imapd/pop3d age:561.385s pid:16966
> exited, status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:imap path:/usr/lib/cyrus-imapd/imapd age:561.387s pid:16958 exited,
> status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:imap path:/usr/lib/cyrus-imapd/imapd age:561.389s pid:16960 exited,
> status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.389s pid:16965 exited,
> status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:lmtpunix path:/usr/lib/cyrus-imapd/lmtpd age:561.388s pid:16967
> exited, status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:notify path:/usr/lib/cyrus-imapd/notifyd age:561.388s pid:16968
> exited, status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.388s pid:16969 exited,
> status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.388s pid:16970
> signaled to death by signal 15 (Terminated)
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.388s pid:16971 exited,
> status 75
> Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
> name:pop3s path:/usr/lib/cyrus-imapd/pop3d age:561.388s pid:16972
> exited, status 75
> Sep 29 14:57:59 kolabv.vagrant.dev systemd[1]: Stopped Cyrus-imapd
> IMAP/POP3 email server.
> 
> 2. Remove all content of /var/lib/imap: rm -rf 

Re: Invalid mailbox name

[Bron Gondwana via Info-cyrus]

If anyone is interested in looking in the code on master to see how this works:

https://github.com/cyrusimap/cyrus-imapd/

You're looking for mbname_t, and in particular mbname_from_*.

EXPORTED mbname_t *mbname_from_extname(const char *extname, const struct 
namespace *ns, const char *userid)

And their inverses, which are just the same without the from, i.e.

EXPORTED const char *mbname_extname(const mbname_t *mbname, const struct 
namespace *ns, const char *userid)

The struct behind mbname_t allows for caching, parsing, processing the parts 
differently, and generally much nicer manipulation of names, all the while 
converting everything at the last possible moment - so the internal 
representation inside an mbname_t has no separators at all, instead it's a 
strarray_t of individual names, allowing magic like:

EXPORTED char *mbname_pop_boxes(mbname_t *mbname)

and the inverse:

EXPORTED void mbname_push_boxes(mbname_t *mbname, const char *item)

Allowing you to create parent and child mailboxes programatically.

One thing I'm _not_ doing here, and might switch to doing, is converting 
to-and-from modified UTF7.  It seems a little pointless while there is no 
unicode representation of names - but when we do JMAP we'll need it, and at 
that point the internal representation should be as canonical as possible.  We 
can also enforce the correctness of the UTF7 more easily then, because it has 
to be correct to be parsed!

Bron.

On Fri, 23 Sep 2016, at 14:10, Bron Gondwana wrote:
> On Thu, 22 Sep 2016, at 22:11, Eric Luyten via Info-cyrus wrote:
> > On Thu, September 22, 2016 9:46 am, Michael Menge via Info-cyrus wrote:
> > >
> > 
> > > Quoting Paul van der Vlis via Info-cyrus 
> > > :
> > >
> > 
> > >> I am wondering about the dot. So far I know I cannot use it in a mailbox
> > >> name, but it is in the list.
> > >>
> > >
> > > I suspect that your cyrus is configured to use the . as hierarchy 
> > > seperator.
> > > see "unixhierarchysep:" in imapd.conf manpage for details.
> > 
> > 
> > 
> > Of course one can use a '.' as part of a Cyrus mailbox name, internally it
> > gets translated into a '^' (arrow/caret).
> 
> In master I've changed the behaviour to expose ^ if you're not in unixhs, so 
> that names are fully translatable between the two settings.  We run in a 
> mixed mode at FastMail - the old imapds listening on mail.messagingengine.com 
> (and our internal stuff) are still using the '.' separator for legacy 
> reasons, but new clients connecting to imap.fastmail.com get both 
> altnamespace (which is now robust) and '/' separators, so they can use dots 
> nicely.
> 
> We're not yet at the point that we want to enable dots in usernames, but 
> we're close.
> 
> > I modified our GOODCHARS definition heavily when we migrated to Cyrus 2.2,
> > ten years ago, and never had an issue with square brackets and such.
> 
> Yep, they seem fine.  We've had them enabled for about a year now.
> 
> Bron.
> 
> -- 
>   Bron Gondwana
>   br...@fastmail.fm


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Invalid mailbox name

[Bron Gondwana via Info-cyrus]

On Fri, 23 Sep 2016, at 02:37, Paul van der Vlis via Info-cyrus wrote:
> Op 22-09-16 om 09:46 schreef Michael Menge via Info-cyrus:
> > 
> > Quoting Paul van der Vlis via Info-cyrus :
> > 
> >> Op 21-09-16 om 14:11 schreef Michael Menge via Info-cyrus:
> >>> Hi,
> >>>
> >>> Quoting Paul van der Vlis via Info-cyrus
> >>> :
> >>>
>  Hello,
> 
>  I am syncing many mailboxes from one IMAP server to another. Now I get
>  sometimes errors on mailbox names. I found out that Cyrus does not
>  accept brackets in a mailbox name. Is there documentation about what
>  characters are accepted in mailbox names??
> >>>
> >>> The allowed ASCII-Chars are defined in the macro GOODCHARS in
> >>> imap/mboxname.c
> >>> (https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/mboxname.c#L1495).
> >>>
> >>>
> >>> non-ASCII-Chars are handled by RFC 3501 5.1.3.
> >>>
> >>> This subject has been discussed a few years ago on this list, and
> >>> GOODCHARS
> >>> has been changed between cyrus versions.
> >>>
> >>> 2.2:#define GOODCHARS "
> >>> +,-.0123456789:=@ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~"
> >>> 2.3:#define GOODCHARS "
> >>> #$'+,-.0123456789:=@ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~"
> >>>
> >>> Master: #define GOODCHARS "
> >>> #$'()*+,-.0123456789:=?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz~"
> >>>
> >>
> >> Thanks for your answer!
> >>
> >> I am wondering about the dot. So far I know I cannot use it in a mailbox
> >> name, but it is in the list.
> >>
> > 
> > I suspect that your cyrus is configured to use the . as hierarchy
> > seperator.
> 
> Correct.
> 
> > see "unixhierarchysep:" in imapd.conf manpage for details.
> 
> Ah, I know about that.
> 
> But I think it's strange to have a dot in the list of GOODCHARS when
> it's used as a hierarchy seperator...

The mailboxname used to be tested against GOODCHARS in its entirety, including 
separator.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Invalid mailbox name

[Bron Gondwana via Info-cyrus]

On Thu, 22 Sep 2016, at 22:11, Eric Luyten via Info-cyrus wrote:
> On Thu, September 22, 2016 9:46 am, Michael Menge via Info-cyrus wrote:
> >
> 
> > Quoting Paul van der Vlis via Info-cyrus :
> >
> 
> >> I am wondering about the dot. So far I know I cannot use it in a mailbox
> >> name, but it is in the list.
> >>
> >
> > I suspect that your cyrus is configured to use the . as hierarchy seperator.
> > see "unixhierarchysep:" in imapd.conf manpage for details.
> 
> 
> 
> Of course one can use a '.' as part of a Cyrus mailbox name, internally it
> gets translated into a '^' (arrow/caret).

In master I've changed the behaviour to expose ^ if you're not in unixhs, so 
that names are fully translatable between the two settings.  We run in a mixed 
mode at FastMail - the old imapds listening on mail.messagingengine.com (and 
our internal stuff) are still using the '.' separator for legacy reasons, but 
new clients connecting to imap.fastmail.com get both altnamespace (which is now 
robust) and '/' separators, so they can use dots nicely.

We're not yet at the point that we want to enable dots in usernames, but we're 
close.

> I modified our GOODCHARS definition heavily when we migrated to Cyrus 2.2,
> ten years ago, and never had an issue with square brackets and such.

Yep, they seem fine.  We've had them enabled for about a year now.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: How to handle special use folders in Sieve?

[Bron Gondwana via Info-cyrus]

On Tue, 13 Sep 2016, at 17:25, OBATA Akio via Info-cyrus wrote:
> Hi,
> 
> Since 3.0.0-beta2 release note's "Major changes since the 2.5.x series":
> 
> * Sieve now supports special use folders. See Cyrus Sieve
> 
>   http://cyrusimap.org/imap/admin/sieve.html#cyrus-sieve-specialuse
> 
> But I cannot find how to do it in sieve script.
> 
> For example, how to rewrite following script with such feature?
> 
>if header :is "X-Spam-Flag" "YES" {
>  fileinto "INBOX.Junk"
>}

if header :is "X-Spam-Flag" "YES" {
   fileinto "\\Junk"
}

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 2.5.9 imapd children and tcp_timeout questions

[Bron Gondwana via Info-cyrus]

On Tue, 13 Sep 2016, at 10:44, ellie timoney via Info-cyrus wrote:
> Note that it talks about the process being used for new connections. 
> Each imapd process serves one connection at a time (so if you have 50
> client connections, you will have 50 imapd processes to serve them, plus
> whatever you have preforked ready to serve new connections).  When one
> of these connections disconnects, the imapd process that was serving
> that connection goes back into the pool, ready to serve a new incoming
> connection -- unless it has already served $uses connections, in which
> case it exits, and master spawns a new one instead if necessary.
> 
> So you can see how, if you have mostly long-lived connections, each
> imapd's count of how many connections it has served would grow very very
> slowly -- many may not ever reach their limit, because you end up
> needing to reboot the server (OS security updates, data centre works,
> etc) before that occurs.

If you look in /var/lib/imap/proc/* you should see one file for each process id,
which tells you who is connected to that process, and which folder they have
selected (if any).  This is quite useful for tracking down if there's a single 
user
causing issues.

Bron.


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: how to deal with mail retention/archival.

[Bron Gondwana via Info-cyrus]

On Sat, 27 Aug 2016, at 02:13, Shawn Bakhtiar via Info-cyrus wrote:
>
>> On Aug 26, 2016, at 8:35 AM, Giuseppe Ravasio (LU) via Info-cyrus > cy...@lists.andrew.cmu.edu> wrote:
>>
>> I saw that someone proposed to make a sort of abuse of delayed
>> expunge,
>>  but I think that in order to comply with regulatory retention
>>  should be
>>  better considering some specific software.
>>
>
> I don't see how using delayed expunge would really be consider abuse,
> the documentation makes mention of its use for this very reason.

You will hit performance issues. All expunged messages need to be
scanned over at every select and refresh.

> "In any case, the time between a message arriving and being deleted
> may not be sufficient to ensure the message is replicated, included in
> the next backup cycle, and generally available for recovery or
> compliance with the regulatory environment."
> --src  https://cyrusimap.org/imap/features/delayed-expunge.html

If you are replicating to a system that's backing up, then you need
delayed delete to ensure everything gets replicated.

The backup system that's in development uses the sync protocol to feed
the data from Cyrus into the backups.

> We use rsync to make a duplicate of the email spool to a file server
> at regular intervals, which eventually makes its way to tape. Although
> we don't have regulatory requirements I've had to do a few recoveries
> and have done so without problem.
>
> in our case we have cyr_expire set to 3 days (I believe the default
> config), which provides more than enough time to in case a backup
> fails for some reason or another.
>
>
>> For example:
>> http://www.mailpiler.org (Fully Free Software)
>> https://www.mailarchiva.com (the old version is opensource but the
>>  latest is closed)
>>
>> The specific software will be much better for searching the archive.
>>  Finding something in the delayed_expunge folders after many years of
>>  archive will absolutely be a nightmare!
>>
>
>
> Of course these tools offer features but that all comes at a price.
>
>
>>
>
>
>> Giuseppe
>>
>>
>>  On 08/26/2016 03:09 PM, Alvin Starr via Info-cyrus wrote:
>>
>>> A company I am working with is facing issues of regulatorymail
>>> retention.
>>>
>>>  Some searching has yielded little useful results other than
>>>  putting a
>>>  system in front to store all incoming messages.
>>>
>>>  What are others doing for mail archival?
>>>
>>>  An ideal solution would let the users carry on using current use
>>>  patterns and not impose extra restrictions.
>>>
>>>  --
>>>  Alvin Starr   ||   voice: (905)513-7688
>>>  Netvel Inc.   ||   Cell:  (416)806-0133
>>> al...@netvel.net ||
>>>
>>>
>>>
>>>  
>>>  Cyrus Home Page: http://www.cyrusimap.org/
>>>  List Archives/Info:
>>>  http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>  To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>
>> 
>>  Cyrus Home Page: http://www.cyrusimap.org/
>>  List Archives/Info:
>>  http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>  To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Replication failing - IMAP_SYNC_CHECKSUM Checksum Failure

[Bron Gondwana via Info-cyrus]

Sounds like you have been fiddling mailboxes.db entries and/or didn't clean it 
up properly?

I would recommend sync_reset on the user on the replica for a better cleanup 
than delete.

Oh yeah, don't create the mailbox on the replica before replicating it in, 
that's crazy talk - you'll get a new user with a different uniqueid.  You 
shouldn't be creating users on replicas.

Bron.

On Tue, 23 Aug 2016, at 10:38, Tod A. Sandman via Info-cyrus wrote:
> I resorted to deleting the mailbox on the replication slave and trying to 
> start from scratch, but I get nowhere.
> 
> On slave:
> 
> cyrus@cyrus2c:~> cyradm --user mailadmin `hostname`
> 
>   cyrus2c.mail.rice.edu> dm user/lamemm7
>   cyrus2c.mail.rice.edu> cm --partition cyrus2g user/lamemm7
>   cyrus2c.mail.rice.edu> setacl user/lamemm7 mailadmin lrswipkxtecda
> 
> 
> On master:
> 
> cyrus@cyrus2a:~> sync_client -v -v -l -m "user/lamemm7"
> .
> <1471911949 imap/sync_client[4386]: MAILBOX received NO response: Mailbox has been moved 
> to another server
> imap/sync_client[4386]: do_folders(): update failed: user.lamemm7 'The remote 
> Server(s) denied the operation'
> Error from do_mailboxes(): bailing out!
> imap/sync_client[4386]: Error in do_mailboxes(): bailing out!
> >1471911949>EXIT
> <1471911949 
> 
> What the ??
> 
> Syslog on the master shows
> 
> Aug 22 19:25:49 cyrus2a imap/sync_client[4386]: MAILBOX received NO response: 
> Mailbox has been moved to another server
> Aug 22 19:25:49 cyrus2a imap/sync_client[4386]: do_folders(): update failed: 
> user.lamemm7 'The remote Server(s) denied the operation'
> Aug 22 19:25:49 cyrus2a imap/sync_client[4386]: Error in do_mailboxes(): 
> bailing out!
> 
> syslog on the slave shows only
> 
>   Aug 22 19:25:49 cyrus2c imap/syncserver[15637]: Mailbox uniqueid changed 
> user.lamemm7 (351531f5-4353-4976-b1ef-a8a0bdc243bb => 705dd252531801ab) - 
> retry
> 
> 
> 
> 
> Tod Sandman
> Sr. Systems Administrator
> Office of Information Technology
> Rice University
> Voice: 713.348.5816
> Email: sandm...@rice.edu
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Replication failing - IMAP_SYNC_CHECKSUM Checksum Failure

[Bron Gondwana via Info-cyrus]

What do you see in syslog?  (both for the reconstruct and later when the 
sync_client runs)

On Tue, 23 Aug 2016, at 03:49, Tod A. Sandman via Info-cyrus wrote:
> I'm using rolling replication with cyrus-imapd-2.5.9.  sync_client died and I 
> am not able to get replication working again.  I've narrowed it down to one 
> mailbox, user.lamemm7, and I've successfully reconstructed the mailbox on 
> both replication partners with various options, such as
> 
> reconstruct -r user/lamemm7
> reconstruct -r -G user/lamemm7
> reconstruct -r -R user/lamemm7
> reconstruct -r -V max user/lamemm7
> 
> These all seem to run fine, but still sync_client fails:
> 
> % sync_client -v -v -l -m user/lamemm7
> .
> <1471887110 imap/sync_client[21652]: MAILBOX received NO response: IMAP_SYNC_CHECKSUM 
> Checksum Failure
> imap/sync_client[21652]: do_folders(): update failed: user.lamemm7 
> 'Replication inconsistency detected'
> Error from do_mailboxes(): bailing out!
> imap/sync_client[21652]: Error in do_mailboxes(): bailing out!
> >1471887110>EXIT
> <1471887110 
> and the sync_client log shows:
> 
> CRC failure on sync for user.lamemm7, trying full update
> do_folders(): update failed: user.lamemm7 'Replication inconsistency detected'
> Error in do_mailboxes(): bailing out!
> 
> Anyone with ideas on what to try next?
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sync_client errors out after 2.3.16 -> 2.5.9 upgrade

[Bron Gondwana via Info-cyrus]

2.3.2 wasn't version 10, it was version 7!  It would have upgraded through 8, 
9, 10 - and maybe you needed to reconstruct to get GUIDs for those versions - 
that sounds familiar.

Bron.

On Tue, Aug 2, 2016, at 12:22, Kenneth Marshall wrote:
> Hi Bron and Ellie,
> 
> I think that is what happened. Replication was working and the 'reconstruct 
> -G'
> was optional in terms of working, so some of the oldest mailboxes could be in
> that state. I will check the versions and see if there are any older than 10,
> but we started with version 2.3.2 so unless the version changed within the
> patch series, they will all be 10.
> 
> Regards,
> Ken
> 
> On Tue, Aug 02, 2016 at 11:39:01AM +1000, Bron Gondwana via Info-cyrus wrote:
> > Yeah, could be.  In that case then the mailbox needs a reconstruct -G first 
> > :(
> > Version 10 mailboxes have a GUID space available, but I guess they could
> > wind up zero depending on how they got upgraded in the past.
> > 
> > Bron.
> > 
> > On Tue, Aug 2, 2016, at 11:26, ellie timoney via Info-cyrus wrote:
> > > I've been under the impression that Ken's mailboxes were version 10,
> > > which seems like they *should* have guids in them.   If this is the
> > > case, then it's interesting that the replica is coming up with zeroed
> > > ones.
> > > 
> > > If his mailboxes are older than version 10, then it all makes sense,
> > > nothing to see here... though it would be good if the sync code detected
> > > mailboxes being too old to replicate and reported that, instead of
> > > naively trying to sync them and crashing out on the guid check.
> > > 
> > > I guess it's plausible that version 10 mailboxes had a field in the
> > > index for the guid, but that it wasn't necessarily populated?  That
> > > might complicate things?  I don't know the history here.
> > > 
> > > On Tue, Aug 2, 2016, at 10:01 AM, Bron Gondwana via Info-cyrus wrote:
> > > > You can't sync a mailbox without GUID for messages.  You need to upgrade
> > > > the mailboxes before you can use them for replication.  The GUID is used
> > > > for replication - if we allowed zero GUIDs, then every message would
> > > > deduplicate to the same message!
> > > > 
> > > > On Tue, Aug 2, 2016, at 07:56, Kenneth Marshall via Info-cyrus wrote:
> > > > > Hi Cyrus Developers,
> > > > > 
> > > > > Thank you for your patch to address the folder move problem between
> > > > > un-reconstructed mailboxes after the 2.3.16 -> 2.5.9 upgrade. I am
> > > > > not sure, but it looks like there may be another overly aggressive
> > > > > check. I keep getting these fatal errors from sync_client:
> > > > > 
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid 
> > > > > mismatch user.robot 1696 (56412de8678bfb53f6cdb5fe4502031af5484056 
> > > > > )
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid 
> > > > > mismatch user.robot 1697 (1b0024218a4419973b83ae3e84ac7133a4ab7d40 
> > > > > )
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid 
> > > > > mismatch user.robot 1698 (f17084425d83bccb28a4dfa195846c7ef88c7567 
> > > > > )
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid 
> > > > > mismatch user.robot 1699 (7a751e41e1d3a58e541298ab724be4c29d96e49d 
> > > > > )
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid 
> > > > > mismatch user.robot 1700 (724a013d0ae97d27a1da33832487df1719681659 
> > > > > )
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: Fatal error: 
> > > > > Internal error: assertion failed: imap/mailbox.c: 2850: 
> > > > > !message_guid_isnull(>guid)
> > > > > 
> > > > > And then the sync_client has to be run manually and if lucky, it will
> > > > > process the full log successfully. I was looking in imap/mailbox.c
> > > > > and it looks like the assert at line 2850 may need a similar override
> > > > > for non-upgraded folders:
> > > > > 
> > > > > ---imap/mailbox.c---
> > > &

Re: sync_client errors out after 2.3.16 -> 2.5.9 upgrade

[Bron Gondwana via Info-cyrus]

Yeah, could be.  In that case then the mailbox needs a reconstruct -G first :(
Version 10 mailboxes have a GUID space available, but I guess they could
wind up zero depending on how they got upgraded in the past.

Bron.

On Tue, Aug 2, 2016, at 11:26, ellie timoney via Info-cyrus wrote:
> I've been under the impression that Ken's mailboxes were version 10,
> which seems like they *should* have guids in them.   If this is the
> case, then it's interesting that the replica is coming up with zeroed
> ones.
> 
> If his mailboxes are older than version 10, then it all makes sense,
> nothing to see here... though it would be good if the sync code detected
> mailboxes being too old to replicate and reported that, instead of
> naively trying to sync them and crashing out on the guid check.
> 
> I guess it's plausible that version 10 mailboxes had a field in the
> index for the guid, but that it wasn't necessarily populated?  That
> might complicate things?  I don't know the history here.
> 
> On Tue, Aug 2, 2016, at 10:01 AM, Bron Gondwana via Info-cyrus wrote:
> > You can't sync a mailbox without GUID for messages.  You need to upgrade
> > the mailboxes before you can use them for replication.  The GUID is used
> > for replication - if we allowed zero GUIDs, then every message would
> > deduplicate to the same message!
> > 
> > On Tue, Aug 2, 2016, at 07:56, Kenneth Marshall via Info-cyrus wrote:
> > > Hi Cyrus Developers,
> > > 
> > > Thank you for your patch to address the folder move problem between
> > > un-reconstructed mailboxes after the 2.3.16 -> 2.5.9 upgrade. I am
> > > not sure, but it looks like there may be another overly aggressive
> > > check. I keep getting these fatal errors from sync_client:
> > > 
> > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> > > user.robot 1696 (56412de8678bfb53f6cdb5fe4502031af5484056 
> > > )
> > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> > > user.robot 1697 (1b0024218a4419973b83ae3e84ac7133a4ab7d40 
> > > )
> > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> > > user.robot 1698 (f17084425d83bccb28a4dfa195846c7ef88c7567 
> > > )
> > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> > > user.robot 1699 (7a751e41e1d3a58e541298ab724be4c29d96e49d 
> > > )
> > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> > > user.robot 1700 (724a013d0ae97d27a1da33832487df1719681659 
> > > )
> > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: Fatal error: Internal 
> > > error: assertion failed: imap/mailbox.c: 2850: 
> > > !message_guid_isnull(>guid)
> > > 
> > > And then the sync_client has to be run manually and if lucky, it will
> > > process the full log successfully. I was looking in imap/mailbox.c
> > > and it looks like the assert at line 2850 may need a similar override
> > > for non-upgraded folders:
> > > 
> > > ---imap/mailbox.c---
> > > /* append a single message to a mailbox - also updates everything
> > >  * automatically.  These two functions are the ONLY way to modify
> > >  * the contents or tracking fields of a message */
> > > EXPORTED int mailbox_append_index_record(struct mailbox *mailbox,
> > > struct index_record *record)
> > > {
> > > indexbuffer_t ibuf;
> > > unsigned char *buf = ibuf.buf;
> > > size_t offset;
> > > int r;
> > > int n;
> > > struct utimbuf settime;
> > > uint32_t recno;
> > > 
> > > assert(mailbox_index_islocked(mailbox, 1));
> > > 
> > > /* Append MUST be a higher UID than any we've yet seen */
> > > assert(record->uid > mailbox->i.last_uid)
> > > 
> > > /* Append MUST have a message with data */
> > > assert(record->size);
> > > 
> > > =>/* GUID must not be null */
> > > =>assert(!message_guid_isnull(>guid));
> > > 
> > > /* belt AND suspenders - check the previous record too */
> > > if (mailbox->i.num_records) {
> > > struct index_record prev;
> > > 
> > > ---imap/mailbox.c---
> 

Re: sync_client errors out after 2.3.16 -> 2.5.9 upgrade

[Bron Gondwana via Info-cyrus]

You can't sync a mailbox without GUID for messages.  You need to upgrade the 
mailboxes before you can use them for replication.  The GUID is used for 
replication - if we allowed zero GUIDs, then every message would deduplicate to 
the same message!

On Tue, Aug 2, 2016, at 07:56, Kenneth Marshall via Info-cyrus wrote:
> Hi Cyrus Developers,
> 
> Thank you for your patch to address the folder move problem between
> un-reconstructed mailboxes after the 2.3.16 -> 2.5.9 upgrade. I am
> not sure, but it looks like there may be another overly aggressive
> check. I keep getting these fatal errors from sync_client:
> 
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> user.robot 1696 (56412de8678bfb53f6cdb5fe4502031af5484056 
> )
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> user.robot 1697 (1b0024218a4419973b83ae3e84ac7133a4ab7d40 
> )
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> user.robot 1698 (f17084425d83bccb28a4dfa195846c7ef88c7567 
> )
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> user.robot 1699 (7a751e41e1d3a58e541298ab724be4c29d96e49d 
> )
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch 
> user.robot 1700 (724a013d0ae97d27a1da33832487df1719681659 
> )
> Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: Fatal error: Internal error: 
> assertion failed: imap/mailbox.c: 2850: !message_guid_isnull(>guid)
> 
> And then the sync_client has to be run manually and if lucky, it will
> process the full log successfully. I was looking in imap/mailbox.c
> and it looks like the assert at line 2850 may need a similar override
> for non-upgraded folders:
> 
> ---imap/mailbox.c---
> /* append a single message to a mailbox - also updates everything
>  * automatically.  These two functions are the ONLY way to modify
>  * the contents or tracking fields of a message */
> EXPORTED int mailbox_append_index_record(struct mailbox *mailbox,
> struct index_record *record)
> {
> indexbuffer_t ibuf;
> unsigned char *buf = ibuf.buf;
> size_t offset;
> int r;
> int n;
> struct utimbuf settime;
> uint32_t recno;
> 
> assert(mailbox_index_islocked(mailbox, 1));
> 
> /* Append MUST be a higher UID than any we've yet seen */
> assert(record->uid > mailbox->i.last_uid)
> 
> /* Append MUST have a message with data */
> assert(record->size);
> 
> =>/* GUID must not be null */
> =>assert(!message_guid_isnull(>guid));
> 
> /* belt AND suspenders - check the previous record too */
> if (mailbox->i.num_records) {
> struct index_record prev;
> 
> ---imap/mailbox.c---
> 
> What do you think?
> 
> Regards,
> Ken
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: version of cyrus.index

[Bron Gondwana via Info-cyrus]

On Fri, Jul 29, 2016, at 18:51, Deniss via Info-cyrus wrote:
> Hello,
> 
> Is there any simple way to check version of cyrus.index format in 2.5
> installation ?

# perl -e 'sysread(\*STDIN, $buf, 12); (undef, undef, $v) = unpack('NNN', 
$buf); print "$v\n"' < ./data/user/foo/cyrus.index
13

Should work on any version of Perl going back ages.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Problem moving between folders during 2.3.16 -> 2.5.9 upgrade

[Bron Gondwana via Info-cyrus]

Thanks for reporting this.  Ellie, if you get a chance can you look at it?  I'm 
in the middle of the the FastMail security rollout right now, so I can't do 
anything today.

Bron.

On Mon, Jul 25, 2016, at 11:45, Kenneth Marshall wrote:
> Hi,
> 
> I accidentally hijacked another thread. So here is a new message. I am
> testing a Cyrus IMAP 2.3.16 to 2.5.9 upgrade and I am having problems
> moving messages between folders before the 'reconstruct -V max' has been
> run on the mailboxes. Here is the error from the IMAP client:
> 
> imap_copy_messages [a0080 NO Mailbox format corruption detected]?
> 
> and the details from the IMAP logs:
> 
> <1469384341 >1469384341>a0077 OK Completed
> <1469384354 >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> * LIST (\HasChildren) "/" DSPAM_notspam
> * LIST (\HasChildren) "/" DSPAM_spam
> * LIST (\HasNoChildren) "/" Drafts
> * LIST (\HasChildren) "/" Mail
> * LIST (\HasNoChildren) "/" Sent
> * LIST (\HasNoChildren) "/" Spam
> * LIST (\HasNoChildren) "/" Trash
> a0078 OK Completed (0.240 secs 356 calls)
> <1469384361 >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
> a0079 OK Completed
> <1469384361 >1469384362>a0080 NO Mailbox format corruption detected
> <1469384370 >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> * LIST (\HasChildren) "/" DSPAM_notspam
> * LIST (\HasChildren) "/" DSPAM_spam
> * LIST (\HasNoChildren) "/" Drafts
> * LIST (\HasChildren) "/" Mail
> * LIST (\HasNoChildren) "/" Sent
> * LIST (\HasNoChildren) "/" Spam
> * LIST (\HasNoChildren) "/" Trash
> a0081 OK Completed (0.230 secs 356 calls)
> <1469384372 a0083 MYRIGHTS "Drafts"
> a0084 SELECT "Drafts"
> >1469384372>a0082 OK Completed
> >1469384372>* MYRIGHTS Drafts lrswipkxtecda
> a0083 OK Completed
> >1469384372>* 0 EXISTS
> * 0 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old 
> \*)] Ok
> * OK [UIDVALIDITY 1345619846] Ok
> * OK [UIDNEXT 6] Ok
> * OK [HIGHESTMODSEQ 1] Ok
> * OK [URLMECH INTERNAL] Ok
> * OK [ANNOTATIONS 65536] Ok
> a0084 OK [READ-WRITE] Completed
> >1469384355>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> * LIST (\HasChildren) "/" DSPAM_notspam
> * LIST (\HasChildren) "/" DSPAM_spam
> * LIST (\HasNoChildren) "/" Drafts
> * LIST (\HasChildren) "/" Mail
> * LIST (\HasNoChildren) "/" Sent
> * LIST (\HasNoChildren) "/" Spam
> * LIST (\HasNoChildren) "/" Trash
> a0078 OK Completed (0.240 secs 356 calls)
> <1469384361 >1469384361>* STATUS Drafts (UIDVALIDITY 1345619846)
> a0079 OK Completed
> <1469384361 >1469384362>a0080 NO Mailbox format corruption detected
> <1469384370 >1469384370>* LIST (\Noinferiors \HasNoChildren) "/" INBOX
> * LIST (\HasChildren) "/" DSPAM_notspam
> * LIST (\HasChildren) "/" DSPAM_spam
> * LIST (\HasNoChildren) "/" Drafts
> * LIST (\HasChildren) "/" Mail
> * LIST (\HasNoChildren) "/" Sent
> * LIST (\HasNoChildren) "/" Spam
> * LIST (\HasNoChildren) "/" Trash
> a0081 OK Completed (0.230 secs 356 calls)
> <1469384372 a0083 MYRIGHTS "Drafts"
> a0084 SELECT "Drafts"
> >1469384372>a0082 OK Completed
> >1469384372>* MYRIGHTS Drafts lrswipkxtecda
> a0083 OK Completed
> >1469384372>* 0 EXISTS
> * 0 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $mdnsent Old 
> \*)] Ok
> * OK [UIDVALIDITY 1345619846] Ok
> * OK [UIDNEXT 6] Ok
> * OK [HIGHESTMODSEQ 1] Ok
> * OK [URLMECH INTERNAL] Ok
> * OK [ANNOTATIONS 65536] Ok
> a0084 OK [READ-WRITE] Completed
> <1469384392 >1469384392>a0085 OK Completed
> <1469384412 >1469384412>a0086 OK Completed
> 
> Is this a known issue with the upgrade process? Messages can be delivered to
> the folders via a sieve script, but both message moves and copies fail. If 
> there
> is any fix for this issue, it would be greatly appreciated. I have been 
> waiting
> since the release of 2.4.x for a version that would handle the I/O storm 
> caused
> by automatically upgrading the mailbox format on first access, and 2.5.x does
> manage that successfully. Thank you for any assistance.
> 
> Regards,
> Ken
> 
> On Fri, Jul 22, 2016 at 

Re: Quick advise regarding Cyrus migration

[Bron Gondwana via Info-cyrus]

Skiplist should be clean for a 32 bit to 64 bit architecture change.  It's all 
very type-careful internally.  It's also endian-change safe, because it uses 
network-byte-order for everything internally.

We definitely did a 32 bit to 64 bit conversion (probably slightly earlier - 
Debian 6 to 7) at FastMail, and we didn't have any problems.

Regards,

Bron.

On Sat, Jul 23, 2016, at 20:18, Roman Medina-Heigl Hernandez via Info-cyrus 
wrote:
> Hi guys,
> 
> I'm running Cyrus Imap 2.4.16 on Debian 7 (x86) 32 bits and I need to
> migrate to 2.4.17(nocaldav) on Debian 8 (x86) ** 64 bits **.
> 
> I've been reading some articles (like this one, which is 6 years old: 
> https://cynici.wordpress.com/2010/12/06/how-to-migrate-32-bit-cyrus-imapd-mailboxes-to-64-bit/
> ).
> 
> My question is: is it really needed to convert .db from skiplist to
> flag, and then back to skiplist , as suggested by many articles? (or
> even rebuild mailboxes like in last pointed article). Both machines are
> x86, being the only difference 32 vs 64 bits (endianness is same, etc).
> 
> I did a quick test simulating the migration, by simply just copying
> /var/lib/imap, /var/spool/mail and /var/spool/sieve. It seems it worked
> fine (or at least I couldn't find any problem when running imap with the
> migrated data). But I'd need your confirmation this should be sufficient
> and I won't find problems in the future.
> 
> I also played with db conversions (skiplist -> flat -> skiplist) and I
> have some questions:
> - I converted annotations.db (skiplist) to flat and then back to
> skiplist. It passed from 11K to 144 bytes (both skiplist)!!! Is it normal?
> - I did same exercise obtaining new skiplist db in both machines from a
> same flat. I got a vey similar (but not exactly the same) result .db in
> both 32 & 64 bits (some minor bytes may change but it didn't seem like
> big differences/changes). Maybe some different metadata (I mean not
> important), or I should concern about it?
> - apart from .db's from /var/lib/imap, should I worry about other
> skiplist like .seen files in /var/spool/mail?
> 
> Sorry if there are too much questions but this migration is important, I
> cannot loose any data (email data + email metadata: read marks, etc).
> 
> Please, advise!
> 
> Thank you so much in advance and have a nice day!
> 
> -- 
> Saludos,
> -Román
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Huge performance problems after updating from 2.4 to 2.5.8

[Bron Gondwana via Info-cyrus]

On Fri, Jul 15, 2016, at 23:52, Bron Gondwana via Info-cyrus wrote:
> On Fri, Jul 15, 2016, at 23:46, Andy Dorman via Info-cyrus wrote:
> > So if the issue apparently lies with twoskip, can we keep our dbs using 
> > skiplist and do the 2.4 -> 2.5 upgrade?  Is it possible -h could revert 
> > back to skiplist?
> 
> You can convert a db back just by changing the setting in imapd.conf and 
> restarting - it will convert itself.
> 
> mboxlist_db: skiplist
> 
> > If it could help we could test upgrading to 2.5.8 on our dev server 
> > while leaving our database(s) as skiplist.
> 
> Twoskip should be fixed in 2.5.9.  I've worked out what went wrong, and am 
> working on patches now :)

Patches have passed all the tests - they're on master and 2.5.  I'm pushing to 
FastMail's testing stores and then going to bed :)

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Huge performance problems after updating from 2.4 to 2.5.8

[Bron Gondwana via Info-cyrus]

On Fri, Jul 15, 2016, at 23:46, Andy Dorman via Info-cyrus wrote:
> So if the issue apparently lies with twoskip, can we keep our dbs using 
> skiplist and do the 2.4 -> 2.5 upgrade?  Is it possible -h could revert 
> back to skiplist?

You can convert a db back just by changing the setting in imapd.conf and 
restarting - it will convert itself.

mboxlist_db: skiplist

> If it could help we could test upgrading to 2.5.8 on our dev server 
> while leaving our database(s) as skiplist.

Twoskip should be fixed in 2.5.9.  I've worked out what went wrong, and am 
working on patches now :)

Bron.


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Huge performance problems after updating from 2.4 to 2.5.8

[Bron Gondwana via Info-cyrus]

So we've discussed this at length in #cyrus on freenode, and concluded that the 
issue is that twoskip is doing far too many munmap / mmap calls during an 
unlocked foreach (which is what the LIST command uses).

I've filed a bug:

https://github.com/cyrusimap/cyrus-imapd/issues/5

And I'm looking at what it would take to fix this behaviour in twoskip now 
(yes, I know I should be sleeping, but I'm not going to be able to sleep until 
I understand how this got broken!)

Bron.

On Fri, Jul 15, 2016, at 20:41, Hynek Schlawack via Info-cyrus wrote:
> Hello,
> 
> we’ve updated one of our Cyrus IMAP backends from 2.4 to 2.5.8 on FreeBSD 
> 10.3 with ZFS and now we have an operational emergency.
> 
> Cyrus IMAPd starts fine and keeps working for about 5 to 20 minutes (rather 
> sluggishly tho).  At some point the server load starts growing and explodes 
> eventually until we have to restart the IMAP daemons which gives us another 5 
> to 20 minutes.
> 
> It doesn’t really matter if we run `reconstruct` in the background or not.
> 
> 
> # Observations:
> 
> 1. While healthy, the imapd daemons’s states are mostly `select` or `RUN`.  
> Once things get critical they all are mostly in `zfs` (but do occasionally 
> switch).
> 2. Customers report that their mail clients are downloading all e-mails.  
> That’s obviously extra bad given we seem to run in some kind of I/O problems. 
>  Running `truss` on busy imapd processes seem to confirm that.
> 3. Once hell breaks loose, IO collapses even on other file systems/hard disks.
> 4. `top` mentions processes in `lock` state – sometimes even more than 200.  
> That’s nothing we see on our other backends.
> 5. There seems to be a correlation between processes hanging in `zfs` state 
> and `truss` showing them accessing mailboxes.db.  Don’t know if it’s related, 
> but soon after the upgrade, mailboxes.db broke and we had to reconstruct it.
> 
> 
> # Additional key data:
> 
> - 25,000 accounts
> - 4.5 TB data
> - 64 GB RAM, no apparent swapping
> - 16 cores CPU
> - nginx in front of it.
> 
> ## zpool iostat 5
> 
>capacity operationsbandwidth
> poolalloc   free   read  write   read  write
> --  -  -  -  -  -  -
> tank4.52T   697G144  2.03K  1.87M  84.2M
> tank4.52T   697G 84730  2.13M  3.94M
> tank4.52T   697G106904  2.78M  4.52M
> tank4.52T   697G115917  3.07M  5.11M
> tank4.52T   697G101   1016  4.04M  5.06M
> tank4.52T   697G124  1.03K  3.27M  6.59M
> 
> Which doesn’t look special.
> 
> The data used to be on HDDs and worked fine with an SSD ZIL.  After the 
> upgrade and ensuing problems we tried a Hail Mary by replacing the HDDs thru 
> SSDs to no avail (migrated a ZFS snapshot for that).
> 
> So we do *not* believe it’s really a traditional I/O bottleneck since it only 
> started *after* the upgrade to 2.5 and did not go away by adding SSDs.  The 
> change notes led us to believe that there shouldn’t be any I/O storm due to 
> mailbox conversions but is it true in any case?  How could we double check?  
> Observation #2 from above leads us to believe that there are in fact some 
> meta data problems.  We’re reconstructing in the background but that’s going 
> to take days; which is sadly time we don’t really have.
> 
> ## procstat -w 1 of an active imapd
> 
>   PID  PPID  PGID   SID  TSID THR LOGINWCHAN EMUL  COMM   
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor - FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor - FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor - FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor *vm objec FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor - FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd  
>  
> 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 

Re: Huge performance problems after updating from 2.4 to 2.5.8

[Bron Gondwana via Info-cyrus]

Squatter indexes will be broken without the latest patches on the 2.5 branch.  
The data gets indexed differently, and I had to patch the squatter code to 
search both styles.

Bron.

On Fri, Jul 15, 2016, at 22:17, Andre Felipe Machado via Info-cyrus wrote:
> Hello,
> Maybe, after such upgrade, squatter metadata indexes were lost and you should 
> run an incremental  squatter again on your mailboxes. 
> Even before the scheduled run at events section on /etc/cyrus.conf.
> Regards.
> Andre Felipe
> 
> 
> 
> Hynek Schlawack via Info-cyrus  wrote ..
> > Hello,
> > 
> > we’ve updated one of our Cyrus IMAP backends from 2.4 to 2.5.8 on FreeBSD 
> > 10.3
> > with ZFS and now we have an operational emergency.
> > 
> > Cyrus IMAPd starts fine and keeps working for about 5 to 20 minutes (rather 
> > sluggishly
> > tho).  At some point the server load starts growing and explodes eventually 
> > until
> > we have to restart the IMAP daemons which gives us another 5 to 20 minutes.
> > 
> > It doesn’t really matter if we run `reconstruct` in the background or not.
> > 
> > 
> > # Observations:
> > 
> > 1. While healthy, the imapd daemons’s states are mostly `select` or `RUN`.  
> > Once
> > things get critical they all are mostly in `zfs` (but do occasionally 
> > switch).
> > 2. Customers report that their mail clients are downloading all e-mails.  
> > That’s
> > obviously extra bad given we seem to run in some kind of I/O problems.  
> > Running
> > `truss` on busy imapd processes seem to confirm that.
> > 3. Once hell breaks loose, IO collapses even on other file systems/hard 
> > disks.
> > 4. `top` mentions processes in `lock` state – sometimes even more than 200. 
> > That’s nothing we see on our other backends.
> > 5. There seems to be a correlation between processes hanging in `zfs` state 
> > and
> > `truss` showing them accessing mailboxes.db.  Don’t know if it’s related, 
> > but
> > soon after the upgrade, mailboxes.db broke and we had to reconstruct it.
> > 
> > 
> > # Additional key data:
> > 
> > - 25,000 accounts
> > - 4.5 TB data
> > - 64 GB RAM, no apparent swapping
> > - 16 cores CPU
> > - nginx in front of it.
> > 
> > ## zpool iostat 5
> > 
> >capacity operationsbandwidth
> > poolalloc   free   read  write   read  write
> > --  -  -  -  -  -  -
> > tank4.52T   697G144  2.03K  1.87M  84.2M
> > tank4.52T   697G 84730  2.13M  3.94M
> > tank4.52T   697G106904  2.78M  4.52M
> > tank4.52T   697G115917  3.07M  5.11M
> > tank4.52T   697G101   1016  4.04M  5.06M
> > tank4.52T   697G124  1.03K  3.27M  6.59M
> > 
> > Which doesn’t look special.
> > 
> > The data used to be on HDDs and worked fine with an SSD ZIL.  After the 
> > upgrade
> > and ensuing problems we tried a Hail Mary by replacing the HDDs thru SSDs 
> > to no
> > avail (migrated a ZFS snapshot for that).
> > 
> > So we do *not* believe it’s really a traditional I/O bottleneck since it 
> > only
> > started *after* the upgrade to 2.5 and did not go away by adding SSDs.  The 
> > change
> > notes led us to believe that there shouldn’t be any I/O storm due to mailbox
> > conversions but is it true in any case?  How could we double check?  
> > Observation
> > #2 from above leads us to believe that there are in fact some meta data 
> > problems.
> > We’re reconstructing in the background but that’s going to take days; which
> > is sadly time we don’t really have.
> > 
> > ## procstat -w 1 of an active imapd
> > 
> >   PID  PPID  PGID   SID  TSID THR LOGINWCHAN EMUL  COMM 
> >   
> > 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor - FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor - FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor - FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor *vm objec FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor zfs   FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor - FreeBSD ELF64 imapd
> >   
> > 45016 43150 43150 43150 0   1 toor zfs   

git.cyrus.foundation deprecated

[Bron Gondwana via Info-cyrus]

Hi all,

Phabricator is being deprecated.  The project is moving to github at:

https://github.com/cyrusimap/

I'm working on migrating all the bugs from bugzilla and phabricator to be 
github issues.  For now the website is still hosted at CMU.

One thing I'm doing is applying a "kill commit" to the top of the git 
repositories at the other locations.  This doesn't lose any history, you can 
just reset --hard HEAD^ to get back to the latest master commit - but it does 
discourage any further pushes of commits that get lost, because they won't 
merge or rebase very happily on top of the kill commit!

The kill commit contains a small README.txt which tells you how to update your 
git repository to point to the new location.

Cheers,

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squat broken after upgrade from 2.4 to 2.5?

[Bron Gondwana via Info-cyrus]

This could very well be a bug in the upgrade :(  It would be great to have a 
test case to check this.  I can't do that today, but maybe ellie can do 
something.

Sadly our bug tracking situtation for Cyrus is a mess at the moment, but I'm 
hoping to have that cleaned up soon.

Bron.

On Mon, Jun 20, 2016, at 19:33, Wolfgang Breyha via Info-cyrus wrote:
> Hi!
> 
> I upgraded some backend hosts from 2.4.18 to 2.5.8. Currently they are all
> still running with index version 12.
> 
> Some of these mailboxes have squatter activated as annotation and my
> cyrus.conf has
> squatter  cmd="squatter -asir *" at=0500
> 
> I see that it runs at night and logs entries like:
> Jun 20 05:00:15 lauren squatter[27593]: indexing mailbox user.
> 
> There are no errors in the log from squatter.
> 
> But today I recognized on my own mailbox that squatter returns nothing while
> searching with thunderbird. The resulting imap request looked like:
> <1466414132<18 UID Search UNDELETED (OR (OR OR TO "search" HEADER CC "search"
> SUBJECT "search") BODY "search")
> <1466414132>* SEARCH
> 18 OK Completed (0 msgs in 0.010 secs)
> 
> I then moved the current squat db and the same search returned 214 messages.
> 
> Rebuiling the squat db and searching again resulted in the exact same 214
> messages.
> 
> It seems that something changed within the squat DB and 2.5 can't search in
> the old 2.4 DBs. That's bad in two ways.
> 
> *) search doesn't fail with an error and simply returns no result. no fallback
> to searching without squat db.
> *) the squat db is not refreshed/rebuilt after upgrade.
> 
> Is this a bug or is something broken on my side?
> 
> Greetings, Wolfgang
> -- 
> Wolfgang Breyha  | http://www.blafasel.at/
> Vienna University Computer Center | Austria
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus and scale-out

[Bron Gondwana via Info-cyrus]

Funny you should ask :)

http://asg.andrew.cmu.edu/archive/message.php?mailbox=archive.cyrus-devel=4939

I definitely have plans of allowing everything to be written back in a reliable 
way so that losing an IMAP server is guaranteed(within the bounds of software 
reliability and all the parts following their contracts) to not lose anything 
which has been acknowledged back to the user!

Bron.

On Sat, Jun 11, 2016, at 01:18, Sebastian Hagedorn via Info-cyrus wrote:
> Hi,
> 
> our systems guys keep telling us that we are doing things in an 
> old-fashioned way and should get with the program.
> 
> We are currently using a single Cyrus server with roughly 13 TB of storage 
> provided by a SAN. We used to have a Red Hat High Availability cluster, but 
> we traded that in for a VMWare HA setup earlier this year. So far we have 
> scaled up. We have added processors, RAM and storage to that single 
> (virtual) machine whenever necessary.
> 
> According to our systems people, we should scale out instead, the way 
> Exchange 2013 and Dovecot Pro apparently do. The idea, as I understand it, 
> is to have multiple backends that all provide access to the same mailboxes. 
> It should be possible to add and remove backends completely transparently. 
> Dovecot Pro seems to realize that by storing all mails in local caches 
> backed by shared object storage (e.g. Ceph), in conjunction with Dovecot 
> Director.
> 
> Now I'm trying to understand if anything like that is on the roadmap for 
> Cyrus. I see that Cyrus 3.0 (experimentally) supports object storage, but 
> only for archive partitions. Are there plans for Cyrus 3.1 or later to add 
> support for regular mail partitions as well?
> 
> Personally I'm stil happy with our setup, but I'm told that future storage 
> hardware won't easily support what we're doing anymore. I'm aware that both 
> clustering and replication are already possible with Cyrus, but my 
> understanding is that you can't trivially and automatically switch to a 
> replicated backend if one goes down. You also need to replicate all 
> messages to each new backend you introduce, which isn't quite what our 
> systems people would like to have.
> 
> Thanks
> Sebastian
> -- 
> .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
>  .:.Regionales Rechenzentrum (RRZK).:.
>.:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> Email had 1 attachment:
> + Attachment1.2
>   1k (application/pgp-signature)


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Bron Gondwana via Info-cyrus]

On Fri, Jun 10, 2016, at 09:41, Jason L Tibbitts III wrote:
> > "BG" == Bron Gondwana  writes:
> 
> BG> Just to be really clear what this is.  It's per mailbox name - if
> BG> you create and delete the SAME mailbox more 20 times, it only keeps
> BG> the most recent 20 of that mailbox.
> 
> Hmm.  That's much less problematic, but it still allows someone to force
> something to be deleted if they really want it to be deleted.  That's
> not really an issue for me because my users wouldn't figure it out, but
> I can imagine that someone using delayed expiry to easily implement some
> sort of legal requirement might be unhappy.  But that's somewhat of a
> stretch.

Yep.

Anyway, magic numbers are bad, so I will make this configurable.  It's easy to
do, and if people with different systems need it changed, then that's fine.

With uniqueid based storage it will all be nicer anyway :)

Bron.


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Bron Gondwana via Info-cyrus]

On Fri, Jun 10, 2016, at 04:38, Jason L Tibbitts III wrote:
> >>>>> "BG" == Bron Gondwana via Info-cyrus <info-cyrus@lists.andrew.cmu.edu> 
> >>>>> writes:
> 
> BG> How would you suggest we protect against exploiting delayed delete
> BG> to fill the server without going over quota?
> 
> Well, I don't even run quotas.  But I do keep deleted messages around
> for 12 weeks, and even if I didn't, I do delete accounts occasionally.
> Deleting one account would go over the limit, and though I suck the
> messages out to mbox format for the final archiving, an instant nuke of
> older mailboxes would prevent an "easy" restore.
> 
> BG> Maybe a new quota
> BG> field for "total mailbox usage including deleted stuff" that can be
> BG> set to a high enough value that no reasonable user will ever hit it?
> 
> As long as I can just set it to 'unlimited', I don't care.  Disk is
> cheap and I don't have enough users to worry about it.  But I've had
> people delete all 100+ of their mailboxes before, and come screaming.

Just to be really clear what this is.  It's per mailbox name - if you create 
and delete
the SAME mailbox more 20 times, it only keeps the most recent 20 of that 
mailbox.

If you accidentally delete 100 mailboxes, they'll all still be there.

And it doesn't stop you deleting mailboxes or anything - it just immediately 
cleans
up the 21st one when you delete the mailbox again.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Bron Gondwana via Info-cyrus]

On Thu, Jun 9, 2016, at 03:02, Andre Felipe Machado via Info-cyrus wrote:
> Hello,
> At future release notes I read
> "Under delete_mode: delayed, only the 20 most recently deleted mailboxes are 
> kept for any given name."
> https://cyrusimap.org/imap/release-notes/3.0/x/3.0.0-beta2.html
> Is there any configuration parameter to increase this limit?
> Why this limit is needed?

denial of service / space wastage protection.  There's no config option 
available right now.  I could be convinced to change it.

How would you suggest we protect against exploiting delayed delete to fill the 
server without going over quota?  Maybe a new quota field for "total mailbox 
usage including deleted stuff" that can be set to a high enough value that no 
reasonable user will ever hit it?

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sieve runtime error Fileinto: Permission denied

[Bron Gondwana via Info-cyrus]

On Tue, May 24, 2016, at 20:16, Patrick Boutilier via Info-cyrus wrote:
> On 05/24/2016 01:56 AM, OBATA Akio via Info-cyrus wrote:
> > On Tue, 24 May 2016 12:32:07 +0900, Bron Gondwana via Info-cyrus
> > <info-cyrus@lists.andrew.cmu.edu> wrote:
> >
> >> On Tue, May 24, 2016, at 10:44, OBATA Akio via Info-cyrus wrote:
> >>> On Tue, 24 May 2016 07:25:42 +0900, Bron Gondwana via Info-cyrus
> >>> <info-cyrus@lists.andrew.cmu.edu> wrote:
> >>>
> >>> > On Mon, May 23, 2016, at 22:47, Sundeep Singh Nanuwa via Info-cyrus
> >>> wrote:
> >>> >> On 23/05/16 13:35, Bron Gondwana via Info-cyrus wrote:
> >>> >> > You need to have "anyone p" acl to fileinto anything other than
> >>> inbox.
> >>> >> That didn't work unfortunately.
> >>> >
> >>> > Deliver into INBOX works, fileinto doesn't.  It's ACLs for sure.
> >>> You need the 'p'
> >>> > ACL set for the user that lmtpd runs as (or anyone).
> >>> >
> >>> > Unless there's a bug in your particular version of Cyrus, that
> >>> should work.  If you
> >>> > could give us that and a copy of your lam output again with the
> >>> anyone ACLs set,
> >>> > maybe that will help.
> >>>
> >>> In which version of Cyrus release, 'p' permission is required even
> >>> with "sieve fileinto"?
> >>> I know that only direct lmtp deliver with subaddress require it.
> >>
> >> Within lmtpd, subaddress delivery and fileinto are identical.  I've
> >> just checked back to the 2.3 branch and the logic is the same there
> >> too - if there's an error delivering to the named mailbox, we fall
> >> back to the INBOX with an authstate based on the username, which is
> >> why you don't need 'p' on the INBOX.
> >
> > I'm using "fileinto" without 'anyone p' permission on 2.4.18.
> > I believe that sieve scripts will run as the user, whereas subaddress is
> > lmtpd user.
> >
> 
> Same here. I don't have "anyone p" set and fileinto works on 2.4.18 .

Hrm, OK.  I guess I haven't spent as much time in that area of the code,
and my understanding was wrong.  Doesn't explain why default delivery
works but fileinto doesn't for this case then...

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sieve runtime error Fileinto: Permission denied

[Bron Gondwana via Info-cyrus]

On Tue, May 24, 2016, at 10:44, OBATA Akio via Info-cyrus wrote:
> On Tue, 24 May 2016 07:25:42 +0900, Bron Gondwana via Info-cyrus 
> <info-cyrus@lists.andrew.cmu.edu> wrote:
> 
> > On Mon, May 23, 2016, at 22:47, Sundeep Singh Nanuwa via Info-cyrus wrote:
> >> On 23/05/16 13:35, Bron Gondwana via Info-cyrus wrote:
> >> > You need to have "anyone p" acl to fileinto anything other than inbox.
> >> That didn't work unfortunately.
> >
> > Deliver into INBOX works, fileinto doesn't.  It's ACLs for sure.  You need 
> > the 'p'
> > ACL set for the user that lmtpd runs as (or anyone).
> >
> > Unless there's a bug in your particular version of Cyrus, that should work. 
> >  If you
> > could give us that and a copy of your lam output again with the anyone ACLs 
> > set,
> > maybe that will help.
> 
> In which version of Cyrus release, 'p' permission is required even with 
> "sieve fileinto"?
> I know that only direct lmtp deliver with subaddress require it.

Within lmtpd, subaddress delivery and fileinto are identical.  I've just 
checked back to the 2.3 branch and the logic is the same there too - if there's 
an error delivering to the named mailbox, we fall back to the INBOX with an 
authstate based on the username, which is why you don't need 'p' on the INBOX.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sieve runtime error Fileinto: Permission denied

[Bron Gondwana via Info-cyrus]

On Mon, May 23, 2016, at 22:47, Sundeep Singh Nanuwa via Info-cyrus wrote:
> On 23/05/16 13:35, Bron Gondwana via Info-cyrus wrote:
> > You need to have "anyone p" acl to fileinto anything other than inbox.
> That didn't work unfortunately.

Deliver into INBOX works, fileinto doesn't.  It's ACLs for sure.  You need the 
'p'
ACL set for the user that lmtpd runs as (or anyone).

Unless there's a bug in your particular version of Cyrus, that should work.  If 
you
could give us that and a copy of your lam output again with the anyone ACLs set,
maybe that will help.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sieve runtime error Fileinto: Permission denied

[Bron Gondwana via Info-cyrus]

You need to have "anyone p" acl to fileinto anything other than inbox.

Bron.  

On Mon, May 23, 2016, at 21:52, Sunny via Info-cyrus wrote:
> Hi,
> 
> I'm seeing the following error logs with some users sieve filtering
> May 23 11:41:31 imap02 lmtpunix[26005]: sieve runtime error for test2 id 
> <5742de5a.3040...@org.ac.uk>: Fileinto: Permission denied
> 
> Essentially emails are not being put into the specific folders.
> 
> Their cyrus:mail permissions are in place
> 
> [root@imap02 test2]# ls -ls /var/spool/imap/t/user/test2/
>   4 -rw--- 1 cyrus mail  1663 May 23 10:36 747.
>   4 -rw--- 1 cyrus mail  1904 May 23 10:53 748.
>   4 -rw--- 1 cyrus mail  1661 May 23 10:54 749.
>   4 -rw--- 1 cyrus mail  1666 May 23 10:56 750.
>   4 -rw--- 2 cyrus mail  1613 May 23 11:00 751.
>   4 -rw--- 1 cyrus mail  1541 May 23 11:12 752.
>   4 -rw--- 1 cyrus mail  1541 May 23 11:20 753.
>   4 -rw--- 1 cyrus mail  1537 May 23 11:22 754.
>   4 -rw--- 1 cyrus mail  1537 May 23 11:23 755.
>   4 -rw--- 1 cyrus mail  1539 May 23 11:40 756.
>   4 -rw--- 1 cyrus mail  1541 May 23 11:41 757.
> 20 -rw--- 1 cyrus mail 16424 May 23 11:41 cyrus.cache
>   4 -rw--- 1 cyrus mail   184 May 23 11:41 cyrus.header
>   4 -rw--- 1 cyrus mail  1768 May 23 11:41 cyrus.index
>   4 drwx-- 2 cyrus mail  4096 May 23 11:41 Drafts
>   4 drwx-- 2 cyrus mail  4096 May 23 11:41 for-sale
>   4 drwx-- 2 cyrus mail  4096 May 23 11:41 hello
>   4 drwx-- 2 cyrus mail  4096 May 23 12:11 Sent
> 12 drwx-- 2 cyrus mail 12288 May 23 11:41 Trash
> 
> 
> cyrus permissions - (I changed user cyrus permission to full to see if 
> it worked - but it didn't)
> 
> localhost.localdomain> lam user.test2*
> user.test2:
>test2 lrswipkxtea
>cyrus lrswipkxtecda
> user.test2.Drafts:
>test2 lrswipkxtea
>cyrus lrswipkxtecda
> user.test2.Sent:
>test2 lrswipkxtea
>cyrus lrswipkxtecda
> user.test2.Trash:
>test2 lrswipkxtea
>cyrus lrswipkxtecda
> user.test2.for-sale:
>test2 lrswipkxtea
>cyrus lrswipkxtecda
> user.test2.hello:
>test2 lrswipkxtea
>cyrus lrswipkxtecda
> user.test2.ssn:
>test2 lrswipkxtea
>cyrus lrswipkxtecda
> 
> cyrus.conf
> SERVICES {
># add or remove based on preferences
>imapcmd="imapd" listen="imap" prefork=5
>imapscmd="imapd -s" listen="imaps" prefork=1
>pop3cmd="pop3d" listen="pop3" prefork=3
>pop3scmd="pop3d -s" listen="pop3s" prefork=1
>sievecmd="timsieved" listen="sieve" prefork=0
>oldsieve  cmd="timsieved" listen="2000" prefork=0
> 
> 
> Other sieve functions work like "reject" "vacation" but not fileinto.
> 
> Are there any other suggestions?
> 
> Regards
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: lmtpd triggering a delivery.db checkpointing (Cyrus 2.3.16)

[Bron Gondwana via Info-cyrus]

> What we do at FastMail to make deliver.db not suck is store it on tmpfs.  The 
> repack is tons faster.  Sure you lose it over a full server restart, but all 
> you lose is the duplicate suppression.  If you wanted to be really clever 
> about it, you could copy the file during the shutdown script and maybe once 
> per hour otherwise, and copy it back onto tmpfs during startup.
> 
> duplicate_db_path: /var/run/cyrus/duplicate.db

oh right, 2.3.x doesn't have duplicate_db_path.

I think your choices are either to hack that option into your codebase so that 
you can move the duplicate DB onto tmpfs, live with what you've got (possibly 
by putting /var/imap on fast disk/SSD), or upgrade to a Cyrus that isn't 10 
years old!

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: lmtpd triggering a delivery.db checkpointing (Cyrus 2.3.16)

[Bron Gondwana via Info-cyrus]

On Tue, May 17, 2016, at 22:51, Eric Luyten via Info-cyrus wrote:
> On Tue, May 17, 2016 11:45 am, Simon Matter wrote:
> >> Hi,
> >>
> >>
> >>
> >> Several times a month our server freezes up on deliveries and the system
> >> load average shoots up into the hundreds. Things quickly return to normal
> >> between one and two minutes later but this has always puzzled me.
> >>
> >> Today I was watching the system from up close when it happened.
> >>
> >>
> >>
> >> May 17 10:59:14  lmtp[24980]: skiplist: checkpointed
> >> /ssd/cyrs/imap/deliver.db (223062 records, 25295200 bytes) in 119 seconds
> >>
> >>
> >>
> >>
> >> I took a quick dive into the code but could not find where and when lmtpd
> >> is supposed to trigger a delivery.db checkpointing action.
> >
> > Isn't it controlled by 'checkpointcmd="ctl_cyrusdb -c" period=30' in
> > cyrus.conf?
> 
> 
> Okay, I think I found the code in   lib/cyrusdb_skiplist.c
> 
> We do indeed have the (default) 'checkpoint  cmd="ctl_cyrusdb -c" period=30'
> entry in cyrus.conf, 30 referring to the number of minutes between 
> invocations.
> 
> We prune deliver.db every night at 00:55 with -E 1
> 
> 
> So I guess the phenomenon I witnessed this morning correlates with server
> business in the area of deliveries.
> A Cyrus Wiki page hints at reducing the number of minutes down from 30.
> 
> "The most common one is that you need to checkpoint the cyrusdb more often.
>  This can be done with a simple ctl_cyrusdb -c If you do this very often,
>  the amount of log that needs to be recovered will be significantly shorter.
>  We recommend doing this at least once every half hour, and more often on
>  busy sites. "
> (http://cyrusimap.web.cmu.edu/mediawiki/index.php/FAQ)

Urgh: 2.3.x.

Sadly, that's not really hooked up nicely and the terminology is really muddy.
Skiplist databases will rewrite themselves as a more compact version when they
reach a certain ratio of ADD records to INORDER records.

This isn't exposed outside cyrusdb_skiplist.c until 2.5, and it's not hooked 
into
ctl_cyrusdb's "checkpoint" operation, which just calls a sync on each database
engine:

case CHECKPOINT:
r2 = (*(dblist[i].env))->sync();

and then takes a backup of the files with:
r2 = (*(dblist[i].env))->archive((const char**) archive_files,
 backup1);


sync does nothing:

static int mysync(void)
{
return 0;
}


archive takes copies of the files (without even locking!)

static int myarchive(const char **fnames, const char *dirname)
{
int r;
const char **fname;
char dstname[1024], *dp;
int length, rest;

strlcpy(dstname, dirname, sizeof(dstname));
length = strlen(dstname);
dp = dstname + length;
rest = sizeof(dstname) - length;

/* archive those files specified by the app */
for (fname = fnames; *fname != NULL; ++fname) {
syslog(LOG_DEBUG, "archiving database file: %s", *fname);
strlcpy(dp, strrchr(*fname, '/'), rest);
r = cyrusdb_copyfile(*fname, dstname);
if (r) {
syslog(LOG_ERR,
   "DBERROR: error archiving database file: %s", *fname);
return CYRUSDB_IOERROR;
}
}

return 0;
}

...

These are identical right up to 3.0, though they're factored out into
"generic sync" and "generic archive".  So ctl_cyrusdb checkpoint
doesn't actually do much worthwhile work.

At least in 3.0 you can use cyr_dbtool to checkpoint a database
explicitly if you want to:

sudo -u cyrus cyr_dbtool /var/imap/deliver.db skiplist repack

But you're running 2.3.x, so none of my last 6 years of work are
available to you!

---

What we do at FastMail to make deliver.db not suck is store it on tmpfs.  The 
repack is tons faster.  Sure you lose it over a full server restart, but all 
you lose is the duplicate suppression.  If you wanted to be really clever about 
it, you could copy the file during the shutdown script and maybe once per hour 
otherwise, and copy it back onto tmpfs during startup.

duplicate_db_path: /var/run/cyrus/duplicate.db

(where /var/run is a tmpfs on our systems)

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: altnamespace doesn't seem to be working in 2.4.17

[Bron Gondwana via Info-cyrus]

Indeed, never use a user listed in admins as your actual email account. It has 
weird behavior around inbox itself too.

On Sun, May 1, 2016, at 16:23, Bron Gondwana wrote:
> Is your user listed in admins in imapd.conf? Admin users never get alt 
> namespace.
> 
> On Sat, Apr 30, 2016, at 23:23, Brian J. Murrell via Info-cyrus wrote:
> > In 2.4.17 of Cyrus IMAPD I have the following /etc/imapd.conf:
> > 
> > configdirectory: /var/lib/imap
> > partition-default: /var/spool/imap
> > admins: cyrus brian
> > sievedir: /var/lib/imap/sieve
> > sendmail: /usr/sbin/sendmail
> > hashimapspool: true
> > sasl_pwcheck_method: saslauthd
> > sasl_mech_list: LOGIN GSSAPI PLAIN
> > allowplaintext: no
> > defaultdomain: mail
> > tls_cert_file: /etc/pki/cyrus-imapd/cert.pem
> > tls_key_file: /etc/pki/cyrus-imapd/privkey.pem
> > tls_ca_file: /etc/pki/cyrus-imapd/fullchain.pem
> > altnamespace: yes
> > autocreatequota: 0
> > lmtp_downcase_rcpt: yes
> > sasl_keytab: /etc/krb5_cyrus.keytab
> > autocreate_post: 1
> > 
> > Yet I am still seeing my folders as children of INBOX.  Is there
> > something I am missing?
> > 
> > Cheers,
> > b.
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > Email had 1 attachment:
> > + signature.asc
> >   1k (application/pgp-signature)
> 
> 
> -- 
>   Bron Gondwana
>   br...@fastmail.fm


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: altnamespace doesn't seem to be working in 2.4.17

[Bron Gondwana via Info-cyrus]

Is your user listed in admins in imapd.conf? Admin users never get alt 
namespace.

On Sat, Apr 30, 2016, at 23:23, Brian J. Murrell via Info-cyrus wrote:
> In 2.4.17 of Cyrus IMAPD I have the following /etc/imapd.conf:
> 
> configdirectory: /var/lib/imap
> partition-default: /var/spool/imap
> admins: cyrus brian
> sievedir: /var/lib/imap/sieve
> sendmail: /usr/sbin/sendmail
> hashimapspool: true
> sasl_pwcheck_method: saslauthd
> sasl_mech_list: LOGIN GSSAPI PLAIN
> allowplaintext: no
> defaultdomain: mail
> tls_cert_file: /etc/pki/cyrus-imapd/cert.pem
> tls_key_file: /etc/pki/cyrus-imapd/privkey.pem
> tls_ca_file: /etc/pki/cyrus-imapd/fullchain.pem
> altnamespace: yes
> autocreatequota: 0
> lmtp_downcase_rcpt: yes
> sasl_keytab: /etc/krb5_cyrus.keytab
> autocreate_post: 1
> 
> Yet I am still seeing my folders as children of INBOX.  Is there
> something I am missing?
> 
> Cheers,
> b.
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mailboxes.db invalid entries

[Bron Gondwana via Info-cyrus]

The tool is safe to use with Cyrus running

Of course if you delete a mailboxes db entry without deleting the underlying 
files and then recreate the folder, things can get confused because there are 
already files on disk...

Bron

On Sat, Apr 23, 2016, at 06:58, Jan Kowalsky via Info-cyrus wrote:
> Hi all,
> 
> Am 22.04.2016 um 22:28 schrieb Jan Kowalsky:
> > Hi Wolfgang,
> > 
> > thanks for your answer.
> > 
> > Am 22.04.2016 um 11:06 schrieb Wolfgang Breyha:
> >> Jan Kowalsky via Info-cyrus wrote on 22/04/16 01:28:
> >>> First I tried to dump the mailbox.db with ctl_mboxlist -d 
> >>> /tmp/mailboxes.txt
> >>>
> >>> After deleting the wrong entry manually I wanted to reload the mailbox
> >>> again with ctl_mboxlist -u /tmp/mailboxes.txt. All operation with
> >>> stopped cyrus.
> >>
> >> Have you renamed your mailboxes.db after using -d and before using -u?
> >> Otherwise ctl_mboxlist will import your dump into the existing 
> >> mailboxes.db.
> >>
> >> And are this exactly the commands you used?
> > 
> > Yes. But I'll give this a try:
> > 
> >> I think
> >> ctl_mboxlist -d >/tmp/mailboxes.txt
> >> and
> >> ctl_mboxlist -f /tmp/mailboxes.db.new -u  >> should be used since both read/write to STDIN/OUT.
> > 
> > unfortunately with the same result:
> > 
> > root@mail:~/cyrus-debug# /usr/lib/cyrus-imapd/ctl_mboxlist -d >
> > /tmp/mboxlist.txt
> > root@mail:~/cyrus-debug# vi /tmp/mboxlist.txt
> > root@mail:~/cyrus-debug# /usr/lib/cyrus-imapd/ctl_mboxlist -f
> > /tmp/mailboxes.db.new -u  < /tmp/mboxlist.txt
> > fatal error: failed to mmap /tmp/mailboxes.db.new.NEW file
> > root@mail:~/cyrus-debug# ls -l /tmp/mailboxes.db.new*
> > -rw--- 1 cyrus mail 2219712 Apr 22 22:06 /tmp/mailboxes.db.new
> > -rw--- 1 cyrus mail   11400 Apr 22 22:06 /tmp/mailboxes.db.new.NEW
> > 
> > I found the cyr_dbtool command.
> > 
> > Is the "cyr_dbtool delete key" maybe meant for this too?
> 
> well, this worked:
> 
> /usr/lib/cyrus-imapd/cyr_dbtool /srv/imap/config/mailboxes.db skiplist
> delete example.com\!user.kontakt.Calendar
> 
> good to know. I did it with stopped Cyrus. Anybody know if this tool is
> also intended to use it while cyrus is running?
> 
> https://cyrusimap.org/imap/admin/commands/cyr_dbtool.html
> 
> Regards Jan
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Shared folder, COPY, OK, message nowhere to be seen

[Bron Gondwana via Info-cyrus]

It not only should be, I'm really sure it is.  I can't see a possible codepath 
that this could happen with - there's no such thing as "in core" here - 2.4.x 
doesn't even have the in-memory caching of cyrus.index changes.

This is from the cyrus-imapd-2.4.17 tag in git:

in imap/index.c: index_copy()

r = append_copy(mailbox, , copyargs.nummsg,
copyargs.copymsg, nolink);
if (!r) {
r = append_commit(, totalsize,
  , , , );
}
...
if (!r) {
/* we log the first name to get GUID-copy magic */
mailbox_close();
sync_log_mailbox_double(mailbox->name, name);
}

And then back in imapd.c: cmd_copy

   /* local mailbox -> local mailbox */
if (!r) {
r = index_copy(imapd_index, sequence, usinguid, mailboxname,
   , 
!config_getswitch(IMAPOPT_SINGLEINSTANCESTORE));
}

imapd_check(NULL, usinguid);

  done:
if (r && !(usinguid && r == IMAP_NO_NOSUCHMSG)) {
prot_printf(imapd_out, "%s NO %s%s\r\n", tag,
(r == IMAP_MAILBOX_NONEXISTENT &&
 mboxlist_createmailboxcheck(mailboxname, 0, 0,
 imapd_userisadmin,
 imapd_userid, imapd_authstate,
 NULL, NULL, 0) == 0)
? "[TRYCREATE] " : "", error_message(r));
}
else if (copyuid) {
prot_printf(imapd_out, "%s OK [COPYUID %s] %s\r\n", tag,
copyuid, error_message(IMAP_OK_COMPLETED));
free(copyuid);
}
else {
prot_printf(imapd_out, "%s OK %s\r\n", tag,
error_message(IMAP_OK_COMPLETED));
}



As you can see, we haven't send the response until we not only did an 
append_commit() and a mailbox_close(), we have also done an imapd_check().  
There's no way that we haven't committed everything before replying with OK.

So unless the RPMs have some funky patches that change this behaviour, there's 
no way the messages are not fully copied before returning OK.  Either it hadn't 
returned OK yet, or it would never have finished.

I believe you saw something weird, but I think you are wrong about the exact 
failure.  I don't suppose you can reproduce it?

Bron.


On Sat, Apr 23, 2016, at 01:21, Janne Peltonen wrote:
> Hi!
> 
> Both. The message cannot be seen via IMAP in the destination mailbox and it
> doesn't exist on the filesystem. 
> 
> But the message is clearly somewhere, probably in core: When I killed all the
> other imapd processes that could've been accessing the same mailbox (that had
> users who were mentioned in the acl of that mailbox), the messages I'd tried 
> to
> copy all both appeared on the file system (all at the same time, as confirmed
> by calling "stat" on the message files - the Change timestamps were all the
> same, and coincided with the time of the process kill) and could be fetched by
> imap, too.
> 
> This is a replicating setup, if that makes any difference.
> 
> Right now, everything seems to be working, and I haven't seen this behaviour
> earlier in the ~17 years I've been using Cyrus with shared folders in here - 
> or
> the 10 years I've been using replication - so I don't really know how to even
> recreate the problem.
> 
> The semantics of the OK response *should* be "Message is safely written on 
> disk
> (so you can delete the original message)", shouldn't it?
> 
> 
> --Janne
> 
> On Sat, Apr 23, 2016 at 12:56:15AM +1000, Bron Gondwana wrote:
> > When you say the messages can't be seen... Do you mean they aren't on the
> > file system, or that they aren't showing up via IMAP, or... ?
> > 
> > On Sat, Apr 23, 2016, at 00:48, Janne Peltonen wrote:
> > > And Centos 5.11, Linux 2.6.18-407.el5.
> > > 
> > > 
> > > --Janne
> > > 
> > > On Fri, Apr 22, 2016 at 05:43:11PM +0300, Janne Peltonen via Info-cyrus 
> > > wrote:
> > > > Hi!
> > > > 
> > > > Oh. Sorry. I thought I'd forgot something important, must be the fact 
> > > > it's
> > > > Friday night.
> > > > 
> > > > 2.4.17 from Simon Matter's srpm. Revision 6 of said rpm.
> > > > 
> > > > 
> > > > --Janne
> > > > 
> > > > On Fri, Apr 22, 2016 at 11:27:40PM +1000, Bron Gondwana via Info-cyrus 
> > > > wrote:
> > > > > Version?
> > > > > 
> > > > > On Fri, Apr 22, 2016, at 21:47, Janne Peltonen via Info-cyrus wrote:
> > > > > > Hi!
> > > > > > 
> > &g

Re: Shared folder, COPY, OK, message nowhere to be seen

[Bron Gondwana via Info-cyrus]

When you say the messages can't be seen... Do you mean they aren't on the file 
system, or that they aren't showing up via IMAP, or... ?

On Sat, Apr 23, 2016, at 00:48, Janne Peltonen wrote:
> And Centos 5.11, Linux 2.6.18-407.el5.
> 
> 
> --Janne
> 
> On Fri, Apr 22, 2016 at 05:43:11PM +0300, Janne Peltonen via Info-cyrus wrote:
> > Hi!
> > 
> > Oh. Sorry. I thought I'd forgot something important, must be the fact it's
> > Friday night.
> > 
> > 2.4.17 from Simon Matter's srpm. Revision 6 of said rpm.
> > 
> > 
> > --Janne
> > 
> > On Fri, Apr 22, 2016 at 11:27:40PM +1000, Bron Gondwana via Info-cyrus 
> > wrote:
> > > Version?
> > > 
> > > On Fri, Apr 22, 2016, at 21:47, Janne Peltonen via Info-cyrus wrote:
> > > > Hi!
> > > > 
> > > > I've always thought that IMAP COPY should only say OK once the message 
> > > > file is
> > > > safely on the disk. So that it should block, should another imapd 
> > > > process be
> > > > holding a write lock to the folder.
> > > > 
> > > > Now today I experienced something very weird. There was a shared 
> > > > folder. If I
> > > > tried to copy a message to it (speaking raw imap), I immediately got an 
> > > > OK. And
> > > > the folder metadata reflected the change. However, the message wasn't 
> > > > on the
> > > > disk. And wasn't retrievable using IMAP.
> > > > 
> > > > I tried this multiple times. Frustrated, I ended up killing the active 
> > > > imapd
> > > > processes of all other users that had ACL rights to the folder. My 
> > > > messages
> > > > immediately appeared on disk and were retrievable by IMAP. As if they'd 
> > > > been
> > > > in-core of my imapd process and waiting for a chance to be flushed to 
> > > > disk.
> > > > Even if the imapd process had returned OK, which I'd thought would've 
> > > > meant
> > > > they'd already been written to disk.
> > > > 
> > > > Now, apparently, one of the users had copied mission critical messages 
> > > > to said
> > > > folder and they had disappeared. Disappeared from the original folder 
> > > > and not
> > > > shown up in the destination folder. Which was the reason I started
> > > > investigating this.
> > > > 
> > > > Now, after having killed the imapd processes, I seem to have permanently
> > > > deleted all the messages that might have been in-core of any of those 
> > > > processes
> > > > (that had told their clients that the messages would've been on disk, 
> > > > i.e.
> > > > COPY -> OK). I killed the imapd processes while thinking that 1) there 
> > > > could in
> > > > no way be the slightest possibility that the messages hadn't been 
> > > > flushed to
> > > > disk from the core of the imapd processes after COPY, as the imapd had 
> > > > answered
> > > > OK to the client's COPY command, SO 2) one of the other users' clients 
> > > > must
> > > > have had a rule that immediately moves the messages away. But 
> > > > apparently, this
> > > > wasn't the case.
> > > > 
> > > > Has anyone else encountered something like this?
> > > > 
> > > > 
> > > > --Janne Peltonen
> > > > 
> > > > Cyrus Home Page: http://www.cyrusimap.org/
> > > > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > > > To Unsubscribe:
> > > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > > 
> > > 
> > > -- 
> > >   Bron Gondwana
> > >   br...@fastmail.fm
> > > 
> > > Cyrus Home Page: http://www.cyrusimap.org/
> > > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > > To Unsubscribe:
> > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Shared folder, COPY, OK, message nowhere to be seen

[Bron Gondwana via Info-cyrus]

Version?

On Fri, Apr 22, 2016, at 21:47, Janne Peltonen via Info-cyrus wrote:
> Hi!
> 
> I've always thought that IMAP COPY should only say OK once the message file is
> safely on the disk. So that it should block, should another imapd process be
> holding a write lock to the folder.
> 
> Now today I experienced something very weird. There was a shared folder. If I
> tried to copy a message to it (speaking raw imap), I immediately got an OK. 
> And
> the folder metadata reflected the change. However, the message wasn't on the
> disk. And wasn't retrievable using IMAP.
> 
> I tried this multiple times. Frustrated, I ended up killing the active imapd
> processes of all other users that had ACL rights to the folder. My messages
> immediately appeared on disk and were retrievable by IMAP. As if they'd been
> in-core of my imapd process and waiting for a chance to be flushed to disk.
> Even if the imapd process had returned OK, which I'd thought would've meant
> they'd already been written to disk.
> 
> Now, apparently, one of the users had copied mission critical messages to said
> folder and they had disappeared. Disappeared from the original folder and not
> shown up in the destination folder. Which was the reason I started
> investigating this.
> 
> Now, after having killed the imapd processes, I seem to have permanently
> deleted all the messages that might have been in-core of any of those 
> processes
> (that had told their clients that the messages would've been on disk, i.e.
> COPY -> OK). I killed the imapd processes while thinking that 1) there could 
> in
> no way be the slightest possibility that the messages hadn't been flushed to
> disk from the core of the imapd processes after COPY, as the imapd had 
> answered
> OK to the client's COPY command, SO 2) one of the other users' clients must
> have had a rule that immediately moves the messages away. But apparently, this
> wasn't the case.
> 
> Has anyone else encountered something like this?
> 
> 
> --Janne Peltonen
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mailboxes.db invalid entries

[Bron Gondwana via Info-cyrus]

Ouch, yeah - spaces in group names?  That's gotta be bogus.   I'm surprised we 
accept those in the first place.

I guess the code should probably check that keys are valid identifiers though 
:(  So that is a legitimate bug.

Bron.

On Fri, Apr 22, 2016, at 09:28, Jan Kowalsky via Info-cyrus wrote:
> Hi all,
> 
> I have an unresolvable problem with one mailbox:
> 
> It got an entry inside the mailboxes.db and a spool directory - but I
> can't change it or access it. I always got "invalid identifier" on cyradm.
> 
> Dumping the mailbox with ctl_mboxlist
> 
> example.com!user.kontakt.Calendar 0 (null) (null)
> 
> Converted the mailbox with cvt_cyrusdb /srv/imap/default/mailboxes.db
> skiplist /tmp/mailboxes.txt flat
> 
> I saw this invalid entry in group entry -> there is a white space
> between group: and group name:
> 
> example.com!user.kontakt.Calendar %(A %(kont...@example.com
> lrswipkxtecdan group: teamgroup lrs) I 0ba6aced567131fb P default V
> 1456854556 M 1461271748)
> 
> In the past I had once a similar problem. The reasons where also invalid
> acl entries - (because of buggy ptloader - and the missing prove if the
> aci subject even exists. Anyway...)
> 
> I tried to ways:
> 
> First I tried to dump the mailbox.db with ctl_mboxlist -d /tmp/mailboxes.txt
> 
> After deleting the wrong entry manually I wanted to reload the mailbox
> again with ctl_mboxlist -u /tmp/mailboxes.txt. All operation with
> stopped cyrus.
> 
> This gave me the error:
> 
> fatal error: failed to mmap /srv/imap/config/mailboxes.db file
> 
> As a second way I tried to convert mailbox with the cvt_cyrusdb to flat
> format - edit the wrong entry and convert it back (with cyrus stopped).
> 
> No cyrus didn't start again:
> 
> In the logs:
> 
> Apr 22 00:53:35 mail master[19881]: setrlimit: Unable to set file
> descriptors limit to -1: Operation not permitted
> Apr 22 00:53:35 mail master[19881]: retrying with 4096 (current max)
> Apr 22 00:53:35 mail ctl_cyrusdb[19882]: skiplist: clean shutdown file
> missing, updating recovery stamp
> Apr 22 00:53:35 mail ctl_cyrusdb[19882]: recovering cyrus databases
> Apr 22 00:53:35 mail ctl_cyrusdb[19882]: converting
> /srv/imap/config/mailboxes.db from skiplist to twoskip
> Apr 22 00:53:35 mail ctl_cyrusdb[19882]: skiplist: recovered
> /srv/imap/config/mailboxes.db (15000 records, 1962048 bytes) in 0 seconds
> Apr 22 00:53:35 mail ctl_cyrusdb[19882]: skiplist: checkpointed
> /srv/imap/config/mailboxes.db (15000 records, 1962048 bytes) in 0.143 sec
> Apr 22 00:53:39 mail ctl_cyrusdb[19882]: IOERROR: mapping
> /srv/imap/config/mailboxes.db.NEW.NEW file: Cannot allocate memory
> Apr 22 00:53:39 mail master[19881]: process type:START name:recover
> path:/usr/lib/cyrus-imapd/ctl_cyrusdb age:0.000s pid:19882 exited, status 75
> Apr 22 00:53:39 mail master[19881]: can't run startup
> Apr 22 00:53:39 mail master[19881]: exiting
> 
> As the cyrus database troubleshooting isn't documented very well, any
> help is very appreciated.
> 
> What is the right way to correct database entries?
> And one more question related: How to delete entries from database where
> no spool directory exists for the mailbox?
> 
> Thanks and regards
> Jan
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: ACL inheritance in Shared Namespace

[Bron Gondwana via Info-cyrus]

Ack. It still works like this

On Thu, Apr 14, 2016, at 03:51, Ken Murchison via Info-cyrus wrote:
> Unless someone other than me changed the code, ACL inheritance only 
> applies to mailbox creation.  Once a mailbox exists, its ACL is 
> independent of all others.
> 
> 
> On 04/13/2016 12:27 PM, Chris via Info-cyrus wrote:
> > All,
> >
> > is ACL inheritance possible in shared namespace?
> >
> > If I revoke access for someone in folderA/, does this also apply to
> > folderA/subfolderA1?
> >
> > Thank you in advance.
> >
> > - Chris
> >
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> -- 
> Kenneth Murchison
> Principal Systems Software Engineer
> Carnegie Mellon University
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: IPv6

[Bron Gondwana via Info-cyrus]

Ellie, can you please fix the listen statement to accept correctly bracketed 
ipv6 and backport to at least 2.5 and 2.4, shouldn't be many changes in that 
code.

Nicola, can you check that we document how it's working now, and of course the 
fixed version :)

Bron (on phone in bush somewhere)

On Thu, Mar 24, 2016, at 05:10, Hajimu UMEMOTO via Info-cyrus wrote:
> Hi,
> 
> > On Wed, 23 Mar 2016 17:43:30 +0100
> > Sebastian Hagedorn  said:
> 
> >  * Parse the "listen" parameter as one of the forms:
> >  *
> >  * hostname
> >  * hostname ':' port
> >  * ipv4-address
> >  * ipv4-address ':' port
> >  * '[' ipv4-address ']'
> >  * '[' ipv4-address ']' ':' port
> >  * '[' ipv6-address ']'
> >  * '[' ipv6-address ']' ':' port
> 
> The `ipv6-address ':' port' notation is also accepted.  However, it is
> not mentioned even in the comment.
> 
> Hagedorn> Thank you! Perhaps that should go into the manpage of cyrus.conf?
> 
> I'm not sure but I think the bracket notation is recommended.
> 
> Sincerely,
> 
> --
> Hajimu UMEMOTO
> u...@mahoroba.org  u...@freebsd.org
> http://www.mahoroba.org/~ume/
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Sieve for shared mailboxes

[Bron Gondwana via Info-cyrus]

We don't use shared mailboxes or non-user sieve at all at FastMail, so
it gets less testing than most things. I'd love to have contributions of
good test cases to Cassandane from someone who knows exactly how this
should work so we can make sieve good for the shared case too.
 
It's somewhat tricky now that FastMail is sponsoring the bulk of Cyrus
work, obviously the stuff we use gets the most love!
 
Cheers,
 
Bron.
 
 
On Fri, Mar 18, 2016, at 23:30, Alvin Starr via Info-cyrus wrote:
> I have had mixed luck with direct mailbox delivery(+ addressing) and
> when I say mixed I mean mostly bad.
>  I wanted to try using it for direct delivery of copies of all sent
>  messages.
>  I found I needed special permissions on all the mailboxes and
>  duplicate delivery detection would not work in that configuration.
>  So I dropped it.
>
>  My guess would be that you may need to setup a shared mailbox or
>  group mail account that would process the mail into the shared
>  mailbox and then you could use sieve.
>
>
>
>
> On 03/18/2016 07:35 AM, Merlin Hartley wrote:
>> Sorry I wasn’t clear: we are using an alias with +plus addressing and
>> lmtp to deliver into the shared mailbox
>> I have the shared mailbox working just fine… it is only the sieve on
>> shared mailboxes that is failing...
>>
>>
>> aliases file:
>> mbujobs: +shared/mbujobs
>>
>> exim router:
>> imap_shared_accept:
>> driver = accept
>> local_parts = ^\\+[^/]+/.+
>> transport = local_delivery_cyrus
>> cannot_route_message = Unknown  user
>>
>> exit transport:
>> local_delivery_cyrus:
>> driver = lmtp
>> socket = /var/lib/imap/socket/lmtp
>> batch_max = 20
>> user = cyrus
>> group = mail
>> transport_filter = /usr/bin/tr -d  \\000
>> delivery_date_add
>> envelope_to_add
>> return_path_add
>>
>> --
>>  Merlin Hartley
>>  IT Systems Engineer
>>  MRC Mitochondrial Biology Unit
>> Cambridge, CB2 0XY
>> United Kingdom
>>
>>> On 18 Mar 2016, at 11:29, Alvin Starr via Info-cyrus >> cy...@lists.andrew.cmu.edu> wrote:
>>>
>>> Sieve is run as part of the mail delivery into the mailbox.
>>>  That would either be deliver or the lmtp interface.
>>>
>>>  If your users are dragging and dropping the messages into the
>>>  shared mailboxes then I don't believe that you can have sieve run
>>>  on the mbox to mbox transfer.
>>>
>>>
>>>
>>> On 03/18/2016 06:48 AM, Merlin Hartley via Info-cyrus wrote:
 Greetings

 I have been managing our mail domain on a cyrus-imap () server
 since 2005 and it has always been great for our ~150 users.

 Of course, over-time more complexity is always required and I have
 recently implemented a few shared mailboxes (rather than just
 sharing user mailboxes).
 Inevitably, the users are now asking for an auto-reply to be
 configured for some of these shared mailboxes…

 We are already using sieve scripts (managed with Roundcubemail
 talking through the firewall to timsieved) so it seems natural to
 use this technology here too...

 I have followed the instructions on this page:
 https://cyrusimap.org/imap/admin/sieve.html?highlight=sieve#managing-sieve-scripts

 But the last step doesn’t seem to do anything…

 So I have a few related questions:

 1) how can I query a mailbox to read the flags set by mboxconfig?
 2) has anyone got sieve working with shared mailboxes?
 3) is it possible to invoke a sieveshell in the context of a shared
mailbox?

 I seem to have successfully created the global scripts (a ‘global’
 folder has appeared in the sievedir) - just can’t seem to attach it
 to a shared mailbox.

 Many thanks!


 Merlin


 P.S. Here is some relevant server info:

 [root@mercury merlin]# sieveshell -u cyrus -a cyrus localhost
 connecting to localhost
 Please enter your password:
 > list
 mbu_jobs  <- active script

 [root@mercury ~]# ls /var/lib/imap/sieve/global/
 *defaultbc*  mbu_jobs.bc  mu_jobs.script

 [root@mercury ~]# cyrus-admin
 verify error:num=19:self signed certificate in certificate chain
 localhost> mboxconfig shared/mbujobs sieve mbu_jobs
 localhost>

 [root@mercury ~]# yum list cyrus-imapd
 Installed Packages
 cyrus-imapd.x86_64                                     2.5.0-4.9
 @cyrus-imapd_2.5.x
 (which is a Kolab repository I installed onto my CentOS 6 server)

 [root@mercury ~]# uname -a
 Linux mercury.mrc-mbu.cam.ac.uk[1] 2.6.32-573.18.1.el6.x86_64 #1
 SMP Tue Feb 9 22:46:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

 --
  Merlin Hartley
  IT Systems Engineer
  MRC Mitochondrial Biology Unit
 Cambridge, CB2 0XY
 United Kingdom



  Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info:
 http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>
>>> 

Re: Replication CRC error

[Bron Gondwana via Info-cyrus]

Master has:

commit 6ef874319ecd98700e682ef30fcad5245ddfdb32 Author: Bron Gondwana
 Date:   Wed Oct 22 14:51:24 2014 -0400

sync_client: do ALL mailboxes, not just all users, for -A flag


It should be pretty cherry-pickable back to 2.5 I would imagine.

Bron.

On Thu, Mar 3, 2016, at 09:17, Artyom Aleksandrov via Info-cyrus wrote:
> Hi,
>
> When using the “-A” flag (sync all users) no non-user mailboxes are
> synced. As the man page imapd.conf(5) notes, ”... this could be
> considered a bug and maybe it should do those mailboxes
> independently.” Source:
> https://cyrusimap.org/imap/admin/sop/replication.html
>
> But how in this case restore replication for shared folders? Run sync
> for each folder separately?
>
> On Wed, Mar 2, 2016, 6:01 PM Konstantin Udalov via Info-cyrus  cy...@lists.andrew.cmu.edu> wrote:
>> Hello.
>>
>>
I configured IMAP rolling replication from master to slave as was
>>
described in https://cyrusimap.org/imap/admin/sop/replication.html And
>>
everything was fine. Until some strange records appear into logs. Here
>>
is one of them.
>>
>>
Mar  2 04:48:55 hostname cyrus/sync_client[725]: MAILBOX received NO
>>
response: IMAP_SYNC_CHECKSUM Checksum Failure
>>
Mar  2 04:48:55 hostname cyrus/sync_client[725]: CRC failure on sync for
>> shared.folder.name, trying full update
>>
Mar  2 04:49:08 hostname cyrus/sync_client[725]: SYNCERROR: only exists
>>
on master shared.folder.name 286227
>>
(d41e384300c3f13f42a546ce14ec3d23bc89f1a6)
>>
Mar  2 04:49:13 hostname cyrus/sync_client[725]: SYNCERROR: only exists
>>
on master shared.folder.name 286227
>>
(d41e384300c3f13f42a546ce14ec3d23bc89f1a6)
>>
Mar  2 04:49:13 hostname cyrus/sync_client[725]: mailbox: longlock
>> shared.folder.name for 11.4 seconds
>>
>>
I stopped rolling replication and launched full (-A) replication. No
>>
more related records in logs, still message 286227 wasn't replicated and
>>
is missing on replica.
>>
If somebody faced same problem, help would be much appreciated.
>>
>>
cyrus-imap 2.5.3
>>

>>
Cyrus Home Page: http://www.cyrusimap.org/
>>
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>
To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info:
> http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
  Bron Gondwana
  br...@fastmail.fm
 
 

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

cyrusimap.org website updated

[Bron Gondwana via Info-cyrus]

Hi Everyone,

Thanks to the awesome work by Nic, Nicola and others, the cyrusimap.org website 
is now entirely generated from Sphinx (not the same sphinx as the search engine 
- yay for namespaces) from the git repository which is currently at:

https://git.cyrus.foundation/diffusion/D/

Feel free to clone that and suggest patches :)

I have kept the old directory structure intact, so links into 
http://cyrusimap.org/docs/cyrus-imapd/ and friends that are linked from Google 
will continue to work.  Likewise with /mediawiki/

I for one welcome our autogenerated overlords.

Cheers,

Bron.


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: determine whan the mailbox was last accessed

[Bron Gondwana via Info-cyrus]

You'd be looking for the "recenttime" field in the mailbox.  Interestingly, it
appears you can get it via the 'fud' daemon, but it's not exposed via an
annotation.

If you create a bug in bugzilla (or phabricator) to remind us... it's probably
not too hard to create an annotation that you can read via cyradm - the
data is there.

Bron.

On Mon, Dec 28, 2015, at 20:11, Eugene M. Zheganin via Info-cyrus wrote:
> Hi.
> 
> Some time ago I used the seen per-user databases to determine when the
> mailbox was last accessed (and it was easy - it was the time of seen
> database modification - for IMAP users, of course). Now the seen
> database has migrated into one file for all users, and I'm not sure how
> to do this. I will really appreciate if someone will describe how to do
> this.
> 
> Thanks.
> Eugene.
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Archiving old email?

[Bron Gondwana via Info-cyrus]

On Mon, Dec 28, 2015, at 02:28, Patrick Goetz via Info-cyrus wrote:
> On 12/26/2015 04:44 PM, Bron Gondwana via Info-cyrus wrote:
> >
> > Each mailbox can have its storage split between two drives - one that's
> > high speed with recent mail, and slower speed drives for bulk storage.
> >
> > At FastMail, we store the current week's email on SSDs in RAID1, and the
> > remainder (almost all of it!) on big SATA drives in RAID6.
> >
> 
> 
> I guess the real problem for me is that applications like Thunderbird 
> like to cache the mail folders locally in order to improve performance, 
> so it sounds like the cyrus archive feature is not going to help with 
> that particular problem.

No, probably not.  You'd be looking more for automatically moving messages to
different folders after a period of time.

Bron.
-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Archiving old email?

[Bron Gondwana via Info-cyrus]

On Sun, Dec 27, 2015, at 04:22, Patrick Goetz via Info-cyrus wrote:
> I have some users whose mail folders are approaching 2GB in size. 
> Unfortunately they can't delete any of this mail, but that doesn't mean 
> that it needs to be lugged around, indexed, and copied to local mail 
> caches as part of a live mail system.  I find that the probability of 
> needing to look at mail more than 5 years old is roughly 0%/year.
> 
> I was going to ask if anyone had any ideas about archiving old mail, but 
> then noticed that cyrus 3.0 is going to support some kind of archiving 
> system:
> 
>Archive support has arrived! Requires addition of an archive 
> partition. (See archive_* options in imapd.conf)
> 
> Can anyone elaborate on what this actually means?

Each mailbox can have its storage split between two drives - one that's
high speed with recent mail, and slower speed drives for bulk storage.

At FastMail, we store the current week's email on SSDs in RAID1, and the
remainder (almost all of it!) on big SATA drives in RAID6.

There are potential plans to switch to using an object store rather than
the RAID6, if we can make it stable enough - we can reduce the number
of copies required without reducing the safety/redundancy - which is
always nice for managing costs.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Ongoing inaccessible mailbox woes

[Bron Gondwana via Info-cyrus]

On Sun, Dec 27, 2015, at 01:14, Patrick Goetz via Info-cyrus wrote:
> Hi Bron -
> 
>[root@www cyrus]# pwd
>/etc/cyrus
>[root@www cyrus]# grep improved_mboxlist_sort imapd.conf
>improved_mboxlist_sort: yes
> 
> But in any case, it is seeing the mailbox.  The "Archives Staff" mailbox 
> shows up with messages and one sub-mailbox -- the other sub-folders are 
> all missing.
> 
> Question:  when I'm running reconstruct, what does it even mean to get 
> the message "Mailbox already exists"?  Of course the mailbox already 
> exists; else there would be no point in running reconstruct!
> 
> Next, when I look in
> 
>/var/imap/user/d/djones/djones.sub
> 
> the folders I'm not seeing and which give the error messages are missing 
> from this list -- could this be the problem?  How can I force cyrus to 
> reconstruct this list, or is this one of the things that reconstruct does?

The subs file isn't significant at all.

create_namecheck returns MAILBOX_EXISTS, as does a rename to the same
name or on the same partition.  It's obviously create_namecheck that will be
triggering this.

I'd be very interested in seeing the exact files that are causing this.  Is it 
possible
for me to get a login to the system where this is running?  Happy to sign any
non-disclosure you need.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Ongoing inaccessible mailbox woes

[Bron Gondwana via Info-cyrus]

It smells like maybe you have improved_mboxlist_sort set to the wrong value 
(off),
and hence it's unable to see the mailbox at all!  This is really bogus of the 
way that
mailboxes are stored, and I have master plans of fixing it better... *sigh*

Bron.

On Sat, Dec 26, 2015, at 02:47, Patrick Goetz via Info-cyrus wrote:
> When I upgraded from 2.4.17 to 2.5.3, I had one user who experienced 
> inaccessible mail folders post upgrade.  Running reconstruct:
> 
> # systemctl stop cyrus-master
> # su - cyrus
> # /usr/lib/cyrus/bin/reconstruct -r -f user/djones
> # -d
> # systemctl start cyrus-master
> 
> temporarily resolved the problem, but then it came back, so I thought 
> this might be a 2.5.3 bug.  I just upgraded to 2.5.7, and the problem 
> persists.  Furthermore, running reconstruct no longer brings the mail 
> messages back and I get the following error messages during the 
> reconstruction on precisely the folders which are not accessible:
> 
> 
> user/djones/World Mission Conference
> user/djones/World Mission Conference/2008
> user/djones/World Mission Conference/2011
> createmailbox user.djones.Archives Staff.Keely Davis: Mailbox already exists
> createmailbox user.djones.Archives Staff.Lauren Kelsey: Mailbox already 
> exists
> createmailbox user.djones.Archives Staff.Stacey Williams: Mailbox 
> already exists
> createmailbox user.djones.Archives Staff.Drew Smith: Mailbox already exists
> createmailbox user.djones.Archives Staff.Whitney: Mailbox already exists
> createmailbox user.djones.Archives Staff.Delia Alexander: Mailbox 
> already exists
> createmailbox user.djones.Archives Staff.Whitney Howell: Mailbox already 
> exists
> createmailbox user.djones.Archives Staff.Former Staff: Mailbox already 
> exists
> createmailbox user.djones.Archives Staff.Julia Miller: Mailbox already 
> exists
> createmailbox user.djones.Archives Staff.Weekly Reports: Mailbox already 
> exists
> createmailbox user.djones.Archives Staff.Eleanor Meuller: Mailbox 
> already exists
> createmailbox user.djones.Archives Staff.Corrine Smith.Inquiries: 
> Mailbox already exists
> createmailbox user.djones.Archives Staff.Corrine Smith.Destruction: 
> Mailbox already exists
> createmailbox user.djones.Archives Staff.Corrine Smith.SharePoint: 
> Mailbox already exists
> 
> 
> The folders and files in the filesystem look completely ordinary and 
> have the correct permissions.  However, under user/djones, this user has 
> the following mail folders:
> 
> drwx--  18 cyrus mail   98304 Dec  7 11:11 Archives Staff
> drwx--   5 cyrus mail4096 Oct  3 05:21 Archives Staff Retreat
> 
> 
> Now I'm starting to wonder if there isn't some kind of folder name 
> length bug at work here (maybe just in reconstruct?) so that the mail 
> folders
> 
>Archives Staff
>Archives Staff Retreat
> 
> are being seen by reconstruct as the same folder?
> 
> I'm kind of grasping at straws here, unfortunately.  Last time this 
> happened, Bron suggested that I needed to set up logging with debugging 
> turned on.  If this is the only way to get to the bottom of this, I 
> guess I'll have to do that.  (Since the mail server is a systemd system, 
> traditional syslog logging is not turned on by defaul and will require 
> some effort to enable.)
> 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Retrospective on this year in the Cyrus Project

[Bron Gondwana via Info-cyrus]

For those of you not following FastMail's Advent calendar (shame on you, surely 
you don't have anything better to do), I posted this today:

https://blog.fastmail.com/2015/12/22/the-year-of-the-cyrus-foundation/

For all that we haven't done another beta release for ages (mainly because 
putting the changelog together is SO MUCH WORK because we're changing so much 
right now)

The project feels a little dead in this list, but there's actually heaps going 
on.  Robert S's JMAP changes are really exciting and recently landed on master. 
 Ken's added a dropbox style WebDAV storage backend, and is working on the 
latest changes coming out of CalConnect (Neil and I will also be attending the 
CalConnect conference in January.

I've also posted about another change recently:

https://blog.fastmail.com/2015/12/05/reverse-acls-making-imap-list-fast/

This deprecates foolstupidclients, disable_{user|shared}_namespace, 
imapmagicplus (kinda).  Basically all the hacks to make LIST fast on big 
servers are gone, because LIST doesn't need to scan all those mailboxes any 
more.

(though note: it doesn't have group ACL support yet)

And FastMail has tons of development plans.  As I say in the blog post - if you 
want a say in the direction of the project, then get involved - post on the 
list, join the #cyrus channel, even come to our meetings.  We're focusing on 
the features that FastMail want (and CMU with Ken's work) because we're the 
ones paying for the developer time, but we're also building for everyone, so 
your needs won't be ignored if we know what they are...

Merry Christmas!

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Weird Characters in unexpunge

[Bron Gondwana via Info-cyrus]

It's dumping the raw literal data from the file.  I have no idea why we're 
storing a literal in there, given that it's already a length-delimited record.  
We've seen the same bug at FastMail in our search system for a while until I 
patched it to parse a literal out of the cache.  It would be easy to patch 
unexpunge the same way (or abstract the whole thing behind the message API, 
which is probably a better idea)

Bron.

On Wed, Dec 16, 2015, at 23:20, Stephan via Info-cyrus wrote:
> 
> Hello all,
>  
> unexpunge (cyrus 2.5.7) is showing weird behaviour when it comes to subject 
> lines containing the double quotation mark ("). When running reconstruct -l 
> the subject lines containg a '"' show up as
>  
> {}Subject, for instance
> 
>     Subj: {109}
> Cron 
>  
> I redirected the output to a file and opened it with vim, which shows a ^M 
> before end of the line, which is a carriage return if I'm not mistaken. So I 
> guess there is a  involved.
> 
> The mail as such looks perfectly fine. Is this a feature or should I open a 
> bug report ?
>  
> Thanks,
> Stephan
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus-imapd-2.4 branch, segfault in lmtpd

[Bron Gondwana via Info-cyrus]

Are you able to get a backtrace on the segfaulting processes?

Thanks,

Bron.

On Wed, Dec 16, 2015, at 04:00, Sergey via Info-cyrus wrote:
> Hello.
> 
> I attempted to use last cyrus-imapd-2.4 branch and got error:
> 
> kernel: [8211701.360899] lmtpd[20104]: segfault at 1 ip b717f3fc sp bfd1d8e0 
> error 4 in libc-2.17.so[b70f9000+172000]
> kernel: [8211713.497625] lmtpd[20109]: segfault at 1 ip b718b3fc sp bff849f0 
> error 4 in libc-2.17.so[b7105000+172000]
> 
> There is no error when commit b2dab34ccd80e7a0753bce52bc07ed18663a484f
> is reverted.
> 
> -- 
> Regards, Sergey
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

[POLL] Cyrus ACLs and group names

[Bron Gondwana via Info-cyrus]

For those of you using Cyrus with group ACLs, how are your groups named?

I know with the auth_unix backend, they are 'group:'.  What I've 
seen from CMU's groups is that they are of the form ':'.

What I want to know is - do group names always contain a colon?  If so it will 
make implementing reverse ACLs for groups a lot easier.

Thanks,

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus