Re: Replication and Deleted Files

[Michael Menge]

Hi,

Quoting Ian Batten via Info-cyrus :


Hi, long-time Cyrus user (25 years, I think), but stumped on this one…

I have an ancient Cyrus 2.5.11 on Solaris 11 installation I am  
trying to migrate off.  The strategy is to run rolling replication  
onto the new server (3.0.8-6+deb10u4 on Debian 10.4), and then point  
the DNS record at the new server.  With Covid, this has become more  
protracted than I would like, as I don’t want to accidentally mess  
up users who are isolating, so the replication has been running for  
some weeks.


The replication structure is old-server -> new-server -> (backup1,  
backup2) where backup1 and backup2 are configured as separate  
channels on new-server.  This has been running seemingly correctly  
for about three months now.


Today I decided to check all was well by using rsync -an to confirm  
that the replicas have everything that is on the master.  They do,  
in that using 


rsync -anvO --size-only  --exclude='cyrus.*'  
root@mail:/var/imap/partition1/user/ /var/imap/partition1/user 


where “mail” is the old server shows that there are no messages  
missing (—size-only because there’s some time slew in a few places,  
usually only of a few seconds, but up to a day in others).


However, reversing it:

rsync -anvO --size-only  --exclude='cyrus.*'  
/var/imap/partition1/user/ root@mail:/var/imap/partition1/user


Shows that there are a _lot_ of files on the replicas which are not  
on the master, some of them relating to recent deletions, but some  
of them seemingly quite old.  I am using:


delete_mode: delayed
expunge_mode: delayed

everywhere, running cyr_expire on the master but not on the  
replicas.  I have enough bandwidth that sync_reset and re-sync is  
realistic, but I’d rather not have to do that immediately prior to a  
cut-over.   These old files are a worry because if I ever had to  
reconstruct one of the mailboxes, presumably the deleted (I think)  
messages would all reappear.  Does anyone have any suggestions?




you also need to run cyr_expire on the "new_server" to remove the old
expunged mails and deleted folders.

I haven't used replication for backup but I suspect that cyr_expire is
also required your backup servers






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Is it safe to wipe: /var/lib/cyrus/lock/

[Michael Menge]

Hi,

Quoting Jan Vales :


Hello,

I just stumbled into this directory: /var/lib/cyrus/lock/

It seems to contain many dead things, like .lock-files that - based on
their names - must be older than 2+ years, including .../user/$NAME
where $NAME does not have a mailbox/account anymore.

Is it safe to shut down cyrus and do a
find /var/lib/cyrus/lock/ -mindepth 2 -delete

Its currently cyrus 3.0.13 on debian 10.



/var/lib/cyrus/lock/ and /var/lib/cyrus/proc should be empty after  
shutting down cyrus


You can also use tempfs for these directories,
- it is faster
- reboot will clean up the left over files in case the
  shut down was not clean
- these directories only contain small files





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.0 released

[Michael Menge]

Quoting ellie timoney :


Hi Marco,


But it really seems that the LD_PRELOAD or the syslog.so doesn't
work for me.


Thanks for including that detail.  Hopefully someone familiar with  
RedHat can chime in with some insight into the LD_PRELOAD issue!  At  
a guess, "preventing library injection from intercepting syslog  
calls" might be a security default of some sort.




RedHat systems use SELinux by default and SELinux has the habit to
block access to files, sockets, ... Especial if software from
non-Redhat repos is used.

SELinux logs to /var/log/audit/audit.log, but these logs are not easy
to understand, therefor I often use audit2allow to see which SELinux
permissions are missing.

# cat /var/log/audit/audit.log | audit2allow

You can change SELinux temporary to permissive mode so that the log will
indicate all missing permissions and not only the first few.

# setenforce 0


Michael Menge


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13

[Michael Menge]

Quoting Olaf Frączyk :



Yes, at the beginning I was also thinking if initial sync is  
necessary, but there was nothing in docs about it, something started  
replicating and I simply assumed it does initial resync. I'll try it  
this evening. :)


Since you use replication - are sieve scripts replicated as well?  
There is -s option called sieve mode but it needs to specify which  
users' files are to replicate and there is written that it is mostly  
for debugging.




Yes, sieve scripts are replicated.

The way the rolling replication works is, that every time something is changed
on the master a "hint" is written in the sync log,

"MAILBOX user.foo.bar" indicates that the mailbox bar of the user foo  
has changed

and the sync_client will sync this (and only this folder)
There are other "hints" e.g for changed subscription or changed sieve script.

But if the  sieve script is not changed sync_client in rolling replication
will not try to sync it. Using the -A or -u Option will sync the all/some
users, including all mailboxes, folder subscriptions and sieve scripts.

The -s option is only needed if you change a compiled sieve script so
that it is not logged in the replication log.





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13

[Michael Menge]

Hi,

Quoting Olaf Frączyk :

I managed to get strace on both sides, however it doesn't make me  
wiser - there is nothing obvious for me.


Additionally I see that replication works more or less for new  
messages, but older are not processed.


I have several subfolders in my mailbox, some of them unreplicated.  
If I change anything in the subfolder now - the folder is  
replicated, but other subfolders remain not replicated - unless I  
change anything in them.


you are trying to use rolling replication.
For rolling replication cyrus logs which the location and type where  
changes ocure
e.g "MAILBOX navi.pl!user.olaf" indicates that something has changed  
in the INBOX of

the user o...@navi.pl.

the rolling replication will keep the master and replica in sync, but  
it requires

that there was an initial replication of all users.

you can use sync_client with -A oder -u to do this synchronization  
(see the manpage for details)


I think using rolling replication without the initial sync may be the  
cause of the errors.

stop the "sync_client -r" and wait for the initial sync to finish.




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13

[Michael Menge]

Quoting Olaf Frączyk :


Thank you for the telemetry hint :)

I don't use the syncserver - the replication is done via IMAP port  
on the replica side. I have no idea how to have strace spawned by  
cyrus master process. When I attach later to imapd using strace -p  
I'm afraid some info already will be lost.




You can try "/usr/bin/strace  /imapd  
" as cmd

or you can use prefork=1 in the service to prefok one imap process and connect
to this one before you start the sync_client.



The syncserver is marked as deprecated in the docs, so I went with  
the more modern option. Maybe here is the problem ;)




i am using syncserver since cyrus-imapd 2.3. So I had no reason to change it.


The funny thing is that from time to time the replication progresses  
a little. I don't like non-repetitive behavior ;)


Thanks again for the hints.

On 2020-04-21 14:13, Michael Menge wrote:

Hi,

Quoting Olaf Frączyk :


The current situation is:

1. Replica:

stopped and started the replica

no activity on replica - iotop and top show nothing

the only messages on replica is incoming connection from master

2. Master:

when I run sync_client -r I still get:

Apr 21 12:38:36 ifs sync_client[29518]: Reprocessing sync log file  
/var/lib/imap/sync/log-run
Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length  
response to MAILBOXES (idle for too long)
Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length  
response to RESTART (idle for too long)
Apr 21 12:43:27 ifs sync_client[29518]: Error in do_sync():  
bailing out! Bad protocol
Apr 21 12:43:27 ifs sync_client[29518]: Processing sync log file  
/var/lib/imap/sync/log-run failed: Bad protocol


3. There is 27 GB of about 45 GB replicated and there is no  
further progress


4. How to find out why the replica doesn't respond?



You can enable telemetry logging on the replica by creating a  
folder /var/lib/imap/log/
where  is the value of the  sync_authname. If you  
give cyrus write permissions for this folder
it will create log-files for each process and what it received and  
send from/to the sync_client with timestamps.


Also you can try to use strace on the syncserver process to figure  
out which files it is accessing




On 2020-04-21 11:18, Olaf Frączyk wrote:

I also found out that when I see on master:

Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length  
response to MAILBOXES (idle for too long)
Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length  
response to RESTART (idle for too long)
Apr 21 11:12:38 ifs sync_client[27996]: Error in do_sync():  
bailing out! Bad protocol
Apr 21 11:12:38 ifs sync_client[27996]: Processing sync log file  
/var/lib/imap/sync/log-run failed: Bad protocol


I also get on the replica:

Apr 21 11:12:38 skink1 imap[5775]: Connection reset by peer,  
closing connection


But I also see, that before it happens, there is no activity both  
on the replica and on the master for some time.


So maybe the imap server process is not recovering correctly in  
the longlock situation?


On 2020-04-21 11:07, Olaf Frączyk wrote:

Hi,

When I run sync_client -r on the master I see the following on  
the replica:


Apr 21 10:56:15 skink1 imap[5775]: mailbox: longlock  
navi.pl!user.olaf for 1.7 seconds
Apr 21 10:56:20 skink1 imap[5775]: mailbox: longlock  
navi.pl!user.piotr for 2.0 seconds
Apr 21 10:56:23 skink1 imap[5775]: mailbox: longlock  
navi.pl!user.olaf for 2.9 seconds
Apr 21 10:56:26 skink1 imap[5775]: mailbox: longlock  
navi.pl!user.piotr for 3.0 seconds


The mailboxes have several GB in Inbox folder and several GB in  
subfolders. The inbox folders have about 20,000 messages, the  
subfolders upto 15,000


Could it cause problems?


the longlock is normaly not the problem. While on process has the lock
no other process can write to the mailbox, but on the replica there normaly
is no other process that should access the mailbox



Maybe I should move the sync_client from START section to  
SERVICES, it seems that it is not automatically restarted



I havend tried starting sync_client in the service section.
I start the sync_client via systemd.


On 2020-04-21 08:47, Michael Menge wrote:

Hi Olaf


Quoting Olaf Frączyk :


Hi,

I upgraded to 3.0.13 but it didn't help.

This time it copied about 18GB

in the logs I still see:

1 - inefficient replication

2 - IOERROR: zero length response to MAILBOXES (idle for too long)
IOERROR: zero length response to RESTART (idle for too long)
Error in do_sync(): bailing out! Bad protocol

But I have no idea what can I do next and why it fails

Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length  
response to MAILBOXES (idle for too long)
Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length  
response to RESTART (idle for too long)
Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync():  
bailing out! Bad protocol


do you see any errors on the syncserver side. The error look like the
sync_client is waiting for a reply of 

Re: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13

[Michael Menge]

Hi,

Quoting Olaf Frączyk :


The current situation is:

1. Replica:

stopped and started the replica

no activity on replica - iotop and top show nothing

the only messages on replica is incoming connection from master

2. Master:

when I run sync_client -r I still get:

Apr 21 12:38:36 ifs sync_client[29518]: Reprocessing sync log file  
/var/lib/imap/sync/log-run
Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length  
response to MAILBOXES (idle for too long)
Apr 21 12:43:27 ifs sync_client[29518]: IOERROR: zero length  
response to RESTART (idle for too long)
Apr 21 12:43:27 ifs sync_client[29518]: Error in do_sync(): bailing  
out! Bad protocol
Apr 21 12:43:27 ifs sync_client[29518]: Processing sync log file  
/var/lib/imap/sync/log-run failed: Bad protocol


3. There is 27 GB of about 45 GB replicated and there is no further progress

4. How to find out why the replica doesn't respond?



You can enable telemetry logging on the replica by creating a folder  
/var/lib/imap/log/
where  is the value of the  sync_authname. If you give  
cyrus write permissions for this folder
it will create log-files for each process and what it received and  
send from/to the sync_client with timestamps.


Also you can try to use strace on the syncserver process to figure out  
which files it is accessing




On 2020-04-21 11:18, Olaf Frączyk wrote:

I also found out that when I see on master:

Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length  
response to MAILBOXES (idle for too long)
Apr 21 11:12:38 ifs sync_client[27996]: IOERROR: zero length  
response to RESTART (idle for too long)
Apr 21 11:12:38 ifs sync_client[27996]: Error in do_sync(): bailing  
out! Bad protocol
Apr 21 11:12:38 ifs sync_client[27996]: Processing sync log file  
/var/lib/imap/sync/log-run failed: Bad protocol


I also get on the replica:

Apr 21 11:12:38 skink1 imap[5775]: Connection reset by peer,  
closing connection


But I also see, that before it happens, there is no activity both  
on the replica and on the master for some time.


So maybe the imap server process is not recovering correctly in the  
longlock situation?


On 2020-04-21 11:07, Olaf Frączyk wrote:

Hi,

When I run sync_client -r on the master I see the following on the replica:

Apr 21 10:56:15 skink1 imap[5775]: mailbox: longlock  
navi.pl!user.olaf for 1.7 seconds
Apr 21 10:56:20 skink1 imap[5775]: mailbox: longlock  
navi.pl!user.piotr for 2.0 seconds
Apr 21 10:56:23 skink1 imap[5775]: mailbox: longlock  
navi.pl!user.olaf for 2.9 seconds
Apr 21 10:56:26 skink1 imap[5775]: mailbox: longlock  
navi.pl!user.piotr for 3.0 seconds


The mailboxes have several GB in Inbox folder and several GB in  
subfolders. The inbox folders have about 20,000 messages, the  
subfolders upto 15,000


Could it cause problems?


the longlock is normaly not the problem. While on process has the lock
no other process can write to the mailbox, but on the replica there normaly
is no other process that should access the mailbox



Maybe I should move the sync_client from START section to  
SERVICES, it seems that it is not automatically restarted



I havend tried starting sync_client in the service section.
I start the sync_client via systemd.


On 2020-04-21 08:47, Michael Menge wrote:

Hi Olaf


Quoting Olaf Frączyk :


Hi,

I upgraded to 3.0.13 but it didn't help.

This time it copied about 18GB

in the logs I still see:

1 - inefficient replication

2 - IOERROR: zero length response to MAILBOXES (idle for too long)
IOERROR: zero length response to RESTART (idle for too long)
Error in do_sync(): bailing out! Bad protocol

But I have no idea what can I do next and why it fails

Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length  
response to MAILBOXES (idle for too long)
Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length  
response to RESTART (idle for too long)
Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync():  
bailing out! Bad protocol


do you see any errors on the syncserver side. The error look like the
sync_client is waiting for a reply of the server.




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13

[Michael Menge]

Hi Olaf


Quoting Olaf Frączyk :


Hi,

I upgraded to 3.0.13 but it didn't help.

This time it copied about 18GB

in the logs I still see:

1 - inefficient replication

2 - IOERROR: zero length response to MAILBOXES (idle for too long)
IOERROR: zero length response to RESTART (idle for too long)
Error in do_sync(): bailing out! Bad protocol

But I have no idea what can I do next and why it fails

Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length  
response to MAILBOXES (idle for too long)
Apr 21 02:24:46 ifs sync_client[12656]: IOERROR: zero length  
response to RESTART (idle for too long)
Apr 21 02:24:46 ifs sync_client[12656]: Error in do_sync(): bailing  
out! Bad protocol


do you see any errors on the syncserver side. The error look like the
sync_client is waiting for a reply of the server.






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Confused about Deleted and Expunged

[Michael Menge]

Hi

I suspect a confusion about "marked as Deleted", Expunged and Expired

See  
https://www.cyrusimap.org/imap/reference/faqs/o-deleted-expired-expunged-purged.html

of more information.

E-Mails "marked as Deleted" need to be Expunged by the imap
EXPUNGE or UID EXPUNGE command.

The cyr_expire cmd is for removing Expunged mails from the index and disk
if  "expunge_mode: delayed" is used, which is the default since 2.5

You may want to take a look at the "/vendor/cmu/cyrus-imapd/expire"  
mailbox annotation




Quoting Adam Tauno Williams :


This command seems to work on Ubuntu 18.04. (cyrus-imapd 2.5.10-
3ubuntu1)
In /usr/sbin there is no cyr_expire but now i found a
/usr/lib/cyrus/bin/cyr_expire
The result is the same, no files were deleted, the log file entry is
the same as when i
use the "/usr/sbin/cyrus expire [..]" command
Can anybody confirm this "/usr/sbin/cyrus expire [..]" command on
Ubuntu?


What happens if you su to the cyrus user and try to run the command you
see in cyrus.conf ?

--
Adam Tauno Williams, awill...@whitemice.org
Multi-Modal Activists Against Auto Dependent Development
resisting the unAmerican socialists of the Motorist hegemony
http://www.mmaaadd.org

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Transfer existing quotas to new cyrus imap service

[Michael Menge]

Hi James,

Quoting "James B. Byrne via Info-cyrus" :


We are transferring an existing cyrus-imapd 3.0.11 mailstore to
another host also running cyrus-imapd 3.0.11.  I cannot find any
documentation on how one transfers user quotas.  Can someone provide
me with the link to the documentation or explain how it is done?

Thanks,



Cyrus stores important information in different paths.
See imapd.conf manpage:

configdirectory:
/mailboxes.db: list of all mailboxes and acls
/annotations.db: mailbox Annotations
/deliver.db: Information about recent delivered mails
  (needed for duplicate delivery suppression and  
sieve vacation)

/user/**: subscribed folders and seen information
/quota/**: Quota-Information

sievedir: Sieve Scripts


Depending on the configured partitions one ore more

partition-: Mailboxes/Mails of the users on the partition 

and optional

metapartition-: Meta-Files of the Mailboxes on the partition 
archivepartition-: Old Mails of Mailboxes on the partition 


You can use cyrus-replication to transfer all user/information to the
second server and keep them in sync.

https://www.cyrusimap.org/imap/reference/admin/sop/replication.html

Michael



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Sieve runtime error claiming mailbox doesn't exist

[Michael Menge]

Hi Patrick


Quoting Patrick Boutilier :

At first glance it looks like you need to change your sieve rules to  
use . as the mailbox separator. Something like:


user.jim.System Messages



I don't think the hierarchy separator is the problem. Some log outputs use the
internal format e.g. "delivered  to mailbox: user.jim" but  
the posted conf

shows he is using "/" as separator.

On December 13, 2019 11:08:54 AM AST, Jim Sculley  
 wrote:

Hello.  I'm a new member of this list but a long time user of
cyrus-imapd with postfix on my little home network.


I recently replaced CentOS 7 with CentOS 8 (CentOS Linux release
8.0.1905) on my  server which had been running
cyrus-imapd with sieve for years without problems.  I thought I had
gotten everything set up properly on the new system
(cyrus-imapd  3.0.7-15.el8_0.1 with cyrus-sasl 2.1.27-0.3rc7.el8 and
postfix 2:3.3.1-8.el8) but sieve is not moving
messages into the correct folders.



which cyrus version did you use with CentOS7?


For example, if I try to send a simple mail on the local network:


[root@buttercup quota]# mail j...@home.lan
Subject: Test Message
Test
.
EOT



The mail log will show:

=
Dec  6 12:51:39 buttercup postfix/pickup[27670]: 07FBA216825B: uid=0
from=
Dec  6 12:51:39 buttercup postfix/cleanup[27810]: 07FBA216825B:
message-id=<20191206175139.07fba2168...@mail.home.lan>
Dec  6 12:51:39 buttercup postfix/qmgr[27671]: 07FBA216825B:
from=, size=419, nrcpt=1 (queue active)
Dec  6 12:51:39 buttercup cyrus/lmtp[27814]: sieve runtime error for
jim id <20191206175139.07fba2168...@mail.home.lan>:
Fileinto (user/jim/System Messages): Mailbox does not exist



Can you post the sieve script. I don't know if the log shows the full path,
but make sure that the fileinto command does not contain the "user/jim" part
as the fileinto is relative to the INBOX of the user.


Dec  6 12:51:39 buttercup cyrus/lmtp[27814]: LOSTQUOTA: unable to
record change of 697 bytes and 1 messages in quota
user.jim: Quota root does not exist


something else is not working correct. Is the configdirectory configured?
Does it exist? Do directory and quotafiles exists?


Dec  6 12:51:39 buttercup cyrus/lmtp[27814]: Delivered:
<20191206175139.07fba2168...@mail.home.lan> to mailbox: user.jim
Dec  6 12:51:39 buttercup cyrus/lmtp[27814]: USAGE jim user: 0.012321
sys: 0.005097
Dec  6 12:51:39 buttercup postfix/lmtp[27813]: 07FBA216825B:
to=, relay=localhost[127.0.0.1]:24,
delay=0.27, delays=0.03/0.01/0.03/0.2, dsn=2.1.5, status=sent (250
2.1.5 Ok
SESSIONID=)
Dec  6 12:51:39 buttercup postfix/qmgr[27671]: 07FBA216825B: removed
=

So, sieve is running, and sees the rule in the script, but fails. This
behavior is seen with all my rules, none of which
are doing anything more than a fileinto based on sender addresses or
message subjects.

The mailbox does exist, as seen with cyradm:


buttercup.home.lan> lm user/jim/System\ Messages
user/jim/System Messages (\HasNoChildren)



This was all working fine before the CentOS 8 install. The relevant (I
think) imapd.conf entries:


==
defaultpartition: default
partition-default: /data/spool/imap
sievedir: /var/lib/imap/sieve
virtdomains: off
unixhierarchysep: yes
===

The spool directory looks like this:

===
[root@buttercup jim]# ls -l /data/spool/imap/j/user/jim
-rw---. 1 cyrus mail    336 Nov 30 17:57  cyrus.annotations
-rw---. 1 cyrus mail 373712 Dec  6 13:01  cyrus.cache
-rw---. 1 cyrus mail    189 Apr 19  2013  cyrus.header
-rw---. 1 cyrus mail  23424 Dec  6 13:01  cyrus.index
-rw---. 1 cyrus mail    112 Nov 26 05:31  cyrus.squat
drwx--. 2 cyrus mail  20480 Nov 30 17:57 'System Messages'
===

I'm not sure if I have some sort of configuration error or if this is
an actual bug. I would appreciate any assistance.






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mailboxes.db disagree between master and replica

[Michael Menge]

Quoting Tod Pike :


Hello:
  I am running a 3.X cyrus setup with master/slave replication. I
have discovered that one of my replicas has gotten out of sync with
it's master server, not in the sense that the mailboxes are not
replicated, but that the mailboxes live in different partitions on
the master and the slave.

  As an example, if I do a "ctl_mboxlist -d user." on the master,
I get back that it lives on partition 2. If I do the same thing on
the replica, I get back that it lives on partition 4. What happened
at one time is that the user got moved from partition 4 to partition 2
and it worked correctly on the master, but never happened on the slave.
BTW, the mupdate server agrees with the master, not the replica.


Make sure that the same partitions are configured on master and replica



  As an experiment, I tried to do a "ctl_mboxlist -m -w" on the
replica server, but it returns "couldn't get mupdate server name" since
it doesn't even know about the mupdate server in it's role as a replica.



Cyrus replication is not integrated with the murder setup.
A replica MUST NOT have a mupdate server configured.


  What are my options here - I know that I could just copy the
mailboxes.db file from the master to the replica, then manually move
the mailboxes that are in the "wrong" place to the "right" place, but
this seems a little drastic. Any way I can just tell the replica to
update the location of the mailbox one at a time?



Have you tried calling sync_client on the master with the "-u" option
(See sync_client manpage for details https://linux.die.net/man/8/sync_client)
This should sync all mailboxes, of the user and move the mailboxes to the
correct partition.

If this fails check for errors in the logs.

An other option would be to move the mailboxes to the correct partition with
cyradm using the renamemailbox command

  renamemailbox oldname newname [partition]





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [Help] Cyrus 2.4.17 segfault

[Michael Menge]

Quoting Marco :


Hello Michael,
  I've generated the dump, but it isn't easy for me to understand  
the content...


[root@cyrus ccpp-2019-11-13-07:04:30-19177]# gdb coredump
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Missing separate debuginfo for the main executable file


here is the why only addresses are show and not function names
you need the debug package for your cyrus imapd

Try: yum --disablerepo='*' --enablerepo='*-debug*' install  
/usr/lib/debug/.build-id/59/d016b26d3d62fcd00eb1400ff1be9c0142ee7c

[New Thread 19177]
Core was generated by `imapd:'.
Program terminated with signal 7, Bus error.
#0  0x7f3af70cd122 in ?? ()
"/var/spool/abrt/ccpp-2019-11-13-07:04:30-19177/coredump" is a core file.
Please specify an executable to debug.
(gdb) bt full
#0  0x7f3af70cd122 in ?? ()
No symbol table info available.
#1  0x7f3af757fcb0 in ?? ()
No symbol table info available.
#2  0x7f3af70cf008 in ?? ()
No symbol table info available.
#3  0x7fff in ?? ()
No symbol table info available.
#4  0x7f3af757fcb0 in ?? ()
No symbol table info available.
#5  0x7f3af757fcc0 in ?? ()
No symbol table info available.
#6  0x7f3af757fcc8 in ?? ()
No symbol table info available.
#7  0x7fff6dec5770 in ?? ()
No symbol table info available.
#8  0x7f3af757c8b0 in ?? ()
No symbol table info available.
#9  0x0001 in ?? ()
No symbol table info available.
#10 0x7f3a0fbc in ?? ()
No symbol table info available.
#11 0x7fff6dec5820 in ?? ()
No symbol table info available.
#12 0x in ?? ()
No symbol table info available.






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [Help] Cyrus 2.4.17 segfault

[Michael Menge]

Hi Marco,

can you generate a coredump and generate a backtrace (e.g. with gdb
https://stackoverflow.com/questions/5115613/core-dump-file-analysis )
You might need to install some debug packages.




Quoting Marco :


Il 12/11/2019 22:58, ellie timoney ha scritto:

On Tue, Nov 12, 2019, at 2:51 AM, Marco wrote:

An user user/a has full ACL to another mailbox user/b. When the user/a
SELECT a folder on user/b where he has access the imap process crashes.


If you set up a couple of test accounts with the same sharing  
arrangement, do those crash in the same way?  Or is it specific to  
the two original accounts?


Hi Ellie,

 I made other tests... It's specific to the user/a account.

When user/a SELECT a shared folder where it has access, the imap  
process crashes.


I updated to cyrus-imapd-2.4.20 and the issue still happens.

I also tried to rebuild mailboxes.db with ctl_mboxlist -d -u, I  
tried to rebuild mailboxes.db with an old backup... I tried to  
delete the statuscache and to delete the skipstamp db... the problem  
still happens.


Thank you
Marco

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: kick_mupdate refused

[Michael Menge]

Hi,

Quoting Miguel Mucio Santos Moreira :


Hello everybody

I've had a weird log information on my frontends servers regards to  
mupdate connection from frontend to mupdate master.

Mupdate Master would be refusing connections from frontend servers.

srvimapfprd01 cyrus/imaps[29677]: kick_mupdate: can't connect to  
target: Connection refused


These messages don't appear all day long and I don't notice any  
problem with the murder cluster


All my cluster has cyrus 2.4.16-4 version.

Any hints about it?



Does your mupdate master use TLS?

https://github.com/cyrusimap/cyrus-imapd/issues/2774

The problem was obvious in Cyrus 3.0 but I hat those connection  
problems also with 2.4

and I now think it was non-thread-safe usage of openssl





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2FA and IMAP

[Michael Menge]

Hi, 

You can configure a different IMAP Service in cyrus.conf for roundcube 
(differnt IP or port) and use pam with for authentication or block login for 
the other services wither cyrus denydb



Am 21. April 2019 23:09:06 MESZ schrieb Marcus Schopen :
>Hi,
>
>a friend wants to restrict access to his mailbox with 2FA. As webmailer
>I use Roundcube, which offers a 2FA plugin. But in the end this is
>pointless, because besides the webmailer there is also the native IMAP
>access available. Is it therefore possible to restrict the access to a
>single IMAP account to a certain IP so that this mailbox can only be
>accessed via the Roundcube?
>
>Ciao!
>Marcus
>
>
>Cyrus Home Page: http://www.cyrusimap.org/
>List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>To Unsubscribe:
>https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Which imap command to rename a root mailbox while maintaining its partition

[Michael Menge]

Hi,

Quoting Sebastian Hagedorn :


Hi,


On cyrus 2.4.x we used to run
A1 RENAME mbox1 mbox2 our_partition

When we run that on cyrus 3.0.8 we get :
A1 NO Cross-server or cross-partition move w/rename not supported

Is there a way to do this rename with one command with cyrus3.
If we run A1 RENAME mbox1 mbox2
the mailbox is renamed as expected but it moves from our_partition to
default which is not what we want.

Any suggestion ?


I have no idea, because we're not far enough along in our migration  
to Cyrus 3, but this is troubling me. We currently use RENAME  
specifically for moving users cross-partition. If that isn't  
supported anymore, is there a new alternative in Cyrus 3?



I have not tested this, but I think you can move cross-partition  
without rename

and rename without move cross-partition.

A1 RENAME mbox1 mbox1 our_partition
A2 RENAME mbox1 mbox2


Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sieve filters being ignored

[Michael Menge]

Quoting hwilmer :


Never mind, I forgot to activate the filter, sorry.

BTW, is there a way to have the destination folders automatically  
created with 'fileinto'?




there was a patch older versions that was included into the official
build in version 2.5.0.


https://github.com/cyrusimap/cyrus-imapd/blob/master/doc/README.autosievefolder
The Patch for 2.3.16 can be found here,
http://code.uoa.gr/p/cyrus/autosievefolder/

I was able to apply the patch to 2.4, but i don't remember if I had
to change the patch.




On 3/8/19 1:46 PM, hwilmer wrote:


Hi,

with cyrus installed on Centos 7, my sieve filter is being ignored  
in that no incoming mail is being filtered into destination folders  
according to the filter rules.


The filter file is basically the same as I am using on another  
cyrus installation where it works fine.  The filter can also be  
found under /var/lib/imap/sieve in a sub directory, as I would  
expect.


I adjusted the filter rules because apparently there is no INBOX  
folder anymore.  Here's an example rule:



if header :contains "List-Id" "Discussion group for Cyrus email system" {
  fileinto "lists.software.cyrus";
  stop;
}


The folder does exist; cyradm shows:


user.hwilmer.lists.software.cyrus (\HasNoChildren)


Is there a way to debug the sieve filtering?  What can I do to get  
sieve to work?


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Sieve script not working

[Michael Menge]

Hi


Quoting Willem Offermans :


Dear Michael and Cyrus friends,

Where is this documented?




I cannot find anything about this in

https://www.cyrusimap.org/imap/reference/admin/sieve.html  



The example shown in section "Testing the sieve server"
https://www.cyrusimap.org/imap/reference/admin/sieve.html#testing-the-sieve-server

# To place this script on the server run the following command:
#
# sieveshell servername
# Please enter your password: **
#> put myscript.script
#> activate myscript
#> quit
#
#This should place your script on the server and make it the active script.

As timsieved implements rfc5804 there is also
https://tools.ietf.org/html/rfc5804#section-1.4



How should a user/administrator know?

Pay attention to

https://www.cyrusimap.org/imap/reference/manpages/usercommands/sieveshell.html#std:cyrusman-sieveshell(1)  



and look at activate/deactivate.

This is sufficient documentation according to the developer, who  
spent many hours in the code, I guess.
However, an ignorant user only gets a hint at most. Where is  
defaultbc mentioned? Be aware that the
text the user reads is far away off what the user experiences, when  
he uses sieveshell. There is no

(visual) connection between the two. It is a challenge to find your way.



I don't know if the defaultbc is documented somewhere, google found
some old list mails from 2007 and a thread on a german linux forum.
So there is room for improvement.

The users can't normally see this file, it is only visible indirect as the
active script is shown / marked in timsieved / sieveshell "list" command.






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Sieve script not working

[Michael Menge]

Hi

Quoting Andrea Venturoli :


On 3/1/19 12:04 PM, Willem Offermans wrote:


(big snip on the rest)
$ man timsieved
TIMSIEVED(8)   
 TIMSIEVED(8)


Just to make sure I've clarified this: the script *is* there.


ls /var/imap/sieve/u/user1
user1.sieve.bc  user1.sieve.script




is the script activated? a user can have multiple scripts stored on the server
but only one can be activated at the same time. On the filesystem there should
be a "defaultbc"  which is a soft link to the active bc file.

The content of user1.sieve.script is the script I installed via  
sieveshell; user1.sieve.bc is a bytecode file where some strings  
(i.e. regex and folder names) match the original script, so I think  
it's right.


As I said, the same goes for user2 and user3, but for those users  
the script also works.


 bye & Thanks
av.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: another problem with conversations db

[Michael Menge]

Replying to myself,

I still see

IOERROR: conversations_audit on store

and

IOERROR: conversations_audit on load



Quoting Michael Menge :


Quoting Bron Gondwana :



ctl_conversationsdb -b should update the cid. BUT - if you're  
running old mailboxes which have a format which doesn't support  
saving the CID, that would for sure break things!




I did run "reconstruct -V max" during the upgrade. So all mailboxes  
should have been updated but

checking with

mbexamine user/* | grep "  Minor Version: 12"

I found some mailboxes that where not updated correctly. I didn't  
see any errors during the
initial upgrade, but I don't have the logs anymore so i don't know  
what happened.

Any idea what could lead to "reconstruct -V max" failing for some mailboxes?

I did rerun "reconstruct -r -V max user/LoginID" for those users,  
but I had two accounts where
the reconstruct command didn't update the index for the INBOX of  
that account. Rerunning it

again did update it.

The rebuild of the conversation_db worked fine after the Index was updated..
Do this did resolve the endless loop in ctl_conversationsdb. I would suggest
to add a check for the index version in ctl_conversationsdb

I will monitor my log files if the IOERRORS will reappear, or if the  
wrong index version

was the root of that problem too.

@Bron Thanks for the help





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: another problem with conversations db

[Michael Menge]

Quoting Bron Gondwana :



ctl_conversationsdb -b should update the cid. BUT - if you're  
running old mailboxes which have a format which doesn't support  
saving the CID, that would for sure break things!




I did run "reconstruct -V max" during the upgrade. So all mailboxes  
should have been updated but

checking with

mbexamine user/* | grep "  Minor Version: 12"

I found some mailboxes that where not updated correctly. I didn't see  
any errors during the
initial upgrade, but I don't have the logs anymore so i don't know  
what happened.

Any idea what could lead to "reconstruct -V max" failing for some mailboxes?

I did rerun "reconstruct -r -V max user/LoginID" for those users, but  
I had two accounts where
the reconstruct command didn't update the index for the INBOX of that  
account. Rerunning it

again did update it.

The rebuild of the conversation_db worked fine after the Index was updated..
Do this did resolve the endless loop in ctl_conversationsdb. I would suggest
to add a check for the index version in ctl_conversationsdb

I will monitor my log files if the IOERRORS will reappear, or if the  
wrong index version

was the root of that problem too.

@Bron Thanks for the help




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: disk space used by a mailbox without expunged

[Michael Menge]

Hi Marcus,

Quoting Marcus Schopen :


Hi,

is there a way to count the disk space used by a mailbox without
expunged messages?



mbexamine user/LoginID | grep Size

on older cyrus versions (2.3 and 2.4) mbexamine did examine all  
subfolders as well

in 3.0 only the info for the given folder is shown




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: another problem with conversations db

[Michael Menge]

Hi Bron,

sorry, i had to rearrange some quotes to put them my answers in a more  
meaningful order.



Quoting Bron Gondwana :


On Mon, Feb 4, 2019, at 22:00, Michael Menge wrote:


Quoting Bron Gondwana :

> On Mon, Feb 4, 2019, at 20:21, Michael Menge wrote:
>>



>> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
>> /srv/cyrus-be/ssd-part/L/user//2185.: No such file or directory
>> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
>> /srv/cyrus-be/ssd-part/L/user//2185.: No such file or directory
>> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive
>> user. 2185 failed to copyfile
>> (/srv/cyrus-be/ssd-part/L/user//2185. =>
>> /srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.): Unknown code
>>  255
>
>
> Ouch. Yeah, that could have been caused by a bug in delivery, and
> would definitely cause conversations DB corruption if the index file
> was updated but the conversations DB wasn't or vice versa.
>
>> The file was already at  
/srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.


I was able to fix these problems with reconstruct, and the didn't  
reappear till now.
Also there where other accounts which had IOERRORS regarding the  
conversation db,
with no cyr_expire archive errors, so i believe that these problems  
are not related.


I tried rebuilding the conversation db for the accounts with errors,  
but some other
accounts will show up with errors some time later. I counldn't find a  
some thing in

common jet.



>> > Anyway, I don't think that would break anything.
>> >
>> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
>> > metapartition_files: header index cache expunge squat annotations
>> > lock dav archivecache
>> >
>> > Ooh, I haven't tested having cache and archivecache on the same
>> > location. That's really interesting. Again, I'd be in favour of
>> > separation here, give them different paths. That might be tricky
>> > with ssd though, the way this is laid out. I assume you have some
>> > kind of symlink farm going on?
>> >
>>
>> I didn't know that there could be a problem with cache and archivecache.
>> At the time we decided on the configuration for cyrus 3.0 I looked at the
>> imapd.conf man page and for metapartition_files decided that I want all
>> meta files on the ssd storage. There was no indication in the man page
>> that there could be a problem.
>
> Fair. I'd have to test that to see if it works correctly. I would
> hope so, but I haven't tested that configuration. This is the
> downside with having lots of different ways to do things!
>
>> How do I separate location of archivecache from the other
>> metapartition path?
>> And fix the cache and archivecache files?
>
> This I don't know a good answer for. I will test if having the same
> path for cache and archivecache could fail. I THINK that I made the
> code safe for it, but I'm not sure that it's been tested.
>
>> No there is no sysmlink farm. We have mounted different iSCSI volumes to
>> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be
>
> Right. That makes sense.


Did you have time to look into the cache/archivecache situation jet?



> Right! I do wonder if there are some bugs in 3.0.x which are fixed
> on master around delivery to archive partition. We definitely had
> bugs on master, but I thought they were newly introduced on master
> as well, which is why the fixes weren't backported. But if you're
> having files be in the wrong location, maybe there are bugs on 3.0.x
> as well.


Are all fixes from master backported to 3.0?

Is the new Commit "I will try your new commits regarding CID" related to the
"IOERROR: conversations_audit on load:" and "IOERROR:  
conversations_audit on store"?


I will try your new commits in the next days on my test servers to sea  
if the fix

the endless loop in ctl_conversationsdb I have seen for some accounts.

Quoting myself (Re: prblems rebuilding conversations db) Jan 24, 2019


The program loops in build_cid_cb (imap/ctl_conversationsdb.c:189)

For the problematic mailbox that I tested, for every message
record->cid was NULLCONVERSATION, so mailbox_cacherecord,
message_update_conversations and mailbox_rewrite_index_record
where called, each returned 0.

After iterating trough all messages in the mailbox count was > 0, and r==0,
so the while condition (!r && count) was true for the next run.
At this point record->cid was still NULLCONVERSATION for every message,
which I guess should not be the case.


Michael


M.MengeTel.: (49

Re: another problem with conversations db

[Michael Menge]

Quoting Bron Gondwana :


On Mon, Feb 4, 2019, at 20:21, Michael Menge wrote:

Hi,

Quoting Bron Gondwana :

> Hi Michael,
>
> Sorry about the delay in looking at this - I was mad crazy busy
> getting ready to go overseas. At Fosdem now, about to give a talk
> about JMAP!
>
> OK, let's start with the things that give me a little bit of hives...
>
> configdirectory: /srv/cyrus-be
> partition-default: /srv/cyrus-be
> partition-ssd: /srv/cyrus-be/ssd-part
>
> Ouch. There's a couple of things I wouldn't do there - having the
> partition be the same as the config directory, and having a separate
> partition be a subdirectory of a partition. They're both asking for
> trouble. I would probably lay my system out like:
>
> configdirectory: /srv/cyrus-be/conf
> partition-default: /srv/cyrus-be/default-part
> partition-ssd: /srv/cyrus-be/ssd-part
>

partition-default isn't used any more. To use the metapartition we moved
all accounts form the default partition to the ssd partition which is the
the new defaultpartition ("defaultpartition: ssd")


Right - that makes sense.


> And then each tree would only have one type of thing in it.
>
> Anyway, I don't think that would break anything.
>
> metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
> metapartition_files: header index cache expunge squat annotations
> lock dav archivecache
>
> Ooh, I haven't tested having cache and archivecache on the same
> location. That's really interesting. Again, I'd be in favour of
> separation here, give them different paths. That might be tricky
> with ssd though, the way this is laid out. I assume you have some
> kind of symlink farm going on?
>

I didn't know that there could be a problem with cache and archivecache.
At the time we decided on the configuration for cyrus 3.0 I looked at the
imapd.conf man page and for metapartition_files decided that I want all
meta files on the ssd storage. There was no indication in the man page
that there could be a problem.


Fair. I'd have to test that to see if it works correctly. I would  
hope so, but I haven't tested that configuration. This is the  
downside with having lots of different ways to do things!


How do I separate location of archivecache from the other  
metapartition path?

And fix the cache and archivecache files?


This I don't know a good answer for. I will test if having the same  
path for cache and archivecache could fail. I THINK that I made the  
code safe for it, but I'm not sure that it's been tested.



No there is no sysmlink farm. We have mounted different iSCSI volumes to
/srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be


Right. That makes sense.


> Otherwise it all looks OK. Are you getting other IOERRORs in your
> log files which could show things aborting? It really looks like
> your conversations DB is getting out of sync due to other failures.

I found a few instances of 3 related errors.

Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
/srv/cyrus-be/ssd-part/L/user//2185.: No such file or directory
Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
/srv/cyrus-be/ssd-part/L/user//2185.: No such file or directory
Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive
user. 2185 failed to copyfile
(/srv/cyrus-be/ssd-part/L/user//2185. =>
/srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.): Unknown code
 255



Ouch. Yeah, that could have been caused by a bug in delivery, and  
would definitely cause conversations DB corruption if the index file  
was updated but the conversations DB wasn't or vice versa.



The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.


Right! I do wonder if there are some bugs in 3.0.x which are fixed  
on master around delivery to archive partition. We definitely had  
bugs on master, but I thought they were newly introduced on master  
as well, which is why the fixes weren't backported. But if you're  
having files be in the wrong location, maybe there are bugs on 3.0.x  
as well.


Do you have the syslog lines at the time that email was delivered?


I dont' have the log, for that message, but I will search for a
more recent example.


From the mail headers i know that it was not dilivered to the archive  
partition

but moved by cyr_expire. The conversation db was not used at that time.

PS.: the timesamp of the file is not the internal date but the time
the mail was moved to the archive partition. I was wondering about the reason.

Michael



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Arch

Re: another problem with conversations db

[Michael Menge]

Hi,

Quoting Bron Gondwana :


Hi Michael,

Sorry about the delay in looking at this - I was mad crazy busy  
getting ready to go overseas. At Fosdem now, about to give a talk  
about JMAP!


OK, let's start with the things that give me a little bit of hives...

configdirectory: /srv/cyrus-be
partition-default: /srv/cyrus-be
partition-ssd: /srv/cyrus-be/ssd-part

Ouch. There's a couple of things I wouldn't do there - having the  
partition be the same as the config directory, and having a separate  
partition be a subdirectory of a partition. They're both asking for  
trouble. I would probably lay my system out like:


configdirectory: /srv/cyrus-be/conf
partition-default: /srv/cyrus-be/default-part
partition-ssd: /srv/cyrus-be/ssd-part



partition-default isn't used any more. To use the metapartition we moved
all accounts form the default partition to the ssd partition which is the
the new defaultpartition ("defaultpartition: ssd")


And then each tree would only have one type of thing in it.

Anyway, I don't think that would break anything.

metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
metapartition_files: header index cache expunge squat annotations  
lock dav archivecache


Ooh, I haven't tested having cache and archivecache on the same  
location. That's really interesting. Again, I'd be in favour of  
separation here, give them different paths. That might be tricky  
with ssd though, the way this is laid out. I assume you have some  
kind of symlink farm going on?




I didn't know that there could be a problem with cache and archivecache.
At the time we decided on the configuration for cyrus 3.0 I looked at the
imapd.conf man page and for metapartition_files decided that I want all
meta files on the ssd storage. There was no indication in the man page
that there could be a problem.

How do I separate location of archivecache from the other metapartition path?
And fix the cache and archivecache files?

No there is no sysmlink farm.  We have mounted different iSCSI volumes to
/srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be


Otherwise it all looks OK. Are you getting other IOERRORs in your  
log files which could show things aborting? It really looks like  
your conversations DB is getting out of sync due to other failures.


I found a few instances of 3 related errors.

Feb  4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening  
/srv/cyrus-be/ssd-part/L/user//2185.: No such file or directory
Feb  4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening  
/srv/cyrus-be/ssd-part/L/user//2185.: No such file or directory
Feb  4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive  
user. 2185 failed to copyfile  
(/srv/cyrus-be/ssd-part/L/user//2185. =>  
/srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.): Unknown code  
 255


The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user//2185.

stat 2185.
  File: '2185.'
  Size: 424542  Blocks: 832IO Block: 4096   regular file
Device: 860h/2144d  Inode: 4677438462  Links: 1
Access: (0600/-rw---)  Uid: (   96/   cyrus)   Gid: (   12/mail)
Context: system_u:object_r:unlabeled_t:s0
Access: 2018-11-27 01:12:46.585864239 +0100
Modify: 2018-11-27 01:12:46.585864239 +0100
Change: 2018-11-27 01:12:46.585864239 +0100
 Birth: -




Cheers,

Bron.



On Thu, Jan 31, 2019, at 19:36, Michael Menge wrote:


Quoting Bron Gondwana :

> On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote:
>> Hi Bron
>>
>>
>> Quoting Bron Gondwana :
>>
>> > Sorry I haven't been following along with this earlier. Can you post
>> > your imapd.conf and cyrus.conf as well as let her know if you run
>> > anything which directly messes with files on disk.
>>
>> Thanks for looking into it. I have attached the backend and replica
>> configs. There should be nothing messing with the files on disk
>> directly. Some times we need to restore mails from file based
>> backup (bacula) followed by a reconstruct but this was not the
>> case this time.
>
> Do you always rebuild the conversations db after doing the
> reconstruct? That will be necessary now. We switched to doing all
> our restores using IMAP append a while back so we're never fiddling
> the file system under Cyrus.
>
>> >
>> > Also, what operating system is this on and what Cyrus version?
>> >
>>
>> We are running a cyrus 3.0.8 compiled with the following Options
>> (./configure --enable-murder --enable-http --enable-calalarmd
>> --enable-replication --enable-backup --enable-idled
>> --enable-autocreate CFLAGS="-fPIC -g")
>> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs.
>
> They should be fine. I'll have a read of the config when I'm at a
> real computer.
>

did you find anything in the config that would ex

Re: another problem with conversations db

[Michael Menge]

Quoting Bron Gondwana :


On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote:

Hi Bron


Quoting Bron Gondwana :

> Sorry I haven't been following along with this earlier. Can you post
> your imapd.conf and cyrus.conf as well as let her know if you run
> anything which directly messes with files on disk.

Thanks for looking into it. I have attached the backend and replica
configs. There should be nothing messing with the files on disk
directly. Some times we need to restore mails from file based
backup (bacula) followed by a reconstruct but this was not the
case this time.


Do you always rebuild the conversations db after doing the  
reconstruct? That will be necessary now. We switched to doing all  
our restores using IMAP append a while back so we're never fiddling  
the file system under Cyrus.



>
> Also, what operating system is this on and what Cyrus version?
>

We are running a cyrus 3.0.8 compiled with the following Options
(./configure --enable-murder --enable-http --enable-calalarmd
--enable-replication --enable-backup --enable-idled
--enable-autocreate CFLAGS="-fPIC -g")
in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs.


They should be fine. I'll have a read of the config when I'm at a  
real computer.




did you find anything in the config that would explain the problems?



> Bron.
>
> On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote:
>> Hi,
>>
>> I have discovered an other problem with the conversations db:
>>
>> Thousends of lines with "IOERROR: conversations_audit on load:" and
>> "IOERROR: conversations_audit on store:"
>> A look at the source code shows that these errors are logged after
>> "_sanity_check_counts" is called.
>> The log level LOG_ERR and the prefix IOERROR indicate that I have a
>> serious problem. Do I?
>>
>> This problem occurred for accounts where the rebuild of the
>> conversations db was successful.
>>
>> I don't want to rebuild the conversations db every few days.
>>
>> Any help is appreciated.
>>
>> Kind regards
>>
>>
>> Michael Menge
>>
>>
>>
>>
>>
>>  


>> M.Menge Tel.: (49) 7071/29-70316
>> Universität Tübingen Fax.: (49) 7071/29-5912
>> Zentrum für Datenverarbeitung mail:
>> michael.me...@zdv.uni-tuebingen.de
>> Wächterstraße 76
>> 72074 Tübingen
>>
>> 
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
> --
> Bron Gondwana
> br...@fastmail.fm




M.Menge Tel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

*Attachments:*
 * imapd_be.conf
 * cyrus_be.conf
 * cyrus_re.conf
 * imapd_re.conf


--
 Bron Gondwana
 br...@fastmail.fm





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: another problem with conversations db

[Michael Menge]

Hi,


Quoting Bron Gondwana :


On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote:

Hi Bron


Quoting Bron Gondwana :

> Sorry I haven't been following along with this earlier. Can you post
> your imapd.conf and cyrus.conf as well as let her know if you run
> anything which directly messes with files on disk.

Thanks for looking into it. I have attached the backend and replica
configs. There should be nothing messing with the files on disk
directly. Some times we need to restore mails from file based
backup (bacula) followed by a reconstruct but this was not the
case this time.


Do you always rebuild the conversations db after doing the  
reconstruct? That will be necessary now. We switched to doing all  
our restores using IMAP append a while back so we're never fiddling  
the file system under Cyrus.




We didn't have to restore any mails from backup since we enabled  
conversation db
a few weeks ago. It is good to know that the rebuild is necessary, but  
shouldn't

reconstruct also update the conversation db if it re-appends the message?
At least a hint should be put in the reconstruct man page, like the  
one for "quota -f"




>
> Also, what operating system is this on and what Cyrus version?
>

We are running a cyrus 3.0.8 compiled with the following Options
(./configure --enable-murder --enable-http --enable-calalarmd
--enable-replication --enable-backup --enable-idled
--enable-autocreate CFLAGS="-fPIC -g")
in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs.


They should be fine. I'll have a read of the config when I'm at a  
real computer.




> Bron.
>
> On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote:
>> Hi,
>>
>> I have discovered an other problem with the conversations db:
>>
>> Thousends of lines with "IOERROR: conversations_audit on load:" and
>> "IOERROR: conversations_audit on store:"
>> A look at the source code shows that these errors are logged after
>> "_sanity_check_counts" is called.
>> The log level LOG_ERR and the prefix IOERROR indicate that I have a
>> serious problem. Do I?
>>
>> This problem occurred for accounts where the rebuild of the
>> conversations db was successful.
>>
>> I don't want to rebuild the conversations db every few days.
>>
>> Any help is appreciated.
>>
>> Kind regards
>>
>>
>> Michael Menge





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: another problem with conversations db

[Michael Menge]

Hi Bron


Quoting Bron Gondwana :

Sorry I haven't been following along with this earlier. Can you post  
your imapd.conf and cyrus.conf as well as let her know if you run  
anything which directly messes with files on disk.


Thanks for looking into it. I have attached the backend and replica
configs. There should be nothing messing with the files on disk
directly. Some times we need to restore mails from file based
backup (bacula) followed by a reconstruct but this was not the
case this time.



Also, what operating system is this on and what Cyrus version?



We are running a cyrus 3.0.8 compiled with the following Options  
(./configure --enable-murder --enable-http --enable-calalarmd  
--enable-replication --enable-backup --enable-idled  
--enable-autocreate CFLAGS="-fPIC -g")

in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs.



Bron.

On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote:

Hi,

I have discovered an other problem with the conversations db:

Thousends of lines with "IOERROR: conversations_audit on load:" and
"IOERROR: conversations_audit on store:"
A look at the source code shows that these errors are logged after
"_sanity_check_counts" is called.
The log level LOG_ERR and the prefix IOERROR indicate that I have a
serious problem. Do I?

This problem occurred for accounts where the rebuild of the
conversations db was successful.

I don't want to rebuild the conversations db every few days.

Any help is appreciated.

Kind regards


 Michael Menge






M.Menge Tel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


--
 Bron Gondwana
 br...@fastmail.fm





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen
# Configuration for Backend/Failover Instance
# Template MD5SUM: b5b04095d552bb1cbc2b178f7edfd1e2  conf/imapd_master.template
servername: ma01.mail.localhost
configdirectory: /srv/cyrus-be
partition-default: /srv/cyrus-be
partition-ssd: /srv/cyrus-be/ssd-part
metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
metapartition_files: header index cache expunge squat annotations lock dav 
archivecache
archivepartition-ssd: /srv/cyrus-hdd-be/archive/ssd-part
archive_enabled: 1
archive_days: 31
archive_maxsize: 10240

proc_path: /srv/tmpfs/proc-be
mboxname_lockpath: /srv/tmpfs/lock-be
defaultpartition: ssd
admins:  

mupdate_server: mupdate.mail.localhost 
mupdate_port: 3905
# mupdate_authname verwenden nicht mupdate_username
# sonst wird login als root/cyrus versucht
#mupdate_username: 
mupdate_authname: 
mupdate_password: 
proxy_authname: 
proxy_password: 
proxyservers: 

allowusermoves: 1
allowallsubscribe: 1

sync_host: sl01.mail.localhost 
sync_authname: 
sync_password: 
sync_port: 2005
guid_mode: sha1
sync_log: 1
sync_shutdown_file: /srv/cyrus-be/sync/shutdown

sievedir: /srv/cyrus-be/sieve
sieve_extensions: fileinto reject vacation imapflags notify include envelope 
body relational regex subaddress copy
sieve_maxscriptsize: 150

sasl_pwcheck_method: saslauthd
sasl_mech_list: plain login

allowanonymouslogin: no
reject8bit: no
quotawarn: 90
timeout: 45
poptimeout: 10
dracinterval: 0
drachost: localhost
lmtp_over_quota_perm_failure: 1
altnamespace: 1
#flushseenstate: 1
unixhierarchysep: 1
hashimapspool: 1
fulldirhash: 1
duplicatesuppression: 0
expunge_mode: delayed
delete_mode: delayed
deletedprefix: DELETED
anysievefolder: 1
#annotation_db: skiplist
#duplicate_db: skiplist
#mboxlist_db: skiplist
#ptscache_db: skiplist
quota_db: quotalegacy
#seenstate_db: skiplist
subscriptions_db: flat
xlist-sent: Mail.sent
xlist-trash: Mail.trash
xlist-drafts: Mail.drafts
xlist-junk: Mail.v-spam
xlist-spam: Mail.v-spam
specialusealways: 1
syslog_prefix: be 

# https://bugzilla.cyrusimap.org/show_bug.cgi?id=3850
# Vermutlich gefixed
#suppress_capabilities: LIST-EXTENDED


tls_server_cert: /etc/pki/cyrus-imapd/cyrus-imapd.pem
tls_server_key: /etc/pki/cyrus-imapd/cyrus-imapd.pem
tls_client_ca_file: /etc/pki/tls/certs/ca-bundle.crt
tls_ciphers: 
EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA:EECDH:EDH+AESGCM:EDH:+3DES:ECDH+AESGCM:ECDH+AES:ECDH:AES:HIGH:MEDIUM:!RC4:!CAMELLIA:!SEED:!aNULL:!MD5:!eNULL:!LOW:!EXP:!DSS:!PSK:!SRP:!3DES:!IDEA
tls_prefer_server_ciphers: 1

# Änderungen 27.06.2018
reverseacls: 1
search_engine: squat
delete_unsubscribe: 1

another problem with conversations db

[Michael Menge]

Hi,

I have discovered an other problem with the conversations db:

Thousends of lines with "IOERROR: conversations_audit on load:" and  
"IOERROR: conversations_audit on store:"
A look at the source code shows that these errors are logged after  
"_sanity_check_counts" is called.
The log level LOG_ERR and the prefix IOERROR indicate that I have a  
serious problem. Do I?


This problem occurred for accounts where the rebuild of the  
conversations db was successful.


I don't want to rebuild the  conversations db every few days.

Any help is appreciated.

Kind regards


   Michael Menge






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: prblems rebuilding conversations db

[Michael Menge]

Hi

Quoting ellie timoney :
...

On Tue, Jan 22, 2019, at 12:32 AM, Michael Menge wrote:

...

Quoting ellie timoney :

...

> On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote:
>> Hi,
>>
>> because conversations db seems to be required for search indexes, I
>> enabled this option
>> on our production servers today and tried to rebuild the conversations
>> db for all users with
>>
>> ctl_conversationsdb -v -b UID
>>
>> For most users this did take less than a second. But for some users
>> this process would
>> not finish. I did kill one process after about 30 Minutes (most others
>> after 3 Minutes).
>> The UserID.conversations has grown over 2 GB (the mailbox itself has
>> only ~700 MB of mails,
>> and the conversations files from finnished rebuilds are less then 20 MB)

...

I did run ctl_conversationsd -b with strace and discovered that the inbox
(cyrus.index O_RDWR, cyrus.header O_RDONLY, cache  cyrus.O_RDWR )  
of that user

was opened over and over again, no other folder for that user was accessed.
For other users the inbox was only opened once and then the other
folders followed.
So not stopping "ctl_conversationsd -b", it would have run till my
filesystem was
full.



I guess something about that inbox is confusing it, and making it  
redo it over and over?  Very curious



...

> Though, that raises the question: was the user accessing the mailbox
> while the rebuild was in progress?  I wonder if their client is
> doing something funny and tripping things up?

We did this while the server was accessible (didn't want to have
several hour downtime for ~44000 users)


Fair enough!


But as I did my recent testing on a test-server without user access,
only ctl_conversationsdb and cyr_dbtool did accessed the mailbox during
the conversation rebuild mentioned in this mail. So I don't think that
user access is part of the problem.


Thanks for testing that, good to rule it out as a culprit


So for me it looks like we have two problems:

1. multiple entries for the same key in skiplist db files.
As skiplist is a Key/Value store this should not be possible
to have duplicate keys?


I don't know the internals of our database formats well, but I  
_think_ maybe it writes new key/value pairs by appending to the end  
of the database, and then updating the skiplist structure to point  
to the newer version instead of the older version.  If this is  
correct, then if it's rewriting the keys over and over for some  
reason, I would expect the raw file to contain a lot of old,  
unlinked records for the same key, but which aren't visible over the  
API.  I believe there's a process that runs occasionally that  
rewrites these databases with only the "live" keys and gets back the  
wasted space?




I did discover, because of the loop (see below) the point of
database commit was not reached, so I guess that could explain
that there was no duplicated key cleanup done.


Is twoskip a better alternative?


This is before my time, but twoskip was written to solve problems  
FastMail experienced with using skiplist with conversations  
("post-crash recovery was too slow").  So if there's a pathological  
case in the more recent conversations code that trips a bug in  
skiplist, we probably wouldn't see it.  So, I'd recommend using  
twoskip for conversationsdb just based on that :)


Same problem with twoskip, but as I said above, the commit point
was not reached.




2. Why didn't ctl_conversationsdb continue to process
the next mailbox but re-did the INBOX / the last mailbox


It kinda sounds to me like it considered the operation to be  
unsuccessful, and tried again?  Not sure what would cause this.  Is  
there anything interesting in syslog?



I will try to run a debugger on ctl_conversationsdb and
will test with twoskip as conversations db format




The program loops in build_cid_cb (imap/ctl_conversationsdb.c:189)

For the problematic mailbox that I tested, for every message
record->cid was NULLCONVERSATION, so mailbox_cacherecord,
message_update_conversations and mailbox_rewrite_index_record
where called, each returned 0.

After iterating trough all messages in the mailbox count was > 0, and r==0,
so the while condition (!r && count) was true for the next run.
At this point record->cid was still NULLCONVERSATION for every message,
which I guess should not be the case.

Thanks, it'd be very interesting to see if this issue reproduces  
with the twoskip format!




Yes, it did.


I did notice one other problem, recreating the conversations db
on the replica confused the sync protocol as the rebuild did increase
the modseq.


Do you mean to say that this user's conversations db could be  
rebuilt successfully on the replica?  Or was this with one of the  
"good" users?




I did activated the conversation db first o

prblems rebuilding conversations db

[Michael Menge]

Hi,

because conversations db seems to be required for search indexes, I  
enabled this option
on our production servers today and tried to rebuild the conversations  
db for all users with


ctl_conversationsdb -v -b UID

For most users this did take less than a second. But for some users  
this process would
not finish. I did kill one process after about 30 Minutes (most others  
after 3 Minutes).
The UserID.conversations has grown over 2 GB (the mailbox itself has  
only ~700 MB of mails,

and the conversations files from finnished rebuilds are less then 20 MB)

Cyrus Logs show many "ctl_conversationsdb[933]: mailbox: longlock  
user.UserID for 1.5 seconds"

every few seconds.

I did try reconstruct -r -G user/UserID, followed by   
ctl_conversationsdb -v -z UID
and tried to rebuild the db again. Reconstruct reported no errors, and  
the new rebuild

hat the same problem.

Has anybody seen a similar problem?

Any ideas how to fix this?


We are running cyrus-imapd 3.0.8


Kind regards

   Michael Menge


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus-imapd not starting after upgrade

[Michael Menge]

Quoting Daniel Bareiro :


Hi, Michael.

Thanks for your reply.

On 15/1/19 17:47, Michael Menge wrote:


After that I managed to get the service is up and running (although I
have still doubt about the "http" line).



http is used for CalDAV / CardDav so is most likely in the cyrus-caldav
package.
This enables your cyrus server to store address books and calendars.


It seems that "http" does indeed enable CalDAV / CardDAV support with
the cyrus-caldav package. I had not thought of it :-) After installing
the package and uncommenting the "http" line, the error was not shown. I
will need this to synchronize calendars and tasks.

I guess it would have to work with this but Thunderbird shows a message
saying "the calendar is momentarily unavailable". It's weird because it
brought the information from Horde calendar and in fact I added a new
event in Thunderbird and it was shown by Horde. But when I pressed the
sync button this message was shown with an exclamation mark. But this
has already happened to me with Cyrus 2.4.17 and Thunderbird 60.x. I'm
thinking if it has to do with something on the side of Thunderbird 60.x
or Horde or the result of the interaction of both.



Horde (Kronolith and Turba) have there own ClaDAV / CardDav Server,
AFAIK Horde is at the moment not able to use cyrus as an Backend for
its calendars and address books.

Horde had support for Kolab, and the CalDAV / CardDAV in Cyrus is
based on what Kolab did there. But the support in Horde has not been
updated yet.

So most likely you have configured your thunderbird to synchronize
to horde and do not use the cyrus CalDAV / CardDAV feature


I have also come across this error now:

--
Jan 15 15:32:16 mail cyrus/imaps[21978]: IDLE: error sending message
INIT to idled for mailbox user.jsmith: No such file or directory.
Falling back to polling every 60 seconds.
--



is idled (man idled) configured in cyrus.conf?


When I upgraded to Stretch I initially installed the versions of
configurations provided by the package maintainer and then passed the
changes from the previous configuration files to the new versions of the
configuration files. I thought that this would be the best.

The "idled" line was commented out with Cyrus 2.4.17 and I don't
remember having ever seen those messages in the log. So here I also kept
it commented out (as it came initially). Maybe this is because something
changed in the new version of Cyrus. Now I have activated that line and
the message has vanished.


It seems this is preventing access to IMAP mailboxes. I am investigating
the cause. Any contribution is welcome.



It should not preventing access to IMAP mailboxes. This is a cpu friendly
"inform me if there is new mail" feature which results in
"Falling back to polling every 60 seconds." if it is missing


Sorry. I expressed myself inaccurately. The content of the mailboxes is
accessed. What I meant is it seems that from time to time the connection
is lost. This is what I see in the log when that happens:

--
Jan 15 19:12:22 mail cyrus/imaps[27754]: Fatal error: Exists wrong 257
256 11 16585
Jan 15 19:12:22 mail cyrus/master[26739]: process type:SERVICE
name:imaps path:/usr/lib/cyrus/bin/imapd age:0.265s pid:27754 exited,
status 75
--

For example, that happens when I try to access a folder with more than
2000 messages. I am investigating the cause. If you know what it can be,
any info will be welcome. It's weird because I don't remember seeing
this with Cyrus 2.4.17.

Thanks for your time.

Kind regards,
Daniel





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus-imapd not starting after upgrade

[Michael Menge]

Quoting Daniel Bareiro :


On 15/1/19 15:05, Daniel Bareiro wrote:


The fact is that if I also comment out the "http" line, now it seems
that I get a syntax error:

--
Jan 15 14:14:28 mail cyrus/master[20778]: configuration file
/etc/cyrus.conf: bad character '_' in name on line 101
Jan 15 14:14:28 mail cyrus/master[20778]: exiting
--

I think the configuration file already came with the "squatter_a" line
and I just removed the comment. Unless this is a bug in the
configuration file provided by Stretch for Cyrus 2.5.10:

--
 99 # reindex all mailboxes (fulltext) daily
100 # DGB - 20190114
101 squatter_a  cmd="/usr/sbin/cyrus squatter" at=0517
--


H... it seems this is a bug in the Debian Stretch configuration file
because of what I see in the examples shown here [1]. The file also has
the line "squatter_1" and it should be "squatter1".

After that I managed to get the service is up and running (although I
have still doubt about the "http" line).


http is used for CalDAV / CardDav so is most likely in the  
cyrus-caldav package.

This enables your cyrus server to store address books and calendars.



I have also come across this error now:

--
Jan 15 15:32:16 mail cyrus/imaps[21978]: IDLE: error sending message
INIT to idled for mailbox user.jsmith: No such file or directory.
Falling back to polling every 60 seconds.
--



is idled (man idled) configured in cyrus.conf?


It seems this is preventing access to IMAP mailboxes. I am investigating
the cause. Any contribution is welcome.



It should not preventing access to IMAP mailboxes. This is a cpu friendly
"inform me if there is new mail" feature which results in
"Falling back to polling every 60 seconds." if it is missing


Thanks in advance for your time.


Kind regards,
Daniel

[1] https://www.cyrusimap.org/2.5/imap/admin/systemcommands/squatter.html





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus-imapd not starting after upgrade

[Michael Menge]

Quoting Daniel Bareiro :


Hi all!

After quite some time, today I decided to update the mail server from
Debian Jessie (cyrus-imapd 2.4.17) to Debian Stretch (cyrus-imapd 2.5.10-3).

All without problems until I reach the part of cyrus-imapd that does not
start. This is what I see in the log:

--
Jan 14 23:10:45 mail systemd[1]: Started Cyrus IMAP/POP3 daemons.
Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: skiplist: clean shutdown
file missing, updating recovery stamp
Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: recovering cyrus databases
Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: done recovering cyrus
databases
Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: Repacking mailbox
user.admin.TareasCron version 12
Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: Expired 0 and expunged 0
out of 28809 messages from 80 mailboxes
Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: duplicate_prune: pruning
back 3.00 days
Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: duplicate_prune: purged 0
out of 438 entries
Jan 14 23:10:46 mail cyrus/tls_prune[5335]: twoskip: invalid magic
header: /var/lib/cyrus/tls_sessions.db
Jan 14 23:10:46 mail cyrus/tls_prune[5335]: cyrusdb: opening
/var/lib/cyrus/tls_sessions.db with backend skiplist (requested twoskip)
Jan 14 23:10:46 mail cyrus/tls_prune[5335]: skiplist: recovered
/var/lib/cyrus/tls_sessions.db (223 records, 41200 bytes) in 0 seconds
Jan 14 23:10:46 mail cyrus/tls_prune[5335]: skiplist: checkpointed
/var/lib/cyrus/tls_sessions.db (223 records, 41200 bytes) in 0.091 sec
Jan 14 23:10:46 mail cyrus/tls_prune[5335]: tls_prune: purged 2 out of
223 entries
Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for
service 'nntp'
Jan 14 23:10:46 mail cyrus/master[5311]: exiting
Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Main process
exited, code=exited, status=78/n/a
Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Unit entered
failed state.
Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Failed with result
'exit-code'.
--

I'm not sure what the problem is but that "invalid magic header" makes
me think that maybe it changed the header format of
/var/lib/cyrus/tls_sessions.db and the migration process did not do the
corresponding conversion. Can that be the reason why it doesn't start or
I'm missing something else? Any ideas that can bring more light?


some defaults for database format have changed between 2.4 and 2.5 but this
is automatically handled.

You Problem lies in

Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for  
service 'nntp'

Jan 14 23:10:46 mail cyrus/master[5311]: exiting

Either the executables are installed in the wrong place, or cyrus was  
compiled/packaged
without news support. You can comment out nntp in /etc/cyrus.conf to  
try to start without

running an nntp server


The associated problem is that because of this it seems that Postfix can
not deliver the mails since there is no /var/run/cyrus/socket/lmtp.


As cyrus aborted the start there is no process listening




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Upgrade to cyrus-imapd and connection drops when searching mailboxes

[Michael Menge]

Hi,

Quoting "James B. Byrne via Info-cyrus" :


cyrus-imapd30-3.0.8_2
Name   : cyrus-imapd30
Version: 3.0.8_2
Installed on   : Tue Jan  8 11:07:34 2019 EST
Origin : mail/cyrus-imapd30
Architecture   : FreeBSD:11:amd64
Prefix : /usr/local
Categories : ipv6 mail
Licenses   : BSD4CLAUSE
Maintainer : u...@freebsd.org

Since upgrading to this version from 3.0.8_1 on January 8 we have seen
a considerable increase of messages in the maillog that look similar
to this:

CYRUS/lmtpunix[79768]: mailbox: longlock user.name.delivery for 1.4
seconds

Where the number of seconds varies between 1.x and 11.x.  When I say
considerably I mean from between 10 and 30 per log rollover to over
800 today and it is not yet noon.

Perhaps coincidently, users are reporting imap connections dropping
when they are performing text searches on their mailboxes.  The user
mail client being squirrelmail.



have you configured a search engine? and have you enabled the conversation db.
Xapian and Squatter seam to require enabled conversation db to use the
search index. At least for squatter there is even a performance regression
for TEXT search even if conversation db is enabled
(https://github.com/cyrusimap/cyrus-imapd/issues/2598)



Are the longlock reports and the imap connection drop related?


An cyrus processes tires to gain access to the mailbox while an
other process is still accessing it. If this takes to long the client
might disconnect. You can try telemetry logging to discover what is
happening .



If imap drops a connection is the reason for that event logged?  Can
it be logged?


The question is which side drops the connection the cyrus-imapd process
or the imap client (squirrelmail)? tcpdum and telemetry logging might
reveal the answer.

What timeout is configured in squirrelmail?




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: upgrade to cyrus_imap or saslauth or both gon horribly wrong

[Michael Menge]

Hi,


Quoting James B Byrne :


FreeBSD-11.2p7
cyrus-imapd30-3.0.8_2
cyrus-sasl-saslauthd-2.1.27
cyrus-sasl-2.1.27

This morning we upgraded our cyrus_imap server using the FreeBSD  
pkg package manager.  Following this we are unable to authenticate  
with imap.  The error we receive is this:


Jan  8 14:05:37 inet17 CYRUS/imaps[40533]: SASL cannot connect to  
saslauthd server: Permission denied
Jan  8 14:05:37 inet17 CYRUS/imaps[40533]: badlogin: servername  
[server address] plaintext username SASL(-1): generic failure:  
checkpass failed


imapd.conf was not changed.  it contains this:

sasl_mech_list: PLAIN
sasl_pwcheck_method:saslauthd


I am posting this from a temporary email because, duhh, I cannot  
access my regular mailbox.


I am open to any reasonable suggestions as to how to fix this, quickly.


However, if I do this then I get a success.

read -ers -p 'prompt: ' PASSWD ; echo -e '\n'
prompt:

read -ers -p 'prompt: ' USERNAME ; echo -e '\n'
prompt:

testsaslauthd -u $USERNAME -p $PASSWD
0: OK "Success."

Does anyone here know what is going on or how to fix it?


are you using SELinux in enforced mode? If yes check your SELinux  
audit log (/var/log/audit/audit.log)





Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian searches of the body of an email

[Michael Menge]

Hi,

Quoting Egoitz Aurrekoetxea :


And, by the way

when using Squatter instead of Xapian as a search engine what do we
really lost?. Just the fact of having a statistical worse results?. Is
it Xapian faster than squat engine?.



I didn't test Xapian but the Squatter search index is not used in  
cyrus-imapd 3.0


See https://github.com/cyrusimap/cyrus-imapd/issues/2598



Sorry for having so many questions but... I suppose I don't have the
implications of each one totally clear :)

Cheers!

---

EGOITZ AURREKOETXEA
Departamento de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es
www.sarenet.es [1]
Antes de imprimir este correo electrónico piense si es necesario
hacerlo.

El 07-01-2019 14:51, Egoitz Aurrekoetxea escribió:


Hi mate!

This seems to take ages... I'm trying to figure the best way of  
implementing this and of clarifying concepts I'm running  
Squatter in rolling replication mode and exist the concept of  
conversations then. What is the exact role of each of them?.  
Squatter seems to index the mailbox but when something is not  
properly indexed instead of running Squatter you use  
ctl_conversations for reindexing some part again or...


Thanks a lot!!

---

EGOITZ AURREKOETXEA
Departamento de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es
www.sarenet.es [1]
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

El 07-01-2019 10:19, Sebastian Hagedorn escribió:
That sounds like the conversationsdb issue I was talking about.  
Have you tried these steps?


ctl_conversationsdb -z USER
ctl_conversationsdb -b USER

I have been testing Xapian searches. Have seen, it's not able to find
strings inside the body of the email. If I set in imap.conf
"search_fuzzy_always: 1" no content is displayed in the searches of a
Roundcube stock webmail. If I remove that config value from imap.conf
and restart services, then search results appear. Does Xapian not index
the body of emails?. Does Xapian, just index the headers?. But this
affirmation does not seem to be possible in my case too... as I have in
the config "search_index_headers: no".

I'm using the following config :

conversations: 1
search_engine: xapian
search_index_headers: no
search_batchsize: 8192
defaultsearchtier: t1
t1searchpartition-default: /expert/search
t1searchpartition-expert2: /expert2/search
t1searchpartition-expert3: /expert3/search

Could anyone help me mates?.

--
Sebastian Hagedorn - Weyertal 121, Zimmer 2.02
Regionales Rechenzentrum (RRZK)
Universität zu Köln / Cologne University - Tel. +49-221-470-89578



Links:
--
[1] http://www.sarenet.es





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

bug renameing/deliting special use folders in murder setup.

[Michael Menge]

Hi,

we discovered a bug in cyrus imapd 3.0.8 in murder/aggregator configuration.

We have the following default stucture:

a LIST "" "*"
* LIST (\HasNoChildren) "/" INBOX
* LIST (\HasChildren) "/" Mail
* LIST (\HasNoChildren \Drafts) "/" Mail/drafts
* LIST (\HasNoChildren) "/" Mail/s-spam
* LIST (\HasNoChildren \Sent) "/" Mail/sent
* LIST (\HasNoChildren \Trash) "/" Mail/trash
* LIST (\HasNoChildren \Junk) "/" Mail/v-spam
a OK Completed (0.000 secs 7 calls)

ctl_mboxlist -d | grep ^user.zrstes1
user.zrstes11 ma05.mail.localhost!ssd zrstes1   lrswipkxtecdan
user.zrstes1.Mail   1 ma05.mail.localhost!ssd zrstes1   lrswipkxtecdan
user.zrstes1.Mail.drafts1 ma05.mail.localhost!ssd zrstes1   
lrswipkxtecdan
user.zrstes1.Mail.s-spam1 ma05.mail.localhost!ssd zrstes1   
lrswipkxtecdan
user.zrstes1.Mail.sent  1 ma05.mail.localhost!ssd zrstes1   lrswipkxtecdan
user.zrstes1.Mail.trash 1 ma05.mail.localhost!ssd zrstes1   lrswipkxtecdan
user.zrstes1.Mail.v-spam1 ma05.mail.localhost!ssd zrstes1   
lrswipkxtecdan

Renaming/Deleting a Mailbox with Special-Use Flag fails

b RENAME Mail/v-spam Mail/spam
b NO SPECIAL-USE flag conflict
c DELETE Mail/v-spam
c NO SPECIAL-USE flag conflict

but Renaming/Deleting the Folder above will succeed

d RENAME Mail Test
* OK rename Mail Test
* OK rename Mail/drafts Test/drafts
* OK rename Mail/s-spam Test/s-spam
* OK rename Mail/sent Test/sent
* OK rename Mail/trash Test/trash
* OK rename Mail/v-spam Test/v-spam
d OK Completed

but results in broken entries in the mailboxdb on mupdate master and frontends

ctl_mboxlist -C /etc/imapd_fe.conf -d | grep ^user.zrstes1
user.zrstes11 ma05.mail.localhost!ssd zrstes1   lrswipkxtecdan
user.zrstes1.Mail.drafts1 ma05.mail.localhost!
user.zrstes1.Mail.sent  1 ma05.mail.localhost!
user.zrstes1.Mail.trash 1 ma05.mail.localhost!
user.zrstes1.Mail.v-spam1 ma05.mail.localhost!
user.zrstes1.Test   1 ma05.mail.localhost!ssd zrstes1   lrswipkxtecdan
user.zrstes1.Test.drafts1 ma05.mail.localhost!ssd zrstes1   
lrswipkxtecdan
user.zrstes1.Test.s-spam1 ma05.mail.localhost!ssd zrstes1   
lrswipkxtecdan
user.zrstes1.Test.sent  1 ma05.mail.localhost!ssd zrstes1   lrswipkxtecdan
user.zrstes1.Test.trash 1 ma05.mail.localhost!ssd zrstes1   lrswipkxtecdan
user.zrstes1.Test.v-spam1 ma05.mail.localhost!ssd zrstes1   
lrswipkxtecdan

see the missing partition for old names of the special use folders.
it seams that the sepecial use annotation was moved with the folders without
a problem

e LIST (SPECIAL-USE) "" "*"
* LIST (\HasNoChildren \Drafts) "/" Test/drafts
* LIST (\HasNoChildren \Sent) "/" Test/sent
* LIST (\HasNoChildren \Trash) "/" Test/trash
* LIST (\HasNoChildren \Junk) "/" Test/v-spam
e OK Completed (0.000 secs 7 calls)

On the backend the mailboxdb entries look fine. But the user is unable  
to rename/move the folders back,
or create the new folders with the old name because the folder are  
still reserved on the mupdate server.


f RENAME Test Mail
* OK rename Test Mail
* NO rename Test/drafts Mail/drafts: unable to reserve mailbox on  
mupdate server

f NO unable to reserve mailbox on mupdate server
g LIST "" "*"
* LIST (\HasNoChildren) "/" INBOX
* LIST (\HasNoChildren) "/" Mail
* LIST (\HasNoChildren \Drafts) "/" Test/drafts
* LIST (\HasNoChildren) "/" Test/s-spam
* LIST (\HasNoChildren \Sent) "/" Test/sent
* LIST (\HasNoChildren \Trash) "/" Test/trash
* LIST (\HasNoChildren \Junk) "/" Test/v-spam
g OK Completed (0.010 secs 7 calls)

a "ctl_mboxlist -m -a" on the backend will fix the mailboxdb on the  
mupdate master/frontend


I am not sure if moving a special use folder should work or not this  
should be consistent to

renaming a top folder.

At the moment our users are unable to restore their folder structure if they
renamed/moved their folders unintended.

Regards


   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: imapd and count of deleted messages

[Michael Menge]

Hi,


Quoting Sergey :


Hello

Can I see number of deleted messages per IMAP sessions?
This is useful when communicating with the user sometimes.

For imap I can see only this:

cyrus/imap[862231]: login:  User logged in  
SESSIONID=

cyrus/imap[862231]: USAGE  user: 0.228965 sys: 0.244962

And much more for pop:

cyrus/pop3[19165]:  User logged in  
SESSIONID=

cyrus/pop3[19165]: Expunged 160 messages from 
cyrus/pop3[19165]: USAGE  user: 0.564000 sys: 0.516000
cyrus/pop3[19165]: counts: retr=<160> top=<0> dele=<160>




Are you sure messages have been "Deleted" via IMAP EXPUNGE command?

To clarify this: Deleting mails is a 2 or 3 stage process depending on
your configuration.

1. the mails are marked as deleted.

   = Example from RFC 3501 =

   C: A003 STORE 2:4 +FLAGS (\Deleted)
   S: * 2 FETCH (FLAGS (\Deleted \Seen))
   S: * 3 FETCH (FLAGS (\Deleted))
   S: * 4 FETCH (FLAGS (\Deleted \Flagged \Seen))
   S: A003 OK STORE completed

   ===

   This is normally not logged by cyrus but you can see the number of Messages
   with the Deleted Flag with

mbexamine user/userID | grep "^  Deleted:"

The Client can still see these messages. Some Clients use this to allow the
user to "undelete" Mails other kopie/move the mails to a trash folder and
use the second step to remove the mails form the original folder.
You can try enable the "auditlog" option in your imapd.conf if you really
need to know this.

2.  the mails marked as deleted are removed form the mailbox with the  
expunge command.


 Quoting RFC 3501 =


Example:C: A202 EXPUNGE
S: * 3 EXPUNGE
S: * 3 EXPUNGE
S: * 5 EXPUNGE
S: * 8 EXPUNGE
S: A202 OK EXPUNGE completed

Note: In this example, messages 3, 4, 7, and 11 had the
\Deleted flag set.  See the description of the EXPUNGE
response for further explanation.

===

This is logged by cyrus in the same way you have seen for pop3  
But can happen

multiple times in the same imap session.


3.  depending of your "expunge_mode" setting in imapd.conf the mails  
deleted in step 2

are still on your disk and are deleted by cyr_expire at a later time.

you can see and restore the mails that have been EXPUNGE by the client,
but not deleted by cyr_expire with the "unexpung" command see man page
for more details

Deleting mails form disk by cyr_expire was logged for each  
mailbox in cyrus 2.3

and 2.4 in cyrus 3.0 it is logged only once per cyr_expire run.
restoring mails with unexpung is also logged by cyrus.

unexpunge[363]: restored 1 expunged messages in mailbox 'user/userID'


Regarding POP3 the messages Deleted in the Session are marked as  
deleted and are expunged

automatically with the quit command by the client at the end of the session.




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Missing Email & Folders

[Michael Menge]

Hi,

Quoting Robert Covell :


If you suspect this is due to a client related problem, you could
enable telemetry logging to find out who/what is causeing the emails to go

missing.


https://www.cyrusimap.org/imap/reference/faqs/o-telemetry.html


Good idea will turn this on.



If the purpose is to (mostly) copy emails into the folder and rarely
delete, you could restrict delete access to a specific account via ACL.

https://www.cyrusimap.org/imap/reference/admin/access-control/rights-
reference.html


Believe it is setup like this, been awhile since I did it. Will confirm.

Also forgot to say the one account is ~90GB if that makes any difference...

-Bob

(sent twice)



We have seen the same kind of problems with Outlook with much smaller  
mailboxes.
I didn't have the time to debug it. So fare I suspected that some Data  
in the Outlook

profile got somehow corrupted.

Regards,

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: squatter not used after upgrade to cyrus 3.0.8

[Michael Menge]

Hi,

Quoting Michael Menge :


Hi,

Quoting Albert Shih :


Le 17/09/2018 à 14:01:52+0200, Michael Menge a écrit

Hi,

we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some initial
problems
which we could fix cyrus imapd 3.0.8 is running stable. The one remaining
problem
we receive reports about is, that the search is not working/too slow.

First we discovered that one of the options for Squatter are not supported
anymore, "-s Skip mailboxes whose index file is older than their current
squat file (within a small time delta)." and that squatter does not like
"-r" in combination with the source "user"

  > squatter -C /etc/imapd_be.conf -r  user
  fatal error: Internal error: assertion failed: lib/cyrusdb_twoskip.c:
2339: keylen


But after reindexing all mailboxes the search is still slow. I tried to
debug this and
found with strace that cyrus didn't try to open the cyrus.squat  
file for the

mailbox.

I suspect that I am missed some configuration change. So here is our
imapd.conf for our backends


If I'm correct you need :

 search_fuzzy_always: on

in your config.

If you can tell me if it's work...I would really appreciate. Because I
activated that but I'm not able to see if it's work really.


Thanks for your help.

I did tried it on my test server and it seems to be a bit faster,
but that could be due to caching. Strace still didn't show any access
to the cyrus.squat file.

For information: We only use squatter. No Xapia. And we had much faster
search with Cyrus-Imapd 2.3.x and 2.4.x. I don't have the timings form
the old system but our users are complaining and they receive timeouts
in our horde/imp webmailer, which they did't receive before.

Any other ideas are appreciated.


I still have the problem that search in cyrus imap 3.0.8 with search engine
squatter is slow compared to 2.4.20. I try to figure out if the squatter
search engine is working in cyurs imapd 3.0 and I messed up my configuration,
or if my configuration should work but squatter is broken.

I did set up a test environment to compare the old and new versions.
To verify that the search is indeed slower with 3.0.8

I used two different searches 'B SEARCH TEXT "Test"' and 'B SEARCH  
HEADER X-comment Unirundmail'


=== 2.4.20 === SEARCH TEXT

A SELECT INBOX
* 4321 EXISTS
* 4321 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1540372444] Ok
* OK [UIDNEXT 93369] Ok
* OK [HIGHESTMODSEQ 2] Ok
* OK [URLMECH INTERNAL] Ok
A OK [READ-WRITE] Completed
B SEARCH TEXT "Test"
* SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23  
24 25 26 27 28 29 30 31 33 34 35 36 37 38 39


4292 4294 4295 4296 4298 4299 4300 4301 4303 4306 4307 4309 4310 4315  
4316 4317 4318 4321

B OK Completed (1996 msgs in 0.690 secs)


=== 3.0.8 === SEARCH TEXT

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk $Forwarded  
$mdnsent $label1 $label2 $label3 hordetest testflag)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk  
$Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag \*)] Ok

* OK [UNSEEN 3737] Ok
* OK [UIDVALIDITY 1238498084] Ok
* OK [UIDNEXT 93373] Ok
* OK [HIGHESTMODSEQ 98491] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
a OK [READ-WRITE] Completed
B SEARCH TEXT "Test"
* SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23  
24 25 26 27 28 29 30 31 33 34 35 36 37 38 39


4274 4275 4276 4277 4278 4279 4285 4286 4287 4288 4292 4294 4295 4296  
4298 4299 4300 4301 4303 4306 4307 4309 4310 4315 4316 4317 4318 4321

B OK Completed (1935 msgs in 2.580 secs)

 2.4.20 === SEARCH HEADER

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1540372444] Ok
* OK [UIDNEXT 93369] Ok
* OK [HIGHESTMODSEQ 2] Ok
* OK [URLMECH INTERNAL] Ok
a OK [READ-WRITE] Completed
b SEARCH HEADER X-comment Unirundmail
* SEARCH 4283 4291 4313 4319 4320
b OK Completed (5 msgs in 0.020 secs)

 3.0.8 === SEARCH HEADER

a SELECT INBOX
* 4321 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk $Forwarded  
$mdnsent $label1 $label2 $label3 hordetest testflag)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk  
$Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag \*)] Ok

* OK [UNSEEN 3737] Ok
* OK [UIDVALIDITY 1238498084] Ok
* OK [UIDNEXT 93373] Ok
* OK [HIGHESTMODSEQ 98491] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
a OK [READ-WRITE] Completed
b SEARCH HEADER X-comment Unirundmail
* SEARCH 4283 4291 4313 4319 4320
b OK Completed (5 msgs in 0.370 secs)

===

There is also a big discrepancy between time indicated in the "OK  
Completed" and the 

Re: squatter not used after upgrade to cyrus 3.0.8

[Michael Menge]

Hi,

Quoting Albert Shih :


Le 17/09/2018 à 14:01:52+0200, Michael Menge a écrit

Hi,

we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some initial
problems
which we could fix cyrus imapd 3.0.8 is running stable. The one remaining
problem
we receive reports about is, that the search is not working/too slow.

First we discovered that one of the options for Squatter are not supported
anymore, "-s Skip mailboxes whose index file is older than their current
squat file (within a small time delta)." and that squatter does not like
"-r" in combination with the source "user"

   > squatter -C /etc/imapd_be.conf -r  user
   fatal error: Internal error: assertion failed: lib/cyrusdb_twoskip.c:
2339: keylen


But after reindexing all mailboxes the search is still slow. I tried to
debug this and
found with strace that cyrus didn't try to open the cyrus.squat file for the
mailbox.

I suspect that I am missed some configuration change. So here is our
imapd.conf for our backends


If I'm correct you need :

  search_fuzzy_always: on

in your config.

If you can tell me if it's work...I would really appreciate. Because I
activated that but I'm not able to see if it's work really.


Thanks for your help.

I did tried it on my test server and it seems to be a bit faster,
but that could be due to caching. Strace still didn't show any access
to the cyrus.squat file.

For information: We only use squatter. No Xapia. And we had much faster
search with Cyrus-Imapd 2.3.x and 2.4.x. I don't have the timings form
the old system but our users are complaining and they receive timeouts
in our horde/imp webmailer, which they did't receive before.

Any other ideas are appreciated.


Regards

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

squatter not used after upgrade to cyrus 3.0.8

[Michael Menge]

Hi,

we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some  
initial problems
which we could fix cyrus imapd 3.0.8 is running stable. The one  
remaining problem

we receive reports about is, that the search is not working/too slow.

First we discovered that one of the options for Squatter are not  
supported anymore, "-s Skip mailboxes whose index file is older than  
their current squat file (within a small time delta)." and that  
squatter does not like  "-r" in combination with the source "user"


   > squatter -C /etc/imapd_be.conf -r  user
   fatal error: Internal error: assertion failed:  
lib/cyrusdb_twoskip.c: 2339: keylen



But after reindexing all mailboxes the search is still slow. I tried  
to debug this and
found with strace that cyrus didn't try to open the cyrus.squat file  
for the mailbox.


I suspect that I am missed some configuration change. So here is our  
imapd.conf for our backends



=== Beginq Imapd.conf =
servername: ma0X.mail.localhost
configdirectory: /srv/cyrus-be
partition-default: /srv/cyrus-be
partition-ssd: /srv/cyrus-be/ssd-part
metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
metapartition_files: header index cache expunge squat annotations lock  
dav archivecache

archivepartition-ssd: /srv/cyrus-hdd-be/archive/ssd-part
archive_enabled: 1
## archive_days = 4*365
archive_days: 1460

proc_path: /srv/tmpfs/proc-be
mboxname_lockpath: /srv/tmpfs/lock-be
defaultpartition: ssd
admins: XXX

mupdate_server: mupdate.mail.localhost
mupdate_port: 3905
mupdate_authname: XXX
mupdate_password: XXX
proxy_authname: XXX
proxy_password: XXX
proxyservers: XXX

allowusermoves: 1
allowallsubscribe: 1
sync_host: sl0X.mail.localhost
sync_authname: XXX
sync_password: XXX
sync_port: 2005
guid_mode: sha1
sync_log: 1
sync_shutdown_file: /srv/cyrus-be/sync/shutdown

sievedir: /srv/cyrus-be/sieve
sieve_extensions: fileinto reject vacation imapflags notify include  
envelope body relational regex subaddress copy

sieve_maxscriptsize: 150

sasl_pwcheck_method: saslauthd
sasl_mech_list: plain login

allowanonymouslogin: no
reject8bit: no
quotawarn: 90
timeout: 45
poptimeout: 10
dracinterval: 0
drachost: localhost
lmtp_over_quota_perm_failure: 1
altnamespace: 1
#flushseenstate: 1
unixhierarchysep: 1
hashimapspool: 1
fulldirhash: 1
duplicatesuppression: 0
expunge_mode: delayed
delete_mode: delayed
deletedprefix: DELETED
anysievefolder: 1
quota_db: quotalegacy
subscriptions_db: flat
xlist-sent: Mail.sent
xlist-trash: Mail.trash
xlist-drafts: Mail.drafts
xlist-junk: Mail.v-spam
xlist-spam: Mail.v-spam
specialusealways: 1
syslog_prefix: be

tls_server_cert: XXX
tls_server_key: XXX
tls_client_ca_file: XXX
tls_ciphers: XXX
tls_prefer_server_ciphers: 1

auditlog: 1
reverseacls: 1
search_engine: squat
delete_unsubscribe: 1
statuscache: 1

=== End Imapd.conf =


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: master-master replication

[Michael Menge]

@.conf"
}

SERVICES {
   @@TYPE@@syncserver  cmd="sync_server -C /etc/imapd_@@TYPE@@.conf"  
listen="@@HOSTNAME@@:csync" prefork=1 maxfds=1024
   @@TYPE@@imapcmd="imapd -U 50 -C /etc/imapd_@@TYPE@@.conf"  
listen="@@HOSTNAME@@:imap" prefork=1 maxfds=1024

}

EVENTS {
   @@TYPE@@checkpointcmd="ctl_cyrusdb -c -C  
/etc/imapd_@@TYPE@@.conf" period=30
   @@TYPE@@delprune  cmd="cyr_expire -E 3 -X 60 -D 60 -C  
/etc/imapd_@@TYPE@@.conf" at=0100

}

===


 Configuration for Backend/Failover Instance
# Template MD5SUM: @@MD5SUM@@
servername: @@HOSTNAME@@
configdirectory: /srv/cyrus-@@TYPE@@
partition-default: /srv/cyrus-@@TYPE@@
partition-ssd: /srv/cyrus-@@TYPE@@/ssd-part
metapartition-ssd: /srv/cyrus-ssd-@@TYPE@@/meta/ssd-part
metapartition_files: header index cache expunge squat annotations lock  
dav archivecache

archivepartition-ssd: /srv/cyrus-hdd-@@TYPE@@/archive/ssd-part
archive_enabled: 1
proc_path: /srv/tmpfs/proc-@@TYPE@@
mboxname_lockpath: /srv/tmpfs/lock-@@TYPE@@
defaultpartition: ssd
admins: XXX

mupdate_server: @@MUPDATEHOSTNAME@@
mupdate_port: 3905
mupdate_authname: XXX
mupdate_password: XXX
proxy_authname: XXX
proxy_password: XXX
proxyservers: XXX

allowallsubscribe: 1

sync_host: @@SYNCHOST@@
sync_authname: XXX
sync_password: XXX
sync_port: 2005
guid_mode: sha1
sync_log: 1
sync_shutdown_file: /srv/cyrus-@@TYPE@@/sync/shutdown

sievedir: /srv/cyrus-@@TYPE@@/sieve
sieve_extensions: fileinto reject vacation imapflags notify include  
envelope body relational regex subaddress copy

sieve_maxscriptsize: 150

syslog_prefix: @@TYPE@@

== Imapd Replic ===
# Configuration for Slave (Replica) Instance
# Template MD5SUM: @@MD5SUM@@
servername: @@HOSTNAME@@
configdirectory: /srv/cyrus-@@TYPE@@
partition-default: /srv/cyrus-@@TYPE@@
partition-ssd: /srv/cyrus-@@TYPE@@/ssd-part
metapartition-ssd: /srv/cyrus-ssd-@@TYPE@@/meta/ssd-part
metapartition_files: header index cache expunge squat annotations lock  
dav archivecache

archivepartition-ssd: /srv/cyrus-hdd-@@TYPE@@/archive/ssd-part
archive_enabled: 1

proc_path: /srv/tmpfs/proc-@@TYPE@@
mboxname_lockpath: /srv/tmpfs/lock-@@TYPE@@
defaultpartition: ssd
admins: XXX

allowusermoves: 1
allowallsubscribe: 1

proxy_authname: XXX
proxy_password: XXX
proxyservers: XXX

sievedir: /srv/cyrus-@@TYPE@@/sieve
sieve_extensions: fileinto reject vacation imapflags notify include  
envelope body relational regex subaddress copy

sieve_maxscriptsize: 150

sasl_pwcheck_method: saslauthd
sasl_mech_list: plain login
allowanonymouslogin: no
syslog_prefix: @@TYPE@@
=

The sync client is started as own service

I hope it helps

Regards

   Michael


Best regards.

Четверг, 13 сентября 2018, 13:22 +05:00 от Michael Menge  
:


Hi,

This setup is NOT SUPPORTED and WILL BREAK if the replication process
is triggered
from the wrong server (user is active on both servers, user switched
from one server
to the other while the sync-log file is still processed, after split
brain) and
some mailboxes have been subscribed, renamed created deleted.

Also there is the risk of a race condition with subscriptions, if a
user subscribes
to multiple folders, the first will trigger a sync from A to B, but as
the folder
is subscribed on B it will trigger a sync from B to A, witch can  
undo the next

folder subscription.

These are only some cases that came to my mind. There will be more
cases and it
will be hard to debug. So DON'T DO IT!

What we do is, that we have distributed our users between multiple
instances, and each server is running one instance as master and one other
as replic. In case of failure or maintenance we stop the master  
instance, and

promote the corresponding replic and configure them so that they will sync
them back. If the old master is up to date we switch them back.

We use cyrus aggregator aka cyrus murder, and AFAIK fastmail also uses
multiple
instances on one server with nginx frontends

Regards,

Michael





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Re[3]: master-master replication

[Michael Menge]

Hi,

This setup is NOT SUPPORTED and WILL BREAK if the replication process  
is triggered
from the wrong server (user is active on both servers, user switched  
from one server
to the other while the sync-log file is still processed, after split  
brain) and

some mailboxes have been subscribed, renamed created deleted.

Also there is the risk of a race condition with subscriptions, if a  
user subscribes
to multiple folders, the first will trigger a sync from A to B, but as  
the folder

is subscribed on B it will trigger a sync from B to A, witch can undo the next
folder subscription.

These are only some cases that came to my mind. There will be more  
cases and it

will be hard to debug. So DON'T DO IT!

What we do is, that we have distributed our users between multiple
instances, and each server is running one instance as master and one other
as replic. In case of failure or maintenance we stop the master instance, and
promote the corresponding replic and configure them so that they will sync
them back. If the old master is up to date we switch them back.

We use cyrus aggregator aka cyrus murder, and AFAIK fastmail also uses  
multiple

instances on one server with nginx frontends

Regards,

   Michael





Quoting Evgeniy Kononov via Info-cyrus :


Sorry! Previous message was sent by mistake.

For example, I can configure both servers as follows.

Server A.
-
/etc/cyrus.conf
START {
...
syncclient       cmd="sync_client -r"
...
}
SERVICES {
...
syncserver       cmd="sync_server" listen="csync"
...
}

/etc/imapd.conf
...
sync_host: SERVER-B
sync_authname: admin
sync_password: password
sync_log: 1
sync_repeat_interval: 30
sync_timeout: 600
sync_shutdown_file: /var/lib/imap/syncstop And the same on server B.
-
/etc/cyrus.conf
START {
...
syncclient       cmd="sync_client -r"
...
}
SERVICES {
...
syncserver       cmd="sync_server" listen="csync"
...
}

/etc/imapd.conf
...
sync_host: SERVER-A
sync_authname: admin
sync_password: password
sync_log: 1
sync_repeat_interval: 30
sync_timeout: 600
sync_shutdown_file: /var/lib/imap/syncstop
Both server will be as master and as slave in one time.

Will there be any problems with this configuration?
Thank you. --
Evgeniy Kononov





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: master-master replication

[Michael Menge]

Hi,



Quoting Evgeniy Kononov via Info-cyrus :


Hello!

I have two servers with  
cyrus-imapd cyrus-imapd-2.5.8-13.3.el7.centos.kolab_16.x86_64

One server as master and second as replica.
All worked fine when users login on master server, but when I  
temporary move users on replica I found some trouble
Messages synchronisation from replica to master goes fine if  
sync_client sees a mismatch on the master, but if user create folder  
on replica it isn't sync on master.
Instead of it folder is unsubscribes from the master server and  
removed from both server


grep UNSUB maillog
Sep 10 13:31:41 master sync_client[1456]: UNSUB u...@example.com  
example.com!user.user.foldername


Why is it happend ?
When I tried manual sync from replica to master server, folder was  
subscribed. Is it possible that both servers will be master and  
replica in same time.




Cyrus does not support master-master replication. Because of
CONDSTORE (https://tools.ietf.org/html/rfc4551) Cyrus is able
to handle messages on a master-master setup, but the information
about folder operations is not tracked and so cyrus is unable to
distinguish is a folder was subscribed on one server, or if a folder
was unsubscribed on the other. Same is true for folder
creation/rename/deletion

The master-master replication is a wanted feature but even the 3.0.x
does not support it. I am not sure about the upcoming 3.1.x or master
branch.


Regards,

   Michael




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Subscriptions and sieve filters lost after upgrade to Cyrus 3.0

[Michael Menge]

Hi,


Quoting Martin Svec :


Hi Ellie,

Dne 19.7.2018 v 4:57 ellie timoney napsal(a):

Hi,

This was an lmtpd bug in earlier versions of 3.0.  It was fixed in  
3.0.6, and this was noted in the release notes:


https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.6.html:
Fixed Issue #2303: lmtpd(8) now looks for sieve scripts in the  
same directories as the rest of cyrus (including timsieved(8)).
Please note that if you had manually installed sieve scripts to  
the directories lmtpd used to look in, these will need to be moved.
It sounds like your setup was taking the bug into account (using  
the '^' paths that lmtpd expected), and since lmtpd has been fixed  
to use dots, you're now seeing the reverse of the problem.


I don't think #2303 is related to our problem. Issue #2303 was a  
sieve path inconsistency between
LMTP and the rest of Cyrus between 3.0.0-beta2 and 3.0.6. We've  
upgraded directly from 2.5.7 to

3.0.6. Also, in our case subscription file names are affected too.



You should be able to rename the name^with^dots paths to  
name.with.dots and things will work normally for you.


Of course, we already renamed them using a simple script. However,  
I'm curious if this breakage
between 2.5 and 3.0 series is well known and documented. Let me  
repeat again: all subscription and
sieve paths in previous Cyrus versions had dots translated to "^"  
for years. Since 3.0 Cyrus uses
"." instead of "^" and so the upgrade leads to missing subscriptions  
and sieve scripts. This is a
significant incompatible change that should be documented in 3.0  
Upgrade Guide. I'm also afraid if

there're no more similar "silent" changes that we didn't find yet.



Just to make sure, you have seen


Important config options: unixhierarchysep: and altnamespace:  
defaults have changed in imapd.conf(5). Implications are outlined in  
the Note in User Namespace Mode and Switching the Alternative  
Namespace. Please also see “Sieve Scripts,” below.


   unixhierarchysep: on
   altnamespace: on


https://www.cyrusimap.org/3.0/imap/download/upgrade.html#copy-config-files-and-update




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Help! Cyrus 2.5.11 segmentation fault

[Michael Menge]

Hi,


Quoting Marco Chesi :


Hello,
we have a Cyrus 2.5.11 on Debian 7 using murder (2 frontends, 3  
backends, 1 mupdate, about  5000 mailboxes) in production  
environment, hosted by a VMware cluster 6.5.


Suddenly, ALL mailboxes have becomed inaccessible.

In the log, we found many messages like this:

   master[5965]: process type:SERVICE name:imaps  
path:/usr/cyrus/bin/proxyd age:82.255s pid:5987 signaled to death by  
signal 11 (Segmentation fault)


I tried many times to restart the cyrus service on all servers  
without success.


Users are authenticated correctly, error occurs when they try to  
access the mailbox (trying with a manual connection using telnet,  
client dies on a "SELECT INBOX" command after a successfull login)


This is the output of one core dump:

    Core was generated by `proxyd -s'.
Program terminated with signal 11, Segmentation fault.
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:32
32    ../sysdeps/x86_64/multiarch/../strlen.S: File o directory  
non esistente.

(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:32
#1  0x7f58caffa656 in xstrdup (str=0x0) at lib/xmalloc.c:95
#2  0x7f58cb574b93 in parse_capability (s=s@entry=0x2223580,  
str=) at imap/backend.c:282
#3  0x7f58cb576362 in parse_capability (str=,  
s=0x2223580) at imap/backend.c:279
#4  backend_login (noauth=6486368, auth_status=0x0, cb=0x2233b20,  
userid=0x2235360 "", ret=0x2223580) at imap/backend.c:833
#5  backend_connect (ret_backend=ret_backend@entry=0x0, server=0xc  
, server@entry=0x2233d90 "imap2",  
prot=0x7ffccb275a40, prot@entry=0x62f840,
    userid=userid@entry=0x2235360 "", cb=0x2233b20,  
cb@entry=0x0, auth_status=auth_status@entry=0x0,  
logfd=logfd@entry=-1) at imap/backend.c:1027
#6  0x00425915 in proxy_findserver (server=0x2233d90  
"imap2", prot=prot@entry=0x62f840, userid=0x2235360 "",  
cache=0x62fb70, current=0x62fb78, inbox=0x62fb80, clientin=0x21ed470)

    at imap/proxy.c:173
#7  0x00417a0d in proxy_findinboxserver (userid=out>) at imap/imap_proxy.c:129
#8  0x00412f63 in cmd_list (tag=0x221ad00 "6",  
listargs=listargs@entry=0x7ffccb276a60) at imap/imapd.c:6728

#9  0x00420afc in cmdloop () at imap/imapd.c:1638
#10 0x00424f4e in service_main (argc=,  
argv=, envp=envp@entry=0x7ffccb27a9e0) at  
imap/imapd.c:984
#11 0x00416962 in main (argc=,  
argv=, envp=0x7ffccb27a9e0) at master/service.c:606

(gdb)

Any suggestions or support?




As far as i can tell is that the frontend imapd-proxy process is  
receiving the SIGSEGV
after it connected and authenticate to the backend. The imapd-proxy is  
parsing the

the capability output of the backend.

What is strange is that the parse_cpability is called with a NULL  
pointer (str=0x0)

which is then used in xstrdup.

So a check to ensure that str != NULL  would help to prevent the  
SIGSEGV, but a am

not sure what should be done in the str == NULL case.


The other question is why is str == NULL in the first place?
Can you try to manual connect to the backend using telnet, and check  
the CAPABILITY output.



And as always the frustrating question: Did something change on your backends?

Regards,

   Michael Menge


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: all messages in inbox have been expunged...any idea how this happened ?

[Michael Menge]

Hi,


Quoting Chentao Credungtao via Info-cyrus :


Hi,

Today a user complained all his messages in the inbox have disappeared

I checked the log and saw this (I replaced domain and user by  
example.com and jdoe) :


Jul  9 16:42:49 mail cyrus/imaps[14116]: client id:  
"x-originating-ip" "192.168.0.217, 10.10.20.12"
Jul  9 16:42:50 mail cyrus/imaps[14116]: Rename:  
example.com!user.jdoe -> example.com!user.jdoe.
Jul  9 16:42:50 mail cyrus/imaps[14116]: Expunged 2149 messages from  
example.com!user.jdoe


Any idea how this could happen ? The client is a SOGo webmail



I suspect that "Rename:" line is cut off. It looks like the user tried to
move his INBOX, to an folder (unintended drag'n'drop). As the user can  
not delete his own INBOX,
this results in an new folder and in moving the mails from the inbox  
to that new folder.


PS : I easily restored all email with "unexpunge -a", so no harm  
done, but I'd really like to understand what happened...


yes, delayed expunge mode is a great help for admins. Don't want to  
work without it.




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Monitoring cyrusimapd

[Michael Menge]

Hi,

Quoting Albert Shih :


Hi everyone,

I would like to know what kind of monitoring you perform on a cyrus-imapd.
Beside classic (check_imap, check_disk, check_cpu etc...) do you have any
special thing to monitor about cyrus-imapd.


In addition to the normal System monitoring (CPU-, Memory and Disk usage and
the Checks if tie services are running, we monitor the Cyrus logs for
Critical errors like IOERROR, or Mupdate commit errors.


For example do you launch any check on each database ? How can I  
check if the replication work fine ?



No database checks.

For Replication we check the size of the "sync" folder to see if the rolling
replication log is growing over 1 MB or if average size of the logs was over
500 KB for more than 15 min




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Restart from....? (DRP)

[Michael Menge]

Hi,

Quoting Albert Shih :


Hi everyone

I've a question about DRP (Disaster Recovery Plan), what's the easiest (=
fastest) way to rebuild a server (with the data) after a server «  
disappear » (fire,

water flood, etc.).

I see three way to « backup »  the data :

  Replication,

  Backup service (inside cyrusimapd 3),

  Filesystem backup (whatever the technic)

For replication my concern is the speed of the replication, the main server
(I got only one server) got lots of RAM, got SSD, and SAS disk, the
replication got SATA disks (lots of RAM too). When I check I think
everything are indeed replicated on the « slave » but with some delays
(1/2 days).



We have distributed our users across 6 (virtual) servers in an cyrus  
2.4 murder setup. The servers
are grouped in pairs, so that one is running on hardware in one  
building and the other in the other.
On each server there are 3 cyrus instances running, one frontend, and  
one backend and one replic.


In case of disaster, or planed maintenance we will start the replic as  
normal backend (we use service
ip addresses for each backend and move this ip to the other server so  
we don't have to update the

mupdate master mailbox.db.


The rolling replication is able to keep up. So normally there is only  
a small delay (2-5 Secs). If there
is a traffic peak (many newsletters) it may take up to 1-2h. I have  
only seen longer delays in case of a
corrupt mailbox where the replication bailed out. We are monitoring  
the size of the replication log.


We have ~ 41000 Accounts ~13.5 TB Mails, The VMs are running in an  
RHEV System.
Each Server has 20 GB Ram, 8 CPU-Cores, the Mails are stored on  
EUROstor iSCSI Systems with SATA disks
Recently we migrated the metadata onto a new EUROstor iSCSI System  
with SSDs. At the moment we plan to migrate
to Cyrus 3.0 to use archive partitions so that the recent mails will  
be stored on a iSCSI System with SAS disks,

and the older mails will be moved to the old iSCSI system with SATA disks

In addition to the disaster recovery plan we use "expunge_mode:  
delayed" and "delete_mode: delayed" and normal file based backup for  
the "I deleted my very importent mail by accident" use case.



Regards

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: IMAPD Missing quota root

[Michael Menge]

Hi,


Quoting "James B. Byrne via Info-cyrus" :


We have a problem with quota settings on a new v3.0.5 service. The
mailboxes were transferred via rsync from oldserver:/var/spool/imap/
to newserver:/var/spool/imap

The mailboxes.db was unloaded on the old server using mboxelist,
transferred to the new server, and then loaded into an empty
mailboxes.db.



Did you also rsync the config dir (see man imapd.conf configdirectory:)?
It is often set to /var/lib/imap





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Moving mailboxes from CIMAP-2.3.16 to CIMAP-3.0.5

[Michael Menge]

Hi,

Quoting "James B. Byrne via Info-cyrus" :


We are in the process of migrating our mail store from CentOS-6.9
running Cyrus-IMAPD-2.3.16 to FreeBSD-11.1 running Cyrus-IMAPD-3.0.5.

I am observing several behaviours that strike me a odd and which
likely indicate that something is wrong with my procedure.

I have tried moving things over using imapsync-1.836 and while this
appears to work we end up with vastly enlarged user mailbox sizes.
Also I cannot see the quota setting for most users although a few are
reported.

I then restarted the process and used rsync natively with --hard-links
in order to see if that reduced the disc space. Following the rsync of
all the user mailboxes I re-initialised

I have run 'reconstruct -fr user' and rebuilt mailboxes.db.  This is
what I get from running 'quota -f user' thereafter:


[root@inet17 /var/spool/imap]#  sudo -u cyrus
/usr/local/cyrus/sbin/quota -f user
   Quota   % Used Used Resource Root
[root@inet17 /var/spool/imap]#

At a guess this means that all of the user mailbox quota settings are
gone.  However,  I would expect the usage to be shown. And is there
not a default quota?


did you rsync the configdirectory too, or only /var/spool/imap ?





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Synchro for migration

[Michael Menge]

Quoting Albert Shih :



It seems like the following imapsync options allow you to use proxy
authentication

--authmech1 --authmech2
--authuser1 --authuser2
--proxyauth1 --proxyauth2


I'm not sure to understand how proxyauth works. I google it and find lots
of doc about...proxy imap.

Currently I need to authenticate my user against a LDAP(openldap),  
so I configure

saslauthd with ldap and works perfectly.

So one of my solution is to have a getpwent (local passwd) with real login
and fake password. And after the migration switch to ldap auth.



This will work but it is not necessary.

imapsync --host1 oldserver.yourdomain --port1 143 --authmech1 PLAIN  
--proxyauth1
 --authuser1 DovecotAdmin --user1 UserYouWantToMigrate  
--password1 DovecotAdminPW
 --host2 newserver.yourdomain --port1 143 --authmech2 PLAIN  
--proxyauth2
 --authuser2 CyrusAdmin   --user2 UserYouWantToMigrate  
--password2 CyrusAdminPW


Depending on your setup you will need other settings as well, and you  
may want to use the

passfile options so that the adminpassword is not shown in the process list




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Synchro for migration

[Michael Menge]

Hi,

Quoting Albert Shih :


Hi everyone.

Totally new in the world of cyrusimapd, I'm trying to migrate my current
imap server from dovecot to cyrusimapd.



welcome to the list.


I'm still in the prototype stage.

I've ~ 4To of mailbox for ~ 2000 users.

I would like to know what's the best way to migrate all mailbox and
subscription.



Using IMAP is the best way I can think of to migrate the Mails.
Depending of the tool you use it might not be the fastest way but
you can migrate all information with out worrying about the on disk
formats. May years ago (~10), at the time we migrated form UW-Imap to Cyrus
we uses imapcopy (it was part of the UW-Imap library) because it
was much faster as imapsync aat that time.



I already try imapsync, but how can I do that without knowing the password
of my user ? I think they are something call master user in dovecot, but is
they are anything similar in cyrusimapd ? And if yes how I choose the user
name in imapsync ?



Cyrus allows admins to proxy authenticate for users, if the sasl mech
support it (PLAIN does suppot it, LOGIN does not)
https://www.cyrusimap.org/sasl/sasl/authentication_mechanisms.html

It seems like the following imapsync options allow you to use proxy  
authentication


--authmech1 --authmech2
--authuser1 --authuser2
--proxyauth1 --proxyauth2






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: annotation and PHP IMAP extension

[Michael Menge]

Hi,


Quoting Marco :


Hello,

 Cyrus-imapd make wide use of annotations.

I already have a PHP program to query many IMAP info about  
mailboxes. I would like to query annotations too.


But official PHP IMAP extension doesn't support getannotation or  
setannotation command:


   https://bugs.php.net/bug.php?id=67956

Really, now Cyrus 3 adopts METADATA commands, I suppose. Now I'm  
working with Cyrus 2.4.


Could you suggest me some PHP libraries to use these commands?


You can take a look at the horde project https://www.horde.org/
Part of this webmail/groupware is a imap library that supports many
IMAP extensions and features and sould be usable as standalone library,
but I have only used it as part of the webmail/groupware




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Mailboxes messed up after migration

[Michael Menge]

Hi,

Quoting Hiago Prata :


Sorry, forgot to mention I'm using Roundcube as client.

Em 2018-02-05 11:26, Hiago Prata escreveu:


Hi!

After the migration of all users I was supposed to migrate to a new  
server, some of them reported that in some of their mailboxes  
subdirectories the display order of the messages is all messed up,  
with messages received or sent months ago being shown before them  
most recent ones. I thought it might be date format or something  
related, but all dates of creation and access of all mailboxes look  
the same.


Any thoughts about that?



From what System (Cyrus / non-Cyrus, Version ) to what System  
(Version) did you migrate,

and how was the migration done? (rsync, scp, cyrus replication, via imap).

How are the mails sorted in roundcube (Date, UID)?


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: install certificate how to

[Michael Menge]

Hi,


Quoting Nikos Gatsis - Qbit :


Hello list
I have a mailserver which serve about 40 virutal domains and many users
per domain using cyrus-imapd-2.4.17-13.el7.x86_64 and
sendmail-8.14.7-5.el7.x86_64.
How can I install a certificate per domain? Is that possible?

Now I use what cyrus manual suggest:

imapd.conf:
...
tls_cert_file: /var/lib/imap/server.pem
tls_key_file: /var/lib/imap/server.pem
3tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
...



The problem with configuring multiple certificates in cyrus ist that at the
moment it would require using one IP for each domain and one imap(s)/pop(s)
service listeing only on this IP and configuring the certs and keys for
each of these service names


In /etc/cyrus.conf Services you would have

doaminaimap cmd="imapd" listen="ipa:imap"
domainaimaps cmd="imapd -s " listen="ipa:imaps"
domainbimap cmd="imapd" listen="ipb:imap"
domainbimaps cmd="imapd -s " listen="ipb:imaps"
...
domainzimap cmd="imapd" listen="ipz:imap"
domainzimaps cmd="imapd -s " listen="ipz:imaps"

and in /etc/imapd.conf

domainaimap_tls_cert_file: /var/lib/imap/domaina.pem
domainaimap_tls_key_file: /var/lib/imap/domaina.pem
domainaimaps_tls_cert_file: /var/lib/imap/domaina.pem
domainaimaps_tls_key_file: /var/lib/imap/domaina.pem
domainbimap_tls_cert_file: /var/lib/imap/domainb.pem
domainbimap_tls_key_file: /var/lib/imap/domainb.pem
domainbimaps_tls_cert_file: /var/lib/imap/domainb.pem
domainbimaps_tls_key_file: /var/lib/imap/domainb.pem
...
domainzimap_tls_cert_file: /var/lib/imap/domainz.pem
domainzimap_tls_key_file: /var/lib/imap/domainz.pem
domainzimaps_tls_cert_file: /var/lib/imap/domainz.pem
domainzimaps_tls_key_file: /var/lib/imap/domainz.pem

There is the SSL Extension SNI  
https://de.wikipedia.org/wiki/Server_Name_Indication
which would allow to using multiple certificates on one IP, but as far  
as i know

that is only implemented by webservers and browsers, but i could be wrong and
that the mail clients will use it because the ssl libraries use this  
extensions

by default.

A few years ago i have seen thread about SNI on this list
https://lists.andrew.cmu.edu/pipermail/info-cyrus/2014-July/thread.html#37461

Depending on how static your list of domains is you could also use one  
certificate

with 40 SubjectAlternativNames


Thank you in advance,
Nikos


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübinge


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Adding archiving to an existing Cyrus installation

[Michael Menge]

Hi,

Quoting Reinaldo Gil Lima de Carvalho <reinal...@gmail.com>:


I think that singleinstancestore (message hard links) will not survive when
moving from one partition to the other and storage total size will increase
significantly.



thanks for the hint. This was not a problem while migration to the  
meta-data partition,
as the mails stayed on the same partition (as in file-system and not  
cyrus-partition)

and only hardlinks where change at all.

So one more reason for an other migration path.




2017-11-03 12:22 GMT-03:00 Michael Menge <michael.me...@zdv.uni-tuebingen.de

:



Hi Nic,

Quoting Nic Bernstein <n...@onlight.com>:

Friends,

I have a client with Cyrus 2.5.10 installed.  Last year we migrated their
old 2.3.18 system to 2.5.10, with an eye towards an eventual move to
3.0.x.  Based on Bron's most excellent email of last year, ([Subject: Cyrus
database and file usage data] from Cyrus Devel of 8 January 2016) we used a
tiered layout for the storage:

The main categories are:

 * Config directory (ssd) [/var/lib/imap]
 o sieve
 o seen
 o sub
 o quota
 o mailboxes.db
 o annotations.db
 * Ephemeral [/var/run/cyrus -- in tmpfs]
 o tls_sessions.db
 o deliver.db
 o statuscache.db
 o proc (directory)
 o lock (directory)
 * Mailbox data [typical 2.5.X usage]
 o Meta-data (ssd)
 + header
 + index
 + cache
 + expunge
 + squat (search index)
 + annotations
 o Spool data (disk: raidX)
 + messages (rfc822 blobs)

We sized the Fast SSD pool (this is three-drive mirrors on ZFS) to be
extra large, so it could eventually handle "Hot" data, and left about 300GB
free there.  Data, on spinning media, is currently 5.74TB with 4.8TB free
(RAID10).  Metadata is 35GB and /var/lib/imap is 8GB, all of which is in
the Fast pool.

Now the client is ready to take the dive into v3.0, and I'm trying to
figure out how to put "archive" operation in effect.

I have read the documentation (hell, I wrote most of it) and understand
the settings, but what I cannot quite wrap my brain around is this: There
is already all of this data sitting in all of these data partitions (we use
a total of 34 separate partitions each for data & metadata) so how do I
make the transition to separate archive partitions, since all that data is
on the "slow" drives? Can I just reassign all of the current data
partitions to archivedata partitions, define the new set of "Hot" data
partitions on the Fast pool, and let 'er rip, or what?

I promise, if you tell me, I'll write it up as real documentation. :-)



We are interested in such a migration too. Our fallback plan, if we don't
find a
better way to do it is, do use the same method as we introduced the ssd
meta-data
partition.


1. We created a new partition in our cyrus configuration,
2. we moved moved the accounts from one partition to the other one by one.
3. (this will be new for the archive partition) run cyrus expire to move
the old mails back to the slow disks.

This method will have two downsides.
1. we have to copy all mails to the fast storage, and move the old mails
   back to the slow storage. So we have to move most of the mails twice.
2. the path of the old mail will change so they will be stored again in
   our file based backup

so a method without these downsides will be appreciated

Regards

   Michael



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Adding archiving to an existing Cyrus installation

[Michael Menge]

Hi Nic,

Quoting Nic Bernstein :


Friends,
I have a client with Cyrus 2.5.10 installed.  Last year we migrated  
their old 2.3.18 system to 2.5.10, with an eye towards an eventual  
move to 3.0.x.  Based on Bron's most excellent email of last year,  
([Subject: Cyrus database and file usage data] from Cyrus Devel of 8  
January 2016) we used a tiered layout for the storage:


The main categories are:

 * Config directory (ssd) [/var/lib/imap]
 o sieve
 o seen
 o sub
 o quota
 o mailboxes.db
 o annotations.db
 * Ephemeral [/var/run/cyrus -- in tmpfs]
 o tls_sessions.db
 o deliver.db
 o statuscache.db
 o proc (directory)
 o lock (directory)
 * Mailbox data [typical 2.5.X usage]
 o Meta-data (ssd)
 + header
 + index
 + cache
 + expunge
 + squat (search index)
 + annotations
 o Spool data (disk: raidX)
 + messages (rfc822 blobs)

We sized the Fast SSD pool (this is three-drive mirrors on ZFS) to  
be extra large, so it could eventually handle "Hot" data, and left  
about 300GB free there.  Data, on spinning media, is currently  
5.74TB with 4.8TB free (RAID10).  Metadata is 35GB and /var/lib/imap  
is 8GB, all of which is in the Fast pool.


Now the client is ready to take the dive into v3.0, and I'm trying  
to figure out how to put "archive" operation in effect.


I have read the documentation (hell, I wrote most of it) and  
understand the settings, but what I cannot quite wrap my brain  
around is this: There is already all of this data sitting in all of  
these data partitions (we use a total of 34 separate partitions each  
for data & metadata) so how do I make the transition to separate  
archive partitions, since all that data is on the "slow" drives? Can  
I just reassign all of the current data partitions to archivedata  
partitions, define the new set of "Hot" data partitions on the Fast  
pool, and let 'er rip, or what?


I promise, if you tell me, I'll write it up as real documentation. :-)



We are interested in such a migration too. Our fallback plan, if we  
don't find a
better way to do it is, do use the same method as we introduced the  
ssd meta-data

partition.


1. We created a new partition in our cyrus configuration,
2. we moved moved the accounts from one partition to the other one by one.
3. (this will be new for the archive partition) run cyrus expire to move
the old mails back to the slow disks.

This method will have two downsides.
1. we have to copy all mails to the fast storage, and move the old mails
   back to the slow storage. So we have to move most of the mails twice.
2. the path of the old mail will change so they will be stored again in
   our file based backup

so a method without these downsides will be appreciated

Regards

   Michael


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: messages lost in the mailbox

[Michael Menge]

Hi,

Quoting Domenico - OpenPC :


Hi,
I have an error occurred with cyrus 2.5.7_1, after a system crashed,  
a mailbox not show all message, but in the directory the files  
exist. I have tried "reconstruct" command but the issue persist.


Can someone help me please?



Are you using delayed expunge mode? And have you checked (with  
unexpunge -l) if the messages

have been deleted?

Regards

   Michael


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: I need to check if stored messages are well formed.

[Michael Menge]

Hi,

Quoting Alvin Starr <al...@netvel.net>:

Through a system crash and recovery I found a number of messages  
that were badly stored.


I am now concerned that I may have other bad messages.

Is there any tool/program that will check the message files  
themselves for things like correct headers and bodies?


it depends on your cyrus Version. I think since version 2.4 cyrus uses  
the checksums

of the message as GUID and stores it the header files.
"reconstruct -G -n" should do the trick. But take a look at the  
manpage for additional Options.



Regards

   Michael Menge


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Mailbox administration in cyrus

[Michael Menge]

Hi,



Quoting "Dr. Peer-Joachim Koch" :


Hi,

is there an esay way to remove ALL mailboxes of an user ?

I did not find any info about something like

dm -r user.OLDUSER



in cyradm

dm user.OLDUSER

should delete the user (all mailboxes and sievscripts), at least since  
cyrus 2.3


which version of cyrus are you using?


removing every folder within OLDUSER and OLDUSER as well.



also you can use wildcards in cyradm


Grüße aus Tübingen

   Michael


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Error in /var/log/maillog ...

[Michael Menge]

Hi,


Quoting "Walter H. via Info-cyrus" :


Hello,

on my CentOS6 I see this error

Jun 12 08:02:33 mail master[1941]: setrlimit: Unable to set file
descriptors limit to -1: Operation not permitted



This is a common error message that is logged on cyrus start/restart,
which is normally followed by a message like "retrying with 8192  
(current max)"



what is causing this?



Cyrus need to handle many open files/connections, so cyrus tries to unlimit
the number of file descriptors on start, which is not allowed on all systems
(depending on OS and/or configuration)



Thanks,
Walter


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus discrete murder issue : CREATE creates mail folder on frontend

[Michael Menge]

Quoting Tim Piessens :


Hi all,

I have created a murder cluster containing 1 update server, 1  
frontend and 2 backends.

Mail reading woks ok, also moving folders between backends.
However when a client creates a new folder, it gets created on the  
frontend and not on a backend.


Is this an Admin user? If a normal user creates a new folder it should  
be created on the same

backend+partition as the inbox of the user.


I am using cyrus-imapd version 2.5.10 from debian testing

# My frontend Configuration
configdirectory: /var/lib/cyrus
proc_path: /run/cyrus/proc
mboxname_lockpath: /run/cyrus/lock
mupdate_config: standard
default server:imapbe2.***


Options in imapd.conf don't contain space. Not sure if it matters in this case


server list:imapbe1.*** imapbe2.***


see above


partition-default: /var/spool/cyrus/mail
altnamespace: yes
unixhierarchysep: no
lmtp_downcase_rcpt: yes
admins: cyrus root murder
proxyservers: cyrus murder


quoting man imapd.conf
proxyservers: 

A list of users and groups that are allowed to proxy for other users,  
separated by spaces. Any user listed in this will be allowed to login  
for any other user: use with caution. In a standard murder this option  
should ONLY be set on backends. DO NOT SET on frontends or things  
won’t work properly.



proxy_authname: murder
allowanonymouslogin: no
popminpoll: 1
umask: 077
sendmail: /usr/sbin/sendmail
sieveusehomedir: false
sievedir: /var/spool/sieve
hashimapspool: true
allowplaintext: no
sasl_mech_list: GSSAPI PLAIN
allowapop: no
sasl_minimum_layer: 1
auth_mech: krb5
unix_group_enable: 1
sasl_pwcheck_method: saslauthd
sasl_auto_transition: no
mupdate_server: update.***
mupdate_username: murder
mupdate_authname: murder


any help is heavily appreciated.

Tim

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: lmtpd segfaults after update to 2.4.19, seems it wants to act as proxy

[Michael Menge]

Quoting Simon Matter :


Hi,

I'm resending this with new subject and one again attached the straces of
a delivery with 2.4.18 and 2.4.19.


I've just updated our cyrus-imapd rpms from 2.4.18 -> 2.4.19. All

seems well with just some small changes to the build.


However, I just can't get it to work: lmtpd segfaults on mail delivery.

The logs show something like this:
May 17 14:21:42 mh1 kernel: lmtpd[7433]: segfault at 5ebc24d7 ip

7f216292973b sp 7ffe5ebc1f58 error 4 in

lmtpd[7f21628c1000+169000]
May 17 14:21:42 mh1 master[7423]: process 7433 exited, signaled to death
by 11

First I've rebuilt without some of the custom patches in the package like
the autocreate patches. But without a change.

I've then straced an lmtpd during mail delivery, once with 2.4.18 and

once

with 2.4.19 in the hope to find something. The traces are attached.

I see that there were quite some change concerning lmtp, does anyone

of the developers have an idea where the problem in lmtpd changes
could be?


It turns out that lmtpd wants to act as proxy, because only then it

creates the spoolfile in /tmp/. But why?

Regards,
Simon



Are you running a murder setup? Can you produce a coredump and brovide a

backtrace with symbols loaded?

Hi,

I did two things meanwhile:

1) Built a completely vanilla cyrus-imapd-2.4.19 without any patches to
make sure it's not anything wrong with additional patches. The crash was
the same.

2) Ran the normal build with an unstripped "lmtpd" binary and produced
core and backtrace. So below:

Core was generated by `lmtpd'.
Program terminated with signal 11, Segmentation fault.
#0  strlcpy (dst=0x7ffaa64e00d0 "", src=0x5d73c6c7 , len=1024) at xstrlcpy.c:65
65  if ((dst[n] = src[n]) == '\0') break;
Missing separate debuginfos, use: debuginfo-install
cyrus-sasl-gssapi-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-md5-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-22.el6.x86_64
glibc-2.12-1.209.el6_9.1.x86_64 keyutils-libs-1.4-5.el6.x86_64
krb5-libs-1.10.3-65.el6.x86_64 libcom_err-1.41.12-23.el6.x86_64
libselinux-2.0.94-7.el6.x86_64 nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64
openssl-1.0.1e-57.el6.x86_64 pcre-7.8-7.el6.x86_64
tcp_wrappers-libs-7.6-58.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  strlcpy (dst=0x7ffaa64e00d0 "", src=0x5d73c6c7 , len=1024) at xstrlcpy.c:65
#1  0x7ffaa5150c2e in proxy_adddest (dlist=0x7ffc5d73c288,
rcpt=0x7ffaa64d26d0 "si...@imap.bi.corp.invoca.ch", rcpt_num=0,
server=0x5d73c6c7 , authas=0x0) at
proxy.c:81


here the bt is already in the wrong path and server has not been set
wich leads to the SIGSEV


#2  0x7ffaa514fbae in deliver (msgdata=0x7ffaa64d0950, authuser=0x0,
authstate=0x7ffc5d73c280) at lmtpd.c:749


This is the point where the wrong path is taken.


r = mlookup(namebuf, , NULL, NULL);
if (!r && server) {
/* remote mailbox */
proxy_adddest(, rcpt, n, server, authuser);
status[n] = nosieve;
}

mlookup did return something that is not 0, and server has not been set
in mlookup. In 2.4.17 there has was "if (server) *server = NULL;"

proxy_adddest sould only be called if the mailbox is a remote mailbox,
possible other cases are IMAP_IOERROR, IMAP_MAILBOX_RESERVED,  
IMAP_MAILBOX_MOVED

IMAP_MAILBOX_NONEXISTENT



Any help would be much appreaciated!






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 2.4.19 released

[Michael Menge]

Hi,

Quoting Simon Matter :


The Cyrus team is proud to announce the immediate availability of a new
version of Cyrus IMAP: 2.4.19.


I've just updated our cyrus-imapd rpms from 2.4.18 -> 2.4.19. All seems
well with just some small changes to the build.

However, I just can't get it to work: lmtpd segfaults on mail delivery.

The logs show something like this:
May 17 14:21:42 mh1 kernel: lmtpd[7433]: segfault at 5ebc24d7 ip
7f216292973b sp 7ffe5ebc1f58 error 4 in lmtpd[7f21628c1000+169000]
May 17 14:21:42 mh1 master[7423]: process 7433 exited, signaled to death
by 11

First I've rebuilt without some of the custom patches in the package like
the autocreate patches. But without a change.

I've then straced an lmtpd during mail delivery, once with 2.4.18 and once
with 2.4.19 in the hope to find something. The traces are attached.

I see that there were quite some change concerning lmtp, does anyone of
the developers have an idea where the problem in lmtpd changes could be?


It turns out that lmtpd wants to act as proxy, because only then it
creates the spoolfile in /tmp/. But why?
Regards,
Simon



Are you running a murder setup? Can you produce a coredump and brovide  
a backtrace with symbols loaded?


Michael





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Sieve RFC5490 checks user quota usage

[Michael Menge]

Hi,

Quoting Paolo Cravero :


And for Nic, yes, I mean the "IMAP STORAGE quota". I would like to warn the
user that his quota is about to fill up through an email, triggered on new
mail arrival or login. Why? Because not all clients support reading the
quota over IMAP or handling an alert (think of some smartphone IMAP client
or an (active)sync system).

Is there a way to achieve the same result somehow, with stock cyrus?



We have a daily cron-job which checks the quota usage and informs
the user via mail if more than 85% is used. The scripts limits the  mails
send to each user to one every week. An other mail is send if 100%
usage is reached.

Two other ways i can think of is parsing cyrus logs, or using the  
notify daemon




Michael


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mbox operations performance issue

[Michael Menge via Info-cyrus]

Hi,


Quoting Miguel Mucio Santos Moreira via Info-cyrus  
:



Dears,


I've had a problem for some days with mailboxes operations like sam, dam etc.
Those operations are too slow, I have some mailboxes with many  
subfolders when I applied a sam command to allow other user access  
it, the command spends too much time (e.g 20hours).

There's nothing weird in logs or etc.
We have murder environment with cyrus-.4.16-4 on Debian 7.



mailbox operations need to be acknowledged by/committed on the MUPDATE server.

Is the mupdate server still running?

Do you see connections from the backends to the mupdate on port 3905?

Can you create new mailboxes?

Did you try restarting cyrus on mupdate server


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: imap impersonate

[Michael Menge via Info-cyrus]

Quoting Gabriele Bulfon :


Thanks,
my imapd.conf has already :
admins: sonicle
sasl_mech_list: plain
if I try an imap session with:
A01 AUTHENTICATE PLAIN
+
xxx
where xxx comes from 'echo -en "\0sonicle\0pass" | base64' , I  
get authenticated as sonicle.

Now, how do I switch to the desired user?
Once I understand how to do it via imap protocol, I need to  
replicate it in java code through:

store.connect(host,143,user,pass);
Thanks in advance!
Gabriele


Quoting from https://tools.ietf.org/html/rfc4616


2.  PLAIN SASL Mechanism

  The mechanism consists of a single message, a string of [UTF-8]
  encoded [Unicode] characters, from the client to the server.  The
  client presents the authorization identity (identity to act as),
  followed by a NUL (U+) character, followed by the authentication
  identity (identity whose password will be used), followed by a NUL
  (U+) character, followed by the clear-text password.  As with
  other SASL mechanisms, the client does not provide an authorization
  identity when it wishes the server to derive an identity from the
  credentials and use that as the authorization identity.


so it is UserID\0AdminID\0AdminPass






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: imap impersonate

[Michael Menge via Info-cyrus]

Quoting Gabriele Bulfon via Info-cyrus :


Hi,
is there any mechanism with Cyrus imap to impersonate another user?
I've seen other imap servers scenarios where one may use plain  
authentication and sending user as mailboxuser plus a separator plus  
adminuser and use only adminpassword, to get access to the  
mailboxuser as is (dovecot, exchange).

Anything like this in Cyrus?
Gabriele


Cyrus can use the PLAIN mech to allow admin access as the user.
You need to add plain to sasl_mech_list in imapd.conf
And the "admin" account has to be listed in admins or proxyservers
in imapd.conf

Regards,

   Michael



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Problems with SSL

[Michael Menge via Info-cyrus]

Hi,


Quoting Infraestructura TIC - UNNOBA via Info-cyrus  
:



Hello!
I'm using cyrus on Debian vm for several years but now, SSL starts to fail:

Nov 29 13:05:58 server1 cyrus/imaps[9595]: inittls: Loading
hard-coded DH parameters
Nov 29 13:05:58 server1 cyrus/imaps[9595]: imaps TLS negotiation
failed: [2801:0:140:f42:f3fa:b0b2:4ab1:8d10]

I tried with self-signed certificates, and third-party ones, but the
result is the same.
I spent two days trying to figure out what happened, without results.

#openssl s_client -connect mail.server.test:993 -crlf -state
CONNECTED(0003)
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL3 alert read:fatal:handshake failure
SSL_connect:error in SSLv3/TLS write client hello
140019483313280:error:14094410:SSL routines:ssl3_read_bytes:sslv3
alert handshake failure:ssl/record/rec_layer_s3.c:1388:SSL alert number
40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 176 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)


I believe the server and client have no SSL/TLS version and/or Cipher  
in common and

therefore can't establish an encrypted connection.

Some time ago i found an ssl server test suite  
https://github.com/drwetter/testssl.sh
witch tries to do what https://www.ssllabs.com/ does for web servers  
but for all protocols

and server not reachable form the internet.

You might want to check your server with ./testssl.sh mail.server.test:993



Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol  : TLSv1.2
Cipher: 
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1480435442
Timeout   : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---


I'm using this versions:

cyrus-admin   2.5.10-2
cyrus-clients 2.5.10-2
cyrus-common  2.5.10-2
cyrus-doc 2.5.10-2
cyrus-imapd   2.5.10-2
cyrus-murder  2.5.10-2
cyrus-pop3d   2.5.10-2
cyrus-replication 2.5.10-2



Both, certificate and key, are accesibles by user cyrus. Certificate is
up-to-date.

This is the config:

$sudo -u cyrus /usr/lib/cyrus/bin/cyr_info  conf
[...]
tls_ciphers: TLSv1+HIGH:!aNULL:@STRENGTH
tls_client_ca_dir: /etc/ssl/certs
tls_client_ca_file: /etc/ssl/certs/cyrus.pem
tls_server_cert: /etc/ssl/certs/cyrus.pem
tls_server_key: /etc/ssl/private/cyrus.key
tls_session_timeout: 0
[...]


And before I declared myself "I'm completely lost", I was watching
entropy ... but is ok.

#cat /proc/sys/kernel/random/entropy_avail
2354



¿Any suggestions?

Thanks in advance!



Javier.-



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Bugs in cyrus murder 2.5 (was: Front End not forwarding requests to the backend)

[Michael Menge via Info-cyrus]

Hi,

Quoting Wolfgang Breyha via Info-cyrus <info-cyrus@lists.andrew.cmu.edu>:


Michael Menge via Info-cyrus wrote on 07/11/16 10:30:

Quoting Alberto Cardenas via Info-cyrus <info-cyrus@lists.andrew.cmu.edu>:


I'm trying to setup Cyrus Murder for testing and have three virtual
machines for this purpose (frontend, mupdate and backend). The three
machines are running Centos 6.5, SASL 2.1.23 and Cyrus 2.3.16



you might want to consider using a more recent cyrus Version. Cyrus 2.3.16
is about 7 years old. 2.5.10 is the latest stable, 3.0 will be coming out
soon. I don't think it has something to do with your problems but, there
have been many bugfixes and improvements since 2.3. And if you set up a
new system it would make things easier in the future.


For murder I recommend the latest 2.4.x release. 2.5 has still several issues
open with murder and I can not recommend using it without in deep knowledge
how to handle them (eg. maintain your own patches). 2.4 simply works compared
to 2.5.



Thanks for the info, we are also still running cyrus 2.4 on our  
production environment.
But mostly because that was the stable version at the time we migrated  
from stand alone
to murder setup and the "never touch a running system" mentality. I  
was not aware that

there was decreased stability for murder setup in the cyrus 2.5 version.

Do you have some specific bugs in mind?


Greetings Michael




Greetings, Wolfgang
--
Wolfgang Breyha <wbre...@gmx.net> | http://www.blafasel.at/
Vienna University Computer Center | Austria


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Sieve login issue. Please help.

[Michael Menge via Info-cyrus]

Hi,


Quoting Müfit Eribol via Info-cyrus :


Hello,

I am a happy user of cyrus-imapd for years without any major problem  
for  small user base.


Currently, I am having login problem for sieve. I have been trying  
to find the problem for days.


Please find below information about my configuration:

1. Installed software: cyrus-imapd-2.4.17, postfix-2.10.1,  
cyrus-sasl-2.1.26, cyrus-sasl-plain-2.1.26, cyrus-sasl-lib-2.1.26 on  
CentOS 7.


2. Authentication is done through saslauthd, pam and mysql.

3. pwcheck_method: saslauthd, mech_list: plain login

4. There is no problem with login to imapd or smtpd.

5. cyrus.conf

SERVICES {
imaplocal cmd="imapd -C /etc/imapd-local.conf"  
listen="127.0.0.1:imap" prefork=0

  imaps cmd="imapd -s" listen="imaps" prefork=1
imapslocalcmd="imapd -C /etc/imapd-local.conf"  
listen="127.0.0.1:imaps" prefork=0

  sieve cmd="timsieved" listen="sieve" prefork=0


You did not define an ip address here, so sieve will use 0.0.0.0:sieve

sievelocal  cmd="timsieved -C /etc/imapd-local.conf"  
listen="127.0.0.1:sieve" prefork=0


this will likely fail, as the "sieve" service above will is already  
listening on 0.0.0.0

and blocking 127.0.0.1


  lmtpunix  cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1
}

6. imapd.conf

postmaster: postmaster
configdirectory: /var/lib/imap
partition-default: /var/spool/imap
#admins: cyrus
allowanonymouslogin: no
allowplaintext: no
#tls_require_cert: 1
sasl_minimum_layer: 128
servername: mail.x.com
autocreatequota: 20
maxmessagesize: 0
reject8bit: 0
munge8bit: 0
quotawarn: 90
timeout: 30
poptimeout: 10
dracinterval: 0
drachost: localhost
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN
sievedir: /var/lib/imap/sieve
sieve_maxscriptsize: 32
sieve_maxscripts: 5
sieve_allowplaintext: 1
sendmail: /usr/sbin/sendmail
tls_cert_file: /etc/pki/tls/certs/imap.pem
tls_key_file: /etc/pki/tls/certs/imap.pem
tls_ca_file: /etc/pki/tls/certs/imap.pem

7. imapd-local.conf

postmaster: postmaster
configdirectory: /var/lib/imap
partition-default: /var/spool/imap
admins: cyrus
allowanonymouslogin: no
allowplaintext: yes
servername: mail.xx.com
autocreatequota: 100
maxmessagesize: 0
reject8bit: no
quotawarn: 90
timeout: 30
poptimeout: 10
dracinterval: 0
drachost: localhost
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN
sievedir: /var/lib/imap/sieve
sieve_maxscriptsize: 32
sieve_maxscripts: 5
sendmail: /usr/sbin/sendmail

8. shell:

[root@server ~]#  sieveshell -u user1 -a user1 localhost
connecting to localhost
unable to connect to server at /usr/bin/sieveshell line 170.

maillog:

Sep 22 10:34:45 server sieve[15050]: Lost connection to client -- exiting

9. shell:

[root@server ~]# telnet localhost sieve
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
"IMPLEMENTATION" "Cyrus timsieved v2.4.17-Fedora-RPM-2.4.17-8.el7_1"
"SASL" ""
"SIEVE" "comparator-i;ascii-numeric fileinto reject vacation  
imapflags notify envelope relational regex subaddress copy"

"STARTTLS"
"UNAUTHENTICATE"
OK

10. When I try to login using smartsieve

maillog:

Sep 22 10:38:32 server sieve[16029]: STARTTLS failed: localhost[127.0.0.1]



you are not connecting to sievelocal but to sieve and therefore  
"allowplaintext: no" from
imapd.conf is preventing auth:login and auth:plain from showing  
without usage of startls


I don't understand why STARTTLS is being called when connecting from  
localhost? Is it normal? Obviously, I am doing something wrong.





I would appreciate any help. Thank you.



Cheers,

   Michael


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Invalid mailbox name

[Michael Menge via Info-cyrus]

Quoting Paul van der Vlis via Info-cyrus <info-cyrus@lists.andrew.cmu.edu>:


Op 21-09-16 om 14:11 schreef Michael Menge via Info-cyrus:

Hi,

Quoting Paul van der Vlis via Info-cyrus <info-cyrus@lists.andrew.cmu.edu>:


Hello,

I am syncing many mailboxes from one IMAP server to another. Now I get
sometimes errors on mailbox names. I found out that Cyrus does not
accept brackets in a mailbox name. Is there documentation about what
characters are accepted in mailbox names??


The allowed ASCII-Chars are defined in the macro GOODCHARS in
imap/mboxname.c
(https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/mboxname.c#L1495).

non-ASCII-Chars are handled by RFC 3501 5.1.3.

This subject has been discussed a few years ago on this list, and GOODCHARS
has been changed between cyrus versions.

2.2:#define GOODCHARS "
+,-.0123456789:=@ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~"
2.3:#define GOODCHARS "
#$'+,-.0123456789:=@ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~"
Master: #define GOODCHARS "
#$'()*+,-.0123456789:=?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz~"


Thanks for your answer!

I am wondering about the dot. So far I know I cannot use it in a mailbox
name, but it is in the list.



I suspect that your cyrus is configured to use the . as hierarchy seperator.
see "unixhierarchysep:" in imapd.conf manpage for details.


And what's "master" exactly? So far I see, I cannot use e.g. brackets in
a mailbox name.


Master is the name of the main development branch in git. A new branch for
cyrus imapd 3.0 will be forked from the master branch with the release of
the new version.



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Invalid mailbox name

[Michael Menge via Info-cyrus]

Hi,

Quoting Paul van der Vlis via Info-cyrus :


Hello,

I am syncing many mailboxes from one IMAP server to another. Now I get
sometimes errors on mailbox names. I found out that Cyrus does not
accept brackets in a mailbox name. Is there documentation about what
characters are accepted in mailbox names??


The allowed ASCII-Chars are defined in the macro GOODCHARS in imap/mboxname.c
(https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/mboxname.c#L1495).
non-ASCII-Chars are handled by RFC 3501 5.1.3.

This subject has been discussed a few years ago on this list, and GOODCHARS
has been changed between cyrus versions.

2.2:#define GOODCHARS "  
+,-.0123456789:=@ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~"
2.3:#define GOODCHARS "  
#$'+,-.0123456789:=@ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~"
Master: #define GOODCHARS "  
#$'()*+,-.0123456789:=?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz~"









M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: unable to delete corrupted mail box on cyrus v2.3.16

[Michael Menge via Info-cyrus]

Quoting Sophie Loewenthal via Info-cyrus :


Hi!

I have a broken mailbox that I would like to delete.

This is Cyrus v2.3.16 on CentOS 6.

 I tried reconstructing the mailbox from scratch ( Because I suspect  
this was manually deleted from disc ).



mkdir imap-store/spool/imap/domain/example.com/user/kat^long
cd imap-store/spool/imap/domain/example.com/user/kat^long
chmod 755 .
chown cyrus:mail .
touch cyrus.header
chown cyrus:mail cyrus.header

log into cyradm:
localhost> lam user/kat.long
kat.l...@example.com lrswipkxtecda
localhost> reconstruct -r user/kae.long
reconstruct: Mailbox has an invalid format
localhost> dm user/kat.long
deletemailbox: Permission denied


you must give your admin account the permission to delete the mailbox

localhost> sam user/kat.long cyrus all
localhost> dm user/kat.long




Names and domain names replaced with false entries.

How could I remove this?

Kind regards,
Sophie.





Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: migrating mails between two cyrus servers keeping seen states

[Michael Menge via Info-cyrus]

Hi,


Quoting Marcus Schopen via Info-cyrus :


Hi,

I'm planning to move some mailboxes from a cyrus server (2.1.18 - which
is a kind of mail archive) to another one (2.4.17+caldav~beta9-3).
Playing around on a test system with copying a user's mailbox and doing
a following "reconstruct -r" came out with a mailbox without seen states
on read mails on the new system; all mails unread. Normally I move
mailboxes using imapsync to keep states etc., but on a number of really
large mailboxes the initial sync takes days. Any other suggestions than
(slow) imapsync tools?


Befor 2.4.x the seen information was stored in extra files in /var/imap/user
Did you copy these files too?



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: migrating mails between two cyrus servers keeping seen states

[Michael Menge via Info-cyrus]

Quoting Marcus Schopen <li...@localguru.de>:


Am Freitag, den 18.12.2015, 14:09 +0100 schrieb Michael Menge via
Info-cyrus:

Hi,


Quoting Marcus Schopen via Info-cyrus <info-cyrus@lists.andrew.cmu.edu>:

> Hi,
>
> I'm planning to move some mailboxes from a cyrus server (2.1.18 - which
> is a kind of mail archive) to another one (2.4.17+caldav~beta9-3).
> Playing around on a test system with copying a user's mailbox and doing
> a following "reconstruct -r" came out with a mailbox without seen states
> on read mails on the new system; all mails unread. Normally I move
> mailboxes using imapsync to keep states etc., but on a number of really
> large mailboxes the initial sync takes days. Any other suggestions than
> (slow) imapsync tools?
>
Befor 2.4.x the seen information was stored in extra files in /var/imap/user
Did you copy these files too?


Ah, it's an old Debian system. These files are stored in
eg. /var/lib/cyrus/user/t/testuser.seen. Where do I have to copy this
file on the new server to? And there is a testuser.sub in the same dir,
which contains the subfolders. What about this file?



The seen information is now included in the cyrus.* files (i think in  
the index file),
but upgrading form an older version with reconstruct should check the  
.seen file at
the old location. The location depends on your setting of  
configdirectory, fulldirhash

and hashimapspool in impad.conf.

The handling of the sub file has not changed. But you should copy them  
to the new server,

if you want to keep the subscription information.


You should check  
https://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-upgrade.php




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Replication problem do_folders(): failed to rename

[Michael Menge via Info-cyrus]

Hi,


Quoting Patrick Boutilier via Info-cyrus :


On 12/14/2015 06:25 AM, Marcus Schopen via Info-cyrus wrote:

Am Freitag, den 11.12.2015, 19:10 +0100 schrieb Marcus Schopen via
Info-cyrus:

Hi,

I have a problem with a single mailbox. The user's Outlook crashed and
since then the sync_client is running wild on this user account and
produces high load on the master. I stopped sync_client on master side
for the moment.

When I try to sync the user by hand

/bin/su - cyrus -c "/usr/lib/cyrus/bin/sync_client -S replicaserver -v
-u testuser

I do get the following error.

Dec 11 17:54:48 master cyrus/sync_client[22727]: RENAME received NO
response: Operation is not supported on mailbox
Dec 11 17:54:48 master cyrus/sync_client[22727]: do_folders(): failed to
rename: user.elsa-secgen -> user.testuser.Archives.Gesch
2014-15.25-Jahr-Feier
Dec 11 17:54:48 master cyrus/sync_client[22727]: IOERROR: The remote
Server(s) denied the operation
Dec 11 17:54:48 master cyrus/sync_client[22727]: Error in
do_user(testuser): bailing out!

Comparing master and slave on filesystem I do see the subfolder
"25-Jahr-Feier" in "user.testuser.Archives.Gesch 2014-15.",
but only on master but not on slave side. And why does sync_client want
to rename and where does it get this order from?

I can login into the users' mailbox on master side and new message are
shown in the INBOX.

How can I fix it?

Should I try a "reconstruct -r user.testuser" on master and slave or
just on slave? (do I have to shutdown cyrus for a reconstruct -r on a
user box?)

Or can I delete the complete mailbox on slave side start an "sync_client
-S replicaserver -v -u testuser"?

Thanks for helping
Marcus


I did a reconstruct on the replica whichs runs through on the 12 GB
mailbox ot the user within a second (too fast?).

/bin/su - cyrus -c "/usr/lib/cyrus/bin/reconstruct -r user.testuser"

A following sync ended up with the same error:

/bin/su - cyrus -c "/usr/lib/cyrus/bin/sync_client -S replicaserver -v
-u testuser

Any ideas?


No, but this seems weird. Was this user ever renamed?

user.elsa-secgen -> user.testuser.Archives.Gesch



Cyrus uses the folder "Unique ID" for syncronisation. If this "Unique  
ID" is NOT unique

it will confuse syncronisation.





Ciao
Marcus



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus imap 2.3.11-xx with postfix 2.9.4

[Michael Menge via Info-cyrus]

Quoting Josef Karliak via Info-cyrus :


Good morning,
  we had some issue tomorow's morning - some users couldn't login to an
email system. We use ypages for distributing passwd maps, authorizing
daemon is saslauthd.
  Some users logged in, some not. "Unauthorized" users made a record to
the syslog:
Nov 24 08:01:23 email1 saslauthd[10396]: DEBUG: auth_pam: pam_authenticate
failed: User not known to the underlying authentication module
Nov 24 08:01:23 email1 saslauthd[10396]: do_auth : auth failure:
[user=username] [service=imap] [realm=] [mech=pam] [reason=PAM auth error]
Nov 24 08:01:23 email1 imap[26411]: badlogin: localhost [127.0.0.1]
plaintext username SASL(-13): authentication failure: checkpass failed

  About this time +/- seconds I see in the mail log this complains of the
postfix to the cyrus's lmtp:
Nov 24 08:01:25 email1 postfix/lmtp[29859]: warning: 19535581B: non-LMTP
response from email1.fnhk.cz[public/lmtp]: do_ypcall: clnt_call: RPC:
Timed out
Nov 24 08:01:25 email1 postfix/lmtp[29859]: warning: to prevent loss of
mail, turn off command pipelining for public/lmtp with the
lmtp_discard_lhlo_keyword_address_maps parameter
Nov 24 08:01:50 email1 postfix/lmtp[29859]: warning: 19535581B: non-LMTP
response from email1.fnhk.cz[public/lmtp]: do_ypcall: clnt_call: RPC:
Timed out
Nov 24 08:01:50 email1 postfix/lmtp[29859]: warning: to prevent loss of
mail, turn off command pipelining for public/lmtp with the
lmtp_discard_lhlo_keyword_address_maps parameter

  The username was in all 3 hosts maps, another users logged in, there is
no complains about network connections.

  This worked fine for an years, but this issue happened twice within 2
weeks.

  An administrators of the ypages servers don't see any problem or logs
about theirs server.

  What caused this issue ? What can I do to prevent it.



The errors "User not known to the underlying authentication module" and
"do_ypcall: clnt_call: RPC: Timed out" indicate that your system is unable
to query the ypages in time. As this happens to postfix and cyrus I think
it is not a problem in postfix or cyrus but in your ypages client,
ypages server or the network connection.




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Advise needed for new mail server

[Michael Menge via Info-cyrus]

Hi Mufit,

Quoting Mufit Eribol via Info-cyrus <info-cyrus@lists.andrew.cmu.edu>:


Hello,

I have been successfully using postfix+cyrus-imapd (2.4.17) for our  
small company for years on our local server. The emails are now  
accounting to a size of  some 160GB. As we are having power and  
internet problems quite often, I rented a VPS from a world renowned  
hosting company and installed postfix+cyrus-imapd there.


My question is, as I have limited hard disk space (40GB) on VPS, I  
can't (and don't want to) copy all of my local emails to the VPS.  
The new mail server will have a fresh start. But old emails needs to  
be accessible on the local server as well.


Currently, I am planning to change the names of local domains to  
some non-existent name just for the internal lookup (example.com -->  
example2.com), so that we can setup example2.com on our email  
clients on lan. The real domain example.com will be setup on our  
desktop email clients as usual. I think, using example2.com on local  
lan just for reading mails by cyrus will work, but it is not an  
elegant solution.


I would appreciate any ideas.




Cyrus Murder my be the solution for you.
https://cyrusimap.org/mediawiki/index.php/Cyrus_Murder

The Clients will only connect to one server (VPS), and the mails can  
be stored on different
backend servers. There is only minimal configuration change needed on  
your old server, but you

would need to rename the folders.


1 backend (VPS) for the INBOX and folders with new Mail
1 backend (old System) with all old folders in an archive subfolder  
for each user

1 mupdate master (VPS)
1 frontend (VPS)


Regards

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: PATCH for lmtp was [frontend lmtp connections to mupdate master]

[Michael Menge via Info-cyrus]

Hi,

https://cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Failure_Modes
"2. Mail delivery via the frontends is deferred (lmtpproxyd cannot  
locate mailbox). "

should be updated too

Quoting Ken Murchison <mu...@andrew.cmu.edu>:


Done.


On 11/17/2015 03:29 AM, Michael Menge wrote:

Hi Ken

thanks. Do you intend to patch 2.4.x and 2.5.x ?

I am a bit confused by the recent changes of cyrusimap and cyrus.fondation.
Is the bugtracker https://bugzilla.cyrusimap.org still used by the project?
If yes, you should close  
https://bugzilla.cyrusimap.org/show_bug.cgi?id=3866,
if not, someone should add some information where the new  
bugtracker can be found


Regards

  Michael



Quoting Ken Murchison <mu...@andrew.cmu.edu>:

I committed a revised version just a little while ago:  
https://git.cyrus.foundation/rI0fd86ca759ffa32e8d22dbdde78ea82d63650827




On 11/16/2015 04:41 AM, Michael Menge via Info-cyrus wrote:

Hi,

is there anything missing from my side, or something that I can do to help
that my patch can be included?

Regards

  Michael


Quoting Nic Bernstein via Info-cyrus <info-cyrus@lists.andrew.cmu.edu>:


Bron,
We're starting to run into this same issue -- lmtp proxies fail  
to deliver mail when mupdate master craps -- and so started to  
investigate this thread.  However, it doesn't look like Michael  
Menge's patch made it in.  Your thoughts?

  -nic

On 10/07/2014 08:17 AM, Bron Gondwana wrote:
I'm convinced :)  I'll probably add it with Ken while we're at  
CMU in a couple of weeks when we can test it together.  This is  
precisely the kind of thing we love to see!  Thanks.


Bron.

On Tue, Oct 7, 2014, at 10:23 PM, Michael Menge wrote:

Hi,

as we have had an other crash of mupdate master last week, we used the
patch on our production system.

After the patch was added the load on mupdate master was dropping form
about 10 to about 1.

After one week of using it we have not found any problems.
As I have not received any other feedback, I added it to the
bugtracker.

https://bugzilla.cyrusimap.org/show_bug.cgi?id=3866

Pleas consider adding it to 2.5 and 2.4.next

Regards

  Michael


Quoting Michael Menge <michael.me...@zdv.uni-tuebingen.de>:


Hi,


Quoting Michael Menge <michael.me...@zdv.uni-tuebingen.de>:


Quoting Michael Menge <michael.me...@zdv.uni-tuebingen.de>:

By thew way, the reason I was so surprised in the first  
place was, that

I have been fooled by the documentation:
Quoting  
https://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-murder.php



Configuring the frontends
[...]
However, because the frontends only talk to the mupdate master
via   a  slave running on the local machine, you will also need
to set  up  a  slave on the same machine, in the SERVICES section
of   cyrus.conf,  like so

And an other misleading DOC !
Quoting
http://www.cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Mail_Delivery

UMich patch

Patch Patches are against 2.2 but are being moved to 2.3
Lmtpproxyd   will now use the local mailboxes.db

Quoting Wesley Craig <wescr...@gmail.com>:


On 29 Sep 2014, at 10:08, Michael Menge
<michael.me...@zdv.uni-tuebingen.de> wrote:

thanks for your mail. By any chance do you still have the patch?

In fact, I don't.  Dusting off my notes, I recall now that instead
of patching this (bogus) code, we instead decided to configure the
frontends as "unified", while leaving the backends "standard".  The
only downside of this approach was that a naive admin could
potentially create a user's mailbox on a frontend.

Sorry I said that we ported that code forward: in fact we simply
got  the effect of consulting mailboxes.db without porting forward.

:wes

I have patched my lmtpd to use the local mailbox.db in all cases,
unified Murder, and standalone cyrus already did this.
I tested it with a few testmails on my test system,
but as i don't have a unified setup or a standalone test system
at the moment i didn't test these setups.

I would appreciate a review.


Regards,

 Michael Menge


 M.MengeTel.: (49)  
7071/29-70316

Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen




 M.MengeTel.: (49)  
7071/29-70316

Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Email had 1 attachment:
+ smime.p7s
7k (application/pkcs7-signature)

Re: PATCH for lmtp was [frontend lmtp connections to mupdate master]

[Michael Menge via Info-cyrus]

Hi Ken

thanks. Do you intend to patch 2.4.x and 2.5.x ?

I am a bit confused by the recent changes of cyrusimap and cyrus.fondation.
Is the bugtracker https://bugzilla.cyrusimap.org still used by the project?
If yes, you should close https://bugzilla.cyrusimap.org/show_bug.cgi?id=3866,
if not, someone should add some information where the new bugtracker  
can be found


Regards

   Michael



Quoting Ken Murchison <mu...@andrew.cmu.edu>:

I committed a revised version just a little while ago:  
https://git.cyrus.foundation/rI0fd86ca759ffa32e8d22dbdde78ea82d63650827




On 11/16/2015 04:41 AM, Michael Menge via Info-cyrus wrote:

Hi,

is there anything missing from my side, or something that I can do to help
that my patch can be included?

Regards

   Michael


Quoting Nic Bernstein via Info-cyrus <info-cyrus@lists.andrew.cmu.edu>:


Bron,
We're starting to run into this same issue -- lmtp proxies fail to  
deliver mail when mupdate master craps -- and so started to  
investigate this thread.  However, it doesn't look like Michael  
Menge's patch made it in.  Your thoughts?

   -nic

On 10/07/2014 08:17 AM, Bron Gondwana wrote:
I'm convinced :)  I'll probably add it with Ken while we're at  
CMU in a couple of weeks when we can test it together.  This is  
precisely the kind of thing we love to see!  Thanks.


Bron.

On Tue, Oct 7, 2014, at 10:23 PM, Michael Menge wrote:

Hi,

as we have had an other crash of mupdate master last week, we used the
patch on our production system.

After the patch was added the load on mupdate master was dropping form
about 10 to about 1.

After one week of using it we have not found any problems.
As I have not received any other feedback, I added it to the
bugtracker.

https://bugzilla.cyrusimap.org/show_bug.cgi?id=3866

Pleas consider adding it to 2.5 and 2.4.next

Regards

   Michael


Quoting Michael Menge <michael.me...@zdv.uni-tuebingen.de>:


Hi,


Quoting Michael Menge <michael.me...@zdv.uni-tuebingen.de>:


Quoting Michael Menge <michael.me...@zdv.uni-tuebingen.de>:

By thew way, the reason I was so surprised in the first place  
was, that

I have been fooled by the documentation:
Quoting  
https://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-murder.php



Configuring the frontends
[...]
However, because the frontends only talk to the mupdate master
via   a  slave running on the local machine, you will also need
to set  up  a  slave on the same machine, in the SERVICES section
of   cyrus.conf,  like so

And an other misleading DOC !
Quoting
http://www.cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Mail_Delivery

UMich patch

Patch Patches are against 2.2 but are being moved to 2.3
Lmtpproxyd   will now use the local mailboxes.db

Quoting Wesley Craig <wescr...@gmail.com>:


On 29 Sep 2014, at 10:08, Michael Menge
<michael.me...@zdv.uni-tuebingen.de> wrote:

thanks for your mail. By any chance do you still have the patch?

In fact, I don't.  Dusting off my notes, I recall now that instead
of patching this (bogus) code, we instead decided to configure the
frontends as "unified", while leaving the backends "standard".  The
only downside of this approach was that a naive admin could
potentially create a user's mailbox on a frontend.

Sorry I said that we ported that code forward: in fact we simply
got  the effect of consulting mailboxes.db without porting forward.

:wes

I have patched my lmtpd to use the local mailbox.db in all cases,
unified Murder, and standalone cyrus already did this.
I tested it with a few testmails on my test system,
but as i don't have a unified setup or a standalone test system
at the moment i didn't test these setups.

I would appreciate a review.


Regards,

  Michael Menge


 M.MengeTel.: (49)  
7071/29-70316

Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen




 M.MengeTel.: (49)  
7071/29-70316

Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Email had 1 attachment:
+ smime.p7s
 7k (application/pkcs7-signature)




--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
6525 W Bluemound Road, Suite 24   v. 414.272.4477
Milwaukee, Wisconsin  53213-4073




 M.Menge 

Re: PATCH for lmtp was [frontend lmtp connections to mupdate master]

[Michael Menge via Info-cyrus]

Hi,

is there anything missing from my side, or something that I can do to help
that my patch can be included?

Regards

Michael


Quoting Nic Bernstein via Info-cyrus <info-cyrus@lists.andrew.cmu.edu>:


Bron,
We're starting to run into this same issue -- lmtp proxies fail to  
deliver mail when mupdate master craps -- and so started to  
investigate this thread.  However, it doesn't look like Michael  
Menge's patch made it in.  Your thoughts?

-nic

On 10/07/2014 08:17 AM, Bron Gondwana wrote:
I'm convinced :)  I'll probably add it with Ken while we're at CMU  
in a couple of weeks when we can test it together.  This is  
precisely the kind of thing we love to see!  Thanks.


Bron.

On Tue, Oct 7, 2014, at 10:23 PM, Michael Menge wrote:

Hi,

as we have had an other crash of mupdate master last week, we used the
patch on our production system.

After the patch was added the load on mupdate master was dropping form
about 10 to about 1.

After one week of using it we have not found any problems.
As I have not received any other feedback, I added it to the
bugtracker.

https://bugzilla.cyrusimap.org/show_bug.cgi?id=3866

Pleas consider adding it to 2.5 and 2.4.next

Regards

Michael


Quoting Michael Menge <michael.me...@zdv.uni-tuebingen.de>:


Hi,


Quoting Michael Menge <michael.me...@zdv.uni-tuebingen.de>:


Quoting Michael Menge <michael.me...@zdv.uni-tuebingen.de>:


By thew way, the reason I was so surprised in the first place was, that
I have been fooled by the documentation:
Quoting https://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-murder.php


Configuring the frontends
[...]
However, because the frontends only talk to the mupdate master
via   a  slave running on the local machine, you will also need
to set  up  a  slave on the same machine, in the SERVICES section
of   cyrus.conf,  like so

And an other misleading DOC !
Quoting
http://www.cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Mail_Delivery

UMich patch

Patch Patches are against 2.2 but are being moved to 2.3
Lmtpproxyd   will now use the local mailboxes.db

Quoting Wesley Craig <wescr...@gmail.com>:


On 29 Sep 2014, at 10:08, Michael Menge
<michael.me...@zdv.uni-tuebingen.de> wrote:

thanks for your mail. By any chance do you still have the patch?

In fact, I don't.  Dusting off my notes, I recall now that instead
of patching this (bogus) code, we instead decided to configure the
frontends as "unified", while leaving the backends "standard".  The
 only downside of this approach was that a naive admin could
potentially create a user's mailbox on a frontend.

Sorry I said that we ported that code forward: in fact we simply
got  the effect of consulting mailboxes.db without porting forward.

:wes

I have patched my lmtpd to use the local mailbox.db in all cases,
unified Murder, and standalone cyrus already did this.
I tested it with a few testmails on my test system,
but as i don't have a unified setup or a standalone test system
at the moment i didn't test these setups.

I would appreciate a review.


Regards,

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Email had 1 attachment:
+ smime.p7s
  7k (application/pkcs7-signature)




--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
6525 W Bluemound Road, Suite 24   v. 414.272.4477
Milwaukee, Wisconsin  53213-4073





M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Dying on Import from PST

[Michael Menge]

Hi Cyrus

this "IOERROR" will also be reported if importing is aborted.  
"unexpected end of file"
is ab bit misleading in this case of message append via IMAP.

One reason can be malformed headers in the mail.
You can use telemetry logging to find the problematic message and the reason.


Quoting Paul Bronson :

> Does anyone else have trouble importing a PST?
>
> I just connected up my cyrus to outlook 2013. Imported a PST (roughly 2k
> emails) and I go onto roudncube and only about 400 show up. I go back to
> Outlook and hit Update Folder, it sends up another 10;15 messages then dies
> again. I am seeing this in the logs:
>
>  imap[3588]: Expunged 1 messages from j...@domain.com
>
>
> and
>
> IOERROR: reading message: unexpected end of file
>
>
> Is there something wrong that I am missing or doing wrong?




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: force relication to all mailboxes

[Michael Menge]

Hi,

Quoting Marcus Schopen li...@localguru.de:

 Hi Bron,

 Am Sonntag, den 19.07.2015, 18:19 -0700 schrieb Bron Gondwana:
 On Sun, Jul 19, 2015, at 12:49, Marcus Schopen wrote:
  Am Sonntag, den 19.07.2015, 18:16 +0200 schrieb Marcus Schopen:
   Hi,
  
   I set a cyrus (2.4.12 / Ubuntu 12.04 LTS) with existing mailboxes in
   replication with an cyrus (2.4.12 / Ubuntu 12.04 LTS). Replication works
   fine, but a sync of existing mailboxes is only triggered on by new
   incoming mails to inboxes or by changes inside subfolders. How to I
   force an initial replication of all mailboxes including subfolders from
   the master to the replica?
 
  Calling sync_client for each mailbox works fine:
 
   sync_client -S imapsync.server -v -u mailboxname


-u is for user

-m is for mailbox

-s is for sieve scripts

   ...
 
  Any better ways?

 There will be '-A' in the future, but it's not available in LTS.   
 You can script
 it up with something that uses cyr_dbtool or ctl_mboxlist to get a listing.


 For understanding: after a restart of a replica the way to get master
 und replica fully in sync again is to start the syn_cclient on master
 and call sync_client -S imapsync.server -v -u mailboxname for each
 mailbox, right?


-u username will sync all subfolders of user.username (user/username),
the sieve scrips and the mailbox subscription. So -u for each user
and -m for shared and news mailboxes.

You don't need the -S option if the syncserver is configured in  
/etc/imapd.conf


 And are there any scripts available to check if master and replica
 storage completely in sync or mails are missing?

 Ciao
 Marcus


 
 Cyrus Home Page: http://www.cyrusimap.org/
 List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
 To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

how to configure dhparms in cyrus-imapd 2.4.x

[Michael Menge]

Hi,

to prevent the logjam Attack https://weakdh.org/ I want to configure
my own set of dh parameters.
I didn't find an option to configure the dh parameter length, or a dh
parameter file generated by
openssl dhparam


Regards

   Michael


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen


smime.p7s
Description: S/MIME-Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Hot to recovry userd deleted

[Michael Menge]

Hi

Quoting Manuel Vazquez titocer...@gmail.com:


Hi.

It's possible recovery an users deleted with the rename cyardm command ?


It depends if delayed expunge is enabled or not.
You you have delete_mode: delayed in your imapd.conf

Deleting folders will them move to the DELETED folder (or
wathever is configured as deletedprefix)

Depending on the client that deleted the mailbox you might
need use unxepunge on the folder to restore the mails in the folder.



It is not possible how I can to do this?


Restore the files form backup and run reconstruct.



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: restore from cyrdump

[Michael Menge]

Quoting Andrew Morgan mor...@orst.edu:


On Mon, 15 Dec 2014, Patrick Goetz wrote:


On 12/10/2014 03:47 AM, Willy Offermans wrote:

I'm not sure what you mean with ``all the metadata'', but there are user
flags saved into the cyrdump file as well. I never performed the whole
cycle of dump and restore (probably nobody did so far), so I cannot tell
you that all metadata is available in the dump file. See my question above!




A while back I was working on an email server that (unbeknownst to me)
was connected to a UPS, but with an external disk array that was plugged
in to an outlet on the UPS that was not battery-backed.  This site had
frequent power problems, so it turns out that power cycling a disk array
while the server stays up is an awesome way to corrupt your entire file
system.

Since I didn't know what I was doing at the time, I restored

  partition-default: /home/cyrus

without also restoring

  configdirectory: /var/lib/cyrus

I was consequently confused when no mailboxes showed up, and had to then
learn about and use reconstruct -r on each individual mailbox
(cyrreconstruct on debian/Ubuntu) in order to reconstruct the
/var/lib/cyrus/*.db files.

I think the main database files you need are mailboxes.db and
annotations.db (can someone confirm this?)

This still leaves the question of how best to back up a cyrus mailstore.

Bron mentioned that most people are using LVM snapshots.  I don't see
how using btrfs/LVM/ZFS snapshots can save you from a race condition
between when the cyrus user directory is updated and when mailboxes.db
is updated.  The only way I would trust this is by doing this:

   1. Stop cyrus
   2. Snapshot
   3. Restart cyrus


cyrdump:  near as I can tell the only useful purpose this serves is to
assemble all email messages into a single mbox file (can anyone
confirm this)?

ctl_mboxlist: this seems useful for making a human readable copy of the
mailboxes.db file, but I'm not sure how this could be useful for
disaster recovery, given the previously mentioned issue about keeping
the mailboxes.db file synchronized with the contents of the user dir.

So, given a simple mail server (i.e. no murder + replication), and when
using a filesystem (e.g. ext4 or XFS) which doesn't do snapshots, it
would appear that the only safe way to backup up a cyrus mailstore is to
either using something like imapsync, or

   1. Stop cyrus
   2. tar cvf /some/safe/place/user.tar {default-partion}
   3. tar cvf /some/safe/place/cyrusdb.tar {configdirectory}
   4. Restart cyrus

The way I've used imapsync in the past required copying mail folders per
authenticated user account; i.e. something like

imapsync --host1 my_host1 --authmech1 LOGIN --user1 my_user1 --password1
x --host2 my_host2 --authmech2 PLAIN --user2 my_user2 --password2 x

which in particular means knowing everyone's passwords.  This is
entirely unworkable for larger sites, and I'm not sure if there is a
trick for getting around this.


We are a large site.  We have 3 backend servers in a Murder cluster with
about 25,000 mailboxes per backend.

Unless you are going to use Fastmail's fancy backup method that actually
locks the mailbox in Cyrus, I don't think there is a way to take a
perfectly consistent backup.

That said, we didn't care about achieving perfection, so we just perform a
normal filesystem backup.  We use EMC Networker, but it isn't doing
anything fancy.  It just walks the entire directory tree looking for
changed files to backup.

Yeah, there is the potential for a race condition.  You really only need
each mailbox to be consistent though, and the odds of a mailbox changing
while it's being backed up are sufficiently low here.  In the event of a
disaster (total loss of filesystem), we'll be restoring from our most
recent backup, which will be anywhere from 0-24 hours old.  If we need to
recover an individual mailbox, we can get everything using delayed delete.
Either way, we'll be running reconstruct on the mailbox(es).

I see people talk a lot about fancy ways to back up Cyrus, but we've just
never had a need.

Andy


We also don't use snapshots or stop cyrus for backup.

But as a complete restore of our mail storage with normal backuptools
would take fare too long, we uses cyrus sync for disaster recovery.
For the normal backup we use a combination of delayed expung (14 days)
and normal filesystem backup. In most cases where the mail is still
in the filesystem and can be restored by unexpung, and in the rare
cases where the mail has been expunged i have to run reconstruct anyway.


   Michael






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/

Re: Locations for folders, etc, for previously-existing users

[Michael Menge]

Hi Don

Quoting Don Levey cyrus.i...@the-leveys.us:


If this has been covered elsewhere, please excuse me - I have not turned
up the info I need in any of my searches.  Recently my CentOS home server
crashed, and in the process of reconstruction I have brought up a new
configuration into which I am trying to migrate the data from my old
users.  What's more, my old server was first configured over a decade ago,
and when it came time to rebuild I decided on Cyrus rather than the
previously-installed Dovecot (as I've found Cyrus is now the default on
CentOS).  Before I worry about the various client applications trying to
connect, am am working on getting Squirrelmail up and running properly.

Now, the details.  At the moment, I am able to send via Squirrelmail, but
am unable to access the existing mail inbox or the numerous folders my
users have previously created on the server (currently reading mail via
mutt, inbox only).  This appears to be due to the differing directory
structures in the two different IMAP packages.  While it might be nice to
simply alter a config var and have everything magically appear, I am
certainly willing to manually move data files to the new appropriate
locations - the problem is that I don't know the locations.

Previously, I had the following directory structure
/home/user
~mail (I honestly don't know how it got set up this way; the tilda
gives me the willies)
Folder 1 (data file with mail messages)
Folder 2
Folder 3
.imap
Folder 1 (dir)
Folder 2 (dir)
Folder 3 (dir)
dovecot.index
dovecot.index.cache
dovecot.index.log

..And so on.  I know that the Cyrus structure seems to use:
/var/spool/imap/user/u/username

but that username file seems to be... not a data file. Is there a way to
migrate the old configuration into the new one?  I'm not sure what other
info will be necessary from me.


Cyrus and Dovecot use differen structures to store the mails and
meta infos on disk. Cyrus is not able to pares the Dovecote data
and as far as I know dovecote can't parse the cyrus structures
either directly. There might be tools to convert the data. Most
tools I know of use the IMAP protocol which requires a running imap
server that can access the old mails.

If i remember correctly the dovecot data files use the mbox format.
There are some tools that can upload mails from mbox files to IMPA
Servers. But i don't know if it is possible to convert all mailbox
features with them (ACLs, seen status for shared mailboxes).

Regards,


   Michael




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Strange load issue with 2.4.17

[Michael Menge]

Hi,

Quoting Sebastian Hagedorn haged...@uni-koeln.de:

--On 13. Oktober 2014 17:39:23 +0200 Simon Matter  
simon.mat...@invoca.ch wrote:



--On 13. Oktober 2014 17:35:25 +0200 Simon Matter
simon.mat...@invoca.ch wrote:


Hi,

for the last week we have seen strange load issues on our Cyrus server.
All
of a sudden the load increases to several thousands, user CPU goes down
to basically zero, system CPU spikes. In the past we've had trouble
with
poor I/O performance, but that went along with an increase in Wait I/O.
We don't
see that now. vmstat shows a massive increase in context switches. When
the
system reaches this state, all we can do is restart Cyrus or reboot the
machine if that doesn't work anymore.

I'm attaching a Ganglia screenshot that shows the problem clearly. When
the
problem exists, there's not much we can do to analyze it. A colleague
suggested that what we see could be related to this bug:

https://bugzilla.cyrusimap.org/show_bug.cgi?id=3744

It was reported for 2.4.16, and it sounds as if it has been fixed, but
is
that fix really part of 2.4.17? Any other ideas?


Is this a physical host or running virtualized?


It's virtualized, but it's been that way for more than a year.


Is this by any chance running on KVM, maybe on an AMD cpu?


No, it's VMware ESX on Intel CPUs.


How is the memory usage? Is the system swaping?



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: PATCH for lmtp was [frontend lmtp connections to mupdate master]

[Michael Menge]

Hi,

as we have had an other crash of mupdate master last week, we used the
patch on our production system.

After the patch was added the load on mupdate master was dropping form
about 10 to about 1.

After one week of using it we have not found any problems.
As I have not received any other feedback, I added it to the
bugtracker.

https://bugzilla.cyrusimap.org/show_bug.cgi?id=3866

Pleas consider adding it to 2.5 and 2.4.next

Regards

   Michael


Quoting Michael Menge michael.me...@zdv.uni-tuebingen.de:


Hi,


Quoting Michael Menge michael.me...@zdv.uni-tuebingen.de:


Quoting Michael Menge michael.me...@zdv.uni-tuebingen.de:


By thew way, the reason I was so surprised in the first place was, that
I have been fooled by the documentation:
Quoting https://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-murder.php


Configuring the frontends
[...]
However, because the frontends only talk to the mupdate master  
via   a  slave running on the local machine, you will also need  
to set  up  a  slave on the same machine, in the SERVICES section  
of   cyrus.conf,  like so


And an other misleading DOC !
Quoting
http://www.cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Mail_Delivery

UMich patch

Patch Patches are against 2.2 but are being moved to 2.3  
Lmtpproxyd   will now use the local mailboxes.db




Quoting Wesley Craig wescr...@gmail.com:

On 29 Sep 2014, at 10:08, Michael Menge   
michael.me...@zdv.uni-tuebingen.de wrote:

thanks for your mail. By any chance do you still have the patch?


In fact, I don't.  Dusting off my notes, I recall now that instead   
of patching this (bogus) code, we instead decided to configure the   
frontends as unified, while leaving the backends standard.  The  
 only downside of this approach was that a naive admin could   
potentially create a user's mailbox on a frontend.


Sorry I said that we ported that code forward: in fact we simply  
got  the effect of consulting mailboxes.db without porting forward.


:wes


I have patched my lmtpd to use the local mailbox.db in all cases,
unified Murder, and standalone cyrus already did this.
I tested it with a few testmails on my test system,
but as i don't have a unified setup or a standalone test system
at the moment i didn't test these setups.

I would appreciate a review.


Regards,

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:   
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Migrate cyrus 2.3.8 to 2.4.17

[Michael Menge]

Quoting Giuseppe Ravasio giuseppe_rava...@modiano.com:


Hi,
I'm planning and testing the migration of a IMAP mail server based on
cyrus 2.3.8 with about 3TB of mailstore, 300 users and a lot of shared
folders (with shared flags (group permission with +w)).

The old server is a 32bit one and the new one is 64bit virutual machine.
as now I've succesfully used this procedure:
- Rsync of all mailstores /var/lib/imap and /var/lib/sieve
- rm -v /var/lib/imap/db/* /var/lib/imap/db.backup1/* \
   /var/lib/imap/db.backup2/* /var/lib/imap/deliver.db \
   /var/lib/imap/tls_sessions.db /var/lib/imap/mailboxes.db
- ctl_mboxlist -u  /tmp/mboxlist.txt
- ctl_cyrusdb -r
- tls_prune
- ctl_cyrusdb -c

- cyr_expire -E 3

The last command took about 24 hours to complete and if understand right
It converted cyrus mailbox DBs from v9 to v12.

I wonder if the first cyr_expire could run even with the service started
and if it should be better to run a reconstruct on all the mailboxes in
place of this cyr_expire.



Every access of a mailbox (cyr_rxpire, squatter, unexpunge, imapd,  
pop3d, ...) will trigger the update of that mailbox on the fly.  
Depending on the size

of the mailbox this will only take a few seconds.
But access to many mailboxes will add up and my slow the system down,
so it might be a good idea to stop cyr_expire, squatter before work time.

Maybe cyr_expire -E 3 -x will not update the mailbox, but i don't know
for sure. Starting cyrus before cyr_expire has been run should not be a
problem.


Last but not least, do you think that sieve filters should be recompiled?


If you where only updating form 2.3.x to 2.4 that should not be necessary,
but as you are changing the architecture I don't know.




Any other hints?

Thanks
Giuseppe

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus







M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: frontend lmtp connections to mupdate master

[Michael Menge]

Hi

Quoting Fabio S. Schmidt fa...@improve.inf.br:


Hi Michael,

- Sorry if I am annoying answering this thread to the list, but maybe
someone has better ideias.

We are using Cyrus 2.4.14 with dedicated Backends, Frontends and
Mupdate servers. We have deployed a Cyrus Frontend on each MTA servers
so when a message is delivered it will only query locally the
locations of the mailboxes. This way the real Frontend servers are
only loaded with the IMAP connections from the clients.

These are our configurations:

# cyrus.conf

imapcmd=proxyd -a listen=localhost:imap prefork=5  
maxchild=100

lmtpcmd=lmtpproxyd -C /etc/imapd-local-lmtpd.conf
listen=lmtp prefork=10 maxchild=500

#/etc/imapd-local-lmtpd.conf

mupdate_server: 127.0.0.1
304 mupdate_port: 3905
305 mupdate_authname: admin
306 mupdate_username: admin
307 mupdate_password: PASSWORD
308 mupdate_retry_delay: 10

# /etc/postfix/transport

ourdomain.com.br  lmtp:127.0.0.1:lmtp



thanks for the config. So you are tricking lmtp to querry the local
mupdate over tcp. This should work in our case, but I'm looking for
an other solution, because this would still add an overhead (tcp  
connection, ssl encryption), which is not needed IMHO.


By thew way, the reason I was so surprised in the first place was, that
I have been fooled by the documentation:
Quoting https://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-murder.php


Configuring the frontends
[...]
However, because the frontends only talk to the mupdate master via a  
slave running on the local machine, you will also need to set up a  
slave on the same machine, in the SERVICES section of cyrus.conf,  
like so


So frontend imapd and pop3d will ask the frontend mailboxdb.
I'm not sure  about timsieved.
But IHMO there is no need that lmtpd/lmtpproxyd will ask the
mupdate_master only in in standard murder.

I will try to patch lmtpd to use the local mailboxdb and will send a
patch after i have tested it.

Regards

   Michael







M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: frontend lmtp connections to mupdate master

[Michael Menge]

Quoting Michael Menge michael.me...@zdv.uni-tuebingen.de:


By thew way, the reason I was so surprised in the first place was, that
I have been fooled by the documentation:
Quoting https://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-murder.php


Configuring the frontends
[...]
However, because the frontends only talk to the mupdate master via  
a  slave running on the local machine, you will also need to set up  
a  slave on the same machine, in the SERVICES section of  
cyrus.conf,  like so


And an other misleading DOC !
Quoting  
http://www.cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Mail_Delivery

UMich patch

Patch Patches are against 2.2 but are being moved to 2.3 Lmtpproxyd  
will now use the local mailboxes.db






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

PATCH for lmtp was [frontend lmtp connections to mupdate master]

[Michael Menge]

Hi,


Quoting Michael Menge michael.me...@zdv.uni-tuebingen.de:


Quoting Michael Menge michael.me...@zdv.uni-tuebingen.de:


By thew way, the reason I was so surprised in the first place was, that
I have been fooled by the documentation:
Quoting https://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-murder.php


Configuring the frontends
[...]
However, because the frontends only talk to the mupdate master via  
 a  slave running on the local machine, you will also need to set  
up  a  slave on the same machine, in the SERVICES section of   
cyrus.conf,  like so


And an other misleading DOC !
Quoting   
http://www.cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Mail_Delivery

UMich patch

Patch Patches are against 2.2 but are being moved to 2.3 Lmtpproxyd  
 will now use the local mailboxes.db




Quoting Wesley Craig wescr...@gmail.com:

On 29 Sep 2014, at 10:08, Michael Menge  
michael.me...@zdv.uni-tuebingen.de wrote:

thanks for your mail. By any chance do you still have the patch?


In fact, I don't.  Dusting off my notes, I recall now that instead  
of patching this (bogus) code, we instead decided to configure the  
frontends as unified, while leaving the backends standard.  The  
only downside of this approach was that a naive admin could  
potentially create a user's mailbox on a frontend.


Sorry I said that we ported that code forward: in fact we simply got  
the effect of consulting mailboxes.db without porting forward.


:wes


I have patched my lmtpd to use the local mailbox.db in all cases,
unified Murder, and standalone cyrus already did this.
I tested it with a few testmails on my test system,
but as i don't have a unified setup or a standalone test system
at the moment i didn't test these setups.

I would appreciate a review.


Regards,

   Michael Menge



M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen--- cyrus-imapd-2.4.17/imap/lmtpd.c	2012-12-01 20:57:54.0 +0100
+++ cyrus-imapd-2.4.17.lmtp_patch/imap/lmtpd.c	2014-09-29 15:27:55.0 +0200
@@ -186,56 +186,44 @@
 
 global_sasl_init(1, 1, mysasl_cb);
 
-if (config_mupdate_server 
-	(config_mupdate_config == IMAP_ENUM_MUPDATE_CONFIG_STANDARD) 
-	!config_getstring(IMAPOPT_PROXYSERVERS)) {
-	/* proxy only -- talk directly to mupdate master */
-	r = mupdate_connect(config_mupdate_server, NULL, mhandle, NULL);
-	if (r) {
-	syslog(LOG_ERR, couldn't connect to MUPDATE server %s: %s,
-		   config_mupdate_server, error_message(r));
-	fatal(error connecting with MUPDATE server, EC_TEMPFAIL);
-	}
-}
-else {
-	dupelim = config_getswitch(IMAPOPT_DUPLICATESUPPRESSION);
+dupelim = config_getswitch(IMAPOPT_DUPLICATESUPPRESSION);
 
 #ifdef USE_SIEVE
-	mylmtp.addheaders = xzmalloc(2 * sizeof(struct addheader));
-	mylmtp.addheaders[0].name = X-Sieve;
-	mylmtp.addheaders[0].body = SIEVE_VERSION;
+mylmtp.addheaders = xzmalloc(2 * sizeof(struct addheader));
+mylmtp.addheaders[0].name = X-Sieve;
+mylmtp.addheaders[0].body = SIEVE_VERSION;
 
-	/* setup sieve support */
-	sieve_interp = setup_sieve();
+/* setup sieve support */
+sieve_interp = setup_sieve();
 #else
-	if (dupelim)
+if (dupelim)
 #endif
-	{
-	/* initialize duplicate delivery database */
-	if (duplicate_init(NULL, 0) != 0) {
-		fatal(lmtpd: unable to init duplicate delivery database,
-		  EC_SOFTWARE);
-	}
+{
+/* initialize duplicate delivery database */
+	if (duplicate_init(NULL, 0) != 0) {
+	fatal(lmtpd: unable to init duplicate delivery database,
+	   EC_SOFTWARE);
 	}
+}
 
-	/* so we can do mboxlist operations */
-	mboxlist_init(0);
-	mboxlist_open(NULL);
-
-	/* so we can do quota operations */
-	quotadb_init(0);
-	quotadb_open(NULL);
-
-	/* Initialize the annotatemore db (for sieve on shared mailboxes) */
-	annotatemore_init(0, NULL, NULL);
-	annotatemore_open(NULL);
+/* so we can do mboxlist operations */
+mboxlist_init(0);
+mboxlist_open(NULL);
 
-	/* setup for statuscache invalidation */
-	statuscache_open(NULL);
+/* so we can do quota operations */
+quotadb_init(0);
+quotadb_open(NULL);
 
-	/* setup for sending IMAP IDLE notifications */
-	idle_enabled();
-}
+/* Initialize the annotatemore db (for sieve on shared mailboxes) */
+annotatemore_init(0, NULL, NULL);
+annotatemore_open(NULL);
+
+/* setup for statuscache invalidation */
+statuscache_open(NULL);
+
+/* setup for sending IMAP IDLE notifications */
+idle_enabled();
+
 
 /* Set namespace */
 if ((r = mboxname_init_namespace(lmtpd_namespace, 0)) != 0) {
@@ -288,32 +276,8 @@
 snmp_increment(TOTAL_CONNECTIONS, 1);
 snmp_increment(ACTIVE_CONNECTIONS, 1);
 
-/* get

Re: frontend lmtp connections to mupdate master

[Michael Menge]

Hi Fabio,



Quoting Fabio S. Schmidt fa...@improve.inf.br:


Hi Michael,

I would suggest you to read this documentation:
http://www.cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Mail_Delivery


thanks for the link, i have not seen this document before.



It is possible to change the delivery to use the local mailbox
database from the frontends when using Cyrus Aggregator. We have
already implemented it on our environment and it works very well. In
this scenario, the messages deliveries do not load the Mupdate master
and in a case of failure of the Master the messages will continue to
be delivered, so only the change operations from the Backends will be
unavailable.


The link to the UMich patch is borken. Did you use the UMich patch?
Can you send me your patch?
Can/should this be included in Cyrus Imapd 2.5.x

Regards

   Michael



---
M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: frontend lmtp connections to mupdate master

[Michael Menge]

Quoting Fabio S. Schmidt fa...@improve.inf.br:


Hi Michael,

You are most welcome.

We use cyrus 2.4 and there is no need to use the patch.


Sorry I had forgotten to write in my first post:
This was cyrus-imapd 2.4.17 :-(

Do you use unified murder?

Regards

   Michael


M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen

smime.p7s
Description: S/MIME Signatur

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus