Re: how to deal with mail retention/archival.
I saw that someone proposed to make a sort of abuse of delayed expunge, but I think that in order to comply with regulatory retention should be better considering some specific software. For example: http://www.mailpiler.org (Fully Free Software) https://www.mailarchiva.com (the old version is opensource but the latest is closed) The specific software will be much better for searching the archive. Finding something in the delayed_expunge folders after many years of archive will absolutely be a nightmare! Giuseppe On 08/26/2016 03:09 PM, Alvin Starr via Info-cyrus wrote: > A company I am working with is facing issues of regulatorymail retention. > > Some searching has yielded little useful results other than putting a > system in front to store all incoming messages. > > What are others doing for mail archival? > > An ideal solution would let the users carry on using current use > patterns and not impose extra restrictions. > > -- > Alvin Starr || voice: (905)513-7688 > Netvel Inc. || Cell: (416)806-0133 > al...@netvel.net || > > > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
IDLE: error sending message
Hi,
I've just migrated a quite busy cyrus install to 2.5.7.
It seems all going well except for the fact that in the logs there are a
lot of messages like:
Feb 8 09:07:06 mail cyrus/imaps[13868]: IDLE: error sending message
DONE to idled for mailbox ${SHARED_FOLDER}: Resource temporarily
unavailable.
and
Feb 8 09:14:27 thot cyrus/imaps[10661]: IDLE: error sending message
INIT to idled for mailbox ${SHARED_FOLDER}: Resource temporarily
unavailable. Falling back to polling every 60 seconds.
All the log messages refers to two folders that are shared to and
subscribed by all users.
I couldn't find any hint about this error with google.
I hope that someone could explain me this message.
Thanks
Giuseppe
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Server Backup
Hi, I'm rewriting my backup script and I cannot find any hint about how to backup the cyrus server in a good way. Actually I'm using this method: - Stop Postfix and Cyrus - making tar of /var/lib/sieve - making tar of /var/lib/imap - dumping mailboxes.db in plain text (ctl_mboxlist -d) - taking snapshots of all the mailstores - Restarting the Services - taking mailstore backups from the snapshots The last night the imaps was not restarted because the socket for IMAPS was busy and so I was wondering if today it's really needed to stop the service before taking the snapshots. Avoiding the service stop could also be useful in order to stop having a 1/2min downtime (I also take vmware snapshots of the machine) I was thinking to move in this direction: - Checkpoint and archive the databases (with ctl_cyrusdb -r) - making tar of /var/lib/imap - making tar of /var/lib/sieve - dumping mailboxes.db in plain text (ctl_mboxlist -d) - taking snapshots of all the mailstores - taking mailstore backups from the snapshots What kind of backup strategy are you using? Thanks Giuseppe Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus 2.3.8 to 2.5.6
Hi, I'm testing the migration of a quite large IMAP server from 2.3.8 to the latest version. I'm syncing: - mailboxes partitions - sieve - var/lib/imap/user - var/lib/imap/quota - var/lib/imap/annotations.db After the sync: - recompile sieve scripts - ctl_mboxlist -u guid) cyrus/master[5994]: process type:SERVICE name:imaps path:/usr/local/lib/cyrus/bin/imapd age:43.479s pid:11803 exited, status 75 The files will be moved to the new folder and sometimes reappears in the original folder, other times disappears whithout beeing shown with: unexpunge -l $FOLDER It's possible to insert expernal messages in thoose old mailbox but It's not possible to delete because the message reappears and the error above is shown in the logs. If I try to delete with SHIFT-DEL the message disappears from Thunderbird, the error is shown in the logs, the file is still on the disk but "unexpunge -l" doesn't show the missing email. It's a bug or I'm not supposed to open the service until the full reconstruct? Thanks Giuseppe it also get deleted from the client (Thunderbird) from the original one, but thoose mail aren't sown via IMAP until the reconstruct. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: [POLL] Cyrus ACLs and group names
Same here Giuseppe On 11/17/2015 12:23 PM, Adam Tauno Williams via Info-cyrus wrote: > On Tue, 2015-11-17 at 07:40 +1100, Bron Gondwana via Info-cyrus wrote: >> For those of you using Cyrus with group ACLs, how are your groups >> named? >> I know with the auth_unix backend, they are >> 'group:'. What I've seen from CMU's groups is that they >> are of the form ':'. > > Ours are group: > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Migrate from 2.2.13 to 2.4.17 disasters
I think that the key difference between using imapsync and upgrading cyrus is the amount of states that will be lost. I almost sure that with imapsync you cannot recreate: 1. Subscribed folders 2. Seen states for shared folders (also for INBOX if imapsync isn't used properly) 3. Shared seen states 4. Email flags (also for INBOX if imapsync isn't used properly) 5. Custom permissions Therefore I think that imapsync is not the best choice for corporate email servers or big installations. Giuseppe On 08/24/2015 01:44 PM, bs...@vsvinc.com wrote: Apologizes for the thread diversion here but I've been seeing a lot of discussion about using imapsync to migrate servers. From what I have read in the documentation, you must have each user's password. How is this possible in a business environment where you can't have (or shouldn't have) that information because of legal/privacy/ethical reasons? Is there something I'm missing in the docs? Brian *From:* Marcus Schopen li...@localguru.de *Sent:* Aug 24, 2015 3:54 AM *To:* mog...@fumlersoft.dk *Cc:* info-cyrus@lists.andrew.cmu.edu *Subject:* Re: Migrate from 2.2.13 to 2.4.17 disasters Am Sonntag, den 23.08.2015, 20:36 +0200 schrieb Mogens Melander: For a task like this, I would use imapsync, a well documented, well supported and open source tool. https://github.com/imapsync/imapsync I've used imapsync to migrate an internal server from 2.1.18 to 2.4.17 without any problems. Good tool. Ciao! Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Lock Folder and cyr_expire
I'm using a tmpfs for the Cyrus {configdir}/proc directory, like so:
tmpfs /var/spool/cyrus/config/proctmpfs
size=25M,nr_inodes=10k 0 0
Maybe it would be better to create {configdir}/lock as a separate tmpfs?
Something like:
tmpfs /var/spool/cyrus/config/proctmpfs
size=25M,nr_inodes=1k 0 0
Great! I tough that the nr_inodes for a tmpfs could not be set so high
as inode/size ratio! Thnaks.
Shuld be interesting to know how much ram space will need a tmpfs volume
with 1k inodes used by 0k files!
I couldn't find any information about inode size in tmpfs.
I also did a small test and on my workstation and it turns out that:
Creating 1 0k files on tmpfs
(-o size=25M,nr_inodes=1k) took 8sec
Creating 1 0k files on XFS loopback device on my SSD
(mkfs.xfs -i maxpct=0 ) took 9sec
Creating 1 0k files on XFS loopback device on tmpfs
(mkfs.xfs -i maxpct=0) took 8sec
There is no reason for lock files to persist between Cyrus restarts, right?
No. the doc say:
There is a new lock folder which defaults to configdir/lock/ and
contains one zerobyte file per mailbox. These can get pretty hot, and
don't need to persist over reboots (they will be auto-created when
needed) - so you may want to define mboxname_lockpath to be on tmpfs or
ramfs or similar. It certainly makes sense to clean it out on restart,
because names will persist in there forever otherwise. Even on mailbox
delete these files aren't removed (to avoid potential race conditions)
But if you're using cyr_expire I think that cleaning do not make sense
because at the first run all the files will be recreated.
Giuseppe
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Lock Folder and cyr_expire
Hi, recently I faced an outage for one of my IMAP server that runs out of inodes. We have about 500k growing (that aren't opened daily!) imap folders and the 0k lock files are filling the inode table of the partition containing the mboxname_lockpath I was hoping that cleaning mboxname_lockpath on cyrus restart will solve the issue but when cyr_expire job was started it created all the locks filling the inode table. Obiouvsly I will address the issue creating a specific filesystem for the mboxname_lockpath with XFS (that allows me to keep the inode/space ratio at maximum). There is a clean workaround that I'm missing? Anyway I think that would be useful to address this problem in some way (I don't know the reason of thoose lock files, so I don't have any ideas) or at least say in the Doc that mboxname_lockpath should be able to contain as many 0k files as the number of folders, because in 24h it will reach the top ( I think that 99% of cyrus users are running cyr_expire ) Thanks Giuseppe Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Migrate cyrus 2.3.8 to 2.4.17
On 10/08/2014 11:10 AM, jayesh shinde wrote: Hi Giuseppe , I want to learn few points from your migration scenario. Few days back I migrated my 32 bit cyrus-imapd-2.3.7-12.el5_7.2 to 64 bit cyrus-imapd-2.4.17-6.el6.x86_64 server I faced the below problems , how you are handling the below points on 64 bit server in your case ? 1) How do you maintained the seen-unseen mail flags. In my test case I hadn't any problems with seen and other flags. I to be told that both source and destination are using skiplist as seen database format. 2) I came across duplicate email download issue in email client . When IP of 32bit server shifted to 64 bit server , then for those client who have setup POP3 account with leave copy on server in email clients for them emails got download again. On 64 bit server there was no changes in /etc/hosts and DNS A records and /etc/resolve.conf , it was exactly same as 32 bit. To be honest this server is an IMAP only service, so I haven't tested the POP3 cases. Anyway I think that this problem is related with the first one, because if a message is downloaded via POP3 the server sould set the seen flag on it. I did a public cyrus server migration months ago from 2.2.x to 2.4.17 and I don't recall to have had thoose kind of problems (and I'm sure that there where 30/50 users using POP3 with copy on server). Did you rsync the /var/lib/imap/user/ from the old server? Bye Giuseppe Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Migrate cyrus 2.3.8 to 2.4.17
Hi, I'm planning and testing the migration of a IMAP mail server based on cyrus 2.3.8 with about 3TB of mailstore, 300 users and a lot of shared folders (with shared flags (group permission with +w)). The old server is a 32bit one and the new one is 64bit virutual machine. as now I've succesfully used this procedure: - Rsync of all mailstores /var/lib/imap and /var/lib/sieve - rm -v /var/lib/imap/db/* /var/lib/imap/db.backup1/* \ /var/lib/imap/db.backup2/* /var/lib/imap/deliver.db \ /var/lib/imap/tls_sessions.db /var/lib/imap/mailboxes.db - ctl_mboxlist -u /tmp/mboxlist.txt - ctl_cyrusdb -r - tls_prune - ctl_cyrusdb -c - cyr_expire -E 3 The last command took about 24 hours to complete and if understand right It converted cyrus mailbox DBs from v9 to v12. I wonder if the first cyr_expire could run even with the service started and if it should be better to run a reconstruct on all the mailboxes in place of this cyr_expire. Last but not least, do you think that sieve filters should be recompiled? Any other hints? Thanks Giuseppe Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Remove empty partition
Hi, I'm doing some fall cleaning on my imap server and now I've a couple of empty partitions because I moved the mails away. I would like to remove those partitions, it's safe removing the lines from imapd.conf and restarting cyrus? or this may impact some internal db? Thanks Giuseppe PS: I'm running cyrus 2.3.8 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Hide tree in one service
Hi, on my imap server I've a shared folders setup, for example: Shared_one/ Shared_one/Sub1 Shared_one/Sub2 Shared_two/Sub1 Shared_two/Sub2 user/* I would like to setup a different imap service on a different imap port where users can see only user subtree (alias they can only see their inbox) There is a way to setup such a service? Thanks Beppe Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
two imapd istances with different altnamespace
Hi, my mail server runs cyrus 2.3 with altnamespace: no parameter. We make extensive use of shared folders and we are happy with this setting and with personal folders unser INBOX. But now need to use a software that only understands alternative namespace and i was thinking about using two different imapd.conf files that differ only in the line: altnamespace:no altnamespace: yes in cyrus.conf i will put: imap cmd=imapd listen=imap prefork=0 imap2 cmd=imapd -C /etc/imapd2.conf listen=10143 prefork=0 It's possible to do this setting? And it's safe using different namespaces on the same mailstore? Thanks Giuseppe Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: two imapd istances with different altnamespace
Ouch... I missed that the doc says When configured to use one (or both) of the options below and i also find the right google keys to find out this post: http://blog.fastmail.fm/2008/08/11/alternate-namespace-imap-port-may-help-outlook-ol-express-apple-mail-and-bis-users/ So it seems that i can use both namespaces at the same time! About sieve scripts if my sieve cmd=timsieved listen=127.0.0.1:sieve prefork=0 points to the altnamespace: no config it seems to me that there is no need to convert all my users sieve scripts. I'm right? Thanks Giuseppe On 10/15/2010 03:32 PM, Dan White wrote: On 15/10/10 10:01 +0200, Giuseppe Ravasio wrote: Hi, my mail server runs cyrus 2.3 with altnamespace: no parameter. We make extensive use of shared folders and we are happy with this setting and with personal folders unser INBOX. But now need to use a software that only understands alternative namespace and i was thinking about using two different imapd.conf files that differ only in the line: altnamespace: no altnamespace: yes in cyrus.conf i will put: imap cmd=imapd listen=imap prefork=0 imap2 cmd=imapd -C /etc/imapd2.conf listen=10143 prefork=0 It's possible to do this setting? And it's safe using different namespaces on the same mailstore? The altnamespace doc says that should work, with the caveat that sieve scripts will need to be updated. http://www.cyrusimap.org/docs/cyrus-imapd/2.3.16/altnamespace.php Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Migration 32 to 64 bit
Hi, I'm planning the migration of our main cyrus server. Actually the server is running cyrus imap 2.2.3 on a SuSE 9.1 i586, with about 130Gb of mailboxes. My idea is moving to OpenSuse 10.3 with cyrus 2.3.8 on 64bit System. I googled a bit, but i couldn't find anything useful; i would like to know if there are issues moving from 32bit to 64bit and/or moving from 2.2.3 to 2.3.8 In particular i would like to minimize users impact, preserving mailstores, subscriptions and all seen status. Any hints/comments/precautions??? Thanks G.Ravasio Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Ghost emails
Alle 22:11, martedì 5 giugno 2007, Florian Gleixner ha scritto: As far as i understand cyrus imapd, it does not delete the message instantly. It marks the message and the /usr/lib/cyrus/bin/cyr_expire job deletes the message some days later. The delete job is configured in /etc/cyrus.conf # this is only necessary if using duplicate delivery suppression delprune cmd=cyr_expire -E 3 at=0400 deletes duplicte messages older than 3 days every day at 4am. But i did not yet fully understand the entire process. In the logs i see thoose lines every times a message is delivered (and i see this one, and every other message in my inbox!)... Jun 8 09:34:44 thot lmtpunix[10927]: accepted connection Jun 8 09:34:44 thot lmtpunix[10927]: lmtp connection preauth'd as postman Jun 8 09:34:44 thot lmtpunix[10927]: duplicate_check: [EMAIL PROTECTED] user.gravasio0 Jun 8 09:34:44 thot lmtpunix[10927]: mystore: starting txn 2147490592 Jun 8 09:34:44 thot lmtpunix[10927]: mystore: committing txn 2147490592 Jun 8 09:34:44 thot lmtpunix[10927]: duplicate_mark: [EMAIL PROTECTED] user.gravasio 1181288084 2913 Jun 8 09:34:44 thot lmtpunix[10927]: mystore: starting txn 2147490593 Jun 8 09:34:44 thot lmtpunix[10927]: mystore: committing txn 2147490593 Jun 8 09:34:44 thot lmtpunix[10927]: duplicate_mark: [EMAIL PROTECTED] [EMAIL PROTECTED] 1181288084 0 I think that the duplicate_mark is only a Debug message meaning that this ID is committed to the duplicate check DB. I'm wrong? Anyone could explain how duplicate check works? and/or how to read thoose log lines? Beppe Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Ghost emails
Alle 15:57, venerdì 8 giugno 2007, Scott M. Likens ha scritto:
Looks like one of those is running a sieve script, what is your current
script look like?
Yes...
my courrent sieve script is something like:
require
[fileinto,reject,vacation,imapflags,relational,comparator-i;ascii-numeric,regex,notify];
if
header :contains X-Spam-Flag YES
{
fileinto INBOX/Z_Spam;
stop;
}
the sieve script of the user with gost emails is:
if address :all :comparator i;ascii-casemap :is
[From, Sender, Resent-From] [[EMAIL PROTECTED], [EMAIL PROTECTED]] {
discard;
stop;
}
if address :all :comparator i;ascii-casemap :is
[From, Sender, Resent-From [[EMAIL PROTECTED], [EMAIL PROTECTED]] {
discard;
stop;
}
Beppe
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Ghost emails
Alle 07:30, martedì 5 giugno 2007, Florian Gleixner ha scritto: Your logs say: May 31 20:37:37 server lmtpunix[31094]: duplicate_mark: [EMAIL PROTECTED],it [EMAIL PROTECTED] this means, that cyrus thinks this message is a duplicate of a existing mail in the users mailbox. Some broken email clients may generate non-unique Message-Ids (groupwise has this bug in one version, some outlook versions, ...). Search your mail logs (/var/log/mail) whether you see the Id [EMAIL PROTECTED],it for different incoming mails. I think that when a message is marked as duplicate i should see a line like: Jun 5 09:11:24 thot lmtpunix[23224]: dupelim: eliminated duplicate message to user.beppe id [EMAIL PROTECTED] (delivery) in fact searching the id in a month logs returns only the lines included in my first email... so it seems that there was no duplicate suppression. Last but not least it doesn't explain why the message is on the FS but not in the IMAP folder's index! Giuseppe Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Ghost emails
Hi, I'm running a Postfix+Cyrus+Ldap Auth mail (IMAP) server on a Suse 9.1 with about 200 users. The software versions are: cyrus-imapd-2.2.3-83.22 postfix-2.1.1-1.20 It's working fine since 2005, but now i'm facing a strange problem on a user mailbox. An email is received by the MTA, then postfix delivers this email to CyrusImap with /usr/lib/cyrus/bin/deliver. From the /var/log/messages i see: May 31 20:37:37 server lmtpunix[31094]: accepted connection May 31 20:37:37 server lmtpunix[31094]: lmtp connection preauth'd as postman May 31 20:37:37 server lmtpunix[31094]: duplicate_check: [EMAIL PROTECTED] user.beppe 0 May 31 20:37:37 server lmtpunix[31094]: mystore: starting txn 2147483677 May 31 20:37:37 server lmtpunix[31094]: mystore: committing txn 2147483677 May 31 20:37:37 server lmtpunix[31094]: duplicate_mark: [EMAIL PROTECTED] user.beppe 1180636657 43699 May 31 20:37:37 server lmtpunix[31094]: mystore: starting txn 2147483678 May 31 20:37:37 server lmtpunix[31094]: mystore: committing txn 2147483678 May 31 20:37:37 server lmtpunix[31094]: duplicate_mark: [EMAIL PROTECTED],it [EMAIL PROTECTED] 1180636657 0 So it seems that the message was correctly delivered, and in fact in the user FS folder there is a file containing the email! But this email doesn't appear via IMAP, and the only way to make it visible was: reconstruct -f user/beppe It isn't the first time that i've this strange behaviour, and since today i saw this issue only on a single user folder. Any ideas? Thanks G.Ravasio Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
