Re: IMAP processes out of control

[Shaheen Bakhtiar]

2 x AMD quad Core 64bit 
4G RAM

This morning I woke up to a plethora of complaints that people were not able to 
access their emails. I remove the aforementioned maxchild from the 
configurations and restart to server. Once I did that people were able to 
re-connect with no problems.

I did not have these types of problems with the older version (I believe was 
2.3.19). Just since I upgraded to the latest version of Cyrus. 

Current version is:
[root@postoffice ~]# dnf info cyrus-imapd
Last metadata expiration check performed 1:06:02 ago on Wed Sep 23 07:12:41 
2015.
Installed Packages
Name: cyrus-imapd
Arch: x86_64
Epoch   : 0
Version : 2.4.17
Release : 9.fc22

Running on Fedora Core 22 64bit

> On Sep 23, 2015, at 7:44 AM, signaldevelo...@gmail.com wrote:
> 
> Again this is active sync devices that are connecting with a ton of pushed 
> folders. The more you tell it to sync (folders) the more processes it's going 
> to fork for each user folder. Is this affecting performance that bad? What's 
> your hardware? 
> 
> - Paul
> 
>> On Sep 22, 2015, at 7:43 PM, Moby <m...@mobsternet.com> wrote:
>> 
>> 
>> 
>> On 9/22/2015 18:12, Shaheen Bakhtiar wrote:
>>>> On Sep 22, 2015, at 2:17 PM, Andrew Morgan <mor...@orst.edu> wrote:
>>>> 
>>>>> On Tue, 22 Sep 2015, Shaheen Bakhtiar wrote:
>>>>> 
>>>>> It happened again….. although it took longer for it to happen, this has 
>>>>> been happening only since the upgrade in Jun.
>>>>> 
>>>>> The number of imap processes continues to increase until the server is 
>>>>> completely OOM. the increase is drastic and all of a sudden.
>>>> You should probably set maxchild to a value that won't run your server out 
>>>> of memory.  :)
>>>> 
>>>> Have you looked at the processes to see what they have in common?  For 
>>>> example, sometimes an IMAP client will run amok and make hundreds or 
>>>> thousands of connections.  Or perhaps the processes are all stuck waiting 
>>>> on a lock, etc.
>>>> 
>>>> lsof, strace, netstat, and your Cyrus logs can help a lot.
>>>> 
>>>>   Andy
>>> 
>>> 
>>> [shawn@postoffice ~]$ ps aux | grep imapd | wc -l
>>> 255
>>> [shawn@postoffice ~]# ps aux | grep imapds | wc -l
>>> 1
>>> [shawn@postoffice ~]# ps aux | grep pop3d | wc -l
>>> 9
>>> [shawn@postoffice ~]# ps aux | grep timseived | wc -l
>>> 1
>>> [shawn@postoffice ~]# ps aux | grep lmtpunix | wc -l
>>> 1
>>> 
>>> Based on that output I changed the configuration file (below) adding 
>>> maxchild. Most likely all my users have their clients open, and from 
>>> previous monitoring I average about 200 instances of imapd:
>>> 
>>> # standard standalone server implementation
>>> 
>>> START {
>>>  # do not delete this entry!
>>>  recover   cmd="ctl_cyrusdb -r"
>>> 
>>>  # this is only necessary if using idled for IMAP IDLE
>>>  idled cmd="idled"
>>> }
>>> 
>>> # UNIX sockets start with a slash and are put into /var/lib/imap/sockets
>>> SERVICES {
>>>  # add or remove based on preferences
>>>  imap  cmd="imapd" listen="imap" prefork=5 maxchild=300
>>>  imaps cmd="imapd -s" listen="imaps" prefork=1 maxchild=100
>>>  pop3  cmd="pop3d" listen="pop3" prefork=3 maxchild=5
>>>  pop3s cmd="pop3d -s" listen="pop3s" prefork=1 maxchild=5
>>>  sieve cmd="timsieved" listen="sieve" prefork=0
>>> 
>>>  # these are only necessary if receiving/exporting usenet via NNTP
>>> #  nntp cmd="nntpd" listen="nntp" prefork=3
>>> #  nntpscmd="nntpd -s" listen="nntps" prefork=1
>>> 
>>>  # at least one LMTP is required for delivery
>>> #  lmtp cmd="lmtpd" listen="lmtp" prefork=0
>>>  lmtpunix  cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1
>>> 
>>>  # this is only necessary if using notifications
>>> #  notify   cmd="notifyd" listen="/var/lib/imap/socket/notify" 
>>> proto="udp" prefork=1
>>> }
>>> 
>>> EVENTS {
>>>  # this is required
>>>  checkpointcmd="ctl_cy

Re: IMAP processes out of control

[Shaheen Bakhtiar]

> On Sep 22, 2015, at 2:17 PM, Andrew Morgan <mor...@orst.edu> wrote:
> 
> On Tue, 22 Sep 2015, Shaheen Bakhtiar wrote:
> 
>> 
>> It happened again….. although it took longer for it to happen, this has been 
>> happening only since the upgrade in Jun.
>> 
>> The number of imap processes continues to increase until the server is 
>> completely OOM. the increase is drastic and all of a sudden.
> 
> You should probably set maxchild to a value that won't run your server out of 
> memory.  :)
> 
> Have you looked at the processes to see what they have in common?  For 
> example, sometimes an IMAP client will run amok and make hundreds or 
> thousands of connections.  Or perhaps the processes are all stuck waiting on 
> a lock, etc.
> 
> lsof, strace, netstat, and your Cyrus logs can help a lot.
> 
>   Andy



[shawn@postoffice ~]$ ps aux | grep imapd | wc -l
255
[shawn@postoffice ~]# ps aux | grep imapds | wc -l
1
[shawn@postoffice ~]# ps aux | grep pop3d | wc -l
9
[shawn@postoffice ~]# ps aux | grep timseived | wc -l
1
[shawn@postoffice ~]# ps aux | grep lmtpunix | wc -l
1

Based on that output I changed the configuration file (below) adding maxchild. 
Most likely all my users have their clients open, and from previous monitoring 
I average about 200 instances of imapd:

# standard standalone server implementation

START {
  # do not delete this entry!
  recover   cmd="ctl_cyrusdb -r"

  # this is only necessary if using idled for IMAP IDLE
  idled cmd="idled"
}

# UNIX sockets start with a slash and are put into /var/lib/imap/sockets
SERVICES {
  # add or remove based on preferences
  imap  cmd="imapd" listen="imap" prefork=5 maxchild=300
  imaps cmd="imapd -s" listen="imaps" prefork=1 maxchild=100
  pop3  cmd="pop3d" listen="pop3" prefork=3 maxchild=5
  pop3s cmd="pop3d -s" listen="pop3s" prefork=1 maxchild=5
  sieve cmd="timsieved" listen="sieve" prefork=0

  # these are only necessary if receiving/exporting usenet via NNTP
#  nntp cmd="nntpd" listen="nntp" prefork=3
#  nntpscmd="nntpd -s" listen="nntps" prefork=1

  # at least one LMTP is required for delivery
#  lmtp cmd="lmtpd" listen="lmtp" prefork=0
  lmtpunix  cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1

  # this is only necessary if using notifications
#  notify   cmd="notifyd" listen="/var/lib/imap/socket/notify" proto="udp" 
prefork=1
}

EVENTS {
  # this is required
  checkpointcmd="ctl_cyrusdb -c" period=30

  # this is only necessary if using duplicate delivery suppression,
  # Sieve or NNTP
  delprune  cmd="cyr_expire -E 3" at=0400

  # this is only necessary if caching TLS sessions
  tlsprune  cmd="tls_prune" at=0400
}



Comments, concerns??? I’m going to keep observium open on a separate screen and 
watch when it starts going up, than do the lsof,netstat, and watch logs to see 
if I can tell why the sudden upticks.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus mailbox authentication changing from NIS to LDAP

[Shaheen Bakhtiar]

using linux you can run autconfig with the varying options to enable sssd with 
the appropriate settings 

THIS IS ONLY AN EXAMPLE YOU’LL WANT TO TAKE APPROPRIATE SECURITY MEASURES SUCH 
AS TLS ETC.. but you can test this way first. 

IE:
authconfig --enablesssd --enablesssdauth --enablelocauthorize --enableldap 
--enableldapauth --ldapserver=ldap://ldap.example.com:389 --disableldaptls 
--ldapbasedn=dc=example,dc=com --enablerfc2307bis --enablemkhomedir 
--enablecachecreds —update

Or you can directly adit your sssd config /etc/ssd/ssd.conf:

[sssd]
domains = default, LDAP
services = nss, pam, autofs
config_file_version = 2

[nss]
filter_groups = root
filter_users = root

[pam]

[domain/LDAP]
#debug_level = 9
ldap_tls_reqcert = never
auth_provider = ldap
id_provider = ldap
chpass_provider = ldap
ldap_schema = rfc2307bis
ldap_uri = ldap://ldap.example.com
ldap_search_base = dc=example,dc=com
cache_credentials = false
enumerate = False

Verify that PAM actually uses SSSD:

By enabling debug_level in the above file you can also look at /var/log/sssd 
files for more details on where (if any) auth is failing.

[root@postoffice ~]# more /etc/pam.d/system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
authrequired  pam_env.so
authsufficientpam_fprintd.so
auth[default=1 success=ok] pam_localuser.so
auth[success=done ignore=ignore default=die] pam_unix.so nullok 
try_first_pass
authrequisite pam_succeed_if.so uid >= 1000 quiet_success
authsufficientpam_sss.so forward_pass
authrequired  pam_deny.so

account required  pam_unix.so broken_shadow
account sufficientpam_localuser.so
account sufficientpam_succeed_if.so uid < 1000 quiet
account [default=bad success=ok user_unknown=ignore] pam_sss.so
account required  pam_permit.so

passwordrequisite pam_pwquality.so try_first_pass local_users_only 
retry=3 authtok_type=
passwordsufficientpam_unix.so sha512 shadow nullok try_first_pass 
use_authtok
passwordsufficientpam_sss.so use_authtok
passwordrequired  pam_deny.so

session optional  pam_keyinit.so revoke
session required  pam_limits.so
-session optional  pam_systemd.so
session optional  pam_oddjob_mkhomedir.so skel=/etc/skel/ umask=0022
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet 
use_uid
session required  pam_unix.so
session optional  pam_sss.so


You should be set.



> On Sep 18, 2015, at 7:48 AM, Sunny  wrote:
> 
> Hi, 
> 
> I've inherited a cyrus mail server and I'm currently learning how it's setup 
> and would like some advice changing from a NIS to LDAP authentication.
> 
> At the moment, the imap server uses NIS to authenticate ssh connections and I 
> believe to also authenticate users to their mailboxes 
> 
> imapd.conf
> sasl_pwcheck_method: saslauthd
> sasl_mech_list: PLAIN
> 
> /etc/sysconfig/saslauthd 
> MECH=pam
> 
> From the above output I believe that cyrus will use the pam service to lookup 
> authentication information to authenticate a users cyrus mailbox.
> 
> I want the imap server to use LDAP (via sssd) for ssh authentication and 
> authenticating users to their mailboxes.
> 
> If I configure the mail server to use sssd (also stop NIS) and update 
> /etc/pam.d/system-auth with the required pam_sss.so entries, does anyone know 
> or have experience if this change will allow users to authenticate to their 
> mailboxes using LDAP?
> 
> Regards
> 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Problems with port 993 (SSL)

[Shaheen Bakhtiar]

> On Sep 1, 2015, at 8:35 AM, Patrick Boutilier  wrote:
> 
> On 09/01/2015 12:26 PM, Paul van der Vlis wrote:
>> Hello,
>> 
>> Since yesterday I get phone calls from Apple users about port 993 not
>> working anymore. Some other users don't have problems using port 993.
>> 
>> No problems on port 143 with starttls.
>> 
>> I've tested it myself on Linux, same problem.
>> Starttls works, but SSL/TLS does not.
>> 
>> I think it has to do with security problems in the SSL-protocol, and
>> updates in mailclients. Is here someone who knows more?
>> 
>> My mail setup is a bit old (Debian Squeeze), I am testing a new setup at
>> the moment with Debian Jessie.
>> 
>> With regards,
>> Paul van der Vlis.
>> 
>> 
>> 
> 
> What version of Cyrus?
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Ran into a similar problem when users upgraded to OS X 10.10.4 on the SMTP side 
of the equation. Perhaps you are running into it on the IMAP side??

Apple (as Apple does often) simply changed implementation (without much notice) 
no longer excepting “weak” public key exchange. In order for Mail.app to work 
you need to use a Diffie–Hellman key greater than 1024. I switched ours to 2048 
just to be safe. 

However, we never had issues with IMAP only SMTP.

Discussions:
https://discussions.apple.com/thread/7104500 

https://discussions.apple.com/thread/7105583 


Solutions was found:
https://discussions.apple.com/thread/7104500?start=60=0 


Hope this helps :)
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Number of imap process increasing over time

[Shaheen Bakhtiar]

Good morning,

Rebuilt our IMAP server from scratch using Cyrus 2.4.17 on FC22 x86_64. The 
server is a single process 2.3GH 8 core AMD 64bit with 4G of memory.

Ever since the rebuild we are experience an ever growing number of imapd 
processes, when we first boot the server we have ~200 using 2.4G of memory. In 
about 3 to 4 days we have ~1500 imapd processes taking up all available 
physical memory and all all available swap memory (an additional 4G).

and our logs are filled with messages like:
Aug 14 06:26:01 postoffice kernel: Out of memory: Kill process 15427 (imapd) 
score 1 or sacrifice child
Aug 14 06:26:01 postoffice kernel: Killed process 15427 (imapd) 
total-vm:179648kB, anon-rss:7756kB, file-rss:672kB

SNMP Graphs of what’s happening:
https://www.dropbox.com/s/7bnu82h54alrzkz/postoffice%20OOM.png?dl=0

I’m not ruling out a DoS, but the logs from the tcpdump (currently) don’t 
support that, neither do the SNMP from the router. Checking the IDS, but I find 
it more than coincidental that this started happening after we upgraded. We had 
no such issues prior to the upgrade.

Any ideas would be greatly appreciated, 
Shawn


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Number of imap process increasing over time

[Shaheen Bakhtiar]

 On Aug 14, 2015, at 8:03 AM, Dan White dwh...@olp.net wrote:
 
 On 08/14/15 07:46 -0700, Shaheen Bakhtiar wrote:
 Rebuilt our IMAP server from scratch using Cyrus 2.4.17 on FC22 x86_64.
 The server is a single process 2.3GH 8 core AMD 64bit with 4G of memory.
 
 Ever since the rebuild we are experience an ever growing number of imapd
 processes, when we first boot the server we have ~200 using 2.4G of
 memory. In about 3 to 4 days we have ~1500 imapd processes taking up all
 available physical memory and all all available swap memory (an additional
 4G).
 
 and our logs are filled with messages like:
 Aug 14 06:26:01 postoffice kernel: Out of memory: Kill process 15427 (imapd) 
 score 1 or sacrifice child
 Aug 14 06:26:01 postoffice kernel: Killed process 15427 (imapd) 
 total-vm:179648kB, anon-rss:7756kB, file-rss:672kB
 
 How many processes spawn is configurable within /etc/cyrus.conf. How do you
 have your imap entries configured?
 
 -- 
 Dan White

[shawn@postoffice ~]$ more /etc/cyrus.conf 
# standard standalone server implementation

START {
  # do not delete this entry!
  recover   cmd=ctl_cyrusdb -r

  # this is only necessary if using idled for IMAP IDLE
  idled cmd=idled
}

# UNIX sockets start with a slash and are put into /var/lib/imap/sockets
SERVICES {
  # add or remove based on preferences
  imap  cmd=imapd listen=imap prefork=5
  imaps cmd=imapd -s listen=imaps prefork=1
  pop3  cmd=pop3d listen=pop3 prefork=3
  pop3s cmd=pop3d -s listen=pop3s prefork=1
  sieve cmd=timsieved listen=sieve prefork=0

  # these are only necessary if receiving/exporting usenet via NNTP
#  nntp cmd=nntpd listen=nntp prefork=3
#  nntpscmd=nntpd -s listen=nntps prefork=1

  # at least one LMTP is required for delivery
#  lmtp cmd=lmtpd listen=lmtp prefork=0
  lmtpunix  cmd=lmtpd listen=/var/lib/imap/socket/lmtp prefork=1

  # this is only necessary if using notifications
#  notify   cmd=notifyd listen=/var/lib/imap/socket/notify proto=udp 
prefork=1
}

EVENTS {
  # this is required
  checkpointcmd=ctl_cyrusdb -c period=30

  # this is only necessary if using duplicate delivery suppression,
  # Sieve or NNTP
  delprune  cmd=cyr_expire -E 3 at=0400

  # this is only necessary if caching TLS sessions
  tlsprune  cmd=tls_prune at=0400
}


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: migration seen from cyrus 2.3.16 to 2.4.17

[Shaheen Bakhtiar]

Out of curiosity….

What is the difference between 2.4 and 2.5 (why are there three versions all 
being updated?).

I’m upgrading our servers to FC 22 and they only come with the 2.4.18 not 2.5.

Does anyone know if the 2.4.18 have the CalDAV module in it? or do I have to 
download source and build the 2.5?

On Jul 7, 2015, at 1:38 AM, Bron Gondwana br...@fastmail.fm wrote:

 On Tue, Jul 7, 2015, at 04:23 PM, Christoph Moench-Tegeder wrote:
 ## Frank Patzig (f...@mdlink.de):
 
 In the mailclient are the mails unseen. The data migration from *.seen
 to cyrus.index not work. What can i do? In strace not found not data und
 in the logging not importent informations.
 
 I believe you missed -V max to reconstruct. (And why not upgrade to 2.5?).
 
 There is no -V max on 2.4.  Maybe those seen files weren't there when it 
 upgraded from 2.3 to 2.4, in which case you're kinda stuck, because you're 
 not supposed to copy the seen files across later.
 
 (upgrading to 2.5 won't help there either)
 
 Bron.
 -- 
  Bron Gondwana
  br...@fastmail.fm
 
 Cyrus Home Page: http://www.cyrusimap.org/
 List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
 To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Documentation update

[Shaheen Bakhtiar]

Hi all,

Not sure where to report this the #irc channel recommended I reach out here.

The Fedora documentation bellow seems out of date as of at least Fedora Core 18 
(Currently 22) .

https://docs.cyrus.foundation/imap/installation/distributions/fedora.html


yum has been replaced with dnf as of FC18[1]

yum install cyrus-imapd cyrus-sasl cyrus-sasl-plain
should be
dnf install cyrus-imapd cyrus-sasl cyrus-sasl-plain

and service and chkconfig have been replaced with systemctl[2]

# service saslauthd start
Starting saslauthd:[  OK  ]
# chkconfig saslauthd on
should be

# systemctl enable cyrus-imapd
# systemctl start cyrus-imapd

I’m going to be upgrading / re-installing a FC22 server, I’ll be more than 
happy to update the documents or submit updates as needed. Just let me know 


[1] https://fedoraproject.org/wiki/Features/DNF
[2] https://fedoraproject.org/wiki/Systemd


On Jul 2, 2015, at 3:59 PM, Bron Gondwana br...@fastmail.fm wrote:

 
 
 On Fri, Jul 3, 2015, at 08:52 AM, Bron Gondwana wrote:
 
 
 On Thu, Jul 2, 2015, at 12:25 PM, Shaw, Brian wrote:
 Jul  1 21:52:15 mail-server postfix/lmtp[12036]: EB0C1E5B11: 
 to=u...@example.com, relay=mail-server[/var/lib/imap/socket/lmtp], 
 delay=32177, delays=32169/6/2.6/0.02, dsn=4.3.0, status=deferred (host 
 mail-server[/var/lib/imap/socket/lmtp] said: 421 4.3.0 lmtpd: Internal 
 error: assertion failed: lib/cyrusdb_twoskip.c: 600: record-level = 
 MAXLEVEL (in reply to end of DATA command))
 
 Here's your problem!  That's a corrupted database file.  Almost certainly 
 your mailboxes.db.
 
 You're running Cyrus 2.5.x, obviously.
 
 I haven't seen any twoskip errors in the wild for a long time, so I'd love 
 to see a copy of your mailboxes.db if you don't mind.  I guess it could also 
 be a .seen file or the annotations.db, the error doesn't give enough 
 information to know :(
 
 (in answer to _that_ part of the problem at least, I've written a patch to 
 twoskip.c which will give better error messages on reading a corrupted DB 
 from disk.  It will be in the next stable 2.5 release)
 
 Bron.
 
 -- 
  Bron Gondwana
  br...@fastmail.fm
 
 Cyrus Home Page: http://www.cyrusimap.org/
 List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
 To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus mailboxes format

[Shaheen Bakhtiar]

Not sure what the size the user base is, but one strategy that helps cleanup 
the mail boxes in the process, is to simply leave the old mail server in place, 
create new accounts on each client to the new server, and put out a memo 
stating that the old server will be up for X months, showing users how to 
drag-n-drop the old email they need/want to the new server.

This will keep the new mail server stores “clean, users will only drag-n-drop 
the emails they need/want to keep.

I find that most users actually appreciate the ability to clean house :)


On Mar 4, 2015, at 7:59 AM, Andres Tarallo atara...@acm.org wrote:

 Hi 
 
 We're about to migrate a very old installation based on cyrus 2.3.11.  The 
 target system is zimbra collaboration. Many users have years of mail stored 
 in their maiboxes, we want to migrate those mails to the nuew server. 
 
 I have the idea to do so with the aid of lmtpinject, a tool provided by 
 zimbra. This tool insert into the mailboxes mails in maildir format. I need 
 to confirm that cyrus stores mails in that format. 
 
 Thanks
 
 Andrés 
 
 Cyrus Home Page: http://www.cyrusimap.org/
 List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
 To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus