Re: cyrus-imapd not starting after upgrade

[Andrew Morgan]

On Tue, 15 Jan 2019, Daniel Bareiro wrote:


Hi all!

After quite some time, today I decided to update the mail server from
Debian Jessie (cyrus-imapd 2.4.17) to Debian Stretch (cyrus-imapd 2.5.10-3).

All without problems until I reach the part of cyrus-imapd that does not
start. This is what I see in the log:

--
Jan 14 23:10:45 mail systemd[1]: Started Cyrus IMAP/POP3 daemons.
Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: skiplist: clean shutdown
file missing, updating recovery stamp
Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: recovering cyrus databases
Jan 14 23:10:45 mail cyrus/ctl_cyrusdb[5318]: done recovering cyrus
databases
Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: Repacking mailbox
user.admin.TareasCron version 12
Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: Expired 0 and expunged 0
out of 28809 messages from 80 mailboxes
Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: duplicate_prune: pruning
back 3.00 days
Jan 14 23:10:46 mail cyrus/cyr_expire[5332]: duplicate_prune: purged 0
out of 438 entries
Jan 14 23:10:46 mail cyrus/tls_prune[5335]: twoskip: invalid magic
header: /var/lib/cyrus/tls_sessions.db
Jan 14 23:10:46 mail cyrus/tls_prune[5335]: cyrusdb: opening
/var/lib/cyrus/tls_sessions.db with backend skiplist (requested twoskip)
Jan 14 23:10:46 mail cyrus/tls_prune[5335]: skiplist: recovered
/var/lib/cyrus/tls_sessions.db (223 records, 41200 bytes) in 0 seconds
Jan 14 23:10:46 mail cyrus/tls_prune[5335]: skiplist: checkpointed
/var/lib/cyrus/tls_sessions.db (223 records, 41200 bytes) in 0.091 sec
Jan 14 23:10:46 mail cyrus/tls_prune[5335]: tls_prune: purged 2 out of
223 entries
Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for
service 'nntp'
Jan 14 23:10:46 mail cyrus/master[5311]: exiting
Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Main process
exited, code=exited, status=78/n/a
Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Unit entered
failed state.
Jan 14 23:10:46 mail systemd[1]: cyrus-imapd.service: Failed with result
'exit-code'.
--

I'm not sure what the problem is but that "invalid magic header" makes
me think that maybe it changed the header format of
/var/lib/cyrus/tls_sessions.db and the migration process did not do the
corresponding conversion. Can that be the reason why it doesn't start or
I'm missing something else? Any ideas that can bring more light?

The associated problem is that because of this it seems that Postfix can
not deliver the mails since there is no /var/run/cyrus/socket/lmtp.


It wants tls_sessions.db to be a twoskip-format file, but the current 
format is skiplist.  However, it was able to detect this and open it as 
skiplist.  You can fix this issue by stopping Cyrus, removing 
tls_sessions.db, and starting Cyrus.


However, your real problem seems to be the missing nntp executable:

Jan 14 23:10:46 mail cyrus/master[5311]: cannot find executable for service 
'nntp'

Do you use NNTP?  You could comment it out of cyrus.conf in order to get 
the rest of Cyrus up and running.


Take a look at the release notes for v2.5.0:

  https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.0.html

It covers important changes from v2.4 to v2.5.  You may need to update 
your cyrus.conf and imapd.conf files.


Thanks,
Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Backup methods

[Andrew Morgan]

On Fri, 11 May 2018, Anatoli wrote:


There may be an argument that could be made for 2 backup stratagies


That's the point. In the context of SME environments (Small and Medium-sized 
Enterprises, i.e. from 5 to 50 employees normally, up to 250 in some 
countries) that we were talking about, a replication is an overkill, IMO. But 
for large enterprises like MNCs, large universities, public mail providers 
(Fastmail) of course multiple masters and backups via replication is the way 
to go. For large deployments there are good backup solutions in Cyrus, but 
for the small businesses admins I don't know any to recommend.


Anatoli,

I think you're making this harder than it needs to be...

For a small system with a few hundred mailboxes, a simple unix filesystem 
backup is sufficient.  You can dump the Cyrus mailboxes.db to a flat file 
every hour with cron (keep a few days worth).  Backup everything with your 
regular backup system (tar, rsync, etc).


If you suffer a complete loss of the system and have to restore from the 
backup, you won't care much about a few database file inconsistencies, 
which can be repaired with Cyrus' reconstruct tool.  You would recover the 
whole backup, recover mailboxes.db from the most recent flat file export, 
and then run reconstruct on every mailbox.


If you need to recover some messages or mailboxes that were deleted by a 
user, then just recover those individual files or directories from you 
backup.  Run reconstruct -rf on the mailbox.


Naturally, delayed expunge and delayed delete are fantastic ways to avoid 
all this work.  Purge them only after a few weeks or a month has passed. 
It is much easier to restore using those delayed delete/expunge features.



Thanks,
Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Can't authorize as different user in cyradm and sieveshell

[Andrew Morgan via Info-cyrus]

I'm using Debian packages for sasl.  Here is what libsasl2-modules 
includes:


/usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
/usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
/usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
/usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
/usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
/usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25

But in my imapd.conf, I'm not specifying an auxprop plugins:

# grep sasl /etc/imapd.conf
sasl_mech_list: PLAIN
sasl_minimum_layer: 0
#sasl_maximum_layer: 256
sasl_pwcheck_method: saslauthd

Since we are using saslauthd, we don't use auxprop plugins, I think...

Andy

On Mon, 21 Nov 2016, Michael Ulitskiy wrote:


I'm trying to read the code and it seems that it tries to lookup authorization 
id
in auxprop plugin. since I don't have any auxprop plugins that returns 
SASL_NOMECH and results
in the error I'm seeing.

By any chance do you have any auxprop plugin defined?

On Monday, November 21, 2016 10:07:23 AM Andrew Morgan wrote:

Maybe there is something wrong with your saslauthd parameters or PAM
config?

Here is what I use:

saslauthd -a pam -c -t 300 -m /var/run/saslauthd -n 5

# cat /etc/pam.d/sieve
# PAM configuration file for Cyrus IMAP service

authsufficient  pam_ldap.so
authrequiredpam_unix.so

account sufficient  pam_ldap.so
account requiredpam_unix.so


(pretty simple!)

In your original email, you showed that you could authenticate as the
target user successfully.  Can you connect to sieve as the admin user (no
proxy-auth)?

Thanks,
Andy


On Mon, 21 Nov 2016, Michael Ulitskiy wrote:


Andrew,

Thanks for the reply. It's good to know it works for someone.
I've tried to downgrade cyrus to 2.4.18, but that didn't help.
sivtest doesn't provide much clue:

root@rway-imap-vm:~# sivtest -a proxyadmin -u t...@virtualcrap.com localhost
S: "IMPLEMENTATION" "Cyrus timsieved v2.4.18"
S: "SASL" "PLAIN"
S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify 
envelope imap4flags relational regex subaddress copy"
S: "UNAUTHENTICATE"
S: OK
Please enter your password:
C: AUTHENTICATE "PLAIN" {48+}

S: NO "Authentication Error"
Authentication failed. generic failure
Security strength factor: 0

while log is saying:
Nov 21 12:01:57 rway-imap-vm saslauthd[1169]: pam_userdb(sieve:auth): user 
'proxyadmin' granted access
Nov 21 12:01:57 rway-imap-vm sieve[21483]: badlogin: localhost[127.0.0.1] PLAIN 
no mechanism available

the same happens if I use admin user.
i also tried to change to sasl_pwcheck_method to 'alwaystrue' to make sure no 
authentication problems stand in the way, but that also didn't help.
I'm at loss now. Anymore troubleshooting clues?

Thanks,
Michael

On Sunday, November 20, 2016 07:34:58 PM Andrew Morgan wrote:

This works for me under v2.4.18.  I'm able to run sieveshell against a
frontend or backend authenticating as a cyrus "admins" user or a
"proxyservers" user (on the backend).

Against a frontend:

# sieveshell -u morgan -a cyrus imap.onid.oregonstate.edu
connecting to imap.onid.oregonstate.edu
Please enter your password:

list

onid-web
real  <- active script

quit



Against a backend:

# sieveshell -u morgan -a cyr_proxy cyrus-be1.onid.oregonstate.edu
connecting to cyrus-be1.onid.oregonstate.edu
Please enter your password:

list

onid-web
real  <- active script

quit



My imapd.conf settings:

admins: cyrus
allowplaintext: 0
sasl_mech_list: PLAIN
sasl_minimum_layer: 0
sasl_pwcheck_method: saslauthd
sieve_allowreferrals: 0
sieve_allowplaintext: 1


Have you tried using the "sivtest" program?  It will show you the protocol
handshakes, which might help.  Here is an example for me:

# sivtest -u morgan -a cyrus localhost
S: "IMPLEMENTATION" "Cyrus timsieved (Murder) v2.4.18"
S: "SASL" "PLAIN"
S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags
notify envelope body relational regex subaddress copy"
S: "STARTTLS"
S: "UNAUTHENTICATE"
S: OK
Please enter your password:
C: AUTHENTICATE "PLAIN" {28+}

S: OK
Authenticated.
Security strength factor: 0
C: LOGOUT
OK "Logout Complete"
Connection closed.


Andy

On Sun, 20 Nov 2016, Michael Ulitskiy via Info-cyrus wrote:


Since nobody answered, I guess, nobody has any idea.
I wonder if anybody uses this feature and it works for you?
I mean I'd like to know if that's just me and something is wrong with my setup 
or may be that feature isn't functional at all?
Thanks in advance,

Michael

On Thursday, November 17, 2016 06:30:18 PM Michael Ulitskiy via Info-cyrus 
wrote:

Hello,

I'm playing with cyrus-imap 2.5.10 and cyrus-sasl 2.1.26.
i'm trying to use sieveshell to setup users sieve scripts, but since
i don't know users passw

Re: Can't authorize as different user in cyradm and sieveshell

[Andrew Morgan via Info-cyrus]

Maybe there is something wrong with your saslauthd parameters or PAM 
config?


Here is what I use:

saslauthd -a pam -c -t 300 -m /var/run/saslauthd -n 5

# cat /etc/pam.d/sieve
# PAM configuration file for Cyrus IMAP service

authsufficient  pam_ldap.so
authrequiredpam_unix.so

account sufficient  pam_ldap.so
account requiredpam_unix.so


(pretty simple!)

In your original email, you showed that you could authenticate as the 
target user successfully.  Can you connect to sieve as the admin user (no 
proxy-auth)?


Thanks,
Andy


On Mon, 21 Nov 2016, Michael Ulitskiy wrote:


Andrew,

Thanks for the reply. It's good to know it works for someone.
I've tried to downgrade cyrus to 2.4.18, but that didn't help.
sivtest doesn't provide much clue:

root@rway-imap-vm:~# sivtest -a proxyadmin -u t...@virtualcrap.com localhost
S: "IMPLEMENTATION" "Cyrus timsieved v2.4.18"
S: "SASL" "PLAIN"
S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify 
envelope imap4flags relational regex subaddress copy"
S: "UNAUTHENTICATE"
S: OK
Please enter your password:
C: AUTHENTICATE "PLAIN" {48+}

S: NO "Authentication Error"
Authentication failed. generic failure
Security strength factor: 0

while log is saying:
Nov 21 12:01:57 rway-imap-vm saslauthd[1169]: pam_userdb(sieve:auth): user 
'proxyadmin' granted access
Nov 21 12:01:57 rway-imap-vm sieve[21483]: badlogin: localhost[127.0.0.1] PLAIN 
no mechanism available

the same happens if I use admin user.
i also tried to change to sasl_pwcheck_method to 'alwaystrue' to make sure no 
authentication problems stand in the way, but that also didn't help.
I'm at loss now. Anymore troubleshooting clues?

Thanks,
Michael

On Sunday, November 20, 2016 07:34:58 PM Andrew Morgan wrote:

This works for me under v2.4.18.  I'm able to run sieveshell against a
frontend or backend authenticating as a cyrus "admins" user or a
"proxyservers" user (on the backend).

Against a frontend:

# sieveshell -u morgan -a cyrus imap.onid.oregonstate.edu
connecting to imap.onid.oregonstate.edu
Please enter your password:

list

onid-web
real  <- active script

quit



Against a backend:

# sieveshell -u morgan -a cyr_proxy cyrus-be1.onid.oregonstate.edu
connecting to cyrus-be1.onid.oregonstate.edu
Please enter your password:

list

onid-web
real  <- active script

quit



My imapd.conf settings:

admins: cyrus
allowplaintext: 0
sasl_mech_list: PLAIN
sasl_minimum_layer: 0
sasl_pwcheck_method: saslauthd
sieve_allowreferrals: 0
sieve_allowplaintext: 1


Have you tried using the "sivtest" program?  It will show you the protocol
handshakes, which might help.  Here is an example for me:

# sivtest -u morgan -a cyrus localhost
S: "IMPLEMENTATION" "Cyrus timsieved (Murder) v2.4.18"
S: "SASL" "PLAIN"
S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags
notify envelope body relational regex subaddress copy"
S: "STARTTLS"
S: "UNAUTHENTICATE"
S: OK
Please enter your password:
C: AUTHENTICATE "PLAIN" {28+}

S: OK
Authenticated.
Security strength factor: 0
C: LOGOUT
OK "Logout Complete"
Connection closed.


Andy

On Sun, 20 Nov 2016, Michael Ulitskiy via Info-cyrus wrote:


Since nobody answered, I guess, nobody has any idea.
I wonder if anybody uses this feature and it works for you?
I mean I'd like to know if that's just me and something is wrong with my setup 
or may be that feature isn't functional at all?
Thanks in advance,

Michael

On Thursday, November 17, 2016 06:30:18 PM Michael Ulitskiy via Info-cyrus 
wrote:

Hello,

I'm playing with cyrus-imap 2.5.10 and cyrus-sasl 2.1.26.
i'm trying to use sieveshell to setup users sieve scripts, but since
i don't know users passwords i want to use a special user for authentication
and authorize as the target user.
Here's what I have.

imapd.conf:
admins: mailadmin
proxyservers: proxyadmin
sasl_pwcheck_method: saslauthd
#sasl_pwcheck_method: alwaystrue
sasl_mech_list: PLAIN
allowplaintext: yes

here's what i do:

root@rway-imap-vm:~# sieveshell -a proxyadmin -u t...@virtualcrap.com localhost
connecting to localhost
Please enter your password:
unable to connect to server at /usr/bin/sieveshell line 191,  line 1.

here's the log:
Nov 17 18:24:44 rway-imap-vm sieve[2256]: TLS is available.
Nov 17 18:24:46 rway-imap-vm saslauthd[1169]: pam_userdb(sieve:auth): user 
'proxyadmin' granted access
Nov 17 18:24:46 rway-imap-vm sieve[2256]: badlogin: localhost [127.0.0.1] PLAIN 
no mechanism available
Nov 17 18:24:46 rway-imap-vm sieve[2256]: Lost connection to client -- exiting

as you can see user proxyadmin authenticated successfully, but then something 
(authorization?) went wrong
and it says "PLAIN no mechanism available".
this only happens if i try to a

Re: Can't authorize as different user in cyradm and sieveshell

[Andrew Morgan via Info-cyrus]

This works for me under v2.4.18.  I'm able to run sieveshell against a 
frontend or backend authenticating as a cyrus "admins" user or a 
"proxyservers" user (on the backend).


Against a frontend:

# sieveshell -u morgan -a cyrus imap.onid.oregonstate.edu
connecting to imap.onid.oregonstate.edu
Please enter your password:

list

onid-web
real  <- active script

quit



Against a backend:

# sieveshell -u morgan -a cyr_proxy cyrus-be1.onid.oregonstate.edu
connecting to cyrus-be1.onid.oregonstate.edu
Please enter your password:

list

onid-web
real  <- active script

quit



My imapd.conf settings:

admins: cyrus
allowplaintext: 0
sasl_mech_list: PLAIN
sasl_minimum_layer: 0
sasl_pwcheck_method: saslauthd
sieve_allowreferrals: 0
sieve_allowplaintext: 1


Have you tried using the "sivtest" program?  It will show you the protocol 
handshakes, which might help.  Here is an example for me:


# sivtest -u morgan -a cyrus localhost
S: "IMPLEMENTATION" "Cyrus timsieved (Murder) v2.4.18"
S: "SASL" "PLAIN"
S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags 
notify envelope body relational regex subaddress copy"

S: "STARTTLS"
S: "UNAUTHENTICATE"
S: OK
Please enter your password:
C: AUTHENTICATE "PLAIN" {28+}

S: OK
Authenticated.
Security strength factor: 0
C: LOGOUT
OK "Logout Complete"
Connection closed.


Andy

On Sun, 20 Nov 2016, Michael Ulitskiy via Info-cyrus wrote:


Since nobody answered, I guess, nobody has any idea.
I wonder if anybody uses this feature and it works for you?
I mean I'd like to know if that's just me and something is wrong with my setup 
or may be that feature isn't functional at all?
Thanks in advance,

Michael

On Thursday, November 17, 2016 06:30:18 PM Michael Ulitskiy via Info-cyrus 
wrote:

Hello,

I'm playing with cyrus-imap 2.5.10 and cyrus-sasl 2.1.26.
i'm trying to use sieveshell to setup users sieve scripts, but since
i don't know users passwords i want to use a special user for authentication
and authorize as the target user.
Here's what I have.

imapd.conf:
admins: mailadmin
proxyservers: proxyadmin
sasl_pwcheck_method: saslauthd
#sasl_pwcheck_method: alwaystrue
sasl_mech_list: PLAIN
allowplaintext: yes

here's what i do:

root@rway-imap-vm:~# sieveshell -a proxyadmin -u t...@virtualcrap.com localhost
connecting to localhost
Please enter your password:
unable to connect to server at /usr/bin/sieveshell line 191,  line 1.

here's the log:
Nov 17 18:24:44 rway-imap-vm sieve[2256]: TLS is available.
Nov 17 18:24:46 rway-imap-vm saslauthd[1169]: pam_userdb(sieve:auth): user 
'proxyadmin' granted access
Nov 17 18:24:46 rway-imap-vm sieve[2256]: badlogin: localhost [127.0.0.1] PLAIN 
no mechanism available
Nov 17 18:24:46 rway-imap-vm sieve[2256]: Lost connection to client -- exiting

as you can see user proxyadmin authenticated successfully, but then something 
(authorization?) went wrong
and it says "PLAIN no mechanism available".
this only happens if i try to authorize as different user. if i don't 
everything works fine:

root@rway-imap-vm:~# sieveshell -a t...@virtualcrap.com -u t...@virtualcrap.com 
localhost
connecting to localhost
Please enter your password:




log:
Nov 17 18:24:11 rway-imap-vm sieve[2247]: TLS is available.
Nov 17 18:24:15 rway-imap-vm saslauthd[1167]: pam_userdb(sieve:auth): user 
't...@virtualcrap.com' granted access
Nov 17 18:24:15 rway-imap-vm sieve[2247]: login: localhost [127.0.0.1] 
t...@virtualcrap.com PLAIN User logged in

the same happends to cyradm:
root@rway-imap-vm:~# cyradm --user=proxyadmin --authz=t...@virtualcrap.com 
--auth=plain localhost
Password:
IMAP Password:

log:
Nov 17 18:26:27 rway-imap-vm saslauthd[1166]: pam_userdb(imap:auth): user 
'proxyadmin' granted access
Nov 17 18:26:27 rway-imap-vm imap[2277]: badlogin: localhost [127.0.0.1] PLAIN 
[SASL(-4): no mechanism available: Unable to find a callback: 32773]

but ok without trying to authorize as different user:
root@rway-imap-vm:~# cyradm --user=t...@virtualcrap.com --auth=plain localhost
Password:
localhost>
Nov 17 18:27:31 rway-imap-vm saslauthd[1167]: pam_userdb(imap:auth): user 
't...@virtualcrap.com' granted access
Nov 17 18:27:31 rway-imap-vm imap[2276]: login: localhost [127.0.0.1] 
t...@virtualcrap.com PLAIN User logged in 
SESSIONID=

Can somebody tell me what I am doing wrong?
Thanks a lot,

Michael


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.4.17 --> 2.5.3 Delayed expunge?

[Andrew Morgan via Info-cyrus]

On Thu, 13 Oct 2016, Sergey via Info-cyrus wrote:


On Wednesday 12 October 2016, Sergey via Info-cyrus wrote:


I'm wrong, "expunge_mode: immediate" works. I was expecting
quick delete, but it is slow: about 30 seconds or more.


and a lot time for big mailboxes: some minutes.


If I remember correctly, this "lazy" delete of message files is a 
performance optimization so that IMAP clients don't have to wait for the 
deletion to happen.  Also, expunged messages don't count against the 
mailbox quota.


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: how to deal with mail retention/archival.

[Andrew Morgan via Info-cyrus]

Could your retention needs be satisfied with Cyrus' delayed_delete and 
delayed_expunge functionality?


Thanks,
Andy

On Fri, 26 Aug 2016, Alvin Starr via Info-cyrus wrote:

Well the MTA still does not deal with archival because it will need to be 
passed through to Yet Another MDA to handle the archival and management 
process.


For the pure archival of the input/output stream including duplicate 
deliveries and all spam always_bcc into YAMDA would work.


In my thinking Cyrus is responsible for the storage and management of email 
so archival would be a part of that process.




On 08/26/2016 09:17 AM, Nic Bernstein wrote:

Alvin,
This is really more of an issue for your MTA, such as Postfix or Exim.  The 
MDA -- Cyrus in this case -- has little or nothing to do with the sort of 
archiving/retention you need for compliance. Take a look at always_bcc and 
similar directives in Postfix, or the equivalent in whatever your MTA is.

-nic

On 08/26/2016 08:09 AM, Alvin Starr via Info-cyrus wrote:

A company I am working with is facing issues of regulatorymail retention.

Some searching has yielded little useful results other than putting a 
system in front to store all incoming messages.


What are others doing for mail archival?

An ideal solution would let the users carry on using current use patterns 
and not impose extra restrictions.


--
Alvin Starr   ||   voice: (905)513-7688
Netvel Inc.   ||   Cell:  (416)806-0133
al...@netvel.net   ||



Cyrus Home Page:http://www.cyrusimap.org/
List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


--
Nic bernstein...@onlight.com
Onlight Inc.www.onlight.com
6525 W Bluemound Rd., Ste 24  v. 414.272.4477
Milwaukee, Wisconsin  53213-4073  f. 414.290.0335


--
Alvin Starr   ||   voice: (905)513-7688
Netvel Inc.   ||   Cell:  (416)806-0133
al...@netvel.net  ||




Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: prefork and IPv6

[Andrew Morgan via Info-cyrus]

On Thu, 9 Jun 2016, Wolfgang Breyha via Info-cyrus wrote:


Hi!

I recently wondered why some of my preforked processes on my murder backends
never get used. I detected them because some quite old lmtpd's were holding
locks on an already deleted deliver.db.

After some debugging I recognized that cyrus-master seems to fork the
configured amount of "prefork" daemons twice. One half listening on IPv4 and
the other half on IPv6. Since IPv6 is practically never used from our
frontends they stay forever doing nothing on the backends.

Is there some reasonable way to prevent this other than setting prefork=0?

I'm only using SERVICE entries like:
 Bimap  cmd="imapd" listen="imap" prefork=5

Only the port is used for listen= without interface/IP.


Use the proto argument:

  proto=tcp
The protocol used for this service (tcp,  tcp4,  tcp6,  udp,  udp4,  udp6). 
  This
string argument is optional.

tcp4, udp4: These arguments are used to bind the service to IPv4 only.
tcp6,  udp6:  These  arguments  are  used to bind the service to IPv6 only, 
if the
operating system supports this.
tcp, udp: These arguments are used to bind to both IPv4 and IPv6 if 
possible.


Here is my cyrus.conf entry:

  imap   cmd="/usr/local/cyrus/bin/imapd" listen="imap" proto="tcp4" prefork=10 
maxchild=4000


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Andrew Morgan via Info-cyrus]

On Thu, 9 Jun 2016, Andre Felipe Machado via Info-cyrus wrote:


Bron Gondwana via Info-cyrus  wrote ..

On Thu, Jun 9, 2016, at 03:02, Andre Felipe Machado via Info-cyrus wrote:

Hello,
At future release notes I read
"Under delete_mode: delayed, only the 20 most recently deleted mailboxes are

kept for any given name."

https://cyrusimap.org/imap/release-notes/3.0/x/3.0.0-beta2.html
Is there any configuration parameter to increase this limit?
Why this limit is needed?


denial of service / space wastage protection.  There's no config option 
available
right now.  I could be convinced to change it.

How would you suggest we protect against exploiting delayed delete to fill the
server without going over quota?  Maybe a new quota field for "total mailbox 
usage
including deleted stuff" that can be set to a high enough value that no 
reasonable
user will ever hit it?

Bron.

--
  Bron Gondwana
  br...@fastmail.fm



Hello, Bron
I understand the problem.
But at a corporate scenario, it is a rare event, because of jobs at stake, 
tracked user accounts,  antispam measures, etc.
It is more likely a "rogue" client,  bug/misconfiguration on a smartphone 
causing such problems.
We stay with official debian repositories versions as long as we could, 
receiving security patches.
So, mantaining an unofficial patch will be a big problem.
The sysadmin configurable parameters will be a more elegant solution.
Having configurations at sysadmin control will mantain cyrus flexible for use 
at different usage scenarios.
For the DoS / waste space problems, the 2 quota limits configurations are more 
suitable than counting folders quantity.
What if each folder contains 1 TB deleted messages?
Maybe a reasonable default (10 times user quota?) for those not wanting to 
configure is good idea.
Even better to have also a way to control individual accounts total quotas, for those 
corporate accounts like "sa...@foo.bar" that  receive lots of legitimate emails 
and have to
delete them after processing.
We have zabbix monitoring space at our cyrus backends, and need unlimited  or 
configurable delayed expunge limits for recovering messages and folders for 
years at corporate
scenario.
Thanks .
Andre Felipe


Remember, this is a limit on the number of deleted *mailboxes* kept, not 
messages.


Bron, this could impact Pine/Alpine users that frequently postpone 
messages.  Pine creates a folder named "postponed-msgs" to store drafts. 
The folder is created when a draft is saved and deleted when all drafts 
have been deleted/sent.


Here is my personal deleted folders list, right now:

DELETED.user.morgan.postponed-msgs.5755CF0C 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F446 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F486 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F4D1 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F4E4 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F50E 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F65F 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F844 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5756ECFC 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5756F602 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.575706F8 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.57585C5D 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.57587FE1 0 p2 morgan lrswipkxtecda

We are removing deleted mailboxes after 7 days:

delprune  cmd="/usr/local/cyrus/bin/cyr_expire -E 1 -X 7 -D 7" at=0100


I don't know if other IMAP clients have similar quirky behavior, but I 
could see myself running into this limit.  However, I certainly don't care 
about recovering my old postponed-msgs mailboxes.


Hmmm, is this a limit per-mailbox (user.morgan.postponed-msgs) or per-user 
(all mailboxes under user.morgan)?


Thanks,
Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Problems with murder upgrade from 2.2.13 to 2.5.8

[Andrew Morgan via Info-cyrus]

I've found that backends should be upgraded before frontends...

You'll run into frontends trying to use features that don't exist on the 
backends.  Usually, you can work around that with the 
suppress_capabilities setting in imapd.conf, but it may require less 
testing to upgrade the frontends last.


Regarding you specific permissions problem, I think Mathieu has already 
posted the answer.  Although, I wonder if the frontend is enforcing 
permissions that can't exist on the backend yet...


For reference, these are the permissions on my v2.4.18 mailbox:

localhost> lam user.morgan
morgan lrswipkxtecda


Andy

On Mon, 6 Jun 2016, Jean Charles Delépine via Info-cyrus wrote:


Hello,

I'm on the way to make a big (late) upgrade. 

My murder config is composed of 16 1To backends. I can't upgrade 
all of them simultaneously. So I planed to :


 - upgrade mupdate server (make a new one, and update frontend's and
   backend's conf)
 - replace frontends with upgraded one's
 - upgrade backends one after the other, nightly, on serveral night

mupdate server upgrade is ok. But I have problems with 2.5 frontends and 2.2
backends interaction. All seems fine (no error), but users can't create new sub 
mailboxes (admin can create mailboxes and sub mailboxes) :


loggued as mailbox owner :
imap-01> lam INBOX
delepine lrswipcda
anyone p
imap-01> cm INBOX.hop
createmailbox: Permission denied

My tests say that, whichever mupdate server version :
 Frontend 2.2 can create 2.2 mailboxes and 2.5 mailboxes
 Frontend 2.5 can't create 2.2 mailboxes but can create 2.5 mailboxes

All others tested features work.

The 2.2 is using saslauthd + pam_ldap for authentification. The 2.5 is using 
either
ldapdb or saslauthd + ptoader and ldap.

With or without
 suppress_capabilities: ESEARCH QRESYNC XLIST LIST-EXTENDED WITHIN
on 2.5 frontends.

2 questions :
 - do you have an idea why users can't create submailboxes on 2.2
   backends with 2.5 frontends ? Is there any acl new option I
   miss ? ...
 - what are the risks if I wait for all backends to migrate before
   using 2.5 frontends ? My option with this problem. I didn't find
   any problem... but surely, if there's one, my users will find it.

Options that might be relevant :
On backends :
 proxyservers: proxy
 proxy_authname: proxy

On frontends:
 proxy_authname: proxy
 proxy_password: <>
 proxyd_allow_status_referral: 0
 proxyd_disable_mailbox_referrals: 1

backends are in an internal non routable network.

Sincerly,
 Jean Charles Delépine

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Request: Please sign this list's messages via DKIM or SPF

[Andrew Morgan via Info-cyrus]

On Tue, 5 Apr 2016, lst_hoe02--- via Info-cyrus wrote:



Zitat von Binarus via Info-cyrus :



Combine SPF / DKIM with domain blacklisting, and then you *have* an 
efficient spam fighting tool.




As stated the spam actually reaching our inboxes after around 90% cutoff is 
valid DKIM/SPF signed as it is mostly from the big free providers like 
Outlook.com, Google and Yahoo. Some other big share is from professional spam 
farms with always alternating IP and Domains ranges from all over the world 
with also valid DKIM/SPF. Next big share is from educational servers also 
mostly valid DKIM/SPF. The tiny rest with around 10% is in fact not DKIM/SPF 
signed.
From the valid e-mail around 20% looks like having a valid SPF/DKIM, mostly 
professional newsletters not personal mail from customers.


So No, SPF/DKIM is no useful spam fighting tool at least not in our corner of 
the world.


Another recent standard, DMARC (https://dmarc.org/) allows the domain 
owner to specify what the recipient should do with messages that fail DKIM 
or SPF checks.


We ran into this recently and discovered that Yahoo's DMARC records tell 
the recipient to REJECT messages that fail DKIM or SPF.  Google is 
honoring that DMARC record by putting the message into the Spam folder.


This seems like a pretty effective method to prevent someone from spoofing 
email from your domain.  Of course, it does not prevent an actual Yahoo 
account from sending spam, so you still need traditional spam detection 
tools as well.  However, it is nice that a third-party sender cannot harm 
your domain's reputation through spoofing.


Note: I don't care whether this email list uses SPF or DKIM.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Is there a way to send custom warning to all IMAP users?

[Andrew Morgan via Info-cyrus]

On Mon, 28 Mar 2016, francis picabia via Info-cyrus wrote:


We have migrated all email on a server to a cloud email platform.
The users were notified by email beforehand, but hundreds are still
connecting to the standard IMAP service.  They may not
even remember they have set up devices to connect here.
Is there a way to send a custom warning through some setting,
similar to how quota warnings are generated.  Really if there is
any error I can fake, and customize the message, it would work.
We are using Linux, pam authentication, Cyrus with saslauthd.

Just shutting down the service is also a solution, but given over 600
unique users have logged in today, I'd rather not dump that load on
the service desk.


When we migrated some of our users to Google Mail, we placed a final 
message in their Cyrus mailbox.  When they login, they can see "You've 
been migrated to Google!", and the message tells them how to find their 
email on Google.


To bypass email routing, you can use the "deliver" program on the Cyrus 
server to drop the message in the Cyrus mailbox.


Let me know if you need more information.

Thanks,
Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: drown/SSL issue

[Andrew Morgan via Info-cyrus]

On Thu, 3 Mar 2016, Tony Galecki via Info-cyrus wrote:

Lots of fiddling arround, tls_versions: ssl3 tls1_2 in the imapd.conf 
file also fixed the issue. However, some clients (notably older Mac Mail 
clients) were not able to connect.


Don't you want to include tls1_0 and tls1_1 in the list?  Here at OSU, we 
use the defaults, "tls_versions: tls1_0 tls1_1 tls1_2".


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus Murder with different Cyrus IMAP Server versions

[Andrew Morgan via Info-cyrus]

On Wed, 2 Mar 2016, Jack Snodgrass via Info-cyrus wrote:


I have a older Cyrus 2.2 version setup and running in production.

I want to move to a newer Cyrus 2.4 system with minimal downtime.

The goal is 1) limit down time and 2) keep the SAME ip address for the users 
imap configs.


I can convert my existing Cyrus 2.2 ( Debian v6 ) to Cyrus 2.4 ( Debian v8 ) 
but will be down around 8 ( at least ) for the two debian upgrades and 
converting 200gig of Cyrus 2.2 mail to Cyrus 2.4 - indexes and what not.


I was thinking.. maybe another approach would be to setup Cyrus Murder ( 2.2 
) on my existing Cyrus 2.2 box and connect it up with a new Cyrus 2.4 server 
( on a new Debian v8 box ) and just move mail accounts over one at a time 
until all of the mail was off of the old box.  Once all of the mail was off 
of the old Cyrus 2.2 box, I could then upgrade that to debian v8 and Cyrus 
2.4 and then have 2 systems that the mail could be split between.


Can I run a Murder 2.2 server and have it talk with a Cyrus 2.4 IMAP box or 
do the versions have to be the same?


In a Cyrus Murder, you want the frontend server to be upgraded last.  If a 
newer frontend is used, it will issue newer IMAP commands that the older 
backend doesn't support.  When you are upgrading an existing Murder 
cluster, you upgrade in this order: mupdate master, backends, then 
frontends.


Murder does allow you to (mostly) transparently move mailboxes between 
backends.  I have upgraded many times by simply moving the mailboxes to a 
new backend server with newer versions of the OS and Cyrus.  However, 
you'll need to create 2 new hosts - a frontend and mupdate master.  Then 
you'll need to move the DNS CNAME from the existing 2.2 server to the 
frontend.


A Murder is a bit complicated (don't forget about mail delivery too!), so 
let me suggest an alternative that keeps the downtime short.


Build a new server with Debian 8.  I'd probably install Cyrus v2.5.latest 
by hand.  Compiling Cyrus is very easy on Debian.  Cyrus v2.5 has a major 
advantage over v2.4 - you can run a script to upgrade the mailbox format 
instead of waiting for the user to open the mailbox.  See the release 
notes for upgrade instructions:


  http://cyrusimap.org/imap/release-notes/2.5/x/2.5.0.html

Anyways, build the new server with Debian and whatever version of Cyrus 
makes you comfortable.  Then, weeks before you plan to make the cutover, 
use rsync to copy to the mail from the old server to the new server.  Of 
course, the first run will take a long time to copy 200GB.  Successive 
rsyncs will take less time as the deltas are smaller.  In the week before 
the scheduled outage, run rsync every night.


During your outage window, stop Cyrus on the old server, run a final 
rsync, then swap IP addresses and/or DNS names, and start Cyrus on the new 
server.


There are a couple advantages to this approach.  You'll be able to test 
how the new server works with your actual mail.  You can make 
configuration changes if needed.  You can also time how long the rsync 
will take, so you know how much time to schedule for the outage.  Even if 
there isn't much data to rsync on the final pass, it can still take a long 
time to calculate the differences between the 2 filesystems.


Before I ran Cyrus Murder, this is how I upgraded our Cyrus server to new 
hardware.


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 2.4.18, problem with reconstruct

[Andrew Morgan via Info-cyrus]

On Fri, 5 Feb 2016, Sergey via Info-cyrus wrote:


Hello.

I attempted to reconstruct some damaged mailboxes with empty
folders, but it does not work. I use this command:

su -l cyrus -s /bin/bash -c "/usr/lib/cyrus/reconstruct -f -r user/user@domain"

Mail directory contains "Trash" subdirectory without any files (manualy
created from backup). Reconstruct works if I put any of files cyrus.* to
this subdirectory. At the same time there was the opposite problem:
I can not delete existing directory, reconstruct restores it.

Is this is a bug or require any other settings to run reconstruct ?


I usually use these steps to add a new folder using reconstruct:

  touch cyrus.header
  chown cyrus:mail cyrus.header
  reconstruct -f -r user.

So, I think the behavior you are seeing is expected.  Create an empty 
cyrus.header file, with the correct ownership, before running reconstruct.


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: unable to delete corrupted mail box on cyrus v2.3.16

[Andrew Morgan via Info-cyrus]

On Mon, 11 Jan 2016, Sophie Loewenthal via Info-cyrus wrote:


Hi!

I have a broken mailbox that I would like to delete.

This is Cyrus v2.3.16 on CentOS 6.

I tried reconstructing the mailbox from scratch ( Because I suspect this 
was manually deleted from disc ).



mkdir imap-store/spool/imap/domain/example.com/user/kat^long
cd imap-store/spool/imap/domain/example.com/user/kat^long
chmod 755 .
chown cyrus:mail .
touch cyrus.header
chown cyrus:mail cyrus.header

log into cyradm:
localhost> lam user/kat.long
kat.l...@example.com lrswipkxtecda
localhost> reconstruct -r user/kae.long
reconstruct: Mailbox has an invalid format
localhost> dm user/kat.long
deletemailbox: Permission denied

Names and domain names replaced with false entries.

How could I remove this?


Here are my steps for recreating a mailbox (normally when I'm restoring a 
mailbox from backups):


1. Locate user's mail directory (/var/spool/cyrus/mail/prefix/user/username).
2. Change to that directory.
3. Make a RESTORE directory (mkdir RESTORE).
4. Fix ownership/perms (chown cyrus:mail RESTORE; chmod 700 RESTORE).
5. Change to the directory containing the mail folder the user wants restored.
6. Run 'recover', the Legato backup client.
7. 'changetime' to change the time to recover data from.
8. 'add filename' to add the files to restore.  To restore all the messages in 
the folder, use 'add *.'.
9. 'relocate RESTORE' to recover files into the RESTORE directory instead of 
the current directory.
10. 'recover' to recover the files.
11. 'quit' to quit out of the recover program.
12. Create a dummy cyrus.header file "(touch RESTORE/cyrus.header; chown 
cyrus:mail RESTORE/cyrus.header; chmod 600 RESTORE/cyrus.header).
13. Run "su cyrus -c '/usr/local/cyrus/bin/reconstruct -x -f user.username'".
14. Run "su cyrus -c '/usr/local/cyrus/bin/quota -f user.username'".

I think you're following the same basic steps, but I would try running 
reconstruct externally, not from cyradm.  Don't forget the quota command 
either.


When you run reconstruct, check syslog for errors too.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: delprune on a single mailbox

[Andrew Morgan via Info-cyrus]

On Fri, 6 Nov 2015, Marcus Schopen via Info-cyrus wrote:


Am Mittwoch, den 04.11.2015, 06:36 -0500 schrieb Adam Tauno Williams via
Info-cyrus:

globally in cyrus.conf delprune is set to
> > > > delprunecmd="/usr/sbin/cyrus expire -E 1 -X 7 -D 7"
> > > > at=0501
> > > > For a single mailbox I don't want to keep deleted mails for 7
> > > > days,
> > > > but
> > > > expire them immediately or once a day per cron. How to do that?
> > > Forogt to say that delete_mode and expunge_mode is set to
> > > delayed.
> > > Via cron this should work for an immediate cleanup/expire:
> > You can set an expire annotation per mailbox. 
> How do I do that? From cyr_expire manpage:

> "The value of the /vendor/cmu/cyrus-imapd/expire annotation is
> inherited by all children of the given mailbox, so an entire mailbox
> tree can be expired by seting a single annotation on the root of that
> tree. If a mailbox does not have a /vendor/cmu/cyrus-imapd/expire
> annotation set on it (or does not inherit one), then no messages are
> expired from the mailbox."

Via cyradm -

cyrus.example.com> mboxcfg user.adam expire 365 
cyrus.example.com> info user.adam 
{user.adam}:

  condstore: false
  duplicatedeliver: false
  expire: 365
  lastpop:
  lastupdate: 13-Aug-2008 19:37:31 -0400
  partition: default
  sharedseen: false
  size: 12325671

AFAIK the annotations supported by cyradm/mboxcfg are:

* comment – A free-form text comment or description to be attached to
the mailbox.
* condstore – This annotation is only supported in the 2.3.x release
series starting with 2.3.3 although its use is not recommended until
2.3.8. As of the 2.4.x release series CONDSTORE functionality is
enabled on all mailboxes regardless of annotation and attempting to set
this annotation will result in a permission denied message. On releases
where this annotation is supported setting a value of “true” will
enable CONDSTORE functionality1.
* expire – If an expire value is provided messages will be
automatically deleted from the mailbox once the specified number of
days has elapsed.
* news2mail - 
* sharedseen - Enables the use of a shared \Seen flag on messages

rather than a per-user \Seen flag. The 's' right in the mailbox ACL
still controls whether a user can set the shared \Seen flag.
* sieve – In the case of a shared folder the “sieve” parameter
specifies the name of a global SIEVE script that will be used for every
message delivered to the folder.  This value is ignored for personal
mailboxes (mailboxes including and subordinate to a user's INBOX).
* squat – Flags the mailbox to be included for indexing when the SQUAT
process performs index generation.


> But is it possible to expunge a message immediately when it's deleted
> by client and not with the next expire run?

Not if delayed expunge is enabled AFAIK; that would defeat the purpose.


I set "mboxcfg user.test expire 1" on a test mailbox, but it has no
effect on nightly delprune set in cyrus.conf EVENT: 


 delprune cmd="/usr/sbin/cyrus expire -E 1 -X 7 -D 7" at=0501"

Messages deleted two days ago are still in the file system.

localhost> info user.test
{user.test}:
 duplicatedeliver: false
 expire: 1
 lastpop:
 lastupdate:  4-Nov-2015 17:14:20 +0100
 partition: default
 pop3newuidl: true
 sharedseen: false
 size: 0


The expire annotation causes Cyrus to delete messages older than  
days.  If you have delayed_expunge enabled, the messages still remain on 
the filesystem until you purge them using cyr_expire.


Andy
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: IMAP processes out of control

[Andrew Morgan]

You should be able to have a LOT of imapd processes with that much RAM. 
On a server with 8GB of RAM, I have maxchild=4000 for imap and 
maxchild=1000 for imaps.


However, it is good to leave lots of RAM for caching, so those limits are 
mainly in place to prevent a bad client from causing a low-memory 
condition on the server.


When you see the process count increasing, you need to identify what the 
"extra" processes are doing.  You will probably be able to identify a 
pattern by looking at the cyrus proc files.  Try this:


  cat ${configdir}/proc/* | sort

The format of the proc file is:

  hostname [IP-address] authenticated-username SELECTed-mailbox

I bet you'll see a lot of connections from one host or user.

You can also use lsof and netstat if things are hanging before the proc 
file is created.


Andy

On Wed, 23 Sep 2015, Shaheen Bakhtiar wrote:

2 x AMD quad Core 64bit 
4G RAM


This morning I woke up to a plethora of complaints that people were not able to 
access their emails. I remove the aforementioned maxchild from the 
configurations and restart to server. Once I did that people were able to 
re-connect with no problems.

I did not have these types of problems with the older version (I believe was 2.3.19). Just since I upgraded to the latest version of Cyrus. 


Current version is:
[root@postoffice ~]# dnf info cyrus-imapd
Last metadata expiration check performed 1:06:02 ago on Wed Sep 23 07:12:41 
2015.
Installed Packages
Name: cyrus-imapd
Arch: x86_64
Epoch   : 0
Version : 2.4.17
Release : 9.fc22

Running on Fedora Core 22 64bit


On Sep 23, 2015, at 7:44 AM, signaldevelo...@gmail.com wrote:

Again this is active sync devices that are connecting with a ton of pushed folders. The more you tell it to sync (folders) the more processes it's going to fork for each user folder. Is this affecting performance that bad? What's your hardware? 


- Paul


On Sep 22, 2015, at 7:43 PM, Moby <m...@mobsternet.com> wrote:



On 9/22/2015 18:12, Shaheen Bakhtiar wrote:

On Sep 22, 2015, at 2:17 PM, Andrew Morgan <mor...@orst.edu> wrote:


On Tue, 22 Sep 2015, Shaheen Bakhtiar wrote:

It happened again….. although it took longer for it to happen, this has been 
happening only since the upgrade in Jun.

The number of imap processes continues to increase until the server is 
completely OOM. the increase is drastic and all of a sudden.

You should probably set maxchild to a value that won't run your server out of 
memory.  :)

Have you looked at the processes to see what they have in common?  For example, 
sometimes an IMAP client will run amok and make hundreds or thousands of 
connections.  Or perhaps the processes are all stuck waiting on a lock, etc.

lsof, strace, netstat, and your Cyrus logs can help a lot.

  Andy



[shawn@postoffice ~]$ ps aux | grep imapd | wc -l
255
[shawn@postoffice ~]# ps aux | grep imapds | wc -l
1
[shawn@postoffice ~]# ps aux | grep pop3d | wc -l
9
[shawn@postoffice ~]# ps aux | grep timseived | wc -l
1
[shawn@postoffice ~]# ps aux | grep lmtpunix | wc -l
1

Based on that output I changed the configuration file (below) adding maxchild. 
Most likely all my users have their clients open, and from previous monitoring 
I average about 200 instances of imapd:

# standard standalone server implementation

START {
 # do not delete this entry!
 recover   cmd="ctl_cyrusdb -r"

 # this is only necessary if using idled for IMAP IDLE
 idled cmd="idled"
}

# UNIX sockets start with a slash and are put into /var/lib/imap/sockets
SERVICES {
 # add or remove based on preferences
 imap  cmd="imapd" listen="imap" prefork=5 maxchild=300
 imaps cmd="imapd -s" listen="imaps" prefork=1 maxchild=100
 pop3  cmd="pop3d" listen="pop3" prefork=3 maxchild=5
 pop3s cmd="pop3d -s" listen="pop3s" prefork=1 maxchild=5
 sieve cmd="timsieved" listen="sieve" prefork=0

 # these are only necessary if receiving/exporting usenet via NNTP
#  nntp cmd="nntpd" listen="nntp" prefork=3
#  nntpscmd="nntpd -s" listen="nntps" prefork=1

 # at least one LMTP is required for delivery
#  lmtp cmd="lmtpd" listen="lmtp" prefork=0
 lmtpunix  cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1

 # this is only necessary if using notifications
#  notify   cmd="notifyd" listen="/var/lib/imap/socket/notify" proto="udp" 
prefork=1
}

EVENTS {
 # this is required
 checkpointcmd="ctl_cyrusdb -c" period=30

 # this is only necessary if using duplicate delivery suppression,
 # Sieve or NNTP
 delprune  cmd="cyr_expire -E 3" at=0400

 # this is only necessary if caching TLS sessions
 tlsprune  cmd="tls_prune" at

Re: IMAP processes out of control

[Andrew Morgan]

On Tue, 22 Sep 2015, Shaheen Bakhtiar wrote:



It happened again….. although it took longer for it to happen, this has 
been happening only since the upgrade in Jun.


The number of imap processes continues to increase until the server is 
completely OOM. the increase is drastic and all of a sudden.


You should probably set maxchild to a value that won't run your server out 
of memory.  :)


Have you looked at the processes to see what they have in common?  For 
example, sometimes an IMAP client will run amok and make hundreds or 
thousands of connections.  Or perhaps the processes are all stuck waiting 
on a lock, etc.


lsof, strace, netstat, and your Cyrus logs can help a lot.

Andy
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: If you want a faster Kolab, read this.

[Andrew Morgan]

On Sat, 12 Sep 2015, Paul Bronson wrote:

> Cyrus gurus - can you help us diagnose the IMAP debug below that I gave and
> help us understand the slow down. I am no imap pro, but the imap_debug
> below seems to show a lot of in-and-out's for a single message click. Again
> this is roundcube, centos 6, kolab 3.4, cyrus version:

The debug log shows that the connection and execution of all those 
commands took place in about 1-2 seconds (19:53:31 - 19:53:32).

Here is a summary of the commands the client issued:

A0001 STARTTLS
A0002 CAPABILITY
A0003 ID ("name" "Roundcube" "version" "1.1.2" "php" "5.3.3" "os" "Linux" 
"command" 
"/webmail/8d61c34e132a834f/?_task=mail&_action=preview&_uid=11&_mbox=INBOX&_framed=1&_caps=pdf%3D1%2Cflash%3D1%2Ctif%3D0")
A0004 AUTHENTICATE PLAIN ** [57]
A0005 GETMETADATA
A0006 LIST ""
A0007 MYRIGHTS
A0008 SELECT Configuration
A0009 LSUB "" "*"
A0010 LIST "" "*"
A0011 GETMETADATA Archive 
(/private/vendor/kolab/folder-type/shared/vendor/kolab/folder-type)
A0012 MYRIGHTS Tasks
A0013 GETMETADATA Tasks 
(/private/vendor/kolab/displayname/shared/vendor/kolab/displayname)
A0014 GETMETADATA Tasks (/private/vendor/kolab/color/shared/vendor/kolab/color)
A0015 MYRIGHTS Contacts
A0016 MYRIGHTS "Contacts/Personal Contacts"
A0017 GETMETADATA Contacts 
(/private/vendor/kolab/displayname/shared/vendor/kolab/displayname)
A0018 GETMETADATA "Contacts/Personal Contacts" 
(/private/vendor/kolab/displayname/shared/vendor/kolab/displayname)
A0019 GETMETADATA Contacts 
(/private/vendor/kolab/uniqueid/shared/vendor/cmu/cyrus-imapd/uniqueid/shared/vendor/kolab/uniqueid)
A0020 GETMETADATA "Contacts/Personal Contacts" 
(/private/vendor/kolab/uniqueid/shared/vendor/cmu/cyrus-imapd/uniqueid/shared/vendor/kolab/uniqueid)
A0021 SELECT Contacts
A0022 LOGOUT

1-2 seconds for all those commands is pretty damn fast.  I don't know if 
that generates much disk I/O, but there must be at least a few I/Os 
required to do that.

I'm with you - the place to optimize this is in Kolab, possibly with in 
conjuction with an imapproxy.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus tweaks (slow on roundcube)

[Andrew Morgan]

On Thu, 10 Sep 2015, signaldevelo...@gmail.com wrote:

> Is there some type of log I can provide from Cyrus / sasl to help 
> diagnose this better to the kolab guys? Other kolab guys I know say 
> their entropy is right where I'm at and they aren't experiencing these 
> slowness issues.
>
> Are their sasl or Cyrus logs I can provide?

Maybe I missed this detail earlier in the thread, but why not run an IMAP 
proxy to reduce the rate of new connections to Cyrus?  Making a new IMAP 
connection with every click seems abusive! :)

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus tweaks (slow on roundcube)

[Andrew Morgan]

I use imapproxy with Horde Webmail here.  Assuming the proxy is using 
cached connections instead of making a new connection each click, then I 
would look into performance problems within Cyrus itself.  It would be 
interesting to see what IMAP commands Roundcube is issuing to Cyrus. 
Perhaps it is doing something "stupid" like retrieving all the message 
bodies on each click?

If you haven't already, try enabling telemetry logging for a single user 
and check the telemetry log files.  If you can post some of those logs 
here, we may be able to identify the problem.

Thanks,
Andy

On Fri, 11 Sep 2015, signaldevelo...@gmail.com wrote:

> I tried imapproxy. It is the same speed. And again, definitely not hardware 
> related.
>
> I see in the logs in queries the proxy and that works fine but not sure why 
> it's still the same speed.
>
>
> - Paul
>
>> On Sep 11, 2015, at 2:47 PM, Andrew Morgan <mor...@orst.edu> wrote:
>>
>>> On Thu, 10 Sep 2015, signaldevelo...@gmail.com wrote:
>>>
>>> Is there some type of log I can provide from Cyrus / sasl to help diagnose 
>>> this better to the kolab guys? Other kolab guys I know say their entropy is 
>>> right where I'm at and they aren't experiencing these slowness issues.
>>>
>>> Are their sasl or Cyrus logs I can provide?
>>
>> Maybe I missed this detail earlier in the thread, but why not run an IMAP 
>> proxy to reduce the rate of new connections to Cyrus?  Making a new IMAP 
>> connection with every click seems abusive! :)
>>
>>Andy
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 2.4.18 released

[Andrew Morgan]

On Tue, 7 Jul 2015, Sebastian Hagedorn wrote:

 --On 6. Juli 2015 13:38:16 -0700 Andrew Morgan mor...@orst.edu wrote:

 On Mon, 6 Jul 2015, Sebastian Hagedorn wrote:
 
 --On 6. Juli 2015 14:23:11 +1000 ellie timoney el...@fastmail.com
 wrote:
 
 Please consult the release notes before upgrading to 2.4.18:

   https://docs.cyrus.foundation/imap/release-notes/2.4-current.html
 
 The big one is this: Disable use of SSLv2/SSLv3
 
 When I look at our log files, I see that there are still several hundred
 SSLv3 connections per day. I'm worried that not all clients used by our
 users  support TLSv1. One such client appears to be Outlook 2003. Has
 anybody else  (especially in education) already turned off SSLv3? What
 were your  experiences?
 
 I had similar concerns when I was making SSLv3 and cipher changes to my
 LDAP service.  I wanted to proactively identify any clients that would be
 affected so we could fix them in advance.
 
 I used tshark to sniff the ciphers for all my incoming connections, but
 you can also get the TLS version used from the output.
 
 I wrote it up in a blog post here:
 
 
 http://blogs.oregonstate.edu/sysadmin/2015/07/01/tracking-ssltls-cipher-u
 sage/

 Thanks for your reply! Our Cyrus server is still running RHEL 5, and its 
 tshark binary doesn't yet support the -2 flag. I see that it's supposed to 
 Perform a two-pass analysis, but I'm unclear on why that is useful or even 
 necessary? I removed the flag for my tests, and at first glance it still 
 seems to work. FWIW, I had to modify the pattern matching in the Perl script, 
 because in our instance there are two tabs before the first IP address.

I copied the basic tshark parameters from someone else.  When I run the 
capture without -2, the output is slightly different, although it seems to 
capture the same basic information.

It appears the parameters -R, -2, and -Y have been changing between 
versions.  Current versions of tshark have -Y, which applies a display 
filter.  My version (v1.8.10 on Oracle Linux 6) doesn't have -Y though.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 2.4.18 released

[Andrew Morgan]

On Mon, 6 Jul 2015, Sebastian Hagedorn wrote:

 --On 6. Juli 2015 14:23:11 +1000 ellie timoney el...@fastmail.com wrote:

 Please consult the release notes before upgrading to 2.4.18:

   https://docs.cyrus.foundation/imap/release-notes/2.4-current.html

 The big one is this: Disable use of SSLv2/SSLv3

 When I look at our log files, I see that there are still several hundred 
 SSLv3 connections per day. I'm worried that not all clients used by our users 
 support TLSv1. One such client appears to be Outlook 2003. Has anybody else 
 (especially in education) already turned off SSLv3? What were your 
 experiences?

I had similar concerns when I was making SSLv3 and cipher changes to my 
LDAP service.  I wanted to proactively identify any clients that would be
affected so we could fix them in advance.

I used tshark to sniff the ciphers for all my incoming connections, but 
you can also get the TLS version used from the output.

I wrote it up in a blog post here:

   
http://blogs.oregonstate.edu/sysadmin/2015/07/01/tracking-ssltls-cipher-usage/

NOTE: This does not require access to your private key because there is no 
decryption of data.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: change to UNIX hierarchy

[Andrew Morgan]

On Tue, 30 Jun 2015, Stephen Ingram wrote:

 Since we support Kerberos, we use standard usernames on our system without
 any domain endings and we also use the Alternate namespace. This being the
 case, can we turn on UNIX hierarchy without any changes in the user's mail
 client or the filesystem itself? From the documentation, it looks like the
 only change would be in the management of the mailboxes (cyradm) where we
 would now use a / instead of a .. For instance, the cyradm command: cm
 user/john/Sent instead of cm user.john.Sent. Am I correct or off base here?

 Steve

The mailbox separator may need to be updated in your IMAP clients too. 
Some clients will detect it automatically (at least when setting up the 
IMAP profile), but you may run into clients that need a manual config 
change.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Murder frontend problem

[Andrew Morgan]

On Fri, 5 Jun 2015, Major Csaba wrote:

 There is one more small question: why the proxied LMTP needs to have admins 
 permission on the backend? I thought the proxyservers setting is for this, 
 but LMTP doesn't work without adding my proxy user in the admins...

Play around with lmtp_admins in imapd.conf.  Our mail relays connect to 
our frontends over lmtp and auth as cyr_lmtp.  That authentication is 
proxied to the backends for delivery.

Here is our admin-related config on the backends:

admins: cyrus cyr_proxy
lmtp_admins: cyr_lmtp cyr_proxy
# Only set proxyservers on Standard Murder BACKENDS
proxyservers: cyr_proxy

and on our frontends:

admins: cyrus
lmtp_admins: cyr_lmtp
proxy_authname: cyr_proxy
proxy_password: redacted


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus aggregate compatibility.

[Andrew Morgan]

Mike, this means that the I/O hit from upgrading will happen at the time 
you XFER the mailbox.  That's good because you can control the I/O by 
spreading out your XFERs, if it's even a problem.  I moved a lot of 
mailboxes (30,000+) without really noticing a problem.  I did try to 
perform the moves during less busy times of the day though.

Andy

On Tue, 21 Apr 2015, Bron Gondwana wrote:

 From 2.3 to 2.4 upgraded automatically.

 From x to 2.5 doesn't upgrade automatically at the moment.  You have to run
 reconstruct -V max on the folder afterwards.

 Maybe for the XFER case we should upgrade automatically... I'll talk to Ellie
 about that when she gets in today.  She's the 2.5 maintainer now.

 Bron .

 On Tue, Apr 21, 2015, at 08:51 AM, Andrew Morgan wrote:
 Does an XFER automatically upgrade the mailbox to the new format?  I don't
 remember having performance problems when I moved users from a v2.3
 backend to a new v2.4 backend (a long time ago).

  Andy

 On Tue, 21 Apr 2015, Bron Gondwana wrote:

 I would wait for 2.5.1, which should be out in a day or so.  There were
 some XFER bugs in 2.5.0.

 The IO hit will have to be taken regardless, it's just deferred
 slightly.  The 2.5 backend will work with 2.2 proxies just fine, though
 of course most of the new features won't be visible to your clients,
 because 2.2 gives a much reduced capability string.

 Longer term, we're looking at a full unified clustering system which might
 still include murder or might be totally separate.  It's going to be very 
 nice,
 but it will only work for 3.0+ servers.

 Bron.

 On Tue, Apr 21, 2015, at 08:07 AM, Michael Sofka wrote:
 On 2015-04-20 17:16, k...@rice.edu wrote:
 On Mon, Apr 20, 2015 at 05:11:00PM -0400, Michael D. Sofka wrote:
 Under the scenario, would 2.5 work better?

 Mike

 Hi Mike,

 In our case, the unconstrained I/O caused by the mandatory mailbox
 format conversion on first use would have necessitated a prolonged
 service outage to prevent overloading the system. 2.5 will allow you
 to schedule your conversions while the system is functional. This
 may not be a concern for you.


 Hum, it might  This would drive up the load on the 2.4 system as
 I'm moving mailboxes?

 This project is driven entirely by the state of the SAN disks.  They
 are either old with controller errors, or expensive to keep on
 service, or needed elsewhere in a chain of updates.  Plan B is to
 clone the existing
 2.3 server, but if I can get a new OS and application image in the
   process, I will be a happy camper.  But even doing that is exceeding
   my mandate.

 But if a 2.5 image will work with 2.2 front-end proxies, the deferred
 conversion is worth considering.  I do anticipate the moves being off-
 hours, but even off-hours is busy.

 Mike


 --
 Michael D. Sofka   sof...@rpi.edu CMT Sr. Systems
 Programmer,   Email, TeX, Epistemology Rensselaer Polytechnic
 Institute, Troy, NY.  http://www.rpi.edu/~sofkam/

 
 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info:
 http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


 --
  Bron Gondwana
  br...@fastmail.fm
 
 Cyrus Home Page: http://www.cyrusimap.org/
 List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
 To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus



 --
  Bron Gondwana
  br...@fastmail.fm


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus aggregate compatibility.

[Andrew Morgan]

Does an XFER automatically upgrade the mailbox to the new format?  I don't 
remember having performance problems when I moved users from a v2.3 
backend to a new v2.4 backend (a long time ago).

Andy

On Tue, 21 Apr 2015, Bron Gondwana wrote:

 I would wait for 2.5.1, which should be out in a day or so.  There were
 some XFER bugs in 2.5.0.

 The IO hit will have to be taken regardless, it's just deferred
 slightly.  The 2.5 backend will work with 2.2 proxies just fine, though
 of course most of the new features won't be visible to your clients,
 because 2.2 gives a much reduced capability string.

 Longer term, we're looking at a full unified clustering system which might
 still include murder or might be totally separate.  It's going to be very 
 nice,
 but it will only work for 3.0+ servers.

 Bron.

 On Tue, Apr 21, 2015, at 08:07 AM, Michael Sofka wrote:
 On 2015-04-20 17:16, k...@rice.edu wrote:
 On Mon, Apr 20, 2015 at 05:11:00PM -0400, Michael D. Sofka wrote:
 Under the scenario, would 2.5 work better?

 Mike

 Hi Mike,

 In our case, the unconstrained I/O caused by the mandatory mailbox
 format conversion on first use would have necessitated a prolonged
 service outage to prevent overloading the system. 2.5 will allow you
 to schedule your conversions while the system is functional. This
 may not be a concern for you.


 Hum, it might  This would drive up the load on the 2.4 system as
 I'm moving mailboxes?

 This project is driven entirely by the state of the SAN disks.  They
 are either old with controller errors, or expensive to keep on
 service, or needed elsewhere in a chain of updates.  Plan B is to
 clone the existing
 2.3 server, but if I can get a new OS and application image in the
   process, I will be a happy camper.  But even doing that is exceeding
   my mandate.

 But if a 2.5 image will work with 2.2 front-end proxies, the deferred
 conversion is worth considering.  I do anticipate the moves being off-
 hours, but even off-hours is busy.

 Mike


 --
 Michael D. Sofka   sof...@rpi.edu CMT Sr. Systems
 Programmer,   Email, TeX, Epistemology Rensselaer Polytechnic
 Institute, Troy, NY.  http://www.rpi.edu/~sofkam/

 
 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info:
 http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


 --
  Bron Gondwana
  br...@fastmail.fm
 
 Cyrus Home Page: http://www.cyrusimap.org/
 List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
 To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Lock Folder and cyr_expire

[Andrew Morgan]

On Wed, 4 Mar 2015, Sebastian Hagedorn wrote:


Hi,

--On 4. März 2015 11:48:19 +0100 Giuseppe Ravasio (LU) 
giuseppe_rava...@modiano.com wrote:



We have about 500k growing (that aren't opened daily!) imap folders and
the 0k lock files are filling the inode table of the partition
containing the mboxname_lockpath


the best solution (IMO) is to use shared memory:

mboxname_lockpath: /dev/shm/cyrus_lock


Interesting!  I haven't looked at the lock directory until just now.  It 
uses a lot of inodes on my system too:


/var/spool/cyrus/config/lock# find . | wc -l
09

It happens to reside on my root partition, and it is using a good chunk of 
the available inodes:


FilesystemInodes   IUsed   IFree IUse% Mounted on
/dev/sdi21189024  628547  560477   53% /


I'm using a tmpfs for the Cyrus {configdir}/proc directory, like so:

  tmpfs   /var/spool/cyrus/config/proctmpfs   size=25M,nr_inodes=10k  0  0


On my system, /dev/shm has an inode limit as well:

FilesystemInodes   IUsed   IFree IUse% Mounted on
tmpfs1025011   1 10250101% /dev/shm


Maybe it would be better to create {configdir}/lock as a separate tmpfs? 
Something like:


  tmpfs   /var/spool/cyrus/config/proctmpfs   size=25M,nr_inodes=1k  0  0


There is no reason for lock files to persist between Cyrus restarts, 
right?


Andy
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sieve and global/default scripts

[Andrew Morgan]

On Fri, 6 Feb 2015, Eugene M. Zheganin wrote:


Hi.

On 06.02.2015 17:05, Niels Dettenbach wrote:

The sieve script is (depending from where or what it should do) in a global
place (or domain) - logged in as cyrus admin - - like imap/sieve/global or by
SIEVE shell - within the cyrus system and then INCLUDED by user scripts which
should use this.

Yeah, but the manual states that global scripts aren’t applied on
incoming messages by default, [...] which made me think that there can
be a way along with the way when users link them manually. Okay, now I
see that linking global scripts is the only way.

Do I understand correctly that now, in order to create a default script
for each  already existing uses I should link the default script for them ?


At our site, when a new mailbox is created we also load a default sieve 
script at the same time.  Have a look at the scripts here:


  http://people.oregonstate.edu/~morgan/cyrus/scripts/

Specifically, look at create_user_inbox.pl and set_user_initial_sieve.pl.

Andy
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Auto create folders

[Andrew Morgan]

On Mon, 26 Jan 2015, John Mok wrote:

 Hi Andy,

 Thank you for your prompt reply.

 How do you create mailboxes now?

 I used cyradm and createmailbox, e.g. createmailbox
 user/username@DOMAIN, to create mailboxes.

 Any idea how to create folder in cyradm? Simply createmailbox
 user/username@DOMAIN/spam, and then set ACL permissions for
 user/username@DOMAIN/spam ?

Yes, that's exactly what you need to do!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Auto create folders

[Andrew Morgan]

On Fri, 23 Jan 2015, John Mok wrote:

 Hi,

 I have been using Cyrus IMAP 2.4.17 on Debian 7 with Kerberos / GSSAPI 
 authentication.

 I would like to auto-create one or more folders upon mailbox creation, 
 e.g. a spam folder to store potential spam mails for spamassassin 
 learning.

 On the other hand, how to prevent such folders from deletion by users?

 Thanks a lot.

How do you create mailboxes now?

We create mailboxes using a Perl script, and that script also creates a 
junk-mail folder with an annotation to delete messages older than 30 
days.

You can also alter the folder's permissions to prevent users from deleting 
them.

Does that help?

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: restore from cyrdump

[Andrew Morgan]

On Tue, 16 Dec 2014, Patrick Goetz wrote:

 On 12/16/2014 4:11 AM, Michael Menge wrote:
 We also don't use snapshots or stop cyrus for backup.

 But as a complete restore of our mail storage with normal backuptools
 would take fare too long, we uses cyrus sync for disaster recovery.
 For the normal backup we use a combination of delayed expung (14 days)
 and normal filesystem backup. In most cases where the mail is still
 in the filesystem and can be restored by unexpung, and in the rare
 cases where the mail has been expunged i have to run reconstruct anyway.


 I haven't used reconstruct in such a long time that I've forgotten what
 can't be reconstructed from the partition-default user mail files.

 Quotas are maintained separately.  Suppose annotations.db and
 mailboxes.db are both corrupted or inaccessible.  Does this mean the
 seen status is gone?  What else?

I forgot about one additional thing we do - we dump the mailboxes.db to a 
flat file once an hour via cron.  That would allow us to (mostly) recover 
from a corrupted mailboxes.db file.  Just like a full restore, we would 
need to run a reconstruct on every mailbox, I think.

I haven't thought about annotations.db.  We do use that for a few things.

Seen status is stored in {configdir}/user/prefix/username.seen files 
(skiplist format) here.  Those are backed up as flat files.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: restore from cyrdump

[Andrew Morgan]

On Tue, 16 Dec 2014, Patrick Goetz wrote:

 On 12/16/2014 01:42 PM, Andrew Morgan wrote:

 I forgot about one additional thing we do - we dump the mailboxes.db to
 a flat file once an hour via cron.  That would allow us to (mostly)
 recover from a corrupted mailboxes.db file.  Just like a full restore,
 we would need to run a reconstruct on every mailbox, I think.


 I thought the whole point of reconstruct was to rebuild mailboxes.db,
 but then I took another look at the reconstruct man page and noticed:


   -m NOTE: CURRENTLY UNAVAILABLE
  Rebuild the mailboxes file. Use whatever data in the
  existing  mailboxes file it can scavenge, then scans
  all partitions listed in the imapd.conf(5) file for
  additional mailboxes.


 now it's no longer clear to me what reconstruct does.  I guess rebuild
 the {configdir}/user/prefix/username/cyrus.* files?

Yes, and add newly found mailboxes to mailboxes.db.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: restore from cyrdump

[Andrew Morgan]

On Mon, 15 Dec 2014, Patrick Goetz wrote:

 On 12/10/2014 03:47 AM, Willy Offermans wrote:
 I'm not sure what you mean with ``all the metadata'', but there are user
 flags saved into the cyrdump file as well. I never performed the whole
 cycle of dump and restore (probably nobody did so far), so I cannot tell
 you that all metadata is available in the dump file. See my question above!



 A while back I was working on an email server that (unbeknownst to me)
 was connected to a UPS, but with an external disk array that was plugged
 in to an outlet on the UPS that was not battery-backed.  This site had
 frequent power problems, so it turns out that power cycling a disk array
 while the server stays up is an awesome way to corrupt your entire file
 system.

 Since I didn't know what I was doing at the time, I restored

   partition-default: /home/cyrus

 without also restoring

   configdirectory: /var/lib/cyrus

 I was consequently confused when no mailboxes showed up, and had to then
 learn about and use reconstruct -r on each individual mailbox
 (cyrreconstruct on debian/Ubuntu) in order to reconstruct the
 /var/lib/cyrus/*.db files.

 I think the main database files you need are mailboxes.db and
 annotations.db (can someone confirm this?)

 This still leaves the question of how best to back up a cyrus mailstore.

 Bron mentioned that most people are using LVM snapshots.  I don't see
 how using btrfs/LVM/ZFS snapshots can save you from a race condition
 between when the cyrus user directory is updated and when mailboxes.db
 is updated.  The only way I would trust this is by doing this:

1. Stop cyrus
2. Snapshot
3. Restart cyrus


 cyrdump:  near as I can tell the only useful purpose this serves is to
 assemble all email messages into a single mbox file (can anyone
 confirm this)?

 ctl_mboxlist: this seems useful for making a human readable copy of the
 mailboxes.db file, but I'm not sure how this could be useful for
 disaster recovery, given the previously mentioned issue about keeping
 the mailboxes.db file synchronized with the contents of the user dir.

 So, given a simple mail server (i.e. no murder + replication), and when
 using a filesystem (e.g. ext4 or XFS) which doesn't do snapshots, it
 would appear that the only safe way to backup up a cyrus mailstore is to
 either using something like imapsync, or

1. Stop cyrus
2. tar cvf /some/safe/place/user.tar {default-partion}
3. tar cvf /some/safe/place/cyrusdb.tar {configdirectory}
4. Restart cyrus

 The way I've used imapsync in the past required copying mail folders per
 authenticated user account; i.e. something like

 imapsync --host1 my_host1 --authmech1 LOGIN --user1 my_user1 --password1
 x --host2 my_host2 --authmech2 PLAIN --user2 my_user2 --password2 x

 which in particular means knowing everyone's passwords.  This is
 entirely unworkable for larger sites, and I'm not sure if there is a
 trick for getting around this.

We are a large site.  We have 3 backend servers in a Murder cluster with 
about 25,000 mailboxes per backend.

Unless you are going to use Fastmail's fancy backup method that actually 
locks the mailbox in Cyrus, I don't think there is a way to take a 
perfectly consistent backup.

That said, we didn't care about achieving perfection, so we just perform a 
normal filesystem backup.  We use EMC Networker, but it isn't doing 
anything fancy.  It just walks the entire directory tree looking for 
changed files to backup.

Yeah, there is the potential for a race condition.  You really only need 
each mailbox to be consistent though, and the odds of a mailbox changing 
while it's being backed up are sufficiently low here.  In the event of a 
disaster (total loss of filesystem), we'll be restoring from our most 
recent backup, which will be anywhere from 0-24 hours old.  If we need to 
recover an individual mailbox, we can get everything using delayed delete. 
Either way, we'll be running reconstruct on the mailbox(es).

I see people talk a lot about fancy ways to back up Cyrus, but we've just 
never had a need.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: I not can register the directory manual added to a new mailbox from backup.

[Andrew Morgan]

On Fri, 12 Dec 2014, Manuel Vazquez wrote:

 I have a cyrus imap to ldap server where the user email are store.

 I convert accidentally a user in Deleted user. I do not know how restore
 this email user in cyrus imap databases.

 I do this step:

 - I maked a new directory mailbox,
 -  I restore the configuration on the imap server
 -  I recovery the information from the directory backup with a rsync command
 -  I do a recontruct command over this mailbox. But this reconstruct not
 register the old directory on the new mailbox.

 Thanks for your help and sorry for my poor english.

If the mailbox still exists in the DELETED hierarchy, then you can simply 
rename the mailbox using cyradm:

   rename DELETED.user.username user.username


This is only true if you are using delayed delete mode in imapd.conf:

   delete_mode: delayed


If the mailbox no longer exists in the DELETED hierarchy, follow these 
steps:

1. Re-create the mailbox using cyradm:

   cm user.username

2. Recover the message files (#. files) but not the cyrus.* files

3. Run su cyrus -c '/usr/local/cyrus/bin/reconstruct -x -f 
user.username'

4. Run su cyrus -c '/usr/local/cyrus/bin/quota -f user.username'


Let us know if you have any questions!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: saslauthd question

[Andrew Morgan]

On Thu, 11 Dec 2014, Patrick Boutilier wrote:

 On 12/11/2014 02:34 PM, Patrick Goetz wrote:
 Surely someone on this list will know the answer to this question.
 
 Given sasl_pwcheck_method: saslauthd, with authentication mechanism=pam
 
 I'm trying to track down how saslauthd knows that the cyrus PAM service
 file is called imap; i.e. /etc/pam.d/imap.
 
 Is this just built in?  I can't find a configuration for it anywhere.
 
 
 
 
 Cyrus Home Page: http://www.cyrusimap.org/
 List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
 To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 


 Harcoded in imapd.c

 if (sasl_server_new(imap, config_servername 


I thought the PAM name was taken from the service name in /etc/cyrus.conf, 
but my own configuration seems to indicate that it must be hardcoded for 
each service.  I only have PAM files for imap, lmtp, and sieve 
although I have other service names for some of them.

I guess it's just the imapd.conf config variables that are allowed to be 
prefixed with the service name.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: saslauthd question

[Andrew Morgan]

On Thu, 11 Dec 2014, Patrick Goetz wrote:

 On 12/11/2014 12:45 PM, Andrew Morgan wrote:
 I only have PAM files for imap, lmtp, and sieve
 although I have other service names for some of them.


 I don't understand why you have PAM files for lmtp and sieve, but most
 particularly lmtp.  lmtpd is just a local daemon that transfers stuff
 from your smtp server to cyrus.  Are you running cyrus and smtpd on
 different servers?  If so, what does the PAM lmtp configuration look like?

 I don't know anything about sieve, but thought the filters where all
 internal, too; hence not in need of authentication.

We have multiple smtp servers that accept incoming mail plus we run a 
Cyrus Murder cluster.  There is a lot of lmtp over the network happening. 
:)

The PAM configuration for lmtp, sieve, and imap is identical (auth against 
LDAP).

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyr_expire: deliver.db inconsistent pre-checkpoint bailing out

[Andrew Morgan]

On Sat, 6 Dec 2014, Vincent Fox wrote:

 Hi,

 We are running quite old Cyrus 2.3.8 (near retirement) and last couple
 of nights it started kicking up this error during nightly expire run.

  cyr_expire[3409]: [ID 386572 local6.error] db
 /var/cyrus/imap/deliver.db, inconsistent pre-checkpoint, bailing out

 Any guidance on best course of action?  We do have nightly snapshots of the
 entire filesystem, so I could roll back to deliver.db from 3 or 4 days ago.
 Or would that create consistency issues with other databases only making
 things worse.  Or could I stop Cyrus and only reconstruct deliver.db?

 The server has thousands of users and 500gig+ of files so I don't relish
 the idea of long downtime for full reconstruct.

I would stop Cyrus and delete (move out of the way) deliver.db.  Let Cyrus 
recreate it at startup.  deliver.db is used for duplicate message 
suppression and tracking vacation responders.  The consequences of 
deleting it are minimal.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Various errors from cyrus maintenance processes

[Andrew Morgan]

On Wed, 29 Oct 2014, Boylan, Ross wrote:

 I recently installed Cyrus2.4.16-4+deb7u2  on a new Debian wheezy system.  
 After an extended period of difficulties authenticating I can now access it.  
 sasldb has 2 users, ross and cyrus.  The only mailbox I created was user.ross.
 Now I see this in my logs (from after the authentication trouble was fixed):
 Oct 28 04:01:00 wheezy4 cyrus/cyr_expire[26711]: DIGEST-MD5 common mech free
 Oct 28 04:01:00 wheezy4 cyrus/tls_prune[26710]: DBERROR: opening 
 /var/lib/cyrus/tls_sessions.db: cyrusdb error
 Oct 28 04:01:00 wheezy4 cyrus/master[7760]: process 26710 exited, status 1
 Oct 28 04:01:00 wheezy4 cyrus/cyr_expire[26711]: IOERROR: opening index 
 user.ross: System I/O error
 Oct 28 04:01:00 wheezy4 cyrus/cyr_expire[26711]: unable to open mailbox 
 user.ross
 Oct 28 04:01:00 wheezy4 cyrus/cyr_expire[26711]: Expunged 0 out of 0 messages 
 from 0 mailboxes

 Do these messages indicate any real problems?  If so, how can I diagnose 
 or fix them?

Yes, those are real problems.  They look suspiciously like the errors you 
would get if the files in /var/lib/cyrus are owned by root instead of 
cyrus.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

RE: Various errors from cyrus maintenance processes

[Andrew Morgan]

On Wed, 29 Oct 2014, Boylan, Ross wrote:

 I've found at least a partial explanation: I forgot to mount the 
 partition with /var/spool/cyrus.  I had installed cyrus onto the file 
 system beneath the mount, but the mount was in effect when I created 
 user.ross.* So it wasn't there to open.

 Does that explain the errors under /var/lib/cyrus as well (DBERROR: 
 opening /var/lib/cyrus/tls_sessions.db: cyrusdb error)?  There was no 
 mount on top of it.

tls_sessions.db may be created on demand, I forget.  I don't know if you 
have SSL/TLS enabled on your host, but you could try making an SSL/TLS 
connection.

 Both before an after the mount all files at and under /var/lib/cyrus and 
 /var/spool/cyrus are owned by cyrus.

 Is there a way I can retrigger the jobs that caused the errors shown in the 
 log?  Do I run them as root or cyrus?
 cryus.conf has
  # this is only necessary if using duplicate delivery suppression
delprunecmd=/usr/sbin/cyrus expire -E 3 at=0401

# this is only necessary if caching TLS sessions
tlsprunecmd=/usr/sbin/cyrus tls_prune at=0401
 I'm guessing I run the commands in quotes.

Sure, just make sure you run them as user cyrus.  You could also see if 
any errors are reported when you start Cyrus or when you make an IMAP 
connection.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Upgrading from 2.2 to 2.4 (slow cyr_expire)

[Andrew Morgan]

On Thu, 16 Oct 2014, Jay Sekora wrote:

 Hi.  I recently tried to upgrade/migrate our Cyrus deployment from
 2.2.13 (on Debian) to 2.4.17 (on Ubuntu 14.04).  In our environment,
 user mailboxes (about 3TB of them) are on iSCSI volumes; everything else
 is on local disk (which I rsync'ed).

 I ran into some delays to do with the storage backend configuration, so
 I didn't actually get to the point of starting imapd on the new server
 until disturbingly close to the end of our announced downtime window.

 When I did, I saw that imapd wasn't responding and cyr_expire was
 running.  I was expecting that, but eyeballing what it was doing via
 strace suggested that it would have taken *over twenty hours* to walk
 all the mailboxes.  (I'm guessing that cyr_expire was doing, perhaps as
 a side effect, the full re-parse of all messages, which may take a
 while mentioned under Upgrading from 2.4.3 at
 http://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-upgrade.php.)  So
 we announced that we were backing out of the upgrade.

 However, as I was getting ready to back out, I killed the cyr_expire by
 hand, and at that point imapd started responding and I was able to log
 in and see my own mail (which I know cyr_expire hadn't gotten to).  It
 was a little slow to initially show my my mail, which suggests that
 maybe Cyrus was running cyr_expire or its equivalent after I
 authenticated and before showing my my inbox, but that led me to wonder
 whether it might be safe (when we repeat the migration) to kill the
 cyr_expire on initial startup so that Cyrus will start talking IMAP
 right away, and run it in the background.

 In case it matters, we have a bunch of emeritus users who occasionally
 check their mail at our site but don't use it on a day-to-day basis, and
 a bunch of users who forward their mail elsewhere and leave a copy on
 our IMAP server as a backup, and a lot of our heavy users are
 sophisticated enough not to leave all their mail in their inboxes so
 when we open the floodgates a very large fraction of that ~3TB is not
 going to be looked at immediately.

You could comment cyr_expire out of cyrus.conf before you upgrade.  After 
a few days, you could uncomment cyr_expire and send a HUP signal to the 
Cyrus master process to have it re-read cyrus.conf.

Remember, the mailbox will be upgraded anytime it is opened.  That will 
occur when a user checks their mailbox AND when new mail is delivered. 
Still, it seems reasonably safe.  Your best bet is to schedule the upgrade 
at a time of low usage and try to touch as many mailboxes as possible 
before things get really busy.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Imapd - more processes than maxchild

[Andrew Morgan]

On Thu, 31 Jul 2014, Fabio S. Schmidt wrote:

 Hi !

 I'm trying to fix an environment which has been growing without proper
 attention.

 There are about 7000 inboxes but only 5000 are active and the maxchilds
 parameter is set as 2000 causing a lot of timeouts when the clients try
 to connect. I thought as a first approach trying to increase this parameter.

 I have noticed that even with the maxchilds parameter set as 2000 there
 are about 2020 processes open, is this behaviour normal?  The version in
 use is 2.4.12.

Are there really 2000+ IMAP clients trying to connect?  Try ps -ef | grep 
imapd | wc -l.

It's also possible that there is an IMAP client running wild, making 
hundreds of connections.  The output of netstat -nt might show you if 
there are a lot of connections from a single IP address.

If you really need to allow more connections, increase the maxchilds 
parameter.  Beware that you don't overload the server, either with too 
much I/O or not enough RAM available!  :)

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: xfer problems between 2.3.15 and 2.4.17

[Andrew Morgan]

On Tue, 10 Jun 2014, gavin.g...@ed.ac.uk wrote:

 Hi Wes,

 It looks like the whole mupdate thing is working perfectly well.
 If I create a folder while connected to my 2.4.17 frontend then the logs show 
 the backend issuing the cmd_set and then a bunch of cmd_find going out 
 including to the frontend in question. Furthermore the new folder really is 
 there in the mailboxes.db on the frontend. So in a way that's reassuring, but 
 then why is the frontend telling email clients that the folder doesn't exist 
 when a request to subscribe to it comes in? We aren't seeing any kick_mupdate 
 getting logged.

I'm pretty sure your problem is that you have the proxyservers variable 
set in imapd.conf on your frontend.  See this message from the archives:

   
http://asg.andrew.cmu.edu/archive/message.php?mailbox=archive.info-cyrussearchterm=Murder%20mailbox%20create%20race%20conditionmsg=54193

I ran into this same problem, which was introduced by changes in v2.4.13. 
The imapd.conf manpage says:

   proxyservers: none
A list of users and groups that are allowed to proxy for  other  users, 
 separated  by
spaces.  Any user listed in this will be allowed to login for any other 
user: use with
caution.  In a standard murder this option should ONLY be set on 
backends.  DO NOT SET
on frontends or things won't work properly.

Let us know if removing proxyservers from your frontends fixes the 
problem!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: xfer problems between 2.3.15 and 2.4.17

[Andrew Morgan]

On Thu, 5 Jun 2014, gavin.g...@ed.ac.uk wrote:

 As you may be aware we are attempting this and have run into various
 problems.

 Currently we have a mixed murder of 2.3.15 backends and 2.4.17 backends.
 We are now fairly confident that we can xfer accounts succesfully between
 these backends. The problems we had appear to have been with a very small
 number of accounts on our older backends that had corrupt cyrus.index
 files.

 However we are now having trouble configuring frontends that will work
 with this mixed murder environment while we xfer our users accross.

 If we use our existing 2.3.15 frontends then users have who have been
 migrated lose the ability to see other accounts in the Other Users name
 space.

 On the other hand if we introduce 2.4.17 frontends then we see strange
 behaviour around folder creation. Clients can create the folders but
 autosubscription fails with the client being told the new folder doesn't
 exist. If one waits a minute or two one can manually subscribe to the
 folder.

This is tickling my memory, but I can't recall exactly what it was.  I 
remember running into a problem like this as well.  Something about the 
frontend's mailbox database not being updated in a timely fashion...

 So far we have not upgraded the mupdate master. Is this a mistake?

 In terms of the frontend config we have added

 suppress_capabilities: ESEARCH QRESYNC WITHIN XLIST LIST-EXTENDED

 to the 2.4.17 frontends, otherwise the config is identical to our 2.3.15
 frontends. Is there any other config changes we should be aware of?

I used the following when I upgraded from 2.3 to 2.4:

   suppress_capabilities: ESEARCH LIST-EXTENDED QRESYNC WITHIN XLIST ENABLE 
SORT=DISPLAY

There was a thread I started back in October 2011 with the subject 2.3 to 
2.4 Murder upgrade where I ran through the upgrade and the workarounds I 
had to make.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Reconstruct a downgrade?

[Andrew Morgan]

On Mon, 14 Apr 2014, Charles Bradshaw wrote:

 I'm trying to move my cyrus imap from Fedora 17 to Centos 6.5,
 unfortunately the package versions of cyrus-imapd appear to be a
 downgrade from version 2.4 to 2.3

 I have copied /var/lib/imap and /var/spool/imap and the necessary /etc/.. 
 conf files

 cyrus-imapd appears to run correctly and I can connect a client (Evolution).
 The clients mailboxes appear, but Evolution throws this error:
 IMAP command failed: Mailbox has an invalid format
 And /var/log/maillog has this messages:
 ...
 Apr 14 16:03:55 dell2600-1 imaps[3058]: fetching user_deny.db entry for 
 'x...@my.domain.com'
 Apr 14 16:04:10 dell2600-1 imaps[3058]: Future index version: 
 my.domain.com!user.xxx (12  10)
 Apr 14 16:04:10 dell2600-1 imaps[3058]: fetching user_deny.db entry for 
 'x...@my.domain.com'

 After deleting cyrus.index, cyrus.header annd cyrus.cache from the user 
 x...@my.domain.com inbox directory
 and a reconstruct -r user/x...@my.domain.com mail boxes and messages are 
 restored successfully.

 But there are now thousands of, presumably, previously deleted messages and 
 the 'seen', 'replied' etc flags
 are gone!

 Is there a way to reconstruct the necessary db files so that I don't loose 
 the flags?

 I tried a build from source of a later version but failed with dozens of 
 compiler errors!

 Thanks in advance, Charles Bradshaw

I sure would try to get Cyrus v2.4.17 to compile.  v2.3 is very old...  We 
would be happy to help you compile v2.4.17 on CentOS 6.5.  Alternatively, 
there are Source RPMs available at:

   http://www.invoca.ch/pub/packages/cyrus-imapd/

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus-imapd memory tuning

[Andrew Morgan]

On Mon, 10 Mar 2014, Marco wrote:

 My server is:
 Red Hat Enterprise Linux Server release 6.3 (Santiago)
 Without problems I read something like this:

  total   used   free sharedbuffers cached
 Mem:   80619767651020 410956  013559643412788
 -/+ buffers/cache:28822685179708
 Swap:  4194296  321804162116

 procs ---memory-- ---swap-- -io --system--
 -cpu-
  r  b   swpd   free   buff  cache   si   sobibo   in   cs us
 sy id wa st
  2  0  32180 386880 1356476 342371200   643   327   25   18
 10  4 81  5  0

Those numbers look okay.  Obviously more memory is nice for caching disk 
I/O, but you're doing fine.

 current cyrus.conf:
 SERVICES {
   # add or remove based on preferences
   imap  cmd=imapd listen=imap prefork=5
   pop3  cmd=pop3d listen=pop3 prefork=3
   sieve cmd=timsieved listen=sieve prefork=0
   lmtp  cmd=lmtpd -a listen=lmtp prefork=0
 }

 I have to prevent memory issue when some oddity forces clients to make
 DOS on Cyrus. So I would like to configure the maxchild cyrus
 parameter for imap. I would like to set this value to avoid memory
 issue during normal work, having a known value of system RAM.

Here is what I'm using on a Cyrus backend with 8GB of RAM:

   imap  cmd=/usr/local/cyrus/bin/imapd listen=imap proto=tcp4 
prefork=10 maxchild=4000
   imaps cmd=/usr/local/cyrus/bin/imapd -s listen=imaps 
proto=tcp4 prefork=10 maxchild=1000
   sieve cmd=/usr/local/cyrus/bin/timsieved listen=sieve 
proto=tcp4 prefork=0 maxchild=100
   lmtp  cmd=/usr/local/cyrus/bin/lmtpd listen=lmtp proto=tcp4 
prefork=1 maxchild=100

I tuned the maxchild setting to balance our usage patterns between the 
imap and imaps ports.  Our highest open connection count is about 1500 
total, so there is quite a bit of headroom.

 I see that an IMAPD process takes in average 22-25MB. With 8GB RAM,
 the server would swap already with less than 400 conns; it not
 happens, so this evaluation is wrong or too many conservative. I think
 that I better consider differences between RSS and SHR memory to
 tuning imapd processes number, but I'm not sure.

 Could you help me in this tuning? In particular I'm interested on
 relation between memory usage and maxchild imapd processes.

I'm running a Cyrus Murder cluster with separate frontends and backends, 
so my numbers won't directly correlate.  On a backend with about 700 imapd 
processes, I have the following memory usage:

  total   used   free sharedbuffers cached
Mem:   82000928136084  64008  027351241614016
-/+ buffers/cache:37869444413148
Swap:  1951736  365441915192

 Meanwhile I would also tune the maxfds parameter. With lsof I measure
 about 60 opened files by each imapd process. If I have 400 imapd
 processes it means a 'ulimit -f' global system of 60*400=24000. This
 is wrong, because I currently have a 4096 limit and I never had
 problems. Maybe do I have to consider only 'Running' processes to
 compute this treshold?

My Cyrus init script does:

# Crank up the limits
ulimit -n 209702
ulimit -u 4096
ulimit -c 102400

This particular backend has:

root@cyrus-be1:~# cat /proc/sys/fs/file-nr
25696   0   819000

Again, this is with about 700 imapd processes.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: replication does not work

[Andrew Morgan]

On Fri, 21 Feb 2014, Marcus Schopen wrote:

 Hi,

 Am Freitag, den 21.02.2014, 17:23 +0100 schrieb Willy Offermans:
 [...]


 I can answer my own question. I was indeed missing the authentication
 mechanism. I added sasl_mech_list: PLAIN LOGIN to imapd.conf on the
 back-end server and the replication worked.

 So I wonder how I can tell sync_client which authentication mechanism to
 use? It seems like a feature request to me? or a hidden option to the
 sync_client executable.

 That's an interesting question. I had a similar problem this week to
 force master and slave to sync via TLS. As long as the banner on slave
 side offered DIGEST-MD5 CRAM-MD5 NTLM LOGIN PLAIN to connection plain.
 I set allowplaintext: no and sasl_mech_list: PLAIN on slave and now
 both are talking PLAIN via TLS. So if there is an option on master side
 to force to login using eg. CRAM-MD5 then there might be an option too
 to force TLS.

 I'm playing with replication now and testing what happens if one deletes
 e-mails on the back-end server and not on the client. Will these mails be
 restored on the back-end by replication and when?

 Don't understand, what is the client, the replica server?

Have you looked at the sasl_minimum_layer option?

sasl_minimum_layer: 0
 The minimum SSF that the server will allow a client to
 negotiate.  A value of 1 requires integrity  protection;  any
 higher value requires some amount of encryption.


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Best distro for Exim/Cyrus

[Andrew Morgan]

On Wed, 12 Feb 2014, Paul O'Rorke wrote:

 So I seem to be getting confused about when the SSL is used. Ideally I'd like 
 to use SSL and authentication for SMTP and IMAP. Is it that the LMTP needs 
 authentication and it's not?  I did use in /etc/cyrus.conf

   lmtpcmd=lmtpd -a listen=localhost:lmtp prefork=0
   maxchild=20

Put the -a inside the quotes, like this:

lmtpcmd=lmtpd -a listen=localhost:lmtp prefork=0 maxchild=20

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Best distro for Exim/Cyrus

[Andrew Morgan]

On Mon, 10 Feb 2014, Paul O'Rorke wrote:

 Hi again Cyrus list,

 still trying to find a definitive resource to use to get this mail server up 
 and running.  Does anyone know of a good howto for setting up 
 Debian/Exim/Cyrus?  I think this is the combination I want to move from the 
 Centos/Exim/Dovecote box I inherited but I must confess to really struggling 
 here.

 It seems there are lots of different variations on the set up, what config 
 files are used for different distros and even versions of Debian. I'm not 
 finding consistency between what I am seeing on my system and the things the 
 guides I'm using are suggesting I should see.

 I appreciate that setting up such a mail server does require significant 
 knowledge, more of which I hope to acquire through this project.  I am 
 surprised however at the difficulty I am experiencing here.  I would be more 
 than happy to get this started with a simple config and spend more time 
 building a better server down the road if anyone can point me in the right 
 direction for a good guide/howto.

 Maybe I've bitten off more than I can chew with Exim/Cyrus. Surely it should 
 be possible to set such a mail server up in a day or so?

 Hoping to find a bone here...  :-(

I'm using Debian here, although I am compiling from source rather than 
using the Debian Cyrus packages.  Back when I started, the Cyrus packages 
were very out of date in Debian.  However, it appears that Debian packages 
in Wheezy (stable) are Cyrus v2.4.16 (plus patches), so I recommend using 
Debian's Cyrus packages for your situation.

I suggest installing the following packages:

cyrus-admin
cyrus-clients
cyrus-doc
cyrus-imapd

which should give you an IMAP server.

One of the tricky things with Cyrus is authentication.  It is very 
flexible.  I'm not sure how the Debian packages will configure 
authentication.  Perhaps they default to local unix authentication or 
cyrusdb (cyrus-only auth).

After that, you need to get Exim to deliver mail to Cyrus via LMTP.  I 
don't use Exim, so I can't comment on that part.  There must be a howto 
out there somewhere!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Postfix + Cyrus Sasl problem

[Andrew Morgan]

On Wed, 18 Dec 2013, Eric Abreu Alamo wrote:


 Hello all:

  Recently I have been trying to install and configure Postfix + Cyrus + 
Sasl auth (with smtp auth) and i found the following problem. I have 
installed and configured Cyrus, Postfix and Sasl, and everything is 
right until smtp auth. When I edit the /etc/default/saslauthd file and I 
change the line OPTIONS=-c -m /var/run/saslauthd by OPTIONS=-c -m 
/var/spool/postfix/var/run/saslauthd where postfix chroot directory is, 
and i run dpkg-stateoverride with 750, 7 for root user owner and 5 sasl 
group, I restart those services and after do that, I got the smtp auth 
but Cyrus authentication service fail, then I can't to access through 
imap service. Somebody have configured those daemons before? Im using 
Ubuntu 12.04 LTS OS.


Try setting this in /etc/imapd.conf:

  sasl_saslauthd_path: /var/spool/postfix/var/run/saslauthd/mux

Found it here:

  http://www.cyrusimap.org/docs/cyrus-sasl/2.1.25/options.php

Andy
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: allowplaintext: no and aggregates

[Andrew Morgan]

On Fri, 6 Dec 2013, sofkam wrote:

 We are running a murder aggregate:

Front-end db
Three front-end servers
One back end server

 Starting next year we are no longer permitting unencrypted connections
 (long time coming).  Our supported authentication mechanisms are:

  sasl_mech_list: PLAIN LOGIN

 When I change allowplaintext to no, will the back-end and front-end
 servers be able to communicate with each other?  Or, do I need
 to add an additional non-plain authentication mechanism?  Will the
 db-server require plain-text logins?

Good question...  My backend servers are still allowing plaintext logins, 
and all the proxy connections from the frontends are using plaintext.  My 
frontends have allowplaintext:0.

I suppose I could try this in my test environment...

Actually, it looks like my test environment has allowplaintext:0 
everywhere, and connections from the frontends use PLAIN+TLS.  Now I just 
need to put this in place in my production environment too!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Undestanding maillog?

[Andrew Morgan]

On Thu, 24 Oct 2013, Charles Bradshaw wrote:

 Hello List

 Sorry about the long post.

 I am trying hard to get to understand my /var/log/maillog when
 connecting to cyrus-imapd.

 When I open Evolution and connect /var/log/maillog says:
 Oct 24 21:52:33 dell2600 imaps[15186]: starttls: SSLv3 with cipher 
 DHE-RSA-CAMELLIA256-SHA (256/256 bits new) no authentication
 Oct 24 21:52:33 dell2600 imaps[15186]: login: testbox.mydomain.com 
 [192.168.0.8] m...@mydomain.com DIGEST-MD5+TLS User logged in 
 SESSIONID=dell2600.bradcan.homelinux.com-15186-1382647953-1

 What does the first log entry above no authentication mean? Imediatly 
 followed by User logged in!

It means the IMAP client did not authenticate with an SSL client 
certificate.  SSL was used for connection encryption, but not 
authentication.  The client authenticated using the DIGEST-MD5 method 
after an encrypted SSL connection was established.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: help with two cyrus systems merge

[Andrew Morgan]

On Mon, 19 Aug 2013, Sandra Regina de Souza wrote:

 Hi there!

We have 2 e-mail servers that have cyrus 2.3 installed on each
 other. But we want to migrate these two ones to only one new server
 with cyrus-2.4.
Is there a way that we could do that to preserve seen flags?
 I tried to merge the two mailboxes.db into a mailboxes.txt
 file , and generate a mailboxes.db, but  it did not work.
I have read that in cyrus-2.4  cyrus.index file
 content is different from cyrus-2.3.
I have 5000 acconunts and  tried to use imapsync,  but it is too
 slow .
Thank you for your help.

I can think of a few ways to do this:

1. Use imapsync
2. Use Cyrus replication (just a guess)
3. Use Cyrus Murder clustering
4. Use Rsync into 2 different partitions on the new server

Imapsync is not a bad option.  Write a script to fire off multiple 
imapsyncs at once.  Run it to completion.  Then run it again.  Then run it 
again.  Schedule your outage.  Run imapsync every night up to the day of 
your outage.  Block your users from accessing Cyrus, run a final imapsync, 
then stop Cyrus on the 2 old servers and start using the new server.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Aw: Re:

[Andrew Morgan]

On Wed, 24 Jul 2013, Stefan Schlörholz wrote:


Hello Simon,


Did you try running reconstruct -r -f ...?


I did try to run reconstruct -r user.paul. The -f switch is not 
known/accepted by my cyradm.


Use the command-line program named reconstruct, not the command 
reconstruct inside of cyradm.


Andy
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyr_expire deadlock

[Andrew Morgan]

On Tue, 21 May 2013, Łukasz Michalski wrote:


Hi,

I am running cyrus imapd 2.4.11 on linux machine.

Today I had a deadlock involving cyr_expire and imapd process.

imapd was locked on (strace):

fcntl64(17, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}^C 
unfinished ...


where fd=17 is a user index file (lsof):

imapd 32314cyrus   17u  REG8,3 30944   10462461 
/var/spool/imap/domain/c/cenbench.pl/a/user/arek^dydo/cyrus.index


Unfortunetaly I did not check cyr_expire with strace, but lsof showed this:

cyr_expir 24356 cyrus0u   CHR1,3  0t0  509 /dev/null
cyr_expir 24356 cyrus1u   CHR1,3  0t0  509 /dev/null
cyr_expir 24356 cyrus2u   CHR1,3  0t0  509 /dev/null
cyr_expir 24356 cyrus3u   CHR1,3  0t0  509 /dev/null
cyr_expir 24356 cyrus4u   CHR1,3  0t0  509 /dev/null
cyr_expir 24356 cyrus5u   REG8,2  144 19196113 
/var/lib/imap/annotations.db
cyr_expir 24356 cyrus6u   REG8,213300 18911268 
/var/lib/imap/mailboxes.db

cyr_expir 24356 cyrus7r  FIFO0,5  0t0  5678136 pipe
cyr_expir 24356 cyrus8w  FIFO0,5  0t0  5678136 pipe
cyr_expir 24356 cyrus9r  FIFO0,5  0t0  5678137 pipe
cyr_expir 24356 cyrus   10w  FIFO0,5  0t0  5678137 pipe
cyr_expir 24356 cyrus   11u   REG8,2   171032 19196126 
/var/lib/imap/deliver.db
cyr_expir 24356 cyrus   12uR  REG8,20 26961663 
/var/lib/imap/lock/domain/c/cenbench.pl/a/user/arek^dydo.lock
cyr_expir 24356 cyrus   13u   REG8,330944 10462461 
/var/spool/imap/domain/c/cenbench.pl/a/user/arek^dydo/cyrus.index


There was 50 imapd processes (my upper limit) in locked on the same file 
and a single cyr_expire. After killing cyr_expire I had to manually kill 
all imapd processes to allow master to spawn new ones.


Not that my cyrus works on really, really slow machine. It is VM running 
under KVM with I/O access varying from 5 to 60MB/s (as shown by hdparm -t)


Please let me know what can I do to trace it better next time.

Regards,
Łukasz


You probably will want to run reconstruct on that user's mailbox because 
the cyrus.index file may be corrupted at this time.  Also, you should 
upgrade to Cyrus v2.4.17 if you can.  There have been a large number of 
bugfixes since your version, one of which may be the cause of your 
deadlock.


Andy
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Refuse IMAP without encryption

[Andrew Morgan]

On Tue, 23 Apr 2013, Paul van der Vlis wrote:

 Hello,

 Is it possible to refuse IMAP-access without encryption like TLS or SSL?
 I think this would be a good idea for security.

 And I would like to make an exception for localhost for the webmail. The
 webmail (Sogo) can do TLS or SSL, but normally I don't do that for
 localhost.

 I am using Cyrus 2.4.16 from Debian 7 (Wheezy).

You can create a second service entry for imapd in cyrus.conf.  Have it 
listen on localhost and on a different port, such as 1143.  In imapd.conf, 
set:

   service_name_allowplaintext: 1

Where service_name is the name of the localhost service in cyrus.conf. 
For example:

   localimap cmd=/usr/local/cyrus/bin/imapd listen=localhost:1143 
proto=tcp4 prefork=10 maxchild=100

Then in imapd.conf:

   localimap_allowplaintext: 1


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: MD5 Passwords in MySql?

[Andrew Morgan]

On Sun, 24 Mar 2013, Charles Bradshaw wrote:

 In my /etc/imapd.conf I'm using:

 sasl_auxprop_plugin:sql
 sasl_sql_engine:mysql

 I want to store MD5 hashed passwords in my database. Is this possible?

 I was thinking about modifying the sql plugin to MD5 the password before
 comparison, but...

 I'm no C programmer so understanding sql.c (the plugin source) is quite
 beyond me. It looks as though we just check for the presence of the
 password and don't actual compare passwords! Surely I'm wrong here?

 I could use a symmetric encryption, eg AES, and place the necessary
 decrypt in the sasl_sql_select statement, but that seems a bit pointless
 since the key is now visible in various logs.

This could be illuminating:

   http://serverfault.com/questions/81958/postfix-sasl-mysql-use-md5-encryption

They suggest using the pam_mysql module so that you can specify the 
password storage format.

It appears the SQL auxprop plugin only works with passwords stored in 
plaintext.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: saslauthd cache / cyrus-imap and several passwords per login

[Andrew Morgan]

On Mon, 28 Jan 2013, Patrick Boutilier wrote:

 On 01/27/2013 09:03 PM, Andrew Morgan wrote:
 On Sat, 5 Jan 2013, Patrick Lamaiziere wrote:
 
 Helo,
 
 We use cyrus-imapd on Centos 6 at work and I've got the following issue
 on authentication:
 
 Users can login via a mailer (imap/pop) or use a webmail (horde). The
 webmail uses a SSO-CAS and horde uses a CAS token to log in
 cyrus-imap). As the CAS tokens are one-time tokens they must been
 cached by saslauthd.
 
 For this we use PAM with saslauthd and 3 PAM modules. pam_cas checks if
 the password is a valid CAS token, then we try ldap and then a local
 account.
 
 cyrus-imap - saslauthd (cache) - PAM (pam_cas, pam_ldap, pam_unix)
 
 That works fine.
 
 The problem is: when a user uses the webmail and uses also a mailer
 (using imap), saslauthd will remove the CAS token previously cached when
 the mailer connects. So the webmail is disconnected.
 
 There is a patch to allow saslauthd to cache several passwords for one
 login but I would like to avoid this.
 
 As far I can see, the cache depends on the service used (ie if I
 connect via pop, the imap password is not cleared from the
 saslauthd cache).
 
 So I'm asking if there is a way to introduce another service on
 cyrus-imap that will be used by the webmail (on another port than 143).
 I mean a service in the saslauthd / PAM way (the parameter '-s' in
 testsaslauthd: imap, pop, sieve).
 
 I don't know where to start. Is there a way to achieve this?
 Thanks, best regards.
 
 Sorry I have taken so long to respond.  I saw this message a while ago but
 I didn't have time to reply then.  It doesn't look like anyone else has
 responded according to the list archives.
 
 You can easily run multiple Cyrus imapd processes with different service
 names.  In your cyrus.conf, make a copy of your imap service and name it
 something like imap_webmail, listening on a different port.  Then make a
 /etc/pam.d/imap_webmail file with your desired PAM config.


 I just gave the above a try since I currently modify the source to force 
 which pam service the imapd binary calls but this entry still calls 
 /etc/pam.d/imap instead of /etc/pam.d/imaptest


 imaptestcmd=imapd listen=imaptest


 imaptest is in /etc/services on port 146

Well shoot, it looks like the SASL service name is hard-coded in imapd.c:

 /* create the SASL connection */
 if (sasl_server_new(imap, config_servername,
 NULL, NULL, NULL, NULL, 0,
 imapd_saslconn) != SASL_OK) {
 fatal(SASL failed initializing: sasl_server_new(), EC_TEMPFAIL);
 }


It would be nice if there was a way to override this somehow...  Perhaps 
file a bug on the bugzilla!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: saslauthd cache / cyrus-imap and several passwords per login

[Andrew Morgan]

On Sat, 5 Jan 2013, Patrick Lamaiziere wrote:

 Helo,

 We use cyrus-imapd on Centos 6 at work and I've got the following issue
 on authentication:

 Users can login via a mailer (imap/pop) or use a webmail (horde). The
 webmail uses a SSO-CAS and horde uses a CAS token to log in
 cyrus-imap). As the CAS tokens are one-time tokens they must been
 cached by saslauthd.

 For this we use PAM with saslauthd and 3 PAM modules. pam_cas checks if
 the password is a valid CAS token, then we try ldap and then a local
 account.

 cyrus-imap - saslauthd (cache) - PAM (pam_cas, pam_ldap, pam_unix)

 That works fine.

 The problem is: when a user uses the webmail and uses also a mailer
 (using imap), saslauthd will remove the CAS token previously cached when
 the mailer connects. So the webmail is disconnected.

 There is a patch to allow saslauthd to cache several passwords for one
 login but I would like to avoid this.

 As far I can see, the cache depends on the service used (ie if I
 connect via pop, the imap password is not cleared from the
 saslauthd cache).

 So I'm asking if there is a way to introduce another service on
 cyrus-imap that will be used by the webmail (on another port than 143).
 I mean a service in the saslauthd / PAM way (the parameter '-s' in
 testsaslauthd: imap, pop, sieve).

 I don't know where to start. Is there a way to achieve this?
 Thanks, best regards.

Sorry I have taken so long to respond.  I saw this message a while ago but 
I didn't have time to reply then.  It doesn't look like anyone else has 
responded according to the list archives.

You can easily run multiple Cyrus imapd processes with different service 
names.  In your cyrus.conf, make a copy of your imap service and name it 
something like imap_webmail, listening on a different port.  Then make a 
/etc/pam.d/imap_webmail file with your desired PAM config.

Another idea, which *might* work, is to run an imap proxy for your Horde 
instance.  We do that here.  That way, from Cyrus' perspective, Horde only 
logs in once so it shouldn't matter if the CAS token is single-use because 
there is only one authentication attempt.  I haven't tried this, so I'm 
not sure if you would see odd behavior if the proxied connection times out 
or something.  Just a thought!

Good luck.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Mailbox does not exist question

[Andrew Morgan]

Yes, the mailbox should be named user.test@mydomain, assuming you actually 
want to use virtual domains.  Do you have virtdomains set in imapd.conf?

Andy

On Fri, 25 Jan 2013, Charles Bradshaw wrote:

 Andrew

 Just a thought, should the mailbox name be 'user.test@mydomain' instead of
 'user.test'?

 Here is a dump of /var/lib/imap/mailboxes.db
 # hexdump -c /var/lib/imap/mailboxes.db
 000 241 002 213  \r   s   k   i   p   l   i   s   t   f   i   l
 010   e  \0  \0  \0  \0  \0  \0 001  \0  \0  \0 002  \0  \0  \0 024
 020  \0  \0  \0 001  \0  \0  \0 001  \0  \0  \0 320   Q 001   4 312
 030  \0  \0 001 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 220
 040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
 *
 080  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 377 377 377 377
 090  \0  \0  \0 001  \0  \0  \0  \t   u   s   e   r   .   t   e   s
 0a0   t  \0  \0  \0  \0  \0  \0 035   0   d   e   f   a   u   l
 0b0   t   t   e   s   t  \t   l   r   s   w   i   p   k   x   t
 0c0   e   c   d   a  \t  \0  \0  \0  \0  \0  \0  \0 377 377 377 377
 0d0

 I see a \tuser.test is the tab correct?

 Also curiously cryadm cannot delete user.test giving Permission denied:

 # cyradm -u cyrus localhost
 Password:
 localhost lm
 user.brad (\HasNoChildren)
 localhost dm user.brad
 deletemailbox: Permission denied
 localhost quit

 Something fishy here.

 Thanks for your help, Charles Bradshaw

 On: Thu, 24 Jan 2013 13:11:02 -0800 (PST), Andrew Morgan wrote:

 On Thu, 24 Jan 2013, Charles Bradshaw wrote:


 On: Thu, 24 Jan 2013 12:37:18 -0800 (PST), Andy wrote:

 On Thu, 24 Jan 2013, Charles Bradshaw wrote:

 Output from cyradm:

 $ cyradm --user cyrus localhost
 Password:
 localhost lm *
 user.test (\HasNoChildren)
 localhost

 Perhaps the user does not permission to see the mailbox?  What does
 lam user.test in cyradm report?

Andy

 # cyradm -u cyrus localhost
 Password:
 localhost lam user.test
 test lrswipkxtecda
 localhost

 Okay, can we confirm that you are connecting as the user test?
  Check your syslog for a message similar to:

 imap[30372]: login: cyrus-fe3.onid.oregonstate.edu [128.193.4.145]
 test PLAIN User logged in

 Perhaps we have a problem with virtualdomains.

  Andy
 --- End of Original Message ---

 Andy
 Here is a complete /var/log/maillog for a session.

 Jan 24 21:16:06 dell2600 imap[4844]: accepted connection
 Jan 24 21:16:06 dell2600 master[5029]: about to exec 
 /usr/lib/cyrus-imapd/imapd
 Jan 24 21:16:06 dell2600 imap[5029]: executed
 Jan 24 21:16:36 dell2600 imap[4844]: fetching user_deny.db entry for
 'test@mydomain'
 Jan 24 21:16:36 dell2600 imap[4844]: login: localhost [::1] test@mydomain
 plaintext User logged in
 SESSIONID=dell2600.bradcan.homelinux.com-4844-1359062166-1
 Jan 24 21:16:36 dell2600 imap[4844]: fetching user_deny.db entry for
 'test@mydomain'
 Jan 24 21:16:51 dell2600 imap[4844]: fetching user_deny.db entry for
 'test@mydomain'
 Jan 24 21:16:59 dell2600 imap[4844]: USAGE test@mydomain user: 0.009998 sys:
 0.009998
 Jan 24 21:18:51 dell2600 master[4485]: process 4844 exited, status 0
 Jan 24 21:19:06 dell2600 master[5036]: about to exec
 /usr/lib/cyrus-imapd/ctl_cyrusdb
 Jan 24 21:19:06 dell2600 ctl_cyrusdb[5036]: checkpointing cyrus databases
 Jan 24 21:19:06 dell2600 ctl_cyrusdb[5036]: archiving database file:
 /var/lib/imap/mailboxes.db
 Jan 24 21:19:06 dell2600 ctl_cyrusdb[5036]: archiving database file:
 /var/lib/imap/annotations.db
 Jan 24 21:19:06 dell2600 ctl_cyrusdb[5036]: done checkpointing cyrus databases
 Jan 24 21:19:06 dell2600 master[4485]: process 5036 exited, status 0

 Also the telemetry log from /var/lib/imap/log/test@mydomain (I figured that I
 need to name the directory user@realm)

 1359062196a1 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte
 QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN
 MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY
 THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN
 QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY X-NETSCAPE LOGINDISABLED
 COMPRESS=DEFLATE IDLE] User logged in
 SESSIONID=dell2600.bradcan.homelinux.com-4844-1359062166-1
 1359062211a2 LIST  *
 1359062211a2 OK Completed (0.000 secs)
 1359062219a3 LOGOUT
 1359062219* BYE LOGOUT received
 a3 OK Completed

 Charles Bradshaw


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Mailbox does not exist question

[Andrew Morgan]

On Thu, 24 Jan 2013, Charles Bradshaw wrote:

 Output from cyradm:

 $ cyradm --user cyrus localhost
 Password:
 localhost lm *
 user.test (\HasNoChildren)
 localhost

Perhaps the user does not permission to see the mailbox?  What does lam 
user.test in cyradm report?

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Mailbox does not exist question (is this the answer?)

[Andrew Morgan]

On Thu, 24 Jan 2013, Charles Bradshaw wrote:

 I have enabled debug. (in imapd.conf debug: yes). Now when I start a telnet
 imap session /var/log/maillog has this:

 Jan 24 13:25:59 dell2600 imap[4507]: accepted connection
 Jan 24 13:25:59 dell2600 master[4549]: about to exec 
 /usr/lib/cyrus-imapd/imapd
 Jan 24 13:25:59 dell2600 imap[4549]: executed
 Jan 24 13:25:59 dell2600 imap[4549]: IOERROR: opening
 /var/lib/imap/user_deny.db: No such file or directory

 Is this the problem?

 How do I create user_deny.db ?

No, user_deny is an optional feature.  There is no error if it is not 
found.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Load spikes when new email arrives

[Andrew Morgan]

On Thu, 24 Jan 2013, francis picabia wrote:

 In another email discussion on the Redhat mailing list, I've confirmed we
 have
 an issue with partition alignment.  This is getting to be quite the mess
 out there.  I saw one posting where it is speculated there are thousands of
 poorly set up disk partitions for their RAID stripe size.  fdisk and
 OS installers were late getting updated for the new TB disks
 and SSD disks as well.  Partition alignment might account
 for 5 to 30% of a performance hit.

Yeah, I read about partition alignment the last time I built a new Cyrus 
server.  I don't remember how it came to my attention, but it was wrong on 
all of my servers too.  The latest stable release of Debian Linux seems to 
do the right thing during installation, but previous versions did not.

I followed the recommendations that I found and set the starting sector to 
2048 for my partition (2048 * 512bytes = 1MB):

root@cyrus-be1:~# fdisk -lu /dev/sda

Disk /dev/sda: 536.9 GB, 536870912000 bytes
214 heads, 31 sectors/track, 158060 cylinders, total 1048576000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x88aa51ee

Device Boot  Start End  Blocks   Id  System
/dev/sda12048  1048575999   524286976   83  Linux

I don't know how much of a performance difference it would actually make, 
but I'm trying to squeeze all I can out of it!

 I've checked and my cyrus lmtpd process count
 never exceeds 11 under work load.
 await jumps up to 150-195 at worst.

 If I'm already at IO saturation, I can't see how a higher lmtpd limit
 would help.

I was going to suggest setting a LOWER lmtpd limit.  :)

It sounds like you have already done that (reading the rest of this email 
thread).

 My goal is to keep the system load reasonable so it is responsive for
 mailbox access by the end users.  Right now we get nagios alerts
 about 6 times a day for excessive load.  If I can move the mail
 queue workload into a hill instead of a sharp peak on the cacti
 load graph, it would be good.  There are minutes around the peaks
 where the queue is emptied and we have only 5 messages
 inbound per minute.

Hmmm, what options are there that don't involve rebuilding the disk...

Definitely check that you have Write-Back caching enabled on the PERC.

I don't know if remounting the filesystem as ext4 would help, but that's 
worth a shot.

Are you mounting the filesystem with the noatime option?  There is no 
need to track atime on a Cyrus mailstore and those extra writes can add 
up.  Here are my mount options:

LABEL=be1data1  /var/spool/cyrus/mail/data1 ext4
rw,auto,data=ordered,noatime   0   2

Perhaps there are some tweaks on the Postfix side that will put less 
strain on Cyrus.  I don't know much about Postfix though.

 In hind sight, I agree RAID 10 should have been implemented.
 At the time, four years ago, getting lots of space was the
 priority as space needs always grow.  We've never seen load
 issues until this month, and it seems to coincide with a
 general increase of all email volume and traffic.  Our primary
 MX is also getting hit more than normal.

Well, if none of the easy stuff helps enough, then maybe you'll get to 
build a new Cyrus filesystem from scratch!  :)

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Mailbox does not exist question

[Andrew Morgan]

On Thu, 24 Jan 2013, Charles Bradshaw wrote:


 On: Thu, 24 Jan 2013 12:37:18 -0800 (PST), Andy wrote:

 On Thu, 24 Jan 2013, Charles Bradshaw wrote:

 Output from cyradm:

 $ cyradm --user cyrus localhost
 Password:
 localhost lm *
 user.test (\HasNoChildren)
 localhost

 Perhaps the user does not permission to see the mailbox?  What does
 lam user.test in cyradm report?

  Andy

 # cyradm -u cyrus localhost
 Password:
 localhost lam user.test
 test lrswipkxtecda
 localhost

Okay, can we confirm that you are connecting as the user test?  Check 
your syslog for a message similar to:

imap[30372]: login: cyrus-fe3.onid.oregonstate.edu [128.193.4.145] test PLAIN 
User logged in

Perhaps we have a problem with virtualdomains.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: How can this happen?

[Andrew Morgan]

On Thu, 24 Jan 2013, Frank Elsner wrote:


 Hello,

 we have the strange situation with our murder environment that

 mailbackend has

 user.x.Sent2 default xlrswipkxtecda

 but the folder is non-existent in the filesystem.

 The mupdate server doesn't know this folder (not in mailboxes.db)

 How can this happen?

Some bug in Cyrus?

If you want to fix this, you can try creating the proper structure on the 
filesystem, run reconstruct to get Cyrus to sync back up with it, then 
delete the folder using cyradm or an IMAP client.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Load spikes when new email arrives

[Andrew Morgan]

On Wed, 23 Jan 2013, francis picabia wrote:

 Here are more stats.  Do these look average for performance?
 It is difficult to understand why the system was working with few
 load spikes before.

 A mailman mailing list sends 10kbyte message to 4000
 users having accounts on this cyrus system.  If I
 grep Delivered in the maillog by the minute I can
 see how fast the messages are stored.

 e.g.:
 # grep Delivered /var/log/maillog | grep 'Jan 23 10:37' | wc -l
696

 That is the best.  This peak event pushed the load to 14
 for 12 minutes, where it averages 604 messages
 delivered to cyrus mailboxes per minute.  Is that
 reasonable for  maximum delivery rate?

 I've also backed out the change (yesterday) to
 /sys/block/sda/queue/nr_requests
 I think it was pushing the load higher and there is no advantage
 in my hardware (SAS with Perc 5/i Raid 5 over 4 disk)
 to run with a low value for nr_requests.

You can certainly achieve higher delivery rates, but that all depends on 
your underlying hardware and how you have partitioned your system.

Why don't you start running iostat -x 5 on the system?  Leave this 
running to give you an idea of the baseline behavior and then look at it 
during periods of high load.  I suspect you will see that your svctm and 
%util will go up dramatically when a large number of messages are being 
delivered.  But, let's not make decisions based on assumptions!  :)

On my Cyrus Murder frontends (3 of them), I have limited LMTP connections 
to 25 in cyrus.conf:

   lmtp  cmd=/usr/local/cyrus/bin/lmtpproxyd listen=lmtp 
proto=tcp4 prefork=0 maxchild=25

This prevents our mail relays (Postfix) from opening too many simultaneous 
LMTP connections, which can cause too much I/O contention.  Take a look 
during your periods of high load to see how many lmtpd processes are 
running.  You may want to limit the number.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Load spikes when new email arrives

[Andrew Morgan]

On Wed, 23 Jan 2013, francis picabia wrote:

 Thanks for the response.  I have been checking my iostat whenever there is
 a number of messages in the active queue.

 Here is a sample snapshot from a script I run (ignoring the first
 iostat output of averages):

 Active in queue: 193
 12:47:01 up 5 days,  5:23,  6 users,  load average: 14.11, 9.22, 4.67

 Device: rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz 
 avgqu-sz   await  svctm  %util
 sda5  3.25   281.00 19.75 129.50   654.00  3384.0027.06 5.53  
  36.24   6.69  99.80

 svctm is about the same as when not under load and it went above 7 only
 once.
 Then there is this comment about the validity of tracking svctm:
 http://www.xaprb.com/blog/2010/09/06/beware-of-svctm-in-linuxs-iostat/

 %util is often reaching close to %100 when there is a queue to process.

 sda5 is where the cyrus mail/imap lives.  Our account names all begin with
 numbers, so almost all mail accounts are under the q folder.

Okay, I didn't realize svctm could be suspect, although I guess that makes 
sense in a RAID array.  What about your await times?  Does await increase 
during peak loads?

It seems pretty clear from iostat that you are IO bound on writes during 
mail delivery.  As Vincent said in his reply, RAID5 performs poorly during 
writes.  Each write actually consumes 4 disk operations (read old data, 
read old parity, write new data, write new parity).  If you can live with 
the slight additional risk, turn on write caching on the Perc 5/i if you 
haven't already.  I think they call it write-back versus 
write-through.

If you can handle it, you would probably be a lot happier converting that 
RAID5 set to RAID10.  You'll lose a disk worth of capacity, but get double 
the write performance.

However, what is your real goal?  Do you want to deliver mail more 
quickly, or do you want to reduce your load average?  You can probably 
reduce your load average and perhaps gain a bit of speed by tweaking the 
lmtp maxchild limit.  If you really need to deliver mail more quickly, 
then you need to throw more IOPS at it.

Let's keep this discussion going!  There are lots of ways to tune for 
performance.  I've probably missed some.  :)

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Mailbox does not exist question

[Andrew Morgan]

On Wed, 23 Jan 2013, Charles Bradshaw wrote:

 I'm seeing the following when I test cyrus-imapd using telnet.

 I seem to be missing some fundamental configuration.

 What am I doing wrong?

 Thanks in advance, Charles Bradshaw

 Telnet imap session:

 # telnet localhost imap
 Trying ::1...
 Connected to localhost.
 Escape character is '^]'.
 * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN
 AUTH=CRAM-MD5 AUTH=DIGEST-MD5 SASL-IR] imap-host.mydomain Cyrus IMAP
 v2.4.14-Fedora-RPM-2.4.14-1.fc17 server ready
 a1 LOGIN test@mydomain ***
 a1 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA
 MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN
 MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY
 THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN
 QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY X-NETSCAPE LOGINDISABLED
 COMPRESS=DEFLATE IDLE] User logged in
 SESSIONID=imap-host.mydomain-1720-1358978359-1
 a2 LIST  *
 a2 OK Completed (0.000 secs)

 I expected something like '* LIST (\HasNoChildren) . INBOX', but the
 response is blank!

 ~ o ~

 Telnet pop session:

 # telnet localhost 110
 Trying ::1...
 Connected to localhost.
 Escape character is '^]'.
 +OK imap-host.mydomain Cyrus POP3 v2.4.14-Fedora-RPM-2.4.14-1.fc17 server
 ready 1420303981.1358957093@imap-host.mydomain
 USER test@mydomain
 +OK Name is a valid mailbox
 PASS **
 -ERR [SYS/PERM] Unable to locate maildrop: Mailbox does not exist

 /val/log/maillog has:
 pop3s[13116]: Unable to locate maildrop mydomain!user.test: Mailbox does not 
 exist

 ~ o ~

 However /var/spool/imap/t/user/test mailbox exists and contains mail:

 # ls -l /var/spool/imap/t/user/test
 total 24
 -rw---. 1 cyrus mail  602 Jan 23 14:36 1.
 -rw---. 1 cyrus mail  606 Jan 23 14:59 2.
 -rw---. 1 cyrus mail  603 Jan 23 15:49 3.
 -rw---. 1 cyrus mail 1884 Jan 23 15:49 cyrus.cache
 -rw---. 1 cyrus mail  154 Jan 21 09:58 cyrus.header
 -rw---. 1 cyrus mail  416 Jan 23 15:49 cyrus.index

 # cat /etc/imapd.conf
 configdirectory: /var/lib/imap
 partition-default: /var/spool/imap
 admins: cyrus
 sievedir: /var/lib/imap/sieve
 sendmail: /usr/sbin/sendmail
 hashimapspool: true
 sasl_pwcheck_method: auxprop
 sasl_mech_list: PLAIN DIGEST-MD5 CRAM-MD5
 sasl_auxprop_plugin:sasldb
 allowplaintext: yes
 virtdomains: userid
 tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
 tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
 tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
 tls_cipher_list: TLSv1:SSLv3:!NULL:!EXPORT:!DES:!LOW:@STRENGTH
 +OK imap-host.mydomain Cyrus POP3 v2.4.14-Fedora-RPM-2.4.14-1.fc17 server
 ready 1420303981.1358957093@imap-host.mydomain
 USER test@mydomain
 +OK Name is a valid mailbox
 PASS **
 -ERR [SYS/PERM] Unable to locate maildrop: Mailbox does not exist

 /val/log/maillog has:
 pop3s[13116]: Unable to locate maildrop mydomain!user.test: Mailbox does not 
 exist

 ~ o ~

 However /var/spool/imap/t/user/test mailbox exists and contains mail:

 # ls -l /var/spool/imap/t/user/test
 total 24
 -rw---. 1 cyrus mail  602 Jan 23 14:36 1.
 -rw---. 1 cyrus mail  606 Jan 23 14:59 2.
 -rw---. 1 cyrus mail  603 Jan 23 15:49 3.
 -rw---. 1 cyrus mail 1884 Jan 23 15:49 cyrus.cache
 -rw---. 1 cyrus mail  154 Jan 21 09:58 cyrus.header
 -rw---. 1 cyrus mail  416 Jan 23 15:49 cyrus.index

 # cat /etc/imapd.conf
 configdirectory: /var/lib/imap
 partition-default: /var/spool/imap
 admins: cyrus
 sievedir: /var/lib/imap/sieve
 sendmail: /usr/sbin/sendmail
 hashimapspool: true
 sasl_pwcheck_method: auxprop
 sasl_mech_list: PLAIN DIGEST-MD5 CRAM-MD5
 sasl_auxprop_plugin:sasldb
 allowplaintext: yes
 virtdomains: userid
 tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
 tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
 tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
 tls_cipher_list: TLSv1:SSLv3:!NULL:!EXPORT:!DES:!LOW:@STRENGTH

Run cyradm --user cyrus localhost and type lm *.  Is the mailbox 
user.test in the output?

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus-imap configuration question

[Andrew Morgan]

On Sat, 19 Jan 2013, Charles Bradshaw wrote:

 I'm tying to configure cyrus-imap on a Fedora 17 system.

 cyrus-imapd version cyrus-imapd.i686 2.4.14-1.fc17

 I have sendmail and saslauthd working using DIGEST-MD5 and CRAM-MD5 working.

 I have gone through the cyrus-imap configuration procedure, but when I try to
 start the server:

 # systemctl start cyrus-imapd.service
 Job failed. See system journal and 'systemctl status' for details.

 # systemctl status cyrus-imapd.service
 cyrus-imapd.service - Cyrus-imapd IMAP/POP3 email server
 Loaded: loaded (/usr/lib/systemd/system/cyrus-imapd.service; disabled)
 Active: failed (Result: exit-code) since Sat, 19 Jan 2013 13:29:32 
 +;
 28s ago
Process: 2049 ExecStartPre=/usr/lib/cyrus-imapd/cyr_systemd_helper 
 start
 (code=exited, status=75)
 CGroup: name=systemd:/system/cyrus-imapd.service

 If I start the master process manually or in debug mode:
 # /usr/lib/cyrus-imapd/cyrus-master -D  (or -d)
 fatal error: can't read mailboxes file
 ctl_cyrusdb: unable to archive environment

 At this point ps -A reports:
 cyrus-master
 imapd defunct
 imapd
 pop3 defunct
 pop3d
 lmtpd

 Top reports imapd  pop3d are sporning and are being zombied at a rate of
 about 1 second!  Surely this is not right?

 Looks like the deamons are crashing imediately!

What do you see in your syslog for Cyrus?  I assume you will see an error 
message about unable to open mailboxes file.  We need to figure out where 
the mailboxes file is located and whether the cyrus user owns it and has 
the correct permissions on it.

To me, this sounds like a problem with an incorrectly created Cyrus 
configuration directory and/or mail spool directory.

I'm not familiar with the Fedora Cyrus package, but maybe there is 
something the package is supposed to do when it is installed?  If someone 
else knows the Fedora package, hopefully they will speak up.

Either way, we can fix this!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: problem with one user after a crash

[Andrew Morgan]

On Thu, 10 Jan 2013, David Lang wrote:

 I has my home mail server crash, and after the crash, one user (me) is unable 
 to
 acess any folders.

 When I manually telnet to the IMAP port, I can login, I can list and run other
 commands, but as soon as I do a select of any folder (mine or any other shared
 folder) I get disconnected.

 Other users have no problems accessing the same folder.

 This is with Cyrus 2.2 on Ubuntu (I need to upgrade, but have not had the time
 to do so yet)

 Any suggestions on what may be wrong and how to diagnose this?

Check your syslog files, whichever one Cyrus is logging to.  I suspect 
you'll see something related to your seen file.

A corrupt seen file could be causing your problem.  Seen files are stored 
in {$configdir}/user/prefix/username.seen.  My seen file is:

   /var/spool/cyrus/config/user/m/morgan.seen

If it is corrupt, you may be able to repair it.  Seen files have been 
stored by default in Skiplist format for quite a while.  You can google 
get skiplist.py, a script to fix corrupted Skiplist files, from:

   http://oss.netfarm.it/python-cyrus.php

Hope this helps!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: problem with one user after a crash

[Andrew Morgan]

On Thu, 10 Jan 2013, David Lang wrote:

 On Thu, 10 Jan 2013, Andrew Morgan wrote:

 On Thu, 10 Jan 2013, David Lang wrote:
 
 I has my home mail server crash, and after the crash, one user (me) is 
 unable to
 acess any folders.
 
 When I manually telnet to the IMAP port, I can login, I can list and run 
 other
 commands, but as soon as I do a select of any folder (mine or any other 
 shared
 folder) I get disconnected.
 
 Other users have no problems accessing the same folder.
 
 This is with Cyrus 2.2 on Ubuntu (I need to upgrade, but have not had the 
 time
 to do so yet)
 
 Any suggestions on what may be wrong and how to diagnose this?
 
 Check your syslog files, whichever one Cyrus is logging to.  I suspect 
 you'll see something related to your seen file.
 
 A corrupt seen file could be causing your problem.  Seen files are stored 
 in {$configdir}/user/prefix/username.seen.  My seen file is:

  /var/spool/cyrus/config/user/m/morgan.seen
 
 If it is corrupt, you may be able to repair it.  Seen files have been 
 stored by default in Skiplist format for quite a while.  You can google get 
 skiplist.py, a script to fix corrupted Skiplist files, from:

  http://oss.netfarm.it/python-cyrus.php
 
 Hope this helps!

 nothing useful shows up in the logs

 Jan 10 13:19:12 asgard cyrus/imap[22884]: login: localhost [127.0.0.1] 
 dl...@lang.hm plaintext Userlogged in
 Jan 10 13:19:47 asgard master[1220]: process 22884 exited, signaled to death 
 by 7
 Jan 10 13:19:47 asgard master[1220]: service imap pid 22884 in BUSY state: 
 terminated abnormally

A corrupted seen file is the only thing that makes sense to me.  If other 
users can open the same folder, then the cyrus.header and cyrus.index 
files must be sane.

As an experiment, you could move your seen file from lang.seen (or 
whatever it's called) to lang.seen.bak.  Then connect to IMAP as yourself 
and try to open the folder.  If it works, then it must have been a 
corrupted seen file, and you can use skiplist.py to recover as much of it 
as possible.

If not...  we can use other tools (strace) to track down the likely 
culprit.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: problem with one user after a crash

[Andrew Morgan]

On Thu, 10 Jan 2013, David Lang wrote:


On Thu, 10 Jan 2013, Andrew Morgan wrote:

A corrupted seen file is the only thing that makes sense to me.  If other 
users can open the same folder, then the cyrus.header and cyrus.index files 
must be sane.


As an experiment, you could move your seen file from lang.seen (or whatever 
it's called) to lang.seen.bak.  Then connect to IMAP as yourself and try to 
open the folder.  If it works, then it must have been a corrupted seen 
file, and you can use skiplist.py to recover as much of it as possible.


Ok, the good news is that this seems to be the problem.

unfortunantly the skiplist recovery tool is not working.

# ./skiplist.py dlang.seen.bak dlang.seen.txt
Traceback (most recent call last):
 File ./skiplist.py, line 172, in module
   values, keys = getkeys(fp)
 File ./skiplist.py, line 152, in getkeys
   spointer = unpack('I', str_p)[0]
struct.error: unpack requires a string argument of length 4

# file dlang.seen.bak
dlang.seen.bak: Cyrus skiplist DB

I tried enabling debug mode in skiplist.py and I'm not seeing anything 
different. This confuses me. I'm not that familiar with python, but as I read 
the code, get_header() should be writing a bunch of stuff before it gets to 
the getkeys() section that failing.


Hmmm, I haven't looked at the code in skiplist.py much.  I have an older 
version of skiplist.py, which I have attached to this email.  Honestly, I 
haven't used this since I upgraded to Cyrus v2.3.something.  I think there 
were some bugs in skiplist on the older versions.  :)


Give the attached skiplist.py a shot!  Worst case, you'll have to start 
over with no Seen history.  :(


Andy#!/usr/bin/env python
# -*- Mode: Python; tab-width: 4 -*-
#
# Cyrus Imapd Skiplist db recovery tool
#
# Copyright (C) 2004 Gianluigi Tiesi sher...@netfarm.it
# Copyright (C) 2004 NetFarm S.r.l.  [http://www.netfarm.it]
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2, or (at your option) any later
# version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTIBILITY
# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.
# ==

__version__= '0.1'
__doc__=Cyrus skiplist db recover

from sys import argv,exit,stdout,stderr
from struct import unpack
from time import localtime, strftime

### User Conf
debug = 0
###

TIMEFMT ='%a, %d %b %Y %H:%M:%S %z'
PADDING = '\xff' * 4
INORDER = 1
ADD = 2
DELETE  = 4
COMMIT  = 255
DUMMY   = 257
HEADER  = -1
MAIN= -2

types = {
1:   'INORDER',
2:   'ADD',
4:   'DELETE',
255: 'COMMIT',
257: 'DUMMY',
-1:  'HEADER',
-2:  '*'
}

def log(rtype, text):
global debug
if debug:
out = '[%s] %s\n' % (types[rtype], text)
stdout.write(out)
stdout.flush()

def roundto4(value):
if value % 4:
return ((value / 4) + 1) * 4
return value

def get_header(fp):
 Magic ??
fp.seek(4)

sign = fp.read(16)
log(HEADER, sign[:-3])

version = unpack('I', fp.read(4))[0]
version_minor = unpack('I', fp.read(4))[0]

log(HEADER, 'Version %d,%d' % (version, version_minor))

maxlevel = unpack('I', fp.read(4))[0]
curlevel = unpack('I', fp.read(4))[0]

log(HEADER, 'Level %d/%d' % (curlevel, maxlevel))

listsize = unpack('I', fp.read(4))[0]
log(HEADER, 'List size %d' % listsize)

logstart = unpack('I', fp.read(4))[0]
log(HEADER, 'Offset %d' % logstart)

lastrecovery = localtime(unpack('I', fp.read(4))[0])
lastrecovery = strftime(TIMEFMT, lastrecovery)

log(HEADER, 'Last Recovery %s' % lastrecovery)

return { 'version': [version, version_minor],
 'level'  : [curlevel, maxlevel],
 'listsize'   : listsize,
 'logstart'   : logstart,
 'lastrecover': lastrecovery
 }

def getkeys(fp):
values = []
keys = {}
keystring = ''
datastring = ''

while 1:
log(MAIN, '-'*78)

stype = fp.read(4)

### EOF
if len(stype) != 4:
break

rtype = unpack('I', stype)[0]
if not types.has_key(rtype):
log(MAIN, 'Invalid type %d' % rtype)
continue

log(rtype, 'Record type %s' % types[rtype])

if rtype == DELETE:
ptr = unpack('I', fp.read(4))[0]
log(rtype, 'DELETE %d (0x%x)' % (ptr, ptr))
continue

if rtype == COMMIT:
continue

ksize = unpack('I', fp.read(4))[0]
log(rtype, 'Key size %d (%d)' % (ksize, roundto4(ksize)))

if ksize:
keystring = fp.read(roundto4(ksize))[:ksize]
log(rtype, 'Key

Re: successful create but unsuccessful subscribe

[Andrew Morgan]

On Wed, 19 Dec 2012, Kerstin Espey wrote:

 On 14.12.2012 20:35, Dan White wrote:

 See if setting

 allowallsubscribe: 1

 on your frontend makes any difference.

 Unfortunately it does not.

 I have reviewed the whole configuration, shortened the config on the
 mupdate master, but nothing helped.

 Now I have reduced the number of preforked mupdate process on the master
 from 5 to 1 - this does the job.
 Increasing the number of preforked processes again leads to the
 well-known misbehaviour. Decreasing again, everything is fine.

 Is this a known behaviour?

This sounds like a bug, either in documentation or behavior.  I could not 
find an existing bug report for it.  Would you be willing to create a bug 
report at https://bugzilla.cyrusimap.org/?

 Our setting on the master is now:

 mupdate   cmd=/usr/lib/cyrus/bin/mupdate -m
 listen=ipaddress:3905 prefork=1 maxchild=20

 How long does it scale?

 Thanks to everybody and I wish you a Merry Christmas!

I don't know why, but we have always operated with prefork=1 here.  As far 
as I can tell, it never runs more than 1 mupdate process.  That single 
mupdate process has 15 connections in total from our 3 frontend servers. 
There doesn't seem to be a need for it to spawn additional mupdate 
processes.  Now that I look closer, I see that mupdate is threaded...

We have 3 frontends and 3 backends.  Each backend has about 20,000 users 
on it.

Here is my cyrus.conf entry on the mupdate master:

   mupdate   cmd=/usr/local/cyrus/bin/mupdate -m listen=3905 
proto=tcp4 prefork=1

and on the frontends:

   mupdate   cmd=/usr/local/cyrus/bin/mupdate listen=3905 proto=tcp4 
prefork=1

Hmm, there is no manpage for mupdate either!  Digging around in the source 
code shows that there are configuration options for mupdate in imapd.conf, 
such as:

mupdate_workers_max: 50
 The maximum number of mupdate worker threads (overall)

mupdate_workers_maxspare: 10
 The maximum number of idle mupdate worker threads

mupdate_workers_minspare: 2
 The minimum number of idle mupdate worker threads

mupdate_workers_start: 5
 The number of mupdate worker threads to start

So the usual process controls in cyrus.conf don't really apply anyways!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Reconstruct mailbox for a specific user.

[Andrew Morgan]

On Thu, 13 Dec 2012, an...@isac.gov.in wrote:

 - Message from mor...@orst.edu -
 Date: Wed, 12 Dec 2012 09:33:03 -0800 (PST)
 From: Andrew Morgan mor...@orst.edu
  Subject: Re: Reconstruct mailbox for a specific user.
   To: an...@isac.gov.in
   Cc: info-cyrus@lists.andrew.cmu.edu


 On Wed, 12 Dec 2012, an...@isac.gov.in wrote:

 - Message from mor...@orst.edu -
Date: Tue, 11 Dec 2012 20:55:04 -0800 (PST)
From: Andrew Morgan mor...@orst.edu
 Subject: Re: Reconstruct mailbox for a specific user.
  To: an...@isac.gov.in
  Cc: info-cyrus@lists.andrew.cmu.edu


 On Wed, 12 Dec 2012, an...@isac.gov.in wrote:

 One of the users mailbox has one more level of sub folder like

 user.xxx.ABC
 user.xxx.ABC.def

 Right now, folders of level user.xxx are seen, but folders at
 user.xxx.ABC including ABC are not seen.

 Should I run,
 /usr/lib/cyrus-imapd/reconstruct -rf user.xxx.ABC now? That is when
 Cyrus-imapd is already running?  Or I should stop the service and run
 reconstruct?  Please advise.

 You can run reconstruct and quota while cyrus-imapd is running.

 If reconstruct does not succeed, verify the mailbox(es) are
 listed within
 the output of 'ctl_mboxlist -d'. If not, you should add them
 via cyradm.
 reconstruct may also fail for a given mailbox if you are missing the
 cyrus.* files within its directory.

 You might wish to backup the contents of the directories in
 question before
 proceeding, in case you end up with missing flags or other data.

 --
 Dan White

 What I found is

 1. /var/spool/imap/user/xxx exists
 2. /var/spool/imap/user has several directories
 3. All other directories except ABC are listed as folders under
 user.xxx and are seen by IMAP clients.
 4. ABC directory has subfolders like
 /var/spool/imap/user/xxx/ABC/1, 2 3 etc and each of these
 subdirectories has cyrus.* files except ABC directory.  As you
 said, as ABC directory does not have cyrus.* files, reconstruct
 has failed to recognise it and hence its subdirectories.

 Should I run reconstruct -r -f user.xxx.ABC or cm user.xxx.ABC
 and then run reconstruct -r -f user.xxx.ABC?

 Create a cyrus.header file in the ABC directory, set the ownership
 and permissions.  Something like this:

 touch cyrus.header
 chown cyrus:mail cyrus.header
 chmod 600 cyrus.header

 Then run:

 reconstruct -x -f user.xxx.ABC


 I should have asked at the beginning - are there any message files
 in (1., 2., 3., etc) in the ABC directory?

  Andy


 NO. There are no message files in ABC directory.  There are only
 directories in ABC directory and each such directory has message
 files and also cyrus.* files.

 I am thinking of another option, move all such directories under
 ABC to one level higher, that is at user.xxx level and run
 reconstruct -r -f user.xxx.

 But, you still suggest, which is the best way.

 What happens when you run:

 reconstruct -x -f user.xxx.ABC.def

 ?

Andy


 NO output.  lm does not list user.xxx.ABC or user.XXX.ABC.def.

 Hmmm.  Could this be a bug in reconstruct?  Maybe it won't
 reconstruct a mailbox is the parent is not also a mailbox...

 Why don't you try creating user.xxx.ABC in cyradm, then running the
 same reconstruct command?

  Andy


 YES. This helped and all folders got recognised.  Thank you for your guidance.

 Regards,
 Anant.

I found an existing bug report that covers this:

   https://bugzilla.cyrusimap.org/show_bug.cgi?id=2125

I updated the bug report with an example of how to reproduce the problem, 
which is still present in v2.4.17.  This bug was originally created 
2003-07-30.  :)

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: successful create but unsuccessful subscribe

[Andrew Morgan]

On Wed, 19 Dec 2012, Frank Elsner wrote:

 On Wed, 19 Dec 2012 10:10:43 -0800 (PST) Andrew Morgan wrote:

  [ ... ]

 I don't know why, but we have always operated with prefork=1 here.  As far
 as I can tell, it never runs more than 1 mupdate process.  That single
 mupdate process has 15 connections in total from our 3 frontend servers.
 There doesn't seem to be a need for it to spawn additional mupdate
 processes.  Now that I look closer, I see that mupdate is threaded...

 We have 3 frontends and 3 backends.  Each backend has about 20,000 users
 on it.

 Here is my cyrus.conf entry on the mupdate master:

mupdate   cmd=/usr/local/cyrus/bin/mupdate -m listen=3905 
 proto=tcp4 prefork=1

 and on the frontends:

mupdate   cmd=/usr/local/cyrus/bin/mupdate listen=3905 
 proto=tcp4 prefork=1

 Hmm, there is no manpage for mupdate either!  Digging around in the source
 code shows that there are configuration options for mupdate in imapd.conf,
 such as:

 mupdate_workers_max: 50
  The maximum number of mupdate worker threads (overall)

 mupdate_workers_maxspare: 10
  The maximum number of idle mupdate worker threads

 mupdate_workers_minspare: 2
  The minimum number of idle mupdate worker threads

 mupdate_workers_start: 5
  The number of mupdate worker threads to start

 Ok, sounds good. On the mupdate master we already have the mupdate_* settings.
 Shall we put mupdate_* settings into the imapd.conf on the frontends too?

I have never set those on either mupdate master or frontends, although I 
have the defaults in imapd.conf commented out.  At least here with our 
systems, the defaults seem to be working well.

How would I know if I am reaching mupdate_workers_max?  Is that logged?

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Fwd: Too many entries of mystore: reusing txn....

[Andrew Morgan]

On Wed, 12 Dec 2012, Adam Tauno Williams wrote:

 On Sun, 2012-12-09 at 10:49 +0530, Anant Athavale wrote:

 As you say, the imap DEBUG logs are coming to maillog.  RHEL 6.3 ships
 with Rsyslogd and also it looks like cyrus-imapd is compiled to use
 MAIL_LOG facility.  (I tried local6.info /var/log/imapd.log. but it
 did log anything in imapd.log ).
 I am attaching rsyslog.conf (Not modified).  What I ultimately want is
 'maillog should not contain imap logs.  And imapd.log should contain
 all logs related to cyrus/imapd with only info level logs.  '
 As I could not achieve it in short span of time, I have released the
 system, but, would like to do that in near future.  Any pointers to
 achieve?

 Yes.  Give up on syslog.  Seriously.  The model provided by syslog is
 very simplistic and kludgy.  Just use syslog as a transport to get
 messages into an NMS, and sort, categorize, and record them there.

 We send all our syslog messages to ZenOSS.  There syslog messages can be
 mapped into categories, prioritized [and discarded], recorded, viewed,
 and generate notifications.  And you get a user interface to do it all
 in, and a coherent way to backup/restore all your machinations.

 Syslog messages from imapd have a tag of imapd, and messages from
 postfix have a tag of postfix, which is almost invisible in syslog
 itself.  So you have the host of origin, the tag, the facility, and the
 level [and the text of the message] all to work with to categorize [and
 potentially discard] any way you want.

 Obviously you want to discard DEBUG messages as the syslog level - that
 is just too much noise for anything.  But a decent host for you NMS can
 handle a surprising load of messages.

Just to add another thought here...  You could use syslog-ng instead of 
rsyslog.  Syslog-ng has more advanced filtering capabilities than rsyslog, 
and you can probably just drop-in replace rsyslog with syslog-ng. 
However, I would not discourage you from looking at ZenOSS too.  Syslog-ng 
might be less work to implement if you do not need ZenOSS features.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Reconstruct mailbox for a specific user.

[Andrew Morgan]

On Wed, 12 Dec 2012, an...@isac.gov.in wrote:

 - Message from mor...@orst.edu -
 Date: Tue, 11 Dec 2012 20:55:04 -0800 (PST)
 From: Andrew Morgan mor...@orst.edu
  Subject: Re: Reconstruct mailbox for a specific user.
   To: an...@isac.gov.in
   Cc: info-cyrus@lists.andrew.cmu.edu


 On Wed, 12 Dec 2012, an...@isac.gov.in wrote:

 One of the users mailbox has one more level of sub folder like

 user.xxx.ABC
 user.xxx.ABC.def

 Right now, folders of level user.xxx are seen, but folders at
 user.xxx.ABC including ABC are not seen.

 Should I run,
 /usr/lib/cyrus-imapd/reconstruct -rf user.xxx.ABC now? That is when
 Cyrus-imapd is already running?  Or I should stop the service and run
 reconstruct?  Please advise.

 You can run reconstruct and quota while cyrus-imapd is running.

 If reconstruct does not succeed, verify the mailbox(es) are listed within
 the output of 'ctl_mboxlist -d'. If not, you should add them via cyradm.
 reconstruct may also fail for a given mailbox if you are missing the
 cyrus.* files within its directory.

 You might wish to backup the contents of the directories in
 question before
 proceeding, in case you end up with missing flags or other data.

 --
 Dan White

 What I found is

 1. /var/spool/imap/user/xxx exists
 2. /var/spool/imap/user has several directories
 3. All other directories except ABC are listed as folders under
 user.xxx and are seen by IMAP clients.
 4. ABC directory has subfolders like
 /var/spool/imap/user/xxx/ABC/1, 2 3 etc and each of these
 subdirectories has cyrus.* files except ABC directory.  As you
 said, as ABC directory does not have cyrus.* files, reconstruct
 has failed to recognise it and hence its subdirectories.

 Should I run reconstruct -r -f user.xxx.ABC or cm user.xxx.ABC
 and then run reconstruct -r -f user.xxx.ABC?

 Create a cyrus.header file in the ABC directory, set the ownership
 and permissions.  Something like this:

 touch cyrus.header
 chown cyrus:mail cyrus.header
 chmod 600 cyrus.header

 Then run:

 reconstruct -x -f user.xxx.ABC


 I should have asked at the beginning - are there any message files
 in (1., 2., 3., etc) in the ABC directory?

Andy


 NO. There are no message files in ABC directory.  There are only
 directories in ABC directory and each such directory has message
 files and also cyrus.* files.

 I am thinking of another option, move all such directories under
 ABC to one level higher, that is at user.xxx level and run
 reconstruct -r -f user.xxx.

 But, you still suggest, which is the best way.

 What happens when you run:

 reconstruct -x -f user.xxx.ABC.def

 ?

  Andy


 NO output.  lm does not list user.xxx.ABC or user.XXX.ABC.def.

Hmmm.  Could this be a bug in reconstruct?  Maybe it won't reconstruct a 
mailbox is the parent is not also a mailbox...

Why don't you try creating user.xxx.ABC in cyradm, then running the same 
reconstruct command?

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Reconstruct mailbox for a specific user.

[Andrew Morgan]

On Tue, 11 Dec 2012, an...@isac.gov.in wrote:


- Message from dwh...@olp.net -
Date: Mon, 10 Dec 2012 13:21:03 -0600
From: Dan White dwh...@olp.net
 Subject: Re: Reconstruct mailbox for a specific user.
  To: an...@isac.gov.in
  Cc: info-cyrus@lists.andrew.cmu.edu



On 12/10/12 16:42 +0530, an...@isac.gov.in wrote:

Dear Experts,

I did reconstruct mailboxes of all users, using the script in
README.HOWTO-recover-mailboxes.db.  Following is the extract of the
script (for reconstruct)

--
find /var/spool/imap/user -maxdepth 1 -mindepth 1 | \
while read i; do
   i=$(basename $i)
   /usr/lib/cyrus-imapd/reconstruct -rf user.${i}
   /usr/lib/cyrus-imapd/quota -f user.${i}
done
--

One of the users mailbox has one more level of sub folder like

user.xxx.ABC
user.xxx.ABC.def

Right now, folders of level user.xxx are seen, but folders at
user.xxx.ABC including ABC are not seen.

Should I run,
/usr/lib/cyrus-imapd/reconstruct -rf user.xxx.ABC now? That is when
Cyrus-imapd is already running?  Or I should stop the service and run
reconstruct?  Please advise.


You can run reconstruct and quota while cyrus-imapd is running.

If reconstruct does not succeed, verify the mailbox(es) are listed within
the output of 'ctl_mboxlist -d'. If not, you should add them via cyradm.
reconstruct may also fail for a given mailbox if you are missing the
cyrus.* files within its directory.

You might wish to backup the contents of the directories in question before
proceeding, in case you end up with missing flags or other data.

--
Dan White


What I found is

1. /var/spool/imap/user/xxx exists
2. /var/spool/imap/user has several directories
3. All other directories except ABC are listed as folders under 
user.xxx and are seen by IMAP clients.
4. ABC directory has subfolders like /var/spool/imap/user/xxx/ABC/1, 2 
3 etc and each of these subdirectories has cyrus.* files except ABC 
directory.  As you said, as ABC directory does not have cyrus.* files, 
reconstruct has failed to recognise it and hence its subdirectories.


Should I run reconstruct -r -f user.xxx.ABC or cm user.xxx.ABC and 
then run reconstruct -r -f user.xxx.ABC?


Create a cyrus.header file in the ABC directory, set the ownership and 
permissions.  Something like this:


touch cyrus.header
chown cyrus:mail cyrus.header
chmod 600 cyrus.header

Then run:

reconstruct -x -f user.xxx.ABC


I should have asked at the beginning - are there any message files in (1., 
2., 3., etc) in the ABC directory?


Andy
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Reconstruct mailbox for a specific user.

[Andrew Morgan]

On Wed, 12 Dec 2012, an...@isac.gov.in wrote:

 One of the users mailbox has one more level of sub folder like

 user.xxx.ABC
 user.xxx.ABC.def

 Right now, folders of level user.xxx are seen, but folders at
 user.xxx.ABC including ABC are not seen.

 Should I run,
 /usr/lib/cyrus-imapd/reconstruct -rf user.xxx.ABC now? That is when
 Cyrus-imapd is already running?  Or I should stop the service and run
 reconstruct?  Please advise.

 You can run reconstruct and quota while cyrus-imapd is running.

 If reconstruct does not succeed, verify the mailbox(es) are listed within
 the output of 'ctl_mboxlist -d'. If not, you should add them via cyradm.
 reconstruct may also fail for a given mailbox if you are missing the
 cyrus.* files within its directory.

 You might wish to backup the contents of the directories in question before
 proceeding, in case you end up with missing flags or other data.

 -- 
 Dan White

 What I found is

 1. /var/spool/imap/user/xxx exists
 2. /var/spool/imap/user has several directories
 3. All other directories except ABC are listed as folders under 
 user.xxx and are seen by IMAP clients.
 4. ABC directory has subfolders like 
 /var/spool/imap/user/xxx/ABC/1, 2 3 etc and each of these 
 subdirectories has cyrus.* files except ABC directory.  As you 
 said, as ABC directory does not have cyrus.* files, reconstruct has 
 failed to recognise it and hence its subdirectories.

 Should I run reconstruct -r -f user.xxx.ABC or cm user.xxx.ABC and 
 then run reconstruct -r -f user.xxx.ABC?

 Create a cyrus.header file in the ABC directory, set the ownership 
 and permissions.  Something like this:

 touch cyrus.header
 chown cyrus:mail cyrus.header
 chmod 600 cyrus.header

 Then run:

 reconstruct -x -f user.xxx.ABC


 I should have asked at the beginning - are there any message files 
 in (1., 2., 3., etc) in the ABC directory?

  Andy


 NO. There are no message files in ABC directory.  There are only 
 directories in ABC directory and each such directory has message files 
 and also cyrus.* files.

 I am thinking of another option, move all such directories under ABC 
 to one level higher, that is at user.xxx level and run reconstruct -r 
 -f user.xxx.

 But, you still suggest, which is the best way.

What happens when you run:

reconstruct -x -f user.xxx.ABC.def

?

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Too many entries of mystore: reusing txn....

[Andrew Morgan]

On Sat, 8 Dec 2012, Anant Athavale wrote:

 Dear Experts,

 I had been sending mails to this list in the last two days with email
 an...@isac.gov.in - with subject - Urgent Help Required.  Based on your
 advise, I have rebuilt all the mailboxes.  I am monitoring the maillog
 entries of cyrus imap after I started the cyrus-imapd.  Everything seems to
 be fine.

 Most of the log entries, I checked in this list and found them harmless and
 hence ignoring them like: setrlimit and IP_TOS etc.

 But, for this one, I did not get any proper comments in this list.

 In the maillog, I have too many lines with the same message, which is as
 below.

 cvt_cyrusdb: mystore: reusing txn .with some value of number.

 This same line repeats, at least 190+ times (including that last number).
 Is this a cause of worry? I have not yet released the system to users.  I
 will be doing it only tomorrow, based on the response for this.  Do I need
 to check something?

 I have rebuilt mailboxes and reconstructed mailboxes in RHEL 6.3 supplied
 Cyrus-IMAP 2.3.16.

 Please advise.

Unless you really want to see all the gory details of Cyrus, turn your 
syslog level down from DEBUG to INFO.  The message you are seeing is a 
DEBUG level log message.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Too many entries of mystore: reusing txn....

[Andrew Morgan]

On Sun, 9 Dec 2012, Anant Athavale wrote:

 On Sat, Dec 8, 2012 at 10:52 PM, Andrew Morgan mor...@orst.edu wrote:

 On Sat, 8 Dec 2012, Anant Athavale wrote:

  Dear Experts,

 I had been sending mails to this list in the last two days with email
 an...@isac.gov.in - with subject - Urgent Help Required.  Based on your
 advise, I have rebuilt all the mailboxes.  I am monitoring the maillog
 entries of cyrus imap after I started the cyrus-imapd.  Everything seems
 to
 be fine.

 Most of the log entries, I checked in this list and found them harmless
 and
 hence ignoring them like: setrlimit and IP_TOS etc.

 But, for this one, I did not get any proper comments in this list.

 In the maillog, I have too many lines with the same message, which is as
 below.

 cvt_cyrusdb: mystore: reusing txn .with some value of number.

 This same line repeats, at least 190+ times (including that last number).
 Is this a cause of worry? I have not yet released the system to users.  I
 will be doing it only tomorrow, based on the response for this.  Do I need
 to check something?

 I have rebuilt mailboxes and reconstructed mailboxes in RHEL 6.3 supplied
 Cyrus-IMAP 2.3.16.

 Please advise.


 Unless you really want to see all the gory details of Cyrus, turn your
 syslog level down from DEBUG to INFO.  The message you are seeing is a
 DEBUG level log message.

 Andy


 I forgot to attach my rsyslog.conf in my previous reply.  Attached.

Check this line:

# Log all the mail messages in one place.
mail.*-/var/log/maillog

to:

# Log all the mail messages in one place.
mail.info -/var/log/imapd.log


Unfortunately, you'll probably end up with your MTA (sendmail, postfix, 
whatever RHEL uses) in the same file.  If they compiled Cyrus to use the 
MAIL facility instead of LOCAL6 as default, then there isn't much you can 
do about it.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Clients creates folders

[Andrew Morgan]

Just a general tip - don't ever login to Cyrus as the admin user with a 
regular IMAP client (Apple Mail, Thunderbird, etc).  Only use the admin 
account with cyradm or other administrative tools.


The admin account sees all mailboxes and uses the internal namespace 
(user.foo or user/foo), so it tends to confuse an IMAP client.


Andy

On Fri, 7 Dec 2012, Jörg Kruse wrote:


Indeed,
that was the case - I  removed this admin user now.
Hope this will solve it -
Thanks for the tip with telemetry logging - I enable it now to see more .

Jorg



Am 07.12.2012 15:51, schrieb Dan White:

On 12/07/12 09:57 +0100, Jörg Kruse wrote:

Dear all,

i am using cyrus-imapd-2.3.11-60.65.64.1 with saslauth against LDAP.
In my installation the apple imap clients create new folder as new
mailboxes.
The are created in the level of partion default - the mailboxes are in
partion-default/user/.
The crreted mailboxes have :
-localhost listacl probe1354832348536
-anyone lrs
 where the number is the unix timestamp.
I tried with anyoneuseracl: no with no succes.

Any idea what happens here ?? and how to prevent it ??


Within your apple imap client, are you connecting as an admin user? The
anyoneuseracl option only applies to non admin users.

Use telemetry logging to verify if it's your client adding the 'anyone
lrs'
acl.



-


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Help with cyrus-imapd, cyrus-sasl, postfix and lmtp

[Andrew Morgan]

On Sun, 4 Nov 2012, Dale J Chatham wrote:

 my intent it so have postfix in the DMZ delivering to cyrus lmtp and
 cyrus internal.

 I'd like to not have to have a map of users, but to use ideally sasldb
 to determine users and passwords, but pam if necessary.  I'd rather use
 stock packages and avoid compiling from scratch.

 Distro is centos 6.3

 I can't seem to get all the pieces talking to each other and have taken
 a week reading everything I can find.  This would seem to be a natural
 way to run, but I can't find docs on it.

 If there is a FAQ out there, someone please point me to it.

sasldb seems ill-suited for this purpose because you have 2 separate 
servers involved.  I suppose you could keep sasldb in sync on both servers 
with a cronjob or some other script that copies one to the other anytime 
there is a change.

Does that make sense?  Maybe I'm missing something in your concept.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mails vanished

[Andrew Morgan]

On Fri, 26 Oct 2012, Andre Bischof wrote:

 Hi,

 I'm using cyrus for years now, and have to say that it is a great piece of
 software, best fitting to my needs.

 Actually I use 2.2 (imap,pop3 amongst others) with Debian stable/testing.

 Two days ago I had to recognize that my mail client (thunderbird) only
 showed a handful of mails in my inbox when there should be thousands,
 reaching approx. 10 years back.

 Strange enough, my snapshot backup (faubackup) showed only a few more,
 going days, weeks and month back.

 I assumed there might be a problem with the disk and tested it thouroughly,
 but neither chkdsk nor SMART or other utilities would show errors or
 problems.

 I checked mail logs and aptitude.log as well, but there was nothing of
 interest to me, neither updates nor lots of deleted mails, only some
 expunged ones, but not too much.

 At this very moment I'm recovering files using testdisk - at least
 something.

 Could one of you tell me whether it's ok to just copy recovered files back
 to /var/spool/cyrus/mail/f/user/f-user/ ? Or am I supposed to recover with
 cyrus admin tools like:

 sudo -u cyrus /usr/sbin/cyrreconstruct -C /etc/imapd.conf -rf 
 user.BENUTZERNAME

 The file cyrus.header only contains:

 Cyrus mailbox header
 The best thing about this system was that it had lots of goals.
--Jim Morris on Andrew
478a94914314724f
 NonJunk Junk $Forwarded $MDNSent $Label1 $Label2 $Label3 $Label4 $Label5
 $has_cal
 friscolrswipcda

 Any hints to recover my mails are VERY ;) appreciated, as well as
 suggestions what might be the cause of the problem.

You can copy the recovered ###. files back into your normal mailbox 
directory and then run reconstruct (and quota -f, if you have a quota) 
afterwards.  Until you run reconstruct, the messages won't appear to IMAP 
clients.

I can't tell you anything about why the messages were deleted though.  :)

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus failure after full disk

[Andrew Morgan]

On Thu, 18 Oct 2012, Tom Plancon wrote:

 Hi all,

 I'm running a mail server with cyrus imap and postfix. I've had a bit of a 
 disaster where a runaway process, not related to email, filled my root 
 directory. I've take care of that and got most of the space back but now I'm 
 not receiving email and maillog is reporting this:

 Oct 18 14:55:53 pelican lmtpunix[32764]: DBERROR db4: Commonly caused by 
 moving a database from one database environment
 Oct 18 14:55:53 pelican lmtpunix[32764]: DBERROR db4: to another without 
 clearing the database LSNs, or by removing all of
 Oct 18 14:55:53 pelican lmtpunix[32764]: DBERROR db4: the log files from a 
 database environment
 Oct 18 14:55:53 pelican lmtpunix[32764]: DBERROR db4: 
 /var/lib/imap/deliver.db: unexpected file type or format
 Oct 18 14:55:53 pelican lmtpunix[32764]: DBERROR: opening 
 /var/lib/imap/deliver.db: Invalid argument
 Oct 18 14:55:53 pelican lmtpunix[32764]: DBERROR: opening 
 /var/lib/imap/deliver.db: cyrusdb error
 Oct 18 14:55:53 pelican lmtpunix[32764]: FATAL: lmtpd: unable to init 
 duplicate delivery database
 Oct 18 14:55:53 pelican master[3213]: process 32764 exited, status 75
 Oct 18 14:55:53 pelican master[3213]: service lmtpunix pid 32764 in READY 
 state: terminated abnormally
 Oct 18 14:55:53 pelican master[32765]: about to exec 
 /usr/lib/cyrus-imapd/lmtpd

 I've had to reconstruct mail boxes before but nothing like this! Any idea 
 how to repair this mess? I'm running CentOS 6.

1. Stop Cyrus
2. Delete /var/lib/imap/deliver.db
3. Delete the contents of /var/lib/imap/db/
4. Start Cyrus

deliver.db contains (mainly) transient data about messages.  If you delete 
it, you may get a repeated message from anyone using a vacation responder, 
but otherwise there are no negative consequences.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: TLS for proxy IMAP connections

[Andrew Morgan]

On Mon, 15 Oct 2012, Andrew Morgan wrote:

 I run a standard Cyrus Murder on v2.4.16.  When I have allowplaintext:0
 on my frontends and allowplaintext:1 on my backends, the frontends will
 not use TLS when proxying the connection to a backend, even if the
 frontend connection from the client used TLS or SSL.

 When I set allowplaintext:0 on the backend, then the frontend will use
 TLS for the proxy connection.

 Shouldn't the frontend attempt to use TLS for the proxy connection if
 STARTTLS is advertised?

Digging through the 2.4.16 source code, I see this in imap/backend.c:

 /* If we don't have a usable mech, do TLS and try again */
 } while (r == SASL_NOMECH  CAPA(s, CAPA_STARTTLS) 
  do_starttls(s, prot-tls_cmd) != -1 

So it appears that backend_authenticate will only use TLS if it is 
required.  I'll look into changing my allowplaintext setting to require 
TLS/SSL.

 On a related note, will a frontend ever make an IMAP-SSL proxy connection
 to a backend?  I ask because I want to set my maxchild parameter correctly
 on my backends.  Right now, all connections seem to be proxied to the
 imap service and none are made on the imaps service.

In my testing, even with allowplaintext:0 on the backend, an IMAP-SSL 
(port 993) frontend connection uses a IMAP-TLS (port 143 with STARTTLS) 
backend connection.

This is fine.  I just needed to know so I can set maxchild correctly on my 
backends.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

TLS for proxy IMAP connections

[Andrew Morgan]

I run a standard Cyrus Murder on v2.4.16.  When I have allowplaintext:0 
on my frontends and allowplaintext:1 on my backends, the frontends will 
not use TLS when proxying the connection to a backend, even if the 
frontend connection from the client used TLS or SSL.

When I set allowplaintext:0 on the backend, then the frontend will use 
TLS for the proxy connection.

Shouldn't the frontend attempt to use TLS for the proxy connection if 
STARTTLS is advertised?

On a related note, will a frontend ever make an IMAP-SSL proxy connection 
to a backend?  I ask because I want to set my maxchild parameter correctly 
on my backends.  Right now, all connections seem to be proxied to the 
imap service and none are made on the imaps service.

Thanks,
Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: TLS wrrors on cyrus imapd log file

[Andrew Morgan]

The code block which generates the log error is:

 if ((!SSL_CTX_load_verify_locations(s_ctx, CAfile, CApath)) ||
 (!SSL_CTX_set_default_verify_paths(s_ctx))) {
 /* just a warning since this is only necessary for client auth */
 syslog(LOG_NOTICE,TLS server engine: cannot load CA data);
 }

If you are not using TLS client auth (x509 client certs), then you could 
ignore the error.  But errors in logs are annoying, so it would be nice to 
fix it.

Is /etc/pki/CA/INFN-CA.pem readable by the user Cyrus runs as?

Does the following openssl command report any errors:

   openssl x509 -in /etc/pki/CA/INFN-CA.pem -text

How about this command:

   openssl s_client -connect imap_server_name:993 -CAfile 
/etc/pki/CA/INFN-CA.pem


Andy

On Thu, 20 Sep 2012, Riccardo Veraldi wrote:

 these are my settings

 tls_cert_file: /etc/pki/tls/certs/iride.pem
 tls_key_file: /etc/pki/tls/private/iride.key
 tls_ca_file: /etc/pki/CA/INFN-CA.pem


 On 9/20/12 8:15 PM, Andrew Morgan wrote:
 On Thu, 20 Sep 2012, Riccardo Veraldi wrote:
 
 Hello,
 I am using cyrus-imapd-2.4.10
 
 I have configured it properly with X509 certificates.
 Everything is working fine but for every client connection I receive
 this error: TLS server engine: cannot load CA data
 
 Sep 16 04:04:42 iride imaps[9363]: TLS server engine: cannot load CA data
 Sep 16 04:04:42 iride imaps[9363]: imapd:Loading hard-coded DH parameters
 Sep 16 04:04:42 iride imaps[9363]: SSL_accept() incomplete - wait
 Sep 16 04:04:42 iride imaps[9363]: SSL_accept() succeeded - done
 Sep 16 04:04:42 iride imaps[9363]: starttls: TLSv1 with cipher
 DHE-RSA-AES256-SHA (256/256 bits reused) no authentication
 Sep 16 04:04:42 iride imaps[9363]: login: wilco.mylocaldomain.org
 [172.16.10.94] username plain+TLS User logged in
 
 X509 certificate is ok it is not expired; it complains about CA
 certificate data, but the certificate path inside imapd.conf is correct.
 
 what the problem could be ?
 
 What are your tls_* settings in imapd.conf?  I am running Cyrus v2.4.16 and 
 do not see the cannot load CA data error in my logs.  Here are my tls_* 
 settings:
 
 tls_ca_path: /etc/ssl/certs
 tls_cert_file: /etc/ssl/certs/imap.onid.oregonstate.edu.crt
 tls_key_file: /etc/ssl/certs/imap.onid.oregonstate.edu.key

 Andy



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: TLS wrrors on cyrus imapd log file

[Andrew Morgan]

On Thu, 20 Sep 2012, Riccardo Veraldi wrote:

 Hello,
 I am using cyrus-imapd-2.4.10

 I have configured it properly with X509 certificates.
 Everything is working fine but for every client connection I receive
 this error: TLS server engine: cannot load CA data

 Sep 16 04:04:42 iride imaps[9363]: TLS server engine: cannot load CA data
 Sep 16 04:04:42 iride imaps[9363]: imapd:Loading hard-coded DH parameters
 Sep 16 04:04:42 iride imaps[9363]: SSL_accept() incomplete - wait
 Sep 16 04:04:42 iride imaps[9363]: SSL_accept() succeeded - done
 Sep 16 04:04:42 iride imaps[9363]: starttls: TLSv1 with cipher
 DHE-RSA-AES256-SHA (256/256 bits reused) no authentication
 Sep 16 04:04:42 iride imaps[9363]: login: wilco.mylocaldomain.org
 [172.16.10.94] username plain+TLS User logged in

 X509 certificate is ok it is not expired; it complains about CA
 certificate data, but the certificate path inside imapd.conf is correct.

 what the problem could be ?

What are your tls_* settings in imapd.conf?  I am running Cyrus v2.4.16 
and do not see the cannot load CA data error in my logs.  Here are my 
tls_* settings:

tls_ca_path: /etc/ssl/certs
tls_cert_file: /etc/ssl/certs/imap.onid.oregonstate.edu.crt
tls_key_file: /etc/ssl/certs/imap.onid.oregonstate.edu.key

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Murder mailbox create race condition

[Andrew Morgan]

Bron helped me track it down.  Starting in Cyrus v2.4.13, there is a check 
to see if we're running on a standard Murder backend.  If so, some code to 
update the mailbox list is skipped.

The code was identifying a server as a backend server by checking for the 
presence of the proxyservers config variable.  I had proxyservers set on 
my frontends (needlessly).  Once I commented out proxyservers, the race 
condition was gone.

Problem solved, and Bron committed a documentation fix to the imapd.conf 
manpage.

Thanks,
Andy

On Wed, 12 Sep 2012, Andrew Morgan wrote:

 I recently upgraded our Cyrus murder cluster from v2.4.12 to v2.4.16.
 Since then, I have come across an interesting race condition.  When
 connected to a frontend server, if I create a mailbox and then immediately
 try to select it, I will get an error message.

 Frontend IMAP telemetry:

 1347491960c34 create foo
 1347491960c34 OK Completed
 1347491960c35 select foo
 1347491960c35 NO Mailbox does not exist

 if I wait a few seconds, it works:

 1347491990c37 create foo
 1347491990c37 OK Completed
 1347491994c38 select foo
 1347491994* 0 EXISTS
 * 0 RECENT
 ...


 When I connect to a backend server, I cannot reproduce this:

 1347492147c34 create foo
 1347492147c34 OK Completed
 1347492147c35 select foo
 1347492147* 0 EXISTS
 * 0 RECENT
 ...


 Is there some reason the frontend server doesn't know about the newly
 created mailbox for a short period of time?

 This error happens everytime I attempt to postpone a message composition
 in Alpine because it creates the postponed-msgs mailbox and then
 immediately attempts to Append the message, which fails.

 I never saw this happen in v2.4.12.

 Thanks,
   Andy
 
 Cyrus Home Page: http://www.cyrusimap.org/
 List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
 To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Murder mailbox create race condition

[Andrew Morgan]

I recently upgraded our Cyrus murder cluster from v2.4.12 to v2.4.16. 
Since then, I have come across an interesting race condition.  When 
connected to a frontend server, if I create a mailbox and then immediately 
try to select it, I will get an error message.

Frontend IMAP telemetry:

1347491960c34 create foo
1347491960c34 OK Completed
1347491960c35 select foo
1347491960c35 NO Mailbox does not exist

if I wait a few seconds, it works:

1347491990c37 create foo
1347491990c37 OK Completed
1347491994c38 select foo
1347491994* 0 EXISTS
* 0 RECENT
...


When I connect to a backend server, I cannot reproduce this:

1347492147c34 create foo
1347492147c34 OK Completed
1347492147c35 select foo
1347492147* 0 EXISTS
* 0 RECENT
...


Is there some reason the frontend server doesn't know about the newly 
created mailbox for a short period of time?

This error happens everytime I attempt to postpone a message composition 
in Alpine because it creates the postponed-msgs mailbox and then 
immediately attempts to Append the message, which fails.

I never saw this happen in v2.4.12.

Thanks,
Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: HOWTO recover mail in maildir(s) from backup onto new server

[Andrew Morgan]

On Sat, 8 Sep 2012, John Mok wrote:

 Hi,

 Due to RAID crash, we have a new server to replace the old one. I would
 like to someone to advise how to recover those old mails in maildir(s)
 from backup onto the new server ?

If you have the full backup, then you should be able to recover the entire 
Cyrus mail spool, including the config directory.  I recommend running 
reconstruct on every mailbox after you restore the files.

Maybe someone else on the list has more detailed recommendations...  Feel 
free to post any errors you are seeing and we'll try to help!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: optimized mode for empty maildrop...

[Andrew Morgan]

On Fri, 24 Aug 2012, Ron Vachiyer wrote:

 Hello,

 I just installed a 2.3 system, and am looking to decrease the syslog 
 verbosity.  This system has mostly POP accounts, and the log message 
 optimized mode for empty maildrop is all the eye can see, hundreds of 
 them per minute.  Is there a way to limit the logging to 
 success/fail/delivered and reduce the rest?

Here is the code that prints that message:

 else if (config_getswitch(IMAPOPT_STATUSCACHE) 
  !(r = statuscache_lookup(inboxname, userid, STATUS_MESSAGES, 
scdata)) 
  !scdata.messages) {
 /* local mailbox (empty) -- don't bother opening the mailbox */
 syslog(LOG_INFO, optimized mode for empty maildrop: %s, popd_userid);

 proc_register(pop3d, popd_clienthost, popd_userid, inboxname);
 }


I suppose you could turn off the statuscache (statuscache:0 in 
imapd.conf).  If you are compiling from src, you could certainly change 
LOG_INFO to LOG_DEBUG and recompile.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 4096 file descriptors

[Andrew Morgan]

On Wed, 22 Aug 2012, Ron Vachiyer wrote:

 Quick question about filedescriptors.  On Centos6, cyrus 2.3.16 seems to 
 be able to open 4096 FDs ;

 master[27121]: retrying with 4096 (current max)

 ulimit -a says 1024;

 open files  (-n) 1024

 I am looking to increase this, and have found some documentation saying 
 to increse file-max in /proc.  However, file-max already has a much 
 larger number;

 cat /proc/sys/fs/file-max
 1201105

 The only way I have found so far is to add a ulimit -n 8192 in 
 /etc/rc.d/init.d/cyrus-imapd


 Is there a more generic/cleaner way to do this?

I've always done it in the cyrus init script:

# Crank up the limits
ulimit -n 209702
ulimit -u 4096
ulimit -c 102400


You may be able to set it in limits.conf (pam_limits), but I'm not sure if 
that applies when starting cyrus from the init script??

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: SASL and default domain

[Andrew Morgan]

On Sun, 19 Aug 2012, brian wrote:

 I'm having some trouble configuring SASL for a new server. Specifically,
 it seems, with realms. I'm now at the point where imtest works with the
 virtual domains but not with the default domain.

 I'm using sasldb through auxprop. In the past I've always done:

 saslpasswd2 -c usern...@domain.tld

 But in order to get SASL working with Postfix this time I had to specify
 the realm with -u and use a bare account name:

 saslpasswd2 -c -u DEFAULT.TLD username
 saslpasswd2 -c -u VDOMAIN1.TLD username
 etc

 After days of struggle, I've got Postfix responding well when testing
 via telnet. The base64 hash was created with:

 perl -MMIME::Base64 -e 'print
 encode_base64(\000user\@DOMAIN.TLD\000password);'

 I mention all that because it seems as if realms are the issue. Or it
 was before and I suppose that's been resolved. Now it's just the default
 domain that's giving me problems. It's been days and days now and I'm so
 close that I'm reluctant to fiddle any more because I know that the
 chances are good that I'll make things worse (as I've probably
 repeatedly done already). I'd appreciate it if someone could suggest
 something to save the rest of my hair.

 FWIW, this server has no DNS records pointing to it yet. My goal is to
 get Postfix  Cyrus working to the point where I can use imapsync, then
 deal with DNS. This is what I've done in the past.

 (And imapsync is working now with the virtual domains.)


 $ hostname -f
 poseidon.DEFAULT.TLD

 $ imtest -v -m plain -a u...@default.tld localhost
 S: * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN
 AUTH=LOGIN AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR] poseidon Cyrus IMAP
 v2.4.12-Debian-2.4.12-2 server ready
 Please enter your password:
 C: A01 AUTHENTICATE PLAIN 
 S: A01 NO authentication failure
 Authentication failed. generic failure
 Security strength factor: 0

Does it work if you use:

   imtest -v -m plain -a user -r DEFAULT.TLD localhost


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: How to be sure that I can remove a mailbox partition

[Andrew Morgan]

On Wed, 27 Jun 2012, Javier Sánchez-Arévalo Díaz wrote:

I have a email server with two local partitions for mailboxes (default and 
part2). Recently these partitions became almost full so I decided create a 
new partition over a NFS mountpoint and migrate all the mailboxes to this new 
partition (part3).


The target is to move all  the mailboxes to part3 in order to leave 
default and part2 completely empty. Once this is done I want to stop 
using them (default and part2) and remove physically the hardisks where 
they are in order to plug new bigger disks.


After moving all the mailboxes to part3 everything is working fine but, 
before removing default and part2, I would like to make a question:


The question is. How I can be completely sure that I can do it safely?

I have done the next tests but I would prefer to ask to experts like you 
before doing It. Its a production server with almost 2 mailboxes:



// list of partitions

pcocol01:~ # cat /etc/imapd.conf | grep -i part
partition-default: /buzonesdir
partition-part2: /mnt/aux
partition-part3: /mnt/celerra
defaultpartition: part3




// mount points of these partitions
pcocol01:~ # mount
/dev/cciss/c0d1p1 on /buzonesdir type reiserfs (rw,acl,user_xattr)
/dev/sdb1 on /mnt/aux type ext3 (rw)
212.145.146.8:/FS_AUX/FS_AUX on /mnt/celerra type nfs (rw,addr=212.145.146.8)





// Check that nobody is using default and part2

pcocol01:~ # fuser -m /mnt/aux/
pcocol01:~ #


pcocol01:~ # fuser -m /buzonesdir/
pcocol01:~ #

pcocol01:~ # fuser -m /mnt/celerra
/mnt/celerra: 6488cm  7467cm  7500cm  7501cm  7504cm 7505cm  7507cm 
7508cm  7513cm  7514cm  7515cm  7519cm  7549cm 7565cm  7567cm  7573cm  7596cm 
7607cm  7623cm  7624cm  7625cm 7626cm 10513c 10521cm 10527cm 24266cm 26056cm 
10233cm 32528cm 26829cm  7667cm 13155cm  5872cm  6020cm 27926cm 27931cm 
27935cm 28826cm  4689cm  4874cm  5893cm  5895cm  6389cm  6446cm  7286cm 
7407cm  7509cm  8716cm  8884cm  8889cm 29283cm 14523cm  3771cm 3772cm  3613cm 
13822cm 22701c 16272cm 11921c  7069c 28817cm 12127cm 28148cm 11318cm   801cm 
1052c  3823c  8338cm  9092cm  9883c 13770cm 19410cm 20014cm 20956cm 24031c 
24371cm 25054cm 25827c 27930c 31191cm 31542cm 31612c 31701cm 31829cm 32249cm 
32357cm 32374cm 32400cm 32462c  1274c  1289c  1756c  1956cm  2260cm  7039c 
7506cm  7613cm 7883cm  7949cm  7956cm  8069c  8445cm  8540c  8611c  9297cm 
9495cm 6cm 14525cm 15235cm 16589cm 19732cm 20811cm 20960cm 21215cm 
21309cm 21606cm 21667cm 22861cm 22878cm 23179c 23222cm 23305cm 23308cm 23321c 
23364cm 23492c 23550c 23557cm 23599cm 23610cm 23616cm 23758cm 23870cm 24103cm 
24144cm 24154cm 24195cm 24200cm 24257cm 24432cm 24748cm 25025cm 25027cm 
25159cm 25160cm 25171cm 25175cm 25190cm 25428cm 25429cm 25484cm 25487cm 
25650cm 25688cm 25923cm 25955cm 25962cm 25968cm 25969cm 26293cm 26482cm 
26529c 26532cm 26534cm 26603cm 26640cm 26649cm 26691cm 26760c 26761cm 26851cm 
26866cm 26891cm 26919cm 26922cm 26935cm 26961cm 27123cm 27208cm 27281cm 
27350cm 27422cm 27432cm 27498cm 27639cm 27640c 27735cm 27741cm 27778cm 
27839cm 27865cm 27868cm 27909cm 27928cm 28018cm 28064cm 28170cm 28198cm 
28201cm 28226cm 28241cm 28258c 28295cm 28315cm 28340cm 28343cm 28348cm 
28371cm 28379cm 28606c 28640cm 28641c 28642c 28646cm 28654cm

pcocol01:~ #




//list of mailboxes to be sure that none is located in default or part2

cyrus@pcocol01:~ ctl_mboxlist -d | wc -l
75444
cyrus@pcocol01:~ ctl_mboxlist -d -p default | wc -l
0
cyrus@pcocol01:~ ctl_mboxlist -d -p part2 | wc -l
0
cyrus@pcocol01:~ ctl_mboxlist -d -p part3 | wc -l
75444


These results from ctl_mboxlist show me that no mailboxes reside on 
default or part2 partitions.





// Contents of default and part2 partitions

pcocol01:~ # ls -l /buzonesdir/
total 0
drwx-- 2 cyrus mail 448 2012-06-20 13:32 stage.
drwxr-xr-x 2 cyrus mail  48 2012-06-27 08:37 user
cyrus@pcocol01:~ ls -l /buzonesdir/user/
total 0
cyrus@pcocol01:~ ls -l /buzonesdir/stage./
total 2094
-rw--- 1 cyrus mail8288 2012-04-26 12:25 10770-1335435917-0
-rw--- 1 cyrus mail   69557 2007-10-04 17:05 13569-1191510346-0
-rw--- 1 cyrus mail  630968 2007-11-12 08:33 17067-1194852808-0
-rw--- 1 cyrus mail 969 2010-06-27 21:15 23403-1277666141-1
-rw--- 1 cyrus mail3054 2012-05-16 08:07 24524-1337148471-0
-rw--- 1 cyrus mail3063 2012-05-16 08:07 24530-1337148469-0
-rw--- 1 cyrus mail3004 2012-05-16 08:07 24539-1337148470-0
-rw--- 1 cyrus mail1564 2010-06-26 18:25 27236-1277569504-1
-rw--- 1 cyrus mail 1397725 2007-09-28 13:20 32238-1190978409-0
-rw--- 1 cyrus mail3457 2007-09-28 13:19 32461-1190978393-0


These are probably junk.  The .stage directory is used to hold messages 
temporarily during delivery.  Sometimes if there 

Re: Outlook not recieving mail

[Andrew Morgan]

On Sat, 23 Jun 2012, JonL wrote:

 ok as per my last e-mail I've been able to login via telnet, but still 
 no mail in the outlook client.

 0 login username password
 0 OK User logged in
 0 select inbox
 * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
 * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
 * 0 EXISTS
 * 0 RECENT

Looks to me like the server is saying there are no messages in the INBOX.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: In preparation of Cyrus IMAP 2.5: autoconf and automake

[Andrew Morgan]

On Mon, 21 May 2012, Bron Gondwana wrote:

 On Mon, May 21, 2012 at 12:03:31PM -0700, Andrew Morgan wrote:
 On Sat, 28 Apr 2012, Jeroen van Meeuwen (Kolab Systems) wrote:

 The canonical build process we think applies, generally speaking, is:

  $ autoreconf -v
  $ ./configure [your-options]
  $ make

 This process currently requires autoconf = 2.67.

 We would appreciate you let us know whether or not such process works for 
 you,
 preferrably though Bugzilla (please use product 'Cyrus IMAP' and component
 'Distribution').

 Why is autoreconf/autoconf required?

 I have been building Cyrus from source for many years.  The tarballs
 already come with the configure script, so I have never needed to install
 the autoconf package before.

 We'll probably keep shipping releases with the configure script - but if 
 you're
 building from git you need to do the whole dance.

Good enough for me then!

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: In preparation of Cyrus IMAP 2.5: autoconf and automake

[Andrew Morgan]

On Sat, 28 Apr 2012, Jeroen van Meeuwen (Kolab Systems) wrote:

 The canonical build process we think applies, generally speaking, is:

  $ autoreconf -v
  $ ./configure [your-options]
  $ make

 This process currently requires autoconf = 2.67.

 We would appreciate you let us know whether or not such process works for you,
 preferrably though Bugzilla (please use product 'Cyrus IMAP' and component
 'Distribution').

Why is autoreconf/autoconf required?

I have been building Cyrus from source for many years.  The tarballs 
already come with the configure script, so I have never needed to install 
the autoconf package before.

Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/