Re: cyrus and 500k to >1M mails per inbox

[Daniel O'Connor]

On Fri, 12 Feb 2010, Derek Yarnell wrote:
> Just as an aside we archive our root mail (lots of crap) into a cyrus
> mailbox and found that past 250k messages it just began to suck at
> the client layer.
>
> What we did was have sieve filter that would be re-generated on a
> monthly basis (using sievec) in the account that would filter into a
> folder for that month.  The only thing was that you would have to
> pre-populate the folders (or maybe I am a sieve noob and not aware of
> a way that sieve can create the folder if it doesn't exist).
>
> Anyway this is a pretty good way to deal with large archival
> mailboxes.

I wrote a script which creates a new folder each month and moves emails 
into it.

This does suck for offline users though (lucky that is not an issue for 
me).

I can send it to you if you like, it uses Python's base imaplib.

-- 
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
  -- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C


signature.asc
Description: This is a digitally signed message part.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: cyrus and 500k to >1M mails per inbox

[Derek Yarnell]

On 2/11/10 10:23 AM, Carsten Hoeger wrote:
> Hi,
>
>
> we have a customer running cyrus imap on RHEL5 cluster with a few thousand
> mailboxes.
> Every single mail is sent to a second host acting as a read-only archive (all
> imap acls set to no write access).
>
> The plan is, that all mail on that second host is kept for a long time.
>
> After some years, old mail will be purged automatically.
>
> It's quite likely, that these mailboxes will grow to 50 or even more then
> 1M mails per mailbox.
>
> Does anybody have experience with such big mailboxes?
> The filesystem is LVM with ext3.


Hi,

Just as an aside we archive our root mail (lots of crap) into a cyrus 
mailbox and found that past 250k messages it just began to suck at the 
client layer.

What we did was have sieve filter that would be re-generated on a 
monthly basis (using sievec) in the account that would filter into a 
folder for that month.  The only thing was that you would have to 
pre-populate the folders (or maybe I am a sieve noob and not aware of a 
way that sieve can create the folder if it doesn't exist).

Anyway this is a pretty good way to deal with large archival mailboxes.

Thanks,
derek

-- 
---
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Sieve and encoded Headers

[Bron Gondwana]

On Mon, Feb 08, 2010 at 11:29:22AM -0500, Ken Murchison wrote:
> 
> 
> Bron Gondwana wrote:
> >On Mon, Feb 08, 2010 at 07:53:21AM +0100, Garry wrote:
> >>Hi,
> >>
> >>after wondering for a while why occasionally my sieve script rules
> >>wouldn't work, I just found the reason (I guess) - Sieve doesn't
> >>(correctly?) decode utf encoded header lines which e.g. are in a format
> >>like this:
> >>
> >>Subject: 
> >>=?utf-8?B?W0xPR10gV2F0Y2hsaXN0OiBzbWVhZ29sIGZvdW5kIEdydcOfIGF1cyBkZXIgVW50ZXJ3ZWx0IChVbmtub3duIENhY2hlKQ==?=
> >>
> >>
> >>I'm using version: "Cyrus timsieved v2.2.13-Debian-2.2.13-19" ... is it
> >>something that is fixed in a newer version? I tried finding something on
> >>the net, but at least the first couple of results pages didn't yield any
> >>insight ...
> >
> >No - it's not fixed in any released version.  It is fixed in the FastMail
> >Cyrus patches - but it's a very invasive change to the charset encoding,
> >so it's been kept out of the stable line for now.
> >
> >I've CC'd Ken on this - I wonder if it's worth going back and doing a
> >"minimal still compatible" set of patches that fixes charset encoding in
> >sieve without actually changing the on disk format of the cyrus.cache
> 
> It might be worth doing this for 2.3.

Done!  I've put lots of testing in to it too :)  Added to the
cyrus-imapd-2_3-tail branch.
 
Bron.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Change "plus" character?

[Charles Boling]

Is there a way [that doesn't require recompiling] to change the
character that Cyrus IMAP uses for "plus addressing" from "+" to
something else?

Thanks.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

How to make sync_client invoke STARTTLS for replication

[Rich Wales]

I'm running Cyrus 2.3.16 (with replication) between two Ubuntu servers.

What do I have to do to make the "sync_client" application invoke STARTTLS
when it connects to "sync_server" on the other host?

I can invoke TLS when I use the "synctest" program, but I can't seem to
figure out how to force "sync_client" to use TLS when actually replicating.

The reason I'm assuming TLS is not happening is that when /var/log/syslog
records the "User logged in" events associated with replication, TLS is
not mentioned as part of the authentication mechanism in use.

Right now, the lack of TLS is not a major issue because one of the servers
is connected to my LAN via a VPN link (so it's encrypted).  But I still
want to know what I'm supposed to do in order for a TLS layer to happen.

Rich Wales
ri...@richw.org

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: cyrus and 500k to >1M mails per inbox

[Bron Gondwana]

On Fri, Feb 12, 2010 at 07:37:20AM +1100, Rob Mueller wrote:
> 
> >>> It's quite likely, that these mailboxes will grow to 50 or
> >>even > more then
> >>>  1M mails per mailbox.
> >>>
> >>> Does anybody have experience with such big mailboxes?
> >>
> >>Is the I/O cost of message adding relative to O(n), n being the
> >>number of msgs
> >>already in the mailbox, or is it independent thereof ?
> >
> >Stacks of stuff is O(n), sorry.  It's not really an ideal way to
> >do things.
> >You'd be better off archiving per month for each mailbox and keeping the
> >messages counts to a reasonable number.
> 
> Well it's not quite as bad as that.
> 
> If the mailbox is mostly just appended to, then it's not a problem.
> The append code just appends to cyrus.index and cyrus.cache I
> believe (Bron?), it doesn't have to read the cyrus.* files each
> time.

It also has to MMAP the the cyrus.index and cyrus.cache files, which
can be an issue on 32 bit architectures due to process size limitations.

It's amazing how quickly you can process a million records on a modern
machine as well!
 
> We have actually seen a mailbox with > 1M messages at FastMail, and
> I remember that I could view it ok. However, I'll add a few caveats
> around that.
> 
> 1. Our web interface has been highly optimised over time to make
> sure it deals ok with large UID lists. I'm not sure how many email
> clients will handle viewing an IMAP folder with 1M messages
> 2. You need a good filesystem as well, because there will be 1M
> files in a directory. We use reiserfs3, and again, it has no problem
> with that many files in a directory, mileage with other filesystems
> might vary.

Yep - and as I said with ever removing old stuff - you'll hit pretty
high expunge costs (I don't know how filesystem unlink cost scales
with the size of the directory entry... probably depends on the
filesystem a fair bit too!)

Bron.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: cyrus and 500k to >1M mails per inbox

[Rob Mueller]

>> > It's quite likely, that these mailboxes will grow to 50 or even 
>> > more then
>> >  1M mails per mailbox.
>> >
>> > Does anybody have experience with such big mailboxes?
>>
>> Is the I/O cost of message adding relative to O(n), n being the number of 
>> msgs
>> already in the mailbox, or is it independent thereof ?
>
> Stacks of stuff is O(n), sorry.  It's not really an ideal way to do 
> things.
> You'd be better off archiving per month for each mailbox and keeping the
> messages counts to a reasonable number.

Well it's not quite as bad as that.

If the mailbox is mostly just appended to, then it's not a problem. The 
append code just appends to cyrus.index and cyrus.cache I believe (Bron?), 
it doesn't have to read the cyrus.* files each time.

We have actually seen a mailbox with > 1M messages at FastMail, and I 
remember that I could view it ok. However, I'll add a few caveats around 
that.

1. Our web interface has been highly optimised over time to make sure it 
deals ok with large UID lists. I'm not sure how many email clients will 
handle viewing an IMAP folder with 1M messages
2. You need a good filesystem as well, because there will be 1M files in a 
directory. We use reiserfs3, and again, it has no problem with that many 
files in a directory, mileage with other filesystems might vary.

Rob


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: cyrus and 500k to >1M mails per inbox

[Bron Gondwana]

On Thu, Feb 11, 2010 at 04:39:25PM +0100, Eric Luyten wrote:
> On Thu, February 11, 2010 4:23 pm, Carsten Hoeger wrote:
> 
> >
> > we have a customer running cyrus imap on RHEL5 cluster with a few thousand
> > mailboxes. Every single mail is sent to a second host acting as a read-only
> > archive (all imap acls set to no write access).
> >
> > The plan is, that all mail on that second host is kept for a long time.
> >
> >
> > After some years, old mail will be purged automatically.
> >
> >
> > It's quite likely, that these mailboxes will grow to 50 or even more 
> > then
> >  1M mails per mailbox.
> >
> >
> > Does anybody have experience with such big mailboxes?
> 
> 
> Carsten,
> 
> No, I do not count more than 50k msgs in our largest mailboxes but I'd also
> welcome information on metadata (cyrus.index and cyrus.cache files) operations
> on huge mailboxes.
> Is the I/O cost of message adding relative to O(n), n being the number of msgs
> already in the mailbox, or is it independent thereof ?
> 
> (question hereby asked to the entire info-cyrus audience)

Stacks of stuff is O(n), sorry.  It's not really an ideal way to do things.
You'd be better off archiving per month for each mailbox and keeping the
messages counts to a reasonable number.

Bron ( it also makes purging 7 years ago or something a viable task )

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus on Amazon EC2 and EBS?

[Adam Tauno Williams]

On Thu, 2010-02-11 at 12:02 -0500, Forrest Aldrich wrote:
> On 2/11/10 11:55 AM, Adam Tauno Williams wrote:
> > On Thu, 2010-02-11 at 10:35 -0500, Zachariah Mully wrote:
> >> Howdy all-
> >> Is anyone out there running Cyrus on an Amazon EC2 instance with EBS?
> >> During these brutal storms we've had in the DC area, we've been
> >> discussing how to improve the resiliency of our operations and getting
> >> our mail systems out of our CO and into a datacenter is high on the
> >> list. I'm looking for any feedback on using Cyrus with Amazon or any
> >> other 'cloud' service (I'm also looking at Rackspace's cloudserver
> >> offering as we're already a customer). Any gotchas, performance,
> >> reliability issues, etc? About 120 users, 300+GB mailstore.
> > I run Cyrus on a Linode VM.  I've been *very* happy with them.  And it
> > is just a 'normal' virtual server and not any odd-ball cloud thing.
> I wonder how people deal with the disk I/O deficiency under virtual 
> systems such as VMWare.   We decided not to virtualize any processes 
> that were disk I/O intensive.

Well, the person quoted is talking about a ~120 user Cyrus box.  Odds
are the I/O load is pretty light.  I've found that for a couple hundred
users you can run Cyrus on just about anything.
-- 
Adam Tauno Williams  LPIC-1, Novell CLA

OpenGroupware, Cyrus IMAPd, Postfix, OpenLDAP, Samba


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus on Amazon EC2 and EBS?

[Zachariah Mully]

On 02/11/2010 12:02 PM, Forrest Aldrich wrote:
>
>
> On 2/11/10 11:55 AM, Adam Tauno Williams wrote:
>> I run Cyrus on a Linode VM.  I've been *very* happy with them.  And it
>> is just a 'normal' virtual server and not any odd-ball cloud thing.
>>
>
>
> I wonder how people deal with the disk I/O deficiency under virtual
> systems such as VMWare.   We decided not to virtualize any processes
> that were disk I/O intensive.

People stripe EBS volumes, but you're restricted by network IO.

Some rough number for I/O speeds:

http://www.thebitsource.com/featured-posts/rackspace-cloud-servers-versus-amazon-ec2-performance-analysis/

http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/

http://victortrac.com/

Z

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus on Amazon EC2 and EBS?

[Simon Fraser]

On Thu, 2010-02-11 at 12:02 -0500, Forrest Aldrich wrote:


> > I run Cyrus on a Linode VM.  I've been *very* happy with them.  And it
> > is just a 'normal' virtual server and not any odd-ball cloud thing.
> 
> 
> I wonder how people deal with the disk I/O deficiency under virtual 
> systems such as VMWare.   We decided not to virtualize any processes 
> that were disk I/O intensive.

The admin next to me, who looks after our VMWare infrastructure, tells
me that you can improve performance for I/O intensive functions by
presenting the storage as a raw device, rather than through the
datastore. Certainly, we run some busy databases and even an exchange
server on virtual machines, and the only complaints raised were when
they were accidentaly put on a datastore and the other VMs using it
suffered.  

Simon.




-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus on Amazon EC2 and EBS?

[Forrest Aldrich]

On 2/11/10 11:55 AM, Adam Tauno Williams wrote:
> On Thu, 2010-02-11 at 10:35 -0500, Zachariah Mully wrote:
>
>> Howdy all-
>> Is anyone out there running Cyrus on an Amazon EC2 instance with EBS?
>> During these brutal storms we've had in the DC area, we've been
>> discussing how to improve the resiliency of our operations and getting
>> our mail systems out of our CO and into a datacenter is high on the
>> list. I'm looking for any feedback on using Cyrus with Amazon or any
>> other 'cloud' service (I'm also looking at Rackspace's cloudserver
>> offering as we're already a customer). Any gotchas, performance,
>> reliability issues, etc? About 120 users, 300+GB mailstore.
>>  
> I run Cyrus on a Linode VM.  I've been *very* happy with them.  And it
> is just a 'normal' virtual server and not any odd-ball cloud thing.
>


I wonder how people deal with the disk I/O deficiency under virtual 
systems such as VMWare.   We decided not to virtualize any processes 
that were disk I/O intensive.



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus on Amazon EC2 and EBS?

[Adam Tauno Williams]

On Thu, 2010-02-11 at 10:35 -0500, Zachariah Mully wrote:
> Howdy all-
> Is anyone out there running Cyrus on an Amazon EC2 instance with EBS? 
> During these brutal storms we've had in the DC area, we've been 
> discussing how to improve the resiliency of our operations and getting 
> our mail systems out of our CO and into a datacenter is high on the 
> list. I'm looking for any feedback on using Cyrus with Amazon or any 
> other 'cloud' service (I'm also looking at Rackspace's cloudserver 
> offering as we're already a customer). Any gotchas, performance, 
> reliability issues, etc? About 120 users, 300+GB mailstore.

I run Cyrus on a Linode VM.  I've been *very* happy with them.  And it
is just a 'normal' virtual server and not any odd-ball cloud thing.

-- 
Adam Tauno Williams  LPIC, Novell CLA

OpenGroupware, Cyrus IMAPd, Postfix, OpenLDAP, Samba


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: cyrus and 500k to >1M mails per inbox

[Eric Luyten]

On Thu, February 11, 2010 4:23 pm, Carsten Hoeger wrote:

>
> we have a customer running cyrus imap on RHEL5 cluster with a few thousand
> mailboxes. Every single mail is sent to a second host acting as a read-only
> archive (all imap acls set to no write access).
>
> The plan is, that all mail on that second host is kept for a long time.
>
>
> After some years, old mail will be purged automatically.
>
>
> It's quite likely, that these mailboxes will grow to 50 or even more then
>  1M mails per mailbox.
>
>
> Does anybody have experience with such big mailboxes?


Carsten,

No, I do not count more than 50k msgs in our largest mailboxes but I'd also
welcome information on metadata (cyrus.index and cyrus.cache files) operations
on huge mailboxes.
Is the I/O cost of message adding relative to O(n), n being the number of msgs
already in the mailbox, or is it independent thereof ?

(question hereby asked to the entire info-cyrus audience)

Eric.



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Cyrus on Amazon EC2 and EBS?

[Zachariah Mully]

Howdy all-
Is anyone out there running Cyrus on an Amazon EC2 instance with EBS? 
During these brutal storms we've had in the DC area, we've been 
discussing how to improve the resiliency of our operations and getting 
our mail systems out of our CO and into a datacenter is high on the 
list. I'm looking for any feedback on using Cyrus with Amazon or any 
other 'cloud' service (I'm also looking at Rackspace's cloudserver 
offering as we're already a customer). Any gotchas, performance, 
reliability issues, etc? About 120 users, 300+GB mailstore.

Thanks,
Z

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Cyrus murder ( IMAP Aggregator )

[Wesley Craig]

On 10 Feb 2010, at 16:54, Richard Pijnenburg wrote:
> How do the backends deceide who services which mailbox?
>

The admin decides, more or less, as you provision mailboxes.

> And what happens if one of those backends dies?
>

The mailboxes on that backend are inaccessible.  The rest of the  
cluster continues to operate as normal, tho your mail queues might be  
negatively impacted.  You might want to investigate replication, for  
how to recover from a failed backend.

> If I understand also correct I’ll need 1 Murder master and then 1  
> or multiple Murder Front-end and back-end servers ?
>

More or less.  It's pretty reasonable to have one of the frontends  
act as the murder master, particularly in a low-load environment.

:wes

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

cyrus and 500k to >1M mails per inbox

[Carsten Hoeger]

Hi,


we have a customer running cyrus imap on RHEL5 cluster with a few thousand
mailboxes. 
Every single mail is sent to a second host acting as a read-only archive (all
imap acls set to no write access).

The plan is, that all mail on that second host is kept for a long time.

After some years, old mail will be purged automatically.

It's quite likely, that these mailboxes will grow to 50 or even more then
1M mails per mailbox.

Does anybody have experience with such big mailboxes?
The filesystem is LVM with ext3.

-- 
With best regards,

Carsten Hoeger


pgpjSUXKVm1zZ.pgp
Description: PGP signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html