OT: Collaboration replacement via Toltec/Bynari (was How many people to admin a Cyrus system?)

[Rob Mueller]

> So you will have a choice of 3 commercial Outlook plug-ins,

What are the 3 commercial Outlook plugins? Obviously the Toltec one, but 
which others?

I've actually always liked the idea of what toltec + bynari were doing. It's 
basically using the IMAP server as a database where folders = tables, emails 
= records. The nice things about using an IMAP server is that:
1) you're reusing a protocol, rather than creating another one
2) emails can't be altered, so an update is a delete + add which makes 
detecting changes easy
3) the server is always authoriative, so syncing tends to be easier

Of course reusing an email protocol to do these other things it was never 
meant to do seems hacky as well. Still, is it really any more hacky than 
using a blatant file protocol (despite calling files "resources" and 
directories "collections") than CalDAV?

As an FYI, last time I tried Bynari and Toltec, my general conclusions were:

Bynari - Annoying they use a binary format for storing the information, but 
good that they use the 0-9 ACLs to designate what folders are 
calendar/contacts/etc folders, means you don't need the damn ANNOTATEMORE 
extension and the patches to make it work with arbitrary values. Seemed a 
bit "flaky". It creates some process that runs next to Outlook, and 
sometimes that process wouldn't quit when you quit Outlook leaving things in 
an odd state. Install/removing the software seemed to be prone to 
failures/screwing up your config.

Toltec - Seemed less prone to bugginess than bynari, and using an XML format 
seemed more open so we could read it ourselves for other features in the 
future. Using ANNOTATEMORE was a pain. There was a showstopper problem 
however. When I tested it seemed that Toltec would display all sub-folders 
of INBOX, and using annotations it would set the Calendar, Contacts, etc 
folders as appropriate to display those items, and all other folders to 
display emails, but it would not show the INBOX! At the time I asked support 
about this it was suggested that you should setup a POP account to download 
the emails from the inbox to the "Local Folders" PST file. This seemed to 
completely defeat the purpose of IMAP to me. Has that been fixed?

Also out of interest, how much caching does toltec do? Basically it seems to 
me to get any decent performance you effectively have to download all the 
calendar/task/etc items from the appropriate folders and cache them all 
inside outlook. Any attempt to get from the server on a "when needed" basis 
seems like it would never perform well?

Rob


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

RE: OT: Re: How many people to admin a Cyrus system?

[Joon Radley]

Hi,

> This was quite some time ago, around the first release of their 2.x
> product IIRC. It was the outlook connector that would save all calendar
> entries from outlook to the IMAP server as binary messages, whereas the
> Horde web front end saved the entries in xml, so Cyrus was at least
> able
> to search them. So if you tried to access through Horde a calendar from
> an Outlook user, it was basically inoperable, as the webclient had to
> download the entire calendar mailbox, parse the binaries, sort them,
> then display them. Useless for anything more than a hundred entries or
> so...

< SELF PROMOTION ON >

All the Outlook plug-ins for Kolab now support the Kolab-XML format. KDE
Kontact already supports Kolab-XML and Horde will support the format in
Kolab 2.2 (coming soon TM - points finger at Gunnar). So you will have a
choice of 3 commercial Outlook plug-ins, one very reasonably priced, and
native Linux and a web clients.

We also have a migration tool from MS Exchange to Kolab, mail and groupware
objects like calendars, contacts, tasks, etc., still in beta.

< SELF PROMOTION OFF >

Kolab might just be worth another look. http://www.kolab.org/

Best Regards

Joon Radley
Radley Network Technologies CC
Cell: +27 (0)83 368 8557
Fax: +27 (0)12 998 4346
E-mail: [EMAIL PROTECTED]
Web: www.toltec.co.za




Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Replication: problems with synctest

[Rich Wales]

My earlier problem with "synctest" turned out to be caused by my not
specifying an authentication name ("-a" argument on the command line).
Since I didn't have an "-a" argument, "synctest" apparently decided
to use my account name (from the USER environment variable?) as the
authentication name -- which was definitely not what I wanted.

I figured this out by decoding my "AUTHENTICATE PLAIN" base64 string
and noticing my own account name included there, even though I was
trying to log in as "admin".

When I started using commands with both -u and -a, such as this . . .

synctest -u admin -a admin -m plain whodunit

. . . my "synctest"s started working just fine.

However, sync_client on one of my servers (flipflop) is still unable
to connect to sync_server on the other server (whodunit).

I've been adding a bunch of syslog() calls to backend.c and
saslclient.c, in an attempt to get "sync_client" to explain what
is going on.  So far, all I can determine is that the call to
sasl_client_start(), near the top of saslclient(), in saslclient.c,
is failing with a SASL_FAIL returned value.  I'm not sure why, yet.

Replication syncing in the other direction (whodunit to flipflop)
works just fine.  Remember, what I'm trying to do here is to set up
replication going both ways between a pair of servers.

This is 2.3.9, BTW.

Any ideas welcome.

-- 
Rich Wales  ===  Palo Alto, CA, USA  === [EMAIL PROTECTED]
http://www.richw.org   ===   http://en.wikipedia.org/wiki/User:Richwales
"The difference between theory and practice is that, in theory,
theory and practice are identical -- whereas in practice, they aren't."

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Squatter causing load spike

[Rob Banz]

>>
> It seems like the default for cyr_expire runs at 4:00AM (delprune
> cmd="cyr_expire -E 3" at=0400) and I start squatter at 3:00AM. Do you
> think that this would cause the spike and server to lock up? We are
> running RHEL4U4

They both compete for a lot of resources... You probably shouldn't run  
them both at the same time.  My cyrii run on Solaris, so I'm not  
familiar with what platform specific issues you may be having.

You might try running just squatter one night without expire and see  
how it does.

Also, make sure you're running squatter with the option (think it's - 
s) that makes it only reindex mailboxes that have changed since the  
last squatter run.

-rob



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Squatter causing load spike

[Gerard]

On Nov 13, 2007 5:07 PM, Rob Banz <[EMAIL PROTECTED]> wrote:
>
>
> On Nov 13, 2007, at 16:47, Gerard wrote:
>
> > I am running squatter at 3am as a cron job on all of my servers. Over
> > the passed week I have one server where squatter spikes the load and
> > ends up locking up the server at around 8am every morning. Yeah, it
> > seems to take that long to run which may be an issue in itself. Has
> > anyone come accross this or have a suggestion on how to get squatter
> > to perform better?
> > 
>
> What OS?
>
> Squatter shouldn't be causing "load spikes" unless there's some
> resource contention for lock files, etc.
>
> You should also make sure that if you're running cyr_expire, you're
> NOT running it concurrently with squatter.  Bad things happen.*  To
> get around this, I created a "nightly" shell script which runs
> cyr_expire, then squatter, and kick that off as a scheduled task in my
> cyrus.conf.
>
> * The bad things come into play if expire & squatter are processing
> the same mailbox at the same time.  Squatter will probably just
> "stop", and not tell you why.
>
> -rob
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>
It seems like the default for cyr_expire runs at 4:00AM (delprune
cmd="cyr_expire -E 3" at=0400) and I start squatter at 3:00AM. Do you
think that this would cause the spike and server to lock up? We are
running RHEL4U4

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Zachariah Mully]

On Tue, 2007-11-13 at 12:22 -0800, Scott M. Likens wrote:
> Ian,
> 
> The only problem with using clamav-filter (or something appropriate)
> as a milter, etc.  Was it did not fall into what Zimbra designed.
> Quite frankly I think it would have been a lot smoother with Sendmail
> +Milter+Clamav+whatever else they wanted.
> 
> However, that is not the direction they picked, either for licensing
> or whatever.
> 
> One thing with Zimbra, is you don't exactly get to pick what you want.
> They throw a ball of software at you, and expect you to work with
> it.  
> 
> I admit, if I didn't review their code as much, and try and see how it
> worked.  I would have been more oblivious and pleased with Zimbra
> maybe.
> 
> Scott

I looked at it when it was announced and it ticked me off that you had
to move over to their proprietary IMAP server. I never understood why
they couldn't have made that aspect of it pluggable. Cool interface
until you're on dialup.

Z




Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Squatter causing load spike

[Rob Banz]

On Nov 13, 2007, at 16:47, Gerard wrote:

> I am running squatter at 3am as a cron job on all of my servers. Over
> the passed week I have one server where squatter spikes the load and
> ends up locking up the server at around 8am every morning. Yeah, it
> seems to take that long to run which may be an issue in itself. Has
> anyone come accross this or have a suggestion on how to get squatter
> to perform better?
> 

What OS?

Squatter shouldn't be causing "load spikes" unless there's some  
resource contention for lock files, etc.

You should also make sure that if you're running cyr_expire, you're  
NOT running it concurrently with squatter.  Bad things happen.*  To  
get around this, I created a "nightly" shell script which runs  
cyr_expire, then squatter, and kick that off as a scheduled task in my  
cyrus.conf.

* The bad things come into play if expire & squatter are processing  
the same mailbox at the same time.  Squatter will probably just  
"stop", and not tell you why.

-rob

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Sieve redirect seems to stop script processing?

[Scott M. Likens]

*smack Ingo*

That's horrid...

require "fileinto";
if header :comparator "i;ascii-casemap" :contains "X-Spam-Flag" "YES"  {
fileinto "INBOX.Spam";

}
else
{
   fileinto "INBOX";}


I don't know what if true { is for... as that really doesn't make sense 
for me.  But for the other portion of it.

Scott

Davin Flatten wrote:
> Hello-
>
> We are using Horde/Ingo against a Cyrus murder with three backend
> servers.  When a user redirects there email the system generates the
> following script:
>
> ##INGO
> # sieve filter generated by Ingo (November 13, 2007, 10:08 am)
>
> require "fileinto";
>
> # Forwards
> if true {
> redirect "[EMAIL PROTECTED]";
> }
>
> # Spam Filter
> if header :comparator "i;ascii-casemap" :contains "X-Spam-Flag" "YES"  {
> fileinto "INBOX.Spam";
> stop;
> }
>
> It seems that the script stops processing at the redirect action and
> does not continue to the Spam filter.  Is this normal behavior of the
> implied keep?
>
> Thanks!
> Davin Flatten
> Systems Administrator
> Engineering Computer Services
> University of Massachusetts
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>
>
> !DSPAM:473a181285876444210739!
>
>
>   


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Adam Tauno Williams]

> > Never had a corrupt PST using ZideLook (the Outlook plugin for
> > OpenGroupware).  Make sue they provide a *real* MAPI provider for
> > outlook and not some background sync thingy (as several Open Source
> > Outlook connectors do or at least did).
> I'll have to give OGo a shot again, when I tested it, their connector
> was still alpha-alpha quality. Any other data on your OGo install

About ~250 users through various means,  only a handful of Outlook
users.  If you download the InstantOGo ISO it comes with a demo license
(I believe) so you can easily just install & test it in VMware, etc...
http://www.instantogo.com/download/index.html

Also WMOGAG (Whitemice Consulting OpenGroupware Adminstrator's Guide)
might be handy,  otherwise documentation other than for ZideLook is
pretty sparse.
http://docs.opengroupware.org/Members/whitemice/wmogag/file_view

> (number of users, clients, etc.) that you'd be willing to part with?
> I've been lucky to avoid the hellhole of exchange, but we've started
> hiring more people from larger companies who can't understand why we
> don't have it. Then I ask them how often they lost email or didn't have
> email, they usually tell me "frequently" or "several times, for over a
> week", and I get to tell them that in 7 years we've only had one
> unplanned email outage during business hours which I think lasted an
> hour... And that was when I bolloxed something ;)
> > Yep,  the Kolab architecture goes in the "what are you, nuts?" column.
> Good to hear someone else thought so as well...



signature.asc
Description: This is a digitally signed message part

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Squatter causing load spike

[Gerard]

I am running squatter at 3am as a cron job on all of my servers. Over
the passed week I have one server where squatter spikes the load and
ends up locking up the server at around 8am every morning. Yeah, it
seems to take that long to run which may be an issue in itself. Has
anyone come accross this or have a suggestion on how to get squatter
to perform better?

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Michael Bacon]

At the risk of being yet one more techie who thinks he has a workaround...

I'm back (in the past two months) doing Cyrus administration after a three 
year break.  I ran Cyrus instance at Duke University before, and am now 
getting up to speed to run the one at UNC.  At Duke we started as a 
multi-host install, and moved to a single instance just as I was leaving. 
Here at UNC, we've been on a single instance for years.  Both places have 
been Solaris all along, and both places had over 50k users and receiving 
several million messages a day.

Part of the way we handle it here is with massive hardware -- An 8 
processor Sun 6800 with the processor boards swapped out to UltraSparc 4s. 
These are still a couple of years old at this point.  That said, our CPU 
load is really pretty minimal.

While we're on a very old version of Cyrus right now (1.6), I think reading 
this that I've got a good feel for what you're looking at.  There's been a 
lot of talk about the linked list in the kernel and the fact that it 
freezes all processes with that file mmap'ed when the file gets written. 
If the spanning of the linked list were really the problem, I think we 
would have seen a total system meltdown here a long time ago.

I'm much more inclined to think that what you're running into is all of the 
processes freezing during the latency period for the re-write of the 
mailboxes file.  This won't show up as I/O blocking on your disk, as there 
won't be any real contingency for that file or even for the channel.  But 
the latency of the write, while only a few milliseconds, is going to kill 
you if your mailboxes file gets big.

I haven't had any role yet in the design and configuration of UNC's system, 
but there's one thing we have that I think saves us an enormous amount of 
pain.  Since we're still on 1.6, and hence using the "plain text" mailboxes 
format, bear in mind that all changes to the mailboxes database involve a 
lock on the file, a complete rewrite of the file next to it on the file 
system, and a rename() system call.  This is SLOOOWWW.  How are we not dead?

Solid state disk for the partition with the mailboxes database.

This thing is amazing.  We've got one of the gizmos with a battery backup 
and a RAID array of Winchester disks that it writes off to if it loses 
power, but the latency levels on this thing are non-existent.  Writes to 
the mailboxes database return almost instantaneously when compared to 
regular spinning disks.  Based on my experience, that's bound to be a much 
bigger chunk of time than traversing a linked list in kernel memory.

For anyone doing a big Cyrus install, I would strongly recommend this.

Michael Bacon
ITS - UNC Chapel Hill

--On Friday, November 09, 2007 10:35 AM -0800 Vincent Fox 
<[EMAIL PROTECTED]> wrote:

> Jure Pečar wrote:
>> In my expirience the "brick wall" you describe is what happens when disks
>> reach a certain point of random IO that they cannot keep up with.
>>
>
> The problem with a technical audience, is that everyone thinks they have
> a workaround
> or probable fix you haven't already thought of.  No offense. I am guilty
> of it myself but
> it's very hard to sometimes say "I DON'T KNOW" and dig through telemetry
> and instrument the software until you know all the answers.
>
> With something as complex as Cyrus, this is harder than you think.
> Unfortunately when it comes to something like a production mail service
> these days it's nearly impossible to get the funding and manhours and
> approvals to run experiments on live guinea pigs to really get to the
> bottom of problems.  We throw systems at the problem and move on.
>
> But in answer to your point, our iostat numbers for busy or service time
> didn't
> indicate there to be any I/O issue.  That was the first thing we looked
> at of course.
> Even by eyeball our array drives are more idle than busy.
>
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html





Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Zachariah Mully]

On Tue, 2007-11-13 at 14:29 -0500, Adam Tauno Williams wrote:

> Never had a corrupt PST using ZideLook (the Outlook plugin for
> OpenGroupware).  Make sue they provide a *real* MAPI provider for
> outlook and not some background sync thingy (as several Open Source
> Outlook connectors do or at least did).

I'll have to give OGo a shot again, when I tested it, their connector
was still alpha-alpha quality. Any other data on your OGo install
(number of users, clients, etc.) that you'd be willing to part with?
I've been lucky to avoid the hellhole of exchange, but we've started
hiring more people from larger companies who can't understand why we
don't have it. Then I ask them how often they lost email or didn't have
email, they usually tell me "frequently" or "several times, for over a
week", and I get to tell them that in 7 years we've only had one
unplanned email outage during business hours which I think lasted an
hour... And that was when I bolloxed something ;)

> Yep,  the Kolab architecture goes in the "what are you, nuts?" column.

Good to hear someone else thought so as well...

Z
-- 
Zachariah Mully
Director, Systems and Networks
SmartBrief, Inc.
[EMAIL PROTECTED]
p: 202-737-5500 x226
c: 202-422-8780
f: 202-737-7577


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Sieve redirect seems to stop script processing?

[Davin Flatten]

Hello-

We are using Horde/Ingo against a Cyrus murder with three backend
servers.  When a user redirects there email the system generates the
following script:

##INGO
# sieve filter generated by Ingo (November 13, 2007, 10:08 am)

require "fileinto";

# Forwards
if true {
redirect "[EMAIL PROTECTED]";
}

# Spam Filter
if header :comparator "i;ascii-casemap" :contains "X-Spam-Flag" "YES"  {
fileinto "INBOX.Spam";
stop;
}

It seems that the script stops processing at the redirect action and
does not continue to the Spam filter.  Is this normal behavior of the
implied keep?

Thanks!
Davin Flatten
Systems Administrator
Engineering Computer Services
University of Massachusetts

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

v2.3.10 build fails, pcreposix problems

[Rosenbaum, Larry M.]

When trying to build Cyrus IMAP v2.3.10 on a Solaris 9 system, I get the 
following error:

gmake[1]: Entering directory `/usr/local/src/cyrus/cyrus-imapd-2.3.10/sieve'
../com_err/et/compile_et ./sieve_err.et
gcc -c -I.. -I./../lib -I../com_err/et -I/usr/local/BerkeleyDB.4.4/include  
-I/usr/local/ssl/include -I/usr/local/include -DHAVE_CONFIG_H  -g -O2  \
sieve_err.c
gcc -c -I.. -I./../lib -I../com_err/et -I/usr/local/BerkeleyDB.4.4/include  
-I/usr/local/ssl/include -I/usr/local/include -DHAVE_CONFIG_H  -g -O2  \
sieve.c
In file included from comparator.h:33,
 from sieve.y:38:
/usr/local/include/pcreposix.h:110: error: syntax error before "int"
/usr/local/include/pcreposix.h:111: error: syntax error before "int"
/usr/local/include/pcreposix.h:113: error: syntax error before "size_t"
/usr/local/include/pcreposix.h:114: error: syntax error before "void"
gmake[1]: *** [sieve.o] Error 1
gmake[1]: Leaving directory `/usr/local/src/cyrus/cyrus-imapd-2.3.10/sieve'
gmake: *** [all] Error 1

There are some definitions in pcreposix.h that depend on some #define's in 
pcre.h, but pcre.h isn't #include'd anywhere.  Here are some relevant lines 
from the configure step:

checking pcreposix.h usability... no
checking pcreposix.h presence... yes
configure: WARNING: pcreposix.h: present but cannot be compiled
configure: WARNING: pcreposix.h: check for missing prerequisite headers?
configure: WARNING: pcreposix.h: see the Autoconf documentation
configure: WARNING: pcreposix.h: section "Present But Cannot Be Compiled"
configure: WARNING: pcreposix.h: proceeding with the preprocessor's result
configure: WARNING: pcreposix.h: in the future, the compiler will take 
precedence
configure: WARNING: ## -- ##
configure: WARNING: ## Report this to the AC_PACKAGE_NAME lists.  ##
configure: WARNING: ## -- ##
checking for pcreposix.h... yes

So the configure script finds that pcreposix.h is unusable, but uses it anyway.

The build works fine on a different Solaris 9 system that doesn't have pcre 
installed.  Is there a fix for this problem, preferable one that doesn't 
involve hacking code?  Is there a config option to pretend pcreposix.h doesn't 
exist?

Thanks, Larry

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Scott M. Likens]

Adam Tauno Williams wrote:

3. Can't handle high load very well, in fact it handles load horribly.



I have a friend who works at a small shop who reports exactly the same
issue with Zimbra, s..ll...ooo.....

  


I'm glad to know that I wasn't alone, even though I was positive I was not.
3) ClamAV.  Do note how much email I said we dealt with a minute.  We 
didn't get a great deal of email.  Maybe 2000 email a day?  Not overly 
much.  However as the ClamAV database would grow, if you restarted 
ClamAV or Zimbra eventually it would take too long for ClamAV to start 
and would not listen on the port assigned and would make mail fail to 
deliver.  (Ouch huh?)



In defense of CLAMAV I can say that we run it on our SMTP server (not on
the IMAP or groupware server which seems like a bad idea).  It works
well and is pretty stable.  If your CLAMAV was causing you this problem
then Zimbra must have boloxed the setup or you just had a bad version.
  
  
It was a bad ClamAV version that they shipped.  I replaced it with a 
more current version (it was .80.7 or some really old version).  Then it 
was working again without it dying on it's own or anything.


Scott

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Scott M. Likens]

Ian,

The only problem with using clamav-filter (or something appropriate) as 
a milter, etc.  Was it did not fall into what Zimbra designed. Quite 
frankly I think it would have been a lot smoother with 
Sendmail+Milter+Clamav+whatever else they wanted.


However, that is not the direction they picked, either for licensing or 
whatever.


One thing with Zimbra, is you don't exactly get to pick what you want.  
They throw a ball of software at you, and expect you to work with it. 

I admit, if I didn't review their code as much, and try and see how it 
worked.  I would have been more oblivious and pleased with Zimbra maybe.


Scott

Ian G Batten wrote:

On 13 Nov 07, at 1335, Adam Tauno Williams wrote:

  
3. Can't handle high load very well, in fact it handles load  
horribly.
  

I have a friend who works at a small shop who reports exactly the same
issue with Zimbra, s..ll...ooo.....



3) ClamAV.  Do note how much email I said we dealt with a minute.  We
didn't get a great deal of email.  Maybe 2000 email a day?  Not  
overly

much.  However as the ClamAV database would grow, if you restarted
ClamAV or Zimbra eventually it would take too long for ClamAV to  
start

and would not listen on the port assigned and would make mail fail to
deliver.  (Ouch huh?)
  
In defense of CLAMAV I can say that we run it on our SMTP server  
(not on

the IMAP or groupware server which seems like a bad idea).  It works
well and is pretty stable.  If your CLAMAV was causing you this  
problem

then Zimbra must have boloxed the setup or you just had a bad version.



Clamav-milter works very well for sendmail shops, without any amavis  
involvement at all.  The slow startup bug is an artefact of one  
particular release: it now comes up in about 15 seconds.  Once it's  
running it's perfectly rapid enough to cope with our complete  
internal load.clamd-milter can do the parsing of archives,  
breaking up of MIME etc at least as well as amavisd.


If you don't have an equivalent to clamav-filter for your MTA of  
choice, then you need to make sure that you start clamd, and then  
pass the material to be scanned with clamdscan (note the d).  clamd  
will need to be running as a user that can read the temporary files,  
because the best way to use clamd is to pass filenames over the  
AF_UNIX domain socket.


We in fact run clamav-milter with its built-in clamd support, for  
reasons I can't offhand remember.  So we fire up clamd, then clamav- 
milter, then clamav-milter passes temporary files to clamd.


If you have to use amavisd, make sure you tell it to use clamdscan  
rather than clamscan.  The latter does indeed take 10 seconds to fire  
up.


clamd likes large pages, Solaris fans.

Our milter startup script: there is some local stuff in there.

#!/bin/sh

case "$1" in
start) mv /var/clamav/clamd.log /var/clamav/clamd.log.old
   LD_PRELOAD=mpss.so.1
   MPSSHEAP=4M
   MPSSSTACK=64K
export LD_PRELOAD MPSSHEAP MPSSSTACK
   newtask -p clam /usr/local/sbin/clamd
   attempt=1
   sleep=5
   while [ $attempt -lt 5 ]; do
  if /usr/local/bin/clamdscan /etc/termcap; then
 break
  else
 attempt=`expr $attempt + 1`
 sleep=`expr $sleep + 5`
 echo sleeping for $sleep seconds, attempt $attempt
 sleep $sleep
  fi
   done

#  [EMAIL PROTECTED] \
#  --postmaster-only \

   newtask -p milter /usr/local/sbin/clamav-milter \
   --dont-blacklist=`/usr/local/bin/fujitsuhosts` \
   --noreject \
   --dont-wait \
   --local \
   --outgoing \
   --quiet \
   --external \
   --pidfile=/var/clamav/milter.pid \
   --whitelist-file=/etc/mail/clamav-whitelist \
   inet:2010

   newtask -p spam /usr/perl5/bin/spamd -s local6 -u spamd -x -d  
--pidfile=/var/run/spamd.pid

   su spamd << \ZZZ
   newtask -p milter /usr/local/sbin/spamassassin_milter -p inet: 
2002 &

ZZZ
   newtask -p milter /usr/local/sbin/mailarchive -u archive -p  
inet:4001
   newtask -p milter /usr/local/sbin/spamtrap -u spamtrap -p inet: 
4000


   ;;
stop) for i in /var/clamav/milter.pid /var/run/spamd.pid; do
  test -f $i && kill `cat $i`
   done
   pkill -u spamd
   pkill -u clamav
   pkill -u archive
   pkill -u spamtrap
   ;;
esac






Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


!DSPAM:4739ffa181401346466276!


  



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Scott M. Likens]

I was using a Tomcat based CalDAV Server.  for awhile, I forget what 
it's name was.  However I have actually been using the CalDav server 
from apple lately (see url 1 in the the quote).  It's been quite pleasant. 

I am sure there is better, there is worse.  They have a great love of 
python that is for sure. 


Scott

Rudy Gevaert wrote:

Scott M. Likens wrote:
  

Have you ever looked at some of the CalDAV Servers out there?

I'll save you some time,

http://trac.macosforge.org/projects/calendarserver

http://rscds.sourceforge.net/

As well as,

http://sourceforge.net/projects/modcaldav/

Truthfully, you don't need Cyrus to support a Calendar.  Because in all 
honesty, it's unrelated to mail.  If you use Kerberos, LDAP, AD, MySQL 
for Authentication.  Take a look at one of those, tie in that 
authentication and you're done.  Then depending on which one you choose, 
you can have users share their calendars or not.
... There is enough F/OSS out there to emulate everything you can get 
with Exchange, and/or any other 'Enterprise' Mail System.  No it's not 
as seamless as Exchange, but it works just fine and it's an open 
standard.  You'll find lots more CalDav Servers, and software in the 
next 6months to a year.



Well I'm impressed, I didn't know it would already be possible.  I'll 
keep an eye on those projects.  When I have some time I'll give them a 
closer look!


Are you running any calendar server?

Rudy


  



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Calendars && Cyrus

[Adam Tauno Williams]

> > > Server is entirely Open Source;  Outlook plugin (MAPI provider) is
> > > commercial.
> > > Works very well with Cyrus; which is its intended IMAP server.
> > Speaking of calendars,... What about Mozilla Sunbird/Lightning?
> > They can speak to WebDAV enabled ICS calendar format. With a special
> > extension, Google Calendar Provider (freely available from
> > addons.mozilla.org), they can talk to Google Calendar.
> > Has anyone seen a calendar server that will work with Cyrus and Sunbird?
> We are using www.egroupware.org as Mail/Calendar/Contact-Frontend and it
> has rudimentary ICS over HTTP support. I spoke to the main developer and
> he says that ICS over HTTP is basically a very stupid idea (which I

Yep,  ICS over HTTP is not a groupware solution.  ICS is just a dumb
"calendar" file,  it doesn't deal with discrete events so trying to
share an ICS file results in a mess.

> concur). Other than the occasional glitch, it's working fine with
> Sunbird/Lightning.

OpenGroupware works with Sunbird, with some glitches.  CalDAV support is
being hammered out.

> Anyway, progress is made on getting GroupDAV (or was it CalDAV?) working
> with eGroupware, but it is still not there, though.


GroupDAV for contacts on OpenGroupware works pretty well.



signature.asc
Description: This is a digitally signed message part

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Adam Tauno Williams]

> > This describes exactly the point of view of administrative staff.  They
> > live in Microsoft Office, and they need a server to support it.  That is
> > the assignment given.
> > I was looking at Open-Xchange on the web .
> > The server provides webmail and MAPI interfaces.  The "Hosting Edition"
> > (and maybe the others, it is not clear) can talk to Cyrus and includes
> > ACL support.
> > (We're still running both Exchange for admin staff and Cyrus for the
> > much larger university community of faculty and students.)
> Perhaps things have improved since my last foray into this area, but
> I've yet to find an "outlook connector" that was stable and didn't
> corrupt the hell out of the users PST. And at least with the Kolab

Never had a corrupt PST using ZideLook (the Outlook plugin for
OpenGroupware).  Make sue they provide a *real* MAPI provider for
outlook and not some background sync thingy (as several Open Source
Outlook connectors do or at least did).

> groupware product when I reviewed it, they made several architecture
> decisions that completely killed performance and then they went on to
> trivialize my concerns. Storing calendar objects as binary attachments
> in an IMAP store, effectively making them unsearchable, meant that every
> calendar access had to pull the entire store down, process it and
> display it. Not cool.

Yep,  the Kolab architecture goes in the "what are you, nuts?" column.


signature.asc
Description: This is a digitally signed message part

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Gunnar Wrobel]

Zachariah Mully <[EMAIL PROTECTED]> writes:

> On Tue, 2007-11-13 at 09:10 -0500, Joseph Brennan wrote:
>> Ian G Batten <[EMAIL PROTECTED]> wrote:
>> > However, people don't want calendaring, they want Outlook.
>> 
>> 
>> This describes exactly the point of view of administrative staff.  They
>> live in Microsoft Office, and they need a server to support it.  That is
>> the assignment given.
>> 
>> I was looking at Open-Xchange on the web .
>> The server provides webmail and MAPI interfaces.  The "Hosting Edition"
>> (and maybe the others, it is not clear) can talk to Cyrus and includes
>> ACL support.
>> 
>> (We're still running both Exchange for admin staff and Cyrus for the
>> much larger university community of faculty and students.)
>> 
>> Joseph Brennan
>> Lead Email Systems Engineer
>> Columbia University Information Technology
>> 
>
> Perhaps things have improved since my last foray into this area, but
> I've yet to find an "outlook connector" that was stable and didn't
> corrupt the hell out of the users PST. And at least with the Kolab
> groupware product when I reviewed it, they made several architecture
> decisions that completely killed performance and then they went on to
> trivialize my concerns. Storing calendar objects as binary attachments
> in an IMAP store, effectively making them unsearchable, meant that every
> calendar access had to pull the entire store down, process it and
> display it. Not cool.

I don't know of ANY Kolab compatible client that would do that. Which
client are you referring to?

Cheers,

Gunnar

>
> Z
>
>
> -- 
> Zachariah Mully
> Director, Systems and Networks
> SmartBrief, Inc.
> [EMAIL PROTECTED]
> p: 202-737-5500 x226
> c: 202-422-8780
> f: 202-737-7577
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Zachariah Mully]

On Tue, 2007-11-13 at 18:27 +0100, Gunnar Wrobel wrote:

> I don't know of ANY Kolab compatible client that would do that. Which
> client are you referring to?
> 
> Cheers,
> 
> Gunnar

This was quite some time ago, around the first release of their 2.x
product IIRC. It was the outlook connector that would save all calendar
entries from outlook to the IMAP server as binary messages, whereas the
Horde web front end saved the entries in xml, so Cyrus was at least able
to search them. So if you tried to access through Horde a calendar from
an Outlook user, it was basically inoperable, as the webclient had to
download the entire calendar mailbox, parse the binaries, sort them,
then display them. Useless for anything more than a hundred entries or
so...

-- 
Zachariah Mully
Director, Systems and Networks
SmartBrief, Inc.
[EMAIL PROTECTED]
p: 202-737-5500 x226
c: 202-422-8780
f: 202-737-7577


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Calendars && Cyrus

[Ulrich Spoerlein]

On Tue, 13.11.2007 at 06:56:21 -0800, Nikola Milutinovic wrote:
> > We use OpenGroupware - http://www.opengroupware.org
> >
> > Server is entirely Open Source;  Outlook plugin (MAPI provider) is
> > commercial.
> >
> > Works very well with Cyrus; which is its intended IMAP server.
> 
> 
> Speaking of calendars,... What about Mozilla Sunbird/Lightning?
> 
> They can speak to WebDAV enabled ICS calendar format. With a special
> extension, Google Calendar Provider (freely available from
> addons.mozilla.org), they can talk to Google Calendar.
> 
> Has anyone seen a calendar server that will work with Cyrus and Sunbird?

We are using www.egroupware.org as Mail/Calendar/Contact-Frontend and it
has rudimentary ICS over HTTP support. I spoke to the main developer and
he says that ICS over HTTP is basically a very stupid idea (which I
concur). Other than the occasional glitch, it's working fine with
Sunbird/Lightning.

Anyway, progress is made on getting GroupDAV (or was it CalDAV?) working
with eGroupware, but it is still not there, though.

> Sunbird is really great and Lightning makes me NOT want to ever touch Outlook.

I prefer the KDE PIM suite Kontact over Mozilla's products any day. It
has tons of bugs, but also some very cool features which I would never
want to miss again.

Cheers,
Ulrich Spoerlein
-- 
It is better to remain silent and be thought a fool,
than to speak, and remove all doubt.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[David Chait]

Darin Perusich wrote:
> Have you looked at Funambol for your OSS push email?
No I hadn't heard of it, but thanks for bringing it to my attention, I 
will give it a shot.

Cheers,
David

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Zachariah Mully]

On Tue, 2007-11-13 at 09:10 -0500, Joseph Brennan wrote:
> Ian G Batten <[EMAIL PROTECTED]> wrote:
> > However, people don't want calendaring, they want Outlook.
> 
> 
> This describes exactly the point of view of administrative staff.  They
> live in Microsoft Office, and they need a server to support it.  That is
> the assignment given.
> 
> I was looking at Open-Xchange on the web .
> The server provides webmail and MAPI interfaces.  The "Hosting Edition"
> (and maybe the others, it is not clear) can talk to Cyrus and includes
> ACL support.
> 
> (We're still running both Exchange for admin staff and Cyrus for the
> much larger university community of faculty and students.)
> 
> Joseph Brennan
> Lead Email Systems Engineer
> Columbia University Information Technology
> 

Perhaps things have improved since my last foray into this area, but
I've yet to find an "outlook connector" that was stable and didn't
corrupt the hell out of the users PST. And at least with the Kolab
groupware product when I reviewed it, they made several architecture
decisions that completely killed performance and then they went on to
trivialize my concerns. Storing calendar objects as binary attachments
in an IMAP store, effectively making them unsearchable, meant that every
calendar access had to pull the entire store down, process it and
display it. Not cool.

Z


-- 
Zachariah Mully
Director, Systems and Networks
SmartBrief, Inc.
[EMAIL PROTECTED]
p: 202-737-5500 x226
c: 202-422-8780
f: 202-737-7577


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Timed Actions in Sieve

[Listaccount]

Zitat von Ian G Batten <[EMAIL PROTECTED]>:

> We've been having a chat about how useful it would be to have timed
> actions in sieve: so that a vacation message could be set up for a
> duration which would automatically revert, so that a forwarding could
> be set up for the duration of a short-term project, etc, etc.  The
> naive way is to add support to the sieve interface of choice (the
> squirrelmail plugin in our case) to handle deferred actions, but I
> can think of all sorts of security problems with that.  Another would
> be a means to auto-generate regexps to match on Date: headers, but
> that's really tacky.  The full solution would be to have the current
> time available in sieve scripts, to then match on.  Has anyone else
> thought about this area?
>
> ian

There seams to be some work is this area already :
http://tools.ietf.org/html/draft-freed-sieve-date-index-07

Anyone aware of the status of this project??

Regards

Andreas



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Timed Actions in Sieve

[Rob Banz]

I had looked into this before, but really haven't had a chance to  
follow up on it.

There is an draft for a sieve date/time extension, which allows you to  
use time comparisons in conditionals -- for example, the vacation  
message defined by a date range.  This looks like its the current  
version of the draft:

http://tools.ietf.org/html/draft-freed-sieve-date-index-07


On Nov 13, 2007, at 06:24, Ian G Batten wrote:

> We've been having a chat about how useful it would be to have timed
> actions in sieve: so that a vacation message could be set up for a
> duration which would automatically revert, so that a forwarding could
> be set up for the duration of a short-term project, etc, etc.  The
> naive way is to add support to the sieve interface of choice (the
> squirrelmail plugin in our case) to handle deferred actions, but I
> can think of all sorts of security problems with that.  Another would
> be a means to auto-generate regexps to match on Date: headers, but
> that's really tacky.  The full solution would be to have the current
> time available in sieve scripts, to then match on.  Has anyone else
> thought about this area?
>
> ian
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Darin Perusich]

Have you looked at Funambol for your OSS push email?

David Chait wrote:
> One key piece of functionality that seems to be missing from every OSS 
> solution mentioned thus far is mobile device push support (Activesync), 
> this is not to be underestimated as it is for us, a key reason why we 
> are ultimately being forced to adopt Exchange en-mass and abandon our 
> current Cyrus infrastructure.
> 
> -David
> 
> Joseph Brennan wrote:
>> Ian G Batten <[EMAIL PROTECTED]> wrote:
>>   
>>> However, people don't want calendaring, they want Outlook.
>>> 
>>
>> This describes exactly the point of view of administrative staff.  They
>> live in Microsoft Office, and they need a server to support it.  That is
>> the assignment given.
>>
>> I was looking at Open-Xchange on the web .
>> The server provides webmail and MAPI interfaces.  The "Hosting Edition"
>> (and maybe the others, it is not clear) can talk to Cyrus and includes
>> ACL support.
>>
>> (We're still running both Exchange for admin staff and Cyrus for the
>> much larger university community of faculty and students.)
>>
>> Joseph Brennan
>> Lead Email Systems Engineer
>> Columbia University Information Technology
>>
>>
>> 
>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>   
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

-- 
Darin Perusich
Unix Systems Administrator
Cognigen Corporation
395 Youngs Rd.
Williamsville, NY 14221
Phone: 716-633-3463
Email: [EMAIL PROTECTED]

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Adam Tauno Williams]

> One key piece of functionality that seems to be missing from every OSS 
> solution mentioned thus far is mobile device push support (Activesync), 
> this is not to be underestimated as it is for us, a key reason why we 
> are ultimately being forced to adopt Exchange en-mass and abandon our 
> current Cyrus infrastructure.

You haven't tried Funambol?  It supports over-the-air mobile devices
quite well.  You can sync IMAP natively, and they have a push
infrastructure.  You can sync with various groupware server's using
plugins.  For instance the GroupDAV plugin works for OpenGroupware,
Citadel, or USA.NET

Funambol:
http://www.funambol.com/opensource/

GroupDAV plugin:
http://bionicmessage.net/?q=node/18
http://bionicmessage.net/files/GroupWare_doc5.pdf


signature.asc
Description: This is a digitally signed message part

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Timed Actions in Sieve

[Juan Pablo Baudoin]

I had the same problem and solved it using a bash programan that 
generates the necesary sieve scripts. This is run with cron every day at 
night It checks for configuration request on a directory, process those 
files and creates sieve scripts for vacation activation and for 
deactivation, finaly it uses sieveshell to upload the scripts. At this 
moment this scripts are in their first version I hope to improve them 
but It would be better to have this functionality in sieve.

bye

Listaccount wrote:
> Zitat von Ian G Batten <[EMAIL PROTECTED]>:
>
>   
>> We've been having a chat about how useful it would be to have timed
>> actions in sieve: so that a vacation message could be set up for a
>> duration which would automatically revert, so that a forwarding could
>> be set up for the duration of a short-term project, etc, etc.  The
>> naive way is to add support to the sieve interface of choice (the
>> squirrelmail plugin in our case) to handle deferred actions, but I
>> can think of all sorts of security problems with that.  Another would
>> be a means to auto-generate regexps to match on Date: headers, but
>> that's really tacky.  The full solution would be to have the current
>> time available in sieve scripts, to then match on.  Has anyone else
>> thought about this area?
>>
>> ian
>> 
>
> This would be for sure a really useful extension for sieve. The Horde  
> Project (www.horde.org) has the same problem for the sieve web  
> interface "Ingo" where it is up until now solved by regexp matching  
> headers. But as headers can be forged and unreliable (wrong system  
> clock at sending site) a sieve extension for access the local system  
> time/timezone and compare against it would be preferable.
>
> Regards
>
> Andreas
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>   


-- 
Juan Pablo Baudoin
Gerente Nacional de sistemas
Kieffer & Asociados
Tel: (591) 2 243 3434
Cel: (591) 720 01485 


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Just in case it is of general interest: ZFS mirroring was the culprit in our case

[Vincent Fox]

Can you expand on this, like a LOT?

I recall a while ago you brought up some performance issues and
said you had found hacks for them.  Were those issues actually unresolved
or are you talking about something else?  I don't see any recent posts by
you about problems with your Cyrus install.

I'm struggling to see the mechanism by which mirroring creates a problem.
Were you resilvering at the time?

Pascal Gienger wrote:
> Our latency problems went away like a miracle when we detached one half of 
> the mirror (so it is no more a mirror).
>
> Read-Rates are doubled (not per device, the total read rate!), latency is 
> cut off. No more latency problems.
>
> When attaching the volume again, resilvering puts the system to a halt - 
> reads and writes do block for seconds (!). We will go on directly with Sun 
> to solve the problem. Their "lowest I/O-priority to resilver disks" does 
> not seem to be effective. It really blocks the kernel and you end up with 
> thousand locks in "zfs_zget".
>
> We have two SAN volumes in different buildings which are NOT the 
> bottleneck, tests show it.
>
> Pascal
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>   


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Joseph Brennan]

David Chait <[EMAIL PROTECTED]> wrote:

> One key piece of functionality that seems to be missing from every OSS
> solution mentioned thus far is mobile device push support (Activesync),
> this is not to be underestimated as it is for us, a key reason why we are
> ultimately being forced to adopt Exchange en-mass and abandon our current
> Cyrus infrastructure.

-
Right.  This is a necessity for us too, if we are to integrate the
Exchange and Cyrus systems.  One of the interesting things about
Open-Xchange is support for push via SyncML.  There is a list of PDAs
that it works with.

The devices we support now need the intermediaries of either GoodLink
or BES attached to Exchange, and if I understand it right (I might not),
they use a MAPI connection to find out when there is new mail.  If so
it appears they would work also with systems like Open-Xchange that
offer MAPI connections.

Joseph Brennan
Lead Email Systems Engineer
Columbia University Information Technology


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Timed Actions in Sieve

[Listaccount]

Zitat von Ian G Batten <[EMAIL PROTECTED]>:

> We've been having a chat about how useful it would be to have timed
> actions in sieve: so that a vacation message could be set up for a
> duration which would automatically revert, so that a forwarding could
> be set up for the duration of a short-term project, etc, etc.  The
> naive way is to add support to the sieve interface of choice (the
> squirrelmail plugin in our case) to handle deferred actions, but I
> can think of all sorts of security problems with that.  Another would
> be a means to auto-generate regexps to match on Date: headers, but
> that's really tacky.  The full solution would be to have the current
> time available in sieve scripts, to then match on.  Has anyone else
> thought about this area?
>
> ian

This would be for sure a really useful extension for sieve. The Horde  
Project (www.horde.org) has the same problem for the sieve web  
interface "Ingo" where it is up until now solved by regexp matching  
headers. But as headers can be forged and unreliable (wrong system  
clock at sending site) a sieve extension for access the local system  
time/timezone and compare against it would be preferable.

Regards

Andreas


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Just in case it is of general interest: ZFS mirroring was the culprit in our case

[Rob Banz]

...though, we have seen super-greedyness of ZFS when resilvering. ;)

On Nov 13, 2007, at 09:17, Dale Ghent wrote:

>
> Interesting. What's your kernel patch level?
>
> We're running on 125101-10 with the exact same configuration as you
> (mirrored to two arrays, in separate buildings even) and haven't seen
> this problem.
>
> /dale
>
>
> On Nov 13, 2007, at 1:23 AM, Pascal Gienger wrote:
>
>> Our latency problems went away like a miracle when we detached one
>> half of
>> the mirror (so it is no more a mirror).
>>
>> Read-Rates are doubled (not per device, the total read rate!),
>> latency is
>> cut off. No more latency problems.
>>
>> When attaching the volume again, resilvering puts the system to a
>> halt -
>> reads and writes do block for seconds (!). We will go on directly
>> with Sun
>> to solve the problem. Their "lowest I/O-priority to resilver disks"
>> does
>> not seem to be effective. It really blocks the kernel and you end up
>> with
>> thousand locks in "zfs_zget".
>>
>> We have two SAN volumes in different buildings which are NOT the
>> bottleneck, tests show it.
>>
>> Pascal
>> 
>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>
>
> --
> Dale Ghent
> Specialist, Storage and UNIX Systems
> UMBC - Office of Information Technology
> ECS 201 - x51705
>
>
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: FW: Retrieving a banned e-mail

[Adam Tauno Williams]

> We have a postfix+cyrus+Spamassassin setup and all the virus or Banned
> or spam e-mails will move to a folder on /var/virusmails.\
> I have a falseposative mail moved due to a banned attachment name.
> Could you please help me to retrieve these e-mails?

If it really is dumped into a folder in /var/virusmails then it isn't in
the Cyrus mail store (seems like a bad idea) and thus can't be accessed
via a mail client.  Why not reconfigure to but the junked e-mail into a
Junk folder?

(Or, IMHO, stop wasting time with overly complex SPAM solutions like
Spamassasin).

-- 
Adam Tauno Williams, Network & Systems Administrator
Consultant - http://www.whitemiceconsulting.com
Developer - http://www.opengroupware.org


signature.asc
Description: This is a digitally signed message part

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Sebastian Hagedorn]

--On 13. November 2007 09:10:08 -0500 Joseph Brennan <[EMAIL PROTECTED]> 
wrote:



Ian G Batten <[EMAIL PROTECTED]> wrote:

However, people don't want calendaring, they want Outlook.



This describes exactly the point of view of administrative staff.


Fortunately that's not true of everyone.


They
live in Microsoft Office, and they need a server to support it.  That is
the assignment given.


If they don't yet have Exchange there's still hope ;-)


I was looking at Open-Xchange on the web .
The server provides webmail and MAPI interfaces.  The "Hosting Edition"
(and maybe the others, it is not clear) can talk to Cyrus and includes
ACL support.

(We're still running both Exchange for admin staff and Cyrus for the
much larger university community of faculty and students.)


OK, to wean them off Exchange is probably impossible, but for new users 
Open-Xchange is (one) possibility. It so happens that they (OX) were here 
today because we (Cologne University) are considering the "Hosting 
Edition". The beauty of it is that it works as a pretty seamless add-on to 
our existing Cyrus infrastructure. We are now waiting for an offer.

--
.:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:.
Zentrum für angewandte Informatik - Universitätsweiter Service RRZK
.:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:.
  .:.:.:.Skype: shagedorn.:.:.:.

pgpLBfbZJahzt.pgp
Description: PGP signature

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Ian G Batten]

On 13 Nov 07, at 1335, Adam Tauno Williams wrote:

>> 3. Can't handle high load very well, in fact it handles load  
>> horribly.
>
> I have a friend who works at a small shop who reports exactly the same
> issue with Zimbra, s..ll...ooo.....
>
>> 3) ClamAV.  Do note how much email I said we dealt with a minute.  We
>> didn't get a great deal of email.  Maybe 2000 email a day?  Not  
>> overly
>> much.  However as the ClamAV database would grow, if you restarted
>> ClamAV or Zimbra eventually it would take too long for ClamAV to  
>> start
>> and would not listen on the port assigned and would make mail fail to
>> deliver.  (Ouch huh?)
>
> In defense of CLAMAV I can say that we run it on our SMTP server  
> (not on
> the IMAP or groupware server which seems like a bad idea).  It works
> well and is pretty stable.  If your CLAMAV was causing you this  
> problem
> then Zimbra must have boloxed the setup or you just had a bad version.

Clamav-milter works very well for sendmail shops, without any amavis  
involvement at all.  The slow startup bug is an artefact of one  
particular release: it now comes up in about 15 seconds.  Once it's  
running it's perfectly rapid enough to cope with our complete  
internal load.clamd-milter can do the parsing of archives,  
breaking up of MIME etc at least as well as amavisd.

If you don't have an equivalent to clamav-filter for your MTA of  
choice, then you need to make sure that you start clamd, and then  
pass the material to be scanned with clamdscan (note the d).  clamd  
will need to be running as a user that can read the temporary files,  
because the best way to use clamd is to pass filenames over the  
AF_UNIX domain socket.

We in fact run clamav-milter with its built-in clamd support, for  
reasons I can't offhand remember.  So we fire up clamd, then clamav- 
milter, then clamav-milter passes temporary files to clamd.

If you have to use amavisd, make sure you tell it to use clamdscan  
rather than clamscan.  The latter does indeed take 10 seconds to fire  
up.

clamd likes large pages, Solaris fans.

Our milter startup script: there is some local stuff in there.

#!/bin/sh

case "$1" in
start) mv /var/clamav/clamd.log /var/clamav/clamd.log.old
   LD_PRELOAD=mpss.so.1
   MPSSHEAP=4M
   MPSSSTACK=64K
export LD_PRELOAD MPSSHEAP MPSSSTACK
   newtask -p clam /usr/local/sbin/clamd
   attempt=1
   sleep=5
   while [ $attempt -lt 5 ]; do
  if /usr/local/bin/clamdscan /etc/termcap; then
 break
  else
 attempt=`expr $attempt + 1`
 sleep=`expr $sleep + 5`
 echo sleeping for $sleep seconds, attempt $attempt
 sleep $sleep
  fi
   done

#  [EMAIL PROTECTED] \
#  --postmaster-only \

   newtask -p milter /usr/local/sbin/clamav-milter \
   --dont-blacklist=`/usr/local/bin/fujitsuhosts` \
   --noreject \
   --dont-wait \
   --local \
   --outgoing \
   --quiet \
   --external \
   --pidfile=/var/clamav/milter.pid \
   --whitelist-file=/etc/mail/clamav-whitelist \
   inet:2010

   newtask -p spam /usr/perl5/bin/spamd -s local6 -u spamd -x -d  
--pidfile=/var/run/spamd.pid
   su spamd << \ZZZ
   newtask -p milter /usr/local/sbin/spamassassin_milter -p inet: 
2002 &
ZZZ
   newtask -p milter /usr/local/sbin/mailarchive -u archive -p  
inet:4001
   newtask -p milter /usr/local/sbin/spamtrap -u spamtrap -p inet: 
4000

   ;;
stop) for i in /var/clamav/milter.pid /var/run/spamd.pid; do
  test -f $i && kill `cat $i`
   done
   pkill -u spamd
   pkill -u clamav
   pkill -u archive
   pkill -u spamtrap
   ;;
esac






Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[David Chait]

One key piece of functionality that seems to be missing from every OSS 
solution mentioned thus far is mobile device push support (Activesync), 
this is not to be underestimated as it is for us, a key reason why we 
are ultimately being forced to adopt Exchange en-mass and abandon our 
current Cyrus infrastructure.

-David

Joseph Brennan wrote:
> Ian G Batten <[EMAIL PROTECTED]> wrote:
>   
>> However, people don't want calendaring, they want Outlook.
>> 
>
>
> This describes exactly the point of view of administrative staff.  They
> live in Microsoft Office, and they need a server to support it.  That is
> the assignment given.
>
> I was looking at Open-Xchange on the web .
> The server provides webmail and MAPI interfaces.  The "Hosting Edition"
> (and maybe the others, it is not clear) can talk to Cyrus and includes
> ACL support.
>
> (We're still running both Exchange for admin staff and Cyrus for the
> much larger university community of faculty and students.)
>
> Joseph Brennan
> Lead Email Systems Engineer
> Columbia University Information Technology
>
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>   

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Calendars && Cyrus

[Nikola Milutinovic]

> We use OpenGroupware - http://www.opengroupware.org
>
> Server is entirely Open Source;  Outlook plugin (MAPI provider) is
> commercial.
>
> Works very well with Cyrus; which is its intended IMAP server.


Speaking of calendars,... What about Mozilla Sunbird/Lightning?

They can speak to WebDAV enabled ICS calendar format. With a special extension, 
Google Calendar Provider (freely available from addons.mozilla.org), they can 
talk to Google Calendar.

Has anyone seen a calendar server that will work with Cyrus and Sunbird?

Sunbird is really great and Lightning makes me NOT want to ever touch Outlook.

Nix.



  

Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: problems with virtual domains and sasl under Debian 4

[Alain Spineux]

On Nov 13, 2007 11:06 AM, Falko Zurell <[EMAIL PROTECTED]> wrote:
> Hello List_members,
>
> I'm using a Debian Linux 4 and the provided Cyrus (2.2.13) and SASL
> packages from Debian. I've configured my Postfix MTA to use virtual
> domains with the SASL library as authentication base. Postfix works
> fine for me but Cyrus makes some trouble.
>
> I can't authenticate on the Cyrus IMAPD with my virtual domain name.
> When I try to login with "[EMAIL PROTECTED]" on cyrus imapd
> authentication fails. When I add a user to the saslpasswd file with
> the default domain (eq. localhost) I can authenticate on IMAPD.
>
> my imapd.conf ist this:
>
> configdirectory: /opt/data/cyrus/lib/
> defaultpartition: default
> partition-default: /opt/data/cyrus/mail
> partition-news: /opt/data/cyrus/news
> newsspool: /var/spool/news
> altnamespace: no
> unixhierarchysep: yes
> reject8bit: yes
> lmtp_downcase_rcpt: yes
> admins: cyrus
> allowanonymouslogin: no
> popminpoll: 1
> autocreatequota: 0
> umask: 077
> sieveusehomedir: false
> sievedir: /opt/data/cyrus/sieve
> hashimapspool: true
> allowplaintext: no
> sasl_mech_list: PLAIN
> loginrealms: domain.com

Can you try to list your other domains here

> virtdomains: on
> defaultdomain: domain.com
> sasl_pwcheck_method: saslauthd
> sasl_auxprop_plugin: sasldb
> sasl_auto_transition: yes
> tls_cert_file: /etc/ssl/certs/post.domain.com.cert.pem
> tls_key_file: /etc/ssl/private/post.domain.com.key.pwless.pem
> tls_ca_path: /etc/ssl/certs
> tls_session_timeout: 1440
> tls_cipher_list: TLSv1+HIGH:!aNULL:@STRENGTH
> lmtpsocket: /var/run/cyrus/socket/lmtp
> idlemethod: poll
> idlesocket: /var/run/cyrus/socket/idle
> notifysocket: /var/run/cyrus/socket/notify
> syslog_prefix: cyrus
>
> Can someone give me a hint where to look? I now this must work because
> I've got it running on a SuSE Linux box with almost the same
> configuration. I can't find the mistake by myself at the moment.
>
>
> Thanks and kind regards
>
> Falko Zurell
>
> /i-d media AG
>
> Ohlauer Straße 43
> D-10999 Berlin
>
> www.idmedia.com
>
> ___
> Board: Regine Haschka-Helmer (CEO), Franz Klose (CFO)
> Chairman of the Supervisory Board: Christian A. Hufnagl
> Headquarters: Berlin, Place of Jurisdiction: Berlin HRB 76342
>
> ___
> This message may contain confidential information and must not be
> copied, disclosed or used by anybody other than the intended
> recipient. If you have received this message in error, please notify
> us immediately by reply e-mail and delete all records of the message
> from your computer. The views represented in this message are solely
> those of the author. Neither the author nor I-D Media AG accepts any
> liability for the contents of this message.
>
>
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>



-- 
Alain Spineux
aspineux gmail com
May the sources be with you

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Just in case it is of general interest: ZFS mirroring was the culprit in our case

[Dale Ghent]

Interesting. What's your kernel patch level?

We're running on 125101-10 with the exact same configuration as you  
(mirrored to two arrays, in separate buildings even) and haven't seen  
this problem.

/dale


On Nov 13, 2007, at 1:23 AM, Pascal Gienger wrote:

> Our latency problems went away like a miracle when we detached one  
> half of
> the mirror (so it is no more a mirror).
>
> Read-Rates are doubled (not per device, the total read rate!),  
> latency is
> cut off. No more latency problems.
>
> When attaching the volume again, resilvering puts the system to a  
> halt -
> reads and writes do block for seconds (!). We will go on directly  
> with Sun
> to solve the problem. Their "lowest I/O-priority to resilver disks"  
> does
> not seem to be effective. It really blocks the kernel and you end up  
> with
> thousand locks in "zfs_zget".
>
> We have two SAN volumes in different buildings which are NOT the
> bottleneck, tests show it.
>
> Pascal
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>

--
Dale Ghent
Specialist, Storage and UNIX Systems
UMBC - Office of Information Technology
ECS 201 - x51705




Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Michael D. Sofka]

On Tuesday 13 November 2007 03:36:55 am Rudy Gevaert wrote:
> Scott M. Likens wrote:
> > Have you ever looked at some of the CalDAV Servers out there?
> >
> > I'll save you some time,
> >
> > http://trac.macosforge.org/projects/calendarserver
> >
> > http://rscds.sourceforge.net/
> >
> > As well as,
> >
> > http://sourceforge.net/projects/modcaldav/

And, Bedework: http://www.bedework.org/

Mike

-- 
Michael D. Sofka   [EMAIL PROTECTED]
C&MT Sr. Systems Programmer,   Email, TeX, Epistemology
Rensselaer Polytechnic Institute, Troy, NY.  http://www.rpi.edu/~sofkam/

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Joseph Brennan]

Ian G Batten <[EMAIL PROTECTED]> wrote:
> However, people don't want calendaring, they want Outlook.


This describes exactly the point of view of administrative staff.  They
live in Microsoft Office, and they need a server to support it.  That is
the assignment given.

I was looking at Open-Xchange on the web .
The server provides webmail and MAPI interfaces.  The "Hosting Edition"
(and maybe the others, it is not clear) can talk to Cyrus and includes
ACL support.

(We're still running both Exchange for admin staff and Cyrus for the
much larger university community of faculty and students.)

Joseph Brennan
Lead Email Systems Engineer
Columbia University Information Technology



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Deleting top-level mailbox with 'delete_mode: delayed'

[David Carter]

On Tue, 13 Nov 2007, Bron Gondwana wrote:

> I have "delete_mode: immediate" on the replica and "delete_mode: 
> delayed" on the master.

sync_server doesn't pay any attention to delete_mode, so the option
shouldn't have any effect on the replica.

-- 
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[David Carter]

On Tue, 13 Nov 2007, Bron Gondwana wrote:

> If you're planning to lift a consistent copy of a .index file, you need
> to lock it for the duration of reading it (read lock at least).

mailbox_lock_index() blocks flag updates (but this doesn't seem to be 
something that imapd worries about when FETCHing data). You don't need to 
worry about expunge or append events once the mailbox is open.

> But since I would like a consistent snapshot of the mailbox state, I 
> lock the cyrus.header and then the cyrus.index and then (if it's there) 
> the cyrus.expunge.  That means no sneaky process could (for example) 
> delete the mailbox and create another one with the same name while I was 
> busy downloading the last file - giving me totally bogus data.

chdir() into the mailbox data directory: with delayed delete and fast 
rename it shouldn't matter if the mailbox is replaced under your feet. 
That's the way replication worked on my 2.1 systems, prior to split-meta.

(Locking isn't a big deal, but safe concurrent access is always nice).

-- 
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Replication: problems with synctest

[Simon Matter]

>> On Nov 13, 2007 8:08 AM, Rich Wales <[EMAIL PROTECTED]> wrote:
>>> OK, so I decided to try what I described earlier (replication in both
>>> directions, with different users using different master servers) . . .
>>> .
>>>
>>> But now I'm running into an authentication problem.  One of my servers
>>> (my original replica) simply refuses to authenticate to the other one
>>> (my original master).
>>>
>>> I've double-checked the user name and password, and I know it's in the
>>> sasldb2.db file on the master, but authentication just will not work.
>>>
>>> I tried synctest to each server, and I can't connect to either server
>>> using synctest.  Examples ("whodunit" is my original master; "flipflop"
>>> is my original replica; and I've obscured the real authentication
>>> strings in the AUTHENTICATE commands):
>>>
>>> % synctest -u admin -m plain whodunit
>>> S: * SASL NTLM LOGIN PLAIN GSSAPI OTP DIGEST-MD5 CRAM-MD5
>>> S: * STARTTLS
>>> S: * OK whodunit.richw.org Cyrus sync server v2.3.9
>>> Please enter your password:
>>> C: AUTHENTICATE PLAIN ***
>>> S: NO user not found
>>> Authentication failed. generic failure
>>> Security strength factor: 0
>>> C: EXIT
>>> Connection closed.
>>>
>>> % synctest -u admin -m plain flipflop
>>> S: * SASL CRAM-MD5 DIGEST-MD5 GSSAPI LOGIN PLAIN NTLM
>>> S: * STARTTLS
>>> S: * OK flipflop Cyrus sync server v2.3.9
>>> Please enter your password:
>>> C: AUTHENTICATE PLAIN ***
>>> S: NO authentication failure
>>> Authentication failed. generic failure
>>> Security strength factor: 0
>>> C: EXIT
>>> Connection closed.
>>>
>>> What's especially weird is the first one (whodunit), which gave a
>>> "user not found" error, even though there IS an account named "admin"
>>> in the sasldb2.db on that machine.
>>>
>>> Any ideas what I might be doing wrong here?
>>
>> Do you have virtdomains enable ?
>> Did you setup defaultdomain ?
>> Did you define user [EMAIL PROTECTED] ?
>> Is $defaultdomain in loginrealms ?
>>
>
> No, my testconfig is very simple:
>
> configdirectory: /var/lib/imap
> partition-default: /var/spool/imap
> admins: cyrus
> sievedir: /var/lib/imap/sieve
> sendmail: /usr/sbin/sendmail
> hashimapspool: true
> sasl_pwcheck_method: saslauthd
> sasl_mech_list: PLAIN
> tls_cert_file: /usr/share/ssl/certs/cyrus-imapd.pem
> tls_key_file: /usr/share/ssl/certs/cyrus-imapd.pem
> tls_ca_file: /usr/share/ssl/certs/ca-bundle.crt
> delete_mode: delayed

Ops, that was the wrong thread, sorry!

Simon


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Adam Tauno Williams]

> 3. Can't handle high load very well, in fact it handles load horribly.

I have a friend who works at a small shop who reports exactly the same
issue with Zimbra, s..ll...ooo.....

> 3) ClamAV.  Do note how much email I said we dealt with a minute.  We 
> didn't get a great deal of email.  Maybe 2000 email a day?  Not overly 
> much.  However as the ClamAV database would grow, if you restarted 
> ClamAV or Zimbra eventually it would take too long for ClamAV to start 
> and would not listen on the port assigned and would make mail fail to 
> deliver.  (Ouch huh?)

In defense of CLAMAV I can say that we run it on our SMTP server (not on
the IMAP or groupware server which seems like a bad idea).  It works
well and is pretty stable.  If your CLAMAV was causing you this problem
then Zimbra must have boloxed the setup or you just had a bad version.


signature.asc
Description: This is a digitally signed message part

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

FW: Retrieving a banned e-mail

[Rajeev R Veedu]

We have a postfix+cyrus+Spamassassin setup and all the virus or Banned or
spam e-mails will move to a folder on /var/virusmails.

I have a falseposative mail moved due to a banned attachment name. Could you
please help me to retrieve these e-mails?

 

Thanks

 

 

Rajeev R. Veedu


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Replication: problems with synctest

[Simon Matter]

> On Nov 13, 2007 8:08 AM, Rich Wales <[EMAIL PROTECTED]> wrote:
>> OK, so I decided to try what I described earlier (replication in both
>> directions, with different users using different master servers) . . . .
>>
>> But now I'm running into an authentication problem.  One of my servers
>> (my original replica) simply refuses to authenticate to the other one
>> (my original master).
>>
>> I've double-checked the user name and password, and I know it's in the
>> sasldb2.db file on the master, but authentication just will not work.
>>
>> I tried synctest to each server, and I can't connect to either server
>> using synctest.  Examples ("whodunit" is my original master; "flipflop"
>> is my original replica; and I've obscured the real authentication
>> strings in the AUTHENTICATE commands):
>>
>> % synctest -u admin -m plain whodunit
>> S: * SASL NTLM LOGIN PLAIN GSSAPI OTP DIGEST-MD5 CRAM-MD5
>> S: * STARTTLS
>> S: * OK whodunit.richw.org Cyrus sync server v2.3.9
>> Please enter your password:
>> C: AUTHENTICATE PLAIN ***
>> S: NO user not found
>> Authentication failed. generic failure
>> Security strength factor: 0
>> C: EXIT
>> Connection closed.
>>
>> % synctest -u admin -m plain flipflop
>> S: * SASL CRAM-MD5 DIGEST-MD5 GSSAPI LOGIN PLAIN NTLM
>> S: * STARTTLS
>> S: * OK flipflop Cyrus sync server v2.3.9
>> Please enter your password:
>> C: AUTHENTICATE PLAIN ***
>> S: NO authentication failure
>> Authentication failed. generic failure
>> Security strength factor: 0
>> C: EXIT
>> Connection closed.
>>
>> What's especially weird is the first one (whodunit), which gave a
>> "user not found" error, even though there IS an account named "admin"
>> in the sasldb2.db on that machine.
>>
>> Any ideas what I might be doing wrong here?
>
> Do you have virtdomains enable ?
> Did you setup defaultdomain ?
> Did you define user [EMAIL PROTECTED] ?
> Is $defaultdomain in loginrealms ?
>

No, my testconfig is very simple:

configdirectory: /var/lib/imap
partition-default: /var/spool/imap
admins: cyrus
sievedir: /var/lib/imap/sieve
sendmail: /usr/sbin/sendmail
hashimapspool: true
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN
tls_cert_file: /usr/share/ssl/certs/cyrus-imapd.pem
tls_key_file: /usr/share/ssl/certs/cyrus-imapd.pem
tls_ca_file: /usr/share/ssl/certs/ca-bundle.crt
delete_mode: delayed

Simon


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Deleting top-level mailbox with 'delete_mode: delayed'

[Simon Matter]

> On Tue, Nov 13, 2007 at 01:11:49PM +0100, Simon Matter wrote:
>> > expunge_mode: delayed
>> > delete_mode: delayed
>>
>> I've just tried a batch delete of mailboxes and hit the same wall.
>>
>> Mailbox deletion doesn't work anymore with 2.3.10 if "delete_mode:
>> delayed". If "delete_mode: immediate" it works, but with delayed I get
>> "deletemailbox: Operation is not supported on mailbox".
>>
>> Did I miss something? Does anybody have a patch?
>
> I have "delete_mode: immediate" on the replica and
> "delete_mode: delayed" on the master.  It doesn't make any sense
> for the replica to do a delayed delete, as the master is already
> generating a "RENAME" event (well, two MAILBOX events actually,
> let's not get picky) with the old and new names for the mailboxes.

My testcase is a single server, no replica whatever.

Simon


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[Bron Gondwana]

On Tue, Nov 13, 2007 at 10:24:22AM +, David Carter wrote:
> On Sun, 11 Nov 2007, Bron Gondwana wrote:
> 
> >> 250,000 mailboxes, 1,000 concurrent users, 60 million emails, 500k 
> >> deliveries/day.  For us, backups are the worst thing, followed by 
> >> reiserfs's use of BLK, followed by the need to use a ton of disks to 
> >> keep up with the i/o.
> >
> > For us backups are hardly a blip on the radar :)  The joy of writing 
> > your own custom backup system that knows more about Cyrus internals than 
> > just about anything else.  It starts with some stat calls, and if any of 
> > the cyrus.header, cyrus.index or cyrus.expunge files have changed then 
> > it will lock them all then stream them all to the backup server.
> 
> Cyrus is pretty ideal for fast incremental updates to a backup system: 
> hence replication. You shouldn't need to lock anything with delayed 
> expunge, delayed delete and fast rename in place.

If you're planning to lift a consistent copy of a .index file, you need
to lock it for the duration of reading it (read lock at least).

Yeah - replication is one way to do it.  We happen to read from the
masters at the moment, but it would be pretty trivial to switch to
using the replicas (change a $Store->MasterSlot() to
$Store->ReplicaSlot() at one place in the code in fact) if we wanted to.

But since I would like a consistent snapshot of the mailbox state,
I lock the cyrus.header and then the cyrus.index and then (if it's
there) the cyrus.expunge.  That means no sneaky process could (for
example) delete the mailbox and create another one with the same
name while I was busy downloading the last file - giving me totally
bogus data.  This is particularly important because I store things
by mailbox uniqueid rather than imap path (with pointers from the
imap path of course) so that a folder rename turns into a symlink
delete (well, replacement with one having an empty target anyway) 
and a symlink create in the tar file.

Bron ( and right now I'm running the process to finish the upgrade
   from MD5 based to SHA1 based internal identifiers in the
   backup system, since all our indexes are upgraded )

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Deleting top-level mailbox with 'delete_mode: delayed'

[Bron Gondwana]

On Tue, Nov 13, 2007 at 01:11:49PM +0100, Simon Matter wrote:
> > expunge_mode: delayed
> > delete_mode: delayed
> 
> I've just tried a batch delete of mailboxes and hit the same wall.
> 
> Mailbox deletion doesn't work anymore with 2.3.10 if "delete_mode:
> delayed". If "delete_mode: immediate" it works, but with delayed I get
> "deletemailbox: Operation is not supported on mailbox".
> 
> Did I miss something? Does anybody have a patch?

I have "delete_mode: immediate" on the replica and 
"delete_mode: delayed" on the master.  It doesn't make any sense
for the replica to do a delayed delete, as the master is already
generating a "RENAME" event (well, two MAILBOX events actually,
let's not get picky) with the old and new names for the mailboxes.

The replica will be doing a rename rather than a delete in the
most frequent case anyway.  If you're unlucky and the two
MAILBOXES calls get split up (probably some other event on the
first mailbox from earlier being run after you've done the 
rename - don't you love concurrency) then it will issue a
DELETE on the replica.  If that causes a rename instead you
will wind up with _TWO_ deleted folders with very similar
names, one containing all the messages that were still on
the replica and one containing all the messages in the
copy on the master.

Better just to re-copy those ones from the master in the
unlucky case if you ask me.

Bron.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Deleting top-level mailbox with 'delete_mode: delayed'

[Simon Matter]

> On Nov 5, 2007 6:15 AM, Bron Gondwana <[EMAIL PROTECTED]> wrote:
>>
>> On Fri, Nov 02, 2007 at 01:15:37PM -0400, Brian Wong wrote:
>> > On Nov 2, 2007 12:39 PM, Rudy Gevaert <[EMAIL PROTECTED]> wrote:
>> > >
>> > > Brian Wong wrote:
>> > > > I was testing out Cyrus 2.3.10 and realized that when I set the
>> option
>> > > >
>> > > > delete_mode: delayed
>> > > >
>> > > > I can not delete top-level mailboxes.
>> > > >
>> > > > localhost.localdomain> lm
>> > > > localhost.localdomain> cm user.bwong
>> > > > localhost.localdomain> sam user.bwong  c
>> > > > localhost.localdomain> dm user.bwong
>> > > > deletemailbox: Operation is not supported on mailbox
>> > > > localhost.localdomain> lm
>> > > > user.bwong (\HasNoChildren)
>> > > >
>> > > > Disabling the delayed delete gives expected results. The mailbox
>> is
>> > > > deleted as normal. Anyone else confirm this?
>> > >
>> > > I'm just back from holiday (and only catching up on mail).  I always
>> set
>> > > the 'x' permission.  Could you try that?  If that doesn't work, I'll
>> try
>> > > to delete a top level mailbox on Monday (I'm running 2.3.10 in
>> test).
>> > >
>> > > Rudy
>> > >
>> >
>> > localhost.localdomain> lam user.bwong
>> > bwong lrswipkxtecda
>> > admin kxc
>> > localhost.localdomain> dm user.bwong
>> > deletemailbox: Operation is not supported on mailbox
>> >
>> > I think if I did not have the right to delete the mailbox, I would get
>> > a "Permission Denied" instead of the error I am receiving. Let me know
>> > what you find when you try it. I feel that if this is really a bug it
>> > would have been caught before release, but then again I can't think of
>> > anything atypical with my setup that would cause this problem.
>>
>> It's almost certainly caused by the code that checks if you're renaming
>> a "top level mailbox" for a user and special cases it in all sorts of
>> ways.  I never liked that code much!
>>
>> My solution was to make DELETED.user.bwong.46A12345 (or similar) also
>> be considered to be an "INBOX" so it was treated as a user rename.
>> This seems not to be working in your environment, and I'm really not
>> sure why.
>>
>> I don't see anything specially different in our config.
>> fast_rename: yes, but that won't work for you anyway because it's
>> using a not-yet-perfect patch at our end.
>>
>> All our mailbox deletes are done as the admin user.  It won't work
>> if you're not a bona-fide admin (not just a user called admin who
>> happens to have an ACL).  Check the 'admins:' parameter in your
>> imapd.conf.
>>
>> Regards,
>>
>> Bron.
>>
>> (P.S. your username is scarily similar to mine!)
>>
>
> admins: cyradm
>
> localhost.localdomain> sam user.bwong cyradm c
> localhost.localdomain> lam user.bwong
> bwong lrswipkxtecda
> cyradm kxc
> localhost.localdomain> dm user.bwong
> deletemailbox: Operation is not supported on mailbox
>
> The 'admins' parameter seems to be fine. I just didnt want to get
> specific with the configuration and showing that the admin was
> 'cyradm' because I didnt want to confuse anyone with having an admin
> user with the same name as the command.
>
> I really do not know what to do. This is a test box where I removed
> and recreated all the relevant cyrus directories just to make sure I
> was starting from scratch.
>
> Configuration:
> imapidresponse: 0
> allowallsubscribe: 1
> allowplaintext: 1
>
> configdirectory: /var/imap
>
> defaultpartition: ms1
> metapartition_files: header index cache expunge
>
> partition-default: /var/spool/imap
>
> partition-ms1: /var/spool/imap/data1
> metapartition-ms1: /var/spool/imap/meta1
> partition-ms2: /var/spool/imap/data2
> metapartition-ms2: /var/spool/imap/meta2
>
> auth_mech: pts
> pts_module: ldap
>
> admins: cyradm
>
> sasl_pwcheck_method: saslauthd
> sasl_mech_list: plain
>
> 
>
> username_tolower: 1
> lmtp_downcase_rcpt: 1
>
> singleinstancestore: 1
>
> munge8bit: 0
> reject8bit: 0
>
> duplicate_db: skiplist
> annotation_db: skiplist
> mboxkey_db: skiplist
> mboxlist_db: skiplist
> ptscache_db: skiplist
> seenstate_db: skiplist
> subscription_db: flat
> tlscache_db: skiplist
>
> expunge_mode: delayed
> delete_mode: delayed

I've just tried a batch delete of mailboxes and hit the same wall.

Mailbox deletion doesn't work anymore with 2.3.10 if "delete_mode:
delayed". If "delete_mode: immediate" it works, but with delayed I get
"deletemailbox: Operation is not supported on mailbox".

Did I miss something? Does anybody have a patch?

Thanks,
Simon


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Replication: problems with synctest

[Alain Spineux]

On Nov 13, 2007 8:08 AM, Rich Wales <[EMAIL PROTECTED]> wrote:
> OK, so I decided to try what I described earlier (replication in both
> directions, with different users using different master servers) . . . .
>
> But now I'm running into an authentication problem.  One of my servers
> (my original replica) simply refuses to authenticate to the other one
> (my original master).
>
> I've double-checked the user name and password, and I know it's in the
> sasldb2.db file on the master, but authentication just will not work.
>
> I tried synctest to each server, and I can't connect to either server
> using synctest.  Examples ("whodunit" is my original master; "flipflop"
> is my original replica; and I've obscured the real authentication
> strings in the AUTHENTICATE commands):
>
> % synctest -u admin -m plain whodunit
> S: * SASL NTLM LOGIN PLAIN GSSAPI OTP DIGEST-MD5 CRAM-MD5
> S: * STARTTLS
> S: * OK whodunit.richw.org Cyrus sync server v2.3.9
> Please enter your password:
> C: AUTHENTICATE PLAIN ***
> S: NO user not found
> Authentication failed. generic failure
> Security strength factor: 0
> C: EXIT
> Connection closed.
>
> % synctest -u admin -m plain flipflop
> S: * SASL CRAM-MD5 DIGEST-MD5 GSSAPI LOGIN PLAIN NTLM
> S: * STARTTLS
> S: * OK flipflop Cyrus sync server v2.3.9
> Please enter your password:
> C: AUTHENTICATE PLAIN ***
> S: NO authentication failure
> Authentication failed. generic failure
> Security strength factor: 0
> C: EXIT
> Connection closed.
>
> What's especially weird is the first one (whodunit), which gave a
> "user not found" error, even though there IS an account named "admin"
> in the sasldb2.db on that machine.
>
> Any ideas what I might be doing wrong here?

Do you have virtdomains enable ?
Did you setup defaultdomain ?
Did you define user [EMAIL PROTECTED] ?
Is $defaultdomain in loginrealms ?

Hope this help.

>
> --
> Rich Wales  ===  Palo Alto, CA, USA  === [EMAIL PROTECTED]
> http://www.richw.org   ===   http://en.wikipedia.org/wiki/User:Richwales
> "The difference between theory and practice is that, in theory,
> theory and practice are identical -- whereas in practice, they aren't."
> 
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>



-- 
Alain Spineux
aspineux gmail com
May the sources be with you

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Calendars && Cyrus

[Adam Tauno Williams]

> > If we could ever get a decent calendar system that works together with
> > Cyrus or other software many people would be happy.

We use OpenGroupware - http://www.opengroupware.org

Server is entirely Open Source;  Outlook plugin (MAPI provider) is
commercial.

Works very well with Cyrus; which is its intended IMAP server.

> We run Cyrus for mail, Oracle Collaboration Suite ($$, but not  
> Exchange-$$) for calendaring.  The Outlook plugin that allows Outlook  
> to believe it has a genuine Exchange instance under it plays nice  
> with Cyrus (its target is Oracle's IMAP server that's bundled with  
> OCS).The OCS plugin for Exchange is smart enough to use CRAM-MD5  
> (or is it DIGEST-MD5) authentication if available, and TLS if available.



signature.asc
Description: This is a digitally signed message part

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Timed Actions in Sieve

[Ian G Batten]

We've been having a chat about how useful it would be to have timed  
actions in sieve: so that a vacation message could be set up for a  
duration which would automatically revert, so that a forwarding could  
be set up for the duration of a short-term project, etc, etc.  The  
naive way is to add support to the sieve interface of choice (the  
squirrelmail plugin in our case) to handle deferred actions, but I  
can think of all sorts of security problems with that.  Another would  
be a means to auto-generate regexps to match on Date: headers, but  
that's really tacky.  The full solution would be to have the current  
time available in sieve scripts, to then match on.  Has anyone else  
thought about this area?

ian


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Multiple skiplist bugs found, patches attached

[David Carter]

On Tue, 13 Nov 2007, Simon Matter wrote:

> I didn't have much troubles with skiplist over the years and it has been 
> a blessing since moving away from BDB. But I did have a few issues with 
> broken skiplist files so your patches are very welcome. I have included 
> the patches in my private rpm packages to try how they work. Do you 
> recommend both for general consumption?

It is certainly very easy to break mailboxes.db using cyr_dbtool.

Kudos to Bron for tracking down the problems.

-- 
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Just in case it is of general interest: ZFS mirroring was the culprit in our case

[David Carter]

On Tue, 13 Nov 2007, Pascal Gienger wrote:

> Our latency problems went away like a miracle when we detached one half 
> of the mirror (so it is no more a mirror).
>
> Read-Rates are doubled (not per device, the total read rate!), latency 
> is cut off. No more latency problems.
>
> When attaching the volume again, resilvering puts the system to a halt -
> reads and writes do block for seconds (!).

Definitely of interest to those of us keeping one eye on ZFS. Thanks. Can 
someone else running ZFS confirm this behaviour?

-- 
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: LARGE single-system Cyrus installs?

[David Carter]

On Sun, 11 Nov 2007, Bron Gondwana wrote:

>> 250,000 mailboxes, 1,000 concurrent users, 60 million emails, 500k 
>> deliveries/day.  For us, backups are the worst thing, followed by 
>> reiserfs's use of BLK, followed by the need to use a ton of disks to 
>> keep up with the i/o.
>
> For us backups are hardly a blip on the radar :)  The joy of writing 
> your own custom backup system that knows more about Cyrus internals than 
> just about anything else.  It starts with some stat calls, and if any of 
> the cyrus.header, cyrus.index or cyrus.expunge files have changed then 
> it will lock them all then stream them all to the backup server.

Cyrus is pretty ideal for fast incremental updates to a backup system: 
hence replication. You shouldn't need to lock anything with delayed 
expunge, delayed delete and fast rename in place.

-- 
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Replication: sync_client -r dies

[David Carter]

On Mon, 12 Nov 2007, Bron Gondwana wrote:

>> It seems to me that the replication code ought to be a bit more robust
>> than this when a replica goes down or loses network connectivity.  Is
>> the 2.3.10 code any better than 2.3.9 in the way this kind of situation
>> is handled?
>
> I believe David Carter has been working on some stuff for this which is
> lined up to go in soon.

The autorestart stuff is already in 2.3.10.

It was Ken's work, based on a suggestion on my part.

-- 
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

problems with virtual domains and sasl under Debian 4

[Falko Zurell]

Hello List_members,

I'm using a Debian Linux 4 and the provided Cyrus (2.2.13) and SASL  
packages from Debian. I've configured my Postfix MTA to use virtual  
domains with the SASL library as authentication base. Postfix works  
fine for me but Cyrus makes some trouble.

I can't authenticate on the Cyrus IMAPD with my virtual domain name.  
When I try to login with "[EMAIL PROTECTED]" on cyrus imapd  
authentication fails. When I add a user to the saslpasswd file with  
the default domain (eq. localhost) I can authenticate on IMAPD.

my imapd.conf ist this:

configdirectory: /opt/data/cyrus/lib/
defaultpartition: default
partition-default: /opt/data/cyrus/mail
partition-news: /opt/data/cyrus/news
newsspool: /var/spool/news
altnamespace: no
unixhierarchysep: yes
reject8bit: yes
lmtp_downcase_rcpt: yes
admins: cyrus
allowanonymouslogin: no
popminpoll: 1
autocreatequota: 0
umask: 077
sieveusehomedir: false
sievedir: /opt/data/cyrus/sieve
hashimapspool: true
allowplaintext: no
sasl_mech_list: PLAIN
loginrealms: domain.com
virtdomains: on
defaultdomain: domain.com
sasl_pwcheck_method: saslauthd
sasl_auxprop_plugin: sasldb
sasl_auto_transition: yes
tls_cert_file: /etc/ssl/certs/post.domain.com.cert.pem
tls_key_file: /etc/ssl/private/post.domain.com.key.pwless.pem
tls_ca_path: /etc/ssl/certs
tls_session_timeout: 1440
tls_cipher_list: TLSv1+HIGH:!aNULL:@STRENGTH
lmtpsocket: /var/run/cyrus/socket/lmtp
idlemethod: poll
idlesocket: /var/run/cyrus/socket/idle
notifysocket: /var/run/cyrus/socket/notify
syslog_prefix: cyrus

Can someone give me a hint where to look? I now this must work because  
I've got it running on a SuSE Linux box with almost the same  
configuration. I can't find the mistake by myself at the moment.


Thanks and kind regards

Falko Zurell

/i-d media AG

Ohlauer Straße 43
D-10999 Berlin

www.idmedia.com

___
Board: Regine Haschka-Helmer (CEO), Franz Klose (CFO)
Chairman of the Supervisory Board: Christian A. Hufnagl
Headquarters: Berlin, Place of Jurisdiction: Berlin HRB 76342

___
This message may contain confidential information and must not be  
copied, disclosed or used by anybody other than the intended  
recipient. If you have received this message in error, please notify  
us immediately by reply e-mail and delete all records of the message  
from your computer. The views represented in this message are solely  
those of the author. Neither the author nor I-D Media AG accepts any  
liability for the contents of this message.



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Replication: does it work in both directions?

[David Carter]

On Sun, 11 Nov 2007, Rich Wales wrote:

> So, I would have replication set up going both directions between my two 
> servers, but the sets of users handled in each direction would be 
> disjoint.  Each user would be assigned to one IMAP server (the master 
> for their mailbox collection), and the other server would be their 
> replica and act as their backup.

We do this. It is quite useful to be able to bounce users back and forth 
between the two machines in a pair so that servers can be maintained 
(patches, O/S upgrades, whatever) without any user visible downtime.

Three caveats:

1) It won't work with shared mailboxes.

2) I'm not running the same replication code as the rest of you (though
replication in 2.3 is based on an old version of my code). I seem to
remember Ken raising an objection when this last discussed a year or
two back now. The objection may just have just been (1).

3) Sanity checks are good:

USER dpc22
NO IMAP_INVALID_USER Attempt to update master for dpc22

-- 
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Ian G Batten]

On 13 Nov 07, at 0812, Scott M. Likens wrote:
> No it's not
> as seamless as Exchange, but it works just fine and it's an open
> standard.

However, people don't want calendaring, they want Outlook.   Offer a  
solution which doesn't allow Outlook to work ``like it should'' and  
you risk learning more about Exchange than you wanted to.   I guess  
Outlook might start to talk CalDAV, but I personally doubt it: it  
would undercut Microsoft's Exchange business which is very lucrative  
for them.  OCS has an Outlook plugin I suspect for this very reason:  
it avoids management refusing to accept a solution which doesn't do  
what they see as the right thing: the rest of us use the OCS Linux/ 
Solaris/Windows/OSX native clients or the web front end, all of which  
work very nicely.

Your management won't regard CalDAV as a standard, they think Outlook  
is a standard.  And ``not as seamless'' is a key admission.

ian


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Rudy Gevaert]

Scott M. Likens wrote:
> Have you ever looked at some of the CalDAV Servers out there?
> 
> I'll save you some time,
> 
> http://trac.macosforge.org/projects/calendarserver
> 
> http://rscds.sourceforge.net/
> 
> As well as,
> 
> http://sourceforge.net/projects/modcaldav/
> 
> Truthfully, you don't need Cyrus to support a Calendar.  Because in all 
> honesty, it's unrelated to mail.  If you use Kerberos, LDAP, AD, MySQL 
> for Authentication.  Take a look at one of those, tie in that 
> authentication and you're done.  Then depending on which one you choose, 
> you can have users share their calendars or not.
> ... There is enough F/OSS out there to emulate everything you can get 
> with Exchange, and/or any other 'Enterprise' Mail System.  No it's not 
> as seamless as Exchange, but it works just fine and it's an open 
> standard.  You'll find lots more CalDav Servers, and software in the 
> next 6months to a year.

Well I'm impressed, I didn't know it would already be possible.  I'll 
keep an eye on those projects.  When I have some time I'll give them a 
closer look!

Are you running any calendar server?

Rudy


-- 
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Rudy Gevaert  [EMAIL PROTECTED]  tel:+32 9 264 4734
Directie ICT, afd. Infrastructuur ICT Department, Infrastructure office
Groep SystemenSystems group
Universiteit Gent Ghent University
Krijgslaan 281, gebouw S9, 9000 Gent, Belgie   www.UGent.be
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Calendars && Cyrus

[Ian G Batten]

On 13 Nov 07, at 0724, Rudy Gevaert wrote:

> If we could ever get a decent calendar system that works together with
> Cyrus or other software many people would be happy.

We run Cyrus for mail, Oracle Collaboration Suite ($$, but not  
Exchange-$$) for calendaring.  The Outlook plugin that allows Outlook  
to believe it has a genuine Exchange instance under it plays nice  
with Cyrus (its target is Oracle's IMAP server that's bundled with  
OCS).The OCS plugin for Exchange is smart enough to use CRAM-MD5  
(or is it DIGEST-MD5) authentication if available, and TLS if available.

ian


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Multiple skiplist bugs found, patches attached

[Bron Gondwana]

On Tue, 13 Nov 2007 09:12:18 +0100 (CET), "Simon Matter" <[EMAIL PROTECTED]> 
said:
> > On Mon, Nov 12, 2007 at 12:34:34AM +1100, Bron Gondwana wrote:
> >> Anyway - here it is.  A "recovery()" that copes if the logstart
> >> parameter in the database header is wrong.  No, I don't have a
> >> clue how that happened unless lseek() lied.  Maybe it sometimes
> >> lies, I don't know.  I'll be writing a test case for that soon
> >> too!
> >
> > I have some more suspicions now, but I wrote it all up in the
> > patch header, so here's the bugfixes only patch, a "robustness"
> > extras patch and the tool I used for testing.
> >
> > Ken, I know you've done some other work on the file changing
> > types.  I'd like to be even more agressive and convert just
> > about everything to bit32 and also rename some variables, but
> > I restricted myself in this to only fixing the most ugly case:
> > offset = htonl(offset).
> >
> > These patches are all against 2.3.10 (in this order), and may
> > need some fuzz fixing to apply against your latest CVS thanks
> > to those changes - sorry I haven't done that, but it's getting
> > on 1am for me, and I've just finished doing a lot of testing
> > and paring these down to simple and clear patches that don't
> > touch more than they need to fix the issues.
> >
> 
> > cyrus-skiplist-bugfixes-2.3.10.diff:
> >
> 
> > cyrus-skiplist-robustify-2.3.10.diff:
> >
> 
> Hi Bron,
> 
> I didn't have much troubles with skiplist over the years and it has been
> a
> blessing since moving away from BDB. But I did have a few issues with
> broken skiplist files so your patches are very welcome. I have included
> the patches in my private rpm packages to try how they work. Do you
> recommend both for general consumption?

They've been running for 24 hours on all our production systems with
no ill effects :)

Seriously - yes, I do.  They are quite short, and they're the culmination
of about 3 days of pretty heavy work over the weekend and Monday after we
lost a mailboxes.db on our busiest store to one of these bugs (my wife and
kids were getting ready to kill me for neglecting them towards the end,
I'm sure!)  I build multiple different patches and tested them over that
time.  I also wrote a Perl module that can read skiplist files natively
and tested some things with that as well.

These couple of patches I have posted are the best bits of those distilled
down into the simplest and clearest small set of changes.  They've been hit
pretty hard with the hammer scripts.


I've also got another patch which I'll attach here that I wrote today which
re-tunes the "how often to checkpoint" calculation.  I want our mailboxes.db
files especially to checkpoint more frequently, as that will make them
less "seeky" - which will help with cachelines at least.  We have enough
memory (and always plenty free) that I'm sure every page is hot in cache
within a few minutes.

The seekyness is mainly an issue with clients doing "LIST", which our web
interface does at login, so we want it to be as quick as possible.

As for seen files - well, they tend to be small and frequently updated,
so they'll just checkpoint about 4 times as often now.  Will save a tiny
bit of disk space but more interestingly reduce the memory footprint to
keep them all in cache.


Bron.
-- 
  Bron Gondwana
  [EMAIL PROTECTED]

Skiplist tuning

With random changes to a mailboxes.db file, it could be nearly 100%
random seeks before it recompressed.

A seen file would need to reach 16kb before even considering
re-compressing, with a real data length of just a couple of hundred
bytes.

This patch reduces the limits to:

4kb overhead
120% rather than 200% of current "sorted" size.
Index: cyrus-imapd-2.3.10/lib/cyrusdb_skiplist.c
===
--- cyrus-imapd-2.3.10.orig/lib/cyrusdb_skiplist.c	2007-11-12 23:53:34.0 -0500
+++ cyrus-imapd-2.3.10/lib/cyrusdb_skiplist.c	2007-11-12 23:57:38.0 -0500
@@ -302,7 +302,7 @@
 SKIPLIST_VERSION = 1,
 SKIPLIST_VERSION_MINOR = 2,
 SKIPLIST_MAXLEVEL = 20,
-SKIPLIST_MINREWRITE = 16834 /* don't rewrite logs smaller than this */
+SKIPLIST_MINREWRITE = 4096 /* don't rewrite logs smaller than this */
 };
 
 #define BIT32_MAX 4294967295U
@@ -1392,8 +1392,8 @@
 }
 
  done:
-/* consider checkpointing */
-if (!r && tid->logend > (2 * db->logstart + SKIPLIST_MINREWRITE)) {
+/* consider checkpointing (journal is 20% of data length) */
+if (!r && tid->logend > (12 * db->logstart / 10 + SKIPLIST_MINREWRITE)) {
 	r = mycheckpoint(db, 1);
 }
 

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: OT: Re: How many people to admin a Cyrus system?

[Scott M. Likens]

Have you ever looked at some of the CalDAV Servers out there?

I'll save you some time,

http://trac.macosforge.org/projects/calendarserver

http://rscds.sourceforge.net/

As well as,

http://sourceforge.net/projects/modcaldav/

Truthfully, you don't need Cyrus to support a Calendar.  Because in all
honesty, it's unrelated to mail.  If you use Kerberos, LDAP, AD, MySQL
for Authentication.  Take a look at one of those, tie in that
authentication and you're done.  Then depending on which one you choose,
you can have users share their calendars or not.

... There is enough F/OSS out there to emulate everything you can get
with Exchange, and/or any other 'Enterprise' Mail System.  No it's not
as seamless as Exchange, but it works just fine and it's an open
standard.  You'll find lots more CalDav Servers, and software in the
next 6months to a year.

Scott

Rudy Gevaert wrote:
> If we could ever get a decent calendar system that works together with 
> Cyrus or other software many people would be happy.
>
> Rudy
>   



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Multiple skiplist bugs found, patches attached

[Simon Matter]

> On Mon, Nov 12, 2007 at 12:34:34AM +1100, Bron Gondwana wrote:
>> Anyway - here it is.  A "recovery()" that copes if the logstart
>> parameter in the database header is wrong.  No, I don't have a
>> clue how that happened unless lseek() lied.  Maybe it sometimes
>> lies, I don't know.  I'll be writing a test case for that soon
>> too!
>
> I have some more suspicions now, but I wrote it all up in the
> patch header, so here's the bugfixes only patch, a "robustness"
> extras patch and the tool I used for testing.
>
> Ken, I know you've done some other work on the file changing
> types.  I'd like to be even more agressive and convert just
> about everything to bit32 and also rename some variables, but
> I restricted myself in this to only fixing the most ugly case:
> offset = htonl(offset).
>
> These patches are all against 2.3.10 (in this order), and may
> need some fuzz fixing to apply against your latest CVS thanks
> to those changes - sorry I haven't done that, but it's getting
> on 1am for me, and I've just finished doing a lot of testing
> and paring these down to simple and clear patches that don't
> touch more than they need to fix the issues.
>

> cyrus-skiplist-bugfixes-2.3.10.diff:
>

> cyrus-skiplist-robustify-2.3.10.diff:
>

Hi Bron,

I didn't have much troubles with skiplist over the years and it has been a
blessing since moving away from BDB. But I did have a few issues with
broken skiplist files so your patches are very welcome. I have included
the patches in my private rpm packages to try how they work. Do you
recommend both for general consumption?

Simon


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html