Re: Cyrus 2.5, xapian, Sphinx and index sizes

2014-09-24 Thread Robert Mueller
# Any time the disk gets over 50%, compress -o single down to data 13 * * * * /home/mod_perl/hm/scripts/ -a -o -d 50 temp data # Copy the temporary search databases down to data during the week 43 1 * * 1,2,3,4,5,6 /home/mod_perl/hm/scripts/ -a temp,meta

Re: How to improve mail search from iPad and other mobile devices

2014-09-24 Thread Robert Mueller
According to Apple's manual, Mail searches all folders like this: Searching looks at the address fields, the subject, and the message body. FYI when we found out about this change, we made all BODY searches from iOS become FUZZY BODY searches to make them work reasonably.

Re: Cannot see entries in folders when using MS Outlook 2013

2013-04-15 Thread Robert Mueller
When I use outlook 2013, I can see the subfolders in my inbox, but NO entries are shown. In Outlook, go to the Advanced settings tab for the account, make sure your root folder path is Inbox not INBOX. Rob Cyrus Home Page: List Archives/Info:

Re: problems with quota files

2013-02-20 Thread Robert Mueller
I realized that quota shown for users with quota command is much less than real filesystem usage This is expected. cyrus quota is sum of size in bytes of each email. Disk quota used might be much higher because minimum size of a file on lots of filesystems is 1 block. Rob Cyrus Home

Re: UTF7 mailbox codification error

2012-02-28 Thread Robert Mueller
It looks like internal debugging junk from our system leaked into a public build to me. Rob? Should we do a release without the 'contact rjlov'? Richard fixed this, and I've pushed the latest code to: You can get it from there for now. Rob

Re: UTF7 mailbox codification error

2012-02-23 Thread Robert Mueller
$IMAPs1-set_unicode_folders(1); $IMAPs2-set_unicode_folders(1); ... How can I avoid this error? Or force that the method do not convert the strings. You explicitly said you want unicode folder support, but then explicitly pass IMAP UTF-7 folder names rather than a perl unicode string. So

Re: Cyrus 2.4.13 memory use

2012-02-08 Thread Robert Mueller
It looks like 3000 IMAP sessions are going to take around 8 GBytes of RAM just to run, and we will need to buy additional RAM for buffer cache. This isn't the end of the world: memory is cheap. I'm just curious if anyone else saw a similar increase when upgrading from 2.3 to 2.4. I

RE: disable IMAP IDLE

2010-11-23 Thread Robert Mueller
I was asked by IT to not permit IDLE since the current server went down after 4-500 blackberries ate up all the (limited) capabilities of that machine. I'd really be surprised if that was a problem these days. We have machines that have 1 connections quite fine. Yes they're fairly loaded

Re: ext3 / XFS [Was: Re: Does anyone allow unlimited or extremely large quotas?]

2010-11-16 Thread Robert Mueller
This is depends on what filesystem you are useing, I have mailboxes with hundreds of thousands of messages in them on XFS and have no problems, but on ext3 I start seeing slowdowns with a bit over ten thousand messages. Was dir_index enabled on that ext3 filesystem? Prior to

Re: how to debug what happens inside mailboxes

2010-04-12 Thread Robert Mueller
That is exactly what I need. Thanks. Is there some sort of documentation for such features in cyrus? I can't seem to find anything beyond basic setup and what is in manual pages. Stuff like annotations (through which I managed to delete 30GB of emails), global sieve skripts, snmp

Re: Nginx configuration for imap

2010-03-17 Thread Robert Mueller
$user['user1'] =; $user['user2'] =; $user[user15000]=; For 15k users this method becomes very heavy. There are too many httpd processes running that suck the resources on the machine. I want to store the userlist in a memcache and look it up through

Re: Nginx configuration for imap

2010-03-17 Thread Robert Mueller
But I thought a memcache lookup will be much more inexpenisve than connecting to a mysql db to do lookup for every cyrus connection Probably slightly. But what happens if the value isn't in memcached? Where do you get the value from? Anyway, it's still WAY better than doing:

Re: what happens when cyrus.squat is old?

2010-01-21 Thread Robert Mueller
Are you using the new incremental mode david carter added? -i Incremental updates where squat indexes already exist. I'm not. This is a very old install. However, we're planning a migration to a new server and I'll possibly try it then. Can you just compile squatter separately

Re: Incorrect sort order when sorting by date

2009-05-18 Thread Robert Mueller (web)
I bet if you look at the Date headers of the problem emails, you'll find that they're not RFC compliant. The cyrus date parser is very strict, and if the header isn't RFC compliant, you'll get a bad value, and bad sorting. Looking at the RFC If the sent

Re: UC Davis Cyrus Incident September 2007

2007-10-18 Thread Robert Mueller
Actually, I don't see a deadlock situation at all... I am guessing that theorettically, it is possible... but the ln -sf option makes the overwriting of the symlink an atomic action (as much as it can), which Not a deadlock situation, but a possible file doesn't exist error. In the Unix/POSIX

Re: Recomendations for a 15000 Cyrus Mailboxes

2007-04-10 Thread Robert Mueller
1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) 2. Which filesystem seems to be the better ? ext3 ? xfs ? reiserfs ? 3. Which options to format the filesystem ? acording to the chosed filesystem 4. Which pop3 / imap proxy to use ? 5. Single instance or multiple instances of


2007-02-27 Thread Robert Mueller
Nice! Is this code available for public consumption? I'd love to use something like this to strip out \0s in messages too. Most of the other solutions for cyrus/postfix that I've seen required an additional exec() in the delivery pipeline, which I would like to avoid. It's all done in

Re: lmtpd memory leak?

2007-02-22 Thread Robert Mueller
all my lmtpd-processes are growing in size and are consuming more and more CPU-time and increasing it's priority. As far as i can see, mail gets deliverd fine, but only from the small lmtpd-processes. The huge processes are not used for delivery, and new lmtp processes get started,

Re: Bounces on local delivery

2006-12-04 Thread Robert Mueller
This usually happens because the lmtp connection got out of sync somehow. There was a bug in older postfix's about this: I haven't seen the problem since then since Wietse fixed it, but I'd try setting lmtp_cache_connect=no in

Bug with rename INBOX - INBOX.blah and replication

2006-12-04 Thread Robert Mueller
Hi Ken There's a bug with replication and renaming INBOX - INBOX.blah. From Renaming INBOX is permitted, and has special behavior. It moves all messages in INBOX to a new mailbox with the given name, leaving INBOX empty. If the server

Re: problems on folders containing @

2006-11-12 Thread Robert Mueller
why does this give an error? x create [EMAIL PROTECTED] x NO Invalid mailbox name Why isn't it allowed at Cyrus that folders contain an @-sign? It was allowed with dovecot and courier-imap and it is possible to create folders with several other characters like Umlauts and spaces. Why is the

Re: performance on large inboxes

2006-11-12 Thread Robert Mueller
I'm using ext3 with dir_hash. I considered using XFS, but there are a lot of benchmarks that show that XFS is not faster in general, also the XFS development seems to be stucked at the moment and from my own experience as well as from other people in a recent thread on this mailinglist there

Re: Too slow

2006-10-25 Thread Robert Mueller
I turned the mail partition now to xfs and it's terribly fast. WOW!!! did you use an ext3 partition with dir_index before? I'm just asking We've just had some experience with filesystems ourselves. Previously we've used reiserfs exclusively for the last 5 years. All up, it's been really

Re: Performance and cheap storage

2006-08-03 Thread Robert Mueller
not sure if we qualify as big enough, but here goes: we typically have 3000 concurrent TLS/SSL connections on each Perdition server during peak hours (although we occasionally see 5000), but the CPU impact is negligible[1]. at peak, 8% system and 12% user out of 400% CPU available (this is

Re: Cyrus 2.3.7 Replication Question

2006-07-12 Thread Robert Mueller
It won't. Cyrus currently only supports unidirectional replication. You can make it work (we do), but you need 2 separate instances of cyrus on each machine, which basically means 2 start/stop scripts, 2 different ports/ips for each service, 2 imapd.confs, 2 cyrus.confs, and lots of -C

Re: Cyrus 2.3.7 Replication Question

2006-07-12 Thread Robert Mueller
You can make it work (we do), but you need 2 separate instances of cyrus on each machine, which basically means 2 start/stop scripts, 2 different ports/ips for each service, 2 imapd.confs, 2 cyrus.confs, and lots of -C command line params to everything so it uses the right conf file. to avoid

Re: Cyrus 2.3.7 Replication Question

2006-07-12 Thread Robert Mueller
Can you detail this A - B replication setup more? I've been trying to figure out how to make use of the new 2.3.7 replication features outside of a murder. The thing I can't wrap my head around is the following situation: 1) Server A dies 2) Users are sent to Server B via DNS redirects 3)

Re: Cyrus 2.3.7 Replication Question

2006-07-12 Thread Robert Mueller
Interesting. This process also implies you want to stop incoming mail for a period of time to both servers. Sound right? I would say the process for failover is: 1. Server A is master (sync_client) replicating to Server B (sync_server) 2. Server A dies/is stopped 3. All IMAP/POP/LMTP

Does anyone else see skiplist recovery errors?

2006-06-14 Thread Robert Mueller
I'm trying to find out if anyone else sees intermittent skiplist recovery problems as sometimes we do, usually after a cyrus restart. Our particular setup is cyrus 2.3.6, but we've seen this problem with everything from 2.1 onwards. This is on on x86 with linux 2.6.x, various versions. We use

Re: What version of BDB are people using?

2006-06-12 Thread Robert Mueller
Actually been over this on the list before. It's completely unrelated to that issue. It's not like some connections get through and some don't. It's completely unreproducible and *NO* POP3 connections get through at all until a reboot. One thing to try, when you connect, see if a new

What version of BDB are people using?

2006-06-09 Thread Robert Mueller
I'm just trying to get an informal survey of which version or Berkeley DB people are using successfully in large cyrus environments. We're currently using: db4-4.2.52-3.1 - old redhat based machines libdb4.2.52-18 - newer debian based machines Both of them seem to be a bit flakey. We only use

Re: Cyrus Patches used at FastMail.FM

2006-05-28 Thread Robert Mueller
accept the message, store it as is and ignore the stuff you don't understand when building indexes. And ignore it when using the indexes. Yes, but where's the patch to do that? I'll have a guess that something like this already happens. My guess is that the charset mapping tables ignore

Re: Cyrus Patches used at FastMail.FM

2006-05-26 Thread Robert Mueller
Comments on the non-local ones: - cyrus-8bit-2.3.3.diff Was it not the position held by CMU in the last five years or so that one such patch would *have* do to it properly and re-encode the header to some default charset, in order not to corrupt the store with lack of charset information (for

Some part of bug from 2.3.4 still in 2.3.5

2006-05-25 Thread Robert Mueller
Hi Ken There's still a serious bug in 2.3.5 that causes copied messages to be pseudo invisible in the moved to folder. I can reproduce it as follows: . select inbox.Archive * FLAGS (\Answered \Flagged \Draft \Deleted \Seen hasnoatt selected medeleted hasatt) * OK [PERMANENTFLAGS (\Answered

Re: Cyrus IMAPd 2.3.4 Released

2006-05-25 Thread Robert Mueller
Is this a good spot to suggest branching from the last stable release (2.3.3 in this case) and just applying the bugfixes if HEAD is only half finished? 2.3 is beta branch. You want stable you go for 2.2. Beta is may be, but still 1. There's no regression testing with cyrus at all. I

Re: Cyrus IMAPd 2.3.4 Released

2006-05-25 Thread Robert Mueller
I agree that a test suite is something that we need. I'll add this to our TODO list. As always, code contributions are always welcome. See below and attached, a copy of what I sent april '04. It's not great, but it's a start... Rob Hi Ken Rob Given that there's currently no

Re: Cyrus Patches used at FastMail.FM

2006-05-23 Thread Robert Mueller
Cool, some of the patches look really interesting and I'm considering to include one or the other into my rpm packages. For example the statuscache patch seems very nice. Just to be sure, are there any license restrictions on the patches? No, no license restrictions. We'd love these to get

Re: Cyrus Patches used at FastMail.FM

2006-05-23 Thread Robert Mueller
Statuscache also piqued my interestDoes it give any win for POP3 clients? We've a *HUGE* number of Outlook users that have the terribly wrong idea that they just MUST poll every minute. That along with the seen state stuff would be a good thing for us. No, it's only for IMAP clients

Re: Cyrus Patches used at FastMail.FM

2006-05-23 Thread Robert Mueller
I know we discussed this in the past, but I can't seem to find the thread. What part of the existing STATUS code causes the bottleneck? Is it STATUS_RECENT and STATUS_UNSEEN? It's a combination of both. The main things are: 1. Status UNSEEN and RECENT both have to loop over the cyrus.index

cyrus 2.3.3 change + solved: permission denied setting flags

2006-05-18 Thread Robert Mueller
Ok, we've tracked down the permissions problem. It has to do with what I believe is a change in behaviour between 2.3 CVS 6 months ago and 2.3.3. Cyrus 2.3 from CVS about 6 months ago... . select ... . OK [READ-WRITE] Completed . examine ... . OK [READ-ONLY] Completed Cyrus

Cyrus virtual domain replication...

2006-04-30 Thread Robert Mueller
I'm interested in the new replication stuff, which is why I want to do the upgrade ... I'm not entirely sure that replication plays nice with virtdomains yet. IIRC, there is a bug in bugzilla for this. So just checking, is there anyone out there currently using cyrus with virtual domains

Re: plus addressing to a mailbox with a space in its name

2006-04-06 Thread Robert Mueller
Has anyone had success with using plus addressing to send to a specific mail folder when the folder name has a space in it? With cyrus 2.3 you can enable: lmtp_fuzzy_mailbox_match: 1 From imapd.conf lmtp_fuzzy_mailbox_match: 0 If enabled, and the mailbox specified in the

BDB and errors...

2006-03-14 Thread Robert Mueller
We're using cyrus 2.3 and everything works fine, except we seem to have intermittent problems with BDB 4.2 (specifically the RPM db4-4.2.52-3.1). We only use BDB for the delivery db. In general it works fine, however if for some reason a server has crashed and we reboot the server, we then

Re: BDB and errors...

2006-03-14 Thread Robert Mueller
I have no real idea what could cause this but I have the following sequence in my db conversion script which is used by the init script in my rpms. The procedure is the best according do lots of my tests using different version of db3 and db4 with cyrus-imapd. As you can see I first try a

Re: improving concurrency/performance

2005-11-06 Thread Robert Mueller
In our experience FS-wise, ReiserFS is the worst performer between ext3, XFS e ReiserFS (with tailBLAH turned on or off) for a Cyrus Backend (1M mailboxes in 3 partitions per backend, 0.5TB each partition). Interesting ... can you provide some numbers, even from memory? I'd also be

Difference between . list * % and . list * *

2004-01-30 Thread Robert Mueller
This is cyrus 2.2.3 with skiplist mailboxes db. Someone pointed this strange behaviour out to me... . list * * * LIST (\HasChildren) . INBOX * LIST (\HasNoChildren) . INBOX.Folder1 * LIST (\HasChildren) . INBOX.Folder2 * LIST (\HasChildren) . INBOX.Folder2.SubFolder1 . list * % * LIST

Slow status calls...

2003-09-18 Thread Robert Mueller
I was noticing that IMAP 'status' calls to update the message counts in our web interface seemed to be taking quite a bit of time. I ran a stand alone script to check this, and found: Time: 0.002, msgs:12, unseen:10, folder: INBOX.spam Time: 0.004, msgs: 18978, unseen: 18978, folder: