Re: squatter segfaulting

[Paolo Cravero]

Hello.

> Vanilla Cyrus, no Xapian. I don't know how to run a debug-enable, I only know 
> the basics :/
> 
> ii  cyrus-imapd   2.5.10-3+deb9u1
> amd64Cyrus mail system - IMAP support
> 

There seems to be an open issue: 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871921

I've seen squatter segfaulting when the index file got too large (in an older 
Cyrus version). Can you check how large it is? How many messages are in the 
mailbox/folder causing the crash?

How about system's ulimit? Can the cyrus user open enough files?

Paolo

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: ACL on root mailbox

[Paolo Cravero]

Hello Albert.

> I would like to known what kind of ACL you give on a shared mailbox. My
> problem is :
> 
>   I have some users who are admin on the mailbox, it's needed because those
>   admin-user can create sub folder.
> 
>   But some user make mistake time to time and move the mailbox under some
>   other mailbox

[...]

> So how can I give enough rights to someone to let he manage their mailbox
> but prevent this kind of wrong manipulation.

See 
https://www.cyrusimap.org/imap/reference/admin/access-control/rights-reference.html
 for an explanation of ACL levels.

I think "lrswc" could do in your case, for non-admin users. You may add "i" 
and/or "t" and "e", depending on the way they operate on the shared mailbox 
(copy mails into | delete mail from the folder).

Greetings,
Paolo

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Why Cyrus?

[Paolo Cravero]

My 2 cents...


>  I have been asked to provide support to a Rube Goldberg style of mail setup 
> and trying to track the installation and configuration of dovecot in a 
> multidomain, virtual-user setup was crazy.
>  It seemed like there were dozens of config files spread over the file system.

I agree on this. It took a me a lot of trial-and-error when I tried to 
replicate in Dovecot our multidomain virtual-user Cyrus setup. Also the use of 
ACLs to share mailboxes was different and I couldn't converge to an identical 
behavior. 

>  I believe Cyrus was designed to solve the mailstore problem at scale from 
> the outset.

With 26 TB of messages stored on Cyrus mailstores in individual files, backups 
can be a nightmare. I really liked the storage bins offered by Dovecot and the 
automatic archiving to lower-performance disks when I played with it, because 
it would cut by a factor of 10-100-1000 the number of files to consider.

As I said: my 2 cents.
Paolo

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

squatter lower limits

[Paolo Cravero]

Hello.

While looking to do low-level disk usage optimization, some simple performance 
tests relied on full-text searches (2.4 branch). Metadata always resides on 
local disks, while messages are on slower hardware.

I noticed that full-text searches with short strings take much longer than 
longer text. For example, a FT search on 3 letters takes >60" while a 9-letter 
long string on the same corpus lasts ~20". These tests have been repeated over 
and over again to exclude disk caching being the culprit: reversing the search 
order - longer first - has no impact.

So I opened up the cyrus source code and looked for search-related code. As I 
understand it, squatter is not used if the search string is shorter than 4 
symbols. From squat.h it's quite clear:

/*
Don't change this unless you're SURE you know what you're doing.
Its only effect on the API is that searches for strings that are
shorter than SQUAT_WORD_SIZE are not allowed.
In SQUAT, a 'word' simply refers to a string of SQUAT_WORD_SIZE
arbitrary bytes.
*/

#define SQUAT_WORD_SIZE 4

So, question to who knows the squatter implementation in cyrus: is this lower 
limit applied to all searches? Body, subject, addresse(s)?

And, does this lower bound still apply to 3.0 branch and the new indexing 
engine Xapian?

Let alone low level disk compression or optimization, a client might not handle 
well long search times without receiving data on the IMAP channel and dismiss 
the connection (or a network device could do it). So, if searching for short 
strings means reading all raw message files, I should warn users through the 
client interface of possible failures since the mail corpus keeps growing and 
growing and growing. That's until we upgrade to 3.0, it that helps.

Thanks,

Paolo

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Sieve RFC5490 checks user quota usage

[Paolo Cravero]

Il 21 marzo 2017 alle 19.19 Nic Bernstein ha scritto:Paolo,Attached is the script we've been using for ages for this purpose. Ouruses LDAP as a source of user ID information, but it could be easilyadapted for other AAA databases.Thank you Nic. That's a good alternative.My request about doing it with sieve was to mimick Sun JES/Messaging behaviour that sent the overrquota warning upon new message receipt (only once) or, in case of no incoming traffic, on user login (not with sieve in this case, obviously).On a multiserver deployment with specialised systems (cyruses on their own machine, postfixes on another, imap proxies on third systems), Merlin's suggested approach is not practical.So, dealing locally with the output of quota helper tool is still the way to go. Simple. Stable. Flexible. Available.Paolo
 

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Sieve RFC5490 checks user quota usage

[Paolo Cravero]

There is no mailbox annotation that exposes quota that could be used with the mboxmetadata extension (RFC5490) and there is no current Sieve extension for checking quota.Thank you Ken. Given the double negative answer I guess that developing such a feature would require a lot of work, or it is not possible at all.And for Nic, yes, I mean the "IMAP STORAGE quota". I would like to warn the user that his quota is about to fill up through an email, triggered on new mail arrival or login. Why? Because not all clients support reading the quota over IMAP or handling an alert (think of some smartphone IMAP client or an (active)sync system).Is there a way to achieve the same result somehow, with stock cyrus?PaoloOn 03/17/2017 11:01 AM, Paolo Cravero wrote:Hello.I am trying to figure out if sieve, with RFC5490 support, is able to read user's disk quota (used) and act accordingly. I would like to trigger a mail to "self" if quota is above a given percent. Something like a vacation message (so once a day or so), triggered on arrival AND if quota is above X %. If sieve doesn't support this, is there another way to do it? Thanks and have a nice weekend,Paolo 
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.0.0-rc4 released

[Paolo Cravero]

Hello Ellie
 
>  > The Cyrus team is proud to announce the immediate availability of the
 >  > third release candidate from the Cyrus IMAP 3.0 series: 3.0.0-rc4.
 
Thanks to all the team for working bringing further our beloved IMAP server.
 
 
I read the release notes and got caught by this statement:
 
"Under delete_mode: delayed, only the 20 most recently deleted mailboxes are
kept for any given name."
 
Not being a native English speaker I may have not a clear idea of the
meaning of "for any given name", so I ask for a clarification.
 
Is it "20 most recently deleted mailboxes regardless of their name"?
Or is it "20 most recently deleted mailboxes with the same name"? "... same
path"?
Or something else?
 
In practice,
 
1) what happens if thirty users delete one mailbox each, all with different
names?
 
2) what happens if we remove 100 accounts and all their tree (dm
user/.@example.com 100 times)?
 
We rely on keeping deleted messages and mailboxes/folders for few days on
the filesystem so that recoveries do not hit the backup guys, but can be
operated with a simple wrapper to unexpunge.
 
Thank you for clarifications,
Paolo
 
 
 
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Sieve RFC5490 checks user quota usage

[Paolo Cravero]

Hello.
I am trying to figure out if sieve, with RFC5490 support, is able to read
user's disk quota (used) and act accordingly.
 
I would like to trigger a mail to "self" if quota is above a given percent.
Something like a vacation message (so once a day or so), triggered on
arrival AND if quota is above X %.
 
If sieve doesn't support this, is there another way to do it?
 
Thanks and have a nice weekend,
Paolo
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP on NFS

[Paolo Cravero via Info-cyrus]

Hello.

>  I know this is a very tipical question but... nowadays... how does Cyrus
> IMAP work with NFS 4?. It supports file locking and I would only mount
> each mail spool in an own unique server... so
> 
>  I think it should be very similar to using a local filesystem... Does
> anyone can tell some experience in this config?..

It works, but you must know what you're doing. First of all: keep metadata
on a local filesystem because that's where most of the action happens.
Messages can then reside on NFS, and all documented considerations still
hold true: risk of loosing connection, network bottleneck, FS going
read-only. There may be more, but in today's world of virtual servers it is
hard to think of local storage as "physically attached set of disks" and
something can go wrong in many hidden places anyway.

If the partition is tied to just one server most probably you will not have
contention problems. Troubles occur if the NFS goes away (ie not remounted
after reboot) and Cyrus rebuilds the tree under local disks, so you must be
quick to intercept that condition.

NFS can be an option to keep online old messages, especially if it costs
much less than your other storage. Since those messages are usually accessed
very rarely, network is not a real bottleneck and you are left with TCP/IP
issues. But do keep metadata (and indexes) on local disks.

>From our storage admins: "One advantage of having most of the
obsolete-but-keep-it-i-might-need-it-one-day data not on a local filesystem
is that in the rare event of system crash you will not have to wait for fsck
to complete, as it must be done for local disks. If you have TB's it makes
quite a difference."


Paolo
(sitting on few NFS spools)

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: how to expunge DELETE.user.username from mailboxes db ?

[Paolo Cravero via Info-cyrus]

 
  
   Il 9 dicembre 2015 alle 10.54 Riccardo Veraldi via Info-cyrus  ha scritto:Hello,after removing a mailbox with cyradm (dm user.username).I have plenty of DELETED.user.username in the mailboxes database.
  
  You should have delayed_delete enabled. And probably delayed_expunge too. Check your imapd.conf for
  delete_mode: delayedexpunge_mode: delayed
  You need to run cyr_expire, either manually or crontabbed within cyrus. See
  https://docs.cyrus.foundation/imap/features/delayed-expunge.html
  https://docs.cyrus.foundation/imap/features/delayed-delete.html
  man 8 cyr_expire
  
  Get ready for some disk I/O activity.
  Paolo
 


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Quota root not removed if folder is on a different partition

[Paolo Cravero]

Hello, I think I found an unexpected feature. Or a bug.

Here's the setup: Cyrus IMAP v2.4.17-Invoca-RPM-2.4.17-6 single instance on
RHEL5.

Amongst other partitions I have an "archive" partition where I attach all
users' "Archive" folders with a separate, very large, quotaroot. Something
like

cm user/me/arch...@example.com archivepartition
sqr user/me/arch...@example.com 123456789

Then the Archive folder is deleted. It goes under user's Trash on the
mailbox primary partition (user/me/Trash/arch...@example.com) and it gains
the default mailbox quota. Delayed delete/expunge is in action, FWIW. So far
so good.

If the Archive folder is created again through a generic IMAP client, it is
created on the mailbox primary partition, which is ok, but it inherits the
original quota root as if it were on the archive partition, which is NOT OK.

It looks like the quotaroot information is not removed when the folder is
deleted, or it is kept without the partition information. Is it related to
delayed delete, perhaps? Or a bug? Maybe fixed in a later release?

Paolo

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Disappearing Mailbox Content

[Paolo Cravero]

Hello.

> A while back we setup a Cyrus install for a client to use as a mail
> repository for customer contact.  They have one account which is shared
> across all users in the office.  Approximate size of the account is 85GB.

Have you checked the size of your cyrus.cache and other metadata files? See
.
Even if the reporting process was squatter, mailbox was behaving
erratically.

Probably not the solution, but worth checking since account is large.
Paolo

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 3.0.0-beta1 delivers mails but doesn't list mailboxes

[Paolo Cravero]

Hello.

> Try lm user/me *

Unfortunately it doesn't seem to work. I tried sereral combinations, some
pseudorandom as well. #1 and #4 should return something, at least they do in
2.4. #2 and #3 follow the pattern you suggested, but nothing.

Should I follow a different rule for creating mailboxes from cyradm? 

tst-msg03.csi.it> lm user/*
tst-msg03.csi.it> lm user/un*   #1
tst-msg03.csi.it> lm user/un * #2
tst-msg03.csi.it> lm uno*
tst-msg03.csi.it> lm uno *
tst-msg03.csi.it> lm user/me * #3
tst-msg03.csi.it> lm user/me* #4
tst-msg03.csi.it> lm user/cyrusadmin
tst-msg03.csi.it> lm user/cyrusadmin *
tst-msg03.csi.it> lm oxcyrus *
tst-msg03.csi.it> lm user/csi.it *
tst-msg03.csi.it> lm user.me *
tst-msg03.csi.it> lm user.uno *
tst-msg03.csi.it> lm user.me*
tst-msg03.csi.it> lm user.cyrusadmin
tst-msg03.csi.it> version
name   : Cyrus IMAPD
version: 3.0.0-beta1 5e06a05d 2015-08-04
vendor : Project Cyrus
support-url: http://www.cyrusimap.org
os : Linux
os-version : 2.6.32-504.el6.x86_64
environment: Built w/Cyrus SASL 2.1.23
 Running w/Cyrus SASL 2.1.23
 Built w/zlib 1.2.3
 Running w/zlib 1.2.3
 CMU Sieve 2.4
 mmap = shared
 lock = fcntl
 nonblock = fcntl
 idle = idled

tst-msg03.csi.it> quit


And the content of mailboxes.db is:

# cyr_dbtool /var/lib/imap/mailboxes.db twoskip show
example.com!user.cyr3^test   %(A %(cyr3.t...@example.com lrswipkxtecdan) P
maildata1 M 1441273381)
example.com!user.cyr3^test.Spam  %(A %(cyr3.t...@example.com lrswipted
anyone p) P maildata1 M 1441273381)
example.com!user.cyr3^test.Trash %(A %(cyr3.t...@example.com
lrswipkxtecdan) P maildata1 M 1441273381)
example.com!user.prova^prova %(A %(prova.pr...@example.com lrswipkxtecdan) P
maildata1 M 1441273381)
example.com!user.strace^1%(A %(strac...@example.com lrswipkxtecdan) P
maildata1 M 1441273381)
example.com!user.uno^archivio%(A %(uno.archi...@example.com
lrswipkxtecdan) P maildata1 M 1441273381)
example.com!user.uno^archivio.Trash  %(A %(uno.archi...@example.com
lrswipkxtecdan) P maildata1 M 1441273381)
etc etc etc

Same result if I specify skiplist in place of twoskip.


> One day I will fix this so you see:
> 
> * user/f...@domain.com
> * user/f...@domain.com/bar
> 
> And at the same time I'm going to ban non-unixhierarchysep and also ban
> the non-alternate namespace. And the world will be a much nicer place.

I agree.

Please advise if this discussion should be moved to another list.
Paolo

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

3.0.0-beta1 delivers mails but doesn't list mailboxes

[Paolo Cravero]

Hi.
I downloaded cyrus-imapd 3.0.0-beta1 and compiled it on a RHEL6.6 machine
(2.6.32-504.el6.x86_64).

This a test server that has never had a Cyrus installed on it. There is a
stale Dovecot installation (off), FWIW.

I have a strange problem. Mails can be delivered via LMTP without errors.
With POP3 I can see and retrieve them. But ALL IMAP activity leads to
nothing, as well as no errors logged whatsoever. Through cyradm I cannot
list mboxes: lm, lm user/*; lq, info user/mailbox return nothing. I straced
the imap process handling cyradm requests, and it stats mailboxes.db.

/var/lib/imap/mailboxes.db has 600 permissions.

The current config is http://pastebin.com/ctaM0rVr . It is adapted from a
working 2.4.17 setup. Basically I reduced the number of partitions.

The source was configured as follows:
./configure  --enable-autocreate --with-ldap --enable-squat
--prefix=/usr/local/cyrus --enable-idled --without-openssl

Authentication is through PAM/saslauthd/LDAP, and is successful both for POP
and IMAP.

I even deleted and recreated mailboxes.db (nothing to loose, it's a test
system) with no positive effect.


Please help me identify what is wrong with this setup or suggest further
checks. I am willing to try the archiving feature introduced in cyrus-imapd
v3.

Thanks,
Paolo

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: 3.0.0-beta1 delivers mails but doesn't list mailboxes

[Paolo Cravero]

Hello.

> I have a suspicion. Can you please post your imapd.conf and cyrus.conf
> files so I
> can have a look. Also what username you're using to authenticate for
> cyradm,
> and whether your usernames have domains on them.

usernames have domains. (m...@example.com, y...@uc.example.com)
cyradm has NO domain. (cyrusadmin)

 cyrus.conf: 

START {
  recover   cmd="ctl_cyrusdb -r"
  idled cmd="idled"
}

SERVICES {
  imap  cmd="imapd" listen="imap" prefork=5 maxchild=5000
  pop3  cmd="pop3d" listen="pop3" prefork=3
  sieve cmd="timsieved" listen="sieve" prefork=0
  lmtp  cmd="lmtpd -a" listen="lmtp" prefork=0
}

EVENTS {
  checkpointcmd="ctl_cyrusdb -c" period=30
  delprune  cmd="cyr_expire -E 3 -D 30 -X 30 -v" at=0100
  squatter  cmd="squatter -s -a -i" at=0200
}


*** imapd.conf *** is up on pastebin. 
configdirectory: /var/lib/imap
partition-maildata1: /var/cyrus/maildata/example.com/maildata1
partition-maildata2: /var/cyrus/maildata/uc.example.com/maildata1
partition-maildatadm: /var/cyrus/maildata/admin.invalid/maildatadm
defaultpartition: maildata1

metapartition_files: header index cache expunge squat
metapartition-maildata1: /var/cyrus/metamaildata1
metapartition-maildata2: /var/cyrus/metamaildata2
metapartition-maildatadm: /var/cyrus/metamaildatadm

altnamespace: 0
serverinfo: on
admins: cyrusadmin
loginuseacl: 1
allowplaintext: 1

delete_mode: delayed
expunge_mode: delayed
annotation_definitions: /etc/annoIMAP.conf

sievedir: /var/lib/imap/sieve
postmaster: postmas...@example.com

sendmail: /usr/sbin/sendmail
fulldirhash: 1
hashimapspool: true
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN
allowapop: 0

quotawarn: 80
statuscache: 1
auditlog: 1
debug: 1
chatty: 1
iolog: 1
commandmintimer: 5
normalizeuid: 1
syslog_prefix: cyrus
unixhierarchysep: 1
autocreate_quota: 0
autocreate_inbox_folders: "Junk | Trash | Sent | Drafts | Archive"
autocreate_users: anyone
implicit_owner_rights: l
autocreate_post: 0
expunge_days: 30
singleinstancestore: 1
defaultdomain: admin.invalid
improved_mboxlist_sort: 1
virtdomains: userid
quotadb: skiplist
specialusealways: 1
xlist-junk: Spam

allowusermoves: 1

sync_log: 1
proxy_authname: xyz  # old config, not using it in 3.0.0-beta1 test setup
proxy_password: ***
unix_group_enable: no

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

How's your server load?

[Paolo Cravero]

Hello,
anyone is willing to comment on this Cyrus server load, or share theirs?

Virtual machine on vSphere 5 with 2x 2 GHz  vCPU;
cyrus-imapd RPM 2.4.16;
kernel 2.6.18 x64_64;
4 GB RAM.

There are 150 simultaneous IMAP connections on average mostly coming 
from a webmail (openXchange), with 240 imapd processes. No POP3. Just 
one instance of cyrus. Messages come over LMTP at a rate of 1 
msg/second, average of course.

Data and metadata partitions are local to the virtual machine (not NFS), 
connected to a SAN via FC.
The system has 20 data partitions (and 20 metadata), for a total of 
3.8TB, of which 550GB are used. Mailboxes (accounts) sum up to 2000.

What is bugging me is the system load that averages at 10 during peak 
hours. CPU usage is equally shared between user/system/idle. As reported 
by vmstat:

us sy id wa st
33 31 32  4  0

These figures obviously vary each second, but don't go too far during 
peak hours.

/proc/meminfo reports:
MemTotal:  4043792 kB
MemFree: 34416 kB
Buffers:305456 kB
Cached:2505672 kB
SwapCached:  0 kB
Active:2393176 kB
Inactive:  1083692 kB
HighTotal:   0 kB
HighFree:0 kB
LowTotal:  4043792 kB
LowFree: 34416 kB
SwapTotal: 4192956 kB
SwapFree:  4192724 kB
Dirty:   86608 kB
Writeback:   0 kB
AnonPages:  665816 kB
Mapped:  95900 kB
Slab:   445736 kB
PageTables:  55056 kB
NFS_Unstable:0 kB
Bounce:  0 kB
CommitLimit:   6214852 kB
Committed_AS:  1018640 kB
VmallocTotal: 34359738367 kB
VmallocUsed:265672 kB
VmallocChunk: 34359472247 kB
HugePages_Total: 0
HugePages_Free:  0
HugePages_Rsvd:  0
Hugepagesize: 2048 kB


The userbase is going to grow. Another group of 3500 users with 2TB/year 
traffic is going to go on a dedicated cyrus server, similar to the one 
detailed above.

Not having a direct experience on large and active userbase under Cyrus, 
the load seems high compared to other solutions we know of. I would 
appreciate any input about the apparent ability of the current server to 
hold more users, or which debugging/characterization steps should be 
undertaken to further optimize our asset.
The Performance Notes page has been studied and applied when possible.

Thank you for reading so far and Season's Greetings, if applies.
Paolo



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Removing sharing of mailbox when granted user is deleted

[Paolo Cravero]

On 27/10/2011 1.07, Andy Bennett wrote:


 Whilst I can't help with your technical problem, I can offer the advice
 that you should think very carefully about reusing user B in the
 future and, therefore deleting user B rather than just disabling them.

 For example, if you reuse the user B@example.net then there's a chance
 that that user will receive mail intended for the old user B.

Sure Andy. Actually re-use of logins is forbidden by our law. But you 
could still have one user coming back to the company in a different 
role, so stale ACLs could still create a problem.

Well, the problem exists even if the user changes role/division without 
interruption (ex. secretary - sales) and should not see old mailboxes 
anymore.

 the social aspects are much wider and once you start going down that
 rabbit hole then you're in for an interesting ride.

We're facing more social aspects too. Like preventing ever growing 
mailboxes, still leaving large quotas and keeping cyrus fluid 
(actually the web client)

Paolo


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Removing sharing of mailbox when granted user is deleted

[Paolo Cravero]

Hi.
I've searched with ACL keyword all my cyrus-info archive and nothing similar
came up in the last 18 months.

The situation is:
- user A shares a mailbox with user B
- user B at some point is deleted

how do I remove the ACL on A's folder for B?

The only way I can think of is a dump of mailboxes.db, grep for B account and
then further removal of the stale ACL. Is there another way to do it?

I know it (probably?) doesn't hurt to keep the broken ACL, but a re-creation of
an account called B would automatically(?) grant access to that folder, 
wouldn't it?

TIA,
Paolo



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Removing sharing of mailbox when granted user is deleted

[Paolo Cravero]

Citazione Dan White dwh...@olp.net:

 On 26/10/11 23:20 +0200, Paolo Cravero wrote:

 The situation is:
 - user A shares a mailbox with user B
 - user B at some point is deleted
 
 how do I remove the ACL on A's folder for B?
 What error are you getting when attempting to remove B's ACLs with cyradm
 (deleteacl)?

Apologies. I used totally wrong words. My question is about a procedure, not an
application error.

The problem is: how do I know which mailboxes B was granted access to, so that I
can remove the ACL(s)?

Like: B is a secretary. A/C/D/E give her access to their mailboxes. That is
under A/C/D/E control, so there is no log of the operation (according to a past
discussion on this list). When B is gone I need to remove all the grants she had
received, but I don't know them.

Can listaccessmailbox be used to filter them out, so that I don't have to
parse through the whole list of mailboxes? Or an IMAP command to get the list,
before B is deleted?

I haven't found any useful hint in the docs.

I reckon I originally asked something else. Once again, apologies.

Paolo



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Ever growing mailboxes and archiving

[Paolo Cravero]

On 18/04/2011 19.41, Bron Gondwana wrote:

 You probably want to add yourself to the CC list on:

 http://bugzilla.cyrusimap.org/show_bug.cgi?id=3384

 Where we're discussing precisely how to implement something like this.

Well, Bron, you ended the discussion pretty fast with an elegant 
solution. :-)

+1
I vote for the feature
etc etc

Add myself to CC on 3384.
Paolo


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Ever growing mailboxes and archiving

[Paolo Cravero]

On 18/04/2011 17.19, Vincent Fox wrote:

 What web-based client?  Why would you expect it to die?

It is Open Xchange. Some users do not organize messages in folders, just 
leave everything in the INBOX. So, depending on the (web)client 
implementation, such a huge number of items can be a performance hog.

For example OX has a cache of some thousand message headers to make 
scrolling smooth. If items in the folder are more than the cache can 
hold, an IMAP LIST is issued at every scrollbar movement...


I have seen Thunderbird choke on a Premier Gmail account with some 
thousands of messages in one folder/label. But that might have well been 
cause by a slow server response.


 As to SAN, this maybe related.

 We have found netbackup on large Cyrus mail-store volumes are a

Curiously I read your reply after greeting Symantec's sales team that 
paid us a visit! So I am now studying netbackup7 on my own.

Thank you for the insight in the problems you're running. I will talk 
with the storage guys here so check how our product is expected to cope 
with many small files (well, it is already doing it with the existing 
email infrastructure anyway).

Paolo



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Ever growing mailboxes and archiving

[Paolo Cravero]

Good morning.
I am looking for a server-side archiving solution to be applied to a 
relatively large cyrus installation.

The customer eats up ca 4 TB of data over about 4000 accounts. No quota.

The competition is: who will die first? The web-based client or our 
SAN+backup system? :-)

I am aware cyrus can handle the folder tree of a single account over 
different cyrus partitions. This might mean more complex maintenance and 
way longer restore times but it does indeed require 100% good user 
behavior in archiving old messages in the proper subtree. Unless 
administratively old messages are moved away once a year (what an impact 
on I/O and backups...).

A good search engine did not return useful links, or I used wrong keywords.

Some commercial email products have solutions to handle old messages or 
just their attachments: is there something that would apply to our 
beloved cyrus?

How do you handle large and rarely accessed old messages?

Thanks,
Paolo

PS: I am not after a regulations compliant archiving, just something to 
offload the main (expensive) storage.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Two cyrus processes, one mail spool

[Paolo Cravero]

Ciro Iriarte wrote:

 Hi, is it safe to run two cyrus processes on the same filesystem/mail
 spool to provide scalability and HA?, in example, two servers
 accessing one clustered filesystem or NFS directory?.

See

http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/CyrusCluster

See also the archives and my message of  2009-09-04 cyrus-imapd in (veritas) 
cluster: anyone? that received no answers (except one private saying they're 
using Sun Cluster).

Then try with more specific questions.

Paolo

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

fyi, problem with compile_sieve on RHEL 5.4 x86_64 (2.3.7-7)

[Paolo Cravero]

For the records, in case someone gets stuck in the same problem.

I installed a test environment x86_64 with RedHat Enterprise Linux 5.4 using 
their official RPMs cyrus-imapd-2.3.7-7.el5.x86_64.rpm and related.

I couldn't get complie_sieve to work, it coredumped right after opening the 
output file for writing.

Solution: compiled invoca's 2.3.14 SRPM.

Since I was still setting up the system I don't know if other 
/usr/lib/cyrus-imapd programs fail.

At the time being there is no more recent RPM on RH official repository.

One more note for x86_64 and PAM authentication with LDAP backend. In 
/etc/pam.d/imap (smtp, pop, sieve, lmtp) files I had to use explicitly 64bit 
libraries:

authsufficient   /lib64/security/pam_ldap.so
account sufficient   /lib64/security/pam_ldap.so

while /lib/security/pam_ldap.so did not log any error but did not authenticate 
at all, not even opened the TCP connection to LDAP server. I'm not that much 
into 64bit yet, so others might find this info useful too.

Paolo


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

cyrus-imapd in (veritas) cluster: anyone?

[Paolo Cravero]

Hi.
I've read Sims message of 2009.08.04 and following replies with interest. 
Unfortunately I couldn't find an answer (in archives) to the following question.

Those using cyrus-imapd in a clustered environment, which cluster environment 
do they use? Veritas, RedHat, ... ?

We're currently running an active-passive cluster with Sun Messaging under 
Veritas. There is a specific Veritas agent for that product that handles 
checks and switchover in case of node failure, but I guess nothing for cyrus.

We will (try to) establish an active-passive cyrus cluster, possibly with 
Veritas. RedHat cluster as second choice. I would appreciate some pointers if 
someone has already been there.

Have a nice weekend,
Paolo Cravero



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Resources for MUA support of IMAP features?

[Paolo Cravero]

Wil Cooley wrote:
 Does anyone know of a mailing list or a web site with information about
 MUA support of various IMAP features? For example, for IMAP IDLE the

I once came across this wiki:
http://www.imapwiki.org/

that links to this http://uplib.parc.com/misc/imapclients.html for your 
specific question.

HTH,
Paolo

PS: nice domain name Wil :)


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

incremental squatter

[Paolo Cravero]

Hi.
Perhaps it is a bug in the documentation.

'man squatter' in the DESCRIPTION says there's no incremental update:

Squatter creates an index of ALL messages in the mailbox, not just those 
since the last time that it was run (i.e.,  it  does  NOT  do  incremental 
updates).

but in the OPTIONS a -i switch mentions incremental.

Is it just a documentation error, so incremental indexing does work in 2.3.14?

TIA,
Paolo



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Removing the Web Changes Notification Service from the wiki

[Paolo Cravero]

Wil Cooley wrote:

 edit a page and see if it could be easily removed. Turns out it's just
 this line:
 
   %INCLUDE{_default.WebNotify}%
 
 Does anyone mind if this is removed from the Cyrus/WebHome page on the
 wiki (and possibly any other pages where I find it)?

I've found that long text still present on several subpages (like 
Specifications and Standards), but I'm not writing because of that. Rather...

http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/WebHome

the last link looks like spam. Or a way to attract more /customers/ ?! :)

Anyone registered on wiki that can remove it?

Paolo


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Compress attachments

[Paolo Cravero]

andreas.moro...@sb-brixen.it wrote:

Hello Andreas,

 Is it possible to configure cyrus the way that it compresses the  
 attachments on the storage ( not for tranfer as in RFC 4978 ) ? I know  

AFAIK there is no cyrus option to do this. Moreover cyrus does not separate 
attachments from the message body, so the whole message would need to be 
gzipped. For each and every message.

 that diskspace is cheap, but when you have to get back thousands of  
 mailboxes from a backup, the smaller the files are the faster the  
 restore is done.

Well, replicate live mailboxes data rather than rely on a single node and 
frequent backups. :-) I'm also starting to love the delayed expunge facility 
with respect to cheap disk space.

Ciao,
Paolo

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: murder and autocreate (I know it is not supported)

[Paolo Cravero]

Thank you Duncan for the info about the patch. I would try it, but I'm sure it 
would not be accepted by the tech stuff as it is not official.

As I said in another thread, this is what happens when autocreate is ON in a 
murder setup with several backends, and a user accesses a shared mailbox 
residing on a different server.

This is an excerpt of backend1 maillog:


canonified backend1 - backend1

mupdate NO response: mailbox already exists

MUPDATE: can't reserve mailbox entry for 'example.com!user.onBE2'

autocreateinbox: User on...@example.com, INBOX failed. unable to reserve 
mailbox on mupdate server

seen_db: user on...@example.com opened 
/var/lib/imap/domain/e/example.com/user/z/onBE2.seen

open: user on...@example.com opened user/onBE1/778899


In words, the backend not owning the mailbox tries to create it, but receives 
a deny from the murder server. Then it opens the target mailbox (onBE1/778899) 
and creates the .seen structure.

So far this means a messy maillog file. I will use autocreate just to populate 
my backends from the backend itself (with some imtest iteration, or whatever) 
and then switch it off.

Looking forward to a MUPDATE protocol update or cyrus official patch.

Paolo



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

murder and cross-backend mailbox sharing

[Paolo Cravero]

Hi.
I have a setup with 1 frontend, 1 murder, 2 backends. I am playing with 
cross-backend mailbox sharing, one of the features we value most.

Problem. While I can access shared mailboxes that reside on the same backend 
just fine, with Thunderbird (1.5/linux) I can see shared folders on other 
backends, but when I subscribe nothing is subscribed.

Thunderbird always connects to the frontend machine. All cyrus servers have 
the same Invoca RPMs (v2.3.14-Invoca-RPM-2.3.14-2). Authentication via PAM 
LDAP, if that matters.

I use virtdomains but all accounts are under the same @domain . ACL are 
anyone lrs.

Summarizing: I can access shared mailboxes on the same backend, but on 
different backends they cannot be subscribed,

Something I should look for?

TIA,
Paolo

Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: murder and cross-backend mailbox sharing

[Paolo Cravero]

Simon Matter wrote:

 I'm not sure it's related but you really should upgrade to the 2.3.14-6
 package. 2.3.14 has a bug which is patched in the 2.3.14-6 RPM.

Thank you Simon. I'm in the staging phase, so still looking for features 
rather than bugs :) BTW, thank you for providing those wonderful RPMs.

Anyway Michael pointed out the missing configuration parameter. Now x-backend 
sharing works and I've also seen in the log why autocreate and murder don't 
work well together. Will post a reply in the proper thread I opened on June 5th.

Paolo


Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

murder and autocreate (I know it is not supported)

[Paolo Cravero]

Hi.
I'm setting up a test environment with frontend, backend, murder (currently 1, 
1, 1 respectively). All using RPMs on recent RedHat systems.

I have read in the docs that the autocreate patch does not work with murder, 
and I have experienced it too. But...

... if I leave the autocreate ON on backends only, and then simulate an IMAP 
access or mail delivery right at the backend level, I should not get into 
troubles, right?

Just to ease the creation of mailboxes, especially during this deployment 
test. Or perhaps even after if our complex tools won't like the cyradm Perl 
interface. (yes, we will have a look at gosa and korreio)

TIA,
Paolo



Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html