Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Bron Gondwana via Info-cyrus]

On Fri, Jun 10, 2016, at 09:41, Jason L Tibbitts III wrote:
> > "BG" == Bron Gondwana  writes:
> 
> BG> Just to be really clear what this is.  It's per mailbox name - if
> BG> you create and delete the SAME mailbox more 20 times, it only keeps
> BG> the most recent 20 of that mailbox.
> 
> Hmm.  That's much less problematic, but it still allows someone to force
> something to be deleted if they really want it to be deleted.  That's
> not really an issue for me because my users wouldn't figure it out, but
> I can imagine that someone using delayed expiry to easily implement some
> sort of legal requirement might be unhappy.  But that's somewhat of a
> stretch.

Yep.

Anyway, magic numbers are bad, so I will make this configurable.  It's easy to
do, and if people with different systems need it changed, then that's fine.

With uniqueid based storage it will all be nicer anyway :)

Bron.


-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Jason L Tibbitts III via Info-cyrus]

> "BG" == Bron Gondwana  writes:

BG> Just to be really clear what this is.  It's per mailbox name - if
BG> you create and delete the SAME mailbox more 20 times, it only keeps
BG> the most recent 20 of that mailbox.

Hmm.  That's much less problematic, but it still allows someone to force
something to be deleted if they really want it to be deleted.  That's
not really an issue for me because my users wouldn't figure it out, but
I can imagine that someone using delayed expiry to easily implement some
sort of legal requirement might be unhappy.  But that's somewhat of a
stretch.

 - J<

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Bron Gondwana via Info-cyrus]

On Fri, Jun 10, 2016, at 04:38, Jason L Tibbitts III wrote:
> > "BG" == Bron Gondwana via Info-cyrus  
> > writes:
> 
> BG> How would you suggest we protect against exploiting delayed delete
> BG> to fill the server without going over quota?
> 
> Well, I don't even run quotas.  But I do keep deleted messages around
> for 12 weeks, and even if I didn't, I do delete accounts occasionally.
> Deleting one account would go over the limit, and though I suck the
> messages out to mbox format for the final archiving, an instant nuke of
> older mailboxes would prevent an "easy" restore.
> 
> BG> Maybe a new quota
> BG> field for "total mailbox usage including deleted stuff" that can be
> BG> set to a high enough value that no reasonable user will ever hit it?
> 
> As long as I can just set it to 'unlimited', I don't care.  Disk is
> cheap and I don't have enough users to worry about it.  But I've had
> people delete all 100+ of their mailboxes before, and come screaming.

Just to be really clear what this is.  It's per mailbox name - if you create 
and delete
the SAME mailbox more 20 times, it only keeps the most recent 20 of that 
mailbox.

If you accidentally delete 100 mailboxes, they'll all still be there.

And it doesn't stop you deleting mailboxes or anything - it just immediately 
cleans
up the 21st one when you delete the mailbox again.

Bron.

-- 
  Bron Gondwana
  br...@fastmail.fm

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Jason L Tibbitts III via Info-cyrus]

> "BG" == Bron Gondwana via Info-cyrus  
> writes:

BG> How would you suggest we protect against exploiting delayed delete
BG> to fill the server without going over quota?

Well, I don't even run quotas.  But I do keep deleted messages around
for 12 weeks, and even if I didn't, I do delete accounts occasionally.
Deleting one account would go over the limit, and though I suck the
messages out to mbox format for the final archiving, an instant nuke of
older mailboxes would prevent an "easy" restore.

BG> Maybe a new quota
BG> field for "total mailbox usage including deleted stuff" that can be
BG> set to a high enough value that no reasonable user will ever hit it?

As long as I can just set it to 'unlimited', I don't care.  Disk is
cheap and I don't have enough users to worry about it.  But I've had
people delete all 100+ of their mailboxes before, and come screaming.

 - J<

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Andre Felipe Machado via Info-cyrus]

Andrew Morgan  wrote ..
> On Thu, 9 Jun 2016, Andre Felipe Machado via Info-cyrus wrote:
> 
> > Bron Gondwana via Info-cyrus  wrote ..
> >> On Thu, Jun 9, 2016, at 03:02, Andre Felipe Machado via Info-cyrus wrote:
> >>> Hello,
> >>> At future release notes I read
> >>> "Under delete_mode: delayed, only the 20 most recently deleted mailboxes 
> >>> are
> >> kept for any given name."
> >>> https://cyrusimap.org/imap/release-notes/3.0/x/3.0.0-beta2.html
> >>> Is there any configuration parameter to increase this limit?
> >>> Why this limit is needed?
> >>
> >> denial of service / space wastage protection.  There's no config option 
> >> available
> >> right now.  I could be convinced to change it.
> >>
> >> How would you suggest we protect against exploiting delayed delete to fill 
> >> the
> >> server without going over quota?  Maybe a new quota field for "total 
> >> mailbox
> usage
> >> including deleted stuff" that can be set to a high enough value that no 
> >> reasonable
> >> user will ever hit it?
> >>
> >> Bron.
> >>
> >> --
> >>   Bron Gondwana
> >>   br...@fastmail.fm
> >> 
> >
> > Hello, Bron
> > I understand the problem.
> > But at a corporate scenario, it is a rare event, because of jobs at stake, 
> > tracked
> user accounts,  antispam measures, etc.
> > It is more likely a "rogue" client,  bug/misconfiguration on a smartphone 
> > causing
> such problems.
> > We stay with official debian repositories versions as long as we could, 
> > receiving
> security patches.
> > So, mantaining an unofficial patch will be a big problem.
> > The sysadmin configurable parameters will be a more elegant solution.
> > Having configurations at sysadmin control will mantain cyrus flexible for 
> > use
> at different usage scenarios.
> > For the DoS / waste space problems, the 2 quota limits configurations are 
> > more
> suitable than counting folders quantity.
> > What if each folder contains 1 TB deleted messages?
> > Maybe a reasonable default (10 times user quota?) for those not wanting to 
> > configure
> is good idea.
> > Even better to have also a way to control individual accounts total quotas, 
> > for
> those corporate accounts like "sa...@foo.bar" that  receive lots of legitimate
> emails and have to
> > delete them after processing.
> > We have zabbix monitoring space at our cyrus backends, and need unlimited  
> > or
> configurable delayed expunge limits for recovering messages and folders for 
> years
> at corporate
> > scenario.
> > Thanks .
> > Andre Felipe
> 
> Remember, this is a limit on the number of deleted *mailboxes* kept, not 
> messages.
> 
> Bron, this could impact Pine/Alpine users that frequently postpone 
> messages.  Pine creates a folder named "postponed-msgs" to store drafts. 
> The folder is created when a draft is saved and deleted when all drafts 
> have been deleted/sent.
> 
> Here is my personal deleted folders list, right now:
> 
> DELETED.user.morgan.postponed-msgs.5755CF0C 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.5755F446 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.5755F486 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.5755F4D1 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.5755F4E4 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.5755F50E 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.5755F65F 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.5755F844 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.5756ECFC 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.5756F602 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.575706F8 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.57585C5D 0 p2 morgan lrswipkxtecda
> DELETED.user.morgan.postponed-msgs.57587FE1 0 p2 morgan lrswipkxtecda
> 
> We are removing deleted mailboxes after 7 days:
> 
> delprune  cmd="/usr/local/cyrus/bin/cyr_expire -E 1 -X 7 -D 7" at=0100
> 
> 
> I don't know if other IMAP clients have similar quirky behavior, but I 
> could see myself running into this limit.  However, I certainly don't care 
> about recovering my old postponed-msgs mailboxes.
> 
> Hmmm, is this a limit per-mailbox (user.morgan.postponed-msgs) or per-user 
> (all mailboxes under user.morgan)?
> 
> Thanks,
>   Andy


Hello, Andrew
Yes, I am aware of being mailboxes limit. This causes it to be even less 
suitable for the intended DoS/waste space control than the 2 quotas idea and 
less yet at corporate 
scenario. And there are the individual total quota idea to evaluate.
We observed that there are corporate users that organize their inboxes at 
extreme levels, containing dozens of folders, classified by project, by 
department, by date, by subject, by 
sender, etc. 
Sometimes their reorganize / delete many of 

Re: prefork and IPv6

[Andrew Morgan via Info-cyrus]

On Thu, 9 Jun 2016, Wolfgang Breyha via Info-cyrus wrote:


Hi!

I recently wondered why some of my preforked processes on my murder backends
never get used. I detected them because some quite old lmtpd's were holding
locks on an already deleted deliver.db.

After some debugging I recognized that cyrus-master seems to fork the
configured amount of "prefork" daemons twice. One half listening on IPv4 and
the other half on IPv6. Since IPv6 is practically never used from our
frontends they stay forever doing nothing on the backends.

Is there some reasonable way to prevent this other than setting prefork=0?

I'm only using SERVICE entries like:
 Bimap  cmd="imapd" listen="imap" prefork=5

Only the port is used for listen= without interface/IP.


Use the proto argument:

  proto=tcp
The protocol used for this service (tcp,  tcp4,  tcp6,  udp,  udp4,  udp6). 
  This
string argument is optional.

tcp4, udp4: These arguments are used to bind the service to IPv4 only.
tcp6,  udp6:  These  arguments  are  used to bind the service to IPv6 only, 
if the
operating system supports this.
tcp, udp: These arguments are used to bind to both IPv4 and IPv6 if 
possible.


Here is my cyrus.conf entry:

  imap   cmd="/usr/local/cyrus/bin/imapd" listen="imap" proto="tcp4" prefork=10 
maxchild=4000


Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Andrew Morgan via Info-cyrus]

On Thu, 9 Jun 2016, Andre Felipe Machado via Info-cyrus wrote:


Bron Gondwana via Info-cyrus  wrote ..

On Thu, Jun 9, 2016, at 03:02, Andre Felipe Machado via Info-cyrus wrote:

Hello,
At future release notes I read
"Under delete_mode: delayed, only the 20 most recently deleted mailboxes are

kept for any given name."

https://cyrusimap.org/imap/release-notes/3.0/x/3.0.0-beta2.html
Is there any configuration parameter to increase this limit?
Why this limit is needed?


denial of service / space wastage protection.  There's no config option 
available
right now.  I could be convinced to change it.

How would you suggest we protect against exploiting delayed delete to fill the
server without going over quota?  Maybe a new quota field for "total mailbox 
usage
including deleted stuff" that can be set to a high enough value that no 
reasonable
user will ever hit it?

Bron.

--
  Bron Gondwana
  br...@fastmail.fm



Hello, Bron
I understand the problem.
But at a corporate scenario, it is a rare event, because of jobs at stake, 
tracked user accounts,  antispam measures, etc.
It is more likely a "rogue" client,  bug/misconfiguration on a smartphone 
causing such problems.
We stay with official debian repositories versions as long as we could, 
receiving security patches.
So, mantaining an unofficial patch will be a big problem.
The sysadmin configurable parameters will be a more elegant solution.
Having configurations at sysadmin control will mantain cyrus flexible for use 
at different usage scenarios.
For the DoS / waste space problems, the 2 quota limits configurations are more 
suitable than counting folders quantity.
What if each folder contains 1 TB deleted messages?
Maybe a reasonable default (10 times user quota?) for those not wanting to 
configure is good idea.
Even better to have also a way to control individual accounts total quotas, for those 
corporate accounts like "sa...@foo.bar" that  receive lots of legitimate emails 
and have to
delete them after processing.
We have zabbix monitoring space at our cyrus backends, and need unlimited  or 
configurable delayed expunge limits for recovering messages and folders for 
years at corporate
scenario.
Thanks .
Andre Felipe


Remember, this is a limit on the number of deleted *mailboxes* kept, not 
messages.


Bron, this could impact Pine/Alpine users that frequently postpone 
messages.  Pine creates a folder named "postponed-msgs" to store drafts. 
The folder is created when a draft is saved and deleted when all drafts 
have been deleted/sent.


Here is my personal deleted folders list, right now:

DELETED.user.morgan.postponed-msgs.5755CF0C 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F446 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F486 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F4D1 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F4E4 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F50E 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F65F 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5755F844 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5756ECFC 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.5756F602 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.575706F8 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.57585C5D 0 p2 morgan lrswipkxtecda
DELETED.user.morgan.postponed-msgs.57587FE1 0 p2 morgan lrswipkxtecda

We are removing deleted mailboxes after 7 days:

delprune  cmd="/usr/local/cyrus/bin/cyr_expire -E 1 -X 7 -D 7" at=0100


I don't know if other IMAP clients have similar quirky behavior, but I 
could see myself running into this limit.  However, I certainly don't care 
about recovering my old postponed-msgs mailboxes.


Hmmm, is this a limit per-mailbox (user.morgan.postponed-msgs) or per-user 
(all mailboxes under user.morgan)?


Thanks,
Andy

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

prefork and IPv6

[Wolfgang Breyha via Info-cyrus]

Hi!

I recently wondered why some of my preforked processes on my murder backends
never get used. I detected them because some quite old lmtpd's were holding
locks on an already deleted deliver.db.

After some debugging I recognized that cyrus-master seems to fork the
configured amount of "prefork" daemons twice. One half listening on IPv4 and
the other half on IPv6. Since IPv6 is practically never used from our
frontends they stay forever doing nothing on the backends.

Is there some reasonable way to prevent this other than setting prefork=0?

I'm only using SERVICE entries like:
  Bimap cmd="imapd" listen="imap" prefork=5

Only the port is used for listen= without interface/IP.

Greetings, Wolfgang
-- 
Wolfgang Breyha  | http://www.blafasel.at/
Vienna University Computer Center | Austria


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [cyrus 3.0] 20 delayed mailbox deleted limit?

[Andre Felipe Machado via Info-cyrus]

Bron Gondwana via Info-cyrus  wrote ..
> On Thu, Jun 9, 2016, at 03:02, Andre Felipe Machado via Info-cyrus wrote:
> > Hello,
> > At future release notes I read
> > "Under delete_mode: delayed, only the 20 most recently deleted mailboxes are
> kept for any given name."
> > https://cyrusimap.org/imap/release-notes/3.0/x/3.0.0-beta2.html
> > Is there any configuration parameter to increase this limit?
> > Why this limit is needed?
> 
> denial of service / space wastage protection.  There's no config option 
> available
> right now.  I could be convinced to change it.
> 
> How would you suggest we protect against exploiting delayed delete to fill the
> server without going over quota?  Maybe a new quota field for "total mailbox 
> usage
> including deleted stuff" that can be set to a high enough value that no 
> reasonable
> user will ever hit it?
> 
> Bron.
> 
> -- 
>   Bron Gondwana
>   br...@fastmail.fm
> 

Hello, Bron
I understand the problem.
But at a corporate scenario, it is a rare event, because of jobs at stake, 
tracked user accounts,  antispam measures, etc.
It is more likely a "rogue" client,  bug/misconfiguration on a smartphone 
causing such problems.
We stay with official debian repositories versions as long as we could, 
receiving security patches.
So, mantaining an unofficial patch will be a big problem.
The sysadmin configurable parameters will be a more elegant solution.
Having configurations at sysadmin control will mantain cyrus flexible for use 
at different usage scenarios.
For the DoS / waste space problems, the 2 quota limits configurations are more 
suitable than counting folders quantity.
What if each folder contains 1 TB deleted messages?
Maybe a reasonable default (10 times user quota?) for those not wanting to 
configure is good idea.
Even better to have also a way to control individual accounts total quotas, for 
those corporate accounts like "sa...@foo.bar" that  receive lots of legitimate 
emails and have to 
delete them after processing.
We have zabbix monitoring space at our cyrus backends, and need unlimited  or 
configurable delayed expunge limits for recovering messages and folders for 
years at corporate 
scenario.
Thanks .
Andre Felipe

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus