Re: How to limit Apple Mail (desktop)?

[María Arrea]

El 31/10/17 a las 15:30, Robert Schetterer escribió:

Am 31.10.2017 um 14:06 schrieb María Arrea:

    If you use Linux you can use netfilter (iptables) mark feature to
track individual connections from a given IP, and above a threshold you
can shape the bandwitdh based on those iptables mark. For example, if an
IMAP connection from ip X exceed 100 megabytes of data, bandwitdh for
that ip would be limited to 100 KB/sec (or whatever number). First 100
megabytes, top speed, above that they get slower.

dont think this will work
tons of cons can come from a nat ip
so you would punish everyone behind that ip ,not only apple mail


Not true. You set a connection tracking using ip+port, not only IP.



i see no real solution for the problem, cause if the server works out
some slow down this would only lead to longer download times, but will
no solve the orig wanted task. It will all be workarounds for a "buggy"
client



    Original post talked about "unbearable server load". Shapping 
network traffic will definitely help with that.



    María.

Re: How to limit Apple Mail (desktop)?

[Joseph Tam]

"@lbutlr"  writes:

(Are you the OP, or have I mistakenly atributed this to Rupert Gallagher?)


So what the composition of all this traffic? Are you saying the mail
client is ultra dumb and repeatedly downloading entire messages, read
and unread, attachment and all (i.e.  you're truly bandwidth limited?)


It most certainly does not do that.


If this is the case for the OP, it's back to square one to find log
entries, and/or nfsstat/iostat/vmstat/etc output to see what is "killing"
the server.  File locking issues, mail_max_userip_connections and process
limit values could also cause service flakiness.  Apple mail readers
seems to gobble up IMAP connections like candy.

Joseph Tam 

Re: How to limit Apple Mail (desktop)?

[@lbutlr]

On Oct 31, 2017, at 5:01 PM, Joseph Tam  wrote:
> Rupert Gallagher writes:
> 
>> By default, Apple Mail downloads all e-mails from? server's account.
>> Previous versions of this client allowed to opt-out.  The latest two
>> versions? however, only allow to opt-out from downloading the
>> attachments. The stress on the server is unbearable.  We cannot ask users to 
>> be
>> considerate: this is the default behaviour of Apple Mail.
> 
> (I'm coming into this thread late.)
> 
> I'm having a hard time visualizing the cause of your problem.  Most big 
> messages have the bulk of their data as attachments (which you state the mail 
> reader can defer downloading).  Also, readers usually cache local copies, so 
> they're downloaded once.

Yep.

> So what the composition of all this traffic?  Are you saying the mail client 
> is ultra dumb and repeatedly downloading entire messages, read and unread, 
> attachment and all (i.e. you're truly bandwidth limited?)

It most certainly does not do that.

> Do your users typically have many concurrent readers for the same mailbox?

Most of my users have 3 devices, a desktop, an iPhone, and an iPad. It doesn't 
seem to cause any issues.

-- 
Apple broke AppleScripting signatures in Mail.app, so no random signatures.

Re: How to limit Apple Mail (desktop)?

[Joseph Tam]

Rupert Gallagher writes:


By default, Apple Mail downloads all e-mails from? server's account.
Previous versions of this client allowed to opt-out.  The latest two
versions? however, only allow to opt-out from downloading the
attachments. 


The stress on the server is unbearable.  We cannot ask users to be
considerate: this is the default behaviour of Apple Mail.


(I'm coming into this thread late.)

I'm having a hard time visualizing the cause of your problem.  Most big
messages have the bulk of their data as attachments (which you state the
mail reader can defer downloading).  Also, readers usually cache local
copies, so they're downloaded once.

So what the composition of all this traffic?  Are you saying the mail
client is ultra dumb and repeatedly downloading entire messages, read
and unread, attachment and all (i.e. you're truly bandwidth limited?)
Do your users typically have many concurrent readers for the same mailbox?
What kind of session numbers are you supporting? 100s? 1000s? 10K+?

Your data outflow has an incoming counterpart: if you're having a
hard time serving it out, is your MTA also having a hard time with the
incoming volume?  Can it simply be the case you've underscaled your
remote mail service?

Joseph Tam 

Re: How to limit Apple Mail (desktop)?

[Rupert Gallagher]

I think I am not mistaken in saying that the original purpose of IMAP was to 
access a remote database, vs POP's approach to download it. When the IMAP 
client opens a folder, the server uploads the *index* of its content. When the 
client requests a specific item, the server uploads the item while keeping the 
original. If the client renames a folder, the server just renames the folder.

Apple Mail behaves like a POP client that wants to download everything while 
keeping the db on the server. Further down Apple's madness, if the client 
renames a folder, Apple mail asks the server to upload the full content of the 
"new" folder. Batshit crazy!

The very best therapy here is, in my opinion, to serve Apple Mail's request for 
mass download as if it were a regular IMAP index request, if technically 
feasible. If it is not feasible, then the alternative is not to rate limit the 
connection, but to ban Apple Mail entirely.

Sent from ProtonMail Mobile

On Tue, Oct 31, 2017 at 2:46 PM, Rupert Gallagher  wrote:

> Aki, the IMAP client can receive the e-mails with an empty body without any 
> damage. This is how IMAP works normally. The full body is queried again by 
> the client when reading the e-mail for real.
>
> On Tue, Oct 31, 2017 at 1:52 PM, Aki Tuomi  wrote:
>
 What's in your mind as solution?
>
>>> When dovecot receives many full body downloads from a client, it could 
>>> respond by sending the header only.
>
>> This sounds rather dangerous. Client is expecting full body download, not 
>> headers. Aki

Re: How to limit Apple Mail (desktop)?

[Robert Schetterer]

Am 31.10.2017 um 14:06 schrieb María Arrea:
> 
>    If you use Linux you can use netfilter (iptables) mark feature to
> track individual connections from a given IP, and above a threshold you
> can shape the bandwitdh based on those iptables mark. For example, if an
> IMAP connection from ip X exceed 100 megabytes of data, bandwitdh for
> that ip would be limited to 100 KB/sec (or whatever number). First 100
> megabytes, top speed, above that they get slower.

dont think this will work
tons of cons can come from a nat ip
so you would punish everyone behind that ip ,not only apple mail

i see no real solution for the problem, cause if the server works out
some slow down this would only lead to longer download times, but will
no solve the orig wanted task. It will all be workarounds for a "buggy"
client




> 
>     Regards
> 
>     María
> 
> 
> El 30/10/17 a las 10:38, Rupert Gallagher escribió:
>> By default, Apple Mail downloads all e-mails from  server's account.
>> Previous versions of this client allowed to opt-out. The latest two
>> versions? however, only allow to opt-out from downloading the
>> attachments.
>>
>> The stress on the server is unbearable. We cannot ask users to be
>> considerate: this is the default behaviour of Apple Mail.
>>
>> We need a server-side solution to the problem.
>>
>> Please share your ideas.



Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: How to limit Apple Mail (desktop)?

[Rupert Gallagher]

Aki, the IMAP client can receive the e-mails with an empty body without any 
damage. This is how IMAP works normally. The full body is queried again by the 
client when reading the e-mail for real.

On Tue, Oct 31, 2017 at 1:52 PM, Aki Tuomi  wrote:

>>> What's in your mind as solution?

>> When dovecot receives many full body downloads from a client, it could 
>> respond by sending the header only.

> This sounds rather dangerous. Client is expecting full body download, not 
> headers. Aki

Re: How to limit Apple Mail (desktop)?

[María Arrea]

   If you use Linux you can use netfilter (iptables) mark feature to 
track individual connections from a given IP, and above a threshold you 
can shape the bandwitdh based on those iptables mark. For example, if an 
IMAP connection from ip X exceed 100 megabytes of data, bandwitdh for 
that ip would be limited to 100 KB/sec (or whatever number). First 100 
megabytes, top speed, above that they get slower.


    Regards

    María


El 30/10/17 a las 10:38, Rupert Gallagher escribió:

By default, Apple Mail downloads all e-mails from  server's account. Previous 
versions of this client allowed to opt-out. The latest two versions? however, 
only allow to opt-out from downloading the attachments.

The stress on the server is unbearable. We cannot ask users to be considerate: 
this is the default behaviour of Apple Mail.

We need a server-side solution to the problem.

Please share your ideas.

Re: How to limit Apple Mail (desktop)?

[Sami Ketola]

> On 31 Oct 2017, at 14.44, Rupert Gallagher  wrote:
> 
>> What's in your mind as solution?
> 
> When dovecot receives many full body downloads from a client, it could 
> respond by sending the header only.


That would not be nice. Breaking IMAP protocol that way. Some rate limiting 
would be better solution.

Sami

Re: How to limit Apple Mail (desktop)?

[Aki Tuomi]

On 31.10.2017 14:44, Rupert Gallagher wrote:
>> What's in your mind as solution?
> When dovecot receives many full body downloads from a client, it could 
> respond by sending the header only.
This sounds rather dangerous. Client is expecting full body download,
not headers.

Aki

> Sent from ProtonMail Mobile
>
> On Tue, Oct 31, 2017 at 11:29 AM,  wrote:
>
>> Hi, On 30.10.2017 10:38, Rupert Gallagher wrote: > We need a server-side 
>> solution to the problem. more powerfull hardware? What's in your mind as 
>> solution? Don't tell the client about the mails -> user will miss old mails. 
>> Stop him from fetching body with a temporary "UNAVAILABLE" failure? -> Don't 
>> know how apple mail will react, but probably present some error to the user 
>> for every mail. The only possibility that would make sense to me is some 
>> form of rate limiting to reduce the server load. Give them some tens of 
>> Megabytes w/o any restriction, then reduce. Regards Moritz

Re: How to limit Apple Mail (desktop)?

[Rupert Gallagher]

> What's in your mind as solution?

When dovecot receives many full body downloads from a client, it could respond 
by sending the header only.

Sent from ProtonMail Mobile

On Tue, Oct 31, 2017 at 11:29 AM,  wrote:

> Hi, On 30.10.2017 10:38, Rupert Gallagher wrote: > We need a server-side 
> solution to the problem. more powerfull hardware? What's in your mind as 
> solution? Don't tell the client about the mails -> user will miss old mails. 
> Stop him from fetching body with a temporary "UNAVAILABLE" failure? -> Don't 
> know how apple mail will react, but probably present some error to the user 
> for every mail. The only possibility that would make sense to me is some form 
> of rate limiting to reduce the server load. Give them some tens of Megabytes 
> w/o any restriction, then reduce. Regards Moritz

Re: How to limit Apple Mail (desktop)?

[ml+dovecot]

Hi,

On 30.10.2017 10:38, Rupert Gallagher wrote:

We need a server-side solution to the problem.


more powerfull hardware?

What's in your mind as solution?

Don't tell the client about the mails -> user will miss old mails.

Stop him from fetching body with a temporary "UNAVAILABLE" failure? -> 
Don't know how apple mail will react, but probably present some error to 
the user for every mail.


The only possibility that would make sense to me is some form of rate 
limiting to reduce the server load. Give them some tens of Megabytes w/o 
any restriction, then reduce.


Regards
Moritz

Re: How to limit Apple Mail (desktop)?

[Sami Ketola]

> On 31 Oct 2017, at 8.42, Rupert Gallagher  wrote:
> 
> When Apple Mail connects to an IMAP account for the very first time, it 
> downloads all e-mails to build a local mirror.
> 
> When the user changes the name of a folder, Apple Mail downloads the whole 
> subtree and erases the old one.
> 
> We have Apple Mail users with >20GB worth of e-mails, downloaded multiple 
> times (horrified emoticon here).
> 
> Sent from ProtonMail Mobile


I can’t think of any easy way to rate limit Apple mail. Only thing that crossed 
my mind is to enable ID response
logging and then with some logwatch process use some traffic shaping solution 
to limit the bandwith of that particular
connection after some period of time.

Sami

Re: How to limit Apple Mail (desktop)?

[Rupert Gallagher]

When Apple Mail connects to an IMAP account for the very first time, it 
downloads all e-mails to build a local mirror.

When the user changes the name of a folder, Apple Mail downloads the whole 
subtree and erases the old one.

We have Apple Mail users with >20GB worth of e-mails, downloaded multiple times 
(horrified emoticon here).

Sent from ProtonMail Mobile

On Mon, Oct 30, 2017 at 10:38 AM, Rupert Gallagher  wrote:

> By default, Apple Mail downloads all e-mails from  server's account. Previous 
> versions of this client allowed to opt-out. The latest two versions? however, 
> only allow to opt-out from downloading the attachments.
>
> The stress on the server is unbearable. We cannot ask users to be 
> considerate: this is the default behaviour of Apple Mail.
>
> We need a server-side solution to the problem.
>
> Please share your ideas.

Re: How to limit Apple Mail (desktop)?

[Robert Schetterer]

Am 30.10.2017 um 17:50 schrieb Robert Schetterer:
> Am 30.10.2017 um 10:38 schrieb Rupert Gallagher:
>> By default, Apple Mail downloads all e-mails from  server's account. 
>> Previous versions of this client allowed to opt-out. The latest two 
>> versions? however, only allow to opt-out from downloading the attachments.
>>
>> The stress on the server is unbearable. We cannot ask users to be 
>> considerate: this is the default behaviour of Apple Mail.
>>
>> We need a server-side solution to the problem.
>>
>> Please share your ideas.
>>
> 
> first check if you can ident Apple Mail versions related, i recent have
> none in my  log , but i.e Android does
> 
> ---log
> ID sent: name=com.samsung.android.email.provider, os=android,
> os-version=7.0; NRD90M, vendor=samsung, x-android-device-model=SM-G930F
> -
> 
> then you need a procedure for limiti have no idea which one
> 
> Best Regards
> MfG Robert Schetterer
> 

I dont know Apple Mail very good, but as workaround you might use sieve
to presort mails on the server at incomming in (sub)folders which arent
synced at default i.e in a date named folder, for sure users ( you
should know your apple mail users ) then need configure an extra
subscribe on these folders. Perhaps a combi with virtual folders may
usefull , for now not better idea , iam nearly sure this is not what you
you expected and/or wanted

seems google has a feature to "hide" mails

see

https://www.guidingtech.com/44581/prevent-mail-app-space-mac/

however if feel very strange with this

Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: How to limit Apple Mail (desktop)?

[Robert Schetterer]

Am 30.10.2017 um 10:38 schrieb Rupert Gallagher:
> By default, Apple Mail downloads all e-mails from  server's account. Previous 
> versions of this client allowed to opt-out. The latest two versions? however, 
> only allow to opt-out from downloading the attachments.
> 
> The stress on the server is unbearable. We cannot ask users to be 
> considerate: this is the default behaviour of Apple Mail.
> 
> We need a server-side solution to the problem.
> 
> Please share your ideas.
> 

first check if you can ident Apple Mail versions related, i recent have
none in my  log , but i.e Android does

---log
ID sent: name=com.samsung.android.email.provider, os=android,
os-version=7.0; NRD90M, vendor=samsung, x-android-device-model=SM-G930F
-

then you need a procedure for limiti have no idea which one

Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

How to limit Apple Mail (desktop)?

[Rupert Gallagher]

By default, Apple Mail downloads all e-mails from  server's account. Previous 
versions of this client allowed to opt-out. The latest two versions? however, 
only allow to opt-out from downloading the attachments.

The stress on the server is unbearable. We cannot ask users to be considerate: 
this is the default behaviour of Apple Mail.

We need a server-side solution to the problem.

Please share your ideas.