On 04/07/2011 12:45 PM, Stan Hoeppner wrote:
Kádár Tamás (KTamas) put forth on 4/7/2011 9:33 AM:
Hi
Okay so we've been having this issue since forever and I figured why
the heck not ask it here since it's mostly related to dovecot. The
thing is, we have a huge amount of public folders (at the moment,
around 1100). Now, with dovecot indexing and caching we're mostly
okay, also being careful with things like Thunderbird 3's default
'Download everything' option ("Keep messages for this account on this
computer") and such. However once in a while, someone goes rogue, we
install a new version of thunderbird, someone accidentally sets up an
email account in thunderbird mistakingly leaving the download
everything option on. This causes high IO on our server, and a single
user can quickly more or less kill the whole server in mere minutes,
load average quickly spiking to 30-40-50 and everything becomes
sloooooow (obviously).
Is there any way I can limit the amount of IO a certain user can use?
TTBOMK Dovecot has no data rate limiting controls, neither globally nor
per user. As I see it you have a few of options:
1. Switch to mdbox or mbox storage format to decrease IOs per email
2. Beef up the server with many more RAID spindles
3. Eliminate all unnecessary filesystem metadata and logging activity
for instance, atime, if not done already
4. Switch to a filesystem matched to your workload EXT3/4/Reiser are
not optimal for high concurrency multi-user server workloads
switch to XFS or JFS if currently using EXT3/4
5. Install a traffic shaper in front of, or on, the Dovecot server
Configure it to clamp any TCP sessions that exceed, say, 10 Mbit/s
for more than 10 seconds down to 1 Mbit/s for a 1 minute duration.
Tune to taste until you get the desired results.
Option 5 is the most sure fire way to solve the problem you describe.
It is the most direct solution, and likely free (not including your time
to set it up), assuming netfilter etc scripts are available to
accomplish this.
I liked option 5 as being the best choice for this, at first.
Then I began to wonder, when a client does this, is there a process
created for each mail directory? I'm not familiar enough with the IMAP
protocol or dovecot to know off hand. If so though, could that be a problem?
Expecting that Timo has probably addressed this already (he's very
thorough to say the least), I looked through the (v2.0.11) configuration
parameters, and found:
#mail_max_userip_connections = 10
in the 20-imap.conf file. This leads me to believe that yes, there can
reasonably be multiple processes per user. With 1100 folders to
download, I can see where the downloading of all that might overload a
system. I would guess that reducing this value somewhat would tend to
limit the impact any one user could have on the system as a whole. At
the same time however, overall performance may suffer the lower this
value is.
BL, your bottleneck (high load) might be more related to the number of
processes the user has at once, more so than the i/o or bandwidth
demands they're causing. Reduce the # of processes, and the others go
down too.
As a quick and easy remedy, I think I'd try adjusting this value some.
--
-Eric 'shubes'
