On Wed, 15 Feb 2017 08:43:23 +0530 Rajesh M wrote: > christian > > the servers i currently own are dell servers. The servers i plan to buy are > Dell R530, 2U rack servers with 8 x 3.5 inch drives, with 64 gb ram each, > Hardware raid. I am thinking of 2 X 300 gb ssds raid1 and 6 x 2 tb drives in > raid10 for data. I do not have any experience in setting up drdb (that would > be my next step) ... primarily using standalone servers with hardware level > redundancy. For backup-ups i rsync data to backup servers, also use qmail > taps for backing up sent/recd emails on a per user basis. > > i am using mysql authentication backend > my mail server is qmailtoaster, ie with vpopmail, spamdyke, simscan. > > you had mentioned as follows > ############## > With 4000 users and 5 logins/s I can't imagine anything that wouldn't be > able to: > a) put all of the relevant data into RAM and > b) thus handle auth requests in very timely manner. > The example I gave earlier about my servers with 140 logins/s > authenticates against LDAP and isn't very busy doing so while also > fitting all 2.7GB of data for about 1 million users into RAM. > ############## > > >>>>>>>>>>>>>>>>>>how do i put all relevant data into the RAM ? > By configuring MySQL accordingly of course, for example: --- my.cnf: # Set buffer pool size to 50-80% of your computer's memory, # but make sure on Linux x86 total memory usage is < 2GB innodb_buffer_pool_size=8G innodb_additional_mem_pool_size=32M --- --- top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 39045 mysql 20 0 9170m 784m 10m S 0 2.4 3:38.69 mysqld ---- --- tuning-primer: INNODB STATUS Current InnoDB index space = 49 M Current InnoDB data space = 109 M Current InnoDB buffer pool free = 98 % Current innodb_buffer_pool_size = 8.00 G Depending on how much space your innodb indexes take up it may be safe to increase this value to up to 2 / 3 of total system memory --- Christian > thanks > rajesh > > > ----- Original Message ----- > From: Christian Balzer [mailto:ch...@gol.com] > To: dovecot@dovecot.org > Sent: Tue, 14 Feb 2017 16:40:01 +0900 > Subject: > > > Hello, > > On Mon, 13 Feb 2017 22:53:23 +0530 Rajesh M wrote: > > > thanks for your help > > > > happy to say that the performance dramatically improved after i use the > > high performance settings from here > > http://wiki.dovecot.org/LoginProcess > > > That's why that page is there. > > > grep Login: /var/log/mail.log.1 |wc -l > > with the mail.log being of a typical, busy day. > > 412992 > > > So about 5 login/s, not that busy, but that of course also depends on your > HW and authentication backend. > > > i also picked up the imap and pop3 connections during peak hours > > [root@ns1 domains]# doveadm who | awk '/imap/{m+=$2}/pop3/{n+=$2}END{print > > m,n}' > > username # proto (pids) > > (ips) > > > > max figures i got was 535 for imap and 97 for pop > > > > > > Question 1 > > i wish to improve the performance further by caching the logins. current > > the same is kept disable because when user's change passwords then they are > > not able to immediately login with the new password for some time. How to > > solve this issue. > > > There is no solution, caching auth data is by default fraught with that > sort of peril. > > And how did you determine that caching auth data would actually improve > things? > > What is your authentication backend? > > With 4000 users and 5 logins/s I can't imagine anything that wouldn't be > able to: > a) put all of the relevant data into RAM and > b) thus handle auth requests in very timely manner. > > The example I gave earlier about my servers with 140 logins/s > authenticates against LDAP and isn't very busy doing so while also > fitting all 2.7GB of data for about 1 million users into RAM. > > > > > > > further to the above. > > i am planning to put up a large mail server with one of the following > > options > > > Don't plan 1, always plan a pair and then use whatever you're comfortable > with to turn them into a HA cluster, like Pacemaker with DRBD. > > > option 1 > > OS and mailqueue : 2 X 600 gb 15k rpm raid 1 > What kind of RAID, HW or MD? > > > data : 4 X 2000 gb raid 10 for data > > backup : 2 X 2000 gb raid 10 for backup > > While with Linux MD a 2 disk RAID10 would be possible, it wouldn't be > particular helpful (slower than a RAID 1 with writes). > > > 32 gb ram > > > > > > option 2 > > os and mail queue : 4 X 600 gb 15k rpm raid 10 > Those 15k HDDs tend to be expensive. > Estimate your expected writes per day (iostat on an existing server can > give you some hints) and get 2 SSDs instead, DC S3520s if 1 DWPD is > sufficient, DC S3610s if you need 3 DWPD (doubtful), S3710 if you need 10 > DWPD (very doubtful). > Similar for Samsung DC level SSDs, check their specs. > > For example, my busiest mailbox servers only write about 250KB/s averaged, > which mean about 50GB/day and thus would be a fit even for a small S3520. > I'm still using 200GB S3710s because I can afford them, like the speed and > never ever want to worry about wearout. > > > data : 4 X 2000 gb raid 10 for data > > 32 gb ram > > > > i am having the OS and mail queue on primary drives > > > > i understand that Raid10 give a faster write and read whereas Raid 1 will > > have slow writes and fast reads > > > While that is true, you're missing the real difference here, the number of > HDDs. With 4 of them in the RAID10 they will be twice as fast as the 2 > disk RAID1. > This can be tweaked even further with the various RAID 10 layout options, > "man md". > > > the question is will there be huge performance difference between Raid1 and > > Raid 10 when it comes small files like mail queue ? > > > Again as explained above, double the IOPS, so yes. > > Christian > > > > thanks > > rajesh > > ----- Original Message ----- > > From: Christian Balzer [mailto:ch...@gol.com] > > To: dovecot@dovecot.org > > Cc: ke...@my.walr.us > > Sent: Mon, 13 Feb 2017 10:46:13 +0900 > > Subject: > > > > > > Hello, > > > > On Sun, 12 Feb 2017 08:27:21 -0500 KT Walrus wrote: > > > > > Thanks for the info. I do have one further question for you. On your > > > servers that are currently handling 50k IMAP sessions, how many users > > > does that correspond to? Since many users will have multiple IMAP > > > sessions on multiple devices, I’d like to hear about some real-world > > > numbers that could be used for budgeting a new project like mine. > > > > > > > Those servers would actually be the wrong ones to look at, as they are > > primarily accessed by the aforementioned broken client, so the numbers are > > somewhat skewed. > > However looking at other servers with a more "normal" user spread things > > aren't actually too different. > > > > The average number of sessions per user tends to be 2. > > The clear majority (over 50%) only has one session open (people with well > > behaved and configured clients watching the INBOX mostly). > > Another 30% has 2 sessions open, again the typical state of this would be > > clients watching another mailbox besides INBOX, typically SENT. > > The rest has 3 and more sessions open. > > > > The number of sessions could of course be drastically reduced if any > > client would support IMAP NOTIFY, alas none that I'm aware of do. > > > > Lastly no more than 60% of the session seem to be in IDLE at any given > > time, so my comments about RAM and IMAP hibernation effectiveness stand. > > > > > Also, do you use Dovecot IMAP proxies in front of your backend servers? > > > If so, how many IMAP sessions can one proxy server handle (assuming the > > > proxy does authorization using MySQL running on a separate server)? And, > > > could the proxy server be tuned to help in optimizing mostly IDLE backend > > > sessions? > > > > > > > Yes to Dovecot Proxying, of course. > > > > No idea about MySQL, with (Open)LDAP nothing is breaking a sweat at an > > average of 140 logins per second (IMAP and POP) on the 2 proxy servers. > > If you can fit your entire dataset into RAM it should be fine, my LDAP > > servers fall into that category and take about 10% of a slow (1.2GHz, 34%, > > power-save mode) core only to handle the that load (also 2 servers). > > And the rate of logins/s is what you need to worry about most and optimize > > for. > > > > The proxies will of course have to do the shuffling of data and SSL > > en/de-coding, but again they're not particular busy with that. > > > > The number of sessions comes into play when looking at the number of login > > processes on the proxies and their memory footprint. > > An IMAP login process on the proxies in performance mode with a client > > limit of 1000 will consume about 55MB at most. > > So assume at least 55KB RAM per session. > > > > Read Timo's mail I linked to about IMAP hibernation, AFAIK nothing has > > happened to make proxies more supportive for this though. > > > > Christian > > > > On Feb 12, 2017, at 1:58 AM, Christian Balzer <ch...@gol.com> wrote: > > > > > > > > > > > > Hello, > > > > > > > > On Fri, 10 Feb 2017 14:50:03 -0500 KT Walrus wrote: > > > > > > > >>> 1. 256GB of real RAM, swap is for chums. > > > >> > > > >> Are you sure that 100,000 IMAP sessions wouldn’t work well with SWAP, > > > >> especially with fast SSD storage (which is a lot cheaper than RAM)? > > > >> > > > > > > > > I'm sure about tax and death, not much else. > > > > > > > > But as a rule of thumb I'd avoid swapping out stuff on production > > > > servers, > > > > even if it were to SSDs. > > > > Incidentally the servers I'm talking about here have their OS and swap > > > > on > > > > Intel DC S3710s (200GB) and the actual storage on plenty of 1.6TB DC > > > > S3610s. > > > > > > > > Relying on the kernel to make swap decisions is likely to result in much > > > > reduced performance even with fast SWAP when you're overcommitting > > > > things > > > > on that scale. > > > > > > > > > > > > But read on. > > > > > > > >> Seems that these IMAP processes are long lived processes (idling most > > > >> of the time) that don’t need that much of the contents of real memory > > > >> available for much of the life of the process. I use a database proxy > > > >> in front of MySQL (for my web apps) so that there can be a large > > > >> number of TCP connections to the proxy where the frontend requests are > > > >> queued for execution using a small number of backend connections. > > > >> > > > >> Could Dovecot IMAP be re-written to be more efficient so it works more > > > >> like MySQL (or other scalable data servers) that could handle a > > > >> million or more IMAP sessions on a server with 32GBs or less of RAM? > > > >> Those IMAP sessions aren’t doing much most of the time and shouldn’t > > > >> really average 2MB of active data per session that needs to be > > > >> resident in main memory at all times. > > > >> > > > > See IMAP hibernation: > > > > https://www.mail-archive.com/dovecot@dovecot.org/msg63429.html > > > > <https://www.mail-archive.com/dovecot@dovecot.org/msg63429.html> > > > > > > > > I'm going to deploy/test this in production in about 2 months from now, > > > > but if you look at the link and the consequent changelog entries you'll > > > > see > > > > that it has certain shortcomings and bug fixes in pretty much each > > > > release > > > > after it was introduced. > > > > > > > > But this is the correct way to tackle things, not SWAP. > > > > > > > > Alas I'm not expecting miracles and if more than 20% of the IMAP > > > > sessions > > > > here will be hibernated at any given time I'd be pleasantly surprised. > > > > > > > > Because between: > > > > > > > > 1. Finding a sensible imap_hibernate_timeout. > > > > > > > > 2. Having well behaved clients that keep idling instead of restarting > > > > the > > > > sequence > > > > (https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/ > > > > <https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/>) > > > > > > > > 3. Having lots of mobile clients who either get disconnected (invisible > > > > to > > > > Dovecot) or have aggressive IDLE timers to overcome carrier NAT timeouts > > > > (a large mobile carrier here times out idle TCP sessions after 2 > > > > minutes, > > > > forcing people to use 1 minute IDLE renewals, making 1. up there a > > > > nightmare). > > > > > > > > 4. Having really broken clients (don't ask, I can't tell) which open > > > > IMAP > > > > sessions, don't put them into IDLE and thus having them expire after 30 > > > > minutes. > > > > > > > > the pool of eligible IDLE sessions isn't as big as it could be, in my > > > > case > > > > at least. > > > > > > > >> My mail server isn’t that large yet as I haven’t fully deployed > > > >> Dovecot outside my own small group yet, but it would be nice if > > > >> scaling Dovecot IMAP to millions of users wasn’t limited to 50,000 > > > >> IMAP sessions on a server... > > > >> > > > > > > > > Scaling up is nice and desirable from a cost (rack space, HW) > > > > perspective, > > > > but the scalability of things OTHER than Dovecot as I pointed out plus > > > > that little detail of failure domains (do you really want half of your > > > > eggs in one basket?) argue for scaling out after a certain density. > > > > > > > > I'm feeling my way there at this time, but expect more than 100k > > > > sessions > > > > per server to be tricky. > > > > > > > > Lastly, when I asked about 500k sessions per server here not so long > > > > ago, > > > > ( http://www.dovecot.org/list/dovecot/2016-November/106284.html > > > > <http://www.dovecot.org/list/dovecot/2016-November/106284.html> ) > > > > Timo mentioned that he's not aware of anybody doing more than 50k per > > > > server, something I got licked already and definitely will go to 100k > > > > eventually. > > > > > > > > Regards, > > > > > > > > Christian > > > >>> On Feb 10, 2017, at 11:07 AM, Christian Balzer <ch...@gol.com> wrote: > > > >>> > > > >>> On Fri, 10 Feb 2017 07:59:52 -0500 KT Walrus wrote: > > > >>> > > > >>>>> 1500 IMAP sessions will eat up about 3GB alone. > > > >>>> > > > >>>> Are you saying that Dovecot needs 2MB of physical memory per IMAP > > > >>>> session? > > > >>>> > > > >>> That depends on the IMAP session, read the mailbox size and index > > > >>> size, > > > >>> etc. > > > >>> Some are significantly larger: > > > >>> --- > > > >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > >>> > > > >>> 1033864 mail 20 0 97600 67m 54m S 0 0.1 0:01.15 imap > > > >>> > > > >>> --- > > > >>> > > > >>> But yes, as somebody who has mailbox servers with 55k+ session the > > > >>> average > > > >>> is around 1.6MB. > > > >>> > > > >>>> If I want to support a max 100,000 IMAP sessions per server, I > > > >>>> should configure the server to have at least 200GBs of SWAP? > > > >>>> > > > >>> You will want: > > > >> > > > >>> 2. Understanding how to tune Dovecot and more importantly the overall > > > >>> system to such a task (see that PID up there?). > > > >>> 3. Be willing to deal with stuff like top and ps taking ages to > > > >>> start/run > > > >>> and others like atop actually killing dovecot (performance wise, not > > > >>> literally) when doing their obviously flawed cleanup on exit. Some > > > >>> things > > > >>> clearly do NOT scale well. > > > >>> > > > >>> My current goal is to have 100k capable servers that work well, 200k > > > >>> in a > > > >>> failover scenario, but that won't be particular enjoyable. > > > >>> > > > >>> Christian > > > >>> > > > >>>>> On Feb 10, 2017, at 3:58 AM, Christian Balzer <ch...@gol.com> wrote: > > > >>>>> > > > >>>>> On Fri, 10 Feb 2017 01:13:20 +0530 Rajesh M wrote: > > > >>>>> > > > >>>>>> hello > > > >>>>>> > > > >>>>>> could somebody with experience let me know the dovecot config file > > > >>>>>> settings to handle around 1500 simultaneous connections over pop3 > > > >>>>>> and 1500 connection over imap simultaneously. > > > >>>>>> > > > >>>>> > > > >>>>> Be very precise here, you expect to see 1500 as the result of > > > >>>>> "doveadm who |grep pop3 |wc -l"? > > > >>>>> > > > >>>>> Because that implies an ungodly number of POP3 connects per second, > > > >>>>> given > > > >>>>> the typically short duration of these. > > > >>>>> > > > >>>>> 1500 IMAP connections (note that frequently a client will have more > > > >>>>> than > > > >>>>> the INBOX open and thus have more than one session and thus process > > > >>>>> on the > > > >>>>> server) are a much easier proposition, provided they are of the > > > >>>>> typical > > > >>>>> long lasting type. > > > >>>>> > > > >>>>> So can you put a number to your expected logins per second (both > > > >>>>> protocols)? > > > >>>>> > > > >>>>>> my server > > > >>>>>> > > > >>>>>> server configuration > > > >>>>>> hex core processor, 16 gb ram 1 X 600 gb 15 k rpm for main drive > > > >>>>>> and 2 X 2000 > > > >>>>>> gb hdd for data (No raid) > > > >>>>>> > > > >>>>> No RAID and no other replication like DRBD? > > > >>>>> Why would you even bother? > > > >>>>> > > > >>>>> How many users/mailboxes in total with what quota? > > > >>>>> > > > >>>>> 1500 IMAP sessions will eat up about 3GB alone. > > > >>>>> You will want more memory, simply to keep all relevant SLAB bits > > > >>>>> (inodes, > > > >>>>> dentries) in RAM. > > > >>>>> > > > >>>>> If you really have several hundreds logins/s, you're facing several > > > >>>>> bottlenecks: > > > >>>>> 1. Login processes themselves (easily fixed by high performance > > > >>>>> mode) > > > >>>>> 2. Auth processes (that will depend on your backends, method mostly) > > > >>>>> 3. Dovecot master process (spawning mail processes) > > > >>>>> > > > >>>>> The later is a single-threaded process, so it will benefit from a > > > >>>>> faster > > > >>>>> CPU core. > > > >>>>> It can be dramatically improved by enabling process re-usage, see: > > > >>>>> http://wiki.dovecot.org/PerformanceTuning > > > >>>>> > > > >>>>> However that also means more memory usage. > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> Christian > > > >>>>> > > > >>>>>> > > > >>>>>> thanks > > > >>>>>> rajesh > > > >>>>>> > > > >>>>> > > > >>>>> [snip] > > > >>>>> -- > > > >>>>> Christian Balzer Network/Systems Engineer > > > >>>>> ch...@gol.com Global OnLine Japan/Rakuten Communications > > > >>>>> http://www.gol.com/ > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> Christian Balzer Network/Systems Engineer > > > >>> ch...@gol.com <mailto:ch...@gol.com> <mailto:ch...@gol.com > > > >>> <mailto:ch...@gol.com>> Global OnLine Japan/Rakuten Communications > > > >>> http://www.gol.com/ <http://www.gol.com/> <http://www.gol.com/ > > > >>> <http://www.gol.com/>> > > > > > > > > > > > > -- > > > > Christian Balzer Network/Systems Engineer > > > > ch...@gol.com <mailto:ch...@gol.com> Global OnLine Japan/Rakuten > > > > Communications > > > > http://www.gol.com/ <http://www.gol.com/> > > > > > > > > > -- Christian Balzer Network/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
