On 08/14/2011 04:50 AM, Robin Horforth wrote: > I "preload" all of the imported mailboxes into RAM via cat * > > /dev/null to reduce the "read cost" to near zero.
I don't think reading the raw mbox files will pose much of a bottleneck. >> A load below 1 indeed sounds suspicious. > > I should have obtained mysql process status, but I only saw one > mysqld worker process during the operation. Mysqld will never show more than one worker process. It's multi-threaded, not forking. mysqladmin processlist should be more informative. > Should I configure innodb differently? Doubtful. Again, the low system load seems to indicate low concurrency. Message insertion is quite intensive on the database. Lots of inserts going on on multiple tables. > Do you have a performance testing framework or set of scripts? I > have the ImapTest suite, but that's more for validating compliance > with the imap4r1 spec than performance. I know about MSTONE > http://mstone.sourceforge.net/ but a tool that targets areas that > developers suspect/worry are weak would seen sensible. There's all the stuff in test-scripts/. But most of that is for regressions, not load. Only real load-tester is testlmtp.py. The torture script was for finding leaks in 2.2. It would need serious updating. I've played with mstone, but never got around to driving development based on it's result. imaptest is another story. That one has helped a *lot* making dbmail a lot more compliant. All Hail Timo! >> I need to keep thread-concurrency at 1 (read: one) to keep >> throughput at acceptable levels with IMAP concurrencies around >> 50-100 connections, POP3 around 10-20, with an added 5 LMTP >> connections for insertions. > > Yuck, are you serious? Maybe splitting the table across multiple > spindles would help. Maybe. But above situation is just my emergency backup when my main bare-iron server goes down (flaky sata backplane). I also suspect that mysql/innodb is not quite mvcc since I've noticed a lot of read-locks blocking on write locks, leading to stampeding herd problems on slow disks: xen host using local sata storage. > I can try a "bare iron" test machine today. What is your "blessed" > or preferred Linux Distro where you do your development and > regression testing? I have no distro religious issues, though being > a control freak, I will usually run "home" to the minimalistic > comfort of Slackware. I do all development on ubuntu desktops. Main servers run debian. But I don't think it should matter. >> That said: dbmail will never be able to achieve the kind of numbers >> you mention for dovecot or panda. The single-instance storage used >> for both mime-blobs and header values, and fully indexed headers >> prevent it. DBMail insertion is not about creating simple files in >> directories. > > I appreciate this difference. I realise that with Dovecot's scheme, > they defer indexing until a user does an IMAP SEARCH TEXT/BODY, > making the user wait a bit for that initial search before updating > the index. Maybe dbmail should have done the same when I first did the header caching tables.... Deferred indexing would speed things up a lot. -- ________________________________________________________________ Paul J Stevens pjstevns @ gmail, twitter, skype, linkedin * Premium Hosting Services and Web Application Consultancy * www.nfg.nl/[email protected]/+31.85.877.99.97 ________________________________________________________________ _______________________________________________ DBmail mailing list [email protected] http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
