Hi Eric, thx for all your help and tests :)
Bye, Norman 2010/5/3 Eric Charles <eric.char...@u-mangate.com>: > Hi Norman, > > I've set ulimit -n 65535 (default was 1024), used -Xmx512m (cfr your last > commit) and connectionBacklog=900 > > lsof -p <pid>| wc -l gives around 550 (stable). > No issue after stop (test with embedded Derby). > > Disk swapping and exceptions came from the 64m limit if you don't set -Xmx > (james is not that memory hungry beast, but 64 is really small). > Will try on real server later this week to rise the throughput. > > Tks, > > Eric > > On 05/02/2010 06:37 PM, Norman Maurer wrote: >> >> Ups I mean >> >> <connectionBacklog>200</connectionBacklog> >> >> in smtpserver.xml >> >> Bye, >> Norman >> >> 2010/5/2 Eric Charles<eric.char...@u-mangate.com>: >> >>> >>> Hi Norman, >>> What do you mean with "...backlock in smtpserver.log" ? >>> Tks, >>> Eric >>> >>> On 05/02/2010 01:15 PM, Norman Maurer wrote: >>> >>>> >>>> Hi Eric, >>>> >>>> for such high load you will need to adjust a few things: >>>> >>>> 1) set a higher ulimit. By default its 1024 on linux which is prolly >>>> not enough ( I use fbsd which use 11095 by default) >>>> 2) set a higher connection backlock in smtpserver.log >>>> >>>> >>>> Bye, >>>> Norman >>>> >>>> >>>> 2010/5/2 Eric Charles<eric.char...@u-mangate.com>: >>>> >>>> >>>>> >>>>> Hi Norman, >>>>> >>>>> Trying to simulate a more real-life test case, I developed a small >>>>> bomber >>>>> class that launches parallel threads. Each thread sends mails with >>>>> attachements (with small pause between each mail sending). >>>>> >>>>> I tested with embedded derby with JDBCDomainList. >>>>> >>>>> Depending on the parameters (10 to 100 parallel threads, 5 ms to 100 ms >>>>> pause, 1KB to 100KB attachement), I could have on my dev laptop 5 to 10 >>>>> mails per second being processed. >>>>> >>>>> The good think is that I have no exception at all during stable phase. >>>>> >>>>> However, I was not able to have more mails being processed on my >>>>> latptop >>>>> due >>>>> to disk access. The CPU and memory are really not charged (20 - 30 %), >>>>> but >>>>> disk is really working non-stop. >>>>> This is probably the price to pay for activemq and the kahadb. >>>>> >>>>> Complete scenario with unstable phase: >>>>> - Bomb 5 minutes >>>>> - Stop bomb. >>>>> - Still some mails are spooled for a while which is quite normal. >>>>> - When all mails are spooled, disk still runs crazy for minutes, >>>>> console >>>>> showing a few messages "Slow KahaDB access: cleanup took 684" >>>>> - rebombing : org.apache.camel.CamelExecutionException: Exception >>>>> occurred >>>>> during execution on the exchange: Exchange[Message: >>>>> org.apache.james.core.maili...@16863f2] Caused by: java.io.IOException: >>>>> Too >>>>> many open files - error message in the bomber >>>>> - Server console show still a few mails being spooled. >>>>> - After, server does not respond on socket - no more exception. >>>>> >>>>> Disk congestion may come from my disk. >>>>> I will setup a server environment with good disk, and will rerun the >>>>> test >>>>> in >>>>> a few days. >>>>> A database on a separate server could also help. >>>>> >>>>> Btw, what should we consider as acceptable/needed as load (xxx mails >>>>> from >>>>> yyy parallel clients) ? >>>>> >>>>> Tks, >>>>> >>>>> Eric >>>>> >>>>> >>>>> >>>>> On 05/01/2010 09:25 AM, Norman Maurer wrote: >>>>> >>>>> >>>>>> >>>>>> About 6000, then I run out of heap space.. >>>>>> >>>>>> Can you maybe try to deploy it with james and bomb it via smtp ? I was >>>>>> not able to see any exception. >>>>>> >>>>>> Thx, >>>>>> Norman >>>>>> >>>>>> >>>>>> 2010/5/1 Eric Charles<eric.char...@u-mangate.com>: >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Hi Norman, >>>>>>> >>>>>>> Similar exceptions with properties.put("openjpa.LockTimeout", >>>>>>> "30000"); >>>>>>> >>>>>>> I think the environment simply can not follow the load (more than one >>>>>>> mail >>>>>>> sent each ms). >>>>>>> Bootleneck can be anywhere (connection creation, database lock, >>>>>>> openjpa,...). >>>>>>> This is why I put a sleep between each mail sending. >>>>>>> Real servers will better hold the pressure. >>>>>>> >>>>>>> So I think I'm limited by my environment : it does not allow me to >>>>>>> have >>>>>>> a >>>>>>> really heavy load to validate that IMAP-137 is resolved. >>>>>>> Well, it is probably, as before you commits, I directly had the >>>>>>> duplicate >>>>>>> key exception. >>>>>>> >>>>>>> How much mails can you hold on your env with h2 inmemory ? >>>>>>> >>>>>>> Tks, >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> >>>>>>> On 05/01/2010 09:08 AM, Norman Maurer wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Hi Eric, >>>>>>>> >>>>>>>> thx for your tests. The order of the uid is caused by the threads, >>>>>>>> thats nothing to worry about. >>>>>>>> >>>>>>>> About the timeout, can you try to add this to the test case: >>>>>>>> >>>>>>>> properties.put("openjpa.LockTimeout", "30000"); >>>>>>>> >>>>>>>> And keep in mind that we don't use any connection pooling in the >>>>>>>> test >>>>>>>> case so the performance should be better when use it in deployment. >>>>>>>> >>>>>>>> Bye, >>>>>>>> Norman >>>>>>>> >>>>>>>> >>>>>>>> 2010/5/1 Eric Charles<eric.char...@u-mangate.com>: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I just tested the JPAStressTest (for IMAP-137 JPA fails to persist >>>>>>>>> MailboxMembership Entity on heavy load) Norman committed yesterday. >>>>>>>>> >>>>>>>>> I made some tests towards derby (memory and embedded) , h2 (memory >>>>>>>>> and >>>>>>>>> embedded) and mysql with following configs (and adding the needed >>>>>>>>> jdbc >>>>>>>>> drivers in pom.xml): >>>>>>>>> >>>>>>>>> // Derby Memory >>>>>>>>> properties.put("openjpa.ConnectionDriverName", >>>>>>>>> org.apache.derby.jdbc.EmbeddedDriver.class.getName()); >>>>>>>>> properties.put("openjpa.ConnectionURL", >>>>>>>>> "jdbc:derby:memory:derbyimap;create=>>>>>>> >>>>>>>>> // Derby Embedded >>>>>>>>> // properties.put("openjpa.ConnectionDriverName", >>>>>>>>> org.apache.derby.jdbc.EmbeddedDriver.class.getName()); >>>>>>>>> // properties.put("openjpa.ConnectionURL", >>>>>>>>> "jdbc:derby:derbyimap;create=r=root;password=root"); >>>>>>>>> >>>>>>>>> // H2 Memory >>>>>>>>> // properties.put("openjpa.ConnectionDriverName", >>>>>>>>> org.h2.Driver.class.getName()); >>>>>>>>> // properties.put("openjpa.ConnectionURL", >>>>>>>>> "jdbc:h2:mem:h2imap;DB_CLOSE_DELAY=>>> >>>>>>>>> // H2 Embedded >>>>>>>>> // properties.put("openjpa.ConnectionDriverName", >>>>>>>>> org.h2.Driver.class.getName()); >>>>>>>>> // properties.put("openjpa.ConnectionURL", >>>>>>>>> "jdbc:h2:~/h2/h2imap;USER=SWORD=root"); >>>>>>>>> >>>>>>>>> //Mysql >>>>>>>>> // properties.put("openjpa.ConnectionDriverName", >>>>>>>>> com.mysql.jdbc.Driver.class.getName()); >>>>>>>>> // properties.put("openjpa.ConnectionURL", >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> "jdbc:mysql://localhost:3306/msqlimap?user=sword=root&createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=utf-8"); >>>>>>>>> >>>>>>>>> >>>>>>>>> I first changed the maximum number of messages from 1.000 to >>>>>>>>> 100.000 >>>>>>>>> and >>>>>>>>> saw >>>>>>>>> on my linux dev PC (james server, stresstest and databases all >>>>>>>>> running >>>>>>>>> on >>>>>>>>> same dev PC): >>>>>>>>> >>>>>>>>> Derby: after many mails sending: >>>>>>>>> java.io.IOException: Too many open files >>>>>>>>> ... >>>>>>>>> Exception in thread "pool-1-thread-1044" >>>>>>>>> java.lang.NoClassDefFoundError: >>>>>>>>> org/apache/james/imap/api/display/HumanReadableText >>>>>>>>> ... >>>>>>>>> >>>>>>>>> H2: after 400 mails: >>>>>>>>> java.io.IOException: Too many open files >>>>>>>>> ... >>>>>>>>> Caused by:<openjpa-1.2.2-r422266:898935 nonfatal general error> >>>>>>>>> org.apache.openjpa.persistence.PersistenceException: Timeout trying >>>>>>>>> to >>>>>>>>> lock >>>>>>>>> table MEMBERSHIP [50200-79] {prepstmnt 406576 >>>>>>>>> INSERT INTO Membership (mailboxId, uid, answered, deleted, draft, >>>>>>>>> flagged, internalDate, recent, seen, size, MESSAGE_ID) >>>>>>>>> VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) >>>>>>>>> [params=, (long) 49, (int) 0, (int) 0, (int) 0, (int) 0, >>>>>>>>> (Timestamp) >>>>>>>>> 2010-05-01 08:37:10.293, (int) 0, (int) 0, (int) 25, (long) 4151]} >>>>>>>>> [codeP200, state= >>>>>>>>> ... >>>>>>>>> >>>>>>>>> MySql : after +/- 1000 (always around 1002 and 1007) mails : >>>>>>>>> Exception in thread "pool-1-thread-84"<openjpa-1.2.2-r422266:898935 >>>>>>>>> nonfatal user error> >>>>>>>>> org.apache.openjpa.persistence.NoResultException: >>>>>>>>> The >>>>>>>>> query on candidate type "class >>>>>>>>> org.apache.james.imap.jpa.mail.model.JPAMailbox" with filter >>>>>>>>> "SELECT >>>>>>>>> mailbox >>>>>>>>> FROM Mailbox mailbox WHERE mailbox.mailboxId =m" was configured to >>>>>>>>> have a unique result, but no instance matched the query. >>>>>>>>> .... >>>>>>>>> org.apache.james.imap.mailbox.MailboxNotFoundException: Mailbox >>>>>>>>> '#mail.INBOX' not found. >>>>>>>>> ... >>>>>>>>> >>>>>>>>> 3 different databases, 3 different behaviour/exceptions. >>>>>>>>> >>>>>>>>> Running 3 components (james server, stresstest, database) on my >>>>>>>>> "not-so-strong-PC" may be an issue. >>>>>>>>> >>>>>>>>> To make the tests successful, I added a Thread.sleep(5) (15 ms for >>>>>>>>> embedded >>>>>>>>> derby and h2) between each mail sending. >>>>>>>>> >>>>>>>>> >>>>>>>>> I suppose we can say that in a real environment, the load will >>>>>>>>> never >>>>>>>>> be >>>>>>>>> that >>>>>>>>> heavy, and that real server will perform better. >>>>>>>>> >>>>>>>>> >>>>>>>>> I also saw that the uid order is not respected: >>>>>>>>> >>>>>>>>> Append message with uidb58 >>>>>>>>> Append message with uidb59 >>>>>>>>> Append message with uidb56 >>>>>>>>> Append message with uidb57 >>>>>>>>> >>>>>>>>> This could come from a print mismatch, and not from uid being >>>>>>>>> generated >>>>>>>>> in a >>>>>>>>> bad order. >>>>>>>>> >>>>>>>>> Tks in advance for your feedbacks, if any, >>>>>>>>> >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>> >>>>> >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>> >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>> For additional commands, e-mail: server-dev-h...@james.apache.org >>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org