Ups I mean <connectionBacklog>200</connectionBacklog>
in smtpserver.xml Bye, Norman 2010/5/2 Eric Charles <eric.char...@u-mangate.com>: > Hi Norman, > What do you mean with "...backlock in smtpserver.log" ? > Tks, > Eric > > On 05/02/2010 01:15 PM, Norman Maurer wrote: >> >> Hi Eric, >> >> for such high load you will need to adjust a few things: >> >> 1) set a higher ulimit. By default its 1024 on linux which is prolly >> not enough ( I use fbsd which use 11095 by default) >> 2) set a higher connection backlock in smtpserver.log >> >> >> Bye, >> Norman >> >> >> 2010/5/2 Eric Charles<eric.char...@u-mangate.com>: >> >>> >>> Hi Norman, >>> >>> Trying to simulate a more real-life test case, I developed a small bomber >>> class that launches parallel threads. Each thread sends mails with >>> attachements (with small pause between each mail sending). >>> >>> I tested with embedded derby with JDBCDomainList. >>> >>> Depending on the parameters (10 to 100 parallel threads, 5 ms to 100 ms >>> pause, 1KB to 100KB attachement), I could have on my dev laptop 5 to 10 >>> mails per second being processed. >>> >>> The good think is that I have no exception at all during stable phase. >>> >>> However, I was not able to have more mails being processed on my latptop >>> due >>> to disk access. The CPU and memory are really not charged (20 - 30 %), >>> but >>> disk is really working non-stop. >>> This is probably the price to pay for activemq and the kahadb. >>> >>> Complete scenario with unstable phase: >>> - Bomb 5 minutes >>> - Stop bomb. >>> - Still some mails are spooled for a while which is quite normal. >>> - When all mails are spooled, disk still runs crazy for minutes, console >>> showing a few messages "Slow KahaDB access: cleanup took 684" >>> - rebombing : org.apache.camel.CamelExecutionException: Exception >>> occurred >>> during execution on the exchange: Exchange[Message: >>> org.apache.james.core.maili...@16863f2] Caused by: java.io.IOException: >>> Too >>> many open files - error message in the bomber >>> - Server console show still a few mails being spooled. >>> - After, server does not respond on socket - no more exception. >>> >>> Disk congestion may come from my disk. >>> I will setup a server environment with good disk, and will rerun the test >>> in >>> a few days. >>> A database on a separate server could also help. >>> >>> Btw, what should we consider as acceptable/needed as load (xxx mails from >>> yyy parallel clients) ? >>> >>> Tks, >>> >>> Eric >>> >>> >>> >>> On 05/01/2010 09:25 AM, Norman Maurer wrote: >>> >>>> >>>> About 6000, then I run out of heap space.. >>>> >>>> Can you maybe try to deploy it with james and bomb it via smtp ? I was >>>> not able to see any exception. >>>> >>>> Thx, >>>> Norman >>>> >>>> >>>> 2010/5/1 Eric Charles<eric.char...@u-mangate.com>: >>>> >>>> >>>>> >>>>> Hi Norman, >>>>> >>>>> Similar exceptions with properties.put("openjpa.LockTimeout", "30000"); >>>>> >>>>> I think the environment simply can not follow the load (more than one >>>>> mail >>>>> sent each ms). >>>>> Bootleneck can be anywhere (connection creation, database lock, >>>>> openjpa,...). >>>>> This is why I put a sleep between each mail sending. >>>>> Real servers will better hold the pressure. >>>>> >>>>> So I think I'm limited by my environment : it does not allow me to have >>>>> a >>>>> really heavy load to validate that IMAP-137 is resolved. >>>>> Well, it is probably, as before you commits, I directly had the >>>>> duplicate >>>>> key exception. >>>>> >>>>> How much mails can you hold on your env with h2 inmemory ? >>>>> >>>>> Tks, >>>>> >>>>> Eric >>>>> >>>>> >>>>> On 05/01/2010 09:08 AM, Norman Maurer wrote: >>>>> >>>>> >>>>>> >>>>>> Hi Eric, >>>>>> >>>>>> thx for your tests. The order of the uid is caused by the threads, >>>>>> thats nothing to worry about. >>>>>> >>>>>> About the timeout, can you try to add this to the test case: >>>>>> >>>>>> properties.put("openjpa.LockTimeout", "30000"); >>>>>> >>>>>> And keep in mind that we don't use any connection pooling in the test >>>>>> case so the performance should be better when use it in deployment. >>>>>> >>>>>> Bye, >>>>>> Norman >>>>>> >>>>>> >>>>>> 2010/5/1 Eric Charles<eric.char...@u-mangate.com>: >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I just tested the JPAStressTest (for IMAP-137 JPA fails to persist >>>>>>> MailboxMembership Entity on heavy load) Norman committed yesterday. >>>>>>> >>>>>>> I made some tests towards derby (memory and embedded) , h2 (memory >>>>>>> and >>>>>>> embedded) and mysql with following configs (and adding the needed >>>>>>> jdbc >>>>>>> drivers in pom.xml): >>>>>>> >>>>>>> // Derby Memory >>>>>>> properties.put("openjpa.ConnectionDriverName", >>>>>>> org.apache.derby.jdbc.EmbeddedDriver.class.getName()); >>>>>>> properties.put("openjpa.ConnectionURL", >>>>>>> "jdbc:derby:memory:derbyimap;create=; >>>>>>> >>>>>>> // Derby Embedded >>>>>>> // properties.put("openjpa.ConnectionDriverName", >>>>>>> org.apache.derby.jdbc.EmbeddedDriver.class.getName()); >>>>>>> // properties.put("openjpa.ConnectionURL", >>>>>>> "jdbc:derby:derbyimap;create=ser=root;password=root"); >>>>>>> >>>>>>> // H2 Memory >>>>>>> // properties.put("openjpa.ConnectionDriverName", >>>>>>> org.h2.Driver.class.getName()); >>>>>>> // properties.put("openjpa.ConnectionURL", >>>>>>> "jdbc:h2:mem:h2imap;DB_CLOSE_DELAY=>>>>> >>>>>>> // H2 Embedded >>>>>>> // properties.put("openjpa.ConnectionDriverName", >>>>>>> org.h2.Driver.class.getName()); >>>>>>> // properties.put("openjpa.ConnectionURL", >>>>>>> "jdbc:h2:~/h2/h2imap;USER=ASSWORD=root"); >>>>>>> >>>>>>> //Mysql >>>>>>> // properties.put("openjpa.ConnectionDriverName", >>>>>>> com.mysql.jdbc.Driver.class.getName()); >>>>>>> // properties.put("openjpa.ConnectionURL", >>>>>>> >>>>>>> >>>>>>> >>>>>>> "jdbc:mysql://localhost:3306/msqlimap?user=assword=root&createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=utf-8"); >>>>>>> >>>>>>> >>>>>>> I first changed the maximum number of messages from 1.000 to 100.000 >>>>>>> and >>>>>>> saw >>>>>>> on my linux dev PC (james server, stresstest and databases all >>>>>>> running >>>>>>> on >>>>>>> same dev PC): >>>>>>> >>>>>>> Derby: after many mails sending: >>>>>>> java.io.IOException: Too many open files >>>>>>> ... >>>>>>> Exception in thread "pool-1-thread-1044" >>>>>>> java.lang.NoClassDefFoundError: >>>>>>> org/apache/james/imap/api/display/HumanReadableText >>>>>>> ... >>>>>>> >>>>>>> H2: after 400 mails: >>>>>>> java.io.IOException: Too many open files >>>>>>> ... >>>>>>> Caused by:<openjpa-1.2.2-r422266:898935 nonfatal general error> >>>>>>> org.apache.openjpa.persistence.PersistenceException: Timeout trying >>>>>>> to >>>>>>> lock >>>>>>> table MEMBERSHIP [50200-79] {prepstmnt 406576 >>>>>>> INSERT INTO Membership (mailboxId, uid, answered, deleted, draft, >>>>>>> flagged, internalDate, recent, seen, size, MESSAGE_ID) >>>>>>> VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) >>>>>>> [params= 1, (long) 49, (int) 0, (int) 0, (int) 0, (int) 0, >>>>>>> (Timestamp) >>>>>>> 2010-05-01 08:37:10.293, (int) 0, (int) 0, (int) 25, (long) 4151]} >>>>>>> [codeP200, state=0] >>>>>>> ... >>>>>>> >>>>>>> MySql : after +/- 1000 (always around 1002 and 1007) mails : >>>>>>> Exception in thread "pool-1-thread-84"<openjpa-1.2.2-r422266:898935 >>>>>>> nonfatal user error> >>>>>>> org.apache.openjpa.persistence.NoResultException: >>>>>>> The >>>>>>> query on candidate type "class >>>>>>> org.apache.james.imap.jpa.mail.model.JPAMailbox" with filter "SELECT >>>>>>> mailbox >>>>>>> FROM Mailbox mailbox WHERE mailbox.mailboxId =ram" was configured to >>>>>>> have a unique result, but no instance matched the query. >>>>>>> .... >>>>>>> org.apache.james.imap.mailbox.MailboxNotFoundException: Mailbox >>>>>>> '#mail.INBOX' not found. >>>>>>> ... >>>>>>> >>>>>>> 3 different databases, 3 different behaviour/exceptions. >>>>>>> >>>>>>> Running 3 components (james server, stresstest, database) on my >>>>>>> "not-so-strong-PC" may be an issue. >>>>>>> >>>>>>> To make the tests successful, I added a Thread.sleep(5) (15 ms for >>>>>>> embedded >>>>>>> derby and h2) between each mail sending. >>>>>>> >>>>>>> >>>>>>> I suppose we can say that in a real environment, the load will never >>>>>>> be >>>>>>> that >>>>>>> heavy, and that real server will perform better. >>>>>>> >>>>>>> >>>>>>> I also saw that the uid order is not respected: >>>>>>> >>>>>>> Append message with uidb58 >>>>>>> Append message with uidb59 >>>>>>> Append message with uidb56 >>>>>>> Append message with uidb57 >>>>>>> >>>>>>> This could come from a print mismatch, and not from uid being >>>>>>> generated >>>>>>> in a >>>>>>> bad order. >>>>>>> >>>>>>> Tks in advance for your feedbacks, if any, >>>>>>> >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>> >>>>> >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>> >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>> For additional commands, e-mail: server-dev-h...@james.apache.org >>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org