Hi Eric,
for such high load you will need to adjust a few things:
1) set a higher ulimit. By default its 1024 on linux which is prolly
not enough ( I use fbsd which use 11095 by default)
2) set a higher connection backlock in smtpserver.log
Bye,
Norman
2010/5/2 Eric Charles <eric.char...@u-mangate.com>:
> Hi Norman,
>
> Trying to simulate a more real-life test case, I developed a small bomber
> class that launches parallel threads. Each thread sends mails with
> attachements (with small pause between each mail sending).
>
> I tested with embedded derby with JDBCDomainList.
>
> Depending on the parameters (10 to 100 parallel threads, 5 ms to 100 ms
> pause, 1KB to 100KB attachement), I could have on my dev laptop 5 to 10
> mails per second being processed.
>
> The good think is that I have no exception at all during stable phase.
>
> However, I was not able to have more mails being processed on my latptop due
> to disk access. The CPU and memory are really not charged (20 - 30 %), but
> disk is really working non-stop.
> This is probably the price to pay for activemq and the kahadb.
>
> Complete scenario with unstable phase:
> - Bomb 5 minutes
> - Stop bomb.
> - Still some mails are spooled for a while which is quite normal.
> - When all mails are spooled, disk still runs crazy for minutes, console
> showing a few messages "Slow KahaDB access: cleanup took 684"
> - rebombing : org.apache.camel.CamelExecutionException: Exception occurred
> during execution on the exchange: Exchange[Message:
> org.apache.james.core.maili...@16863f2] Caused by: java.io.IOException: Too
> many open files - error message in the bomber
> - Server console show still a few mails being spooled.
> - After, server does not respond on socket - no more exception.
>
> Disk congestion may come from my disk.
> I will setup a server environment with good disk, and will rerun the test in
> a few days.
> A database on a separate server could also help.
>
> Btw, what should we consider as acceptable/needed as load (xxx mails from
> yyy parallel clients) ?
>
> Tks,
>
> Eric
>
>
>
> On 05/01/2010 09:25 AM, Norman Maurer wrote:
>>
>> About 6000, then I run out of heap space..
>>
>> Can you maybe try to deploy it with james and bomb it via smtp ? I was
>> not able to see any exception.
>>
>> Thx,
>> Norman
>>
>>
>> 2010/5/1 Eric Charles<eric.char...@u-mangate.com>:
>>
>>>
>>> Hi Norman,
>>>
>>> Similar exceptions with properties.put("openjpa.LockTimeout", "30000");
>>>
>>> I think the environment simply can not follow the load (more than one
>>> mail
>>> sent each ms).
>>> Bootleneck can be anywhere (connection creation, database lock,
>>> openjpa,...).
>>> This is why I put a sleep between each mail sending.
>>> Real servers will better hold the pressure.
>>>
>>> So I think I'm limited by my environment : it does not allow me to have a
>>> really heavy load to validate that IMAP-137 is resolved.
>>> Well, it is probably, as before you commits, I directly had the duplicate
>>> key exception.
>>>
>>> How much mails can you hold on your env with h2 inmemory ?
>>>
>>> Tks,
>>>
>>> Eric
>>>
>>>
>>> On 05/01/2010 09:08 AM, Norman Maurer wrote:
>>>
>>>>
>>>> Hi Eric,
>>>>
>>>> thx for your tests. The order of the uid is caused by the threads,
>>>> thats nothing to worry about.
>>>>
>>>> About the timeout, can you try to add this to the test case:
>>>>
>>>> properties.put("openjpa.LockTimeout", "30000");
>>>>
>>>> And keep in mind that we don't use any connection pooling in the test
>>>> case so the performance should be better when use it in deployment.
>>>>
>>>> Bye,
>>>> Norman
>>>>
>>>>
>>>> 2010/5/1 Eric Charles<eric.char...@u-mangate.com>:
>>>>
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I just tested the JPAStressTest (for IMAP-137 JPA fails to persist
>>>>> MailboxMembership Entity on heavy load) Norman committed yesterday.
>>>>>
>>>>> I made some tests towards derby (memory and embedded) , h2 (memory and
>>>>> embedded) and mysql with following configs (and adding the needed jdbc
>>>>> drivers in pom.xml):
>>>>>
>>>>> // Derby Memory
>>>>> properties.put("openjpa.ConnectionDriverName",
>>>>> org.apache.derby.jdbc.EmbeddedDriver.class.getName());
>>>>> properties.put("openjpa.ConnectionURL",
>>>>> "jdbc:derby:memory:derbyimap;create=");
>>>>>
>>>>> // Derby Embedded
>>>>> // properties.put("openjpa.ConnectionDriverName",
>>>>> org.apache.derby.jdbc.EmbeddedDriver.class.getName());
>>>>> // properties.put("openjpa.ConnectionURL",
>>>>> "jdbc:derby:derbyimap;create=;user=root;password=root");
>>>>>
>>>>> // H2 Memory
>>>>> // properties.put("openjpa.ConnectionDriverName",
>>>>> org.h2.Driver.class.getName());
>>>>> // properties.put("openjpa.ConnectionURL",
>>>>> "jdbc:h2:mem:h2imap;DB_CLOSE_DELAY=;
>>>>>
>>>>> // H2 Embedded
>>>>> // properties.put("openjpa.ConnectionDriverName",
>>>>> org.h2.Driver.class.getName());
>>>>> // properties.put("openjpa.ConnectionURL",
>>>>> "jdbc:h2:~/h2/h2imap;USER=;PASSWORD=root");
>>>>>
>>>>> //Mysql
>>>>> // properties.put("openjpa.ConnectionDriverName",
>>>>> com.mysql.jdbc.Driver.class.getName());
>>>>> // properties.put("openjpa.ConnectionURL",
>>>>>
>>>>>
>>>>> "jdbc:mysql://localhost:3306/msqlimap?user=&password=root&createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=utf-8");
>>>>>
>>>>>
>>>>> I first changed the maximum number of messages from 1.000 to 100.000
>>>>> and
>>>>> saw
>>>>> on my linux dev PC (james server, stresstest and databases all running
>>>>> on
>>>>> same dev PC):
>>>>>
>>>>> Derby: after many mails sending:
>>>>> java.io.IOException: Too many open files
>>>>> ...
>>>>> Exception in thread "pool-1-thread-1044"
>>>>> java.lang.NoClassDefFoundError:
>>>>> org/apache/james/imap/api/display/HumanReadableText
>>>>> ...
>>>>>
>>>>> H2: after 400 mails:
>>>>> java.io.IOException: Too many open files
>>>>> ...
>>>>> Caused by:<openjpa-1.2.2-r422266:898935 nonfatal general error>
>>>>> org.apache.openjpa.persistence.PersistenceException: Timeout trying to
>>>>> lock
>>>>> table MEMBERSHIP [50200-79] {prepstmnt 406576
>>>>> INSERT INTO Membership (mailboxId, uid, answered, deleted, draft,
>>>>> flagged, internalDate, recent, seen, size, MESSAGE_ID)
>>>>> VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
>>>>> [params=g) 1, (long) 49, (int) 0, (int) 0, (int) 0, (int) 0,
>>>>> (Timestamp)
>>>>> 2010-05-01 08:37:10.293, (int) 0, (int) 0, (int) 25, (long) 4151]}
>>>>> [codeP200, state=T00]
>>>>> ...
>>>>>
>>>>> MySql : after +/- 1000 (always around 1002 and 1007) mails :
>>>>> Exception in thread "pool-1-thread-84"<openjpa-1.2.2-r422266:898935
>>>>> nonfatal user error>
>>>>> org.apache.openjpa.persistence.NoResultException:
>>>>> The
>>>>> query on candidate type "class
>>>>> org.apache.james.imap.jpa.mail.model.JPAMailbox" with filter "SELECT
>>>>> mailbox
>>>>> FROM Mailbox mailbox WHERE mailbox.mailboxId =Param" was configured to
>>>>> have a unique result, but no instance matched the query.
>>>>> ....
>>>>> org.apache.james.imap.mailbox.MailboxNotFoundException: Mailbox
>>>>> '#mail.INBOX' not found.
>>>>> ...
>>>>>
>>>>> 3 different databases, 3 different behaviour/exceptions.
>>>>>
>>>>> Running 3 components (james server, stresstest, database) on my
>>>>> "not-so-strong-PC" may be an issue.
>>>>>
>>>>> To make the tests successful, I added a Thread.sleep(5) (15 ms for
>>>>> embedded
>>>>> derby and h2) between each mail sending.
>>>>>
>>>>>
>>>>> I suppose we can say that in a real environment, the load will never be
>>>>> that
>>>>> heavy, and that real server will perform better.
>>>>>
>>>>>
>>>>> I also saw that the uid order is not respected:
>>>>>
>>>>> Append message with uidb58
>>>>> Append message with uidb59
>>>>> Append message with uidb56
>>>>> Append message with uidb57
>>>>>
>>>>> This could come from a print mismatch, and not from uid being generated
>>>>> in a
>>>>> bad order.
>>>>>
>>>>> Tks in advance for your feedbacks, if any,
>>>>>
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
