Re: [IMAP] Over-designed /Some thoughts ?
On Wed, Apr 28, 2010 at 5:55 AM, Norman Maurer norman.mau...@googlemail.com wrote: snip I think it would be a good think to simplify the api a bit to make it a bit easier to understand. So some points which came to me mind: 1) UidChangeTracking: Is this really necessary ? It does some kind of caching but I don't see something else for which its useful. Why not just fire the events directly with a shared MailboxEventDispatcher which is the same for all Mailboxes? i'm not convinced it's needed but beware... this is one of the few areas retained from the design before i started reworking. i had hoped to replace it but never really worked out how to do that without crippling performance or breaking IMAP. I'm currently testing imap without the UidChangeTracker and so far it seems like its not really slower then before.. it's only slower than the alternatives that required to make IMAP work properly ;-) IIRC UIDChangeTracker tracks UID changes made by concurrent sessions accessing the same mailbox. the local caching should work for users own changes. it's possible that some of the changes i might have made it redundant by now but i don't trust the functional concurrency tests. 2) Global Mailbox caching At the moment the Mailbox is cached in a HashMap. The problem with this is it will never get recycled by the GC. This can generate a OOM over long time i run IMAP with approx 1.5G spread over around a hundred mailboxes. i've never had an OOM. so i never bothered changing this. I think you use Torque right ? Maybe it behave a bit different there. i inherited torque and this is one area i left alone ;-) I'm using JPA and its reproducable with feeding a mailbox with ca 1 million emails. You will see the memory usage just grow and grow.. When I took a heap dump it seems like the OpenJPA objects where never released, because the where hold in the HashMap. for torque the session needs to be held to manage concurrency (mailbox access needs to be synchronized). for OpenJPA, sounds like the mailbox structure needs to be there to manage synchronization and caching but a new OpenJPA object needs to be created each time. The other problem with this is, the Mailbox should be tight to the MailboxSession. Let me explain why. For example in JCR we could use the User/Pass which is bound to the MailboxSession to access different parts of the JCR Repository etc.. i thought this too originally but i couldn't work out how to do so without cripple performance or breaking IMAP. Sure good performance is a must, but I would prefer to have a good api first ;) this wasn't a good performance issue but a usable at all one when two sessions are accessing the same mailbox, there are a handful number of operations which require caching and concurrency control to maintain correctness. there are a number of ways that this design could work. mailbox et al is inherited, and probably not my first choice. i would prefer to revise the API by pushing the Mailbox functions into MailboxManager, and so making it an internal feature which could be varied by implementations. the namespace handling is problematic, so i would then model namespace by a Mailbox object which could be passed in to each method in the API. IIRC these are related issues. the essential function is caching and synchronization. in performances terms, i think much higher performance could be achieved by replacement by something asynchronous and event driven using a blocking queue. this would be a substantial change. I agree with you here. But as you outlined already, its not a easy thing todo, without rewrite a lot of stuff. very little rewriting but hard, and risky for the poorly tested concurrent use cases. then again, maybe these don't work ATM. the best place to start would be by using creating some more concurrency tests. there's an application that creates tests in the package org.apache.james.imap.functional.builder in seda. I even tend to believe we should do something similar to what we have in SMTP/POP3. Just have some kind of LineHandler which push data in the processor when a CRLF was detected and so not using blocking streams as input at all. the IMAP protocol makes this approach tricky, but in general yes. the protocol handling foo is intended to address this, and should be quite close now. - robert - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Over-designed /Some thoughts ?
On Wed, Apr 28, 2010 at 5:55 AM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Robert, nice to see you are back ;)... Comments inline.. i'm really busy ATM (exams, projects, reports) :-/ this will probably have to be my last post for a while... 2010/4/27 Robert Burrell Donkin robertburrelldon...@gmail.com: On Tue, Apr 27, 2010 at 10:42 AM, Norman Maurer nor...@apache.org wrote: Hi all, after spending some time over the weekend to fix some issues with IMAP I started to feel its a big over-designed ... i started out with that impression. after digging around i came to the conclusion that IMAP has some annoying requirements.. Oh well... its some kind of a pita. +1 sessions that select a mailbox have to be updated by operations done by other sessions on that mailbox 1) UidChangeTracking: the API uses a listeners and events to manage updates to UIDs etc. this design may need to be revised. 2) Global Mailbox caching At the moment the Mailbox is cached in a HashMap. The problem with this is it will never get recycled by the GC. This can generate a OOM over long time i run IMAP with approx 1.5G spread over around a hundred mailboxes. i've never had an OOM. so i never bothered changing this. I think you use Torque right ? Maybe it behave a bit different there. I'm using JPA and its reproducable with feeding a mailbox with ca 1 million emails. You will see the memory usage just grow and grow.. When I took a heap dump it seems like the OpenJPA objects where never released, because the where hold in the HashMap. AFACT the OpenJPA stuff shouldn't be caching the mailboxes in the manager (caching should be managed internally by OpenJPA). sounds like a session management problem is much more likely. ATM sessions cache entity managers for the duration. if you're running a lot of concurrent connections, they need to be pooled. FWIW in IMAP, a session may: * have a long term interest in a mailbox spanning multiple requests * need to perform multiple operations on one or more mailboxes to execute a single protocol request IMHO the failure to cleanly and clearly separate these is a major flaw in current API. the Mailbox caching issues are just a consequence. fixing these would require a major rewrite with knowledge of the current foibles... - robert - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Over-designed /Some thoughts ?
2010/4/28 Robert Burrell Donkin robertburrelldon...@gmail.com: On Wed, Apr 28, 2010 at 5:55 AM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Robert, nice to see you are back ;)... Comments inline.. i'm really busy ATM (exams, projects, reports) :-/ this will probably have to be my last post for a while... Ok, hope to see you back in some time.. 2010/4/27 Robert Burrell Donkin robertburrelldon...@gmail.com: On Tue, Apr 27, 2010 at 10:42 AM, Norman Maurer nor...@apache.org wrote: Hi all, after spending some time over the weekend to fix some issues with IMAP I started to feel its a big over-designed ... i started out with that impression. after digging around i came to the conclusion that IMAP has some annoying requirements.. Oh well... its some kind of a pita. +1 sessions that select a mailbox have to be updated by operations done by other sessions on that mailbox Yes I'm aware of this.. I'm still trying to get my head around of a good solution to see without breakin to much ;) 1) UidChangeTracking: the API uses a listeners and events to manage updates to UIDs etc. this design may need to be revised. I think the event stuff is good. Just the Tracker is not needed and add a useless layer of complexity to the api.. 2) Global Mailbox caching At the moment the Mailbox is cached in a HashMap. The problem with this is it will never get recycled by the GC. This can generate a OOM over long time i run IMAP with approx 1.5G spread over around a hundred mailboxes. i've never had an OOM. so i never bothered changing this. I think you use Torque right ? Maybe it behave a bit different there. I'm using JPA and its reproducable with feeding a mailbox with ca 1 million emails. You will see the memory usage just grow and grow.. When I took a heap dump it seems like the OpenJPA objects where never released, because the where hold in the HashMap. AFACT the OpenJPA stuff shouldn't be caching the mailboxes in the manager (caching should be managed internally by OpenJPA). sounds like a session management problem is much more likely. ATM sessions cache entity managers for the duration. if you're running a lot of concurrent connections, they need to be pooled. Yeah we should remove the caching of mailboxes at all from the MailboxManager, because (as you already stated) it depend on the implementation if a cache is needed and how a cache is implemented. At the moment I refactored the openjpa stuff to use one EntityManager per request. This seems to work out well so far. Even better would be to use one entitymanager per mailboxsession. FWIW in IMAP, a session may: * have a long term interest in a mailbox spanning multiple requests * need to perform multiple operations on one or more mailboxes to execute a single protocol request IMHO the failure to cleanly and clearly separate these is a major flaw in current API. the Mailbox caching issues are just a consequence. fixing these would require a major rewrite with knowledge of the current foibles... - robert Bye, Norman Ps: And good luck for your exams mate.. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Celebration, plus: Release James 4.0M1?
Hi, Thanks to Norman, Robert and everyone else - over the past month, there have been tons of changes and improvements made to James. Most of them are fantastic: modularization, IMAP, container, async io protocols. I can even deploy James as a WAR file now - This must be the future! Now, how dare we not to release this bag of beauties to our users? Why not have a milestone release adding a disclaimer that this isn't production ready and collect feedback from the field? And even if you'd call it 3.0 instead of the deserved 4.0, I'd be happy. :-) Here's my +1! Non-bindingly, Bernd - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Over-designed /Some thoughts ?
On Wed, Apr 28, 2010 at 8:44 AM, Norman Maurer norman.mau...@googlemail.com wrote: 2010/4/28 Robert Burrell Donkin robertburrelldon...@gmail.com: On Wed, Apr 28, 2010 at 5:55 AM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Robert, nice to see you are back ;)... Comments inline.. i'm really busy ATM (exams, projects, reports) :-/ this will probably have to be my last post for a while... Ok, hope to see you back in some time.. thunderbird 3 hates our IMAP so i'm having mail issues :- 2010/4/27 Robert Burrell Donkin robertburrelldon...@gmail.com: On Tue, Apr 27, 2010 at 10:42 AM, Norman Maurer nor...@apache.org wrote: Hi all, after spending some time over the weekend to fix some issues with IMAP I started to feel its a big over-designed ... i started out with that impression. after digging around i came to the conclusion that IMAP has some annoying requirements.. Oh well... its some kind of a pita. +1 sessions that select a mailbox have to be updated by operations done by other sessions on that mailbox Yes I'm aware of this.. I'm still trying to get my head around of a good solution to see without breakin to much ;) 1) UidChangeTracking: the API uses a listeners and events to manage updates to UIDs etc. this design may need to be revised. I think the event stuff is good. Just the Tracker is not needed and add a useless layer of complexity to the api.. AFACT in https://svn.apache.org/repos/asf/james/imap/trunk UIDChangeTracker is only used by Torque. why not just move it into that package then leave it alone? 2) Global Mailbox caching At the moment the Mailbox is cached in a HashMap. The problem with this is it will never get recycled by the GC. This can generate a OOM over long time i run IMAP with approx 1.5G spread over around a hundred mailboxes. i've never had an OOM. so i never bothered changing this. I think you use Torque right ? Maybe it behave a bit different there. I'm using JPA and its reproducable with feeding a mailbox with ca 1 million emails. You will see the memory usage just grow and grow.. When I took a heap dump it seems like the OpenJPA objects where never released, because the where hold in the HashMap. AFACT the OpenJPA stuff shouldn't be caching the mailboxes in the manager (caching should be managed internally by OpenJPA). sounds like a session management problem is much more likely. ATM sessions cache entity managers for the duration. if you're running a lot of concurrent connections, they need to be pooled. Yeah we should remove the caching of mailboxes at all from the MailboxManager, because (as you already stated) it depend on the implementation if a cache is needed and how a cache is implemented. At the moment I refactored the openjpa stuff to use one EntityManager per request. This seems to work out well so far. Even better would be to use one entitymanager per mailboxsession. for JPA, they should really be pooled, then obtained per request - robert - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Over-designed /Some thoughts ?
2010/4/28 Robert Burrell Donkin robertburrelldon...@gmail.com: On Wed, Apr 28, 2010 at 8:44 AM, Norman Maurer norman.mau...@googlemail.com wrote: 2010/4/28 Robert Burrell Donkin robertburrelldon...@gmail.com: On Wed, Apr 28, 2010 at 5:55 AM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Robert, nice to see you are back ;)... Comments inline.. i'm really busy ATM (exams, projects, reports) :-/ this will probably have to be my last post for a while... Ok, hope to see you back in some time.. thunderbird 3 hates our IMAP so i'm having mail issues :- Just till upgrade ? 2010/4/27 Robert Burrell Donkin robertburrelldon...@gmail.com: On Tue, Apr 27, 2010 at 10:42 AM, Norman Maurer nor...@apache.org wrote: Hi all, after spending some time over the weekend to fix some issues with IMAP I started to feel its a big over-designed ... i started out with that impression. after digging around i came to the conclusion that IMAP has some annoying requirements.. Oh well... its some kind of a pita. +1 sessions that select a mailbox have to be updated by operations done by other sessions on that mailbox Yes I'm aware of this.. I'm still trying to get my head around of a good solution to see without breakin to much ;) 1) UidChangeTracking: the API uses a listeners and events to manage updates to UIDs etc. this design may need to be revised. I think the event stuff is good. Just the Tracker is not needed and add a useless layer of complexity to the api.. AFACT in https://svn.apache.org/repos/asf/james/imap/trunk UIDChangeTracker is only used by Torque. why not just move it into that package then leave it alone? Yeah its still work in progress.. 2) Global Mailbox caching At the moment the Mailbox is cached in a HashMap. The problem with this is it will never get recycled by the GC. This can generate a OOM over long time i run IMAP with approx 1.5G spread over around a hundred mailboxes. i've never had an OOM. so i never bothered changing this. I think you use Torque right ? Maybe it behave a bit different there. I'm using JPA and its reproducable with feeding a mailbox with ca 1 million emails. You will see the memory usage just grow and grow.. When I took a heap dump it seems like the OpenJPA objects where never released, because the where hold in the HashMap. AFACT the OpenJPA stuff shouldn't be caching the mailboxes in the manager (caching should be managed internally by OpenJPA). sounds like a session management problem is much more likely. ATM sessions cache entity managers for the duration. if you're running a lot of concurrent connections, they need to be pooled. Yeah we should remove the caching of mailboxes at all from the MailboxManager, because (as you already stated) it depend on the implementation if a cache is needed and how a cache is implemented. At the moment I refactored the openjpa stuff to use one EntityManager per request. This seems to work out well so far. Even better would be to use one entitymanager per mailboxsession. for JPA, they should really be pooled, then obtained per request - robert Bye, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Over-designed /Some thoughts ?
On Wed, Apr 28, 2010 at 10:19 AM, Norman Maurer norman.mau...@googlemail.com wrote: 2010/4/28 Robert Burrell Donkin robertburrelldon...@gmail.com: On Wed, Apr 28, 2010 at 8:44 AM, Norman Maurer norman.mau...@googlemail.com wrote: 2010/4/28 Robert Burrell Donkin robertburrelldon...@gmail.com: On Wed, Apr 28, 2010 at 5:55 AM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Robert, nice to see you are back ;)... Comments inline.. i'm really busy ATM (exams, projects, reports) :-/ this will probably have to be my last post for a while... Ok, hope to see you back in some time.. thunderbird 3 hates our IMAP so i'm having mail issues :- Just till upgrade ? thunderbird 3 automatically retries operations in the background. so, when it crashes it will retry the next time you open it. it crashes a lot anyway but if you try any long running operations that are likely to time out, it quickly becomes unusable. starting to write something on long operations would probably help by stopping some of the timeouts but basically the rewrite to the IMAP client has been poorly executed. - robert - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
svn commit: r938877 - in /james/imap/trunk/store/src/main/java/org/apache/james/imap/store: AbstractRewindableInputStream.java InputStreamContent.java RewindableInputStream.java
Author: norman
Date: Wed Apr 28 10:59:33 2010
New Revision: 938877
URL: http://svn.apache.org/viewvc?rev=938877view=rev
Log:
Make sure we dispose temporary files after finish processing (IMAP-135)
Modified:
james/imap/trunk/store/src/main/java/org/apache/james/imap/store/AbstractRewindableInputStream.java
james/imap/trunk/store/src/main/java/org/apache/james/imap/store/InputStreamContent.java
james/imap/trunk/store/src/main/java/org/apache/james/imap/store/RewindableInputStream.java
Modified:
james/imap/trunk/store/src/main/java/org/apache/james/imap/store/AbstractRewindableInputStream.java
URL:
http://svn.apache.org/viewvc/james/imap/trunk/store/src/main/java/org/apache/james/imap/store/AbstractRewindableInputStream.java?rev=938877r1=938876r2=938877view=diff
==
---
james/imap/trunk/store/src/main/java/org/apache/james/imap/store/AbstractRewindableInputStream.java
(original)
+++
james/imap/trunk/store/src/main/java/org/apache/james/imap/store/AbstractRewindableInputStream.java
Wed Apr 28 10:59:33 2010
@@ -1,17 +1,37 @@
+/
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information*
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the*
+ * License); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the*
+ * specific language governing permissions and limitations *
+ * under the License. *
+ /
package org.apache.james.imap.store;
-import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
/**
- * {...@link FilterInputStream} which support the get rewinded. This is done
by copy over every byte
+ * {...@link RewindableInputStream} which support the get rewinded. This is
done by copy over every byte
* over to another {...@link OutputStream}. What {...@link OutputStream} to
use is up to the implementations.
*
* The rewinding will get delayed as long as possible. So if you call
* rewind, it will only get performed when needed
*
+ * Be sure to call {...@link #close()} once you done reading from the object.
This will
+ * remove all temporary data
+ *
*
*/
public abstract class AbstractRewindableInputStream extends
RewindableInputStream{
@@ -22,15 +42,41 @@ public abstract class AbstractRewindable
super(in);
}
+/**
+ * Return the OutputStream to which the data should get written when they
are read the
+ * first time
+ *
+ * @return output
+ * @throws IOException
+ */
protected abstract OutputStream getRewindOutputStream() throws IOException;
+/**
+ * Return the InputStream which should get used after the stream was
rewinded
+ *
+ * @return rewindInput
+ * @throws IOException
+ */
protected abstract InputStream getRewindInputStream() throws IOException;
+/**
+ * Dispose all temporary data
+ *
+ * @throws IOException
+ */
protected abstract void dispose() throws IOException;
+/**
+ * Get called after the rewind was complete
+ *
+ * @throws IOException
+ */
protected abstract void afterRewindComplete() throws IOException;
-@Override
+/**
+ * Close the stream and dispose all temporary data
+ *
+ */
public void close() throws IOException {
try {
in.close();
@@ -48,7 +94,9 @@ public abstract class AbstractRewindable
}
-
+/**
+ * Read data and write and store it for later usage once the rewind was
done
+ */
@Override
public int read() throws IOException {
int i;
@@ -70,6 +118,9 @@ public abstract class AbstractRewindable
return i;
}
+/**
+ * Read data and write and store it for later usage once the rewind was
done
+ */
@Override
public int read(byte[] b, int off, int len) throws IOException {
if (len == 0) {
@@ -92,6 +143,9 @@ public abstract class AbstractRewindable
return i;
}
+/**
+ * Read data and
[jira] Resolved: (IMAP-135) RewindableInputStream does not cleanup temporary files in all cases
[ https://issues.apache.org/jira/browse/IMAP-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norman Maurer resolved IMAP-135. Resolution: Fixed RewindableInputStream does not cleanup temporary files in all cases --- Key: IMAP-135 URL: https://issues.apache.org/jira/browse/IMAP-135 Project: JAMES Imap Issue Type: Bug Reporter: Norman Maurer Fix For: 0.1 Sometimes the RewindableInputStream does seem to not cleanup the temporary used file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] Created: (IMAP-137) JPA fails to persist MailboxMembership Entity on heavy load
JPA fails to persist MailboxMembership Entity on heavy load --- Key: IMAP-137 URL: https://issues.apache.org/jira/browse/IMAP-137 Project: JAMES Imap Issue Type: Bug Components: JPA Mailbox Reporter: Norman Maurer Assignee: Norman Maurer On heavy load JPA fails to persist MailboxMembership Enitity objects. Thats because it not use the right locking.. Exception: ** INFO 10:44:16,401 | james.smtpserver | Successfully spooled mail Mail1271954641807-3007 from rgur...@localhost on 127.0.0.1 for [rlgur...@localhost] INFO 10:44:16,404 | james.mailetcontext | Error while storing mail. org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be parsed.; nested exception is: org.apache.james.imap.mailbox.StorageException: failed. Transaction commit failed.; nested exception is: openjpa-1.2.1-r752877:753278 fatal store error org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265) at org.apache.james.MailboxManagerPoster.post(MailboxManagerPoster.java:125) at org.apache.jsieve.mailet.SieveMailboxMailet.storeMessageInbox(SieveMailboxMailet.java:369) at org.apache.jsieve.mailet.SieveMailboxMailet.sieveMessage(SieveMailboxMailet.java:363) at org.apache.jsieve.mailet.SieveMailboxMailet.storeMail(SieveMailboxMailet.java:330) at org.apache.jsieve.mailet.SieveMailboxMailet.service(SieveMailboxMailet.java:263) at org.apache.james.transport.mailets.LocalDelivery.service(LocalDelivery.java:103) at org.apache.james.transport.camel.MailetProcessor.process(MailetProcessor.java:60) at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) at org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93) at org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) at org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49) at org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228) at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:51) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) at org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) at org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) at org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49) at org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228) at org.apache.camel.processor.Pipeline.process(Pipeline.java:74) at org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) at org.apache.camel.spring.spi.TransactionErrorHandler.processByRegularErrorHandler(TransactionErrorHandler.java:90) at org.apache.camel.spring.spi.TransactionErrorHandler.process(TransactionErrorHandler.java:81) at org.apache.camel.processor.MulticastProcessor.doProcess(MulticastProcessor.java:266) at org.apache.camel.processor.MulticastProcessor.access$000(MulticastProcessor.java:59) at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:178) at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:172) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at
[jira] Resolved: (IMAP-137) JPA fails to persist MailboxMembership Entity on heavy load
[ https://issues.apache.org/jira/browse/IMAP-137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norman Maurer resolved IMAP-137. Fix Version/s: 0.1 Resolution: Fixed Fixed.. JPA fails to persist MailboxMembership Entity on heavy load --- Key: IMAP-137 URL: https://issues.apache.org/jira/browse/IMAP-137 Project: JAMES Imap Issue Type: Bug Components: JPA Mailbox Reporter: Norman Maurer Assignee: Norman Maurer Fix For: 0.1 On heavy load JPA fails to persist MailboxMembership Enitity objects. Thats because it not use the right locking.. Exception: ** INFO 10:44:16,401 | james.smtpserver | Successfully spooled mail Mail1271954641807-3007 from rgur...@localhost on 127.0.0.1 for [rlgur...@localhost] INFO 10:44:16,404 | james.mailetcontext | Error while storing mail. org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be parsed.; nested exception is: org.apache.james.imap.mailbox.StorageException: failed. Transaction commit failed.; nested exception is: openjpa-1.2.1-r752877:753278 fatal store error org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265) at org.apache.james.MailboxManagerPoster.post(MailboxManagerPoster.java:125) at org.apache.jsieve.mailet.SieveMailboxMailet.storeMessageInbox(SieveMailboxMailet.java:369) at org.apache.jsieve.mailet.SieveMailboxMailet.sieveMessage(SieveMailboxMailet.java:363) at org.apache.jsieve.mailet.SieveMailboxMailet.storeMail(SieveMailboxMailet.java:330) at org.apache.jsieve.mailet.SieveMailboxMailet.service(SieveMailboxMailet.java:263) at org.apache.james.transport.mailets.LocalDelivery.service(LocalDelivery.java:103) at org.apache.james.transport.camel.MailetProcessor.process(MailetProcessor.java:60) at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) at org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93) at org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) at org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49) at org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228) at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:51) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) at org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) at org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) at org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49) at org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228) at org.apache.camel.processor.Pipeline.process(Pipeline.java:74) at org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) at org.apache.camel.spring.spi.TransactionErrorHandler.processByRegularErrorHandler(TransactionErrorHandler.java:90) at org.apache.camel.spring.spi.TransactionErrorHandler.process(TransactionErrorHandler.java:81) at org.apache.camel.processor.MulticastProcessor.doProcess(MulticastProcessor.java:266) at org.apache.camel.processor.MulticastProcessor.access$000(MulticastProcessor.java:59) at
Re: Celebration, plus: Release James 4.0M1?
non-binding +1 from a trunk user. On 04/28/2010 10:45 AM, Bernd Fondermann wrote: Hi, Thanks to Norman, Robert and everyone else - over the past month, there have been tons of changes and improvements made to James. Most of them are fantastic: modularization, IMAP, container, async io protocols. I can even deploy James as a WAR file now - This must be the future! Now, how dare we not to release this bag of beauties to our users? Why not have a milestone release adding a disclaimer that this isn't production ready and collect feedback from the field? And even if you'd call it 3.0 instead of the deserved 4.0, I'd be happy. :-) Here's my +1! Non-bindingly, Bernd - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
svn commit: r939071 - in /james/imap/trunk: deployment/src/test/java/org/apache/james/imap/functional/jpa/ jpa/src/main/java/org/apache/james/imap/jpa/ jpa/src/main/java/org/apache/james/imap/jpa/mail
Author: norman
Date: Wed Apr 28 20:07:29 2010
New Revision: 939071
URL: http://svn.apache.org/viewvc?rev=939071view=rev
Log:
* Fix for JPA fails to persist MailboxMembership Entity on heavy load by using
persimistic locking (IMAP-137)
* Do not cache Mailbox instances in store api to prevent GC (IMAP-131)
* Fix inmemory implementation
Removed:
james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/openjpa/OpenJPAMailboxMapper.java
Modified:
james/imap/trunk/deployment/src/test/java/org/apache/james/imap/functional/jpa/JPAHostSystem.java
james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/JPAMailbox.java
james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/JPAMailboxManager.java
james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/MailboxSessionEntityManagerFactory.java
james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/JPAMailboxMapper.java
james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/openjpa/OpenJPAMailbox.java
james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/openjpa/OpenJPAMailboxManager.java
james/imap/trunk/mailbox/src/main/java/org/apache/james/imap/mailbox/Content.java
james/imap/trunk/memory/src/main/java/org/apache/james/imap/inmemory/InMemoryMailboxManager.java
james/imap/trunk/store/src/main/java/org/apache/james/imap/store/DelegatingMailboxListener.java
james/imap/trunk/store/src/main/java/org/apache/james/imap/store/StoreMailbox.java
Modified:
james/imap/trunk/deployment/src/test/java/org/apache/james/imap/functional/jpa/JPAHostSystem.java
URL:
http://svn.apache.org/viewvc/james/imap/trunk/deployment/src/test/java/org/apache/james/imap/functional/jpa/JPAHostSystem.java?rev=939071r1=939070r2=939071view=diff
==
---
james/imap/trunk/deployment/src/test/java/org/apache/james/imap/functional/jpa/JPAHostSystem.java
(original)
+++
james/imap/trunk/deployment/src/test/java/org/apache/james/imap/functional/jpa/JPAHostSystem.java
Wed Apr 28 20:07:29 2010
@@ -66,7 +66,11 @@ public class JPAHostSystem extends ImapH
org.apache.james.imap.jpa.mail.model.JPAMessage; +
org.apache.james.imap.jpa.mail.model.JPAProperty; +
org.apache.james.imap.jpa.user.model.JPASubscription));
-
+// persimistic locking..
+properties.put(openjpa.LockManager, pessimistic);
+properties.put(openjpa.ReadLockLevel, read);
+properties.put(openjpa.WriteLockLevel, write);
+properties.put(openjpa.jdbc.TransactionIsolation, repeatable-read);
userManager = new InMemoryUserManager();
entityManagerFactory =
OpenJPAPersistence.getEntityManagerFactory(properties);
MailboxSessionEntityManagerFactory factory = new
MailboxSessionEntityManagerFactory(entityManagerFactory);
Modified:
james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/JPAMailbox.java
URL:
http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/JPAMailbox.java?rev=939071r1=939070r2=939071view=diff
==
---
james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/JPAMailbox.java
(original)
+++
james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/JPAMailbox.java
Wed Apr 28 20:07:29 2010
@@ -25,6 +25,8 @@ import java.util.List;
import javax.mail.Flags;
import javax.persistence.EntityManager;
+import javax.persistence.EntityTransaction;
+import javax.persistence.Query;
import org.apache.james.imap.jpa.mail.JPAMailboxMapper;
import org.apache.james.imap.jpa.mail.JPAMessageMapper;
@@ -105,12 +107,19 @@ public abstract class JPAMailbox extends
}
/**
- * Reserve next Uid in mailbox and return the mailbox. This method needs
to be synchronized
- * to be sure we don't get any race-condition
+ * Reserve next Uid in mailbox and return the mailbox. We use a
transaction here to be sure we don't get any duplicates
+ *
*/
protected MailboxLong reserveNextUid(MailboxSession session) throws
MailboxException {
-final JPAMailboxMapper mapper = createMailboxMapper(session);
-final MailboxLong mailbox = mapper.consumeNextUid(getMailboxId());
+EntityManager entityManager =
entityManagerFactory.getEntityManager(session);
+
+EntityTransaction transaction = entityManager.getTransaction();
+transaction.begin();
+Query query =
entityManager.createNamedQuery(findMailboxById).setParameter(idParam,
getMailboxId());
+org.apache.james.imap.jpa.mail.model.JPAMailbox mailbox =
(org.apache.james.imap.jpa.mail.model.JPAMailbox) query.getSingleResult();
+mailbox.consumeUid();
+entityManager.persist(mailbox);
+transaction.commit();
return mailbox;
-}
+}
}
Modified:
[jira] Resolved: (IMAP-131) StoreManager is caching Mailboxes and so the GC is not able to destroy the objects, which lead to memory which is not free up
[ https://issues.apache.org/jira/browse/IMAP-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norman Maurer resolved IMAP-131. Resolution: Fixed fixed .. :) StoreManager is caching Mailboxes and so the GC is not able to destroy the objects, which lead to memory which is not free up - Key: IMAP-131 URL: https://issues.apache.org/jira/browse/IMAP-131 Project: JAMES Imap Issue Type: Bug Components: Mailbox Reporter: Norman Maurer Assignee: Norman Maurer Fix For: 0.1 Currently the StoreManager is a caching the Mailboxes. This is bad because this will prevent the GC to destroy the Mailboxes if they are not used anymore and so Memory will not free -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
svn commit: r939072 - /james/server/trunk/spring-deployment/src/main/config/james/META-INF/persistence.xml
Author: norman Date: Wed Apr 28 20:12:22 2010 New Revision: 939072 URL: http://svn.apache.org/viewvc?rev=939072view=rev Log: Use persimitic locking for imap jpa .. Modified: james/server/trunk/spring-deployment/src/main/config/james/META-INF/persistence.xml Modified: james/server/trunk/spring-deployment/src/main/config/james/META-INF/persistence.xml URL: http://svn.apache.org/viewvc/james/server/trunk/spring-deployment/src/main/config/james/META-INF/persistence.xml?rev=939072r1=939071r2=939072view=diff == --- james/server/trunk/spring-deployment/src/main/config/james/META-INF/persistence.xml (original) +++ james/server/trunk/spring-deployment/src/main/config/james/META-INF/persistence.xml Wed Apr 28 20:12:22 2010 @@ -36,6 +36,13 @@ -- !-- create tables on startup -- property name=openjpa.jdbc.SynchronizeMappings value=buildSchema(ForeignKeys=true)/ + +!-- persimistic locking -- +property name=openjpa.LockManager value=pessimistic/ +property name=openjpa.ReadLockLevel value=read/ +property name=openjpa.WriteLockLevel value=write/ +property name=openjpa.jdbc.TransactionIsolation value=repeatable-read/ + /properties /persistence-unit /persistence - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
svn commit: r939073 - /james/server/trunk/mailets/src/main/java/org/apache/james/transport/matchers/AbstractStorageQuota.java
Author: norman
Date: Wed Apr 28 20:14:22 2010
New Revision: 939073
URL: http://svn.apache.org/viewvc?rev=939073view=rev
Log:
Call endprocessing after mailbox processing
Modified:
james/server/trunk/mailets/src/main/java/org/apache/james/transport/matchers/AbstractStorageQuota.java
Modified:
james/server/trunk/mailets/src/main/java/org/apache/james/transport/matchers/AbstractStorageQuota.java
URL:
http://svn.apache.org/viewvc/james/server/trunk/mailets/src/main/java/org/apache/james/transport/matchers/AbstractStorageQuota.java?rev=939073r1=939072r2=939073view=diff
==
---
james/server/trunk/mailets/src/main/java/org/apache/james/transport/matchers/AbstractStorageQuota.java
(original)
+++
james/server/trunk/mailets/src/main/java/org/apache/james/transport/matchers/AbstractStorageQuota.java
Wed Apr 28 20:14:22 2010
@@ -175,7 +175,7 @@ abstract public class AbstractStorageQuo
while (results.hasNext()) {
size += results.next().getSize();
}
-manager.startProcessingRequest(session);
+manager.endProcessingRequest(session);
manager.logout(session, true);
return size;
}
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Celebration, plus: Release James 4.0M1?
Hi Bernd, you are right. I will try to prepare everythink needed. We will at least need to eleminate snapshot dependencies before. I will analyze the dependency Graph and see what needs to get done. Bye Norman 2010/4/28, Bernd Fondermann bernd.fonderm...@googlemail.com: Hi, Thanks to Norman, Robert and everyone else - over the past month, there have been tons of changes and improvements made to James. Most of them are fantastic: modularization, IMAP, container, async io protocols. I can even deploy James as a WAR file now - This must be the future! Now, how dare we not to release this bag of beauties to our users? Why not have a milestone release adding a disclaimer that this isn't production ready and collect feedback from the field? And even if you'd call it 3.0 instead of the deserved 4.0, I'd be happy. :-) Here's my +1! Non-bindingly, Bernd - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Celebration, plus: Release James 4.0M1?
Hi Bernd, you are right. I will try to prepare everything needed. We will at least need to eleminate snapshot dependencies before. I will analyze the dependency Graph and see what needs to get done. Bye Norman 2010/4/28, Bernd Fondermann bernd.fonderm...@googlemail.com: Hi, Thanks to Norman, Robert and everyone else - over the past month, there have been tons of changes and improvements made to James. Most of them are fantastic: modularization, IMAP, container, async io protocols. I can even deploy James as a WAR file now - This must be the future! Now, how dare we not to release this bag of beauties to our users? Why not have a milestone release adding a disclaimer that this isn't production ready and collect feedback from the field? And even if you'd call it 3.0 instead of the deserved 4.0, I'd be happy. :-) Here's my +1! Non-bindingly, Bernd - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
