Re: Re: Re: [PROPOSAL] Replace MailRepository with Mailbox / MailboxManager for POP3Server ...
Hi Norman, Many tks for your quick answer. I had defined in my Thunderbird 3.0.3 client (on linux) the pop3 account as the default one. When launching thunderbird, the connection was succefully done Host contacted, sending login information, saying after 1 second Determining which messages to index (never saw that message before...), hanging there a few minutes, and finally saying connection to server ... timed out. After that, Imap account behaved also such a way... I also tried with latest Thundrebird 3.0.3: same result... I have now defined the imap account as the default one, and it's working fine even without any mail in the inbox. However, for the pop3 account, I still have the same behaviour. Anyway, I have a temporary workaround right now. I will try to debug it but it may take some time to jump into the new architecture (I hacked james 3 years ago, but now we have some new great stuff such as jpa, activemq, jackrabbit,...). Regarding the OOM, I saw some days ago the JAMES-981 regarding the ConsumerTemplate. However, the problem was still present even after you commited JAMES-981 into trunk, so I don't think the problem was coming from that. I tried to analyse the 2GB heap dump with eclipse memory analyser, but it failed (too much swapping, my RAM is only 1GB). Currently, the process memory with the new trunk version I deployed today seems stable. I will keep posted if I still have the issue, or if finally, the memory remains stable. Tks a lot and keep up the excellent work, Eric On 01/-10/-28163 08:59 PM, Norman Maurer wrote: Hi Eric, thx for the feedback . Does this happens with pop3 or imap ? Or with both? What client you use? About the OOM Exception, chances are good that this is fixed now. I did something bad with ConsumerTemplate in RemoteDelivery nailet, which could lead to such problems. Bye Norman 2010/3/27, Eric Charleseric.char...@u-mangate.com: Great Norman, I just upgraded to the current trunk to benefit from : - JAMES-983: common POP3/IMAP MailRepository. - JAMES-984: activemq 5.3.1 hoping it will resolve some OutOfMemory exceptions I had (*) I deployed and now (configured with enableVirtualHosting=true and JDBCDomainList), I've got the following: - When my mailbox contains at least one mail, No problem: I can read it without any problem. - When my mailbox is empty, my mail client simply waits and finally give me a timeout (closing my client and launching it again with mails in the inbox, everything is fine again). Can anybody else reproduce it? Is there additional specific configuration I have to apply ? Tks a lot, Eric (*) Just for information, about the OutOfMemory I had with activemq 3.5.0: - Testing different configurations from http://activemq.apache.org/javalangoutofmemory.html didn't help. - Stressing a local deployment with apache-sever-postage, server was holding the pressure during hours, however, memory was slightly but constantly growing. - I have a heap dump to be analyzed (still todo) - Reading 6 memory leak bugs resolved in activemq 3.5.1. (https://issues.apache.org/activemq/secure/ReleaseNote.jspa?projectId=10520styleName=Htmlversion=12183), I decided to try to migrate to activemq 5.3.1, and by chance, Norman checked it in two days ago. I still have to wait a few days to see if the OutOfMemory problem is resolved. On 12/23/-28158 08:59 PM, Norman Maurer wrote: Ok it seems like this works for everyone so far.. I will commit it this weekend .. Thanks again for all the feedback / questions. Bye, Norman 2010/3/26 Stefano Bagnaraapa...@bago.org: 2010/3/24 Norman Maurernor...@apache.org: Hi all, I want to propose some really heavy change in current JAMES trunk, and so next version. As all of you knows we are supporting IMAP in current development version, which ships with its own mail store backend called MailboxManager / Mailbox. For POP3 we use MailRepository as backend. I think this is a no go for a number of reasons, but the major one is that we should be able to switch between IMAP and POP3 without the need to migrate mails. So I rewrote the POP3Server to re-use the MailboxManager / Mailbox stuff which is used by IMAP. I think this is a good think. I also hope MailboxManager/Mailbox API can be improved and simplified, but using a single API is the only way to go. Once we use only one we'll better understand how to improve it. So if a user login via POP3 he will just see the folder called INBOX and nothing else. With IMAP he will see all folders. Thats exactly what dovecot and courier does ( both heavy used unix imap/pop3 servers). A good extension would be to have a login pattern to read custom folders.. e.g: I login as username#FOLDER with username password and this way I get access to FOLDER instead of INBOX. The same could be applied to the delete behaviour (something like the recent: prefix hack in gmail). Another advance is
Re: [PROPOSAL] Replace MailRepository with Mailbox / MailboxManager for POP3Server ...
Hi Norman, With wireshark, thunderbird/james hangs at POP3 STAT command. I saw you patched the StatCmdHandler. It now works even without any message in my mailbox (after update/package/deploy). Great! However: - I received a few exceptions complaining about the transaction (can not initiate a new one while still an active) while retrieving messages and the spool was doing some work. - I saw yesterday some strange behaviour when I made tests with different clients : deleted messages were still fetched by new clients. I will further investigate to post with more consolidated info. Many tks, Eric On 03/29/2010 08:53 AM, Norman Maurer wrote: Hi Eric, I was able to reproduce your problem and just checked in the fix. Please svn up and try again.. Thx for your feedback. Bye, Norman 2010/3/29 Norman Maurernor...@apache.org: Hi Eric, I just tried to acces an empty inbox via thunderbird (pop3) here. It worked without any probs. Im using windows 7 and thunderbird 3.0.3 Bye, Norman 2010/3/28 Norman Maurernor...@apache.org: -- Forwarded message -- From: Norman Maurernor...@apache.org Date: 2010/3/28 Subject: Re: [PROPOSAL] Replace MailRepository with Mailbox / MailboxManager for POP3Server ... To: Eric Charleseric.char...@u-mangate.com Hi Eric, could you maybe use wireshark or something similar to log what thunderbird tries todo exactly.. ? Thx, Norman 2010/3/28 Eric Charleseric.char...@u-mangate.com: Hi Norman, I just made the following tests: - Fresh checkout from trunk and mvn package (some new dependencies were downloaded such as jsieve, mailet-sdk,...) = same problem on a local deployment with thunderbird : it hangs when mailbox is empty. - I after defined my production account on evolution (http://www.gnome.org/projects/evolution) : I didn't have the problem anymore, but really strange, evolution downloaded some messages from yesterday. Those messages had been deleted from my inbox via thunderbird, but was retrieved via evolution... - I tested with evolution the local deployment I just made : I can read the inbox even if empty. The problem seems to reside on the thunderbird/james conversation when inbox is empty. Tks, Eric On 03/27/2010 10:08 PM, Norman Maurer wrote: Hi Eric, I just tested pop3 with imail and no messages in inbox and everything just works. Will try to test it with thunderbird later.. Bye, Norman 2010/3/27 Norman Maurernor...@apache.org: Hi Eric, I will try to find out whats wrong with the pop3 inbox stuff. If you find out anything else please feel free to just report it.. Thx, Norman 2010/3/27 Eric Charleseric.char...@u-mangate.com: Hi Norman, Many tks for your quick answer. I had defined in my Thunderbird 3.0.3 client (on linux) the pop3 account as the default one. When launching thunderbird, the connection was succefully done Host contacted, sending login information, saying after 1 second Determining which messages to index (never saw that message before...), hanging there a few minutes, and finally saying connection to server ... timed out. After that, Imap account behaved also such a way... I also tried with latest Thundrebird 3.0.3: same result... I have now defined the imap account as the default one, and it's working fine even without any mail in the inbox. However, for the pop3 account, I still have the same behaviour. Anyway, I - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
OutOfMemory exception
Hi, The last two weeks, I deployed various trunk snapshots. After james running less than 1 day, I always ran into a OutOfMemory Exception. I analysed the logs (for example STACK TRACE 1 in annex), and always found Camel complaining when trying to deliver on JMS queue. I tried to adapt the ActiveMQ configuration based on http://activemq.apache.org/javalangoutofmemory.html but nothing helped. I also analyzed various heap dump :sometimes, 65% was used by org.apache.mina.transport.socket.nio.NioSocketSession, sometimes by a activemq class,... Yesterday, I had a more explicit stacktrace (see STACK TRACE 2 in annex) talking about Too many open files in the MimeMessageInputStreamSource constructor when trying file = File.createTempFile(key, .m64). Normally, the reference to the file should disappear when invoking the dispose method (doing a file.delete();). But, I've got many old files in my /tmp (one more file each minute), so I suppose that for a reason or another, those files are not deleted. This could cause the Too many open files problem, leaving the JMS Producers in memory with any Consumers... ? Finally, I don't know if the OOM comes from the Too many open files or still from somewhere else ? Please note that I define the address-error processors as Null (don't want to keep those mails on disk) and that I have activated the SpamAssassin mailet. But even without those extra configs, I've got the OOM. Any idea, Eric STACK TRACE 1 INFO 16:23:01,385 | james.smtpserver | Successfully spooled mail Mail1269872581383-7178 from johnsmith...@u-mangate.org on 120.56.194.53 for [johnsmith...@u-mangate.org] INFO 16:23:01,386 | james.smtpserver | Successfully spooled mail Mail1269872581383-7179 from catchthism...@u-mangate.net on 120.56.194.53 for [catchthism...@u-mangate.net] INFO 16:23:20,887 | james.smtpserver | Successfully spooled mail Mail1269872600878-7183 from 3edd9016.4070...@u-mangate.be on 120.56.194.53 for [3edd9016.4070...@u-mangate.be] INFO 16:23:22,007 | james.smtpserver | Successfully spooled mail Mail1269872602002-7185 from 3c3cd8ce0b.4020...@u-mangate.org on 120.56.194.53 for [3c3cd8ce0b.4020...@u-mangate.org] INFO 16:23:23,336 | james.smtpserver | Successfully spooled mail Mail1269872603328-7186 from elizabethsorensen...@bbr-vt.de on 95.169.149.179 for [angelo.bid...@u-mangate.net] INFO 16:23:34,899 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 1115 INFO 16:23:37,254 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 1240 ms, Index update took 0 ms INFO 16:23:37,255 | james.smtpserver | Successfully spooled mail Mail1269872616013-7188 from 3ee83341.3030...@u-mangate.be on 120.56.194.53 for [3ee83341.3030...@u-mangate.be] INFO 16:23:42,140 | james.smtpserver | Successfully spooled mail Mail1269872622133-7190 from m...@u-mangate.net on 120.56.194.53 for [m...@u-mangate.net] INFO 16:23:56,193 | james.smtpserver | Successfully spooled mail Mail1269872633794-7195 from shauna.escoba...@superiorsleep.co.nz on 121.147.241.76 for [angelo.bid...@u-mangate.net] INFO 16:23:56,195 | james.smtpserver | Successfully spooled mail Mail1269872633798-7196 from 3ee822b7.10...@u-mangate.be on 120.56.194.53 for [3ee822b7.10...@u-mangate.be] INFO 16:23:59,740 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 3543 ms, Index update took 0 ms INFO 16:23:59,743 | james.smtpserver | Successfully spooled mail Mail1269872636196-7198 from ric.char...@u-mangate.org on 120.56.194.53 for [ric.char...@u-mangate.org] INFO 16:24:01,066 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 1322 ms, Index update took 0 ms INFO 16:24:01,066 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 1321 ms, Index update took 0 ms INFO 16:24:01,069 | james.smtpserver | Successfully spooled mail Mail1269872639742-7199 from stephane.liu...@u-mangate.net on 120.56.194.53 for [stephane.liu...@u-mangate.net] INFO 16:24:10,625 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 7100 INFO 16:24:10,626 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 2458 ms, Index update took 7099 ms INFO 16:24:19,753 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 5770 INFO 16:24:19,753 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 3358 ms, Index update took 5769 ms INFO 16:24:28,832 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 4584 INFO 16:24:53,501 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 22422 INFO 16:24:53,502 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 0 ms, Index update took
Re: OutOfMemory exception
Norman, Done. Now, we have to wait... I saw some created *.m64 that were removed. But there are other ones that remains in /tmp. I sometimes run a jmap (java memory map) to produce a heap dump and analyse it. At the beginning, everything seems ok, and often, when I come back, the OOM has already occured. I changed the -Xmx512m to -Xmx256m to have a quicker exception (if any, let's hope not). The *.m64 are a clue and we should ensure that they are removed in all circumstances. This may be the easy part, as we have some visible clues (the files). We should also ensure after that there are no other causes to the leaks. Tks, Eric On 03/29/2010 08:00 PM, Norman Maurer wrote: Hi Eric, I found the cause for the not deleted temporary files. Hopefully this is the cause of the OOM. Could try to svn up and run again ? Thx, Norman 2010/3/29 Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. I see exact the same problem today here.. (The OOM). Didn't notice the files in the tmp folder, but I think thats a good pointer. I will try to debug the problem later or tomorrow. But I suspect you are right about the tmp files and jms producers.. Did you run a kill -3 pid to see what threads are active etc ? Thx, Norman Ps: Patches are welcome :) 2010/3/29 Eric Charleseric.char...@u-mangate.com: Hi, The last two weeks, I deployed various trunk snapshots. After james running less than 1 day, I always ran into a OutOfMemory Exception. I analysed the logs (for example STACK TRACE 1 in annex), and always found Camel complaining when trying to deliver on JMS queue. I tried to adapt the ActiveMQ configuration based on http://activemq.apache.org/javalangoutofmemory.html but nothing helped. I also analyzed various heap dump :sometimes, 65% was used by org.apache.mina.transport.socket.nio.NioSocketSession, sometimes by a activemq class,... Yesterday, I had a more explicit stacktrace (see STACK TRACE 2 in annex) talking about Too many open files in the MimeMessageInputStreamSource constructor when trying file = File.createTempFile(key, .m64). Normally, the reference to the file should disappear when invoking the dispose method (doing a file.delete();). But, I've got many old files in my /tmp (one more file each minute), so I suppose that for a reason or another, those files are not deleted. This could cause the Too many open files problem, leaving the JMS Producers in memory with any Consumers... ? Finally, I don't know if the OOM comes from the Too many open files or still from somewhere else ? Please note that I define the address-error processors as Null (don't want to keep those mails on disk) and that I have activated the SpamAssassin mailet. But even without those extra configs, I've got the OOM. Any idea, Eric STACK TRACE 1 INFO 16:23:01,385 | james.smtpserver | Successfully spooled mail Mail1269872581383-7178 from johnsmith...@u-mangate.org on 120.56.194.53 for [johnsmith...@u-mangate.org] INFO 16:23:01,386 | james.smtpserver | Successfully spooled mail Mail1269872581383-7179 from catchthism...@u-mangate.net on 120.56.194.53 for [catchthism...@u-mangate.net] INFO 16:23:20,887 | james.smtpserver | Successfully spooled mail Mail1269872600878-7183 from 3edd9016.4070...@u-mangate.be on 120.56.194.53 for [3edd9016.4070...@u-mangate.be] INFO 16:23:22,007 | james.smtpserver | Successfully spooled mail Mail1269872602002-7185 from 3c3cd8ce0b.4020...@u-mangate.org on 120.56.194.53 for [3c3cd8ce0b.4020...@u-mangate.org] INFO 16:23:23,336 | james.smtpserver | Successfully spooled mail Mail1269872603328-7186 from elizabethsorensen...@bbr-vt.de on 95.169.149.179 for [angelo.bid...@u-mangate.net] INFO 16:23:34,899 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 1115 INFO 16:23:37,254 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 1240 ms, Index update took 0 ms INFO 16:23:37,255 | james.smtpserver | Successfully spooled mail Mail1269872616013-7188 from 3ee83341.3030...@u-mangate.be on 120.56.194.53 for [3ee83341.3030...@u-mangate.be] INFO 16:23:42,140 | james.smtpserver | Successfully spooled mail Mail1269872622133-7190 from m...@u-mangate.net on 120.56.194.53 for [m...@u-mangate.net] INFO 16:23:56,193 | james.smtpserver | Successfully spooled mail Mail1269872633794-7195 from shauna.escoba...@superiorsleep.co.nz on 121.147.241.76 for [angelo.bid...@u-mangate.net] INFO 16:23:56,195 | james.smtpserver | Successfully spooled mail Mail1269872633798-7196 from 3ee822b7.10...@u-mangate.be on 120.56.194.53 for [3ee822b7.10...@u-mangate.be] INFO 16:23:59,740 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 3543 ms, Index update took 0 ms INFO 16:23:59,743 | james.smtpserver | Successfully spooled mail Mail1269872636196-7198 from ric.char...@u-mangate.org on 120.56.194.53 for [ric.char...@u-mangate.org] INFO 16:24:01,066 |
Re: OutOfMemory exception
Hi Norman, The .m64 are all to unkown user but to well known domain (so for unkn...@known.com). They are from various size (with and without attachment). They are well formed (I can open the downloaded file with thunderbird) However, when I sent a mail to unkn...@known.com, I don't see it in the /tmp Tks, Eric On 03/29/2010 08:43 PM, Norman Maurer wrote: Hi Eric, sure.. we have to find the OOM cause. What would be interesting, could you check somehow if the .m64 files are files which are related to successfully delivered mail ? Thx, Norman 2010/3/29 Eric Charleseric.char...@u-mangate.com: Norman, Done. Now, we have to wait... I saw some created *.m64 that were removed. But there are other ones that remains in /tmp. I sometimes run a jmap (java memory map) to produce a heap dump and analyse it. At the beginning, everything seems ok, and often, when I come back, the OOM has already occured. I changed the -Xmx512m to -Xmx256m to have a quicker exception (if any, let's hope not). The *.m64 are a clue and we should ensure that they are removed in all circumstances. This may be the easy part, as we have some visible clues (the files). We should also ensure after that there are no other causes to the leaks. Tks, Eric On 03/29/2010 08:00 PM, Norman Maurer wrote: Hi Eric, I found the cause for the not deleted temporary files. Hopefully this is the cause of the OOM. Could try to svn up and run again ? Thx, Norman 2010/3/29 Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. I see exact the same problem today here.. (The OOM). Didn't notice the files in the tmp folder, but I think thats a good pointer. I will try to debug the problem later or tomorrow. But I suspect you are right about the tmp files and jms producers.. Did you run a kill -3 pid to see what threads are active etc ? Thx, Norman Ps: Patches are welcome :) 2010/3/29 Eric Charleseric.char...@u-mangate.com: Hi, The last two weeks, I deployed various trunk snapshots. After james running less than 1 day, I always ran into a OutOfMemory Exception. I analysed the logs (for example STACK TRACE 1 in annex), and always found Camel complaining when trying to deliver on JMS queue. I tried to adapt the ActiveMQ configuration based on http://activemq.apache.org/javalangoutofmemory.html but nothing helped. I also analyzed various heap dump :sometimes, 65% was used by org.apache.mina.transport.socket.nio.NioSocketSession, sometimes by a activemq class,... - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OutOfMemory exception
Hi Norman, I just deployed your new commit with the new camel DisposeProcess. Good news : I don't see anymore the m64 file in tmp (well I see 1 file, on the second after, it is no more there, so the dispose works as it should). I keep you posted with our eventual future OOM. Tks, Eric On 03/29/2010 09:18 PM, Eric Charles wrote: Hi Norman, The .m64 are all to unkown user but to well known domain (so for unkn...@known.com). They are from various size (with and without attachment). They are well formed (I can open the downloaded file with thunderbird) However, when I sent a mail to unkn...@known.com, I don't see it in the /tmp Tks, Eric On 03/29/2010 08:43 PM, Norman Maurer wrote: Hi Eric, sure.. we have to find the OOM cause. What would be interesting, could you check somehow if the .m64 files are files which are related to successfully delivered mail ? Thx, Norman 2010/3/29 Eric Charleseric.char...@u-mangate.com: Norman, Done. Now, we have to wait... I saw some created *.m64 that were removed. But there are other ones that remains in /tmp. I sometimes run a jmap (java memory map) to produce a heap dump and analyse it. At the beginning, everything seems ok, and often, when I come back, the OOM has already occured. I changed the -Xmx512m to -Xmx256m to have a quicker exception (if any, let's hope not). The *.m64 are a clue and we should ensure that they are removed in all circumstances. This may be the easy part, as we have some visible clues (the files). We should also ensure after that there are no other causes to the leaks. Tks, Eric On 03/29/2010 08:00 PM, Norman Maurer wrote: Hi Eric, I found the cause for the not deleted temporary files. Hopefully this is the cause of the OOM. Could try to svn up and run again ? Thx, Norman 2010/3/29 Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. I see exact the same problem today here.. (The OOM). Didn't notice the files in the tmp folder, but I think thats a good pointer. I will try to debug the problem later or tomorrow. But I suspect you are right about the tmp files and jms producers.. Did you run a kill -3 pid to see what threads are active etc ? Thx, Norman Ps: Patches are welcome :) 2010/3/29 Eric Charleseric.char...@u-mangate.com: Hi, The last two weeks, I deployed various trunk snapshots. After james running less than 1 day, I always ran into a OutOfMemory Exception. I analysed the logs (for example STACK TRACE 1 in annex), and always found Camel complaining when trying to deliver on JMS queue. I tried to adapt the ActiveMQ configuration based on http://activemq.apache.org/javalangoutofmemory.html but nothing helped. I also analyzed various heap dump :sometimes, 65% was used by org.apache.mina.transport.socket.nio.NioSocketSession, sometimes by a activemq class,... - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OutOfMemory exception
Oops, no, the files are still there (only unkn...@known.com). Eric On 03/29/2010 10:16 PM, Eric Charles wrote: Hi Norman, I just deployed your new commit with the new camel DisposeProcess. Good news : I don't see anymore the m64 file in tmp (well I see 1 file, on the second after, it is no more there, so the dispose works as it should). I keep you posted with our eventual future OOM. Tks, Eric On 03/29/2010 09:18 PM, Eric Charles wrote: Hi Norman, The .m64 are all to unkown user but to well known domain (so for unkn...@known.com). They are from various size (with and without attachment). They are well formed (I can open the downloaded file with thunderbird) However, when I sent a mail to unkn...@known.com, I don't see it in the /tmp Tks, Eric On 03/29/2010 08:43 PM, Norman Maurer wrote: Hi Eric, sure.. we have to find the OOM cause. What would be interesting, could you check somehow if the .m64 files are files which are related to successfully delivered mail ? Thx, Norman 2010/3/29 Eric Charleseric.char...@u-mangate.com: Norman, Done. Now, we have to wait... I saw some created *.m64 that were removed. But there are other ones that remains in /tmp. I sometimes run a jmap (java memory map) to produce a heap dump and analyse it. At the beginning, everything seems ok, and often, when I come back, the OOM has already occured. I changed the -Xmx512m to -Xmx256m to have a quicker exception (if any, let's hope not). The *.m64 are a clue and we should ensure that they are removed in all circumstances. This may be the easy part, as we have some visible clues (the files). We should also ensure after that there are no other causes to the leaks. Tks, Eric On 03/29/2010 08:00 PM, Norman Maurer wrote: Hi Eric, I found the cause for the not deleted temporary files. Hopefully this is the cause of the OOM. Could try to svn up and run again ? Thx, Norman 2010/3/29 Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. I see exact the same problem today here.. (The OOM). Didn't notice the files in the tmp folder, but I think thats a good pointer. I will try to debug the problem later or tomorrow. But I suspect you are right about the tmp files and jms producers.. Did you run a kill -3 pid to see what threads are active etc ? Thx, Norman Ps: Patches are welcome :) 2010/3/29 Eric Charleseric.char...@u-mangate.com: Hi, The last two weeks, I deployed various trunk snapshots. After james running less than 1 day, I always ran into a OutOfMemory Exception. I analysed the logs (for example STACK TRACE 1 in annex), and always found Camel complaining when trying to deliver on JMS queue. I tried to adapt the ActiveMQ configuration based on http://activemq.apache.org/javalangoutofmemory.html but nothing helped. I also analyzed various heap dump :sometimes, 65% was used by org.apache.mina.transport.socket.nio.NioSocketSession, sometimes by a activemq class,... - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OutOfMemory exception
Hi Norman, I had defined the Null mailet mailet match=HostIsLocal class=Null processor local-address-error /processor notice550 - Requested action not taken: no such user here/notice /mailet but now, I use the standard config mailet match=HostIsLocal class=ToProcessor processor local-address-error /processor notice550 - Requested action not taken: no such user here/notice /mailet I still have the OOM. I will now deploy a fresh svn with your last commits and enable the ValidRcptHandler. I keep you posted with the result, Tks, Eric On 03/30/2010 06:49 AM, Norman Maurer wrote: Hi Eric, you said all the files are related to address-errors , could you show me your address error processor config? Does the Problem still exist when you enable the ValidRcptHandler in the smtpserver.xml file? Thx Norman 2010/3/30, Eric Charleseric.char...@u-mangate.com: Oops, no, the files are still there (only unkn...@known.com). Eric On 03/29/2010 10:16 PM, Eric Charles wrote: Hi Norman, I just deployed your new commit with the new camel DisposeProcess. Good news : I don't see anymore the m64 file in tmp (well I see 1 file, on the second after, it is no more there, so the dispose works as it should). I keep you posted with our eventual future OOM. Tks, Eric On 03/29/2010 09:18 PM, Eric Charles wrote: Hi Norman, The .m64 are all to unkown user but to well known domain (so for unkn...@known.com). They are from various size (with and without attachment). They are well formed (I can open the downloaded file with thunderbird) However, when I sent a mail to unkn...@known.com, I don't see it in the /tmp Tks, Eric On 03/29/2010 08:43 PM, Norman Maurer wrote: Hi Eric, sure.. we have to find the OOM cause. What would be interesting, could you check somehow if the .m64 files are files which are related to successfully delivered mail ? Thx, Norman 2010/3/29 Eric Charleseric.char...@u-mangate.com: Norman, Done. Now, we have to wait... I saw some created *.m64 that were removed. But there are other ones that remains in /tmp. I sometimes run a jmap (java memory map) to produce a heap dump and analyse it. At the beginning, everything seems ok, and often, when I come back, the OOM has already occured. I changed the -Xmx512m to -Xmx256m to have a quicker exception (if any, let's hope not). The *.m64 are a clue and we should ensure that they are removed in all circumstances. This may be the easy part, as we have some visible clues (the files). We should also ensure after that there are no other causes to the leaks. Tks, Eric On 03/29/2010 08:00 PM, Norman Maurer wrote: Hi Eric, I found the cause for the not deleted temporary files. Hopefully this is the cause of the OOM. Could try to svn up and run again ? Thx, Norman 2010/3/29 Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. I see exact the same problem today here.. (The OOM). Didn't notice the files in the tmp folder, but I think thats a good pointer. I will try to debug the problem later or tomorrow. But I suspect you are right about the tmp files and jms producers.. Did you run a kill -3 pid to see what threads are active etc ? Thx, Norman Ps: Patches are welcome :) 2010/3/29 Eric Charleseric.char...@u-mangate.com: Hi, The last two weeks, I deployed various trunk snapshots. After james running less than 1 day, I always ran into a OutOfMemory Exception. I analysed the logs (for example STACK TRACE 1 in annex), and always found Camel complaining when trying to deliver on JMS queue. I tried to adapt the ActiveMQ configuration based on http://activemq.apache.org/javalangoutofmemory.html but nothing helped. I also analyzed various heap dump :sometimes, 65% was used by org.apache.mina.transport.socket.nio.NioSocketSession, sometimes by a activemq class,... - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OutOfMemory exception
suspect the files are not the cause of the OOM. I didn't get a OOM since yesterday morning (the time I deployed current trunk version), but just found to new .m64 files.. So I'm still searching for the real cause. If nothing helps I will need to use a profiler to find th leak. Bye, Norman 2010/3/31 Eric Charleseric.char...@u-mangate.com: Hi Norman, I had defined the Null mailet mailet match=HostIsLocal class=Null processor local-address-error/processor notice550 - Requested action not taken: no such user here/notice /mailet but now, I use the standard config mailet match=HostIsLocal class=ToProcessor processor local-address-error/processor notice550 - Requested action not taken: no such user here/notice /mailet I still have the OOM. I will now deploy a fresh svn with your last commits and enable the ValidRcptHandler. I keep you posted with the result, Tks, Eric On 03/30/2010 06:49 AM, Norman Maurer wrote: Hi Eric, you said all the files are related to address-errors , could you show me your address error processor config? Does the Problem still exist when you enable the ValidRcptHandler in the smtpserver.xml file? Thx Norman 2010/3/30, Eric Charleseric.char...@u-mangate.com: Oops, no, the files are still there (only unkn...@known.com). Eric On 03/29/2010 10:16 PM, Eric Charles wrote: Hi Norman, I just deployed your new commit with the new camel DisposeProcess. Good news : I don't see anymore the m64 file in tmp (well I see 1 file, on the second after, it is no more there, so the dispose works as it should). I keep you posted with our eventual future OOM. Tks, Eric On 03/29/2010 09:18 PM, Eric Charles wrote: Hi Norman, The .m64 are all to unkown user but to well known domain (so for unkn...@known.com). They are from various size (with and without attachment). They are well formed (I can open the downloaded file with thunderbird) However, when I sent a mail to unkn...@known.com, I don't see it in the /tmp Tks, Eric On 03/29/2010 08:43 PM, Norman Maurer wrote: Hi Eric, sure.. we have to find the OOM cause. What would be interesting, could you check somehow if the .m64 files are files which are related to successfully delivered mail ? Thx, Norman 2010/3/29 Eric Charleseric.char...@u-mangate.com: Norman, Done. Now, we have to wait... I saw some created *.m64 that were removed. But there are other ones that remains in /tmp. I sometimes run a jmap (java memory map) to produce a heap dump and analyse it. At the beginning, everything seems ok, and often, when I come back, the OOM has already occured. I changed the -Xmx512m to -Xmx256m to have a quicker exception (if any, let's hope not). The *.m64 are a clue and we should ensure that they are removed in all circumstances. This may be the easy part, as we have some visible clues (the files). We should also ensure after that there are no other causes to the leaks. Tks, Eric On 03/29/2010 08:00 PM, Norman Maurer wrote: Hi Eric, I found the cause for the not deleted temporary files. Hopefully this is the cause of the OOM. Could try to svn up and run again ? Thx, Norman 2010/3/29 Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. I see exact the same problem today here.. (The OOM). Didn't notice the files in the tmp folder, but I think thats a good pointer. I will try to debug the problem later or tomorrow. But I suspect you are right about the tmp files and jms producers.. Did you run a kill -3 pid to see what threads are active etc ? Thx, Norman Ps: Patches are welcome :) 2010/3/29 Eric Charleseric.char...@u-mangate.com: Hi, The last two weeks, I deployed various trunk snapshots. After james running less than 1 day, I always ran into a OutOfMemory Exception. I analysed the logs (for example STACK TRACE 1 in annex), and always found Camel complaining when trying to deliver on JMS queue. I tried to adapt the ActiveMQ configuration based on http://activemq.apache.org/javalangoutofmemory.html but nothing helped. I also analyzed various heap dump :sometimes, 65% was used by org.apache.mina.transport.socket.nio.NioSocketSession, sometimes by a activemq class,... - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
Re: OutOfMemory exception
, destination = queue://processor.root, transactionId = null, expiration = 0, timestamp = 1270014325685, arrival = 0, brokerInTime = 1270014326893, brokerOutTime = 1270014333023, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.byteseque...@69b568d0, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 10718, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}]. Caused by: [java.lang.OutOfMemoryError - Java heap space] at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055) at org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154) ... I also think the tmp files are not the main cause. I have now james running in eclipse and I made already made a little trip in the code. Quite impressive since last time I looked at it. I will try to stress james with a small smtp/pop3 client (I used postage, but a simple class with commons-net behind) may be easier and see what happens (eventually with eclipse profiler). It will be for this weekend for me. Bye, Eric On 03/31/2010 07:11 PM, Norman Maurer wrote: Hi Eric, thx for keeping us in the loop... I'm still not sure why the .m64 files are still in the tmp folder sometimes.. But I suspect the files are not the cause of the OOM. I didn't get a OOM since yesterday morning (the time I deployed current trunk version), but just found to new .m64 files.. So I'm still searching for the real cause. If nothing helps I will need to use a profiler to find th leak. Bye, Norman 2010/3/31 Eric Charleseric.char...@u-mangate.com: Hi Norman, I had defined the Null mailet mailet match=HostIsLocal class=Null processor local-address-error/processor notice550 - Requested action not taken: no such user here/notice /mailet but now, I use the standard config mailet match=HostIsLocal class=ToProcessor processor local-address-error/processor notice550 - Requested action not taken: no such user here/notice /mailet I still have the OOM. I will now deploy a fresh svn with your last commits and enable the ValidRcptHandler. I keep you posted with the result, Tks, Eric On 03/30/2010 06:49 AM, Norman Maurer wrote: Hi Eric, you said all the files are related to address-errors , could you show me your address error processor config? Does the Problem still exist when you enable the ValidRcptHandler in the smtpserver.xml file? Thx Norman 2010/3/30, Eric Charleseric.char...@u-mangate.com: Oops, no, the files are still there (only unkn...@known.com). Eric On 03/29/2010 10:16 PM, Eric Charles wrote: Hi Norman, I just deployed your new commit with the new camel DisposeProcess. Good news : I don't see anymore the m64 file in tmp (well I see 1 file, on the second after, it is no more there, so the dispose works as it should). I keep you posted with our eventual future OOM. Tks, Eric On 03/29/2010 09:18 PM, Eric Charles wrote: Hi Norman, The .m64 are all to unkown user but to well known domain (so for unkn...@known.com). They are from various size (with and without attachment). They are well formed (I can open the downloaded file with thunderbird) However, when I sent a mail to unkn...@known.com, I don't see it in the /tmp Tks, Eric On 03/29/2010 08:43 PM, Norman Maurer wrote: Hi Eric, sure.. we have to find the OOM cause. What would be interesting, could you check somehow if the .m64 files are files which are related to successfully delivered mail ? Thx, Norman 2010/3/29 Eric Charleseric.char...@u-mangate.com: Norman, Done. Now, we have to wait... I saw some created *.m64 that were removed. But there are other ones that remains in /tmp. I sometimes run a jmap (java memory map) to produce a heap dump and analyse it. At the beginning, everything seems ok, and often, when I come back, the OOM has already occured. I changed the -Xmx512m to -Xmx256m to have a quicker exception (if any, let's hope not). The *.m64 are a clue and we should ensure that they are removed in all circumstances. This may be the easy part, as we have some visible clues (the files). We should also ensure after that there are no other causes to the leaks. Tks, Eric On 03/29/2010 08:00 PM, Norman Maurer wrote: Hi Eric, I found the cause for the not deleted temporary files. Hopefully this is the cause of the OOM. Could try to svn up and run again ? Thx, Norman 2010/3/29 Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. I see exact the same problem today here.. (The OOM). Didn't notice the files in the tmp folder
Re: OutOfMemory exception
Tks Stefano for the precisions.
I keep these in mind and already took last week some head dumps via
-XX:+HeapDumpOnOutOfMemoryError and jmap.
I have to further analyze the dumps and run full profiling this weekend.
For sure increasing the memory will still produce a OOM, but I have a
bit more time before process crash.
Tks a lot,
Eric
On 04/01/2010 09:44 AM, Stefano Bagnara wrote:
2010/4/1 Eric Charleseric.char...@u-mangate.com:
Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets,
[...]
So still a OOM exception that was shown by yet-another-component (in this
case, the StoreMailbox).
OOM are shown by whichever component needs memory once the memory is
exausted. So there's almost no point in taking into consideration the
exception stacktrace when an OOM happens in a complex system.
OOM are the results of (a) real insufficient memory (too big memory
requirements), (b) memory leaks.
So, either some component is configured to use more memory than the
available or some component does not free resources. I guess we are in
(b).
So, either you go for a full profiler, or you at least take heap dumps.
We have to know if memory usage grows constantly to OOM, or if you
have very frequent GC that free space but then once in a while it is
not enough and it throws the OOM, if the memory is full of unused
objects from the same class or instead a full tree of different
objects.
If you don't go for a full profiler, jmap -histo, jmap -dump, jstat,
jmap, jconsole are your friends here.
Also, add the -XX:+HeapDumpOnOutOfMemoryError parameter to your jvm,
so that you have an automatic heap dump on OOM (you can also set this
live with jinfo)
Also some other guessed information can help: the memory usage is
proportional to the processed message? To their sizes? To the uptime?
To the failed message..etc.
There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
job).
All 4 files were 0 bytes.
I have now launched with EXTRA_JVM_ARGUMENTS=-Xms512m -Xmx4g (so 4GB max
memory).
With the previous parameters ( -Xmx512m), the process was taking the whole
512MB.
Increasing the memory is rarely of help in this case: this will only
help if we are in the (a) scenario (some component configured to use
more memory than we thought). You'll probably get the OOM anyway, but
it will take more time. If this happen we cat then exclude (a) and go
for (b) analysis.
Stefano
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OutOfMemory exception
Hello Norman, Done :) Tks, Eric PS1 : I didn't change the config, except for the JDCBDomainList I am using. PS2 : I only checkout http://svn.apache.org/repos/asf/james/server/trunk and not the other projects. From what I read from the poms, snapshot version of imap-mailbox, jsieve,... are taken from the central maven repository and I don't have to rebuild those projects. Hi Eric, its me again ;) Could you please svn up to latest trunk and try again ? Thx, Norman 2010/4/1 Norman Maurernorman.mau...@googlemail.com: Hi Eric, I did a stress test in the last few hours (running in the background here) and I was not able to get a OOM yet with ?broker.useJmx=falseamp;jms.prefetchPolicy.all=1 for the jmsConnectionFactory. I sent 4 emails with 100k to it and 3000 with 5 mb. I'm using -Xmx512m and memory usage is not getting higher then 700m. Bye. Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OutOfMemory exception
James survived the night. 300MB (exactly the same as when it started). No m64 file in tmp. I will post another update this evening. Tks, Eric On 04/01/2010 07:06 PM, Norman Maurer wrote: Yeah thats right.. just checout server/trunk. I'm looking forward to hear your feedback ;) Bye, Norman 2010/4/1 Eric Charleseric.char...@u-mangate.com: Hello Norman, Done :) Tks, Eric PS1 : I didn't change the config, except for the JDCBDomainList I am using. PS2 : I only checkout http://svn.apache.org/repos/asf/james/server/trunk and not the other projects. From what I read from the poms, snapshot version of imap-mailbox, jsieve,... are taken from the central maven repository and I don't have to rebuild those projects. Hi Eric, its me again ;) Could you please svn up to latest trunk and try again ? Thx, Norman 2010/4/1 Norman Maurernorman.mau...@googlemail.com: Hi Eric, I did a stress test in the last few hours (running in the background here) and I was not able to get a OOM yet with ?broker.useJmx=falseamp;jms.prefetchPolicy.all=1 for the jmsConnectionFactory. I sent 4 emails with 100k to it and 3000 with 5 mb. I'm using -Xmx512m and memory usage is not getting higher then 700m. Bye. Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OutOfMemory exception
nop. OutOfMemoryError is back. I will double check my server, jdk and environment config, but other java processes are running on the same machine without problem. I will also have a stress scenario on my local dev PC that runs fine and after, I will rerun it on my production server. Dump analysis and full profiling are also to do. Some stuff are difficult to simulate : See for example (occurs regulary) ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String There are no more .m64 file in /tmp Tks, Eric ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail Mail1270218298023-9820 from jewishnesses...@ecs-solar.com on 68.62.205.100 for [ric.char...@u-mangate.org, ric.char...@u-mangate.com] INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms INFO 16:25:00,099 | james.mailetcontext | Storing mail Mail1270218298023-9820-!226071 in file://var/mail/address-error/ INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail Mail1270218307697-9822 from ours...@stahelin.com on 68.62.205.100 for [ric.char...@u-mangate.be] INFO 16:25:08,723 | james.mailetcontext | Storing mail Mail1270218307697-9822-!70969 in file://var/mail/address-error/ INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 7274 INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 1141 ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 58100 INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 10032 ms, Index update took 52100 ms INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail Mail1270218355606-9824 from inexperienc...@toutgratuit.com on 213.163.116.78 for [angelo.bid...@u-mangate.net] WARN 16:36:27,919 | org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of JMS message listener invoker failed for destination 'processor.local-address-error' - trying to recover. Cause: Java heap space ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String WARN 16:44:23,231 | org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of JMS message listener invoker failed for destination 'processor.spam' - trying to recover. Cause: Java heap space ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught an exception trying to create a JMSException for java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String WARN 17:09:11,055 | org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of JMS message listener invoker failed for destination 'processor.local-address-error' - trying to recover. Cause: Java heap space ERROR 17:09:35,017 | org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retrydelay=3] could not poll endpoint: pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retrydelay=3 caused by: Java heap space java.lang.OutOfMemoryError: Java heap space WARN 17:09:55,361 | org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of JMS message listener invoker failed for destination 'processor.spam' - trying to recover. Cause: Java heap space WARN 17:09:57,557 | org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of JMS message listener invoker failed for destination 'processor.root' - trying to recover. Cause: Java heap space WARN 17:11:00,695 | org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of JMS message listener invoker failed for destination 'processor.local-address-error' - trying to
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, Please also note that we have 2 different exceptions: - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina 2.0.0-RC1 - http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png that occurs with 2.0.0-RC2-SNAPSHOT It happens on jdk 1.6 (and also 1.7) Tks, Eric On 04/05/2010 10:01 AM, Ashish wrote: Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashishpaliwalash...@gmail.com: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurernor...@apache.org wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Yes, One instance of org.apache.mina.core.session.DefaultIoSessionDataStructureFactory$DefaultWriteRequestQueue contains : - for 2.0.0-RC1 : one instance of org.apache.mina.util.CircularQueue with an array of many org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest - for 2.0.0-RC2-SNAPSHOT : many recursive instances of java.util.concurrent.ConcurrentLinkedQueue Tks, Eric On 04/05/2010 10:51 AM, Norman Maurer wrote: I think thats only because mina-trunk is not using CircularQueue anymore. It now seems to use ConcurrentLinkedQueue. Bye, Norman 2010/4/5 Eric Charleseric.char...@u-mangate.com: Hi Ashish, Please also note that we have 2 different exceptions: - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina 2.0.0-RC1 - http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png that occurs with 2.0.0-RC2-SNAPSHOT It happens on jdk 1.6 (and also 1.7) Tks, Eric On 04/05/2010 10:01 AM, Ashish wrote: Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer norman.mau...@googlemail.comwrote: Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashishpaliwalash...@gmail.com: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurernor...@apache.org wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OutOfMemory exception
Hi Norman, james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5 day in prod. jmap -dump:file=... (this morning and right now) shows org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for the rest. However, I had to redefine all accounts as IMAP in thunderbird to be able to read mails: - the pop3 was working fine, and at a time-being, it hanged. - Since there, all pop3 requests hang (thunderbird gives me a timeout after 2 minutes), simply saying downloading message 1 of ... (nothing in the log files) - wireshark shows a correct conversation at the beginning, and afterwards many Malformed Packets. Maybe I could restart james, but I prefer let him work to have a better view on the OOM. Tks, Eric On 04/02/2010 09:04 PM, Norman Maurer wrote: Hi Eric, I forgot to mention that one cause of the OOM could be the current usage of InMemoryMail. At the moment James load the whole MimeMessage into memory while processing the camel routes. This is really a bad practice in terms of performance and could maybe cause a OOM if you processing many big messages which are bigger (the sumary) as the avaible memory. Could this be the problem on your side ? I'm currently workin on storing the MimeMessage on an external storage and just stream in on demand.. So only the meta-data is stored in jms and the rest in a db / jcr etc.. I have a proof-of-concept workin here. But I need a bit more time to see if its the way to go.. I'm also evolating the use of BlobMessages with ActiveMQ. Bye, Norman 2010/4/2 Norman Maurernor...@apache.org: Hi Eric so at least the files not leak anymore, unfortunaly this was the less critical problem. About the errors in pop3server I will have a look and see what could cause this. I'm 100 % sure it's not related to the OOM. Any more infos related to the OOM are welcome. Bye Norman 2010/4/2, Eric Charleseric.char...@u-mangate.com: nop. OutOfMemoryError is back. I will double check my server, jdk and environment config, but other java processes are running on the same machine without problem. I will also have a stress scenario on my local dev PC that runs fine and after, I will rerun it on my production server. Dump analysis and full profiling are also to do. Some stuff are difficult to simulate : See for example (occurs regulary) ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String There are no more .m64 file in /tmp Tks, Eric ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail Mail1270218298023-9820 from jewishnesses...@ecs-solar.com on 68.62.205.100 for [ric.char...@u-mangate.org, ric.char...@u-mangate.com] INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms INFO 16:25:00,099 | james.mailetcontext | Storing mail Mail1270218298023-9820-!226071 in file://var/mail/address-error/ INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail Mail1270218307697-9822 from ours...@stahelin.com on 68.62.205.100 for [ric.char...@u-mangate.be] INFO 16:25:08,723 | james.mailetcontext | Storing mail Mail1270218307697-9822-!70969 in file://var/mail/address-error/ INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 7274 INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 1141 ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 58100 INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 10032 ms, Index update took 52100 ms INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail Mail1270218355606-9824 from inexperienc...@toutgratuit.com on 213.163.116.78 for [angelo.bid...@u-mangate.net] WARN 16:36:27,919 | org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of JMS message listener invoker failed for destination 'processor.local-address-error' - trying to recover. Cause: Java heap space ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String WARN 16:44:23,231 | org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of JMS message listener invoker failed for destination 'processor.spam' - trying to recover. Cause: Java heap space ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an valid
Re: OutOfMemory exception
One more information. Actions taken via IMAP are net reflected on POP3 : deleting a mail via IMAP, you will still have to download it via POP3. I don't know if this can help, neither if the goal is to have the same behaviour via IMAP and POP3. Tks, Eric On 04/06/2010 05:41 PM, Eric Charles wrote: Hi Norman, james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5 day in prod. jmap -dump:file=... (this morning and right now) shows org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for the rest. However, I had to redefine all accounts as IMAP in thunderbird to be able to read mails: - the pop3 was working fine, and at a time-being, it hanged. - Since there, all pop3 requests hang (thunderbird gives me a timeout after 2 minutes), simply saying downloading message 1 of ... (nothing in the log files) - wireshark shows a correct conversation at the beginning, and afterwards many Malformed Packets. Maybe I could restart james, but I prefer let him work to have a better view on the OOM. Tks, Eric On 04/02/2010 09:04 PM, Norman Maurer wrote: Hi Eric, I forgot to mention that one cause of the OOM could be the current usage of InMemoryMail. At the moment James load the whole MimeMessage into memory while processing the camel routes. This is really a bad practice in terms of performance and could maybe cause a OOM if you processing many big messages which are bigger (the sumary) as the avaible memory. Could this be the problem on your side ? I'm currently workin on storing the MimeMessage on an external storage and just stream in on demand.. So only the meta-data is stored in jms and the rest in a db / jcr etc.. I have a proof-of-concept workin here. But I need a bit more time to see if its the way to go.. I'm also evolating the use of BlobMessages with ActiveMQ. Bye, Norman 2010/4/2 Norman Maurernor...@apache.org: Hi Eric so at least the files not leak anymore, unfortunaly this was the less critical problem. About the errors in pop3server I will have a look and see what could cause this. I'm 100 % sure it's not related to the OOM. Any more infos related to the OOM are welcome. Bye Norman 2010/4/2, Eric Charleseric.char...@u-mangate.com: nop. OutOfMemoryError is back. I will double check my server, jdk and environment config, but other java processes are running on the same machine without problem. I will also have a stress scenario on my local dev PC that runs fine and after, I will rerun it on my production server. Dump analysis and full profiling are also to do. Some stuff are difficult to simulate : See for example (occurs regulary) ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String There are no more .m64 file in /tmp Tks, Eric ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail Mail1270218298023-9820 from jewishnesses...@ecs-solar.com on 68.62.205.100 for [ric.char...@u-mangate.org, ric.char...@u-mangate.com] INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms INFO 16:25:00,099 | james.mailetcontext | Storing mail Mail1270218298023-9820-!226071 in file://var/mail/address-error/ INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail Mail1270218307697-9822 from ours...@stahelin.com on 68.62.205.100 for [ric.char...@u-mangate.be] INFO 16:25:08,723 | james.mailetcontext | Storing mail Mail1270218307697-9822-!70969 in file://var/mail/address-error/ INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 7274 INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 1141 ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 58100 INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 10032 ms, Index update took 52100 ms INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail Mail1270218355606-9824 from inexperienc...@toutgratuit.com on 213.163.116.78 for [angelo.bid...@u-mangate.net] WARN 16:36:27,919 | org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of JMS message listener invoker failed for destination 'processor.local-address-error' - trying to recover. Cause: Java heap space ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an valid
Re: OutOfMemory exception
Hi Norman, Strange, POP3 is now working correctly. Regarding expunge, It is disabled by default on thunderbird. If I enable it, it seems to remove all mails from the folder, even if I don't delete them. Remembering the qmail/courrier/... server I was running a few years ago, I had a transparent behaviour between pop/imap. But I have to take more time to make some tests. Tks, Eric On 04/06/2010 07:09 PM, Norman Maurer wrote: Hi Eric, I could only of one change which could maybe improve things and so you don't see the OOM anymore. The patch introduced in revision r930727. This patch changed the behavoir of pop3servers RETR and TOP handler to not load the email in the memory while return it to the client. Do you know which revision you used before ? Some more info would be interesting. Could you tell me after which POP3 command the Mailformed Packets appeared ? Thx, Norman 2010/4/6 Eric Charleseric.char...@u-mangate.com: Hi Norman, james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5 day in prod. jmap -dump:file=... (this morning and right now) shows org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for the rest. However, I had to redefine all accounts as IMAP in thunderbird to be able to read mails: - the pop3 was working fine, and at a time-being, it hanged. - Since there, all pop3 requests hang (thunderbird gives me a timeout after 2 minutes), simply saying downloading message 1 of ... (nothing in the log files) - wireshark shows a correct conversation at the beginning, and afterwards many Malformed Packets. Maybe I could restart james, but I prefer let him work to have a better view on the OOM. Tks, Eric On 04/02/2010 09:04 PM, Norman Maurer wrote: Hi Eric, I forgot to mention that one cause of the OOM could be the current usage of InMemoryMail. At the moment James load the whole MimeMessage into memory while processing the camel routes. This is really a bad practice in terms of performance and could maybe cause a OOM if you processing many big messages which are bigger (the sumary) as the avaible memory. Could this be the problem on your side ? I'm currently workin on storing the MimeMessage on an external storage and just stream in on demand.. So only the meta-data is stored in jms and the rest in a db / jcr etc.. I have a proof-of-concept workin here. But I need a bit more time to see if its the way to go.. I'm also evolating the use of BlobMessages with ActiveMQ. Bye, Norman 2010/4/2 Norman Maurernor...@apache.org: Hi Eric so at least the files not leak anymore, unfortunaly this was the less critical problem. About the errors in pop3server I will have a look and see what could cause this. I'm 100 % sure it's not related to the OOM. Any more infos related to the OOM are welcome. Bye Norman 2010/4/2, Eric Charleseric.char...@u-mangate.com: nop. OutOfMemoryError is back. I will double check my server, jdk and environment config, but other java processes are running on the same machine without problem. I will also have a stress scenario on my local dev PC that runs fine and after, I will rerun it on my production server. Dump analysis and full profiling are also to do. Some stuff are difficult to simulate : See for example (occurs regulary) ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String There are no more .m64 file in /tmp Tks, Eric ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail Mail1270218298023-9820 from jewishnesses...@ecs-solar.com on 68.62.205.100 for [ric.char...@u-mangate.org, ric.char...@u-mangate.com] INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms INFO 16:25:00,099 | james.mailetcontext | Storing mail Mail1270218298023-9820-!226071 in file://var/mail/address-error/ INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail Mail1270218307697-9822 from ours...@stahelin.com on 68.62.205.100 for [ric.char...@u-mangate.be] INFO 16:25:08,723 | james.mailetcontext | Storing mail Mail1270218307697-9822-!70969 in file://var/mail/address-error/ INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 7274 INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup took 1141 ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an valid Object but class java.lang.String INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase | Slow KahaDB access: cleanup
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, You can find some information to download on http://apache.u-mangate.com/james/oom/index.html Tks, Eric On 04/08/2010 10:26 AM, Ashish wrote: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashishpaliwalash...@gmail.com wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, I am the user who has many OOM with the trunk and feed Norman with my issues :) 1. Indeed, you have the dump with CircularQueue from mina 2.0.0-RC1. This is the release we normally use in james and give the issue after a certain period of time. I mean, the server can stay working correctly 2 days or crash after 2 hours (see http://apache.u-mangate.com/james/oom/oom.png) due to a peak in memory usage. T To further investigate, we tried with mina trunk 2.0.0-RC2-SNAPSHOT : There, we see a direct crash. I've uploaded such a dump on http://apache.u-mangate.com/james/oom/index.html (click on shift-f5 to force refresh, I setup the web server in a hurry). You will see there recursive ConcurrentLinkedQueue. 2. As shown on http://apache.u-mangate.com/james/oom/oom.png, the system works while the line is flat (mail are received, spooled, delivered,...). On a few seconds, it peaks and of course, nothing works anymore. So, the messages don't remain in the spool. I made tests with quite huge messages, and they are delivered very fast as soon as they arrive in James. Don't hesitate to ask more questions or propose additional tests. Many tks in advance, Eric On 04/09/2010 12:40 PM, Ashish wrote: Norman, Couple of more queries 1. The heap dump uses circularqueue class, so seem to be taken for an earlier trunk snapshot. Is my take correct? 2. What's the state of the System? are the clients receiving the messages. The queue seems to be holding a very large number of objects. Essentially what I want to know is, if the clients are receiving the messages or the Server is holding them up. Will spend more time with the issue and see what I can figure out. thanks ashish On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer norman.mau...@googlemail.com wrote: Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashishpaliwalash...@gmail.com: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashishpaliwalash...@gmail.com wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Replace MINA by Netty
I can confirm netty implementation is running fine after 1 week with real prod traffic and prod users. Tks, Eric On 04/19/2010 01:22 PM, Manuel Carrasco Moñino wrote: +1, Netty being a popular OpenSource project, If it performs better, it is enough for me. -Manolo On Sun, Apr 18, 2010 at 1:48 PM, Norman Maurernor...@apache.org wrote: Hi all, due the last weeks we encountered some problems realted to memory usage etc. To see if the problem is within MINA (or our usage of MINA), I wrote a socket implementation using NETTY (http://www.jboss.org/netty). Since the release of NETTY 3.x its license under the ASL2. The Framework is written by Trustin Lee which was one of the main authors of MINA, before he dedicited to leave the project. Now it seems like MINA it self is not the problem with the OOM. It seems it was caused by some infinity loop which was caused by some bad usage ( but I still think it should not happen if MINA would be a bit smarter about this...). Anyway after more tests it seems like NETTY just performs better in terms of CPU and Memory. I would even say its better maintained.. I know all the eat you own dogfood voices, but I would like to just remove all the MINA code and only use NETTY. Thoughts ? Bye, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Re: OutOfMemory exception
Hi Norman, It is now working as I would expect: deleted mails via imap are no more dowloaded via pop3, after thunderbird exit of course doing the expunge. Just for the record, I am pretty sure the behaviour was not correct with the versions that were producin oom. Probably nothing to do with each others; your commits seem to have solved much issues. Tks, Eric On 01/-10/-28163 08:59 PM, Norman Maurer wrote: Hmm.. are you sure thunderbird is calling expunge ? Thx, Norman 2010/4/6 Eric Charleseric.char...@u-mangate.com: One more information. Actions taken via IMAP are net reflected on POP3 : deleting a mail via IMAP, you will still have to download it via POP3. I don't know if this can help, neither if the goal is to have the same behaviour via IMAP and POP3. Tks, Eric On 04/06/2010 05:41 PM, Eric Charles wrote: Hi Norman, james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5 day in prod. jmap -dump:file=... (this morning and right now) shows org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for the rest. However, I had to redefine all accounts as IMAP in thunderbird to be able to read mails: - the pop3 was working fine, and at a time-being, it hanged. - Since there, all pop3 requests hang (thunderbird gives me a timeout after 2 minutes), simply saying downloading message 1 of ... (nothing in the log files) - wireshark shows a correct conversation at the beginning, and afterwards many Malformed Packets. Maybe I could restart james, but I prefer let him work to have a better view on the OOM. Tks, Eric - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi, I finally made some tests with mina-rc2 trying to reproduce the oom. I didn't succeed to reproduce it when stressing james with normal mails (well-formed,...). I would tend to follow the pointer Norman gave about the infinite loop in the exceptionCaught() method (writing on the session recause and exception,...) http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/mina/SMTPIoHandler.java?revision=930014view=markup I think that some sockets behaving badly may be the cause and tried to simulate this with long-time opened connection, with garbage being sent on the sockets,... without success (no exception). I will now roolback to the previous version of SMTPIoHandler with james not closing the mina session, and see what I can find. If you've got some idea of patterns, datas,... to send on the socket to make mina throw an exception, just post it. Tks, Eric On 04/13/2010 07:23 PM, Eric Charles wrote: Hi Ashish, James in running with netty since 2 days. I plan to let it run until this Friday (if no problem). Thereafter, I will go back to mina. The logs I've got are not helpfull and show nothing, except a sudden java OOM. I may try this weekend to find a reproducible way to have the oom. Tks, Eric On 04/13/2010 10:47 AM, Norman Maurer wrote: We need to let Eric test it again. Currently he run the netty version of james to see if the cause is really MINA .. Thx, Norman 2010/4/13 Ashishpaliwalash...@gmail.com: Is the OOM fixed after the change? thanks ashish On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I bet its the loop I mention before.. This would make sense when looking at the growing queued writes. I commited a change to trunk which close the session after write the data to the client. We will see if this helps.. Thx, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Integration with Jboss
Hi Amit, I found http://wiki.apache.org/james/Embedded on the wiki, but it goes on phoenix container which is not used anymore. I integrated james 2.3 in tomcat 2 years ago with a ServletContextListener responsible to launch the main class of the spring intregration (now the org.apache.james.container.spring.Main class). It was quite straighforward (put jars found in the james lib directory in WEB-INF/lib). The only difficulty I had was the way james was loading the file ressources from its conf directory: sometimes like a disk file, sometimes like a classpath resource. That obliged me to integrate some files in the jars I had in WEB-INF/lib (the classpath loading), and other files in /conf directory (the file loading). I also had to care about the directoy from where the process was launched (if launched from /etc/init.d/james on linux, the conf directory had to be in the /). I did the integration to have access on the user repository, and also to allow some mailets access some ressources provided by the local tomcat container. I think to remember having read about apache geronimo integrating james, but can't find back the links... Maybe you could tell use a bit more about your goals: do you want jboss serve smtp/pop/imap protocol and control the traffic by application deployed in jboss, are you willing to have a kind of admin-console,... Tks, Eric On 04/20/2010 09:26 AM, Norman Maurer wrote: Hi amit, as we use Spring in current development Version for DI and maven2 for Building it should not be a big Task to create an ear or war file for James. Anyway noone did it yet. Maybe you want to help out here ? Bye Norman 2010/4/20, Amit Vermaamit.ve...@agnity.com: Hi I want to integrate James with Jboss Server version 5.0 . Please let me know what I all I need to change in my Jboss server or james for their integration. Amit - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: programatically create/delete accounts --James 3.x
Hi, 2) Just inject the UsersRepository into your app. I suppose this will work only if you develop in the james' ApplicationContext, meaning you have to integrate the whole james spring-beans.xml into you application. Or maybe you can only use the spring-beans.xml part related to UsersRepository without any other resources (no xml file from conf directory,...) ? Tks, Eric On 04/19/2010 04:32 PM, Norman Maurer wrote: Hi there, there are some options here.. 1) use JMX todo this 2) Just inject the UsersRepository into your app 3) Using commons-net to connect via telnet About a web-imap client... There is HUPA which is a JAMES subproject. But its not 100% complete. Anyway feedback is welcome: http://james.apache.org/hupa An other very good webmail client is roundcube webmail: http://roundcube.net/ Hope this helps, Norman 2010/4/18 Any Joedaffodil_w...@yahoo.com: Gurus! I did search for this topic, but not sure. I need to create accounts on the fly rather than telent ing Is there any solution/code out there that 'can reuse. BTW, what is suggested web-imap client for James 3.x? Thanks a million - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [jira] Created: (IMAP-133) JPAStreamingMessage vs JPAMessage
Hi Tim-Christian, I stopped james 2 days ago for small config change (btw hot-config would be great :)) and I received exactly the same exception (I already saw it once or twice). The exception was shown only when fetching mails on one specific account. I deployed a fresh version of james trunk, and by magic, the error had disappeared . I will report if I still encounter the exception. Tks, Eric On 04/23/2010 04:33 PM, Norman Maurer wrote: As a quick workaround , you should just remove the *Streaming* stuff from the persistence.xml file. It seems like the classloader gets confused otherwise sometimes... Thx for Reporting, Norman 2010/4/23, Tim-Christian Mundt (JIRA)server-dev@james.apache.org: JPAStreamingMessage vs JPAMessage - Key: IMAP-133 URL: https://issues.apache.org/jira/browse/IMAP-133 Project: JAMES Imap Issue Type: Bug Components: JPA Mailbox Reporter: Tim-Christian Mundt Priority: Blocker With newest James I quite often (as always not always) get this error during copy operations which causes them to fail: Exception in thread pool-8-thread-1 java.lang.ClassCastException: org.apache.james.imap.jpa.mail.model.openjpa.JPAStreamingMessage cannot be cast to org.apache.james.imap.jpa.mail.model.JPAMessage at org.apache.james.imap.jpa.mail.model.JPAMailboxMembership.init(JPAMailboxMembership.java:64) at org.apache.james.imap.jpa.JPAMailbox.copyMessage(JPAMailbox.java:99) at org.apache.james.imap.jpa.openjpa.OpenJPAMailbox.copyMessage(OpenJPAMailbox.java:83) at org.apache.james.imap.store.StoreMailbox$5.run(StoreMailbox.java:611) at org.apache.james.imap.store.transaction.AbstractTransactionalMapper.execute(AbstractTransactionalMapper.java:38) at org.apache.james.imap.store.StoreMailbox.copy(StoreMailbox.java:603) at org.apache.james.imap.store.StoreMailbox.copyTo(StoreMailbox.java:637) at org.apache.james.imap.store.StoreMailboxManager.copyMessages(StoreMailboxManager.java:296) at org.apache.james.imap.processor.CopyProcessor.doProcess(CopyProcessor.java:82) at org.apache.james.imap.processor.AbstractMailboxProcessor.doProcess(AbstractMailboxProcessor.java:125) at org.apache.james.imap.processor.AbstractMailboxProcessor.process(AbstractMailboxProcessor.java:80) at org.apache.james.imap.processor.AbstractMailboxProcessor.doProcess(AbstractMailboxProcessor.java:73) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:44) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.main.AbstractImapRequestHandler.doProcessRequest(AbstractImapRequestHandler.java:65) at org.apache.james.imap.main.ImapRequestStreamHandler.handleRequest(ImapRequestStreamHandler.java:82) at org.apache.james.imapserver.netty.ImapStreamChannelUpstreamHandler.processStreamIo(ImapStreamChannelUpstreamHandler.java:64) at org.jboss.netty.handler.stream.StreamHandler$1.run(StreamHandler.java:127) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) streaming is set to
OOM with Netty
Hi All, I just got oom with netty impl. See http://apache.u-mangate.com/james/oom/20090424.png (http://apache.u-mangate.com/james/oom/index.html for history). Everything was fine with the trunk of last week that was really stable. I redeployed 3 days ago a new trunk version after a stop/start that showed ClassCastException of ImapMessage (ClassCastException was solved with the redeploy) Short infos about the today oom: - The logs had IOException Bad file descriptor (see copy of the exception after this mail) - The begining time of these exceptions in the log is 00:10, and there are numerous ones during 1 minute. - The heap dump automatically generated via -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/ occured on 00:10, exactly on the same time the IOException Bad file descriptor began. - I forgot to lsof to see the open files... but when if the error came from too many open files, the received exception in the log would have been java.io.IOException: too many open files. - Seems like an infinite loop as Norman anticipated, but now stacktraces from logs seem to confirm this. - Looking at line 150 of http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup , out.write(line) seem to be the place that triggers the exception, out variable being = (OutputStream) session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM); - The server filesystem seems sane (new hardware, running with enough privilege, other applications running on the same machine without any problem). I don't see why I have a Bad file descriptor There are file in tmp with creation date that span 1 week (so not produced on 00:10): - A few messagestream*.ms - More rewindable*.tmp I restarted james. It still showed some issues (one account via pop3 gave errors in my client). oouch... I updated from trunk, redeployed, copied the var directory to migrate the account/domain/mail database, and restarted. Hopefully, now everything is working as before. So, we've got now a useful stacktrace that clearly shows where the oom trigger could be. I'm pretty sure that jconsole would have shown the same behaviour as http://apache.u-mangate.com/james/oom/oom.png Possible cause on the SMTP channel: - A weird behaviour of the smtp clients (hacking or respectfulness protocols clients) - A design in james causing infinite loop: thread clash (a thread accessing the same file as another thread?,...) - Additional James config (spamassassin or whatever) that leads to such behaviour - Hardware problem Currently, I run with default config + jdbcdomainlist. I except to receive another oom one of these days, unless the oom comes from an unactivated config (spamassassin or whatever). Tks, Eric ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while processing DATA. java.io.IOException: Bad file descriptor at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:247) at org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150) at org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) at org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) at org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54) at org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) at org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) at org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151) at org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) at org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) at org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
Re: OOM with Netty
Hi Norman, You can download it from http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip This error seems to get thrown when you write to a closed stream.. Could it be a thread-safety problem ? (on thread using the session/state of another one ?) I will also monitor with lsof -p pid | grep deleted | wc -l As config change, I activated a few days ago the ValidRcptHandler fastfail handler. Tks, Eric On 04/24/2010 03:13 PM, Norman Maurer wrote: Could you upload the heap dump too ? 2010/4/24, Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. Comments are inline... 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi All, I just got oom with netty impl. See http://apache.u-mangate.com/james/oom/20090424.png (http://apache.u-mangate.com/james/oom/index.html for history). Everything was fine with the trunk of last week that was really stable. I redeployed 3 days ago a new trunk version after a stop/start that showed ClassCastException of ImapMessage (ClassCastException was solved with the redeploy) The ClassCastException should be fixed since yesterday. This was caused by the EntityManager got confused by entities.. Short infos about the today oom: - The logs had IOException Bad file descriptor (see copy of the exception after this mail) This error seems to get thrown when you write to a closed stream... - The begining time of these exceptions in the log is 00:10, and there are numerous ones during 1 minute. - The heap dump automatically generated via -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/ occured on 00:10, exactly on the same time the IOException Bad file descriptor began. Ok.. - I forgot to lsof to see the open files... but when if the error came from too many open files, the received exception in the log would have been java.io.IOException: too many open files. right.. - Seems like an infinite loop as Norman anticipated, but now stacktraces from logs seem to confirm this. - Looking at line 150 of http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup , out.write(line) seem to be the place that triggers the exception, out variable being = (OutputStream) session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM); Good catch will see how this could happen.. - The server filesystem seems sane (new hardware, running with enough privilege, other applications running on the same machine without any problem). I don't see why I have a Bad file descriptor See above.. There are file in tmp with creation date that span 1 week (so not produced on 00:10): - A few messagestream*.ms - More rewindable*.tmp Will investigate .. I restarted james. It still showed some issues (one account via pop3 gave errors in my client). oouch... I updated from trunk, redeployed, copied the var directory to migrate the account/domain/mail database, and restarted. Hopefully, now everything is working as before. So, we've got now a useful stacktrace that clearly shows where the oom trigger could be. I'm pretty sure that jconsole would have shown the same behaviour as http://apache.u-mangate.com/james/oom/oom.png Possible cause on the SMTP channel: - A weird behaviour of the smtp clients (hacking or respectfulness protocols clients) - A design in james causing infinite loop: thread clash (a thread accessing the same file as another thread?,...) - Additional James config (spamassassin or whatever) that leads to such behaviour - Hardware problem Currently, I run with default config + jdbcdomainlist. I except to receive another oom one of these days, unless the oom comes from an unactivated config (spamassassin or whatever). Tks, Eric ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while processing DATA. java.io.IOException: Bad file descriptor at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:247) at org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150) at org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) at org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) at org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54) at org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) at org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) at org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151) at
Re: OOM with Netty
Hi Norman, lsof -p pid | grep deleted | wc -l gives a stable number (around 20, growing and after decreasing : cleanup of gc or os... whatever...). No file in /tmp. I wonder if the files I found were files left further to the few hard-stop I made during last week. Currently, to stop James, I use kill -9 pid Is there another cleaner way to do this ? I saw shutdown command (kills the current JVM (convenient when James is run as a daemon)) in the remotemanager. Is it different from kill -9 ? Should I use this ? Tks, Eric On 04/24/2010 04:23 PM, Norman Maurer wrote: Hi Eric, just downloading it.. About the tread safety. I don't think so. I'm still wonder if the exception was only thrown because of the OOM or if its the cause of the OOM. Hopefully I will have a better view of the whole story once I looked at the heap dump... Bye, Norman 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi Norman, You can download it from http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip This error seems to get thrown when you write to a closed stream.. Could it be a thread-safety problem ? (on thread using the session/state of another one ?) I will also monitor with lsof -ppid | grep deleted | wc -l As config change, I activated a few days ago the ValidRcptHandler fastfail handler. Tks, Eric On 04/24/2010 03:13 PM, Norman Maurer wrote: Could you upload the heap dump too ? 2010/4/24, Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. Comments are inline... 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi All, I just got oom with netty impl. See http://apache.u-mangate.com/james/oom/20090424.png (http://apache.u-mangate.com/james/oom/index.html for history). Everything was fine with the trunk of last week that was really stable. I redeployed 3 days ago a new trunk version after a stop/start that showed ClassCastException of ImapMessage (ClassCastException was solved with the redeploy) The ClassCastException should be fixed since yesterday. This was caused by the EntityManager got confused by entities.. Short infos about the today oom: - The logs had IOException Bad file descriptor (see copy of the exception after this mail) This error seems to get thrown when you write to a closed stream... - The begining time of these exceptions in the log is 00:10, and there are numerous ones during 1 minute. - The heap dump automatically generated via -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath= occured on 00:10, exactly on the same time the IOException Bad file descriptor began. Ok.. - I forgot to lsof to see the open files... but when if the error came from too many open files, the received exception in the log would have been java.io.IOException: too many open files. right.. - Seems like an infinite loop as Norman anticipated, but now stacktraces from logs seem to confirm this. - Looking at line 150 of http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=rkup , out.write(line) seem to be the place that triggers the exception, out variable being =OutputStream) session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM); Good catch will see how this could happen.. - The server filesystem seems sane (new hardware, running with enough privilege, other applications running on the same machine without any problem). I don't see why I have a Bad file descriptor See above.. There are file in tmp with creation date that span 1 week (so not produced on 00:10): - A few messagestream*.ms - More rewindable*.tmp Will investigate .. I restarted james. It still showed some issues (one account via pop3 gave errors in my client). oouch... I updated from trunk, redeployed, copied the var directory to migrate the account/domain/mail database, and restarted. Hopefully, now everything is working as before. So, we've got now a useful stacktrace that clearly shows where the oom trigger could be. I'm pretty sure that jconsole would have shown the same behaviour as http://apache.u-mangate.com/james/oom/oom.png Possible cause on the SMTP channel: - A weird behaviour of the smtp clients (hacking or respectfulness protocols clients) - A design in james causing infinite loop: thread clash (a thread accessing the same file as another thread?,...) - Additional James config (spamassassin or whatever) that leads to such behaviour - Hardware problem Currently, I run with default config + jdbcdomainlist. I except to receive another oom one of these days, unless the oom comes from an unactivated config (spamassassin or whatever). Tks, Eric ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while processing DATA. java.io.IOException: Bad file descriptor at
Re: OOM with Netty
Hi Norman, I launch with (background process) so I can leave the terminal. Would shutdown via remotemanager better ? Tks, Eric On 04/24/2010 06:19 PM, Norman Maurer wrote: Hi Eric, doesn't CTRL-C work in the terminal you started james in ? And you should not use kill-9pid because it will not let JAMES cleanup stuff, if you want to use killpid .. Bye, Norman 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi Norman, lsof -ppid | grep deleted | wc -l gives a stable number (around 20, growing and after decreasing : cleanup of gc or os... whatever...). No file in /tmp. I wonder if the files I found were files left further to the few hard-stop I made during last week. Currently, to stop James, I use kill -9pid Is there another cleaner way to do this ? I saw shutdown command (kills the current JVM (convenient when James is run as a daemon)) in the remotemanager. Is it different from kill -9 ? Should I use this ? Tks, Eric On 04/24/2010 04:23 PM, Norman Maurer wrote: Hi Eric, just downloading it.. About the tread safety. I don't think so. I'm still wonder if the exception was only thrown because of the OOM or if its the cause of the OOM. Hopefully I will have a better view of the whole story once I looked at the heap dump... Bye, Norman 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi Norman, You can download it from http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip This error seems to get thrown when you write to a closed stream.. Could it be a thread-safety problem ? (on thread using the session/state of another one ?) I will also monitor with lsof -ppid| grep deleted | wc -l As config change, I activated a few days ago the ValidRcptHandler fastfail handler. Tks, Eric On 04/24/2010 03:13 PM, Norman Maurer wrote: Could you upload the heap dump too ? 2010/4/24, Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. Comments are inline... 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi All, I just got oom with netty impl. See http://apache.u-mangate.com/james/oom/20090424.png (http://apache.u-mangate.com/james/oom/index.html for history). Everything was fine with the trunk of last week that was really stable. I redeployed 3 days ago a new trunk version after a stop/start that showed ClassCastException of ImapMessage (ClassCastException was solved with the redeploy) The ClassCastException should be fixed since yesterday. This was caused by the EntityManager got confused by entities.. Short infos about the today oom: - The logs had IOException Bad file descriptor (see copy of the exception after this mail) This error seems to get thrown when you write to a closed stream... - The begining time of these exceptions in the log is 00:10, and there are numerous ones during 1 minute. - The heap dump automatically generated via -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=ccured on 00:10, exactly on the same time the IOException Bad file descriptor began. Ok.. - I forgot to lsof to see the open files... but when if the error came from too many open files, the received exception in the log would have been java.io.IOException: too many open files. right.. - Seems like an infinite loop as Norman anticipated, but now stacktraces from logs seem to confirm this. - Looking at line 150 of http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=up , out.write(line) seem to be the place that triggers the exception, out variable being =tputStream) session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM); Good catch will see how this could happen.. - The server filesystem seems sane (new hardware, running with enough privilege, other applications running on the same machine without any problem). I don't see why I have a Bad file descriptor See above.. There are file in tmp with creation date that span 1 week (so not produced on 00:10): - A few messagestream*.ms - More rewindable*.tmp Will investigate .. I restarted james. It still showed some issues (one account via pop3 gave errors in my client). oouch... I updated from trunk, redeployed, copied the var directory to migrate the account/domain/mail database, and restarted. Hopefully, now everything is working as before. So, we've got now a useful stacktrace that clearly shows where the oom trigger could be. I'm pretty sure that jconsole would have shown the same behaviour as http://apache.u-mangate.com/james/oom/oom.png Possible cause on the SMTP channel: - A weird behaviour of the smtp clients (hacking or respectfulness protocols clients) - A design in james causing
Clean shutdown (was: OOM with Netty)
mmh, as requirement to start/stop james, I would like to read java 1.? in the doc :) Would you agree I open a jira for this (rename shutdown to kill and create a clean shutdown). I could also try to contribute a patch for this. Tks, Eric On 04/24/2010 06:34 PM, Norman Maurer wrote: Hi Eric, you should use screen (http://www.gnu.org/software/screen/) for launching james ;) RemotManager would be the same as killpid Bye, Norman 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi Norman, I launch with (background process) so I can leave the terminal. Would shutdown via remotemanager better ? Tks, Eric On 04/24/2010 06:19 PM, Norman Maurer wrote: Hi Eric, doesn't CTRL-C work in the terminal you started james in ? And you should not use kill-9pidbecause it will not let JAMES cleanup stuff, if you want to use killpid.. Bye, Norman 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi Norman, lsof -ppid| grep deleted | wc -l gives a stable number (around 20, growing and after decreasing : cleanup of gc or os... whatever...). No file in /tmp. I wonder if the files I found were files left further to the few hard-stop I made during last week. Currently, to stop James, I use kill -9pid Is there another cleaner way to do this ? I saw shutdown command (kills the current JVM (convenient when James is run as a daemon)) in the remotemanager. Is it different from kill -9 ? Should I use this ? Tks, Eric On 04/24/2010 04:23 PM, Norman Maurer wrote: Hi Eric, just downloading it.. About the tread safety. I don't think so. I'm still wonder if the exception was only thrown because of the OOM or if its the cause of the OOM. Hopefully I will have a better view of the whole story once I looked at the heap dump... Bye, Norman 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi Norman, You can download it from http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip This error seems to get thrown when you write to a closed stream.. Could it be a thread-safety problem ? (on thread using the session/state of another one ?) I will also monitor with lsof -ppid | grep deleted | wc -l As config change, I activated a few days ago the ValidRcptHandler fastfail handler. Tks, Eric On 04/24/2010 03:13 PM, Norman Maurer wrote: Could you upload the heap dump too ? 2010/4/24, Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. Comments are inline... 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi All, I just got oom with netty impl. See http://apache.u-mangate.com/james/oom/20090424.png (http://apache.u-mangate.com/james/oom/index.html for history). Everything was fine with the trunk of last week that was really stable. I redeployed 3 days ago a new trunk version after a stop/start that showed ClassCastException of ImapMessage (ClassCastException was solved with the redeploy) The ClassCastException should be fixed since yesterday. This was caused by the EntityManager got confused by entities.. Short infos about the today oom: - The logs had IOException Bad file descriptor (see copy of the exception after this mail) This error seems to get thrown when you write to a closed stream... - The begining time of these exceptions in the log is 00:10, and there are numerous ones during 1 minute. - The heap dump automatically generated via -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPathÌured on 00:10, exactly on the same time the IOException Bad file descriptor began. Ok.. - I forgot to lsof to see the open files... but when if the error came from too many open files, the received exception in the log would have been java.io.IOException: too many open files. right.. - Seems like an infinite loop as Norman anticipated, but now stacktraces from logs seem to confirm this. - Looking at line 150 of http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view= , out.write(line) seem to be the place that triggers the exception, out variable being =utStream) session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM); Good catch will see how this could happen.. - The server filesystem seems sane (new hardware, running with enough privilege, other applications running on the same machine without any problem). I don't see why I have a Bad file descriptor See above.. There are file in tmp with creation date that span 1 week (so not produced on 00:10): - A few messagestream*.ms - More rewindable*.tmp Will investigate .. I restarted james. It
Re: Clean shutdown
Hi Norman, I will open a jira. Do you see the implementation as a a parametrizable main method in the http://svn.apache.org/viewvc/james/server/trunk/spring-deployment/src/main/java/org/apache/james/container/spring/Main.java?view=markup ? The scripts would pass start or stop parameter and depending on this parameter, the main will do its job. Right? Eric On 04/24/2010 10:13 PM, Norman Maurer wrote: Hi Eric, I think the used name of shutdown is correct. About the start / stop script you are right. It would be very nice to have this. Could you open a jira issue for it ? I think the maven appassember plugin (http://mojo.codehaus.org/appassembler/appassembler-maven-plugin/) could help with it and do all the magic for us.. Thx , Norman 2010/4/24 Eric Charleseric.char...@u-mangate.com: mmh, as requirement to start/stop james, I would like to read java 1.? in the doc :) Would you agree I open a jira for this (rename shutdown to kill and create a clean shutdown). I could also try to contribute a patch for this. Tks, Eric On 04/24/2010 06:34 PM, Norman Maurer wrote: Hi Eric, you should use screen (http://www.gnu.org/software/screen/) for launching james ;) RemotManager would be the same as killpid Bye, Norman 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi Norman, I launch with(background process) so I can leave the terminal. Would shutdown via remotemanager better ? Tks, Eric - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OOM with Netty
Hi Norman, Done. Wait and see. The good think this time is that we knew where to look at. Tks good logging. Tks for your very quick fixes. Eric PS : No file in /tmp but when I killed the process, 2 were left. So the /tmp files I reported came probably from the kill. On 04/24/2010 09:35 PM, Norman Maurer wrote: Hi Eric, please svn up your source and try again.. I found at least one possible problem which could cause a OOM. Thx, Norman 2010/4/24 Norman Maurernorman.mau...@googlemail.com: Hi Eric, just downloading it.. About the tread safety. I don't think so. I'm still wonder if the exception was only thrown because of the OOM or if its the cause of the OOM. Hopefully I will have a better view of the whole story once I looked at the heap dump... Bye, Norman 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi Norman, You can download it from http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip This error seems to get thrown when you write to a closed stream.. Could it be a thread-safety problem ? (on thread using the session/state of another one ?) I will also monitor with lsof -ppid | grep deleted | wc -l As config change, I activated a few days ago the ValidRcptHandler fastfail handler. Tks, Eric On 04/24/2010 03:13 PM, Norman Maurer wrote: Could you upload the heap dump too ? 2010/4/24, Norman Maurernorman.mau...@googlemail.com: Hi Eric, thx for the report. Comments are inline... 2010/4/24 Eric Charleseric.char...@u-mangate.com: Hi All, I just got oom with netty impl. See http://apache.u-mangate.com/james/oom/20090424.png (http://apache.u-mangate.com/james/oom/index.html for history). Everything was fine with the trunk of last week that was really stable. I redeployed 3 days ago a new trunk version after a stop/start that showed ClassCastException of ImapMessage (ClassCastException was solved with the redeploy) The ClassCastException should be fixed since yesterday. This was caused by the EntityManager got confused by entities.. Short infos about the today oom: - The logs had IOException Bad file descriptor (see copy of the exception after this mail) This error seems to get thrown when you write to a closed stream... - The begining time of these exceptions in the log is 00:10, and there are numerous ones during 1 minute. - The heap dump automatically generated via -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath= occured on 00:10, exactly on the same time the IOException Bad file descriptor began. Ok.. - I forgot to lsof to see the open files... but when if the error came from too many open files, the received exception in the log would have been java.io.IOException: too many open files. right.. - Seems like an infinite loop as Norman anticipated, but now stacktraces from logs seem to confirm this. - Looking at line 150 of http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=rkup , out.write(line) seem to be the place that triggers the exception, out variable being =OutputStream) session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM); Good catch will see how this could happen.. - The server filesystem seems sane (new hardware, running with enough privilege, other applications running on the same machine without any problem). I don't see why I have a Bad file descriptor See above.. There are file in tmp with creation date that span 1 week (so not produced on 00:10): - A few messagestream*.ms - More rewindable*.tmp Will investigate .. I restarted james. It still showed some issues (one account via pop3 gave errors in my client). oouch... I updated from trunk, redeployed, copied the var directory to migrate the account/domain/mail database, and restarted. Hopefully, now everything is working as before. So, we've got now a useful stacktrace that clearly shows where the oom trigger could be. I'm pretty sure that jconsole would have shown the same behaviour as http://apache.u-mangate.com/james/oom/oom.png Possible cause on the SMTP channel: - A weird behaviour of the smtp clients (hacking or respectfulness protocols clients) - A design in james causing infinite loop: thread clash (a thread accessing the same file as another thread?,...) - Additional James config (spamassassin or whatever) that leads to such behaviour - Hardware problem Currently, I run with default config + jdbcdomainlist. I except to receive another oom one of these days, unless the oom comes from an unactivated config (spamassassin or whatever). Tks, Eric ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while processing DATA. java.io.IOException: Bad file descriptor at
Re: [jira] Resolved: (JAMES-993) start/stop scripts creation during build
I like ./james stop showing the log INFO 19:04:42,071 | org.apache.james.container.spring.JamesServerApplicationContext | Closing org.apache.james.container.spring.jamesserverapplicationcont...@12dacd1: startup date [Tue Apr 27 18:30:50 CEST 2010]; root of context hierarchy INFO 19:04:42,074 | org.apache.activemq.broker.BrokerService | ActiveMQ Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) is shutting down INFO 19:04:42,102 | james.imapserver | Dispose IMAP Service INFO 19:04:42,105 | james.smtpserver | Dispose SMTP Service INFO 19:04:42,105 | james.pop3server | Dispose POP3 Service INFO 19:04:42,105 | james.remotemanager | Dispose RemoteManager Service INFO 19:04:42,106 | org.apache.xbean.spring.context.ResourceXmlApplicationContext | Closing org.apache.xbean.spring.context.resourcexmlapplicationcont...@e59e40: startup date [Tue Apr 27 18:31:05 CEST 2010]; parent: org.apache.james.container.spring.jamesserverapplicationcont...@12dacd1 INFO 19:04:42,106 | org.apache.activemq.broker.BrokerService | ActiveMQ Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) is shutting down INFO 19:04:42,107 | org.apache.activemq.broker.TransportConnector | Connector openwire Stopped INFO 19:04:42,113 | org.apache.activemq.broker.TransportConnector | Connector vm://localhost Stopped INFO 19:04:42,545 | org.apache.activemq.broker.TransportConnector | Connector openwire Stopped INFO 19:04:42,560 | org.apache.activemq.broker.BrokerService | ActiveMQ JMS Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) stopped INFO 19:04:42,604 | org.apache.activemq.broker.BrokerService | ActiveMQ JMS Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) stopped Really clean stop. Tks Norman Maven assembler plugin, Eric On 04/25/2010 09:31 PM, Norman Maurer (JIRA) wrote: [ https://issues.apache.org/jira/browse/JAMES-993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norman Maurer resolved JAMES-993. - Fix Version/s: 3.0-M1 Resolution: Fixed Done :) start/stop scripts creation during build Key: JAMES-993 URL: https://issues.apache.org/jira/browse/JAMES-993 Project: JAMES Server Issue Type: New Feature Components: Build System Affects Versions: Trunk Reporter: Eric Charles Assignee: Norman Maurer Priority: Minor Fix For: 3.0-M1 Build should create start and stop scripts. See http://mojo.codehaus.org/appassembler/appassembler-maven-plugin/ - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [jira] Resolved: (JAMES-993) start/stop scripts creation during build
Works fine on full 32 bits. Building on 32-bits and deploying on 64-bits does not work: - the generated wrapper-linux-x86-32 is obviously a 32-bits version - running the wrapper-linux-x86-32 on 64-bits gives /bin/./wrapper-linux-x86-32: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory Should I consider this as normal (I suppose so) ? I will probably have to build on 64-bits. Tks, Eric On 04/27/2010 07:28 PM, Eric Charles wrote: I like ./james stop showing the log INFO 19:04:42,071 | org.apache.james.container.spring.JamesServerApplicationContext | Closing org.apache.james.container.spring.jamesserverapplicationcont...@12dacd1: startup date [Tue Apr 27 18:30:50 CEST 2010]; root of context hierarchy INFO 19:04:42,074 | org.apache.activemq.broker.BrokerService | ActiveMQ Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) is shutting down INFO 19:04:42,102 | james.imapserver | Dispose IMAP Service INFO 19:04:42,105 | james.smtpserver | Dispose SMTP Service INFO 19:04:42,105 | james.pop3server | Dispose POP3 Service INFO 19:04:42,105 | james.remotemanager | Dispose RemoteManager Service INFO 19:04:42,106 | org.apache.xbean.spring.context.ResourceXmlApplicationContext | Closing org.apache.xbean.spring.context.resourcexmlapplicationcont...@e59e40: startup date [Tue Apr 27 18:31:05 CEST 2010]; parent: org.apache.james.container.spring.jamesserverapplicationcont...@12dacd1 INFO 19:04:42,106 | org.apache.activemq.broker.BrokerService | ActiveMQ Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) is shutting down INFO 19:04:42,107 | org.apache.activemq.broker.TransportConnector | Connector openwire Stopped INFO 19:04:42,113 | org.apache.activemq.broker.TransportConnector | Connector vm://localhost Stopped INFO 19:04:42,545 | org.apache.activemq.broker.TransportConnector | Connector openwire Stopped INFO 19:04:42,560 | org.apache.activemq.broker.BrokerService | ActiveMQ JMS Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) stopped INFO 19:04:42,604 | org.apache.activemq.broker.BrokerService | ActiveMQ JMS Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) stopped Really clean stop. Tks Norman Maven assembler plugin, Eric On 04/25/2010 09:31 PM, Norman Maurer (JIRA) wrote: [ https://issues.apache.org/jira/browse/JAMES-993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norman Maurer resolved JAMES-993. - Fix Version/s: 3.0-M1 Resolution: Fixed Done :) start/stop scripts creation during build Key: JAMES-993 URL: https://issues.apache.org/jira/browse/JAMES-993 Project: JAMES Server Issue Type: New Feature Components: Build System Affects Versions: Trunk Reporter: Eric Charles Assignee: Norman Maurer Priority: Minor Fix For: 3.0-M1 Build should create start and stop scripts. See http://mojo.codehaus.org/appassembler/appassembler-maven-plugin/ - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [jira] Resolved: (JAMES-993) start/stop scripts creation during build
Yes. yum install glibc.i686 installed the needed lib on fedora and wrapper works fine. The equivalents for ubuntu,... should be easy to find. Maybe something to remind for our future 3.0.M1 documentation :) Tks Norman, Eric On 04/27/2010 07:56 PM, Norman Maurer wrote: Hmm, good question. I will have a look. BTW, you could install the libc32 bit libraries. Bye, Norman 2010/4/27 Eric Charleseric.char...@u-mangate.com: Works fine on full 32 bits. Building on 32-bits and deploying on 64-bits does not work: - the generated wrapper-linux-x86-32 is obviously a 32-bits version - running the wrapper-linux-x86-32 on 64-bits gives /bin/./wrapper-linux-x86-32: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory Should I consider this as normal (I suppose so) ? I will probably have to build on 64-bits. Tks, Eric On 04/27/2010 07:28 PM, Eric Charles wrote: I like ./james stop showing the log INFO 19:04:42,071 | org.apache.james.container.spring.JamesServerApplicationContext | Closing org.apache.james.container.spring.jamesserverapplicationcont...@12dacd1: startup date [Tue Apr 27 18:30:50 CEST 2010]; root of context hierarchy INFO 19:04:42,074 | org.apache.activemq.broker.BrokerService | ActiveMQ Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) is shutting down INFO 19:04:42,102 | james.imapserver | Dispose IMAP Service INFO 19:04:42,105 | james.smtpserver | Dispose SMTP Service INFO 19:04:42,105 | james.pop3server | Dispose POP3 Service INFO 19:04:42,105 | james.remotemanager | Dispose RemoteManager Service INFO 19:04:42,106 | org.apache.xbean.spring.context.ResourceXmlApplicationContext | Closing org.apache.xbean.spring.context.resourcexmlapplicationcont...@e59e40: startup date [Tue Apr 27 18:31:05 CEST 2010]; parent: org.apache.james.container.spring.jamesserverapplicationcont...@12dacd1 INFO 19:04:42,106 | org.apache.activemq.broker.BrokerService | ActiveMQ Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) is shutting down INFO 19:04:42,107 | org.apache.activemq.broker.TransportConnector | Connector openwire Stopped INFO 19:04:42,113 | org.apache.activemq.broker.TransportConnector | Connector vm://localhost Stopped INFO 19:04:42,545 | org.apache.activemq.broker.TransportConnector | Connector openwire Stopped INFO 19:04:42,560 | org.apache.activemq.broker.BrokerService | ActiveMQ JMS Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) stopped INFO 19:04:42,604 | org.apache.activemq.broker.BrokerService | ActiveMQ JMS Message Broker (localhost, ID:ppc005-48633-1272385866569-0:0) stopped Really clean stop. Tks Norman Maven assembler plugin, Eric On 04/25/2010 09:31 PM, Norman Maurer (JIRA) wrote: [ https://issues.apache.org/jira/browse/JAMES-993?page=m.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norman Maurer resolved JAMES-993. - Fix Version/s: 3.0-M1 Resolution: Fixed Done :) start/stop scripts creation during build Key: JAMES-993 URL: https://issues.apache.org/jira/browse/JAMES-993 Project: JAMES Server Issue Type: New Feature Components: Build System Affects Versions: Trunk Reporter: Eric Charles Assignee: Norman Maurer Priority: Minor Fix For: 3.0-M1 Build should create start and stop scripts. See http://mojo.codehaus.org/appassembler/appassembler-maven-plugin/ - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Celebration, plus: Release James 4.0M1?
non-binding +1 from a trunk user. On 04/28/2010 10:45 AM, Bernd Fondermann wrote: Hi, Thanks to Norman, Robert and everyone else - over the past month, there have been tons of changes and improvements made to James. Most of them are fantastic: modularization, IMAP, container, async io protocols. I can even deploy James as a WAR file now - This must be the future! Now, how dare we not to release this bag of beauties to our users? Why not have a milestone release adding a disclaimer that this isn't production ready and collect feedback from the field? And even if you'd call it 3.0 instead of the deserved 4.0, I'd be happy. :-) Here's my +1! Non-bindingly, Bernd - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
JPAStressTest with derby, h2 and mysql
Hi,
I just tested the JPAStressTest (for IMAP-137 JPA fails to persist
MailboxMembership Entity on heavy load) Norman committed yesterday.
I made some tests towards derby (memory and embedded) , h2 (memory and
embedded) and mysql with following configs (and adding the needed jdbc
drivers in pom.xml):
// Derby Memory
properties.put(openjpa.ConnectionDriverName,
org.apache.derby.jdbc.EmbeddedDriver.class.getName());
properties.put(openjpa.ConnectionURL,
jdbc:derby:memory:derbyimap;create=true);
// Derby Embedded
//properties.put(openjpa.ConnectionDriverName,
org.apache.derby.jdbc.EmbeddedDriver.class.getName());
//properties.put(openjpa.ConnectionURL,
jdbc:derby:derbyimap;create=true;user=root;password=root);
// H2 Memory
//properties.put(openjpa.ConnectionDriverName,
org.h2.Driver.class.getName());
//properties.put(openjpa.ConnectionURL,
jdbc:h2:mem:h2imap;DB_CLOSE_DELAY=-1);
// H2 Embedded
//properties.put(openjpa.ConnectionDriverName,
org.h2.Driver.class.getName());
//properties.put(openjpa.ConnectionURL,
jdbc:h2:~/h2/h2imap;USER=root;PASSWORD=root);
//Mysql
//properties.put(openjpa.ConnectionDriverName,
com.mysql.jdbc.Driver.class.getName());
//properties.put(openjpa.ConnectionURL,
jdbc:mysql://localhost:3306/msqlimap?user=rootpassword=rootcreateDatabaseIfNotExist=trueamp;useUnicode=trueamp;characterEncoding=utf-8);
I first changed the maximum number of messages from 1.000 to 100.000 and
saw on my linux dev PC (james server, stresstest and databases all
running on same dev PC):
Derby: after many mails sending:
java.io.IOException: Too many open files
...
Exception in thread pool-1-thread-1044 java.lang.NoClassDefFoundError:
org/apache/james/imap/api/display/HumanReadableText
...
H2: after 400 mails:
java.io.IOException: Too many open files
...
Caused by: openjpa-1.2.2-r422266:898935 nonfatal general error
org.apache.openjpa.persistence.PersistenceException: Timeout trying to
lock table MEMBERSHIP [50200-79] {prepstmnt 406576
INSERT INTO Membership (mailboxId, uid, answered, deleted, draft,
flagged, internalDate, recent, seen, size, MESSAGE_ID)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
[params=(long) 1, (long) 49, (int) 0, (int) 0, (int) 0, (int) 0,
(Timestamp) 2010-05-01 08:37:10.293, (int) 0, (int) 0, (int) 25, (long)
4151]} [code=50200, state=HYT00]
...
MySql : after +/- 1000 (always around 1002 and 1007) mails :
Exception in thread pool-1-thread-84 openjpa-1.2.2-r422266:898935
nonfatal user error org.apache.openjpa.persistence.NoResultException:
The query on candidate type class
org.apache.james.imap.jpa.mail.model.JPAMailbox with filter SELECT
mailbox FROM Mailbox mailbox WHERE mailbox.mailboxId = :idParam was
configured to have a unique result, but no instance matched the query.
org.apache.james.imap.mailbox.MailboxNotFoundException: Mailbox
'#mail.INBOX' not found.
...
3 different databases, 3 different behaviour/exceptions.
Running 3 components (james server, stresstest, database) on my
not-so-strong-PC may be an issue.
To make the tests successful, I added a Thread.sleep(5) (15 ms for
embedded derby and h2) between each mail sending.
I suppose we can say that in a real environment, the load will never be
that heavy, and that real server will perform better.
I also saw that the uid order is not respected:
Append message with uid=6258
Append message with uid=6259
Append message with uid=6256
Append message with uid=6257
This could come from a print mismatch, and not from uid being generated
in a bad order.
Tks in advance for your feedbacks, if any,
Eric
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JPAStressTest with derby, h2 and mysql
Hi Norman,
Similar exceptions with properties.put(openjpa.LockTimeout, 3);
I think the environment simply can not follow the load (more than one
mail sent each ms).
Bootleneck can be anywhere (connection creation, database lock,
openjpa,...).
This is why I put a sleep between each mail sending.
Real servers will better hold the pressure.
So I think I'm limited by my environment : it does not allow me to have
a really heavy load to validate that IMAP-137 is resolved.
Well, it is probably, as before you commits, I directly had the
duplicate key exception.
How much mails can you hold on your env with h2 inmemory ?
Tks,
Eric
On 05/01/2010 09:08 AM, Norman Maurer wrote:
Hi Eric,
thx for your tests. The order of the uid is caused by the threads,
thats nothing to worry about.
About the timeout, can you try to add this to the test case:
properties.put(openjpa.LockTimeout, 3);
And keep in mind that we don't use any connection pooling in the test
case so the performance should be better when use it in deployment.
Bye,
Norman
2010/5/1 Eric Charleseric.char...@u-mangate.com:
Hi,
I just tested the JPAStressTest (for IMAP-137 JPA fails to persist
MailboxMembership Entity on heavy load) Norman committed yesterday.
I made some tests towards derby (memory and embedded) , h2 (memory and
embedded) and mysql with following configs (and adding the needed jdbc
drivers in pom.xml):
// Derby Memory
properties.put(openjpa.ConnectionDriverName,
org.apache.derby.jdbc.EmbeddedDriver.class.getName());
properties.put(openjpa.ConnectionURL,
jdbc:derby:memory:derbyimap;create=ue);
// Derby Embedded
//properties.put(openjpa.ConnectionDriverName,
org.apache.derby.jdbc.EmbeddedDriver.class.getName());
//properties.put(openjpa.ConnectionURL,
jdbc:derby:derbyimap;create=ue;user=root;password=root);
// H2 Memory
//properties.put(openjpa.ConnectionDriverName,
org.h2.Driver.class.getName());
//properties.put(openjpa.ConnectionURL,
jdbc:h2:mem:h2imap;DB_CLOSE_DELAY=);
// H2 Embedded
//properties.put(openjpa.ConnectionDriverName,
org.h2.Driver.class.getName());
//properties.put(openjpa.ConnectionURL,
jdbc:h2:~/h2/h2imap;USER=ot;PASSWORD=root);
//Mysql
//properties.put(openjpa.ConnectionDriverName,
com.mysql.jdbc.Driver.class.getName());
//properties.put(openjpa.ConnectionURL,
jdbc:mysql://localhost:3306/msqlimap?user=otpassword=rootcreateDatabaseIfNotExist=trueamp;useUnicode=trueamp;characterEncoding=utf-8);
I first changed the maximum number of messages from 1.000 to 100.000 and saw
on my linux dev PC (james server, stresstest and databases all running on
same dev PC):
Derby: after many mails sending:
java.io.IOException: Too many open files
...
Exception in thread pool-1-thread-1044 java.lang.NoClassDefFoundError:
org/apache/james/imap/api/display/HumanReadableText
...
H2: after 400 mails:
java.io.IOException: Too many open files
...
Caused by:openjpa-1.2.2-r422266:898935 nonfatal general error
org.apache.openjpa.persistence.PersistenceException: Timeout trying to lock
table MEMBERSHIP [50200-79] {prepstmnt 406576
INSERT INTO Membership (mailboxId, uid, answered, deleted, draft,
flagged, internalDate, recent, seen, size, MESSAGE_ID)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
[params=ong) 1, (long) 49, (int) 0, (int) 0, (int) 0, (int) 0, (Timestamp)
2010-05-01 08:37:10.293, (int) 0, (int) 0, (int) 25, (long) 4151]}
[codeP200, state=HYT00]
...
MySql : after +/- 1000 (always around 1002 and 1007) mails :
Exception in thread pool-1-thread-84openjpa-1.2.2-r422266:898935
nonfatal user error org.apache.openjpa.persistence.NoResultException: The
query on candidate type class
org.apache.james.imap.jpa.mail.model.JPAMailbox with filter SELECT mailbox
FROM Mailbox mailbox WHERE mailbox.mailboxId =idParam was configured to
have a unique result, but no instance matched the query.
org.apache.james.imap.mailbox.MailboxNotFoundException: Mailbox
'#mail.INBOX' not found.
...
3 different databases, 3 different behaviour/exceptions.
Running 3 components (james server, stresstest, database) on my
not-so-strong-PC may be an issue.
To make the tests successful, I added a Thread.sleep(5) (15 ms for embedded
derby and h2) between each mail sending.
I suppose we can say that in a real environment, the load will never be that
heavy, and that real server will perform better.
I also saw that the uid order is not respected:
Append message with uidb58
Append message with uidb59
Append message with uidb56
Append message with uidb57
This could come from a print mismatch, and not from uid being generated in a
bad order.
Tks in advance for your feedbacks, if any,
Eric
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JPAStressTest with derby, h2 and mysql
Hi Norman, Trying to simulate a more real-life test case, I developed a small bomber class that launches parallel threads. Each thread sends mails with attachements (with small pause between each mail sending). I tested with embedded derby with JDBCDomainList. Depending on the parameters (10 to 100 parallel threads, 5 ms to 100 ms pause, 1KB to 100KB attachement), I could have on my dev laptop 5 to 10 mails per second being processed. The good think is that I have no exception at all during stable phase. However, I was not able to have more mails being processed on my latptop due to disk access. The CPU and memory are really not charged (20 - 30 %), but disk is really working non-stop. This is probably the price to pay for activemq and the kahadb. Complete scenario with unstable phase: - Bomb 5 minutes - Stop bomb. - Still some mails are spooled for a while which is quite normal. - When all mails are spooled, disk still runs crazy for minutes, console showing a few messages Slow KahaDB access: cleanup took 684 - rebombing : org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: org.apache.james.core.maili...@16863f2] Caused by: java.io.IOException: Too many open files - error message in the bomber - Server console show still a few mails being spooled. - After, server does not respond on socket - no more exception. Disk congestion may come from my disk. I will setup a server environment with good disk, and will rerun the test in a few days. A database on a separate server could also help. Btw, what should we consider as acceptable/needed as load (xxx mails from yyy parallel clients) ? Tks, Eric On 05/01/2010 09:25 AM, Norman Maurer wrote: About 6000, then I run out of heap space.. Can you maybe try to deploy it with james and bomb it via smtp ? I was not able to see any exception. Thx, Norman 2010/5/1 Eric Charleseric.char...@u-mangate.com: Hi Norman, Similar exceptions with properties.put(openjpa.LockTimeout, 3); I think the environment simply can not follow the load (more than one mail sent each ms). Bootleneck can be anywhere (connection creation, database lock, openjpa,...). This is why I put a sleep between each mail sending. Real servers will better hold the pressure. So I think I'm limited by my environment : it does not allow me to have a really heavy load to validate that IMAP-137 is resolved. Well, it is probably, as before you commits, I directly had the duplicate key exception. How much mails can you hold on your env with h2 inmemory ? Tks, Eric On 05/01/2010 09:08 AM, Norman Maurer wrote: Hi Eric, thx for your tests. The order of the uid is caused by the threads, thats nothing to worry about. About the timeout, can you try to add this to the test case: properties.put(openjpa.LockTimeout, 3); And keep in mind that we don't use any connection pooling in the test case so the performance should be better when use it in deployment. Bye, Norman 2010/5/1 Eric Charleseric.char...@u-mangate.com: Hi, I just tested the JPAStressTest (for IMAP-137 JPA fails to persist MailboxMembership Entity on heavy load) Norman committed yesterday. I made some tests towards derby (memory and embedded) , h2 (memory and embedded) and mysql with following configs (and adding the needed jdbc drivers in pom.xml): // Derby Memory properties.put(openjpa.ConnectionDriverName, org.apache.derby.jdbc.EmbeddedDriver.class.getName()); properties.put(openjpa.ConnectionURL, jdbc:derby:memory:derbyimap;create=); // Derby Embedded //properties.put(openjpa.ConnectionDriverName, org.apache.derby.jdbc.EmbeddedDriver.class.getName()); //properties.put(openjpa.ConnectionURL, jdbc:derby:derbyimap;create=;user=root;password=root); // H2 Memory //properties.put(openjpa.ConnectionDriverName, org.h2.Driver.class.getName()); //properties.put(openjpa.ConnectionURL, jdbc:h2:mem:h2imap;DB_CLOSE_DELAY=; // H2 Embedded //properties.put(openjpa.ConnectionDriverName, org.h2.Driver.class.getName()); //properties.put(openjpa.ConnectionURL, jdbc:h2:~/h2/h2imap;USER=;PASSWORD=root); //Mysql //properties.put(openjpa.ConnectionDriverName, com.mysql.jdbc.Driver.class.getName()); //properties.put(openjpa.ConnectionURL, jdbc:mysql://localhost:3306/msqlimap?user=password=rootcreateDatabaseIfNotExist=trueamp;useUnicode=trueamp;characterEncoding=utf-8); I first changed the maximum number of messages from 1.000 to 100.000 and saw on my linux dev PC (james server, stresstest and databases all running on same dev PC): Derby: after many mails sending: java.io.IOException: Too many open files ... Exception in thread pool-1-thread-1044 java.lang.NoClassDefFoundError: org/apache/james/imap/api/display/HumanReadableText ... H2: after 400 mails: java.io.IOException: Too many open files ... Caused
Re: JPAStressTest with derby, h2 and mysql
Hi Norman, What do you mean with ...backlock in smtpserver.log ? Tks, Eric On 05/02/2010 01:15 PM, Norman Maurer wrote: Hi Eric, for such high load you will need to adjust a few things: 1) set a higher ulimit. By default its 1024 on linux which is prolly not enough ( I use fbsd which use 11095 by default) 2) set a higher connection backlock in smtpserver.log Bye, Norman 2010/5/2 Eric Charleseric.char...@u-mangate.com: Hi Norman, Trying to simulate a more real-life test case, I developed a small bomber class that launches parallel threads. Each thread sends mails with attachements (with small pause between each mail sending). I tested with embedded derby with JDBCDomainList. Depending on the parameters (10 to 100 parallel threads, 5 ms to 100 ms pause, 1KB to 100KB attachement), I could have on my dev laptop 5 to 10 mails per second being processed. The good think is that I have no exception at all during stable phase. However, I was not able to have more mails being processed on my latptop due to disk access. The CPU and memory are really not charged (20 - 30 %), but disk is really working non-stop. This is probably the price to pay for activemq and the kahadb. Complete scenario with unstable phase: - Bomb 5 minutes - Stop bomb. - Still some mails are spooled for a while which is quite normal. - When all mails are spooled, disk still runs crazy for minutes, console showing a few messages Slow KahaDB access: cleanup took 684 - rebombing : org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: org.apache.james.core.maili...@16863f2] Caused by: java.io.IOException: Too many open files - error message in the bomber - Server console show still a few mails being spooled. - After, server does not respond on socket - no more exception. Disk congestion may come from my disk. I will setup a server environment with good disk, and will rerun the test in a few days. A database on a separate server could also help. Btw, what should we consider as acceptable/needed as load (xxx mails from yyy parallel clients) ? Tks, Eric On 05/01/2010 09:25 AM, Norman Maurer wrote: About 6000, then I run out of heap space.. Can you maybe try to deploy it with james and bomb it via smtp ? I was not able to see any exception. Thx, Norman 2010/5/1 Eric Charleseric.char...@u-mangate.com: Hi Norman, Similar exceptions with properties.put(openjpa.LockTimeout, 3); I think the environment simply can not follow the load (more than one mail sent each ms). Bootleneck can be anywhere (connection creation, database lock, openjpa,...). This is why I put a sleep between each mail sending. Real servers will better hold the pressure. So I think I'm limited by my environment : it does not allow me to have a really heavy load to validate that IMAP-137 is resolved. Well, it is probably, as before you commits, I directly had the duplicate key exception. How much mails can you hold on your env with h2 inmemory ? Tks, Eric On 05/01/2010 09:08 AM, Norman Maurer wrote: Hi Eric, thx for your tests. The order of the uid is caused by the threads, thats nothing to worry about. About the timeout, can you try to add this to the test case: properties.put(openjpa.LockTimeout, 3); And keep in mind that we don't use any connection pooling in the test case so the performance should be better when use it in deployment. Bye, Norman 2010/5/1 Eric Charleseric.char...@u-mangate.com: Hi, I just tested the JPAStressTest (for IMAP-137 JPA fails to persist MailboxMembership Entity on heavy load) Norman committed yesterday. I made some tests towards derby (memory and embedded) , h2 (memory and embedded) and mysql with following configs (and adding the needed jdbc drivers in pom.xml): // Derby Memory properties.put(openjpa.ConnectionDriverName, org.apache.derby.jdbc.EmbeddedDriver.class.getName()); properties.put(openjpa.ConnectionURL, jdbc:derby:memory:derbyimap;create=; // Derby Embedded //properties.put(openjpa.ConnectionDriverName, org.apache.derby.jdbc.EmbeddedDriver.class.getName()); //properties.put(openjpa.ConnectionURL, jdbc:derby:derbyimap;create=ser=root;password=root); // H2 Memory //properties.put(openjpa.ConnectionDriverName, org.h2.Driver.class.getName()); //properties.put(openjpa.ConnectionURL, jdbc:h2:mem:h2imap;DB_CLOSE_DELAY= // H2 Embedded //properties.put(openjpa.ConnectionDriverName, org.h2.Driver.class.getName()); //properties.put(openjpa.ConnectionURL, jdbc:h2:~/h2/h2imap;USER=ASSWORD=root); //Mysql //properties.put(openjpa.ConnectionDriverName, com.mysql.jdbc.Driver.class.getName()); //properties.put(openjpa.ConnectionURL, jdbc:mysql://localhost:3306/msqlimap?user=assword=rootcreateDatabaseIfNotExist=trueamp;useUnicode=trueamp;characterEncoding=utf-8); I first changed the
Re: JPAStressTest with derby, h2 and mysql
Hi Norman, I've set ulimit -n 65535 (default was 1024), used -Xmx512m (cfr your last commit) and connectionBacklog=900 lsof -p pid| wc -l gives around 550 (stable). No issue after stop (test with embedded Derby). Disk swapping and exceptions came from the 64m limit if you don't set -Xmx (james is not that memory hungry beast, but 64 is really small). Will try on real server later this week to rise the throughput. Tks, Eric On 05/02/2010 06:37 PM, Norman Maurer wrote: Ups I mean connectionBacklog200/connectionBacklog in smtpserver.xml Bye, Norman 2010/5/2 Eric Charleseric.char...@u-mangate.com: Hi Norman, What do you mean with ...backlock in smtpserver.log ? Tks, Eric On 05/02/2010 01:15 PM, Norman Maurer wrote: Hi Eric, for such high load you will need to adjust a few things: 1) set a higher ulimit. By default its 1024 on linux which is prolly not enough ( I use fbsd which use 11095 by default) 2) set a higher connection backlock in smtpserver.log Bye, Norman 2010/5/2 Eric Charleseric.char...@u-mangate.com: Hi Norman, Trying to simulate a more real-life test case, I developed a small bomber class that launches parallel threads. Each thread sends mails with attachements (with small pause between each mail sending). I tested with embedded derby with JDBCDomainList. Depending on the parameters (10 to 100 parallel threads, 5 ms to 100 ms pause, 1KB to 100KB attachement), I could have on my dev laptop 5 to 10 mails per second being processed. The good think is that I have no exception at all during stable phase. However, I was not able to have more mails being processed on my latptop due to disk access. The CPU and memory are really not charged (20 - 30 %), but disk is really working non-stop. This is probably the price to pay for activemq and the kahadb. Complete scenario with unstable phase: - Bomb 5 minutes - Stop bomb. - Still some mails are spooled for a while which is quite normal. - When all mails are spooled, disk still runs crazy for minutes, console showing a few messages Slow KahaDB access: cleanup took 684 - rebombing : org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: org.apache.james.core.maili...@16863f2] Caused by: java.io.IOException: Too many open files - error message in the bomber - Server console show still a few mails being spooled. - After, server does not respond on socket - no more exception. Disk congestion may come from my disk. I will setup a server environment with good disk, and will rerun the test in a few days. A database on a separate server could also help. Btw, what should we consider as acceptable/needed as load (xxx mails from yyy parallel clients) ? Tks, Eric On 05/01/2010 09:25 AM, Norman Maurer wrote: About 6000, then I run out of heap space.. Can you maybe try to deploy it with james and bomb it via smtp ? I was not able to see any exception. Thx, Norman 2010/5/1 Eric Charleseric.char...@u-mangate.com: Hi Norman, Similar exceptions with properties.put(openjpa.LockTimeout, 3); I think the environment simply can not follow the load (more than one mail sent each ms). Bootleneck can be anywhere (connection creation, database lock, openjpa,...). This is why I put a sleep between each mail sending. Real servers will better hold the pressure. So I think I'm limited by my environment : it does not allow me to have a really heavy load to validate that IMAP-137 is resolved. Well, it is probably, as before you commits, I directly had the duplicate key exception. How much mails can you hold on your env with h2 inmemory ? Tks, Eric On 05/01/2010 09:08 AM, Norman Maurer wrote: Hi Eric, thx for your tests. The order of the uid is caused by the threads, thats nothing to worry about. About the timeout, can you try to add this to the test case: properties.put(openjpa.LockTimeout, 3); And keep in mind that we don't use any connection pooling in the test case so the performance should be better when use it in deployment. Bye, Norman 2010/5/1 Eric Charleseric.char...@u-mangate.com: Hi, I just tested the JPAStressTest (for IMAP-137 JPA fails to persist MailboxMembership Entity on heavy load) Norman committed yesterday. I made some tests towards derby (memory and embedded) , h2 (memory and embedded) and mysql with following configs (and adding the needed jdbc drivers in pom.xml): // Derby Memory properties.put(openjpa.ConnectionDriverName, org.apache.derby.jdbc.EmbeddedDriver.class.getName()); properties.put(openjpa.ConnectionURL, jdbc:derby:memory:derbyimap;create= // Derby Embedded //properties.put(openjpa.ConnectionDriverName, org.apache.derby.jdbc.EmbeddedDriver.class.getName()); //properties.put(openjpa.ConnectionURL, jdbc:derby:derbyimap;create=r=root;password=root); // H2 Memory //
Users Repository
Hi, I understand org.apache.james.impl.jamesuser.LocalJamesUsersRepository should be used for 2.3 compatibility and org.apache.james.impl.user.LocalUsersRepository for fresh 3 installations. Is this the case ? Tks, Eric PS1 : Comment in spring-beans.xml could mention the full classpath (org.apache.james.impl.user.LocalUsersRepository) to avoid any ambiguity. PS2 : Default for our coming M1 release could be org.apache.james.impl.user.LocalUsersRepository, migrating from 2.3 to 3 being a story still to be written.
SVN Projects
Hi, smtp/pop3/imap mail processing use different src projects: http://svn.apache.org/viewvc/james/server/trunk/ http://svn.apache.org/viewvc/james/imap/trunk/ http://svn.apache.org/viewvc/james/jsieve/trunk/ http://svn.apache.org/viewvc/james/mailet/ (you will get more than needed, so you may co ...mailet/standard/trunk/, .../mailet/crypto/trunk/, .../mailet/base/trunk/, ...mailet/api/trunk/) http://svn.apache.org/viewvc/james/protocols/trunk/ ... An aggregated project that gathers all needed trunk sources would be welcome. Much easier for updates, especially if you don't want to rely on snapshots dependencies. Tks, Eric - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Config resource loading in dev
Hi, To run james in my ide (eclipse), I copied config files from server/spring-deployment/src/main/config/james to a directory called conf a level upon the root of my project. The resource loading via the file://conf was ok. After, I wanted to load via classpath to have smoother intergration in my ide and also thinking to later integration with 3rd parties (simply using james jars with embedded resources). I changed all needed file://conf/ to classpath: in the xml files (spring-beans,...). I also had to change in SpringConfigurationRegistry.getForComponent line 56 from Resource r = loader.getResource(file://conf/ + name + .xml); to Resource r = loader.getResource(classpath: + name + .xml); I looked for a system-wide configuration (in James.xml or whatever) that would allow to define the way resources are loaded (file or classpath) but didn't find anything. One way to tackle this would be first try on file://conf and rather than throwing directly the RegistryException, still give a try to classpath. In case of duplicate config present (file + classpath), the file would take the precedence, discarding the classpath. You could have a default classpath config, and simply override some via file. On pom.xml, we could also add the following (direct ide setup after mvn eclipse:eclipse). build resources resource directorysrc/main/resources/directory /resource /resources In a first instance, for mvn package, everything could remain as such (all files copied to conf). Any comment? Tks, Eric
Re: Users Repository
Hi Norman, The patch is here after. A quick scan of the code shows that only only UsersRepository interface is used (JamesUsersRepository that adds alias and forward is no more used - but as we rely now on mappings, this should give no functinality loose). I checked that users defined with LocalJamesUsersRepository were still available and working after changing to LocalJamesUsersRepository (needed for people already using trunk). Tks, Eric Index: src/main/config/james/spring-beans.xml === --- src/main/config/james/spring-beans.xml(revision 942302) +++ src/main/config/james/spring-beans.xml(working copy) @@ -238,14 +238,18 @@ /bean !--This is needed to link the smtpserver to the local user repository -LocalJamesUsersRepository is used for backward compatibility with -James 2.3.0 -If backward compatibility is not need the LocalUsersRepository -implementation +org.apache.james.impl.jamesuser.LocalJamesUsersRepository can be used for +backward compatibility with James 2.3.0 +If backward compatibility is not needed, org.apache.james.impl.user.LocalUsersRepository +implementation is used by default. -- - !-- could be safely used -- + + !-- uncomment this for james 2.3.0 backward compatibility bean id=localusersrepository class=org.apache.james.impl.jamesuser.LocalJamesUsersRepository / +-- + !-- this is the default implementation for the localusersrepository - not james 2.3.0 backward compatible -- + bean id=localusersrepository class=org.apache.james.impl.user.LocalUsersRepository / !-- The context FileSystem implementation -- bean id=filesystem class=org.apache.james.container.spring.SpringFileSystem / On 05/08/2010 05:28 AM, Norman Mauer wrote: +1, makes sense.. How about a patch ;) ? Thx, Norman 2010/5/5 Eric Charleseric.char...@u-mangate.com: Hi, I understand org.apache.james.impl.jamesuser.LocalJamesUsersRepository should be used for 2.3 compatibility and org.apache.james.impl.user.LocalUsersRepository for fresh 3 installations. Is this the case ? Tks, Eric PS1 : Comment in spring-beans.xml could mention the full classpath (org.apache.james.impl.user.LocalUsersRepository) to avoid any ambiguity. PS2 : Default for our coming M1 release could be org.apache.james.impl.user.LocalUsersRepository, migrating from 2.3 to 3 being a story still to be written. - To unsubscribe, e-mail:server-dev-unsubscr...@james.apache.org For additional commands, e-mail:server-dev-h...@james.apache.org
Re: Fwd: Re: Config resource loading in dev
Hi Norman,
I made a first approach making SpringConfigurationRegistry abstract with
2 implementations (SpringFileConfigurationRegistry and
SpringClasspathConfigurationRegistry).
I first wanted to use composition, but this would have asked too much
refactoring, so I used inheritance.
The classpath loader still does not work, as there is at least 3
file://conf definitions in spring-beans.xml. We could ask the
administrator to change these, but I find that tedious.
This is why I would rather think to a global setting in James.xml or
spring-beans.xml that would define the resource loading type. In this
case, the attached patch can be forgotten.
Tks in advance for your comments on the attached patch (will be removed
from the james list, so I copied you).
Eric
On 05/08/2010 05:28 AM, Norman Maurer wrote:
Hi Eric,
I like your idea about loading every config from classpath. I think
you are right, we should change it. About the proposed fallback
strategy I'm not sure. I think we should not try to mix up things to
much. Just let use add two different ConfigurationRegistry
implementations. One for classpath and one for file, so if someone
needs to load things from files he could just swap them out.
Bye,
Norman
2010/5/5 Eric Charleseric.char...@u-mangate.com:
Hi,
To run james in my ide (eclipse), I copied config files from
server/spring-deployment/src/main/config/james to a directory called conf a
level upon the root of my project.
The resource loading via thefile://conf was ok.
After, I wanted to load via classpath to have smoother intergration in my
ide and also thinking to later integration with 3rd parties (simply using
james jars with embedded resources).
I changed all neededfile://conf/ to classpath: in the xml files
(spring-beans,...).
I also had to change in SpringConfigurationRegistry.getForComponent line 56
from
Resource r =oader.getResource(file://conf/ + name + .xml);
to
Resource r =oader.getResource(classpath: + name + .xml);
I looked for a system-wide configuration (in James.xml or whatever) that
would allow to define the way resources are loaded (file or classpath) but
didn't find anything.
One way to tackle this would be first try onfile://conf and rather than
throwing directly the RegistryException, still give a try to classpath.
In case of duplicate config present (file + classpath), the file would take
the precedence, discarding the classpath.
You could have a default classpath config, and simply override some via
file.
On pom.xml, we could also add the following (direct ide setup after mvn
eclipse:eclipse).
build
resources
resource
directorysrc/main/resources/directory
/resource
/resources
In a first instance, for mvn package, everything could remain as such (all
files copied to conf).
Any comment?
Tks,
Eric
-
To unsubscribe, e-mail:server-dev-unsubscr...@james.apache.org
For additional commands, e-mail:server-dev-h...@james.apache.org
Index: src/main/java/org/apache/james/container/spring/lifecycle/SpringClasspathConfigurationRegistry.java
===
--- src/main/java/org/apache/james/container/spring/lifecycle/SpringClasspathConfigurationRegistry.java (revision 0)
+++ src/main/java/org/apache/james/container/spring/lifecycle/SpringClasspathConfigurationRegistry.java (revision 0)
@@ -0,0 +1,12 @@
+package org.apache.james.container.spring.lifecycle;
+
+import org.apache.commons.configuration.HierarchicalConfiguration;
+
+public class SpringClasspathConfigurationRegistry extends SpringConfigurationRegistry {
+
+ public HierarchicalConfiguration getForComponent(String componentname)
+ throws org.apache.james.container.spring.Registry.RegistryException {
+ return getForComponent(componentname, SpringConfigurationRegistry.SpringLoadType.CLASSPATH);
+ }
+
+}
Index: src/main/java/org/apache/james/container/spring/lifecycle/SpringFileConfigurationRegistry.java
===
--- src/main/java/org/apache/james/container/spring/lifecycle/SpringFileConfigurationRegistry.java (revision 0)
+++ src/main/java/org/apache/james/container/spring/lifecycle/SpringFileConfigurationRegistry.java (revision 0)
@@ -0,0 +1,12 @@
+package org.apache.james.container.spring.lifecycle;
+
+import org.apache.commons.configuration.HierarchicalConfiguration;
+
+public class SpringFileConfigurationRegistry extends SpringConfigurationRegistry {
+
+ public HierarchicalConfiguration getForComponent(String componentname)
+ throws org.apache.james.container.spring.Registry.RegistryException {
+ return getForComponent(componentname, SpringConfigurationRegistry.SpringLoadType.FILE);
+ }
+
+}
Index: src/main/java/org/apache/james/container/spring/lifecycle/SpringConfigurationRegistry.java
Re: Users Repository
Done, Eric On 05/09/2010 02:17 PM, Norman Maurer wrote: Hi Eric, thx for the patch. could you please create a jira issue and attach the patch there? Be sure to check the grant asl2 box after uploading. Thx Norman 2010/5/9, Eric Charleseric.char...@u-mangate.com: Hi Norman, The patch is here after. A quick scan of the code shows that only only UsersRepository interface is used (JamesUsersRepository that adds alias and forward is no more used - but as we rely now on mappings, this should give no functinality loose). I checked that users defined with LocalJamesUsersRepository were still available and working after changing to LocalJamesUsersRepository (needed for people already using trunk). Tks, Eric Index: src/main/config/james/spring-beans.xml === --- src/main/config/james/spring-beans.xml(revision 942302) +++ src/main/config/james/spring-beans.xml(working copy) @@ -238,14 +238,18 @@ /bean !--This is needed to link the smtpserver to the local user repository -LocalJamesUsersRepository is used for backward compatibility with -James 2.3.0 -If backward compatibility is not need the LocalUsersRepository -implementation +org.apache.james.impl.jamesuser.LocalJamesUsersRepository can be used for +backward compatibility with James 2.3.0 +If backward compatibility is not needed, org.apache.james.impl.user.LocalUsersRepository +implementation is used by default. -- -!-- could be safely used -- + +!-- uncomment this for james 2.3.0 backward compatibility bean id=localusersrepository class=org.apache.james.impl.jamesuser.LocalJamesUsersRepository / +-- +!-- this is the default implementation for the localusersrepository - not james 2.3.0 backward compatible -- +bean id=localusersrepository class=org.apache.james.impl.user.LocalUsersRepository / !-- The context FileSystem implementation -- bean id=filesystem class=org.apache.james.container.spring.SpringFileSystem / On 05/08/2010 05:28 AM, Norman Mauer wrote: +1, makes sense.. How about a patch ;) ? Thx, Norman 2010/5/5 Eric Charleseric.char...@u-mangate.com: Hi, I understand org.apache.james.impl.jamesuser.LocalJamesUsersRepository should be used for 2.3 compatibility and org.apache.james.impl.user.LocalUsersRepository for fresh 3 installations. Is this the case ? Tks, Eric PS1 : Comment in spring-beans.xml could mention the full classpath (org.apache.james.impl.user.LocalUsersRepository) to avoid any ambiguity. PS2 : Default for our coming M1 release could be org.apache.james.impl.user.LocalUsersRepository, migrating from 2.3 to 3 being a story still to be written. - To unsubscribe, e-mail:server-dev-unsubscr...@james.apache.org For additional commands, e-mail:server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Fwd: Re: Config resource loading in dev
Hi Norman, I will do, but I think this is a partial solution. Administrator will still have replace in spring-beans.xml file://conf/ with classpath: Are we happy with that ? Tks, Eric On 05/09/2010 03:33 PM, Norman Maurer wrote: Hi Eric, I had a quick look and it seems to make sense. could you create a jira issue and attach it there... Thx, Norman 2010/5/9 Eric Charleseric.char...@u-mangate.com: Hi Norman, I made a first approach making SpringConfigurationRegistry abstract with 2 implementations (SpringFileConfigurationRegistry and SpringClasspathConfigurationRegistry). I first wanted to use composition, but this would have asked too much refactoring, so I used inheritance. The classpath loader still does not work, as there is at least 3 file://conf definitions in spring-beans.xml. We could ask the administrator to change these, but I find that tedious. This is why I would rather think to a global setting in James.xml or spring-beans.xml that would define the resource loading type. In this case, the attached patch can be forgotten. Tks in advance for your comments on the attached patch (will be removed from the james list, so I copied you). Eric On 05/08/2010 05:28 AM, Norman Maurer wrote: Hi Eric, I like your idea about loading every config from classpath. I think you are right, we should change it. About the proposed fallback strategy I'm not sure. I think we should not try to mix up things to much. Just let use add two different ConfigurationRegistry implementations. One for classpath and one for file, so if someone needs to load things from files he could just swap them out. Bye, Norman 2010/5/5 Eric Charleseric.char...@u-mangate.com: Hi, To run james in my ide (eclipse), I copied config files from server/spring-deployment/src/main/config/james to a directory called conf a level upon the root of my project. The resource loading via the file://conf was ok. After, I wanted to load via classpath to have smoother intergration in my ide and also thinking to later integration with 3rd parties (simply using james jars with embedded resources). I changed all needed file://conf/ to classpath: in the xml files (spring-beans,...). I also had to change in SpringConfigurationRegistry. getForComponent line 56 from Resource r =der.getResource(file://conf/ + name + .xml); to Resource r =der.getResource(classpath: + name + .xml); I looked for a system-wide configuration (in James.xml or whatever) that would allow to define the way resources are loaded (file or classpath) but didn't find anything. One way to tackle this would be first try on file://conf and rather than throwing directly the RegistryException, still give a try to classpath. In case of duplicate config present (file + classpath), the file would take the precedence, discarding the classpath. You could have a default classpath config, and simply override some via file. On pom.xml, we could also add the following (direct ide setup after mvn eclipse:eclipse). build resources resource directorysrc/main/resources/directory /resource /resources In a first instance, for mvn package, everything could remain as such (all files copied to conf). Any comment? Tks, Eric - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Fwd: Re: Config resource loading in dev
Patch uploaded in JAMES-1002. Tks, Eric On 05/09/2010 05:33 PM, Norman Maurer wrote: Hi Eric, yeah I think 'we' are happy with that. We will even need to change all the file://conf stuff in other configuration files too. Bye, Norman 2010/5/9 Eric Charleseric.char...@u-mangate.com: Hi Norman, I will do, but I think this is a partial solution. Administrator will still have replace in spring-beans.xml file://conf/ with classpath: Are we happy with that ? Tks, Eric On 05/09/2010 03:33 PM, Norman Maurer wrote: Hi Eric, I had a quick look and it seems to make sense. could you create a jira issue and attach it there... Thx, Norman 2010/5/9 Eric Charleseric.char...@u-mangate.com: Hi Norman, I made a first approach making SpringConfigurationRegistry abstract with 2 implementations (SpringFileConfigurationRegistry and SpringClasspathConfigurationRegistry). I first wanted to use composition, but this would have asked too much refactoring, so I used inheritance. The classpath loader still does not work, as there is at least 3 file://conf definitions in spring-beans.xml. We could ask the administrator to change these, but I find that tedious. This is why I would rather think to a global setting in James.xml or spring-beans.xml that would define the resource loading type. In this case, the attached patch can be forgotten. Tks in advance for your comments on the attached patch (will be removed from the james list, so I copied you). Eric On 05/08/2010 05:28 AM, Norman Maurer wrote: Hi Eric, I like your idea about loading every config from classpath. I think you are right, we should change it. About the proposed fallback strategy I'm not sure. I think we should not try to mix up things to much. Just let use add two different ConfigurationRegistry implementations. One for classpath and one for file, so if someone needs to load things from files he could just swap them out. Bye, Norman 2010/5/5 Eric Charleseric.char...@u-mangate.com: Hi, To run james in my ide (eclipse), I copied config files from server/spring-deployment/src/main/config/james to a directory called conf a level upon the root of my project. The resource loading via the file://conf was ok. After, I wanted to load via classpath to have smoother intergration in my ide and also thinking to later integration with 3rd parties (simply using james jars with embedded resources). I changed all needed file://conf/ to classpath: in the xml files (spring-beans,...). I also had to change in SpringConfigurationRegistry. getForComponent line 56 from Resource r Þr.getResource(file://conf/ + name + .xml); to Resource r Þr.getResource(classpath: + name + .xml); I looked for a system-wide configuration (in James.xml or whatever) that would allow to define the way resources are loaded (file or classpath) but didn't find anything. One way to tackle this would be first try on file://conf and rather than throwing directly the RegistryException, still give a try to classpath. In case of duplicate config present (file + classpath), the file would take the precedence, discarding the classpath. You could have a default classpath config, and simply override some via file. On pom.xml, we could also add the following (direct ide setup after mvn eclipse:eclipse). build resources resource directorysrc/main/resources/directory /resource /resources In a first instance, for mvn package, everything could remain as such (all files copied to conf). Any comment? Tks, Eric - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Uid Generation Strategy
Hi, Section 2.3.1.1 of IMAP RFC (http://www.faqs.org/rfcs/rfc3501.html) states that Unique identifiers MUST be strictly ascending in the mailbox at all times. This is currently enforced in org.apache.james.imap.store.mail.model.Mailbox#consumeUid() for the JPA store. A JPAStressTest has been setup by Norman to verify that the uid were correctly generated. This showed that parallel threads could give issues in critical section of org.apache.james.imap.jpa.JPAMailbox#reserveNextUid(mailboxSession) (the section between the transaction begin and commit). After chatting and patching with Norman, at least 4 strategies are identified: 1. Use locking specific mecanism of each store. This is what is actually implemented for database store (via LockModeType.PESSIMISTIC_WRITE and adequate timeout) on JPAMaibox. The JCR store would need something similar. Simply document the reserveNextUid to indicate that a locking mecanism should be implemented. Currently, this solution works for JPA and uid are correctly generated. 2. Implement some ReentrantLock (or equivalent synchronization) an abstraction level higher. All specific implementations would benefit from this mecanism. Lock would be for all maiboxes. 3. Change in a way the api (change org.apache.james.imap.mailbox.Mailbox#appendMessage signature for example) to oblige each implementation to have a threadsafe way of generating the uid. 4. Design and implement a more evolved solution that would, per mailbox, maintain the lastuid and queue all uid generation request per mailbox. So the question is to which strategy should be go ? Eric - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [jira] Created: (IMAP-145) Add a Cassandra Mailbox implementation (NoSQL)
Could be of interest: http://www.jpox.org/servlet/forum/viewthread_thread,6087 (see also page 2) Datanucleus already supports hbase (http://www.datanucleus.org/plugins/store.hbase.html) and is apache-friendly (http://www.datanucleus.org/project/license.html). Eric On 05/14/2010 04:38 PM, Norman Maurer (JIRA) wrote: Add a Cassandra Mailbox implementation (NoSQL) -- Key: IMAP-145 URL: https://issues.apache.org/jira/browse/IMAP-145 Project: JAMES Imap Issue Type: New Feature Reporter: Norman Maurer It would be cool to have a Cassandra implementation for Mailboxes. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [Fixed?] Re: James IMAP and Alfresco and refactoring
Hi Norman, Just svn up, mvn package and tested. I've got: Exception in thread pool-16-thread-1 openjpa-2.0.0-r422266:935683 nonfatal user error org.apache.openjpa.persistence.InvalidStateException: Can only perform operation while a transaction is active. FailedObject: SELECT user FROM JamesUser user WHERE user.name=:name [java.lang.String] at org.apache.openjpa.kernel.BrokerImpl.assertTransactionOperation(BrokerImpl.java:4595) at org.apache.openjpa.kernel.BrokerImpl.beginStore(BrokerImpl.java:1378) at org.apache.openjpa.jdbc.kernel.PessimisticLockManager.ensureStoreManagerTransaction(PessimisticLockManager.java:177) at org.apache.openjpa.jdbc.kernel.PessimisticLockManager.selectForUpdate(PessimisticLockManager.java:86) at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:377) at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:363) at org.apache.openjpa.jdbc.sql.LogicalUnion$UnionSelect.execute(LogicalUnion.java:427) at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:230) at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:220) at org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94) at org.apache.openjpa.lib.rop.EagerResultList.init(EagerResultList.java:34) at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1246) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:861) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:792) at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542) at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:288) at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302) at org.apache.openjpa.persistence.QueryImpl.getSingleResult(QueryImpl.java:326) at org.apache.james.server.jpa.JPAUsersRepository.getUserByName(JPAUsersRepository.java:143) at org.apache.james.server.jpa.JPAUsersRepository.test(JPAUsersRepository.java:290) at org.apache.james.impl.user.LocalUsersRepository.test(LocalUsersRepository.java:127) at org.apache.james.imapserver.UserRepositoryAuthenticator.isAuthentic(UserRepositoryAuthenticator.java:45) at org.apache.james.imap.store.StoreMailboxManager.login(StoreMailboxManager.java:425) at org.apache.james.imap.store.StoreMailboxManager.login(StoreMailboxManager.java:468) at org.apache.james.imap.processor.LoginProcessor.doProcess(LoginProcessor.java:67) at org.apache.james.imap.processor.AbstractMailboxProcessor.doProcess(AbstractMailboxProcessor.java:125) at org.apache.james.imap.processor.AbstractMailboxProcessor.process(AbstractMailboxProcessor.java:80) at org.apache.james.imap.processor.AbstractMailboxProcessor.doProcess(AbstractMailboxProcessor.java:73) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:44) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at
Re: [jira] Created: (IMAP-145) Add a Cassandra Mailbox implementation (NoSQL)
maybe the second page: http://www.jpox.org/servlet/forum/viewthread_thread,6087_offset,10 shortened url : http://j.mp/cqR0n3 Eric On 05/15/2010 08:15 AM, Norman Maurer wrote: Hi Eric the first link doesn't work for me Bye Norman 2010/5/15, Eric Charleseric.char...@u-mangate.com: Could be of interest: http://www.jpox.org/servlet/forum/viewthread_thread,6087 (see also page 2) Datanucleus already supports hbase (http://www.datanucleus.org/plugins/store.hbase.html) and is apache-friendly (http://www.datanucleus.org/project/license.html). Eric On 05/14/2010 04:38 PM, Norman Maurer (JIRA) wrote: Add a Cassandra Mailbox implementation (NoSQL) -- Key: IMAP-145 URL: https://issues.apache.org/jira/browse/IMAP-145 Project: JAMES Imap Issue Type: New Feature Reporter: Norman Maurer It would be cool to have a Cassandra implementation for Mailboxes. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [Fixed?] Re: James IMAP and Alfresco and refactoring
Hi Norman, persistence was up-to-date but spring-beans had been merged with local changes. It's ok now. Tks Eric On 05/15/2010 08:10 AM, Norman Maurer wrote: Hi Eric, your persistence.xml is not up-to-date. Compare it with trunk... Bye Norman 2010/5/15, Eric Charleseric.char...@u-mangate.com: Hi Norman, Just svn up, mvn package and tested. I've got: Exception in thread pool-16-thread-1openjpa-2.0.0-r422266:935683 nonfatal user error org.apache.openjpa.persistence.InvalidStateException: Can only perform operation while a transaction is active. FailedObject: SELECT user FROM JamesUser user WHERE user.name=:name [java.lang.String] at org.apache.openjpa.kernel.BrokerImpl.assertTransactionOperation(BrokerImpl.java:4595) at org.apache.openjpa.kernel.BrokerImpl.beginStore(BrokerImpl.java:1378) at org.apache.openjpa.jdbc.kernel.PessimisticLockManager.ensureStoreManagerTransaction(PessimisticLockManager.java:177) at org.apache.openjpa.jdbc.kernel.PessimisticLockManager.selectForUpdate(PessimisticLockManager.java:86) at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:377) at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:363) at org.apache.openjpa.jdbc.sql.LogicalUnion$UnionSelect.execute(LogicalUnion.java:427) at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:230) at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:220) at org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94) at org.apache.openjpa.lib.rop.EagerResultList.init(EagerResultList.java:34) at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1246) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:861) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:792) at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542) at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:288) at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302) at org.apache.openjpa.persistence.QueryImpl.getSingleResult(QueryImpl.java:326) at org.apache.james.server.jpa.JPAUsersRepository.getUserByName(JPAUsersRepository.java:143) at org.apache.james.server.jpa.JPAUsersRepository.test(JPAUsersRepository.java:290) at org.apache.james.impl.user.LocalUsersRepository.test(LocalUsersRepository.java:127) at org.apache.james.imapserver.UserRepositoryAuthenticator.isAuthentic(UserRepositoryAuthenticator.java:45) at org.apache.james.imap.store.StoreMailboxManager.login(StoreMailboxManager.java:425) at org.apache.james.imap.store.StoreMailboxManager.login(StoreMailboxManager.java:468) at org.apache.james.imap.processor.LoginProcessor.doProcess(LoginProcessor.java:67) at org.apache.james.imap.processor.AbstractMailboxProcessor.doProcess(AbstractMailboxProcessor.java:125) at org.apache.james.imap.processor.AbstractMailboxProcessor.process(AbstractMailboxProcessor.java:80) at org.apache.james.imap.processor.AbstractMailboxProcessor.doProcess(AbstractMailboxProcessor.java:73) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:44) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at
Re: [Fixed?] Re: James IMAP and Alfresco and refactoring
trunk is now deployed and serving real traffic. Simply copied the derbydb directory in jame/var to migrate users and mails. openjpa2 migration from openjpa1 seems OK (no compatibility issue with the db defined via openjpa1). Will keep you updated on stability and eventual exceptions. Tks, Eric On 05/15/2010 09:48 AM, Eric Charles wrote: Hi Norman, persistence was up-to-date but spring-beans had been merged with local changes. It's ok now. Tks Eric On 05/15/2010 08:10 AM, Norman Maurer wrote: Hi Eric, your persistence.xml is not up-to-date. Compare it with trunk... Bye Norman 2010/5/15, Eric Charleseric.char...@u-mangate.com: Hi Norman, Just svn up, mvn package and tested. I've got: Exception in thread pool-16-thread-1openjpa-2.0.0-r422266:935683 nonfatal user error org.apache.openjpa.persistence.InvalidStateException: Can only perform operation while a transaction is active. FailedObject: SELECT user FROM JamesUser user WHERE user.name=:name [java.lang.String] at org.apache.openjpa.kernel.BrokerImpl.assertTransactionOperation(BrokerImpl.java:4595) at org.apache.openjpa.kernel.BrokerImpl.beginStore(BrokerImpl.java:1378) at org.apache.openjpa.jdbc.kernel.PessimisticLockManager.ensureStoreManagerTransaction(PessimisticLockManager.java:177) at org.apache.openjpa.jdbc.kernel.PessimisticLockManager.selectForUpdate(PessimisticLockManager.java:86) at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:377) at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:363) at org.apache.openjpa.jdbc.sql.LogicalUnion$UnionSelect.execute(LogicalUnion.java:427) at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:230) at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:220) at org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94) at org.apache.openjpa.lib.rop.EagerResultList.init(EagerResultList.java:34) at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1246) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:861) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:792) at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542) at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:288) at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302) at org.apache.openjpa.persistence.QueryImpl.getSingleResult(QueryImpl.java:326) at org.apache.james.server.jpa.JPAUsersRepository.getUserByName(JPAUsersRepository.java:143) at org.apache.james.server.jpa.JPAUsersRepository.test(JPAUsersRepository.java:290) at org.apache.james.impl.user.LocalUsersRepository.test(LocalUsersRepository.java:127) at org.apache.james.imapserver.UserRepositoryAuthenticator.isAuthentic(UserRepositoryAuthenticator.java:45) at org.apache.james.imap.store.StoreMailboxManager.login(StoreMailboxManager.java:425) at org.apache.james.imap.store.StoreMailboxManager.login(StoreMailboxManager.java:468) at org.apache.james.imap.processor.LoginProcessor.doProcess(LoginProcessor.java:67) at org.apache.james.imap.processor.AbstractMailboxProcessor.doProcess(AbstractMailboxProcessor.java:125) at org.apache.james.imap.processor.AbstractMailboxProcessor.process(AbstractMailboxProcessor.java:80) at org.apache.james.imap.processor.AbstractMailboxProcessor.doProcess(AbstractMailboxProcessor.java:73) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:44) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46) at org.apache.james.imap.processor.base.AbstractChainedProcessor.process(AbstractChainedProcessor.java:46
Re: Another OpenJPA InvalidStateException
Similar issues on POP3 the last weeks with JPA exceptions not logged in james-server.log. I have now configured openjpa.Log in persistence.xml and added the following line in log4j.properties log4j.logger.org.apache.openjpa=TRACE, CONS, FILE But I still don't see much openjpa in the logs. Eric On 05/17/2010 07:16 AM, Norman Maurer wrote: Hi Vanuj, could you give us some more informations ? * Which revision of trunk ? * Does it happen everytime ? * Any changes made in persistence.xml or spring-beans.xml ? * Could enable the TRACE logging in persistence.xml and show us the full stack trace... : property name=penjpa.Log value=DefaultLevel=TRACE, Runtime=ACE, Tool=TRACE/ Thx, Norman 2010/5/17 Apache Jamesvan.n...@yahoo.com: Hi, Apologies for the Spam, I just built James from trunk and deployed it on tomcat a few hours ago. Everything seems to be working fine but there is a problem in POP3 server. When I hit the 'List' command does not return any response and hits an exception in JPAMessageMapper.findMessagesInMailboxWithUID() openjpa-2.0.0-r422266:935683 fatal user error org.apache.openjpa.persistence.InvalidStateException: The context has been closed. The stack trace at which the context was closed is available if Runtime=ACE logging is enabled. Has anyone faced this recently? Thanks, Vanuj. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Another OpenJPA InvalidStateException
Hi Vanuj, Seems like your eclipse has some confusion with openjpa1 and openjpa2. (James uses openjpa2 since 1 week). I also activated the TRACE log, and didn't receive your message [main] openjpa.Runtime - Not creating a ValidatorImpl because this app is using the JPA 1.0 Spec Both exceptions are related to EntityManager, so I think you have to look at your jpa dependencies. Tks, Eric On 05/17/2010 07:14 PM, Apache James wrote: Hi Norman, I enabled trace and found that there are two exceptions happening. One on james startup and the other when the LIST or RETR command gets called. Only the one on startup gets logged. Also, There is this message during printed during server startup that looked interesting. (Is is supposed to be this way) 312 James TRACE [main] openjpa.Runtime - Not creating a ValidatorImpl because this app is using the JPA 1.0 Spec The Exceptions along with the other details are inline. Thanks, Vanuj. On 5/17/2010 10:46 AM, Norman Maurer wrote: Hi Vanuj, could you give us some more informations ? * Which revision of trunk ? 943607 * Does it happen everytime ? Funny thing. Whenever I try to get mail using thunderbird, the exception occurs during 'LIST' command and thunderbird times out waiting for james to reply. But if I create a raw connection with James and try to hit James, it goes through and the list is returned. But then the same exception at the time of RETR command. * Any changes made in persistence.xml or spring-beans.xml ? Nope. I am building james for the first time. Did not touch any code. * Could enable the TRACE logging in persistence.xml and show us the full stack trace... : property name=openjpa.Log value=DefaultLevel=TRACE, Runtime=TRACE, Tool=TRACE/ EXCEPTION 1 Here is the exception that happens during the startup. java.lang.reflect.UndeclaredThrowableException at $Proxy27.addTransformer(Unknown Source) at org.apache.openjpa.persistence.PersistenceProviderImpl.createContainerEntityManagerFactory(PersistenceProviderImpl.java:169) at org.apache.openjpa.persistence.PersistenceProviderImpl.createContainerEntityManagerFactory(PersistenceProviderImpl.java:65) at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:225) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:308) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1469) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1409) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190) at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.findDefaultEntityManagerFactory(PersistenceAnnotationBeanPostProcessor.java:529) at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.findEntityManagerFactory(PersistenceAnnotationBeanPostProcessor.java:495) at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor$PersistenceElement.resolveEntityManagerFactory(PersistenceAnnotationBeanPostProcessor.java:642) at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor$PersistenceElement.getResourceToInject(PersistenceAnnotationBeanPostProcessor.java:633) at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:159) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:84) at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.postProcessPropertyValues(PersistenceAnnotationBeanPostProcessor.java:338) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1064) at
Re: Uid Generation Strategy
Hi Norman, As discussed, it may be worth to reread the RFC requirements: I've copy/paste the RFC paragraph here after. Can we deduce from that that the uid must be incremented by one for each mail arriving in a mailbox? Tks, Eric 2.3.1.1.Unique Identifier (UID) Message Attribute A 32-bit value assigned to each message, which when used with the unique identifier validity value (see below) forms a 64-bit value that MUST NOT refer to any other message in the mailbox or any subsequent mailbox with the same name forever. Unique identifiers are assigned in a strictly ascending fashion in the mailbox; as each message is added to the mailbox it is assigned a higher UID than the message(s) which were added previously. Unlike message sequence numbers, unique identifiers are not necessarily contiguous. The unique identifier of a message MUST NOT change during the session, and SHOULD NOT change between sessions. Any change of unique identifiers between sessions MUST be detectable using the UIDVALIDITY mechanism discussed below. Persistent unique identifiers are required for a client to resynchronize its state from a previous session with the server (e.g., disconnected or offline access clients); this is discussed further in [IMAP-DISC]. Associated with every mailbox are two values which aid in unique identifier handling: the next unique identifier value and the unique identifier validity value. The next unique identifier value is the predicted value that will be assigned to a new message in the mailbox. Unless the unique identifier validity also changes (see below), the next unique identifier value MUST have the following two characteristics. First, the next unique identifier value MUST NOT change unless new messages are added to the mailbox; and second, the next unique identifier value MUST change whenever new messages are added to the mailbox, even if those new messages are subsequently expunged. Note: The next unique identifier value is intended to provide a means for a client to determine whether any messages have been delivered to the mailbox since the previous time it checked this value. It is not intended to provide any guarantee that any message will have this unique identifier. A client can only assume, at the time that it obtains the next unique identifier value, that messages arriving after that time will have a UID greater than or equal to that value. The unique identifier validity value is sent in a UIDVALIDITY response code in an OK untagged response at mailbox selection time. If unique identifiers from an earlier session fail to persist in this session, the unique identifier validity value MUST be greater than the one used in the earlier session. Note: Ideally, unique identifiers SHOULD persist at all times. Although this specification recognizes that failure to persist can be unavoidable in certain server environments, it STRONGLY ENCOURAGES message store implementation techniques that avoid this problem. For example: 1) Unique identifiers MUST be strictly ascending in the mailbox at all times. If the physical message store is re-ordered by a non-IMAP agent, this requires that the unique identifiers in the mailbox be regenerated, since the former unique identifiers are no longer strictly ascending as a result of the re-ordering. 2) If the message store has no mechanism to store unique identifiers, it must regenerate unique identifiers at each session, and each session must have a unique UIDVALIDITY value. 3) If the mailbox is deleted and a new mailbox with the same name is created at a later date, the server must either keep track of unique identifiers from the previous instance of the mailbox, or it must assign a new UIDVALIDITY value to the new instance of the mailbox. A good UIDVALIDITY value to use in this case is a 32-bit representation of the creation date/time of the mailbox. It is alright to use a constant such as 1, but only if it guaranteed that unique identifiers will never be reused, even in the case of a mailbox being deleted (or renamed) and a new mailbox by the same name created at some future time. 4) The combination of mailbox name, UIDVALIDITY, and UID must refer to a single immutable message on that server forever. In particular, the internal date, [RFC-2822 http://www.faqs.org/rfcs/rfc2822.html] size, envelope, body structure, and message texts (RFC822 http://www.faqs.org/rfcs/rfc822.html,RFC822
Re: Mailbox(Manager)
Hi, I also personally found that the 2 mentioned goals would be fine before 3.0 release: 1.- Migrate imap projects/packages to store and protocol. 2.- Uniform the current apache-... james-... namings. And, yes, in the middle-term, it would be much cleaner. Now, when you talk about apache-james-imap-store, you need to translate the project responsible to access the different mail stores. During some time, I was really confused about this. On the other hand, mailet and jsieve projects are just released with current naming and the amount of work to migrate/rename is really huge without any user functionality counterpart. As Tim, I was also asking-party for a migration and uniform naming. But I just feel now that we should concentrate on releasing 3.0, trying to find and solve bugs in trunk. If we release, we should have more users, helping us to solve issues. The migration and renaming could be strong goals for 3.1 in a near future. Maybe we should in priority have a list of JIRA still to solve before releasing. I also find that the website documentation needs some urgent updates (how to setup and use james 3.0). Tks, Eric On 05/29/2010 12:08 PM, Tim-Christian Mundt wrote: Hey, when doing this, we could as well unify the artifact names. We have names like apache-james-imap-* (both, apache and james) james-server-* (only james) apache-mailet-* (only apache) I'm aware that this is no particularly helpful/urgent contribution of mine, but in the long term consistency and a clear structure pay out, I think. Let me know if I can be of any help with this. Tim Am Samstag, den 29.05.2010, 09:36 +0200 schrieb Norman Maurer: Hi Tim, I agree it should get moved out of the imap namespace. I delayed that to make it not do complicated by just introduce another dependency on a mini-library. But yeah its something we should do... Bye, Norman 2010/5/29 Tim-Christian Mundtd...@tim-erwin.de: Hi, since both IMAP and POP3 use the same backend, now (https://issues.apache.org/jira/browse/JAMES-983), shouldn't the storage packages rather be moved from the IMAP library to the server module? They are not IMAP specific any more. Maybe, they could get their own mailstore library which both the IMAP library and the server depend on. Regards Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: more maven
Hi Tim, I you do 'mvn install' on your dependency, your project should take this into account the 'last' dependency in place of the snapshot. But I'm not sure of this for james because I don't work as such (should test it). I use m2eclipse plugin for eclipse and take a little time to configure as such: - I enable 'enable dependency management' - I enable 'workspace resolution' - I close/open all projects All projects dependency known as source projects in eclipse now really use the projects in eclipse rather than the dependencies in your local repository. Once configured, you have a pratical environment to develop: any modification in your projects are directly taken into account in the others without any 'mvn install' or jar copy. Tks, Eric On 06/01/2010 11:43 PM, Tim-Christian Mundt wrote: Hi guys, for maven stuff I could as well ask google, but this seems kinda james specific: how can I build a part of imap (e.g. jpa) with what I have in my working copy as dependency? It will use the downloaded artifacts instead which are in some cases not compatible (I changed several parts). Do I need to do a full build (which is kinda intelligent and does not rebuild everything, I know)? Thanks Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Character encoding for some senders
Hi,
Since a few weeks, I sometimes receive mails that are correct if I look
at the source, but that are displayed completely wrong in thunderbird.
It occurs systematically for some senders, and since a few weeks after I
migrated migrated (i didn't change anything to thunderbird config).
Example:
�٥�杶+e{�b�W�z֡��^���*���O*^r��z{��m�f��Z�ǭy�[�ܛ��%z֛��v��o
��^� zYlq�e��]�xѩb�pz{^���v֛��v��o��^� zYlq�e��]�x���v�
��Z�('zYlq�e��]�x���v
What could I do to identify the cause (james or not) ?
Tks,
Eric
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
Re: more maven
Hi Tim, Just to complete my input (and probably not related to your question), I launch the Main james class from eclipse and works with /conf and /var a directory above eclipse ./ I only use 'mvn package' when I need to deploy on a real production server, and not during development phase. Tks, Eric On 06/02/2010 08:51 AM, Tim-Christian Mundt wrote: Hi Eric, I'm already using it exactly the way you describe. It works perfectly for development and browsing the code. All dependencies are resolved correctly (eclipse is amazing!!). However, this does not apply for the build command. Even if I invoke the build in eclipse, it will fail. It only works if I build the whole IMAP component, but this takes much longer, that's why I'm asking if there's another possibility. Thanks Tim Am Mittwoch, den 02.06.2010, 05:36 +0200 schrieb Eric Charles: Hi Tim, I you do 'mvn install' on your dependency, your project should take this into account the 'last' dependency in place of the snapshot. But I'm not sure of this for james because I don't work as such (should test it). I use m2eclipse plugin for eclipse and take a little time to configure as such: - I enable 'enable dependency management' - I enable 'workspace resolution' - I close/open all projects All projects dependency known as source projects in eclipse now really use the projects in eclipse rather than the dependencies in your local repository. Once configured, you have a pratical environment to develop: any modification in your projects are directly taken into account in the others without any 'mvn install' or jar copy. Tks, Eric On 06/01/2010 11:43 PM, Tim-Christian Mundt wrote: Hi guys, for maven stuff I could as well ask google, but this seems kinda james specific: how can I build a part of imap (e.g. jpa) with what I have in my working copy as dependency? It will use the downloaded artifacts instead which are in some cases not compatible (I changed several parts). Do I need to do a full build (which is kinda intelligent and does not rebuild everything, I know)? Thanks Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Character encoding for some senders
The issue is not james related (nor thunderbird). It comes from wrong mail header set by sender (saying base64-encoding, but is not): Content-Transfer-Encoding: base64 Sorry for the spam, Eric On 06/02/2010 07:28 PM, Norman Maurer wrote: - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Cut a milestone ?
Hi, I will deploy a fresh trunk this night just to make sure everything is still ok (cfr last commits in protocol,..). The last snapshot I took is 2 weeks-old and is really stable. Tks, Eric On 06/07/2010 04:40 PM, Norman Maurer wrote: Hi all, I think all the stuff in the imap library is now very usable. I think we should cut a milestone and then cut one of james server. Anything you guys want to get refactored before ? Bye, Norman Ps: Even if we discover a bug later we can cut just another one.. Release often, Release early (urgh...) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Cut a milestone ?
Running here without any problem jpa (embedded derby) + jdbc domainlist + spamassassin + forwarding mailet. When do you think to release? Tks, Eric On 06/07/2010 04:51 PM, Norman Maurer wrote: Thx mate.. I deployed fresh trunk and everything seems to work so far without problems :) Bye, Norman Ps: I'm using JCR Mailbox 2010/6/7 Eric Charleseric.char...@u-mangate.com: Hi, I will deploy a fresh trunk this night just to make sure everything is still ok (cfr last commits in protocol,..). The last snapshot I took is 2 weeks-old and is really stable. Tks, Eric On 06/07/2010 04:40 PM, Norman Maurer wrote: Hi all, I think all the stuff in the imap library is now very usable. I think we should cut a milestone and then cut one of james server. Anything you guys want to get refactored before ? Bye, Norman Ps: Even if we discover a bug later we can cut just another one.. Release often, Release early (urgh...) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Cut a milestone ?
Hi Norman, - I will also try to see what happens with the foreign keys. - MESSAGE_PROPERTY and MESSAGE_HEADER are created today during a fresh install. - Yes, input from community is welcome for the primary key generation strategy. Anyone? - I now better understand JPAMessage streaming. Many tks for the explanation. Will also look for derby (maybe with latest 1.6). Back to MailboxMembership, I was happy to see uid is part of the MailboxMembership. It's good that a mail copy get a new UID. As Tim, I don't often copy mails: the space argument should not be retained. The memory argument is interesting for database that does not support streaming. I'm just wondering when the mail is completely loaded: during mail list when a client consults a server or only during attachment download? We should also stick to a common strategy for all stores (jcr, jpa,...). Is JCR also working with a temporary structure linking Mailbox and Mail? Tks, Eric On 06/10/2010 11:09 AM, Norman Maurer wrote: Hi Eric, comments inside... 2010/6/9 Eric Charleseric.char...@u-mangate.com: Hi, A stable database schema would be great. Upgrading jars is straightforward, but changing the db schema (and potentially the datas) is always a pain. I had a look at the generated tables (see the list after the text). All have primary key and unique constraints, but there are no foreign keys (no constraints integrity). The buildSchema(ForeignKeys=ue) does not seem to create something for the @ManyToOne. Or maybe is this specific to derby (I'm using), and other databases such as mysql,... behave differently? I need to check, I'm currently using JCR so no idea atm.. I also didn't find an entity for MESSAGE_PROPERTY nor MESSAGE_HEADER. However the table exists. Maybe somthing from the good old days ? does they get created on a fresh install ? As far I can understand, the primary keys are all generated via the default OPENJPA_SEQUENCE_TABLE. I often let each the table auto_generate the primary keys (strategy=nerationType.IDENTITY). Are there any plans for this in the future, or will we remain with the unique default OPENJPA_SEQUENCE_TABLE? I have no strong opinion here.. Anyone ? About MailboxMembership, Tim, you said the whole message is not loaded tks to streaming. Right now, openjpa.streamingúlse in database.properties but when I look JPAMessage implementation, it seems that the content is loaded via InputStream. So what is the current state? Should the openjpa.streaming property be removed? Let me try to explain it.. When you have a look at JPAMessage you see it use a byte[] object to store the message content. So once you need to read the content, the whole content get loaded in memory. Thats the default and work for every db. If you use openjpa.streaming=ue it will stream the content direct from the db. So it never need to load the content into the memory at all. This only works for a few databases (derby is not one of them). Thats why it is false by default. Sorry for my in-vrac comments. Each of them may worth a separate thread. Tks, Eric MEMBERSHIP JAMESUSER MAILBOX HEADER MESSAGE MESSAGE_HEADER MESSAGE_PROPERTY DOMAIN SUBSCRIPTION PROPERTY BAYESIANANALYSIS_SPAM VIRTUALUSERTABLE BAYESIANANALYSIS_MESSAGECOUNTS BAYESIANANALYSIS_HAM OPENJPA_SEQUENCE_TABLE On 06/09/2010 10:18 AM, Norman Maurer wrote: Right, I think we should at least be sure to not change the layout anymore. I don't have a strong opinion on the double storage vs. single storage. If we don't need the single storage we could just merge Document and MailboxMembership interface. Bye, Norman 2010/6/9 Tim-Christian Mundtd...@tim-erwin.de: Hi! There is this one change of the database layout which should be decided upon before releasing, I think. Messing with code after the release is fine, but changing the database would be bad. I'm talking about Normans idea to unite the Message and Membership interfaces/classes = tables. The original idea of separating those was not to pass the whole mime message around when just the flags are needed. Moreover, the Membership part would serve as a reference to the Message so that a copy would just mean another reference to the same mail. The question: does the increased complexity really pay off? I think saving space is not really an argument because it's not really common to have duplicates (at least in my experience). And loading the whole message would not mean the contents but only a stream handler, not a memory killer, right? Regards Tim Eric Charles: Running here without any problem jpa (embedded derby) + jdbc domainlist + spamassassin + forwarding mailet. When do you think to release? Tks, Eric On 06/07/2010 04:51 PM, Norman Maurer wrote: Thx mate.. I deployed fresh trunk and everything seems to work so far without problems :) Bye, Norman Ps: I'm using JCR Mailbox 2010/6/7 Eric Charleseric.char...@u
Re: [ANNOUNCE] New committer Eric Charles
Hi everyone, Happy to be onboard. Hope to hear you on the ml. Tks, Eric On 06/12/2010 08:14 AM, Norman Maurer wrote: Hi all, say welcome to our new committer Eric Charles. He was voted as new committer the last week :) Welcome on board... Bye, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Cut a milestone ?
Hi Tim, I implemented the standard @OneToMany (not the proprietary extension) without intermediary table and the schema is created. As you described, it hanged the first, but recompiling and relaunching made it happen. I'm working now on an issue when inserting mail (fk is not set). I think a few unit tests in the apache-james-imap-jpa will help. On my side property name=penjpa.Log value=SQL=TRACE/ shows the SQL (in fact the JPQL). Recommendations found on http://openjpa.apache.org/faq.html allow the fk to be created. I finally think the current default pk strategy is the good one. There is still the decision to take for the MailMembership. Finally, I don't think we should take the coherence between all stores as an argument. For example, storing on file system is completely different than storing in DB, than in JCR, than in NoSQL,... The only reason why we should keep it may be the memory economy when listing mails for databases that does not support the streaming. Tks, Eric On 06/11/2010 08:59 AM, Norman Maurer wrote: Hi Tim, comments inside.. 2010/6/10 Tim-Christian Mundtd...@tim-erwin.de: Hi, Am Donnerstag, den 10.06.2010, 19:56 +0200 schrieb Norman Maurer: After thinking a bit more about it, could it be that the two tables are used for mapping ? I think that would make sense because we have a many relation here.. So one for mapping Message and Properties and one for Message and Header. That's true. However, we don't have Many-to-many mappings here, but one to many (as defined in AbstractJPAMessage). So these tables are basically superfluous and causing (little) performance loss. The reason they are created by OpenJPA is that the mappings are unidirectional. You are right.. One-to-Many should not need the join table. Although we don't need a message field in Headers and Properties I think we should introduce it and thus allow for One-To-Many without mapping table. I already tried doing that, adding @JoinColum and mappedBy according to the specification: http://java.sun.com/javaee/5/docs/api/javax/persistence/OneToMany.html However, when mappedBy=essage is added to the AbstractJPAMessage then OpenJPA hangs without any error output (despiteproperty name=penjpa.Log value=SQL=TRACE/) and the tables won't get created. If put into JPAMessage (overriding the field) the tables get created as usual simply with the addition of a join column (in Header). I guess it has to do with the inheritance. Any idea? Otherwise I'd ask on some OpenJPA mailing list. Askin on the OpenJPA mailing list would be a good idea.. Besides that there is a proprietary solution for OpenJPA: @OneToMany(cascade =ascadeType.ALL, fetchþtchType.LAZY) @ElementJoinColumn(name=essage_id, referencedColumnName=id) private ListJPAHeader headers; This fixes the issue but is non-standard. Other framworks like jpox also have such options if I don't err. If nothing helps we could move the mapping stuff to the OpenJPA* classes, but I think it should be possible with using plain JPA. Tim Bye, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [jira] Resolved: (JAMES-402) Add IPv6 support to NetMachter.
Norman, Tks for your support and code review. Eric On 06/13/2010 06:48 PM, Norman Maurer wrote: Very good work mate :) Bye Norman 2010/6/13 Eric Charles (JIRA)server-dev@james.apache.org: [ https://issues.apache.org/jira/browse/JAMES-402?page=m.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Charles resolved JAMES-402. Resolution: Fixed IP v6 introduced with http://svn.apache.org/viewvc?rev•4226view=rev Currently, the IP address must be completely specified (not shortened, as for example ::1) in the Matcher config. The patch is currently enough to further implement IPv6 support with JAMES-403 and JAMES-405. It is worth to come back later one to validate the complete James IPv6 support and to support the complete IPv6 syntax. Therefor, the issue is not closed. It is however of enough quality to support IPv6 matchers. Add IPv6 support to NetMachter. --- Key: JAMES-402 URL: https://issues.apache.org/jira/browse/JAMES-402 Project: JAMES Server Issue Type: Sub-task Components: James Core, SMTPServer Affects Versions: 2.1.3, 2.2.0, 2.3.0 Reporter: Arjan Veenstra Assignee: Eric Charles Priority: Minor Fix For: Trunk org.apache.james.util.NetMacher should be able to accept and match IPv6 network specifications to support relaying of mail that comes in over IPv6. This should add IPv6 support to the AuthorizedAdresses configuration and likely to most subclasses of AbstractNetworkMatcher as well. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Cut a milestone ?
Hi Tim, I'm a big fan of autoincrement pk columns per table. Each table has its own responsiblity, and keys are generated sequentially per table. This gives however requirements when you migrate via export/import to fresh database. For example, with Derby, the schema is create with GENERATED ALWAYS columns and not GENERATED BY DEFAULT. You will have issues on fk constraints even if you apply the procedure described in http://db.apache.org/derby/docs/dev/tools/ctoolsimportidentitycol.html. The current solution uses a pool of 100 keys in memory, and access to database is limited. We could also define a key-generation-table per table, but I find this overkill. I would tend to think to leave it like it is. For the direct @OneToMany, we have issues when implemented in James even if this samples show that it should be correct. Strange... You said JPA 2.0 defines a way to define the association only in the parent (*Message): Can you send me a working sample? One more point is the sql generated. The logs show jpql, and not sql. Do you know if it's possible to view the sql in the logs (I didn't find a way). If not possible, can you log on the database (mysql I think you use)? On my side, I could analyse the derby queryplan and see if we would gain something without intermediate table. Tks, Eric On 06/16/2010 08:37 AM, Tim-Christian Mundt wrote: Hi Eric, You will find attached (removed from ml, so bcc to you): seems like attachments are not stripped via ML, both emails here have the patch attached. I have to define the OneToMany on the concrete class (the JPAMessage). Yes, that's what I tried then. If this is done without changing JPAHeader/JPAProperty it should be ok, right? I spent hours trying to solve this without success. I can see that from the time you sent this mail... oh boy, that's late/early. One more point: for the insert to work, you need to add property name=openjpa.InverseManager value=true/ in persistence.xml (not in the patch) Is this still standard JPA? Just asking because if so, why would we need an extra option? Currently, the patch still gives issues when reading mails (null pointer on the properties list). Apparently, although the tables are created they are not used. Weird. Even if we solve these last points, we will remain with another issue... Indeed, in the sample, the type of the MayToOne needs to be a concrete class, otherwise openjpa complains. This means that you would be obliged in the Property to link to JPAMessage or JPAStreamingMessage. You can't link to AbstractJPAMessage. JPA 2.0 defines a way to define the association only in the parent (*Message). Then that would be ok. Although it's still strange that it works in your tests but not in James (also not for me). About the pk creation, I will reply a bit later. I reconsidered this issue and I think we should switch to Strategy.IDENTITY = autoincrement because that would allow other systems and implementations to work with this database without messing with the sequence table. Moreover, it's faster. Thanks for the files. Will check them out tonight. Best, Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Cut a milestone ?
Hi Tim, Sorry, message_id was indeed needed in the patch, otherwise you've got a id column which is pk and fk. I still made some additional tests, and omitting the @JoinColumn also works. DB is created and mail delivery is correctly working when you apply on JPAProperty. As soon as you apply on JPAProperty and JPAHeader, it hangs. Still looking for. Bye, Eric On 06/17/2010 08:26 AM, Eric Charles wrote: Hi Tim, Comments inside. Tks, Eric On 06/17/2010 12:58 AM, Tim-Christian Mundt wrote: Hey Eric, For example, with Derby, the schema is create with GENERATED ALWAYS columns and not GENERATED BY DEFAULT. Why's that? Wouldn't GENERATED BY DEFAULT solve this problem so we could happily use autoincrements? GENERATED BY DEFAULT would help, but openjpa create with GENERATED ALWAYS. Moreover, when you recreate your database, you need to define a START value depending on your last generated key. We could also define a key-generation-table per table, but I find this overkill. I agree, that's too much. OK I would tend to think to leave it like it is. If you consider the issue above really serious, than lets keep it. On the other hand: nobody would use derby in production, right? Do similar problems exist with other databases? I sometimes use derby in production with low-end PCs as servers. Each database has it own way and syntax for managing pk generation, with its goodness and pitfalls. For the direct @OneToMany, we have issues when implemented in James even if this samples show that it should be correct. Strange... On my side something in my dummy project is just odd. Can't get this running at all. JPA seems to be a bit delicate? OK You said JPA 2.0 defines a way to define the association only in the parent (*Message): Can you send me a working sample? As said: I can't really get it working here. I've seen it on this site: http://en.wikibooks.org/wiki/Java_Persistence/OneToMany#Example_of_a_JPA_2.0_unidirectional_OneToMany_relationship_database The @JoinColumn is put on the other side using referencedColumnName. this sample may be a non working one. One more point is the sql generated. The logs show jpql, and not sql. Do you know if it's possible to view the sql in the logs (I didn't find a way). No, idea, sorry. I'm happy with jpql :) OK. Was just curious to see the generated sql that can show if the database queries are efficient or not. If not possible, can you log on the database (mysql I think you use)? MySQL provides logs for all operations, yes. Will try to get these from derby. I noticed a small issue with your patch: You have to use message_id instead of id for the name of the @JoinColumn. Patch was working on my side with tables and fk created Strange, but I think to remember it was working with both (id and message_id) Lets ask on the OpenJPA ML how to make OpenJPA use the relationship. Maybe they can even have a look at our concrete issue with James. Will you ask? Should I? I will post something in the coming days. But it is working as designed with the samples. They may say it is in the way we are using it. Maybe someone will take time to help us solve it. On the other hand, I'm not sure it is an issue to leave those intermediary table and I don't think it should be a blocker for the release. Best, Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Cut a milestone ?
Hi, that's me again, I just posted some questions on openjpa ml. If we want to keep JPAStreamingMessage without intermediary tables, we may use the proprietary @ElementJoinColumn annotation. For other features, such as index creation, we also probably need proprietary annotations. For example, the current findUnseenMessagesInMailboxOrderByUid SELECT membership FROM Membership membership WHERE membership.mailboxId = :idParam AND membership.seen = FALSE ORDER BY membership.uid ASC could benefit from @Index @Id private long uid; @Index @Id private long mailboxId; So the questions is Should we use those openjpa annotations?. I personally don't see why we couldn't. What do you think of? Btw, I think the original question was Cut a milestone ?... :) Tks, Eric PS: Better use the apache-james-imap-jpa-0.1-M2-SNAPSHOT.jar enhanced via maven in eclipse. If you use the project class folder, the enhancement is not always as it should. Many problems during dev comes from eclipse (or whatever) does not enhance, or take the previous enhancement... F5 is also your friend. On 06/17/2010 11:33 PM, Eric Charles wrote: Hi Tim, Sorry, message_id was indeed needed in the patch, otherwise you've got a id column which is pk and fk. I still made some additional tests, and omitting the @JoinColumn also works. DB is created and mail delivery is correctly working when you apply on JPAProperty. As soon as you apply on JPAProperty and JPAHeader, it hangs. Still looking for. Bye, Eric On 06/17/2010 08:26 AM, Eric Charles wrote: Hi Tim, Comments inside. Tks, Eric On 06/17/2010 12:58 AM, Tim-Christian Mundt wrote: Hey Eric, For example, with Derby, the schema is create with GENERATED ALWAYS columns and not GENERATED BY DEFAULT. Why's that? Wouldn't GENERATED BY DEFAULT solve this problem so we could happily use autoincrements? GENERATED BY DEFAULT would help, but openjpa create with GENERATED ALWAYS. Moreover, when you recreate your database, you need to define a START value depending on your last generated key. We could also define a key-generation-table per table, but I find this overkill. I agree, that's too much. OK I would tend to think to leave it like it is. If you consider the issue above really serious, than lets keep it. On the other hand: nobody would use derby in production, right? Do similar problems exist with other databases? I sometimes use derby in production with low-end PCs as servers. Each database has it own way and syntax for managing pk generation, with its goodness and pitfalls. For the direct @OneToMany, we have issues when implemented in James even if this samples show that it should be correct. Strange... On my side something in my dummy project is just odd. Can't get this running at all. JPA seems to be a bit delicate? OK You said JPA 2.0 defines a way to define the association only in the parent (*Message): Can you send me a working sample? As said: I can't really get it working here. I've seen it on this site: http://en.wikibooks.org/wiki/Java_Persistence/OneToMany#Example_of_a_JPA_2.0_unidirectional_OneToMany_relationship_database The @JoinColumn is put on the other side using referencedColumnName. this sample may be a non working one. One more point is the sql generated. The logs show jpql, and not sql. Do you know if it's possible to view the sql in the logs (I didn't find a way). No, idea, sorry. I'm happy with jpql :) OK. Was just curious to see the generated sql that can show if the database queries are efficient or not. If not possible, can you log on the database (mysql I think you use)? MySQL provides logs for all operations, yes. Will try to get these from derby. I noticed a small issue with your patch: You have to use message_id instead of id for the name of the @JoinColumn. Patch was working on my side with tables and fk created Strange, but I think to remember it was working with both (id and message_id) Lets ask on the OpenJPA ML how to make OpenJPA use the relationship. Maybe they can even have a look at our concrete issue with James. Will you ask? Should I? I will post something in the coming days. But it is working as designed with the samples. They may say it is in the way we are using it. Maybe someone will take time to help us solve it. On the other hand, I'm not sure it is an issue to leave those intermediary table and I don't think it should be a blocker for the release. Best, Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr
OpenJPA usage at Apache James
Hi, Apache James project uses OpenJPA 2.0. We've got @OneToMany relations on AbstractJPAMessage [1] to JPAHeader [2] (headers attribute) and JPAProperty [3] (properties attribute). With current config, intermediary tables are created (MESSAGE_HEADER and MESSAGE_PROPERTY) but we would like to get rid of these. a) We tried to use the @OneToMany 'mappedBy' attribute, adding a JPAMessage attribute on the JPAHeader and JPAProperty. The tables are correctly created if we redefine the annotated headers and properties attributes on the concrete JPAMessage [4] class. We had however to define openjpa.InverseManager=true to make it work at runtime. b) When we set the @OneToMany with 'mappedBy' attribute on the AbstractJPAMessage (and the JPAMessage on the JPAHeader and JPAProperty), the tables are created, but not the FK. At runtime, the application generates SQL, but no response seems to come back (no exception in the log). It simply hangs. c) In a) and b) scenario, we need to define an additional JPAMessage concrete attribute on JPAHeader and JPAProperty. This does not allow to define FK to other types, such as the JPAStreamingMessage [5]. In this case, the class structure we built does not make much sense. Would a Custom Mapping help? (http://openjpa.apache.org/builds/latest/docs/manual/manual.html#ref_guide_mapping_custom) d) If we set @ElementJoinColumn(name=MESSAGE_ID, referencedColumnName=ID) on the headers and properties in the AbstractJPAMessage, everything works fine. This also solve problem mentioned in c) because we don't need any JPAMessage attributed on JPAHeader and JPAProperty. However, we try to stick to the standard JPA API without using OpenJPA extensions if not necessary. e) Apart from the above points, are there any standard way to define indexes other than org.apache.openjpa.persistence.jdbc.Index ? We would like to hear your comments on the above. What do you think we should consider/investigate? Are our interpretations correct? Tks, Eric PS: Please reply-to-all so we can follow the thread on James ml. [1] AbstractJPAMessage http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/AbstractJPAMessage.java?view=log [2] JPAHeader http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/JPAHeader.java?view=log [3] JPAProperty http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/JPAProperty.java?view=log [4] JPAMessage http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/JPAMessage.java?view=log [5] JPAStreamingMessage http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/openjpa/JPAStreamingMessage.java?view=log - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OpenJPA usage at Apache James
Hi Pinaki, We use the maven downloaded jar release 422266:935683 from 22-Apr-2010, so we don't have the patch for the MappedSuperClass. I didn't find a recent 2.0.0-SNAPSHOT on http://people.apache.org/maven-snapshot-repository/ nor on https://repository.apache.org/index.html. Will try with 2.1.0-SNAPSHOT. The entites can be found here: [1] AbstractJPAMessage http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/AbstractJPAMessage.java?view=markup [2] JPAHeader http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/JPAHeader.java?view=markup [3] JPAProperty http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/JPAProperty.java?view=markup [4] JPAMessage http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/JPAMessage.java?view=markup [5] JPAStreamingMessage http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/openjpa/JPAStreamingMessage.java?view=markup Tks, Eric On 06/23/2010 07:04 PM, Pinaki Poddar wrote: Hi, The mappedBy mapping on abstract class should work and the relations need not be redefined in the concrete subclass. But a critical bug on MappedSuperclass had been found recently -- which *may be* impacting the mapping. 1. Which OpenJPA revision being used? $ java -jar /path/to/openjpa.jar shows the revision number. 2. If the used revision is prior to 952893 (dated June 8,2010), is it possible to switch to a post-952893 version and see if that resolves the issue? 3. Can you please post the relevant entity classes (JPAMesaage etc.) for further analysis? - Pinaki - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: OpenJPA usage at Apache James
Hi Pinaki, OpenJPA 2.1.0-SNAPSHOT didn't help. I also made a separate sample with a Message, AbstractMessage and Header (a message has many headers). I defined: - @OneToMany(mappedBy...) with type Header on the AbstractMessage - @ManyToOne with type AbstractMessage on the Header. With that config, the FK is not created (nor the corresponding message_id column in Header table). I receive no exception. Another question was the definition of indexes: Does JPA standard provide this ? Tks, Eric On 06/24/2010 05:26 AM, Eric Charles wrote: Hi Pinaki, We use the maven downloaded jar release 422266:935683 from 22-Apr-2010, so we don't have the patch for the MappedSuperClass. I didn't find a recent 2.0.0-SNAPSHOT on http://people.apache.org/maven-snapshot-repository/ nor on https://repository.apache.org/index.html. Will try with 2.1.0-SNAPSHOT. The entites can be found here: [1] AbstractJPAMessage http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/AbstractJPAMessage.java?view=markup [2] JPAHeader http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/JPAHeader.java?view=markup [3] JPAProperty http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/JPAProperty.java?view=markup [4] JPAMessage http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/JPAMessage.java?view=markup [5] JPAStreamingMessage http://svn.apache.org/viewvc/james/imap/trunk/jpa/src/main/java/org/apache/james/imap/jpa/mail/model/openjpa/JPAStreamingMessage.java?view=markup Tks, Eric On 06/23/2010 07:04 PM, Pinaki Poddar wrote: Hi, The mappedBy mapping on abstract class should work and the relations need not be redefined in the concrete subclass. But a critical bug on MappedSuperclass had been found recently -- which *may be* impacting the mapping. 1. Which OpenJPA revision being used? $ java -jar /path/to/openjpa.jar shows the revision number. 2. If the used revision is prior to 952893 (dated June 8,2010), is it possible to switch to a post-952893 version and see if that resolves the issue? 3. Can you please post the relevant entity classes (JPAMesaage etc.) for further analysis? - Pinaki - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
JPA for imap 0.1 release
Hi, Before releasing IMAP, I propose the following for JPA: 1. JAMES ships with openjpa but is open to other JPA providers via the provider/ tag: - org.apache.openjpa.persistence.PersistenceProviderImpl for openjpa (the standard) - org.hibernate.ejb.HibernatePersistence for hibernate - org.eclipse.persistence.jpa.PersistenceProvider for eclipselink 2. We add @ElementJoinColumn(name=MESSAGE_ID) on JPAProperty and JPAHeader in the AbstractJPAMessage (org.apache.openjpa.persistence.jdbc.ElementJoinColumn) to avoid intermediate tables. 3. We implement @Index (org.apache.openjpa.persistence.jdbc.Index) where needed to increase performance. This will be critical for large (and not so large) mailboxes. 4. The majority of the classes will use openjpa classes: instead of moving them all to org.apache.james.imap.jpa.mail.model.openjpa package, we leave them and we rename this package to org.apache.james.imap.jpa.mail.model.streaming to reflect that it goes on streaming the blobs. The definition of an alternate JPA provider should not clash with the proprietary extensions (still need to test it). What's your opinion on this? Tks, Eric - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] countMailboxesWithName
oops, I didn't mean so patch it but so, patches would be good for those kind of behaviour. Still early and missing coffee :) Tks, Eric On 06/25/2010 05:29 AM, Eric Charles wrote: Hi Tim, Same for the domain creation. Currently, you may have duplicates in db, and when you list domains, it hides it via a Set. A strategy is to check if it already exists before writing it. So patch it :) Tks, Eric On 06/24/2010 10:41 PM, Tim-Christian Mundt wrote: Hi, in IMAP the method MailboxMapper.countMailboxesWithName(mailboxName) is only ever used in StoreMailboxManager.mailboxExists(mailboxName, session). There the result is checked for duplicates. Duplicates should be detected before writing and not when reading, for some stores (e.g. maildir) it is not even possible to have more than one mailbox with the same name. So this is superfluous and should should be moved to the MailboxMapper.existsMailbox(mailboxName) which can provide a more efficient existence test. I think I proposed that before but can't find to reason why it's not there. Thus I thought I'd check back with you before I change it. Any objections? Otherwise I'd submit a patch. Cheers Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] countMailboxesWithName
Hi Tim, Same for the domain creation. Currently, you may have duplicates in db, and when you list domains, it hides it via a Set. A strategy is to check if it already exists before writing it. So patch it :) Tks, Eric On 06/24/2010 10:41 PM, Tim-Christian Mundt wrote: Hi, in IMAP the method MailboxMapper.countMailboxesWithName(mailboxName) is only ever used in StoreMailboxManager.mailboxExists(mailboxName, session). There the result is checked for duplicates. Duplicates should be detected before writing and not when reading, for some stores (e.g. maildir) it is not even possible to have more than one mailbox with the same name. So this is superfluous and should should be moved to the MailboxMapper.existsMailbox(mailboxName) which can provide a more efficient existence test. I think I proposed that before but can't find to reason why it's not there. Thus I thought I'd check back with you before I change it. Any objections? Otherwise I'd submit a patch. Cheers Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JPA for imap 0.1 release
Hi Tim, If we set streaming by default, we can not use derby anymore as default. With a different provider, schema may be different. Do you mean we should ensure with specific annotations that schema will always be the same? Tks, Eric On 06/24/2010 10:31 PM, Tim-Christian Mundt wrote: Hi, 4. The majority of the classes will use openjpa classes: instead of moving them all to org.apache.james.imap.jpa.mail.model.openjpa package, we leave them and we rename this package to org.apache.james.imap.jpa.mail.model.streaming to reflect that it goes on streaming the blobs. With OpenJPA you can still choose whether to use streaming, right? Because some databases don't support it. I think we should make streaming the default. The definition of an alternate JPA provider should not clash with the proprietary extensions (still need to test it). However, the usage of another JPA provider would result in a different database scheme, doesn't it? We could add implementations for a the other two providers you mentioned. Best Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JPA for imap 0.1 release
Hi Tim, Config is the price to pay for the many available options. It has also to see with Spring, and not only with JPA. We may begin another thread later on to talk about this. I'm also happy with OpenJPA and using its proprietary annotations (not classes) doesn't prohibit a developer/deployer to define another JPA provider. What about : - @ElementJoinColumn ? - @Index ? - rename 'openjpa' package to 'streaming' ? I will be off for 2 weeks and won't probably be able to continue the conversation. Tks, Eric On 06/25/2010 10:31 AM, Tim-Christian Mundt wrote: Hi Eric, Am Freitag, den 25.06.2010, 05:16 +0200 schrieb Eric Charles: Hi Tim, If we set streaming by default, we can not use derby anymore as default. well, shipping derby is obviously nice for a quick James test, we should leave that as it is. Maybe we can find a better way to configure streaming, the store, the provider. Would be really nice to just have one simple config file and not mess in the spring.xml and persistence.xml - just a side note. With a different provider, schema may be different. Do you mean we should ensure with specific annotations that schema will always be the same? Yes, something like that. The OpenJPA annotations won't work for others and hence they will again produce (or try to use) linker tables. Why would anybody want to use a different provider, by the way? Being generic is cool, but it's not that a user would benefit from that. I think it's more on the programmer's side that we could prefer another provider sometime and in that unlikely case we could easily adapt the current implementation. I'd suggest being content with OpenJPA and not offer any other provider. That saves us a lot of headache. Regards Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JPA for imap 0.1 release
Hi Tim, I think (not sure) the **/openjpa packages have been defined to place all classes that use specific/proprietary functions of openjpa. If that we introduce specific/proprietary annotations such as @Index a bit everywhere, the above logic does not make much sense anymore. For example, the org.apache.james.imap.jpa.mail.model.openjpa contains classes that support streaming to DB blobs with help of org.apache.openjpa.persistence.Persistent annotation. I proposed to rename the package to org.apache.james.imap.jpa.mail.model.streaming to indicate that it goes on stuff that support streaming. If we have another function such as let's say veryfast (stupid example), we would create org.apache.james.imap.jpa.mail.model.veryfast. Now, if we have a veryfast streaming function, we are a bit in the problem. So it my proposal may not make much sens anymore an we may simply leave it as such. But I remain with the (very little) problem that classes not in the openjpa package contain openjpa proprietary extensions. Well, I'm sure those 2 weeks holiday will make things clearer in my head (I will drink a pastis at your health in South of France :). Bye, Eric On 06/25/2010 06:33 PM, Tim-Christian Mundt wrote: Hey, I'm also happy with OpenJPA and using its proprietary annotations (not classes) doesn't prohibit a developer/deployer to define another JPA provider. Right. What about : - @ElementJoinColumn ? - @Index ? I'd support those. - rename 'openjpa' package to 'streaming' ? We already have a streaming package and there is both streaming and non-streaming for OpenJPA, so why rename the package? Maybe I haven't fully understood your point. I will be off for 2 weeks and won't probably be able to continue the conversation. Hope I may say Happy vacations :) Best Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JPA for imap 0.1 release
Hi Tim, So there is consensus to leave the package naming as-is and move entities with openjpa proprietary extension to the openjpa packages. Currently, I have no well defined patch (only many trials I made). I will implement some @ElementJoinColumn and @Index and test it with real traffic. Depending on the result and timing, we may integrate the changes in our upcoming 3.0 M1 release. I will also need to upgrade the current database schema and datas. Probably the number of users that need this migration is very limited (only users running a 3.0 trunk built snapshot). However, we could use it as a base for the latter migrations and also for the 2.3 to 3.0 migration. I will look if an existing JIRA or create a new one to publish the progress. Tks, Eric On 06/26/2010 10:17 AM, Tim-Christian Mundt wrote: Hi Norman and Eric, I fully agree with simply using OpenJPA annotations. Concerning the openjpa package I think I found what you mean, Eric. It was confusing because there are two OpenJPA packages: org/apache/james/imap/jpa/mail/model/openjpa org/apache/james/imap/jpa/openjpa The latter is there merely to support the useStreaming option, if I don't err. The former is also for streaming, so yes, it would make sense to rename it. On the other hand we could move all OpenJPA stuff to /jpa/openjpa which would basically mean to e.g. put the streaming classes into org/apache/james/imap/jpa/openjpa/mail/model/streaming If everything with proprietary OpenJPA annotations would be in a separate package it would become immediate which classes one needs to implement in order to create a new provider. That's my vote: stick with OpenJPA but still cleanly separate it from standards conforming code. Eric, could you send us a patch of what you've done so far? Then we can finish it (hope you still read this before your trip...) Tim Am Freitag, den 25.06.2010, 19:33 +0200 schrieb Norman Maurer: Ok so to come to some consequence here.. Let us just use the openjpa annotation stuff.. If we really want to support other JPA implementations we could handle it later.. Bye, Norman 2010/6/25 Tim-Christian Mundtd...@tim-erwin.de: Hey, I'm also happy with OpenJPA and using its proprietary annotations (not classes) doesn't prohibit a developer/deployer to define another JPA provider. Right. What about : - @ElementJoinColumn ? - @Index ? I'd support those. - rename 'openjpa' package to 'streaming' ? We already have a streaming package and there is both streaming and non-streaming for OpenJPA, so why rename the package? Maybe I haven't fully understood your point. I will be off for 2 weeks and won't probably be able to continue the conversation. Hope I may say Happy vacations :) Best Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [CANCELED?] Re: [VOTE] Release protocols 1.1
[x] +1 Please release the artifacts Eric (no new vote needed on my side) On 07/09/2010 02:33 PM, Manuel Carrasco Moñino wrote: I don't think a new vote is needed. Cheers -Manolo On Fri, Jul 9, 2010 at 8:02 AM, Norman Maurernor...@apache.org wrote: Hi all, unfortunaly a bug with the SSL enabled services just popped up. Do we need a new vote ? It was just a tiny change: http://svn.apache.org/viewvc?view=visionrevision–2422 https://issues.apache.org/jira/browse/JAMES-1025 Bye, Norman 2010/7/7 Norman Maurernor...@apache.org: I will close the VOTE on friday 09.07.2010 09:00 CEST. Thx, Norman 2010/7/7 Manuel Carrasco Moñinoman...@apache.org: +1 Manolo On Tue, Jul 6, 2010 at 11:29 AM, Stefano Bagnaraapa...@bago.org wrote: to be able to release James Server 3.0-M1 we need to cut a release of the protocols subproject, to remove SNAPSHOT dependencies. The protocols release contains some critical bugfixes for smtp-auth and for correctly shut down the server implementation [...] https://repository.apache.org/content/repositories/orgapachejames-033/ [X] +1 Please release the artifacts Stefano - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JPA for imap 0.1 release
Hi Tim, I still need time to find my way in the hard-work you did with Norman these last 2 weeks :) Upon IMAP-168, are there other JIRA that could impact the database schema/data (IMAP-172,...?) ? Tks, Eric On 07/11/2010 03:12 PM, Tim-Christian Mundt wrote: Hi Eric, that sounds good. Let's see, if we can provide a sql-only migration script. After solving issue IMAP-168 the database schema will change again, so we'll have to take care of that, too. Best Tim Am Sonntag, den 11.07.2010, 14:18 +0200 schrieb Eric Charles: Hi Tim, So there is consensus to leave the package naming as-is and move entities with openjpa proprietary extension to the openjpa packages. Currently, I have no well defined patch (only many trials I made). I will implement some @ElementJoinColumn and @Index and test it with real traffic. Depending on the result and timing, we may integrate the changes in our upcoming 3.0 M1 release. I will also need to upgrade the current database schema and datas. Probably the number of users that need this migration is very limited (only users running a 3.0 trunk built snapshot). However, we could use it as a base for the latter migrations and also for the 2.3 to 3.0 migration. I will look if an existing JIRA or create a new one to publish the progress. Tks, Eric On 06/26/2010 10:17 AM, Tim-Christian Mundt wrote: Hi Norman and Eric, I fully agree with simply using OpenJPA annotations. Concerning the openjpa package I think I found what you mean, Eric. It was confusing because there are two OpenJPA packages: org/apache/james/imap/jpa/mail/model/openjpa org/apache/james/imap/jpa/openjpa The latter is there merely to support the useStreaming option, if I don't err. The former is also for streaming, so yes, it would make sense to rename it. On the other hand we could move all OpenJPA stuff to /jpa/openjpa which would basically mean to e.g. put the streaming classes into org/apache/james/imap/jpa/openjpa/mail/model/streaming If everything with proprietary OpenJPA annotations would be in a separate package it would become immediate which classes one needs to implement in order to create a new provider. That's my vote: stick with OpenJPA but still cleanly separate it from standards conforming code. Eric, could you send us a patch of what you've done so far? Then we can finish it (hope you still read this before your trip...) Tim Am Freitag, den 25.06.2010, 19:33 +0200 schrieb Norman Maurer: Ok so to come to some consequence here.. Let us just use the openjpa annotation stuff.. If we really want to support other JPA implementations we could handle it later.. Bye, Norman 2010/6/25 Tim-Christian Mundtd...@tim-erwin.de: Hey, I'm also happy with OpenJPA and using its proprietary annotations (not classes) doesn't prohibit a developer/deployer to define another JPA provider. Right. What about : - @ElementJoinColumn ? - @Index ? I'd support those. - rename 'openjpa' package to 'streaming' ? We already have a streaming package and there is both streaming and non-streaming for OpenJPA, so why rename the package? Maybe I haven't fully understood your point. I will be off for 2 weeks and won't probably be able to continue the conversation. Hope I may say Happy vacations :) Best Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Summer of Code End
Hi Tim, I checked-out, compiled, deployed and tested without any problem. Your MailDir store is working :) and generates the cur, new, tmp in my user directory. Last but not least, your code is really well structured and readable. You really made great job! Btw, we knew that James architecture was great open to alternative store, and you proved it. Quick comments: - I renamed /var/james/Maildirs/%user to ../var/Maildirs/%user (maybe should be file://var/...) to have relative paths to james deployment (more coherent with other stores) - I configured with JDBCDomainList and defined e...@localhost.net user. The created dir is /var/Maildirs/eric/ We may have user clash if let's say e...@otherdomain.net is defined. Tks, Eric On 08/13/2010 10:48 PM, Tim-Christian Mundt wrote: Hi guys, last Monday was pencils down for the Google Summer of Code project. The days till next Monday - the final dead line - are meant as time for wrap up, testing, documentation and so on. Well, the new Maildir backend works. Until now it has just one parameter which is the path (with variables) where the Maildir folders should reside. It works with pre-existing data from other servers or can build up the data from scratch. The functional tests work with the exception of testSearchCombinations* which is due to the fact that Maildir only stores the _current_ date as internal date, not the one that is given during append (which is perfectly valid for the RFC but not for the tests). The code is attached to this mail and you can also find it in my repository: svn://tim-erwin.de/maildir/trunk/maildir Although GSOC is officially over, I'm still open to cc and as soon as the code is in the James SVN it's open to be teared apart anyways. Thanks for having me in this project. Special thanks to Norman who has been a very helpful and nice to work with mentor. I'm sure this will not be the end of my James activities. Regards Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Summer of Code End
btw, this is the first james store not portable on windows. Did you discover any maildir ports on windows during your GSOC ? Can we consider your impl as a Maildir++ one (http://en.wikipedia.org/wiki/Maildir) ? Tks, Eric On 08/14/2010 11:46 AM, Eric Charles wrote: Hi Tim, I checked-out, compiled, deployed and tested without any problem. Your MailDir store is working :) and generates the cur, new, tmp in my user directory. Last but not least, your code is really well structured and readable. You really made great job! Btw, we knew that James architecture was great open to alternative store, and you proved it. Quick comments: - I renamed /var/james/Maildirs/%user to ../var/Maildirs/%user (maybe should be file://var/...) to have relative paths to james deployment (more coherent with other stores) - I configured with JDBCDomainList and defined e...@localhost.net user. The created dir is /var/Maildirs/eric/ We may have user clash if let's say e...@otherdomain.net is defined. Tks, Eric On 08/13/2010 10:48 PM, Tim-Christian Mundt wrote: Hi guys, last Monday was pencils down for the Google Summer of Code project. The days till next Monday - the final dead line - are meant as time for wrap up, testing, documentation and so on. Well, the new Maildir backend works. Until now it has just one parameter which is the path (with variables) where the Maildir folders should reside. It works with pre-existing data from other servers or can build up the data from scratch. The functional tests work with the exception of testSearchCombinations* which is due to the fact that Maildir only stores the _current_ date as internal date, not the one that is given during append (which is perfectly valid for the RFC but not for the tests). The code is attached to this mail and you can also find it in my repository: svn://tim-erwin.de/maildir/trunk/maildir Although GSOC is officially over, I'm still open to cc and as soon as the code is in the James SVN it's open to be teared apart anyways. Thanks for having me in this project. Special thanks to Norman who has been a very helpful and nice to work with mentor. I'm sure this will not be the end of my James activities. Regards Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
XMLVirtualUserTable, JDBCVirtualUserTable and JDBCAlias mailets deprecated
Hi, I'm trying to find my way in james user mngt and came to the conclusion that mappings are now buit-in via the org.apache.james.impl.vut. classes. Although, there are still mailets in trunk that achieve some mapping functions. They are still mentioned of these mailets in spoolmanager.xml !-- This mailet redirects mail for the user 'postmaster' at any local domain to -- !-- the postmaster address specified for the server. The postmaster address -- !-- is required by rfc822. Do not remove this mailet unless you are meeting -- !-- this requirement through other means (e.g. a XML/JDBCVirtualUserTable mailet) -- mailet match=All class=PostmasterAlias/ This kind of mailets may be considered as giving flexibility is the whole processing, but I sometimes feel that they are a left-over implementations of what existed before .vut. package. And yes, sometimes more is less... The JDBC* classes also ask much maintenance and the future for relational database access should be JPA (or any other mapping framework, but not native SQL via JDBC). So should we mark XMLVirtualUserTable, JDBCVirtualUserTable and JDBCAlias as deprecated ? Tks, Eric - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Summer of Code End
Hi, With file:// , I was referring to james nomenclature (ex: file://conf/... for files in james conf dir, file://var/... for files in james var dir), and not to the URI/URL patterns. You can find plenty of this nomenclature in spring-beans.xml,... (ex: file://conf/database.properties) The other James nomenclature is classpath://. We may remain with confusions with the James vs. URL/URI file://, but i think we could discuss a bit later in the frame of OSGI stuff,... ../var/... can perfectly do the job now. Nice to read we can use %domain and %user variables. May be the default in spring-beans.xml could be ../var/Maildir/%domain/%user/Maildir or whatever, so we know we can play with domain / user structure. Tks, Eric On 16/08/2010 08:26, Norman Maurer wrote: Hi, comments inside.. 2010/8/15 Tim-Christian Mundtd...@tim-erwin.de: Hi Eric, thanks a lot for checking out and for your kind comments. James' Store-Architecture really makes it relatively easy (really relative) to implement new Storages which is definitely a big plus for James. Changing the path is a good idea. However, introducing the scheme (file://) would make the configuration more complicated and suggest that other protocols would also work, right?. If its still better I can add it, shouldn't be a big thing. I think we should not add the scheme stuff. For JAMES Server we just need to subclass it and use the FileSystem interface for lookup the directory. No big deal ;) The path is not really well documented, it needs to be in the config. It can have three different variables: %user, %domain, %fulluser. You would either use something like /Maildirs/%domain/%user/ or /Maildirs/% fulluser When thinking of vpopmail/courier-imap it use: ../%domain/%user/Maildir/ I think we should do the same.. Regards Tim Am Samstag, den 14.08.2010, 11:46 +0200 schrieb Eric Charles: Hi Tim, I checked-out, compiled, deployed and tested without any problem. Your MailDir store is working :) and generates the cur, new, tmp in my user directory. Last but not least, your code is really well structured and readable. You really made great job! Btw, we knew that James architecture was great open to alternative store, and you proved it. Quick comments: - I renamed /var/james/Maildirs/%user to ../var/Maildirs/%user (maybe should be file://var/...) to have relative paths to james deployment (more coherent with other stores) - I configured with JDBCDomainList and defined e...@localhost.net user. The created dir is /var/Maildirs/eric/ We may have user clash if let's say e...@otherdomain.net is defined. Tks, Eric On 08/13/2010 10:48 PM, Tim-Christian Mundt wrote: Hi guys, last Monday was pencils down for the Google Summer of Code project. The days till next Monday - the final dead line - are meant as time for wrap up, testing, documentation and so on. Well, the new Maildir backend works. Until now it has just one parameter which is the path (with variables) where the Maildir folders should reside. It works with pre-existing data from other servers or can build up the data from scratch. The functional tests work with the exception of testSearchCombinations* which is due to the fact that Maildir only stores the _current_ date as internal date, not the one that is given during append (which is perfectly valid for the RFC but not for the tests). The code is attached to this mail and you can also find it in my repository: svn://tim-erwin.de/maildir/trunk/maildir Although GSOC is officially over, I'm still open to cc and as soon as the code is in the James SVN it's open to be teared apart anyways. Thanks for having me in this project. Special thanks to Norman who has been a very helpful and nice to work with mentor. I'm sure this will not be the end of my James activities. Regards Tim - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org Bye, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Proposed store api change (NEXTUID)
... and one more general point :Should the UID API oblige (or not?) each store to use the UIDNEXT from the memory ? (or Should the caching UIDNEXT mecanism be an optional utility, meaning that each store could re-implement it in a different way - cached, persistent or not ?). Tks, Eric On 17/08/2010 17:56, Eric Charles wrote: Hi Norman, I've read the http://www.rfc-editor.org/rfc/rfc3501.txt (section 2.3.1 Message Numbers) and http://www.rfc-editor.org/rfc/rfc2683.txt (section 3.4.3. UIDs and UIDVALIDITY) A first point is RFC talks about backend server not being able to store the UIDs. In this case, the UID are to be regenerated each time, with a different UIDVALIDITY, so a there are no risk to confuse mails. I had also to read twice the sentence and associated explanation It seems to be a common misunderstanding that the UIDVALIDITY and the UID, taken together, form a 64-bit identifier that uniquely identifies a message on a server . However, it is said at another place : The combination of mailbox name, UIDVALIDITY, and UID must refer to a single immutable message on that server forever. Finally, this may give as requirement that the store API should not prevent to implement a store that wouldn't be capable of storing UIDs (seems strange, but considered at numerous places in RFCs). I think the current store API already allows that ? On the UIDVALIDITY, it is now generated for example in JPA with a Math.abs(RANDOM.nextInt()). RFC states: A good UIDVALIDITY value to use in this case is a 32-bit representation of the creation date/time of the mailbox. It seems reasonable to provide utility methods such as existing randomUidValidity() to the store impl, each store having the freedom to use it or not. (was just wondering what is the difference between the imap-mailbox and imap-store projects - not always obvious at first sight to define the responsibilities of each). Coming to the UIDNEXT and as you pointed, I also understand that the returned UIDNEXT value has nothing to do with the UID that will be given to the next coming message. That value needs however to be equals or higher. I suppose the idea would be to have per mailbox a cache in memory. That cache would be used to return the UIDNEXT (that would be the current cache value), but also to assign the UID for coming mails (cache+1). I am wondering how we can ensure in case of abrupt shutdown that the last value of the cache be stored. If we can't ensure that, and this will be probably the case, there we an have a strategy to init the cache with a value recomputed from the all the stored UID (something like give me the highest value from all the UID of that mailbox). This would need an initial step when the cache is not initialized but should not be a penality for a JPA store, even for mailbox with many mails. Don't know for the other stores (jcr, maildir,...) ? Should we care on the cache time-to-live? If we don't care about that, we will have a growing memory, even if for each mailbox, we only need an Integer (that wouldn't represent much KB). But there is also the possibility to define a ttl of a few hours, with a scheduled cache manager that would cleanup things. Tks, Eric On 15/08/2010 17:25, Norman Maurer wrote: Hi there, After looking a bit over the store api again the last days I think there is some room for improvements. This improvements will break the api (again), so I think we should do it now and after that cut the 0.1 release. I will try to explain you why I think there should be some improvements made and whats my point of view. Please feel free to comment .. NEXTUID (IMAP-193): The NEXTUID generation / house-keeping is just a big performance killer. We really guaranteer to use the value of NEXTUID for the next message which will get saved. Thats not needed. We just need to guaranteer its equal or greater then the value returned by NEXTUID. So its prolly more performant to just hold the informations in memory and update it every x writes (or something like that). So the implementation could use an auto-increment field to generate the unique uid when storing the message or just an AtomicInteger for generation. Maybe again with a new abstract class called UIDKeeper ? Does this sound like something which make sense ? - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[IMAP] Projects' structure
Hi Everybody After splitting the tests from functional to their store projects (IMAP-187), we could centralize the test infrastructure present in different projects: - apache-james-imap-deployment - apache-james-imap-protocol-tester All going to apache-james-imap-seda apache-james-imap-decoder shouldn't apache-james-imap-decode be fusioned with apache-james-imap-message which already contains the encode stuff ? This does not bring real direct value, but we'll have less projects (easier for a newbie and to talk about it). Any thoughts ? Tks, Eric - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] store api changes
Hi Norman, I applied the last uploaded file (proposal_v3.diff): - patch works, which is good :) - I don't see any change on MessageMapper. Probably the modif to return only uid was already committed (IMAP-203,...) ? - You migrated methods from StoreMessageManager to AbstractMessageMapper. So now, I'm a bit confused about the class responsiblity. Could you summary us the roles/responsibilities of these 2 classes. That will help to define which methods goes where... - Abstract/extend is widely used in james. I sometimes force myself to thin to favor composition over inheritance. Did you consider composition instead of AbstractMessageMapper ? (maybe stupid question and I should have tried to implement before asking... :) Of course, you can find plenty of discussions and explanations (http://www.google.com/search?q=favor+composition+over+inheritance) on the net - I like the new UpdatedFlag class. When you read it, you understand what it is aimed for. Separating data domain from service domain is always good. Tks, Eric On 08/19/2010 12:20 PM, Norman Maurer wrote: Hi there, I' currently looking for ways to make it easier to create high performant Mailbox implementations while reusing the store api. I think there are some things we should change to allow this. * We should return only the uid of the message if nothing more is needed * We should move some stuff to an abstract MessageMapper implementation to make it easier to override and handle it different and still be able to use the store api. For this I opened a jira and attached a patch whic hshows the idea.. I'm still not 100% sure if its really a good idea, so feedback is welcome :) https://issues.apache.org/jira/browse/IMAP-202 Bye, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] store api changes
OK So AbstractStoreMessageManager doesn't implement MessageMapper (and it was in your javadoc :) ... and implements Mailbox (the org.apache.james.imap.mailbox.Mailbox, not the org.apache.james.imap.store.mail.model.Mailbox...) The org.apache.james.imap.mailbox.Mailbox has much too do with MaiboxSession and not with domain classes, so it respects your goal. Maybe the domain independence could be also documented in the javadoc. Talking about domain model, I'm sometimes a bit confused. We have a nice package org.apache.james.imap.store.mail.model that contains the model, but my 2-cents: - There are a few utils classes (PropertyBuilder,...) that could be moved somewhere else. - Document could be renamed to Message (I think we talked about it some time ago) - When you read code, Mailbox,... is sometimes confusing because it exists somewhere else, for example in org.apache.james.imap.mailbox.Mailbox. This last one has more to do with service than with domain. I don't have a good name right now, but Maiboxable seems the closer to what I'm looking for. Hope I'm not too niggling, Tks, Eric On 20/08/2010 07:36, Norman Maurer wrote: Let me try to explain it The idea is to let the MessageMapper untouched, because its API is really easy to understand and use. Its not the most Performant todo but thats the price to pay. So I introduced a new abstract class which can be used if you want to write a custom store which is not forced to use a MessageMapper and so is a good fit for other implementations which not works so well with the Domain model and lazy loading (nosql for example). Hope this explains it... Thx Norman 2010/8/20, Eric Charlese...@apache.org: Hi Norman, I applied the last uploaded file (proposal_v3.diff): - patch works, which is good :) - I don't see any change on MessageMapper. Probably the modif to return only uid was already committed (IMAP-203,...) ? - You migrated methods from StoreMessageManager to AbstractMessageMapper. So now, I'm a bit confused about the class responsiblity. Could you summary us the roles/responsibilities of these 2 classes. That will help to define which methods goes where... - Abstract/extend is widely used in james. I sometimes force myself to thin to favor composition over inheritance. Did you consider composition instead of AbstractMessageMapper ? (maybe stupid question and I should have tried to implement before asking... :) Of course, you can find plenty of discussions and explanations (http://www.google.com/search?q=favor+composition+over+inheritance) on the net - I like the new UpdatedFlag class. When you read it, you understand what it is aimed for. Separating data domain from service domain is always good. Tks, Eric On 08/19/2010 12:20 PM, Norman Maurer wrote: Hi there, I' currently looking for ways to make it easier to create high performant Mailbox implementations while reusing the store api. I think there are some things we should change to allow this. * We should return only the uid of the message if nothing more is needed * We should move some stuff to an abstract MessageMapper implementation to make it easier to override and handle it different and still be able to use the store api. For this I opened a jira and attached a patch whic hshows the idea.. I'm still not 100% sure if its really a good idea, so feedback is welcome :) https://issues.apache.org/jira/browse/IMAP-202 Bye, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] store api changes
Hi Norman, The following hierarchy is obvious and gives services such as create, login, logout, search on *mailboxes*: MailboxManager (interface) /|\ | DelegatingMailboxManagerimpl (abstract) /|\ | StoreMailboxManager (abstract) /|\ | JPA/JCR/Maildir/...MailboxManager (concrete) This one gives services to append, set flags', expunge on *messages*: Mailbox (interface) /|\ | AbstractStoreMessageManagerId (abstract) /|\ | StoreMessageManagerId (abstract) /|\ | JPA/JCR/Maildir/...MessageManager (concrete) Wouldn't that Mailbox interface finally be named MessageManager ? But np for me to live with the existing names. Will StoreMessageManager remain abstract (it is in the patch) ? (if yes, we will end up with a AbstractStoreMessageManager and a StoreMessageManager, both abstract,...). I suppose the criteria to implement in AbstractStoreMessageManager was 'no need for messageMapper'. Is there a common characteristic to these methods (so we can QualifyStoreMessageManager) ? If I understand well, store implementors could reuse or simply override methods of *StoreMessageManager. I was also talking about composition, so a store implementator would know directly which methods he should implement to take control on a certain behaviour aspect. With the inheritance mecanism, the implementator has to go deep in the class hierarchy to know it. Would you rename Document to Message in IMAP-202 ? (btw, in my first reply, I confused somewhere ...Mapper with ...Manager, sorry) Tks, Eric On 20/08/2010 12:07, Norman Maurer wrote: Comments inline.. 2010/8/20 Eric Charlese...@apache.org: OK So AbstractStoreMessageManager doesn't implement MessageMapper (and it was in your javadoc :) ... and implements Mailbox (the org.apache.james.imap.mailbox.Mailbox, not the org.apache.james.imap.store.mail.model.Mailbox...) yep .. The org.apache.james.imap.mailbox.Mailbox has much too do with MaiboxSession and not with domain classes, so it respects your goal. Maybe the domain independence could be also documented in the javadoc. Fair enough.. Talking about domain model, I'm sometimes a bit confused. We have a nice package org.apache.james.imap.store.mail.model that contains the model, but my 2-cents: - There are a few utils classes (PropertyBuilder,...) that could be moved somewhere else. I see no value here.. - Document could be renamed to Message (I think we talked about it some time ago) +1 - When you read code, Mailbox,... is sometimes confusing because it exists somewhere else, for example in org.apache.james.imap.mailbox.Mailbox. This last one has more to do with service than with domain. I don't have a good name right now, but Maiboxable seems the closer to what I'm looking for. Thats why its still called Mailbox ;) Hope I'm not too niggling, Tks, Eric On 20/08/2010 07:36, Norman Maurer wrote: Let me try to explain it The idea is to let the MessageMapper untouched, because its API is really easy to understand and use. Its not the most Performant todo but thats the price to pay. So I introduced a new abstract class which can be used if you want to write a custom store which is not forced to use a MessageMapper and so is a good fit for other implementations which not works so well with the Domain model and lazy loading (nosql for example). Hope this explains it... Thx Norman 2010/8/20, Eric Charlese...@apache.org: Hi Norman, I applied the last uploaded file (proposal_v3.diff): - patch works, which is good :) - I don't see any change on MessageMapper. Probably the modif to return only uid was already committed (IMAP-203,...) ? - You migrated methods from StoreMessageManager to AbstractMessageMapper. So now, I'm a bit confused about the class responsiblity. Could you summary us the roles/responsibilities of these 2 classes. That will help to define which methods goes where... - Abstract/extend is widely used in james. I sometimes force myself to thin to favor composition over inheritance. Did you consider composition instead of AbstractMessageMapper ? (maybe stupid question and I should have tried to implement before asking... :) Of course, you can find plenty of discussions and explanations (http://www.google.com/search?q=favor+composition+over+inheritance) on the net - I like the new UpdatedFlag class. When you read it, you understand what it is aimed for. Separating data domain from service domain is always good. Tks, Eric On 08/19/2010 12:20 PM, Norman Maurer wrote: Hi there, I' currently looking for ways to make it easier to create high performant Mailbox implementations while reusing the store api. I think there are some things we should change to allow this. * We should return only the uid of the message if nothing more is needed * We should move some stuff to an abstract MessageMapper
Re: [IMAP] store api changes
Hi All, Comments inside. Tks, Eric On 08/20/2010 04:53 PM, Tim-Christian Mundt wrote: Hi guys, I began looking into the API stuff. I'd like to simplify the Mapper hierarchy, which however will have no direct impact on the API. Will upload a patch tomorrow. Will look at it :) I agree, we should name the interface MessageManager, that's what I thought. +1 The proposal_v3 is reasonable, I think. To make it clearer, we should maybe call AbstractStoreMessageManager StoreMessageManager and qualify StoreMessageManager to MapperStoreMessageManager. I think, that's what Eric also proposed. Sounds good (loosely proposal : MappingStoreMessageManager instead of MapperStoreMessageManager :) Concerning composition: I think the composition part in the Store stuff is the mappers which get plugged into the Managers. The only reason so far for extending Store*Manager is the creation of certain objects. Nonetheless, you are right, Eric, there are quite extensive hierarchies in James and maybe we can simplify them. However, inheritance here is not used to change behavior but to model different stages of abstraction which is not the matter of composition. Where/How do you think we should introduce composition? I'm talking about composition in the sense of the Gof Strategy pattern. I think we can say that James has different store strategies. I also can be in line with you when you say the behaviour is not really changed (we always talk about storing stuff). However, fine-grained behaviour is quite different depending on the store (jpa/jcr/file/nosql such as cassandra). Another argument for composition in that underlying api can change without disturbing the existing 3rd party store impl (think what will happen if you rename a method in the abstract class). The enormeous advantage of composition is that the interface of the composed class is the published contract and should not change (between minor releases). So you can change anything in the abstract without breaking 3rd party implementations. But I'd better take my IDE and to create a patch with a composed API However, I've got a number of waiting JIRAs, and if you don't see anything from me soon, let's not wait to implement this refactoring in the inheritance way. We can also come on this later on if appropriate. Regards Tim Am Freitag, den 20.08.2010, 14:17 +0200 schrieb Eric Charles: Hi Norman, The following hierarchy is obvious and gives services such as create, login, logout, search on *mailboxes*: MailboxManager (interface) /|\ | DelegatingMailboxManagerimpl (abstract) /|\ | StoreMailboxManager (abstract) /|\ | JPA/JCR/Maildir/...MailboxManager (concrete) This one gives services to append, set flags', expunge on *messages*: Mailbox (interface) /|\ | AbstractStoreMessageManagerId (abstract) /|\ | StoreMessageManagerId (abstract) /|\ | JPA/JCR/Maildir/...MessageManager (concrete) Wouldn't that Mailbox interface finally be named MessageManager ? But np for me to live with the existing names. Will StoreMessageManager remain abstract (it is in the patch) ? (if yes, we will end up with a AbstractStoreMessageManager and a StoreMessageManager, both abstract,...). I suppose the criteria to implement in AbstractStoreMessageManager was 'no need for messageMapper'. Is there a common characteristic to these methods (so we can QualifyStoreMessageManager) ? If I understand well, store implementors could reuse or simply override methods of *StoreMessageManager. I was also talking about composition, so a store implementator would know directly which methods he should implement to take control on a certain behaviour aspect. With the inheritance mecanism, the implementator has to go deep in the class hierarchy to know it. Would you rename Document to Message in IMAP-202 ? (btw, in my first reply, I confused somewhere ...Mapper with ...Manager, sorry) Tks, Eric On 20/08/2010 12:07, Norman Maurer wrote: Comments inline.. 2010/8/20 Eric Charlese...@apache.org: OK So AbstractStoreMessageManager doesn't implement MessageMapper (and it was in your javadoc :) ... and implements Mailbox (the org.apache.james.imap.mailbox.Mailbox, not the org.apache.james.imap.store.mail.model.Mailbox...) yep .. The org.apache.james.imap.mailbox.Mailbox has much too do with MaiboxSession and not with domain classes, so it respects your goal. Maybe the domain independence could be also documented in the javadoc. Fair enough.. Talking about domain model, I'm sometimes a bit confused. We have a nice package org.apache.james.imap.store.mail.model that contains the model, but my 2-cents: - There are a few utils classes (PropertyBuilder,...) that could be moved somewhere else. I see no value here.. - Document could be renamed to Message
Re: [IMAP] Remove org.apache.james.imap.store.Subscriper
Hi, So Tim, you was also ok with optin 2 and finally Norman reverted to option 1 ? (not sure to be in line :) ?) I've looked at revision 987821: - SubscriptionManager is not used anymore in MailboxManager hirerachy (StoreMailboxManager,...) - The 3 delegate methods present in DelegatingMailboxManager are removed (so, yes, DelegatingMailboxManager is not delegating anymore, and we could rename it :), or merge with StoreMailboxManager - The ImapProcess is now responsible for the SubscriptionManager I suppose the spring-beans.xml is not committed according to these changes (imapProcessor and mailboxmanager constructors should be updated) I like the idea to leave an interface thats clearly describe the Subscription methods. Is there a particular reason to have now the DefaultImapProcessorFactory responsible for the SubscriptionManager ? I can see it's // with the MailboxManager. NioImapServer finally uses the imapProcessor bean. btw, if we talk Gof patterns, we went from a Strategy pattern to a Chain of responsibility pattern. So the question I'm wondering is is it the responsibility of the the NioImapServer to process the mailbox, then the subscription (it does it via the DefaultProcessorChain). I remain in favour of strategy or chain, rather then inheritance. Tim, are the mapper considerations your're talking here related to IMAP-206 ? (not sure to understand the relations, maybe a separate thread would help) Tks, Eric On 21/08/2010 22:53, Tim-Christian Mundt wrote: Norman, I have not yet reviewed your recent commit. However, I'd like to remark that I was in favor of your last attempt. I just wanted to _additionally_: I think we should also merge them which would also simplify the package structure because we wouldn't need the .mail and .user packages anymore. Which means, not only merge the SubscriptionManager into the MailboxManager like you did, but also the SubscriptionMapper into the MailboxMapper. That would make the separation between .user and .mail superfluous. Does that make sense? Maybe your current attempt is still better, I haven't checked yet. Best Tim Am Samstag, den 21.08.2010, 22:08 +0200 schrieb Norman Maurer: Ok another attempt was made.. please review changes made in revision 987821 Thx, Norman 2010/8/21 Norman Maurernor...@apache.org: Well I need to revert it ;) I will do so then.. Bye, Norman 2010/8/21 Norman Maurernor...@apache.org: Hi Tim, comments inside.. 2010/8/21 Tim-Christian Mundtd...@tim-erwin.de: Norman, you are right in that it was kinda double, so there should be some cleanup. My first attempt would have been to remove the subscription stuff from the MailboxManager (your option #1). The reason is that we always have a manager and its respective mapper. Now we have the MailboxManager with two Mappers. I think we should also merge them which would also simplify the package structure because we wouldn't need the .mail and .user packages anymore. Thats true I just thought it would be more easy to have not to many interfaces to implement. Anyway I would also be happy to move the subscripe stuff to any extra interface. I just don't like to have it duplicated so feel free to revert... One more thing concerning naming and stuff: Now the DelegatingMailboxManager is not really delegating anymore. Is there any good reason we should keep it separate from StoreMailboxManager? If not I'd rather have a little bigger class but fewer hierarchy levels. I need to review.. Any thoughts? Tim Am Samstag, den 21.08.2010, 10:23 +0200 schrieb Norman Maurer: I just committed the changes.. If anyone thinks its a bad idea we can revert it anyway.. https://issues.apache.org/jira/browse/IMAP-205 Bye, Norman 2010/8/21 Norman Maurernor...@apache.org: Hi there, after looking again at the IMAP api I'm in favor of removing the org.apache.james.imap.store.Subscriper interface and merge the implementations with the MailboxManager implementations. Thats because the Subsciper interface has 3 methods, all of the methods are already in MailboxManager. So the MailboxManager just wraps the Subscriper implementation and delegate the call to it. So there are two solutions to this: 1) Remove the methods from MailboxManager and move the Subscriper interface to the mailbox api 2) Remove the Subscriper interface from store api and merge the implementations As I stated before I would prefer 2). WDYT ? Bye, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org Bye, Norman - To unsubscribe, e-mail:
Re: [IMAP] Remove org.apache.james.imap.store.Subscriper
Back again. In DefaultProcessorChain, we know have createDefaultChain(chainEndProcessor, mailboxManager, subscriptionManager, statusResponseFactory) so mailbox and subscription can be seen on the same level, each playing role in the different processors. There are now 3 options: 1.- composition/stragety was the initial situation (before the pach) 2.- inheritance was the intermediate commit 3.- chain or responsibility (in the processor) is the current commit. I like the 3rd one. Tks, Eric On 22/08/2010 08:53, Eric Charles wrote: Hi, So Tim, you was also ok with optin 2 and finally Norman reverted to option 1 ? (not sure to be in line :) ?) I've looked at revision 987821: - SubscriptionManager is not used anymore in MailboxManager hirerachy (StoreMailboxManager,...) - The 3 delegate methods present in DelegatingMailboxManager are removed (so, yes, DelegatingMailboxManager is not delegating anymore, and we could rename it :), or merge with StoreMailboxManager - The ImapProcess is now responsible for the SubscriptionManager I suppose the spring-beans.xml is not committed according to these changes (imapProcessor and mailboxmanager constructors should be updated) I like the idea to leave an interface thats clearly describe the Subscription methods. Is there a particular reason to have now the DefaultImapProcessorFactory responsible for the SubscriptionManager ? I can see it's // with the MailboxManager. NioImapServer finally uses the imapProcessor bean. btw, if we talk Gof patterns, we went from a Strategy pattern to a Chain of responsibility pattern. So the question I'm wondering is is it the responsibility of the the NioImapServer to process the mailbox, then the subscription (it does it via the DefaultProcessorChain). I remain in favour of strategy or chain, rather then inheritance. Tim, are the mapper considerations your're talking here related to IMAP-206 ? (not sure to understand the relations, maybe a separate thread would help) Tks, Eric On 21/08/2010 22:53, Tim-Christian Mundt wrote: Norman, I have not yet reviewed your recent commit. However, I'd like to remark that I was in favor of your last attempt. I just wanted to _additionally_: I think we should also merge them which would also simplify the package structure because we wouldn't need the .mail and .user packages anymore. Which means, not only merge the SubscriptionManager into the MailboxManager like you did, but also the SubscriptionMapper into the MailboxMapper. That would make the separation between .user and .mail superfluous. Does that make sense? Maybe your current attempt is still better, I haven't checked yet. Best Tim Am Samstag, den 21.08.2010, 22:08 +0200 schrieb Norman Maurer: Ok another attempt was made.. please review changes made in revision 987821 Thx, Norman 2010/8/21 Norman Maurernor...@apache.org: Well I need to revert it ;) I will do so then.. Bye, Norman 2010/8/21 Norman Maurernor...@apache.org: Hi Tim, comments inside.. 2010/8/21 Tim-Christian Mundtd...@tim-erwin.de: Norman, you are right in that it was kinda double, so there should be some cleanup. My first attempt would have been to remove the subscription stuff from the MailboxManager (your option #1). The reason is that we always have a manager and its respective mapper. Now we have the MailboxManager with two Mappers. I think we should also merge them which would also simplify the package structure because we wouldn't need the .mail and .user packages anymore. Thats true I just thought it would be more easy to have not to many interfaces to implement. Anyway I would also be happy to move the subscripe stuff to any extra interface. I just don't like to have it duplicated so feel free to revert... One more thing concerning naming and stuff: Now the DelegatingMailboxManager is not really delegating anymore. Is there any good reason we should keep it separate from StoreMailboxManager? If not I'd rather have a little bigger class but fewer hierarchy levels. I need to review.. Any thoughts? Tim Am Samstag, den 21.08.2010, 10:23 +0200 schrieb Norman Maurer: I just committed the changes.. If anyone thinks its a bad idea we can revert it anyway.. https://issues.apache.org/jira/browse/IMAP-205 Bye, Norman 2010/8/21 Norman Maurernor...@apache.org: Hi there, after looking again at the IMAP api I'm in favor of removing the org.apache.james.imap.store.Subscriper interface and merge the implementations with the MailboxManager implementations. Thats because the Subsciper interface has 3 methods, all of the methods are already in MailboxManager. So the MailboxManager just wraps the Subscriper implementation and delegate the call to it. So there are two solutions to this: 1) Remove the methods from MailboxManager and move the Subscriper interface to the mailbox api 2) Remove the Subscriper interface from store api and merge the implementations As I stated before I would prefer 2). WDYT ? Bye
Re: [IMAP] Remove org.apache.james.imap.store.Subscriper
See comments after. Eric On 22/08/2010 10:12, Norman Maurer wrote: Which means, not only merge the SubscriptionManager into the MailboxManager like you did, but also the SubscriptionMapper into the MailboxMapper. That would make the separation between .user and .mail superfluous. Does that make sense? Maybe your current attempt is still better, I haven't checked yet. Yes this makes sense too.. The question is do we want to have an extra interface for subscriptions or not ? The problem with the current patch is that its not really optimal how the startProcessing... stopProcessing is used.. Because the SubscriptionManager does not have such methods and so it can lead to leaking resources if the methods in MailboxManager is not called.. Thats a problem cause it would be possible to use the SubscriptionManager alone.. If we go backto option 1) we don't have this problems.. So I'm still not sure what solution is the best .. DefaultProcessorChain is responsible to invoke the MailboxManager and the SubscriptionManager. So the behaviour of both is controlled by an external compontent, the DefaultProcessorChain (they are coordinated/controlled by something). But it's true that SubscriptionManager could be used independently of MailboxManager, and one depending on the other, this could produce unwanted effects (didn't look which one...). An important point is the following: Subscription are links between JamesUser and Mailbox : right ? We are in a mind to let the James Admin user configure the way he want to persist its mails, its users,.. Let's take the example of JCR mailbox store with a JPA user store. Where would the subscription be persisted : JCR or JPA ? And why the user wouldn't be allowed to persist its subscriptions in (I take any example) : another DB, a LDAP, a noSQL store,... If we think we should give freedom and tools to 3rd party to persist where they want the different James domain aspects, leaving the aspects separated in different is not a bad thing. If we think that a Subscription management is on the same level as Mailbox Management (seen from here, it seems), we could also add the start/stop processing and addListerner methods on it (it can always be useful to be notified of a subscription). - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: [IMAP] Remove org.apache.james.imap.store.Subscriper
one more inline :) On 22/08/2010 13:58, Norman Maurer wrote: Not 100%.. Subscriptions links users to Mailboxes. The users are just Strings and nothing more. So its not coupled to james server in any way. Yeah, these are soft links. A Subscription store could be different from the Mailbox store (so we have links via String). Example: - Users are stored in LDAP tks to james-server. - Mailbox in DB via JPA. - Subscription soft link LDAP to DB and these soft links are store anywhere defined by the admin (in spring-beans.xml). (snippped...) Just as a side-note... For example the subscription stuff is not really necessary for using MailboxManager in non-imap enviroments (for example pop3). True enough. I re-read the pop3 case, and I see one more argument to leave the SubscriptionManager independent of the MailboxManager. So http://svn.apache.org/viewvc?rev=987821view=rev Move the Subscriper to the mailbox api and rename it SubscriptionManager... makes all its sense. There's still the open question on start/stopProcessing and listener (to be duplicated in Mailbox/SubscriptionManager or externalized ?). To take a decision , we should also consider that operations need to be atomic, so encapsulate the Mailbox/SubscriptionManager processing. So externalize/encapsulate makes some sense, as you communicated me ? - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
