Raju's stack dump indicated that he was running HotSpot 14.0 Client mode. If possible try running on a more recent Oracle JDK VM in Server mode.
If that's not an option try adding -XX:+UseMembar to your startup.sh and restart James. http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6822370 Cheers, Simon On Sat, Sep 17, 2011 at 8:40 AM, Zach Cox <[email protected]> wrote: > After restarting James several times, 1) all mails continued to be > spooled, and 2) not all mails previously spooled were actually sent > out after restart. Thus, I had to run a new James server and point > DNS to it. > > But I did not terminate the broken server, so it is still there for > debugging purposes. > > Please let me know what information I can pull off of that broken > James server to help find a resolution to this severe problem. > > Thanks, > Zach > > > > On Fri, Sep 16, 2011 at 4:56 PM, Zach Cox <[email protected]> wrote: >> James is currently spooling all of our incoming email and not >> processing it. I took 4 jstack dumps. Anything else I can look at >> quickly before I have to restart James? >> >> >> >> On Fri, Sep 16, 2011 at 6:58 AM, Zach Cox <[email protected]> wrote: >>> mailetcontainer.xml has: >>> >>> <spooler> >>> <threads> 20 </threads> >>> </spooler> >>> >>> After running all night, there are still only 4 blocked Camel threads, >>> and for the most part James appears to be processing all emails >>> properly. I will do some more testing this morning, and try to force a >>> bunch of these HTTP requests and see if I can lock it up. >>> >>> Just taking another look at our stack trace. In this mailet, we use >>> DefaultHttpClient with ThreadSafeClientConnManager. I know this is not >>> the Apache HttpClient mailing list, but does this mean it's not >>> blocking on a request with no response, but it's blocking just trying >>> to get a connection from the connection pool? >>> >>> java.lang.Thread.State: WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <0x7aff2ee0> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) >>> at >>> org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158) >>> at >>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:403) >>> at >>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) >>> at >>> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) >>> at >>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401) >>> at >>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) >>> at >>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) >>> at >>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) >>> >>> That stack trace looks almost identical up towards the top as all of >>> the other "normal" Camel threads like this one: >>> >>> "Camel Thread 72 - Split" prio=10 tid=0x0a1ebc00 nid=0x10b5 waiting on >>> condition [0xb0e7a000] >>> java.lang.Thread.State: WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <0x7a6a2ca0> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) >>> at >>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386) >>> at >>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:679) >>> >>> I just want to make sure I understand how exactly this mailet is blocking. >>> >>> Thanks, >>> Zach >>> >>> >>> >>> >>> >>> >>> On Fri, Sep 16, 2011 at 12:16 AM, Norman Maurer >>> <[email protected]> wrote: >>>> Hi there, >>>> >>>> how many spooling threads did you specifiy in mailetcontainer.xml ? >>>> >>>> About the split threads... If a Matcher "split" and email into parts >>>> Camel will process this new parts in seperate threads. And no, 4 >>>> blocked threads should not block all the others. >>>> >>>> Bye, >>>> Norman >>>> >>>> >>>> 2011/9/16 Zach Cox <[email protected]>: >>>>> I just noticed James starting to spool some emails and did some >>>>> jstacks. I found several threads like this: >>>>> >>>>> "Camel Thread 65 - Split" prio=10 tid=0x0a1c1400 nid=0x10ae waiting on >>>>> condition [0xb105f000] >>>>> java.lang.Thread.State: WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x7aff2ee0> (a >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) >>>>> at >>>>> org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158) >>>>> at >>>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:403) >>>>> at >>>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) >>>>> at >>>>> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) >>>>> at >>>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401) >>>>> at >>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) >>>>> at >>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) >>>>> at >>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) >>>>> <rest is redacted> >>>>> >>>>> >>>>> One of our custom mailets makes an HTTP request to a 3rd party site. >>>>> Usually this works fine, but apparently sometimes this other site does >>>>> not send back an HTTP response, keeps the request open forever, and >>>>> the Apache HttpClient is not doing any kind of timeouts, so this >>>>> request stays open and seems to block a thread forever. >>>>> >>>>> What exactly do these 200 Camel Threads do? Currently we have 4 Camel >>>>> threads blocked like this. But I wouldn't think 4 out of 200 would >>>>> cause James to stop processing all incoming emails? James is still >>>>> processing some emails, but stuck on others. We will let James run all >>>>> night, and see if eventually all 200 get blocked. >>>>> >>>>> Anyways, wanted to share this. In the morning we will add timeouts to >>>>> these HTTP requests (as well as any others we do) and see if that >>>>> helps things. >>>>> >>>>> Thanks, >>>>> Zach >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Sep 15, 2011 at 6:27 PM, Raju Buchi <[email protected]> >>>>> wrote: >>>>>> I guess the attachment was sent over. I have created a Bug report. >>>>>> >>>>>> https://issues.apache.org/jira/browse/JAMES-1317 >>>>>> >>>>>> - Raju >>>>>> >>>>>> On Thu, Sep 15, 2011 at 6:12 PM, Raju Buchi >>>>>> <[email protected]> wrote: >>>>>>> Hi >>>>>>> >>>>>>> I have got the issue again today where the mails are stuck in spool >>>>>>> and all the mails sent to the server will just say >>>>>>> INFO 19:02:25,581 | james.smtpserver | ID=17053510 Successfully >>>>>>> spooled mail Mail1316127745577-3dd610ec-2f53-4dcd-ad87-fa9b8401d84b >>>>>>> from ............... >>>>>>> >>>>>>> >>>>>>> But those mails will never be sent out. >>>>>>> >>>>>>> I tried to run jstack as Norman mentioned and exported the output >>>>>>> (Attached File). >>>>>>> >>>>>>> Please let me know if we can figure out what the issue is. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> - Raju >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 15, 2011 at 3:17 PM, Zach Cox <[email protected]> wrote: >>>>>>>> jstack requires a JDK to be installed, not just a JRE (as I discovered >>>>>>>> earlier today). >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 15, 2011 at 2:59 PM, Raju Buchi >>>>>>>> <[email protected]> wrote: >>>>>>>>> The mails are now stuck up in spool on my server. Can any one tell me >>>>>>>>> how to >>>>>>>>> debug. I tried to use jstack, but it says jstack: command not found. >>>>>>>>> >>>>>>>>> - Raju >>>>>>>>> >>>>>>>>> On Thu, Sep 15, 2011 at 1:16 PM, Zach Cox <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> OK so the cause of this shutdown problem was setting jmx.address= >>>>>>>>>> public IP address of the server in conf/jmx.properties. Just had the >>>>>>>>>> exact same problem on a fresh install of James. I changed jmx.address >>>>>>>>>> back to 127.0.0.1 and James now starts up properly. >>>>>>>>>> >>>>>>>>>> There is also no mention of anything related to jmx in the logs when >>>>>>>>>> James is unable to start up because of using the public IP address >>>>>>>>>> for >>>>>>>>>> jmx.address. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Sep 15, 2011 at 1:02 PM, Zach Cox <[email protected]> wrote: >>>>>>>>>> > Sorry, had to replace that EC2 instance with a new one, need to >>>>>>>>>> > keep >>>>>>>>>> > emails flowing... >>>>>>>>>> > >>>>>>>>>> > So that server is gone, but we still have all of the logs. >>>>>>>>>> > >>>>>>>>>> > If this happens again I will try moving activemq dir. >>>>>>>>>> > >>>>>>>>>> > Thanks, >>>>>>>>>> > Zach >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > On Thu, Sep 15, 2011 at 1:00 PM, Norman Maurer >>>>>>>>>> > <[email protected]> wrote: >>>>>>>>>> >> Hmm.. >>>>>>>>>> >> >>>>>>>>>> >> never saw this problem. But could you try to move the activemq >>>>>>>>>> >> dir out >>>>>>>>>> >> of the way. >>>>>>>>>> >> >>>>>>>>>> >> # mv /path/to/james/var/store/activemq >>>>>>>>>> /path/to/james/var/store/activemq.crc >>>>>>>>>> >> >>>>>>>>>> >> After that try to start again.. >>>>>>>>>> >> >>>>>>>>>> >> Bye, >>>>>>>>>> >> Norman >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> 2011/9/15 Zach Cox <[email protected]>: >>>>>>>>>> >>> And now every time I start James, after it completes startup, it >>>>>>>>>> >>> just >>>>>>>>>> >>> shuts down: >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> 2011-09-15 17:56:26 INFO smtpserver:? - Init SMTP Service done >>>>>>>>>> >>> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - >>>>>>>>>> >>> ConfigurationUtils.locate(): base is null, name is null >>>>>>>>>> >>> 2011-09-15 17:56:26 INFO lmtpserver:? - LMTP Service disabled by >>>>>>>>>> configuration >>>>>>>>>> >>> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - >>>>>>>>>> >>> ConfigurationUtils.locate(): base is null, name is null >>>>>>>>>> >>> 2011-09-15 17:56:26 INFO pop3server:? - POP3 Service disabled by >>>>>>>>>> configuration >>>>>>>>>> >>> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - >>>>>>>>>> >>> ConfigurationUtils.locate(): base is null, name is null >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO imapserver:? - IMAP Service disabled by >>>>>>>>>> configuration >>>>>>>>>> >>> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - >>>>>>>>>> >>> ConfigurationUtils.locate(): base is null, name is null >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO usersrepository23:? - Init >>>>>>>>>> >>> org.apache.james.repository.file.FilePersistentObjectRepository >>>>>>>>>> >>> Store >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO usersrepository23:? - >>>>>>>>>> >>> org.apache.james.repository.file.FilePersistentObjectRepository >>>>>>>>>> >>> opened >>>>>>>>>> >>> in ../var/users >>>>>>>>>> >>> 2011-09-15 17:56:27 DEBUG usersrepository23:? - >>>>>>>>>> >>> org.apache.james.user.file.UsersFileRepository created in >>>>>>>>>> >>> file://var/users/ >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service >>>>>>>>>> >>> done >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service >>>>>>>>>> >>> done >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service >>>>>>>>>> >>> done >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service >>>>>>>>>> >>> done >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO mailspooler:? - >>>>>>>>>> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO mailspooler:? - Stop >>>>>>>>>> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: >>>>>>>>>> >>> dequeuer-1 >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO mailspooler:? - Stop >>>>>>>>>> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: >>>>>>>>>> >>> dequeuer-2 >>>>>>>>>> >>> 2011-09-15 17:56:27 INFO mailspooler:? - >>>>>>>>>> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler thread >>>>>>>>>> >>> shutdown >>>>>>>>>> >>> completed. >>>>>>>>>> >>> 2011-09-15 17:56:30 INFO SocketNode:94 - Caught >>>>>>>>>> >>> java.io.EOFException >>>>>>>>>> >>> closing conneciton. >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> Any advice would be greatly appreciated. >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> On Thu, Sep 15, 2011 at 12:54 PM, Zach Cox <[email protected]> >>>>>>>>>> >>> wrote: >>>>>>>>>> >>>> OK now several minutes after I restarted James, everything was >>>>>>>>>> >>>> working >>>>>>>>>> >>>> fine, then all of a sudden I see this while tailing >>>>>>>>>> >>>> james-server.log: >>>>>>>>>> >>>> >>>>>>>>>> >>>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Timeout >>>>>>>>>> >>>> occurred. Now forcing the routes to be shutdown now. >>>>>>>>>> >>>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - >>>>>>>>>> >>>> Interrupted >>>>>>>>>> >>>> while waiting during graceful shutdown, will force shutdown now. >>>>>>>>>> >>>> 2011-09-15 17:51:24 WARN DefaultInflightRepository:? - >>>>>>>>>> >>>> Shutting down >>>>>>>>>> >>>> while there are still 8 in flight exchanges. >>>>>>>>>> >>>> 2011-09-15 17:51:26 INFO SocketNode:94 - Caught >>>>>>>>>> >>>> java.io.EOFException >>>>>>>>>> >>>> closing conneciton. >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> And James just shuts down. This is incredibly bad to have a >>>>>>>>>> >>>> mail >>>>>>>>>> >>>> server just shut down out of the blue... >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> On Thu, Sep 15, 2011 at 12:46 PM, Zach Cox <[email protected]> >>>>>>>>>> >>>> wrote: >>>>>>>>>> >>>>> Also when James is starting up I see this in logs: >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> 2011-09-15 17:45:05 WARN AMQPersistenceAdapter:? - The >>>>>>>>>> ReferenceStore >>>>>>>>>> >>>>> is not valid - recovering ... >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> I'm guessing that's not a good thing? >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> On Thu, Sep 15, 2011 at 11:54 AM, Zach Cox <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>> I don't think we're seeing the problem again yet, but I just >>>>>>>>>> >>>>>> had to >>>>>>>>>> >>>>>> restart James and noticed this on shutdown: >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO JamesServerApplicationContext:? - >>>>>>>>>> >>>>>> Closing >>>>>>>>>> >>>>>> >>>>>>>>>> org.apache.james.container.spring.context.JamesServerApplicationContext@86fe26 >>>>>>>>>> : >>>>>>>>>> >>>>>> startup date [Thu Sep 15 12:20:21 UTC 2011]; root of context >>>>>>>>>> hierarchy >>>>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service >>>>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service >>>>>>>>>> >>>>>> done >>>>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service >>>>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service >>>>>>>>>> >>>>>> done >>>>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service >>>>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service >>>>>>>>>> >>>>>> done >>>>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service >>>>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service >>>>>>>>>> >>>>>> done >>>>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO mailspooler:? - >>>>>>>>>> >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler >>>>>>>>>> >>>>>> dispose... >>>>>>>>>> >>>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >>>>>>>>>> >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: >>>>>>>>>> >>>>>> dequeuer-2 >>>>>>>>>> >>>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >>>>>>>>>> >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: >>>>>>>>>> >>>>>> dequeuer-1 >>>>>>>>>> >>>>>> 2011-09-15 16:48:51 INFO SocketNode:94 - Caught >>>>>>>>>> java.io.EOFException >>>>>>>>>> >>>>>> closing conneciton. >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> Is that java.io.EOFException normal and expected? Or is it a >>>>>>>>>> >>>>>> sign of >>>>>>>>>> >>>>>> some problems? >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> Thanks, >>>>>>>>>> >>>>>> Zach >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> On Thu, Sep 15, 2011 at 8:35 AM, Norman Maurer >>>>>>>>>> >>>>>> <[email protected]> wrote: >>>>>>>>>> >>>>>>> Using: >>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>>> # jstack <pid> >>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>>> Please take 3 within on minute ... >>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>>> Bye, >>>>>>>>>> >>>>>>> Norman >>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>>> 2011/9/15 Zach Cox <[email protected]>: >>>>>>>>>> >>>>>>>> I had to restart James several times this morning so >>>>>>>>>> >>>>>>>> nothing is >>>>>>>>>> >>>>>>>> blocked at the moment. >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> There is nothing indicated in the log files. Things look >>>>>>>>>> >>>>>>>> normal, >>>>>>>>>> and >>>>>>>>>> >>>>>>>> then they look like this: >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> 2011-09-15 01:25:26 INFO smtpserver:? - ID=8443803 >>>>>>>>>> >>>>>>>> Connection >>>>>>>>>> >>>>>>>> established from x >>>>>>>>>> >>>>>>>> 2011-09-15 01:25:31 INFO smtpserver:? - ID=8443803 >>>>>>>>>> >>>>>>>> Successfully >>>>>>>>>> >>>>>>>> spooled mail >>>>>>>>>> Mail1316049931087-05b8199e-e01f-4d14-84d5-3d7ac2443bb3 >>>>>>>>>> >>>>>>>> from x on x for [x] >>>>>>>>>> >>>>>>>> 2011-09-15 01:25:36 INFO smtpserver:? - ID=8443803 >>>>>>>>>> >>>>>>>> Connection >>>>>>>>>> closed for x >>>>>>>>>> >>>>>>>> 2011-09-15 01:25:47 INFO smtpserver:? - ID=11521429 >>>>>>>>>> >>>>>>>> Connection >>>>>>>>>> >>>>>>>> established from x >>>>>>>>>> >>>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 >>>>>>>>>> >>>>>>>> Successfully >>>>>>>>>> >>>>>>>> spooled mail >>>>>>>>>> Mail1316049948254-abd67e8f-bafe-41ab-8eef-c551420fd56e >>>>>>>>>> >>>>>>>> from x on x for [x] >>>>>>>>>> >>>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 >>>>>>>>>> >>>>>>>> Connection >>>>>>>>>> closed for x >>>>>>>>>> >>>>>>>> 2011-09-15 01:26:41 INFO smtpserver:? - ID=5675333 >>>>>>>>>> >>>>>>>> Connection >>>>>>>>>> >>>>>>>> established from x >>>>>>>>>> >>>>>>>> 2011-09-15 01:26:52 INFO smtpserver:? - ID=5675333 >>>>>>>>>> >>>>>>>> Successfully >>>>>>>>>> >>>>>>>> spooled mail >>>>>>>>>> Mail1316050012621-522b1f2a-2851-46fb-940c-711591f637cc >>>>>>>>>> >>>>>>>> from x on x for [x] >>>>>>>>>> >>>>>>>> 2011-09-15 01:26:57 INFO smtpserver:? - ID=5675333 >>>>>>>>>> >>>>>>>> Connection >>>>>>>>>> closed for x >>>>>>>>>> >>>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 >>>>>>>>>> >>>>>>>> Connection >>>>>>>>>> >>>>>>>> established from localhost (127.0.0.1) >>>>>>>>>> >>>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 >>>>>>>>>> >>>>>>>> Connection >>>>>>>>>> closed >>>>>>>>>> >>>>>>>> for localhost (127.0.0.1) >>>>>>>>>> >>>>>>>> 2011-09-15 01:28:12 INFO smtpserver:? - ID=11673762 >>>>>>>>>> >>>>>>>> Connection >>>>>>>>>> >>>>>>>> established from x >>>>>>>>>> >>>>>>>> 2011-09-15 01:28:22 INFO smtpserver:? - ID=11673762 >>>>>>>>>> >>>>>>>> Successfully >>>>>>>>>> >>>>>>>> spooled mail >>>>>>>>>> Mail1316050102393-6fcc21aa-8f32-4935-b0eb-df892d54a5e8 >>>>>>>>>> >>>>>>>> from x on x for [x] >>>>>>>>>> >>>>>>>> 2011-09-15 01:28:27 INFO smtpserver:? - ID=11673762 >>>>>>>>>> >>>>>>>> Connection >>>>>>>>>> closed for x >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> So every mail that comes in just gets spooled and not >>>>>>>>>> >>>>>>>> processed. >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> When this starts happening again, what's the best way to >>>>>>>>>> >>>>>>>> take a >>>>>>>>>> thread >>>>>>>>>> >>>>>>>> dump from James? >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> Thanks, >>>>>>>>>> >>>>>>>> Zach >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> On Thu, Sep 15, 2011 at 8:13 AM, Norman Maurer >>>>>>>>>> >>>>>>>> <[email protected]> wrote: >>>>>>>>>> >>>>>>>>> Hi there, >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> please take a thread-dump to see why its blocked... >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> Anything in the logs ? >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> Bye, >>>>>>>>>> >>>>>>>>> Norman >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> 2011/9/15 Zach Cox <[email protected]>: >>>>>>>>>> >>>>>>>>>> We upgraded to James 3.0-beta3 on Tuesday night. Things >>>>>>>>>> >>>>>>>>>> worked >>>>>>>>>> fine >>>>>>>>>> >>>>>>>>>> until around 9pm last night, where exactly the same >>>>>>>>>> >>>>>>>>>> problem >>>>>>>>>> started >>>>>>>>>> >>>>>>>>>> occurring: mail would get spooled, but not processed. >>>>>>>>>> >>>>>>>>>> After >>>>>>>>>> >>>>>>>>>> restarting James, some of the mail was then processed >>>>>>>>>> successfully, >>>>>>>>>> >>>>>>>>>> but most was lost. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We consider this an emergency situation for our company >>>>>>>>>> >>>>>>>>>> and need >>>>>>>>>> to >>>>>>>>>> >>>>>>>>>> find a resolution ASAP. What kind of information can we >>>>>>>>>> >>>>>>>>>> provide >>>>>>>>>> to >>>>>>>>>> >>>>>>>>>> help find a resolution? If financial resources need to be >>>>>>>>>> involved to >>>>>>>>>> >>>>>>>>>> find a resolution or move things along, that is an option. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Any help is most greatly appreciated. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Zach >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Sep 13, 2011 at 12:41 PM, Norman Maurer >>>>>>>>>> >>>>>>>>>> <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> No problems... >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> If you still can replicate then please open a issue with >>>>>>>>>> >>>>>>>>>>> a >>>>>>>>>> stack trace >>>>>>>>>> >>>>>>>>>>> attached.. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Bye, >>>>>>>>>> >>>>>>>>>>> Norman >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: >>>>>>>>>> >>>>>>>>>>>> James 3.0-M2. I will try to stand up a new server with >>>>>>>>>> >>>>>>>>>>>> beta3 >>>>>>>>>> and see if I >>>>>>>>>> >>>>>>>>>>>> can replicate the issue. >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> - Raju >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> On Tue, Sep 13, 2011 at 12:37 PM, Norman Maurer < >>>>>>>>>> >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> Do you use beta3 ? >>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> Bye, >>>>>>>>>> >>>>>>>>>>>>> Norman >>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: >>>>>>>>>> >>>>>>>>>>>>> > Hello >>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>> >>>>>>>>>>>>> > We are also facing exactly the same issues. Tried to >>>>>>>>>> >>>>>>>>>>>>> > debug >>>>>>>>>> my custom >>>>>>>>>> >>>>>>>>>>>>> mailets >>>>>>>>>> >>>>>>>>>>>>> > and seems to be everything is working fine. Logs did >>>>>>>>>> >>>>>>>>>>>>> > not >>>>>>>>>> show any >>>>>>>>>> >>>>>>>>>>>>> exception >>>>>>>>>> >>>>>>>>>>>>> > messages except it will always spool a mail and will >>>>>>>>>> >>>>>>>>>>>>> > never >>>>>>>>>> attempt to >>>>>>>>>> >>>>>>>>>>>>> send >>>>>>>>>> >>>>>>>>>>>>> > it. i.e. all mails are stuck in the spool. >>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>> >>>>>>>>>>>>> > - Raju >>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>> >>>>>>>>>>>>> > On Tue, Sep 13, 2011 at 12:20 PM, Zach Cox < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>> >>>>>>>>>>>>> >> We are having a severe problem with James 3.0-M2 >>>>>>>>>> >>>>>>>>>>>>> >> where it >>>>>>>>>> will operate >>>>>>>>>> >>>>>>>>>>>>> >> correctly for hours or days, and then all of a >>>>>>>>>> >>>>>>>>>>>>> >> sudden it >>>>>>>>>> will not >>>>>>>>>> >>>>>>>>>>>>> >> process a lot of the mail it spools. Sometimes it >>>>>>>>>> >>>>>>>>>>>>> >> will >>>>>>>>>> not process >>>>>>>>>> >>>>>>>>>>>>> >> *any* mail that it spools, and other times it will >>>>>>>>>> >>>>>>>>>>>>> >> spool a >>>>>>>>>> bunch of >>>>>>>>>> >>>>>>>>>>>>> >> mail, and then process others seemingly at random. >>>>>>>>>> >>>>>>>>>>>>> >> I have >>>>>>>>>> to restart >>>>>>>>>> >>>>>>>>>>>>> >> James to get that spooled mail to be processed. >>>>>>>>>> >>>>>>>>>>>>> >> But even >>>>>>>>>> then, >>>>>>>>>> >>>>>>>>>>>>> >> sometimes the spooled mail is completely lost after >>>>>>>>>> >>>>>>>>>>>>> >> a >>>>>>>>>> restart. I have >>>>>>>>>> >>>>>>>>>>>>> >> seen several other threads lately with similar >>>>>>>>>> >>>>>>>>>>>>> >> problems, >>>>>>>>>> but they >>>>>>>>>> >>>>>>>>>>>>> >> haven't helped our situation. >>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>>>>> >> We do not store mail in James for personal accounts >>>>>>>>>> >>>>>>>>>>>>> >> and we >>>>>>>>>> have IMAP & >>>>>>>>>> >>>>>>>>>>>>> >> POP3 disabled. Every inbound email has one of two >>>>>>>>>> >>>>>>>>>>>>> >> things >>>>>>>>>> done to it: >>>>>>>>>> >>>>>>>>>>>>> >> - it is processed locally by custom mailets & >>>>>>>>>> >>>>>>>>>>>>> >> matchers >>>>>>>>>> >>>>>>>>>>>>> >> - or it is relayed to our corporate Google Apps >>>>>>>>>> >>>>>>>>>>>>> >> account >>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>>>>> >> A previous thread recommended checking all custom >>>>>>>>>> >>>>>>>>>>>>> >> mailets >>>>>>>>>> and >>>>>>>>>> >>>>>>>>>>>>> >> matchers. As far as I can tell, they are all >>>>>>>>>> >>>>>>>>>>>>> >> working >>>>>>>>>> properly. There >>>>>>>>>> >>>>>>>>>>>>> >> are no exceptions or errors in >>>>>>>>>> >>>>>>>>>>>>> >> log/james-server.log, and >>>>>>>>>> in the >>>>>>>>>> >>>>>>>>>>>>> >> scenario where James spools some mail & processes >>>>>>>>>> >>>>>>>>>>>>> >> others, >>>>>>>>>> the mail >>>>>>>>>> >>>>>>>>>>>>> >> that is processed is handled properly by our custom >>>>>>>>>> mailets & >>>>>>>>>> >>>>>>>>>>>>> >> matchers. >>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>>>>> >> I've tried connecting to James JMX remotely via >>>>>>>>>> >>>>>>>>>>>>> >> JConsole >>>>>>>>>> but have not >>>>>>>>>> >>>>>>>>>>>>> >> been able to connect. Which port does James run >>>>>>>>>> >>>>>>>>>>>>> >> JMX on? >>>>>>>>>> Any help >>>>>>>>>> >>>>>>>>>>>>> >> with JMX would be appreciated. >>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>>>>> >> Does anyone have any other ideas about what we >>>>>>>>>> >>>>>>>>>>>>> >> should look >>>>>>>>>> for? I >>>>>>>>>> >>>>>>>>>>>>> >> noticed 3.0-beta3 is out now, were there fixes in >>>>>>>>>> >>>>>>>>>>>>> >> beta3 >>>>>>>>>> related to >>>>>>>>>> >>>>>>>>>>>>> >> mail being spooled but not processed? >>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>>>>> >> Thanks, >>>>>>>>>> >>>>>>>>>>>>> >> Zach >>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> >>>>>>>>>>>>> >> To unsubscribe, e-mail: >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>>>>> >> For additional commands, e-mail: >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>> >>>>>>>>>>>>> > >>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> >>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> >>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> >>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>> For additional commands, e-mail: >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> >>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>> >>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>> For additional commands, e-mail: >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> >>>>>>>> To unsubscribe, e-mail: >>>>>>>>>> >>>>>>>> [email protected] >>>>>>>>>> >>>>>>>> For additional commands, e-mail: >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> >>>>>>> To unsubscribe, e-mail: >>>>>>>>>> >>>>>>> [email protected] >>>>>>>>>> >>>>>>> For additional commands, e-mail: >>>>>>>>>> >>>>>>> [email protected] >>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>> >>>>>>>>>> >>> >>>>>>>>>> >>> --------------------------------------------------------------------- >>>>>>>>>> >>> To unsubscribe, e-mail: [email protected] >>>>>>>>>> >>> For additional commands, e-mail: >>>>>>>>>> >>> [email protected] >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >> >>>>>>>>>> >> --------------------------------------------------------------------- >>>>>>>>>> >> To unsubscribe, e-mail: [email protected] >>>>>>>>>> >> For additional commands, e-mail: [email protected] >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Raju Buchi >>>>>>> Web Developer | Content Pilot LLC >>>>>>> Mobile: 817.313.2578 >>>>>>> Email: [email protected] | [email protected] >>>>>>> Website: http://www.rajubuchi.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Raju Buchi >>>>>> Web Developer | Content Pilot LLC >>>>>> Mobile: 817.313.2578 >>>>>> Email: [email protected] | [email protected] >>>>>> Website: http://www.rajubuchi.com >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>> >>> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
