James is currently spooling all of our incoming email and not processing it. I took 4 jstack dumps. Anything else I can look at quickly before I have to restart James?
On Fri, Sep 16, 2011 at 6:58 AM, Zach Cox <[email protected]> wrote: > mailetcontainer.xml has: > > <spooler> > <threads> 20 </threads> > </spooler> > > After running all night, there are still only 4 blocked Camel threads, > and for the most part James appears to be processing all emails > properly. I will do some more testing this morning, and try to force a > bunch of these HTTP requests and see if I can lock it up. > > Just taking another look at our stack trace. In this mailet, we use > DefaultHttpClient with ThreadSafeClientConnManager. I know this is not > the Apache HttpClient mailing list, but does this mean it's not > blocking on a request with no response, but it's blocking just trying > to get a connection from the connection pool? > > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7aff2ee0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158) > at > org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:403) > at > org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) > at > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) > at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) > > That stack trace looks almost identical up towards the top as all of > the other "normal" Camel threads like this one: > > "Camel Thread 72 - Split" prio=10 tid=0x0a1ebc00 nid=0x10b5 waiting on > condition [0xb0e7a000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7a6a2ca0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > > I just want to make sure I understand how exactly this mailet is blocking. > > Thanks, > Zach > > > > > > > On Fri, Sep 16, 2011 at 12:16 AM, Norman Maurer > <[email protected]> wrote: >> Hi there, >> >> how many spooling threads did you specifiy in mailetcontainer.xml ? >> >> About the split threads... If a Matcher "split" and email into parts >> Camel will process this new parts in seperate threads. And no, 4 >> blocked threads should not block all the others. >> >> Bye, >> Norman >> >> >> 2011/9/16 Zach Cox <[email protected]>: >>> I just noticed James starting to spool some emails and did some >>> jstacks. I found several threads like this: >>> >>> "Camel Thread 65 - Split" prio=10 tid=0x0a1c1400 nid=0x10ae waiting on >>> condition [0xb105f000] >>> java.lang.Thread.State: WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <0x7aff2ee0> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) >>> at >>> org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158) >>> at >>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:403) >>> at >>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) >>> at >>> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) >>> at >>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401) >>> at >>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) >>> at >>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) >>> at >>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) >>> <rest is redacted> >>> >>> >>> One of our custom mailets makes an HTTP request to a 3rd party site. >>> Usually this works fine, but apparently sometimes this other site does >>> not send back an HTTP response, keeps the request open forever, and >>> the Apache HttpClient is not doing any kind of timeouts, so this >>> request stays open and seems to block a thread forever. >>> >>> What exactly do these 200 Camel Threads do? Currently we have 4 Camel >>> threads blocked like this. But I wouldn't think 4 out of 200 would >>> cause James to stop processing all incoming emails? James is still >>> processing some emails, but stuck on others. We will let James run all >>> night, and see if eventually all 200 get blocked. >>> >>> Anyways, wanted to share this. In the morning we will add timeouts to >>> these HTTP requests (as well as any others we do) and see if that >>> helps things. >>> >>> Thanks, >>> Zach >>> >>> >>> >>> >>> >>> On Thu, Sep 15, 2011 at 6:27 PM, Raju Buchi <[email protected]> >>> wrote: >>>> I guess the attachment was sent over. I have created a Bug report. >>>> >>>> https://issues.apache.org/jira/browse/JAMES-1317 >>>> >>>> - Raju >>>> >>>> On Thu, Sep 15, 2011 at 6:12 PM, Raju Buchi <[email protected]> >>>> wrote: >>>>> Hi >>>>> >>>>> I have got the issue again today where the mails are stuck in spool >>>>> and all the mails sent to the server will just say >>>>> INFO 19:02:25,581 | james.smtpserver | ID=17053510 Successfully >>>>> spooled mail Mail1316127745577-3dd610ec-2f53-4dcd-ad87-fa9b8401d84b >>>>> from ............... >>>>> >>>>> >>>>> But those mails will never be sent out. >>>>> >>>>> I tried to run jstack as Norman mentioned and exported the output >>>>> (Attached File). >>>>> >>>>> Please let me know if we can figure out what the issue is. >>>>> >>>>> Thanks >>>>> >>>>> - Raju >>>>> >>>>> >>>>> >>>>> On Thu, Sep 15, 2011 at 3:17 PM, Zach Cox <[email protected]> wrote: >>>>>> jstack requires a JDK to be installed, not just a JRE (as I discovered >>>>>> earlier today). >>>>>> >>>>>> >>>>>> On Thu, Sep 15, 2011 at 2:59 PM, Raju Buchi >>>>>> <[email protected]> wrote: >>>>>>> The mails are now stuck up in spool on my server. Can any one tell me >>>>>>> how to >>>>>>> debug. I tried to use jstack, but it says jstack: command not found. >>>>>>> >>>>>>> - Raju >>>>>>> >>>>>>> On Thu, Sep 15, 2011 at 1:16 PM, Zach Cox <[email protected]> wrote: >>>>>>> >>>>>>>> OK so the cause of this shutdown problem was setting jmx.address= >>>>>>>> public IP address of the server in conf/jmx.properties. Just had the >>>>>>>> exact same problem on a fresh install of James. I changed jmx.address >>>>>>>> back to 127.0.0.1 and James now starts up properly. >>>>>>>> >>>>>>>> There is also no mention of anything related to jmx in the logs when >>>>>>>> James is unable to start up because of using the public IP address for >>>>>>>> jmx.address. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 15, 2011 at 1:02 PM, Zach Cox <[email protected]> wrote: >>>>>>>> > Sorry, had to replace that EC2 instance with a new one, need to keep >>>>>>>> > emails flowing... >>>>>>>> > >>>>>>>> > So that server is gone, but we still have all of the logs. >>>>>>>> > >>>>>>>> > If this happens again I will try moving activemq dir. >>>>>>>> > >>>>>>>> > Thanks, >>>>>>>> > Zach >>>>>>>> > >>>>>>>> > >>>>>>>> > On Thu, Sep 15, 2011 at 1:00 PM, Norman Maurer >>>>>>>> > <[email protected]> wrote: >>>>>>>> >> Hmm.. >>>>>>>> >> >>>>>>>> >> never saw this problem. But could you try to move the activemq dir >>>>>>>> >> out >>>>>>>> >> of the way. >>>>>>>> >> >>>>>>>> >> # mv /path/to/james/var/store/activemq >>>>>>>> /path/to/james/var/store/activemq.crc >>>>>>>> >> >>>>>>>> >> After that try to start again.. >>>>>>>> >> >>>>>>>> >> Bye, >>>>>>>> >> Norman >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> 2011/9/15 Zach Cox <[email protected]>: >>>>>>>> >>> And now every time I start James, after it completes startup, it >>>>>>>> >>> just >>>>>>>> >>> shuts down: >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> 2011-09-15 17:56:26 INFO smtpserver:? - Init SMTP Service done >>>>>>>> >>> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - >>>>>>>> >>> ConfigurationUtils.locate(): base is null, name is null >>>>>>>> >>> 2011-09-15 17:56:26 INFO lmtpserver:? - LMTP Service disabled by >>>>>>>> configuration >>>>>>>> >>> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - >>>>>>>> >>> ConfigurationUtils.locate(): base is null, name is null >>>>>>>> >>> 2011-09-15 17:56:26 INFO pop3server:? - POP3 Service disabled by >>>>>>>> configuration >>>>>>>> >>> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - >>>>>>>> >>> ConfigurationUtils.locate(): base is null, name is null >>>>>>>> >>> 2011-09-15 17:56:27 INFO imapserver:? - IMAP Service disabled by >>>>>>>> configuration >>>>>>>> >>> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - >>>>>>>> >>> ConfigurationUtils.locate(): base is null, name is null >>>>>>>> >>> 2011-09-15 17:56:27 INFO usersrepository23:? - Init >>>>>>>> >>> org.apache.james.repository.file.FilePersistentObjectRepository >>>>>>>> >>> Store >>>>>>>> >>> 2011-09-15 17:56:27 INFO usersrepository23:? - >>>>>>>> >>> org.apache.james.repository.file.FilePersistentObjectRepository >>>>>>>> >>> opened >>>>>>>> >>> in ../var/users >>>>>>>> >>> 2011-09-15 17:56:27 DEBUG usersrepository23:? - >>>>>>>> >>> org.apache.james.user.file.UsersFileRepository created in >>>>>>>> >>> file://var/users/ >>>>>>>> >>> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service >>>>>>>> >>> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service done >>>>>>>> >>> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service >>>>>>>> >>> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service done >>>>>>>> >>> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service >>>>>>>> >>> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service done >>>>>>>> >>> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service >>>>>>>> >>> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service done >>>>>>>> >>> 2011-09-15 17:56:27 INFO mailspooler:? - >>>>>>>> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... >>>>>>>> >>> 2011-09-15 17:56:27 INFO mailspooler:? - Stop >>>>>>>> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-1 >>>>>>>> >>> 2011-09-15 17:56:27 INFO mailspooler:? - Stop >>>>>>>> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-2 >>>>>>>> >>> 2011-09-15 17:56:27 INFO mailspooler:? - >>>>>>>> >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler thread >>>>>>>> >>> shutdown >>>>>>>> >>> completed. >>>>>>>> >>> 2011-09-15 17:56:30 INFO SocketNode:94 - Caught >>>>>>>> >>> java.io.EOFException >>>>>>>> >>> closing conneciton. >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> Any advice would be greatly appreciated. >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> On Thu, Sep 15, 2011 at 12:54 PM, Zach Cox <[email protected]> >>>>>>>> >>> wrote: >>>>>>>> >>>> OK now several minutes after I restarted James, everything was >>>>>>>> >>>> working >>>>>>>> >>>> fine, then all of a sudden I see this while tailing >>>>>>>> >>>> james-server.log: >>>>>>>> >>>> >>>>>>>> >>>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Timeout >>>>>>>> >>>> occurred. Now forcing the routes to be shutdown now. >>>>>>>> >>>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Interrupted >>>>>>>> >>>> while waiting during graceful shutdown, will force shutdown now. >>>>>>>> >>>> 2011-09-15 17:51:24 WARN DefaultInflightRepository:? - Shutting >>>>>>>> >>>> down >>>>>>>> >>>> while there are still 8 in flight exchanges. >>>>>>>> >>>> 2011-09-15 17:51:26 INFO SocketNode:94 - Caught >>>>>>>> >>>> java.io.EOFException >>>>>>>> >>>> closing conneciton. >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> And James just shuts down. This is incredibly bad to have a mail >>>>>>>> >>>> server just shut down out of the blue... >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> On Thu, Sep 15, 2011 at 12:46 PM, Zach Cox <[email protected]> >>>>>>>> >>>> wrote: >>>>>>>> >>>>> Also when James is starting up I see this in logs: >>>>>>>> >>>>> >>>>>>>> >>>>> 2011-09-15 17:45:05 WARN AMQPersistenceAdapter:? - The >>>>>>>> ReferenceStore >>>>>>>> >>>>> is not valid - recovering ... >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> I'm guessing that's not a good thing? >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> On Thu, Sep 15, 2011 at 11:54 AM, Zach Cox <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>> I don't think we're seeing the problem again yet, but I just >>>>>>>> >>>>>> had to >>>>>>>> >>>>>> restart James and noticed this on shutdown: >>>>>>>> >>>>>> >>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO JamesServerApplicationContext:? - >>>>>>>> >>>>>> Closing >>>>>>>> >>>>>> >>>>>>>> org.apache.james.container.spring.context.JamesServerApplicationContext@86fe26 >>>>>>>> : >>>>>>>> >>>>>> startup date [Thu Sep 15 12:20:21 UTC 2011]; root of context >>>>>>>> hierarchy >>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service >>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service >>>>>>>> >>>>>> done >>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service >>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service >>>>>>>> >>>>>> done >>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service >>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service >>>>>>>> >>>>>> done >>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service >>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service >>>>>>>> >>>>>> done >>>>>>>> >>>>>> 2011-09-15 16:48:31 INFO mailspooler:? - >>>>>>>> >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler >>>>>>>> >>>>>> dispose... >>>>>>>> >>>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >>>>>>>> >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: >>>>>>>> >>>>>> dequeuer-2 >>>>>>>> >>>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >>>>>>>> >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: >>>>>>>> >>>>>> dequeuer-1 >>>>>>>> >>>>>> 2011-09-15 16:48:51 INFO SocketNode:94 - Caught >>>>>>>> java.io.EOFException >>>>>>>> >>>>>> closing conneciton. >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> Is that java.io.EOFException normal and expected? Or is it a >>>>>>>> >>>>>> sign of >>>>>>>> >>>>>> some problems? >>>>>>>> >>>>>> >>>>>>>> >>>>>> Thanks, >>>>>>>> >>>>>> Zach >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> On Thu, Sep 15, 2011 at 8:35 AM, Norman Maurer >>>>>>>> >>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>> Using: >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> # jstack <pid> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> Please take 3 within on minute ... >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> Bye, >>>>>>>> >>>>>>> Norman >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> 2011/9/15 Zach Cox <[email protected]>: >>>>>>>> >>>>>>>> I had to restart James several times this morning so nothing >>>>>>>> >>>>>>>> is >>>>>>>> >>>>>>>> blocked at the moment. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> There is nothing indicated in the log files. Things look >>>>>>>> >>>>>>>> normal, >>>>>>>> and >>>>>>>> >>>>>>>> then they look like this: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2011-09-15 01:25:26 INFO smtpserver:? - ID=8443803 Connection >>>>>>>> >>>>>>>> established from x >>>>>>>> >>>>>>>> 2011-09-15 01:25:31 INFO smtpserver:? - ID=8443803 >>>>>>>> >>>>>>>> Successfully >>>>>>>> >>>>>>>> spooled mail >>>>>>>> Mail1316049931087-05b8199e-e01f-4d14-84d5-3d7ac2443bb3 >>>>>>>> >>>>>>>> from x on x for [x] >>>>>>>> >>>>>>>> 2011-09-15 01:25:36 INFO smtpserver:? - ID=8443803 Connection >>>>>>>> closed for x >>>>>>>> >>>>>>>> 2011-09-15 01:25:47 INFO smtpserver:? - ID=11521429 >>>>>>>> >>>>>>>> Connection >>>>>>>> >>>>>>>> established from x >>>>>>>> >>>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 >>>>>>>> >>>>>>>> Successfully >>>>>>>> >>>>>>>> spooled mail >>>>>>>> Mail1316049948254-abd67e8f-bafe-41ab-8eef-c551420fd56e >>>>>>>> >>>>>>>> from x on x for [x] >>>>>>>> >>>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 >>>>>>>> >>>>>>>> Connection >>>>>>>> closed for x >>>>>>>> >>>>>>>> 2011-09-15 01:26:41 INFO smtpserver:? - ID=5675333 Connection >>>>>>>> >>>>>>>> established from x >>>>>>>> >>>>>>>> 2011-09-15 01:26:52 INFO smtpserver:? - ID=5675333 >>>>>>>> >>>>>>>> Successfully >>>>>>>> >>>>>>>> spooled mail >>>>>>>> Mail1316050012621-522b1f2a-2851-46fb-940c-711591f637cc >>>>>>>> >>>>>>>> from x on x for [x] >>>>>>>> >>>>>>>> 2011-09-15 01:26:57 INFO smtpserver:? - ID=5675333 Connection >>>>>>>> closed for x >>>>>>>> >>>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 >>>>>>>> >>>>>>>> Connection >>>>>>>> >>>>>>>> established from localhost (127.0.0.1) >>>>>>>> >>>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 >>>>>>>> >>>>>>>> Connection >>>>>>>> closed >>>>>>>> >>>>>>>> for localhost (127.0.0.1) >>>>>>>> >>>>>>>> 2011-09-15 01:28:12 INFO smtpserver:? - ID=11673762 >>>>>>>> >>>>>>>> Connection >>>>>>>> >>>>>>>> established from x >>>>>>>> >>>>>>>> 2011-09-15 01:28:22 INFO smtpserver:? - ID=11673762 >>>>>>>> >>>>>>>> Successfully >>>>>>>> >>>>>>>> spooled mail >>>>>>>> Mail1316050102393-6fcc21aa-8f32-4935-b0eb-df892d54a5e8 >>>>>>>> >>>>>>>> from x on x for [x] >>>>>>>> >>>>>>>> 2011-09-15 01:28:27 INFO smtpserver:? - ID=11673762 >>>>>>>> >>>>>>>> Connection >>>>>>>> closed for x >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> So every mail that comes in just gets spooled and not >>>>>>>> >>>>>>>> processed. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> When this starts happening again, what's the best way to take >>>>>>>> >>>>>>>> a >>>>>>>> thread >>>>>>>> >>>>>>>> dump from James? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Zach >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 15, 2011 at 8:13 AM, Norman Maurer >>>>>>>> >>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi there, >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> please take a thread-dump to see why its blocked... >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> Anything in the logs ? >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> Bye, >>>>>>>> >>>>>>>>> Norman >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> 2011/9/15 Zach Cox <[email protected]>: >>>>>>>> >>>>>>>>>> We upgraded to James 3.0-beta3 on Tuesday night. Things >>>>>>>> >>>>>>>>>> worked >>>>>>>> fine >>>>>>>> >>>>>>>>>> until around 9pm last night, where exactly the same problem >>>>>>>> started >>>>>>>> >>>>>>>>>> occurring: mail would get spooled, but not processed. After >>>>>>>> >>>>>>>>>> restarting James, some of the mail was then processed >>>>>>>> successfully, >>>>>>>> >>>>>>>>>> but most was lost. >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> We consider this an emergency situation for our company and >>>>>>>> >>>>>>>>>> need >>>>>>>> to >>>>>>>> >>>>>>>>>> find a resolution ASAP. What kind of information can we >>>>>>>> >>>>>>>>>> provide >>>>>>>> to >>>>>>>> >>>>>>>>>> help find a resolution? If financial resources need to be >>>>>>>> involved to >>>>>>>> >>>>>>>>>> find a resolution or move things along, that is an option. >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> Any help is most greatly appreciated. >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> Thanks, >>>>>>>> >>>>>>>>>> Zach >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> On Tue, Sep 13, 2011 at 12:41 PM, Norman Maurer >>>>>>>> >>>>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>>>> No problems... >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> If you still can replicate then please open a issue with a >>>>>>>> stack trace >>>>>>>> >>>>>>>>>>> attached.. >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> Bye, >>>>>>>> >>>>>>>>>>> Norman >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: >>>>>>>> >>>>>>>>>>>> James 3.0-M2. I will try to stand up a new server with >>>>>>>> >>>>>>>>>>>> beta3 >>>>>>>> and see if I >>>>>>>> >>>>>>>>>>>> can replicate the issue. >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> - Raju >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> On Tue, Sep 13, 2011 at 12:37 PM, Norman Maurer < >>>>>>>> >>>>>>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> Do you use beta3 ? >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> Bye, >>>>>>>> >>>>>>>>>>>>> Norman >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: >>>>>>>> >>>>>>>>>>>>> > Hello >>>>>>>> >>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>> > We are also facing exactly the same issues. Tried to >>>>>>>> >>>>>>>>>>>>> > debug >>>>>>>> my custom >>>>>>>> >>>>>>>>>>>>> mailets >>>>>>>> >>>>>>>>>>>>> > and seems to be everything is working fine. Logs did >>>>>>>> >>>>>>>>>>>>> > not >>>>>>>> show any >>>>>>>> >>>>>>>>>>>>> exception >>>>>>>> >>>>>>>>>>>>> > messages except it will always spool a mail and will >>>>>>>> >>>>>>>>>>>>> > never >>>>>>>> attempt to >>>>>>>> >>>>>>>>>>>>> send >>>>>>>> >>>>>>>>>>>>> > it. i.e. all mails are stuck in the spool. >>>>>>>> >>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>> > - Raju >>>>>>>> >>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>> > On Tue, Sep 13, 2011 at 12:20 PM, Zach Cox < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>> >> We are having a severe problem with James 3.0-M2 >>>>>>>> >>>>>>>>>>>>> >> where it >>>>>>>> will operate >>>>>>>> >>>>>>>>>>>>> >> correctly for hours or days, and then all of a sudden >>>>>>>> >>>>>>>>>>>>> >> it >>>>>>>> will not >>>>>>>> >>>>>>>>>>>>> >> process a lot of the mail it spools. Sometimes it >>>>>>>> >>>>>>>>>>>>> >> will >>>>>>>> not process >>>>>>>> >>>>>>>>>>>>> >> *any* mail that it spools, and other times it will >>>>>>>> >>>>>>>>>>>>> >> spool a >>>>>>>> bunch of >>>>>>>> >>>>>>>>>>>>> >> mail, and then process others seemingly at random. I >>>>>>>> >>>>>>>>>>>>> >> have >>>>>>>> to restart >>>>>>>> >>>>>>>>>>>>> >> James to get that spooled mail to be processed. But >>>>>>>> >>>>>>>>>>>>> >> even >>>>>>>> then, >>>>>>>> >>>>>>>>>>>>> >> sometimes the spooled mail is completely lost after a >>>>>>>> restart. I have >>>>>>>> >>>>>>>>>>>>> >> seen several other threads lately with similar >>>>>>>> >>>>>>>>>>>>> >> problems, >>>>>>>> but they >>>>>>>> >>>>>>>>>>>>> >> haven't helped our situation. >>>>>>>> >>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>> >> We do not store mail in James for personal accounts >>>>>>>> >>>>>>>>>>>>> >> and we >>>>>>>> have IMAP & >>>>>>>> >>>>>>>>>>>>> >> POP3 disabled. Every inbound email has one of two >>>>>>>> >>>>>>>>>>>>> >> things >>>>>>>> done to it: >>>>>>>> >>>>>>>>>>>>> >> - it is processed locally by custom mailets & >>>>>>>> >>>>>>>>>>>>> >> matchers >>>>>>>> >>>>>>>>>>>>> >> - or it is relayed to our corporate Google Apps >>>>>>>> >>>>>>>>>>>>> >> account >>>>>>>> >>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>> >> A previous thread recommended checking all custom >>>>>>>> >>>>>>>>>>>>> >> mailets >>>>>>>> and >>>>>>>> >>>>>>>>>>>>> >> matchers. As far as I can tell, they are all working >>>>>>>> properly. There >>>>>>>> >>>>>>>>>>>>> >> are no exceptions or errors in log/james-server.log, >>>>>>>> >>>>>>>>>>>>> >> and >>>>>>>> in the >>>>>>>> >>>>>>>>>>>>> >> scenario where James spools some mail & processes >>>>>>>> >>>>>>>>>>>>> >> others, >>>>>>>> the mail >>>>>>>> >>>>>>>>>>>>> >> that is processed is handled properly by our custom >>>>>>>> mailets & >>>>>>>> >>>>>>>>>>>>> >> matchers. >>>>>>>> >>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>> >> I've tried connecting to James JMX remotely via >>>>>>>> >>>>>>>>>>>>> >> JConsole >>>>>>>> but have not >>>>>>>> >>>>>>>>>>>>> >> been able to connect. Which port does James run JMX >>>>>>>> >>>>>>>>>>>>> >> on? >>>>>>>> Any help >>>>>>>> >>>>>>>>>>>>> >> with JMX would be appreciated. >>>>>>>> >>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>> >> Does anyone have any other ideas about what we should >>>>>>>> >>>>>>>>>>>>> >> look >>>>>>>> for? I >>>>>>>> >>>>>>>>>>>>> >> noticed 3.0-beta3 is out now, were there fixes in >>>>>>>> >>>>>>>>>>>>> >> beta3 >>>>>>>> related to >>>>>>>> >>>>>>>>>>>>> >> mail being spooled but not processed? >>>>>>>> >>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>> >> Thanks, >>>>>>>> >>>>>>>>>>>>> >> Zach >>>>>>>> >>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>> >> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>>>>>>>>>>>> >> To unsubscribe, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>>>>>> >> For additional commands, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>> >> >>>>>>>> >>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>>>> For additional commands, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>>>>>>>>> To unsubscribe, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>>> For additional commands, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>>>>>>>> To unsubscribe, e-mail: >>>>>>>> >>>>>>>>> [email protected] >>>>>>>> >>>>>>>>> For additional commands, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>>>>>>> To unsubscribe, e-mail: >>>>>>>> >>>>>>>> [email protected] >>>>>>>> >>>>>>>> For additional commands, e-mail: >>>>>>>> [email protected] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>>>>>> To unsubscribe, e-mail: >>>>>>>> >>>>>>> [email protected] >>>>>>>> >>>>>>> For additional commands, e-mail: >>>>>>>> >>>>>>> [email protected] >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>> >>>>>>>> >>>> >>>>>>>> >>> >>>>>>>> >>> --------------------------------------------------------------------- >>>>>>>> >>> To unsubscribe, e-mail: [email protected] >>>>>>>> >>> For additional commands, e-mail: [email protected] >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >> >>>>>>>> >> --------------------------------------------------------------------- >>>>>>>> >> To unsubscribe, e-mail: [email protected] >>>>>>>> >> For additional commands, e-mail: [email protected] >>>>>>>> >> >>>>>>>> >> >>>>>>>> > >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Raju Buchi >>>>> Web Developer | Content Pilot LLC >>>>> Mobile: 817.313.2578 >>>>> Email: [email protected] | [email protected] >>>>> Website: http://www.rajubuchi.com >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Raju Buchi >>>> Web Developer | Content Pilot LLC >>>> Mobile: 817.313.2578 >>>> Email: [email protected] | [email protected] >>>> Website: http://www.rajubuchi.com >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
