OK now several minutes after I restarted James, everything was working fine, then all of a sudden I see this while tailing james-server.log:
2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Timeout occurred. Now forcing the routes to be shutdown now. 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Interrupted while waiting during graceful shutdown, will force shutdown now. 2011-09-15 17:51:24 WARN DefaultInflightRepository:? - Shutting down while there are still 8 in flight exchanges. 2011-09-15 17:51:26 INFO SocketNode:94 - Caught java.io.EOFException closing conneciton. And James just shuts down. This is incredibly bad to have a mail server just shut down out of the blue... On Thu, Sep 15, 2011 at 12:46 PM, Zach Cox <[email protected]> wrote: > Also when James is starting up I see this in logs: > > 2011-09-15 17:45:05 WARN AMQPersistenceAdapter:? - The ReferenceStore > is not valid - recovering ... > > > I'm guessing that's not a good thing? > > > On Thu, Sep 15, 2011 at 11:54 AM, Zach Cox <[email protected]> wrote: >> I don't think we're seeing the problem again yet, but I just had to >> restart James and noticed this on shutdown: >> >> 2011-09-15 16:48:31 INFO JamesServerApplicationContext:? - Closing >> org.apache.james.container.spring.context.JamesServerApplicationContext@86fe26: >> startup date [Thu Sep 15 12:20:21 UTC 2011]; root of context hierarchy >> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service >> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service done >> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service >> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service done >> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service >> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service done >> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service >> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service done >> 2011-09-15 16:48:31 INFO mailspooler:? - >> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... >> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-2 >> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-1 >> 2011-09-15 16:48:51 INFO SocketNode:94 - Caught java.io.EOFException >> closing conneciton. >> >> >> Is that java.io.EOFException normal and expected? Or is it a sign of >> some problems? >> >> Thanks, >> Zach >> >> >> >> >> >> On Thu, Sep 15, 2011 at 8:35 AM, Norman Maurer >> <[email protected]> wrote: >>> Using: >>> >>> # jstack <pid> >>> >>> >>> Please take 3 within on minute ... >>> >>> Bye, >>> Norman >>> >>> 2011/9/15 Zach Cox <[email protected]>: >>>> I had to restart James several times this morning so nothing is >>>> blocked at the moment. >>>> >>>> There is nothing indicated in the log files. Things look normal, and >>>> then they look like this: >>>> >>>> 2011-09-15 01:25:26 INFO smtpserver:? - ID=8443803 Connection >>>> established from x >>>> 2011-09-15 01:25:31 INFO smtpserver:? - ID=8443803 Successfully >>>> spooled mail Mail1316049931087-05b8199e-e01f-4d14-84d5-3d7ac2443bb3 >>>> from x on x for [x] >>>> 2011-09-15 01:25:36 INFO smtpserver:? - ID=8443803 Connection closed for x >>>> 2011-09-15 01:25:47 INFO smtpserver:? - ID=11521429 Connection >>>> established from x >>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 Successfully >>>> spooled mail Mail1316049948254-abd67e8f-bafe-41ab-8eef-c551420fd56e >>>> from x on x for [x] >>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 Connection closed for >>>> x >>>> 2011-09-15 01:26:41 INFO smtpserver:? - ID=5675333 Connection >>>> established from x >>>> 2011-09-15 01:26:52 INFO smtpserver:? - ID=5675333 Successfully >>>> spooled mail Mail1316050012621-522b1f2a-2851-46fb-940c-711591f637cc >>>> from x on x for [x] >>>> 2011-09-15 01:26:57 INFO smtpserver:? - ID=5675333 Connection closed for x >>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 Connection >>>> established from localhost (127.0.0.1) >>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 Connection closed >>>> for localhost (127.0.0.1) >>>> 2011-09-15 01:28:12 INFO smtpserver:? - ID=11673762 Connection >>>> established from x >>>> 2011-09-15 01:28:22 INFO smtpserver:? - ID=11673762 Successfully >>>> spooled mail Mail1316050102393-6fcc21aa-8f32-4935-b0eb-df892d54a5e8 >>>> from x on x for [x] >>>> 2011-09-15 01:28:27 INFO smtpserver:? - ID=11673762 Connection closed for >>>> x >>>> >>>> So every mail that comes in just gets spooled and not processed. >>>> >>>> When this starts happening again, what's the best way to take a thread >>>> dump from James? >>>> >>>> Thanks, >>>> Zach >>>> >>>> >>>> On Thu, Sep 15, 2011 at 8:13 AM, Norman Maurer >>>> <[email protected]> wrote: >>>>> Hi there, >>>>> >>>>> please take a thread-dump to see why its blocked... >>>>> >>>>> Anything in the logs ? >>>>> >>>>> Bye, >>>>> Norman >>>>> >>>>> >>>>> 2011/9/15 Zach Cox <[email protected]>: >>>>>> We upgraded to James 3.0-beta3 on Tuesday night. Things worked fine >>>>>> until around 9pm last night, where exactly the same problem started >>>>>> occurring: mail would get spooled, but not processed. After >>>>>> restarting James, some of the mail was then processed successfully, >>>>>> but most was lost. >>>>>> >>>>>> We consider this an emergency situation for our company and need to >>>>>> find a resolution ASAP. What kind of information can we provide to >>>>>> help find a resolution? If financial resources need to be involved to >>>>>> find a resolution or move things along, that is an option. >>>>>> >>>>>> Any help is most greatly appreciated. >>>>>> >>>>>> Thanks, >>>>>> Zach >>>>>> >>>>>> >>>>>> On Tue, Sep 13, 2011 at 12:41 PM, Norman Maurer >>>>>> <[email protected]> wrote: >>>>>>> No problems... >>>>>>> >>>>>>> If you still can replicate then please open a issue with a stack trace >>>>>>> attached.. >>>>>>> >>>>>>> Bye, >>>>>>> Norman >>>>>>> >>>>>>> >>>>>>> 2011/9/13 Raju Buchi <[email protected]>: >>>>>>>> James 3.0-M2. I will try to stand up a new server with beta3 and see >>>>>>>> if I >>>>>>>> can replicate the issue. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> - Raju >>>>>>>> >>>>>>>> On Tue, Sep 13, 2011 at 12:37 PM, Norman Maurer < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Do you use beta3 ? >>>>>>>>> >>>>>>>>> Bye, >>>>>>>>> Norman >>>>>>>>> >>>>>>>>> >>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: >>>>>>>>> > Hello >>>>>>>>> > >>>>>>>>> > We are also facing exactly the same issues. Tried to debug my custom >>>>>>>>> mailets >>>>>>>>> > and seems to be everything is working fine. Logs did not show any >>>>>>>>> exception >>>>>>>>> > messages except it will always spool a mail and will never attempt >>>>>>>>> > to >>>>>>>>> send >>>>>>>>> > it. i.e. all mails are stuck in the spool. >>>>>>>>> > >>>>>>>>> > - Raju >>>>>>>>> > >>>>>>>>> > On Tue, Sep 13, 2011 at 12:20 PM, Zach Cox <[email protected]> >>>>>>>>> > wrote: >>>>>>>>> > >>>>>>>>> >> We are having a severe problem with James 3.0-M2 where it will >>>>>>>>> >> operate >>>>>>>>> >> correctly for hours or days, and then all of a sudden it will not >>>>>>>>> >> process a lot of the mail it spools. Sometimes it will not process >>>>>>>>> >> *any* mail that it spools, and other times it will spool a bunch of >>>>>>>>> >> mail, and then process others seemingly at random. I have to >>>>>>>>> >> restart >>>>>>>>> >> James to get that spooled mail to be processed. But even then, >>>>>>>>> >> sometimes the spooled mail is completely lost after a restart. I >>>>>>>>> >> have >>>>>>>>> >> seen several other threads lately with similar problems, but they >>>>>>>>> >> haven't helped our situation. >>>>>>>>> >> >>>>>>>>> >> We do not store mail in James for personal accounts and we have >>>>>>>>> >> IMAP & >>>>>>>>> >> POP3 disabled. Every inbound email has one of two things done to >>>>>>>>> >> it: >>>>>>>>> >> - it is processed locally by custom mailets & matchers >>>>>>>>> >> - or it is relayed to our corporate Google Apps account >>>>>>>>> >> >>>>>>>>> >> A previous thread recommended checking all custom mailets and >>>>>>>>> >> matchers. As far as I can tell, they are all working properly. >>>>>>>>> >> There >>>>>>>>> >> are no exceptions or errors in log/james-server.log, and in the >>>>>>>>> >> scenario where James spools some mail & processes others, the mail >>>>>>>>> >> that is processed is handled properly by our custom mailets & >>>>>>>>> >> matchers. >>>>>>>>> >> >>>>>>>>> >> I've tried connecting to James JMX remotely via JConsole but have >>>>>>>>> >> not >>>>>>>>> >> been able to connect. Which port does James run JMX on? Any help >>>>>>>>> >> with JMX would be appreciated. >>>>>>>>> >> >>>>>>>>> >> Does anyone have any other ideas about what we should look for? I >>>>>>>>> >> noticed 3.0-beta3 is out now, were there fixes in beta3 related to >>>>>>>>> >> mail being spooled but not processed? >>>>>>>>> >> >>>>>>>>> >> Thanks, >>>>>>>>> >> Zach >>>>>>>>> >> >>>>>>>>> >> --------------------------------------------------------------------- >>>>>>>>> >> To unsubscribe, e-mail: [email protected] >>>>>>>>> >> For additional commands, e-mail: [email protected] >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> > >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>> For additional commands, e-mail: [email protected] >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
