Sorry, had to replace that EC2 instance with a new one, need to keep emails flowing...
So that server is gone, but we still have all of the logs. If this happens again I will try moving activemq dir. Thanks, Zach On Thu, Sep 15, 2011 at 1:00 PM, Norman Maurer <[email protected]> wrote: > Hmm.. > > never saw this problem. But could you try to move the activemq dir out > of the way. > > # mv /path/to/james/var/store/activemq /path/to/james/var/store/activemq.crc > > After that try to start again.. > > Bye, > Norman > > > 2011/9/15 Zach Cox <[email protected]>: >> And now every time I start James, after it completes startup, it just >> shuts down: >> >> >> 2011-09-15 17:56:26 INFO smtpserver:? - Init SMTP Service done >> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - >> ConfigurationUtils.locate(): base is null, name is null >> 2011-09-15 17:56:26 INFO lmtpserver:? - LMTP Service disabled by >> configuration >> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - >> ConfigurationUtils.locate(): base is null, name is null >> 2011-09-15 17:56:26 INFO pop3server:? - POP3 Service disabled by >> configuration >> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - >> ConfigurationUtils.locate(): base is null, name is null >> 2011-09-15 17:56:27 INFO imapserver:? - IMAP Service disabled by >> configuration >> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - >> ConfigurationUtils.locate(): base is null, name is null >> 2011-09-15 17:56:27 INFO usersrepository23:? - Init >> org.apache.james.repository.file.FilePersistentObjectRepository Store >> 2011-09-15 17:56:27 INFO usersrepository23:? - >> org.apache.james.repository.file.FilePersistentObjectRepository opened >> in ../var/users >> 2011-09-15 17:56:27 DEBUG usersrepository23:? - >> org.apache.james.user.file.UsersFileRepository created in >> file://var/users/ >> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service >> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service done >> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service >> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service done >> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service >> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service done >> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service >> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service done >> 2011-09-15 17:56:27 INFO mailspooler:? - >> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... >> 2011-09-15 17:56:27 INFO mailspooler:? - Stop >> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-1 >> 2011-09-15 17:56:27 INFO mailspooler:? - Stop >> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-2 >> 2011-09-15 17:56:27 INFO mailspooler:? - >> org.apache.james.mailetcontainer.impl.JamesMailSpooler thread shutdown >> completed. >> 2011-09-15 17:56:30 INFO SocketNode:94 - Caught java.io.EOFException >> closing conneciton. >> >> >> Any advice would be greatly appreciated. >> >> >> >> On Thu, Sep 15, 2011 at 12:54 PM, Zach Cox <[email protected]> wrote: >>> OK now several minutes after I restarted James, everything was working >>> fine, then all of a sudden I see this while tailing james-server.log: >>> >>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Timeout >>> occurred. Now forcing the routes to be shutdown now. >>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Interrupted >>> while waiting during graceful shutdown, will force shutdown now. >>> 2011-09-15 17:51:24 WARN DefaultInflightRepository:? - Shutting down >>> while there are still 8 in flight exchanges. >>> 2011-09-15 17:51:26 INFO SocketNode:94 - Caught java.io.EOFException >>> closing conneciton. >>> >>> >>> And James just shuts down. This is incredibly bad to have a mail >>> server just shut down out of the blue... >>> >>> >>> >>> On Thu, Sep 15, 2011 at 12:46 PM, Zach Cox <[email protected]> wrote: >>>> Also when James is starting up I see this in logs: >>>> >>>> 2011-09-15 17:45:05 WARN AMQPersistenceAdapter:? - The ReferenceStore >>>> is not valid - recovering ... >>>> >>>> >>>> I'm guessing that's not a good thing? >>>> >>>> >>>> On Thu, Sep 15, 2011 at 11:54 AM, Zach Cox <[email protected]> wrote: >>>>> I don't think we're seeing the problem again yet, but I just had to >>>>> restart James and noticed this on shutdown: >>>>> >>>>> 2011-09-15 16:48:31 INFO JamesServerApplicationContext:? - Closing >>>>> org.apache.james.container.spring.context.JamesServerApplicationContext@86fe26: >>>>> startup date [Thu Sep 15 12:20:21 UTC 2011]; root of context hierarchy >>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service >>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service done >>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service >>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service done >>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service >>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service done >>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service >>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service done >>>>> 2011-09-15 16:48:31 INFO mailspooler:? - >>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... >>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-2 >>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-1 >>>>> 2011-09-15 16:48:51 INFO SocketNode:94 - Caught java.io.EOFException >>>>> closing conneciton. >>>>> >>>>> >>>>> Is that java.io.EOFException normal and expected? Or is it a sign of >>>>> some problems? >>>>> >>>>> Thanks, >>>>> Zach >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Sep 15, 2011 at 8:35 AM, Norman Maurer >>>>> <[email protected]> wrote: >>>>>> Using: >>>>>> >>>>>> # jstack <pid> >>>>>> >>>>>> >>>>>> Please take 3 within on minute ... >>>>>> >>>>>> Bye, >>>>>> Norman >>>>>> >>>>>> 2011/9/15 Zach Cox <[email protected]>: >>>>>>> I had to restart James several times this morning so nothing is >>>>>>> blocked at the moment. >>>>>>> >>>>>>> There is nothing indicated in the log files. Things look normal, and >>>>>>> then they look like this: >>>>>>> >>>>>>> 2011-09-15 01:25:26 INFO smtpserver:? - ID=8443803 Connection >>>>>>> established from x >>>>>>> 2011-09-15 01:25:31 INFO smtpserver:? - ID=8443803 Successfully >>>>>>> spooled mail Mail1316049931087-05b8199e-e01f-4d14-84d5-3d7ac2443bb3 >>>>>>> from x on x for [x] >>>>>>> 2011-09-15 01:25:36 INFO smtpserver:? - ID=8443803 Connection closed >>>>>>> for x >>>>>>> 2011-09-15 01:25:47 INFO smtpserver:? - ID=11521429 Connection >>>>>>> established from x >>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 Successfully >>>>>>> spooled mail Mail1316049948254-abd67e8f-bafe-41ab-8eef-c551420fd56e >>>>>>> from x on x for [x] >>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 Connection closed >>>>>>> for x >>>>>>> 2011-09-15 01:26:41 INFO smtpserver:? - ID=5675333 Connection >>>>>>> established from x >>>>>>> 2011-09-15 01:26:52 INFO smtpserver:? - ID=5675333 Successfully >>>>>>> spooled mail Mail1316050012621-522b1f2a-2851-46fb-940c-711591f637cc >>>>>>> from x on x for [x] >>>>>>> 2011-09-15 01:26:57 INFO smtpserver:? - ID=5675333 Connection closed >>>>>>> for x >>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 Connection >>>>>>> established from localhost (127.0.0.1) >>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 Connection closed >>>>>>> for localhost (127.0.0.1) >>>>>>> 2011-09-15 01:28:12 INFO smtpserver:? - ID=11673762 Connection >>>>>>> established from x >>>>>>> 2011-09-15 01:28:22 INFO smtpserver:? - ID=11673762 Successfully >>>>>>> spooled mail Mail1316050102393-6fcc21aa-8f32-4935-b0eb-df892d54a5e8 >>>>>>> from x on x for [x] >>>>>>> 2011-09-15 01:28:27 INFO smtpserver:? - ID=11673762 Connection closed >>>>>>> for x >>>>>>> >>>>>>> So every mail that comes in just gets spooled and not processed. >>>>>>> >>>>>>> When this starts happening again, what's the best way to take a thread >>>>>>> dump from James? >>>>>>> >>>>>>> Thanks, >>>>>>> Zach >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 15, 2011 at 8:13 AM, Norman Maurer >>>>>>> <[email protected]> wrote: >>>>>>>> Hi there, >>>>>>>> >>>>>>>> please take a thread-dump to see why its blocked... >>>>>>>> >>>>>>>> Anything in the logs ? >>>>>>>> >>>>>>>> Bye, >>>>>>>> Norman >>>>>>>> >>>>>>>> >>>>>>>> 2011/9/15 Zach Cox <[email protected]>: >>>>>>>>> We upgraded to James 3.0-beta3 on Tuesday night. Things worked fine >>>>>>>>> until around 9pm last night, where exactly the same problem started >>>>>>>>> occurring: mail would get spooled, but not processed. After >>>>>>>>> restarting James, some of the mail was then processed successfully, >>>>>>>>> but most was lost. >>>>>>>>> >>>>>>>>> We consider this an emergency situation for our company and need to >>>>>>>>> find a resolution ASAP. What kind of information can we provide to >>>>>>>>> help find a resolution? If financial resources need to be involved to >>>>>>>>> find a resolution or move things along, that is an option. >>>>>>>>> >>>>>>>>> Any help is most greatly appreciated. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Zach >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Sep 13, 2011 at 12:41 PM, Norman Maurer >>>>>>>>> <[email protected]> wrote: >>>>>>>>>> No problems... >>>>>>>>>> >>>>>>>>>> If you still can replicate then please open a issue with a stack >>>>>>>>>> trace >>>>>>>>>> attached.. >>>>>>>>>> >>>>>>>>>> Bye, >>>>>>>>>> Norman >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: >>>>>>>>>>> James 3.0-M2. I will try to stand up a new server with beta3 and >>>>>>>>>>> see if I >>>>>>>>>>> can replicate the issue. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> - Raju >>>>>>>>>>> >>>>>>>>>>> On Tue, Sep 13, 2011 at 12:37 PM, Norman Maurer < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Do you use beta3 ? >>>>>>>>>>>> >>>>>>>>>>>> Bye, >>>>>>>>>>>> Norman >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: >>>>>>>>>>>> > Hello >>>>>>>>>>>> > >>>>>>>>>>>> > We are also facing exactly the same issues. Tried to debug my >>>>>>>>>>>> > custom >>>>>>>>>>>> mailets >>>>>>>>>>>> > and seems to be everything is working fine. Logs did not show any >>>>>>>>>>>> exception >>>>>>>>>>>> > messages except it will always spool a mail and will never >>>>>>>>>>>> > attempt to >>>>>>>>>>>> send >>>>>>>>>>>> > it. i.e. all mails are stuck in the spool. >>>>>>>>>>>> > >>>>>>>>>>>> > - Raju >>>>>>>>>>>> > >>>>>>>>>>>> > On Tue, Sep 13, 2011 at 12:20 PM, Zach Cox <[email protected]> >>>>>>>>>>>> > wrote: >>>>>>>>>>>> > >>>>>>>>>>>> >> We are having a severe problem with James 3.0-M2 where it will >>>>>>>>>>>> >> operate >>>>>>>>>>>> >> correctly for hours or days, and then all of a sudden it will >>>>>>>>>>>> >> not >>>>>>>>>>>> >> process a lot of the mail it spools. Sometimes it will not >>>>>>>>>>>> >> process >>>>>>>>>>>> >> *any* mail that it spools, and other times it will spool a >>>>>>>>>>>> >> bunch of >>>>>>>>>>>> >> mail, and then process others seemingly at random. I have to >>>>>>>>>>>> >> restart >>>>>>>>>>>> >> James to get that spooled mail to be processed. But even then, >>>>>>>>>>>> >> sometimes the spooled mail is completely lost after a restart. >>>>>>>>>>>> >> I have >>>>>>>>>>>> >> seen several other threads lately with similar problems, but >>>>>>>>>>>> >> they >>>>>>>>>>>> >> haven't helped our situation. >>>>>>>>>>>> >> >>>>>>>>>>>> >> We do not store mail in James for personal accounts and we have >>>>>>>>>>>> >> IMAP & >>>>>>>>>>>> >> POP3 disabled. Every inbound email has one of two things done >>>>>>>>>>>> >> to it: >>>>>>>>>>>> >> - it is processed locally by custom mailets & matchers >>>>>>>>>>>> >> - or it is relayed to our corporate Google Apps account >>>>>>>>>>>> >> >>>>>>>>>>>> >> A previous thread recommended checking all custom mailets and >>>>>>>>>>>> >> matchers. As far as I can tell, they are all working properly. >>>>>>>>>>>> >> There >>>>>>>>>>>> >> are no exceptions or errors in log/james-server.log, and in the >>>>>>>>>>>> >> scenario where James spools some mail & processes others, the >>>>>>>>>>>> >> mail >>>>>>>>>>>> >> that is processed is handled properly by our custom mailets & >>>>>>>>>>>> >> matchers. >>>>>>>>>>>> >> >>>>>>>>>>>> >> I've tried connecting to James JMX remotely via JConsole but >>>>>>>>>>>> >> have not >>>>>>>>>>>> >> been able to connect. Which port does James run JMX on? Any >>>>>>>>>>>> >> help >>>>>>>>>>>> >> with JMX would be appreciated. >>>>>>>>>>>> >> >>>>>>>>>>>> >> Does anyone have any other ideas about what we should look for? >>>>>>>>>>>> >> I >>>>>>>>>>>> >> noticed 3.0-beta3 is out now, were there fixes in beta3 related >>>>>>>>>>>> >> to >>>>>>>>>>>> >> mail being spooled but not processed? >>>>>>>>>>>> >> >>>>>>>>>>>> >> Thanks, >>>>>>>>>>>> >> Zach >>>>>>>>>>>> >> >>>>>>>>>>>> >> --------------------------------------------------------------------- >>>>>>>>>>>> >> To unsubscribe, e-mail: [email protected] >>>>>>>>>>>> >> For additional commands, e-mail: >>>>>>>>>>>> >> [email protected] >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>> For additional commands, e-mail: [email protected] >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> >>>>> >>>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
