OK so the cause of this shutdown problem was setting jmx.address= public IP address of the server in conf/jmx.properties. Just had the exact same problem on a fresh install of James. I changed jmx.address back to 127.0.0.1 and James now starts up properly.
There is also no mention of anything related to jmx in the logs when James is unable to start up because of using the public IP address for jmx.address. On Thu, Sep 15, 2011 at 1:02 PM, Zach Cox <[email protected]> wrote: > Sorry, had to replace that EC2 instance with a new one, need to keep > emails flowing... > > So that server is gone, but we still have all of the logs. > > If this happens again I will try moving activemq dir. > > Thanks, > Zach > > > On Thu, Sep 15, 2011 at 1:00 PM, Norman Maurer > <[email protected]> wrote: >> Hmm.. >> >> never saw this problem. But could you try to move the activemq dir out >> of the way. >> >> # mv /path/to/james/var/store/activemq /path/to/james/var/store/activemq.crc >> >> After that try to start again.. >> >> Bye, >> Norman >> >> >> 2011/9/15 Zach Cox <[email protected]>: >>> And now every time I start James, after it completes startup, it just >>> shuts down: >>> >>> >>> 2011-09-15 17:56:26 INFO smtpserver:? - Init SMTP Service done >>> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - >>> ConfigurationUtils.locate(): base is null, name is null >>> 2011-09-15 17:56:26 INFO lmtpserver:? - LMTP Service disabled by >>> configuration >>> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - >>> ConfigurationUtils.locate(): base is null, name is null >>> 2011-09-15 17:56:26 INFO pop3server:? - POP3 Service disabled by >>> configuration >>> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - >>> ConfigurationUtils.locate(): base is null, name is null >>> 2011-09-15 17:56:27 INFO imapserver:? - IMAP Service disabled by >>> configuration >>> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - >>> ConfigurationUtils.locate(): base is null, name is null >>> 2011-09-15 17:56:27 INFO usersrepository23:? - Init >>> org.apache.james.repository.file.FilePersistentObjectRepository Store >>> 2011-09-15 17:56:27 INFO usersrepository23:? - >>> org.apache.james.repository.file.FilePersistentObjectRepository opened >>> in ../var/users >>> 2011-09-15 17:56:27 DEBUG usersrepository23:? - >>> org.apache.james.user.file.UsersFileRepository created in >>> file://var/users/ >>> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service >>> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service done >>> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service >>> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service done >>> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service >>> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service done >>> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service >>> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service done >>> 2011-09-15 17:56:27 INFO mailspooler:? - >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... >>> 2011-09-15 17:56:27 INFO mailspooler:? - Stop >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-1 >>> 2011-09-15 17:56:27 INFO mailspooler:? - Stop >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-2 >>> 2011-09-15 17:56:27 INFO mailspooler:? - >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler thread shutdown >>> completed. >>> 2011-09-15 17:56:30 INFO SocketNode:94 - Caught java.io.EOFException >>> closing conneciton. >>> >>> >>> Any advice would be greatly appreciated. >>> >>> >>> >>> On Thu, Sep 15, 2011 at 12:54 PM, Zach Cox <[email protected]> wrote: >>>> OK now several minutes after I restarted James, everything was working >>>> fine, then all of a sudden I see this while tailing james-server.log: >>>> >>>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Timeout >>>> occurred. Now forcing the routes to be shutdown now. >>>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Interrupted >>>> while waiting during graceful shutdown, will force shutdown now. >>>> 2011-09-15 17:51:24 WARN DefaultInflightRepository:? - Shutting down >>>> while there are still 8 in flight exchanges. >>>> 2011-09-15 17:51:26 INFO SocketNode:94 - Caught java.io.EOFException >>>> closing conneciton. >>>> >>>> >>>> And James just shuts down. This is incredibly bad to have a mail >>>> server just shut down out of the blue... >>>> >>>> >>>> >>>> On Thu, Sep 15, 2011 at 12:46 PM, Zach Cox <[email protected]> wrote: >>>>> Also when James is starting up I see this in logs: >>>>> >>>>> 2011-09-15 17:45:05 WARN AMQPersistenceAdapter:? - The ReferenceStore >>>>> is not valid - recovering ... >>>>> >>>>> >>>>> I'm guessing that's not a good thing? >>>>> >>>>> >>>>> On Thu, Sep 15, 2011 at 11:54 AM, Zach Cox <[email protected]> wrote: >>>>>> I don't think we're seeing the problem again yet, but I just had to >>>>>> restart James and noticed this on shutdown: >>>>>> >>>>>> 2011-09-15 16:48:31 INFO JamesServerApplicationContext:? - Closing >>>>>> org.apache.james.container.spring.context.JamesServerApplicationContext@86fe26: >>>>>> startup date [Thu Sep 15 12:20:21 UTC 2011]; root of context hierarchy >>>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service >>>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service done >>>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service >>>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service done >>>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service >>>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service done >>>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service >>>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service done >>>>>> 2011-09-15 16:48:31 INFO mailspooler:? - >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... >>>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-2 >>>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-1 >>>>>> 2011-09-15 16:48:51 INFO SocketNode:94 - Caught java.io.EOFException >>>>>> closing conneciton. >>>>>> >>>>>> >>>>>> Is that java.io.EOFException normal and expected? Or is it a sign of >>>>>> some problems? >>>>>> >>>>>> Thanks, >>>>>> Zach >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 15, 2011 at 8:35 AM, Norman Maurer >>>>>> <[email protected]> wrote: >>>>>>> Using: >>>>>>> >>>>>>> # jstack <pid> >>>>>>> >>>>>>> >>>>>>> Please take 3 within on minute ... >>>>>>> >>>>>>> Bye, >>>>>>> Norman >>>>>>> >>>>>>> 2011/9/15 Zach Cox <[email protected]>: >>>>>>>> I had to restart James several times this morning so nothing is >>>>>>>> blocked at the moment. >>>>>>>> >>>>>>>> There is nothing indicated in the log files. Things look normal, and >>>>>>>> then they look like this: >>>>>>>> >>>>>>>> 2011-09-15 01:25:26 INFO smtpserver:? - ID=8443803 Connection >>>>>>>> established from x >>>>>>>> 2011-09-15 01:25:31 INFO smtpserver:? - ID=8443803 Successfully >>>>>>>> spooled mail Mail1316049931087-05b8199e-e01f-4d14-84d5-3d7ac2443bb3 >>>>>>>> from x on x for [x] >>>>>>>> 2011-09-15 01:25:36 INFO smtpserver:? - ID=8443803 Connection closed >>>>>>>> for x >>>>>>>> 2011-09-15 01:25:47 INFO smtpserver:? - ID=11521429 Connection >>>>>>>> established from x >>>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 Successfully >>>>>>>> spooled mail Mail1316049948254-abd67e8f-bafe-41ab-8eef-c551420fd56e >>>>>>>> from x on x for [x] >>>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 Connection closed >>>>>>>> for x >>>>>>>> 2011-09-15 01:26:41 INFO smtpserver:? - ID=5675333 Connection >>>>>>>> established from x >>>>>>>> 2011-09-15 01:26:52 INFO smtpserver:? - ID=5675333 Successfully >>>>>>>> spooled mail Mail1316050012621-522b1f2a-2851-46fb-940c-711591f637cc >>>>>>>> from x on x for [x] >>>>>>>> 2011-09-15 01:26:57 INFO smtpserver:? - ID=5675333 Connection closed >>>>>>>> for x >>>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 Connection >>>>>>>> established from localhost (127.0.0.1) >>>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 Connection closed >>>>>>>> for localhost (127.0.0.1) >>>>>>>> 2011-09-15 01:28:12 INFO smtpserver:? - ID=11673762 Connection >>>>>>>> established from x >>>>>>>> 2011-09-15 01:28:22 INFO smtpserver:? - ID=11673762 Successfully >>>>>>>> spooled mail Mail1316050102393-6fcc21aa-8f32-4935-b0eb-df892d54a5e8 >>>>>>>> from x on x for [x] >>>>>>>> 2011-09-15 01:28:27 INFO smtpserver:? - ID=11673762 Connection closed >>>>>>>> for x >>>>>>>> >>>>>>>> So every mail that comes in just gets spooled and not processed. >>>>>>>> >>>>>>>> When this starts happening again, what's the best way to take a thread >>>>>>>> dump from James? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Zach >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 15, 2011 at 8:13 AM, Norman Maurer >>>>>>>> <[email protected]> wrote: >>>>>>>>> Hi there, >>>>>>>>> >>>>>>>>> please take a thread-dump to see why its blocked... >>>>>>>>> >>>>>>>>> Anything in the logs ? >>>>>>>>> >>>>>>>>> Bye, >>>>>>>>> Norman >>>>>>>>> >>>>>>>>> >>>>>>>>> 2011/9/15 Zach Cox <[email protected]>: >>>>>>>>>> We upgraded to James 3.0-beta3 on Tuesday night. Things worked fine >>>>>>>>>> until around 9pm last night, where exactly the same problem started >>>>>>>>>> occurring: mail would get spooled, but not processed. After >>>>>>>>>> restarting James, some of the mail was then processed successfully, >>>>>>>>>> but most was lost. >>>>>>>>>> >>>>>>>>>> We consider this an emergency situation for our company and need to >>>>>>>>>> find a resolution ASAP. What kind of information can we provide to >>>>>>>>>> help find a resolution? If financial resources need to be involved to >>>>>>>>>> find a resolution or move things along, that is an option. >>>>>>>>>> >>>>>>>>>> Any help is most greatly appreciated. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Zach >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Sep 13, 2011 at 12:41 PM, Norman Maurer >>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>> No problems... >>>>>>>>>>> >>>>>>>>>>> If you still can replicate then please open a issue with a stack >>>>>>>>>>> trace >>>>>>>>>>> attached.. >>>>>>>>>>> >>>>>>>>>>> Bye, >>>>>>>>>>> Norman >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: >>>>>>>>>>>> James 3.0-M2. I will try to stand up a new server with beta3 and >>>>>>>>>>>> see if I >>>>>>>>>>>> can replicate the issue. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> - Raju >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Sep 13, 2011 at 12:37 PM, Norman Maurer < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Do you use beta3 ? >>>>>>>>>>>>> >>>>>>>>>>>>> Bye, >>>>>>>>>>>>> Norman >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: >>>>>>>>>>>>> > Hello >>>>>>>>>>>>> > >>>>>>>>>>>>> > We are also facing exactly the same issues. Tried to debug my >>>>>>>>>>>>> > custom >>>>>>>>>>>>> mailets >>>>>>>>>>>>> > and seems to be everything is working fine. Logs did not show >>>>>>>>>>>>> > any >>>>>>>>>>>>> exception >>>>>>>>>>>>> > messages except it will always spool a mail and will never >>>>>>>>>>>>> > attempt to >>>>>>>>>>>>> send >>>>>>>>>>>>> > it. i.e. all mails are stuck in the spool. >>>>>>>>>>>>> > >>>>>>>>>>>>> > - Raju >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Tue, Sep 13, 2011 at 12:20 PM, Zach Cox <[email protected]> >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> >> We are having a severe problem with James 3.0-M2 where it will >>>>>>>>>>>>> >> operate >>>>>>>>>>>>> >> correctly for hours or days, and then all of a sudden it will >>>>>>>>>>>>> >> not >>>>>>>>>>>>> >> process a lot of the mail it spools. Sometimes it will not >>>>>>>>>>>>> >> process >>>>>>>>>>>>> >> *any* mail that it spools, and other times it will spool a >>>>>>>>>>>>> >> bunch of >>>>>>>>>>>>> >> mail, and then process others seemingly at random. I have to >>>>>>>>>>>>> >> restart >>>>>>>>>>>>> >> James to get that spooled mail to be processed. But even then, >>>>>>>>>>>>> >> sometimes the spooled mail is completely lost after a restart. >>>>>>>>>>>>> >> I have >>>>>>>>>>>>> >> seen several other threads lately with similar problems, but >>>>>>>>>>>>> >> they >>>>>>>>>>>>> >> haven't helped our situation. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> We do not store mail in James for personal accounts and we >>>>>>>>>>>>> >> have IMAP & >>>>>>>>>>>>> >> POP3 disabled. Every inbound email has one of two things done >>>>>>>>>>>>> >> to it: >>>>>>>>>>>>> >> - it is processed locally by custom mailets & matchers >>>>>>>>>>>>> >> - or it is relayed to our corporate Google Apps account >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> A previous thread recommended checking all custom mailets and >>>>>>>>>>>>> >> matchers. As far as I can tell, they are all working >>>>>>>>>>>>> >> properly. There >>>>>>>>>>>>> >> are no exceptions or errors in log/james-server.log, and in the >>>>>>>>>>>>> >> scenario where James spools some mail & processes others, the >>>>>>>>>>>>> >> mail >>>>>>>>>>>>> >> that is processed is handled properly by our custom mailets & >>>>>>>>>>>>> >> matchers. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> I've tried connecting to James JMX remotely via JConsole but >>>>>>>>>>>>> >> have not >>>>>>>>>>>>> >> been able to connect. Which port does James run JMX on? Any >>>>>>>>>>>>> >> help >>>>>>>>>>>>> >> with JMX would be appreciated. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Does anyone have any other ideas about what we should look >>>>>>>>>>>>> >> for? I >>>>>>>>>>>>> >> noticed 3.0-beta3 is out now, were there fixes in beta3 >>>>>>>>>>>>> >> related to >>>>>>>>>>>>> >> mail being spooled but not processed? >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>> >> Zach >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> --------------------------------------------------------------------- >>>>>>>>>>>>> >> To unsubscribe, e-mail: >>>>>>>>>>>>> >> [email protected] >>>>>>>>>>>>> >> For additional commands, e-mail: >>>>>>>>>>>>> >> [email protected] >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>> For additional commands, e-mail: [email protected] >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
