The mails are now stuck up in spool on my server. Can any one tell me how to debug. I tried to use jstack, but it says jstack: command not found.
- Raju On Thu, Sep 15, 2011 at 1:16 PM, Zach Cox <[email protected]> wrote: > OK so the cause of this shutdown problem was setting jmx.address= > public IP address of the server in conf/jmx.properties. Just had the > exact same problem on a fresh install of James. I changed jmx.address > back to 127.0.0.1 and James now starts up properly. > > There is also no mention of anything related to jmx in the logs when > James is unable to start up because of using the public IP address for > jmx.address. > > > > > On Thu, Sep 15, 2011 at 1:02 PM, Zach Cox <[email protected]> wrote: > > Sorry, had to replace that EC2 instance with a new one, need to keep > > emails flowing... > > > > So that server is gone, but we still have all of the logs. > > > > If this happens again I will try moving activemq dir. > > > > Thanks, > > Zach > > > > > > On Thu, Sep 15, 2011 at 1:00 PM, Norman Maurer > > <[email protected]> wrote: > >> Hmm.. > >> > >> never saw this problem. But could you try to move the activemq dir out > >> of the way. > >> > >> # mv /path/to/james/var/store/activemq > /path/to/james/var/store/activemq.crc > >> > >> After that try to start again.. > >> > >> Bye, > >> Norman > >> > >> > >> 2011/9/15 Zach Cox <[email protected]>: > >>> And now every time I start James, after it completes startup, it just > >>> shuts down: > >>> > >>> > >>> 2011-09-15 17:56:26 INFO smtpserver:? - Init SMTP Service done > >>> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - > >>> ConfigurationUtils.locate(): base is null, name is null > >>> 2011-09-15 17:56:26 INFO lmtpserver:? - LMTP Service disabled by > configuration > >>> 2011-09-15 17:56:26 DEBUG ConfigurationUtils:? - > >>> ConfigurationUtils.locate(): base is null, name is null > >>> 2011-09-15 17:56:26 INFO pop3server:? - POP3 Service disabled by > configuration > >>> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - > >>> ConfigurationUtils.locate(): base is null, name is null > >>> 2011-09-15 17:56:27 INFO imapserver:? - IMAP Service disabled by > configuration > >>> 2011-09-15 17:56:27 DEBUG ConfigurationUtils:? - > >>> ConfigurationUtils.locate(): base is null, name is null > >>> 2011-09-15 17:56:27 INFO usersrepository23:? - Init > >>> org.apache.james.repository.file.FilePersistentObjectRepository Store > >>> 2011-09-15 17:56:27 INFO usersrepository23:? - > >>> org.apache.james.repository.file.FilePersistentObjectRepository opened > >>> in ../var/users > >>> 2011-09-15 17:56:27 DEBUG usersrepository23:? - > >>> org.apache.james.user.file.UsersFileRepository created in > >>> file://var/users/ > >>> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service > >>> 2011-09-15 17:56:27 INFO imapserver:? - Dispose IMAP Service done > >>> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service > >>> 2011-09-15 17:56:27 INFO pop3server:? - Dispose POP3 Service done > >>> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service > >>> 2011-09-15 17:56:27 INFO lmtpserver:? - Dispose LMTP Service done > >>> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service > >>> 2011-09-15 17:56:27 INFO smtpserver:? - Dispose SMTP Service done > >>> 2011-09-15 17:56:27 INFO mailspooler:? - > >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... > >>> 2011-09-15 17:56:27 INFO mailspooler:? - Stop > >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-1 > >>> 2011-09-15 17:56:27 INFO mailspooler:? - Stop > >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-2 > >>> 2011-09-15 17:56:27 INFO mailspooler:? - > >>> org.apache.james.mailetcontainer.impl.JamesMailSpooler thread shutdown > >>> completed. > >>> 2011-09-15 17:56:30 INFO SocketNode:94 - Caught java.io.EOFException > >>> closing conneciton. > >>> > >>> > >>> Any advice would be greatly appreciated. > >>> > >>> > >>> > >>> On Thu, Sep 15, 2011 at 12:54 PM, Zach Cox <[email protected]> wrote: > >>>> OK now several minutes after I restarted James, everything was working > >>>> fine, then all of a sudden I see this while tailing james-server.log: > >>>> > >>>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Timeout > >>>> occurred. Now forcing the routes to be shutdown now. > >>>> 2011-09-15 17:51:24 WARN DefaultShutdownStrategy:? - Interrupted > >>>> while waiting during graceful shutdown, will force shutdown now. > >>>> 2011-09-15 17:51:24 WARN DefaultInflightRepository:? - Shutting down > >>>> while there are still 8 in flight exchanges. > >>>> 2011-09-15 17:51:26 INFO SocketNode:94 - Caught java.io.EOFException > >>>> closing conneciton. > >>>> > >>>> > >>>> And James just shuts down. This is incredibly bad to have a mail > >>>> server just shut down out of the blue... > >>>> > >>>> > >>>> > >>>> On Thu, Sep 15, 2011 at 12:46 PM, Zach Cox <[email protected]> wrote: > >>>>> Also when James is starting up I see this in logs: > >>>>> > >>>>> 2011-09-15 17:45:05 WARN AMQPersistenceAdapter:? - The > ReferenceStore > >>>>> is not valid - recovering ... > >>>>> > >>>>> > >>>>> I'm guessing that's not a good thing? > >>>>> > >>>>> > >>>>> On Thu, Sep 15, 2011 at 11:54 AM, Zach Cox <[email protected]> > wrote: > >>>>>> I don't think we're seeing the problem again yet, but I just had to > >>>>>> restart James and noticed this on shutdown: > >>>>>> > >>>>>> 2011-09-15 16:48:31 INFO JamesServerApplicationContext:? - Closing > >>>>>> > org.apache.james.container.spring.context.JamesServerApplicationContext@86fe26 > : > >>>>>> startup date [Thu Sep 15 12:20:21 UTC 2011]; root of context > hierarchy > >>>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service > >>>>>> 2011-09-15 16:48:31 INFO imapserver:? - Dispose IMAP Service done > >>>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service > >>>>>> 2011-09-15 16:48:31 INFO pop3server:? - Dispose POP3 Service done > >>>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service > >>>>>> 2011-09-15 16:48:31 INFO lmtpserver:? - Dispose LMTP Service done > >>>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service > >>>>>> 2011-09-15 16:48:31 INFO smtpserver:? - Dispose SMTP Service done > >>>>>> 2011-09-15 16:48:31 INFO mailspooler:? - > >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler dispose... > >>>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop > >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-2 > >>>>>> 2011-09-15 16:48:32 INFO mailspooler:? - Stop > >>>>>> org.apache.james.mailetcontainer.impl.JamesMailSpooler: dequeuer-1 > >>>>>> 2011-09-15 16:48:51 INFO SocketNode:94 - Caught > java.io.EOFException > >>>>>> closing conneciton. > >>>>>> > >>>>>> > >>>>>> Is that java.io.EOFException normal and expected? Or is it a sign of > >>>>>> some problems? > >>>>>> > >>>>>> Thanks, > >>>>>> Zach > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Thu, Sep 15, 2011 at 8:35 AM, Norman Maurer > >>>>>> <[email protected]> wrote: > >>>>>>> Using: > >>>>>>> > >>>>>>> # jstack <pid> > >>>>>>> > >>>>>>> > >>>>>>> Please take 3 within on minute ... > >>>>>>> > >>>>>>> Bye, > >>>>>>> Norman > >>>>>>> > >>>>>>> 2011/9/15 Zach Cox <[email protected]>: > >>>>>>>> I had to restart James several times this morning so nothing is > >>>>>>>> blocked at the moment. > >>>>>>>> > >>>>>>>> There is nothing indicated in the log files. Things look normal, > and > >>>>>>>> then they look like this: > >>>>>>>> > >>>>>>>> 2011-09-15 01:25:26 INFO smtpserver:? - ID=8443803 Connection > >>>>>>>> established from x > >>>>>>>> 2011-09-15 01:25:31 INFO smtpserver:? - ID=8443803 Successfully > >>>>>>>> spooled mail > Mail1316049931087-05b8199e-e01f-4d14-84d5-3d7ac2443bb3 > >>>>>>>> from x on x for [x] > >>>>>>>> 2011-09-15 01:25:36 INFO smtpserver:? - ID=8443803 Connection > closed for x > >>>>>>>> 2011-09-15 01:25:47 INFO smtpserver:? - ID=11521429 Connection > >>>>>>>> established from x > >>>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 Successfully > >>>>>>>> spooled mail > Mail1316049948254-abd67e8f-bafe-41ab-8eef-c551420fd56e > >>>>>>>> from x on x for [x] > >>>>>>>> 2011-09-15 01:25:48 INFO smtpserver:? - ID=11521429 Connection > closed for x > >>>>>>>> 2011-09-15 01:26:41 INFO smtpserver:? - ID=5675333 Connection > >>>>>>>> established from x > >>>>>>>> 2011-09-15 01:26:52 INFO smtpserver:? - ID=5675333 Successfully > >>>>>>>> spooled mail > Mail1316050012621-522b1f2a-2851-46fb-940c-711591f637cc > >>>>>>>> from x on x for [x] > >>>>>>>> 2011-09-15 01:26:57 INFO smtpserver:? - ID=5675333 Connection > closed for x > >>>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 Connection > >>>>>>>> established from localhost (127.0.0.1) > >>>>>>>> 2011-09-15 01:27:52 INFO smtpserver:? - ID=33404412 Connection > closed > >>>>>>>> for localhost (127.0.0.1) > >>>>>>>> 2011-09-15 01:28:12 INFO smtpserver:? - ID=11673762 Connection > >>>>>>>> established from x > >>>>>>>> 2011-09-15 01:28:22 INFO smtpserver:? - ID=11673762 Successfully > >>>>>>>> spooled mail > Mail1316050102393-6fcc21aa-8f32-4935-b0eb-df892d54a5e8 > >>>>>>>> from x on x for [x] > >>>>>>>> 2011-09-15 01:28:27 INFO smtpserver:? - ID=11673762 Connection > closed for x > >>>>>>>> > >>>>>>>> So every mail that comes in just gets spooled and not processed. > >>>>>>>> > >>>>>>>> When this starts happening again, what's the best way to take a > thread > >>>>>>>> dump from James? > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Zach > >>>>>>>> > >>>>>>>> > >>>>>>>> On Thu, Sep 15, 2011 at 8:13 AM, Norman Maurer > >>>>>>>> <[email protected]> wrote: > >>>>>>>>> Hi there, > >>>>>>>>> > >>>>>>>>> please take a thread-dump to see why its blocked... > >>>>>>>>> > >>>>>>>>> Anything in the logs ? > >>>>>>>>> > >>>>>>>>> Bye, > >>>>>>>>> Norman > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> 2011/9/15 Zach Cox <[email protected]>: > >>>>>>>>>> We upgraded to James 3.0-beta3 on Tuesday night. Things worked > fine > >>>>>>>>>> until around 9pm last night, where exactly the same problem > started > >>>>>>>>>> occurring: mail would get spooled, but not processed. After > >>>>>>>>>> restarting James, some of the mail was then processed > successfully, > >>>>>>>>>> but most was lost. > >>>>>>>>>> > >>>>>>>>>> We consider this an emergency situation for our company and need > to > >>>>>>>>>> find a resolution ASAP. What kind of information can we provide > to > >>>>>>>>>> help find a resolution? If financial resources need to be > involved to > >>>>>>>>>> find a resolution or move things along, that is an option. > >>>>>>>>>> > >>>>>>>>>> Any help is most greatly appreciated. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Zach > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Tue, Sep 13, 2011 at 12:41 PM, Norman Maurer > >>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>> No problems... > >>>>>>>>>>> > >>>>>>>>>>> If you still can replicate then please open a issue with a > stack trace > >>>>>>>>>>> attached.. > >>>>>>>>>>> > >>>>>>>>>>> Bye, > >>>>>>>>>>> Norman > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: > >>>>>>>>>>>> James 3.0-M2. I will try to stand up a new server with beta3 > and see if I > >>>>>>>>>>>> can replicate the issue. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks > >>>>>>>>>>>> > >>>>>>>>>>>> - Raju > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Sep 13, 2011 at 12:37 PM, Norman Maurer < > >>>>>>>>>>>> [email protected]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Do you use beta3 ? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Bye, > >>>>>>>>>>>>> Norman > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> 2011/9/13 Raju Buchi <[email protected]>: > >>>>>>>>>>>>> > Hello > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > We are also facing exactly the same issues. Tried to debug > my custom > >>>>>>>>>>>>> mailets > >>>>>>>>>>>>> > and seems to be everything is working fine. Logs did not > show any > >>>>>>>>>>>>> exception > >>>>>>>>>>>>> > messages except it will always spool a mail and will never > attempt to > >>>>>>>>>>>>> send > >>>>>>>>>>>>> > it. i.e. all mails are stuck in the spool. > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > - Raju > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > On Tue, Sep 13, 2011 at 12:20 PM, Zach Cox < > [email protected]> wrote: > >>>>>>>>>>>>> > > >>>>>>>>>>>>> >> We are having a severe problem with James 3.0-M2 where it > will operate > >>>>>>>>>>>>> >> correctly for hours or days, and then all of a sudden it > will not > >>>>>>>>>>>>> >> process a lot of the mail it spools. Sometimes it will > not process > >>>>>>>>>>>>> >> *any* mail that it spools, and other times it will spool a > bunch of > >>>>>>>>>>>>> >> mail, and then process others seemingly at random. I have > to restart > >>>>>>>>>>>>> >> James to get that spooled mail to be processed. But even > then, > >>>>>>>>>>>>> >> sometimes the spooled mail is completely lost after a > restart. I have > >>>>>>>>>>>>> >> seen several other threads lately with similar problems, > but they > >>>>>>>>>>>>> >> haven't helped our situation. > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> We do not store mail in James for personal accounts and we > have IMAP & > >>>>>>>>>>>>> >> POP3 disabled. Every inbound email has one of two things > done to it: > >>>>>>>>>>>>> >> - it is processed locally by custom mailets & matchers > >>>>>>>>>>>>> >> - or it is relayed to our corporate Google Apps account > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> A previous thread recommended checking all custom mailets > and > >>>>>>>>>>>>> >> matchers. As far as I can tell, they are all working > properly. There > >>>>>>>>>>>>> >> are no exceptions or errors in log/james-server.log, and > in the > >>>>>>>>>>>>> >> scenario where James spools some mail & processes others, > the mail > >>>>>>>>>>>>> >> that is processed is handled properly by our custom > mailets & > >>>>>>>>>>>>> >> matchers. > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> I've tried connecting to James JMX remotely via JConsole > but have not > >>>>>>>>>>>>> >> been able to connect. Which port does James run JMX on? > Any help > >>>>>>>>>>>>> >> with JMX would be appreciated. > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> Does anyone have any other ideas about what we should look > for? I > >>>>>>>>>>>>> >> noticed 3.0-beta3 is out now, were there fixes in beta3 > related to > >>>>>>>>>>>>> >> mail being spooled but not processed? > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> Thanks, > >>>>>>>>>>>>> >> Zach > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > --------------------------------------------------------------------- > >>>>>>>>>>>>> >> To unsubscribe, e-mail: > [email protected] > >>>>>>>>>>>>> >> For additional commands, e-mail: > [email protected] > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > >>>>>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>>>>> To unsubscribe, e-mail: > [email protected] > >>>>>>>>>>>>> For additional commands, e-mail: > [email protected] > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>>> To unsubscribe, e-mail: > [email protected] > >>>>>>>>>>> For additional commands, e-mail: > [email protected] > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>> To unsubscribe, e-mail: > [email protected] > >>>>>>>>>> For additional commands, e-mail: > [email protected] > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>>>> For additional commands, e-mail: > [email protected] > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > --------------------------------------------------------------------- > >>>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>>> For additional commands, e-mail: > [email protected] > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > --------------------------------------------------------------------- > >>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>> For additional commands, e-mail: [email protected] > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: [email protected] > >>> For additional commands, e-mail: [email protected] > >>> > >>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
