Hi,
Yes i am running the default Messaging config. The logging is direct from
the log, no "grep" of some sort have been done. According to the log,
nothing else runs during the delay.
I am checking activemq code right now and and it looks like the delay
appears when creating ActiveMQ Broker but i don't know wky !
André
On Mon, Aug 13, 2012 at 1:31 PM, aj...@virginia.edu <aj...@virginia.edu>wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I think that eliminates my hypothesis.
>
> Are you running with an absolutely bog-standard Messaging module config?
> Is the logging you've shown direct from the log or have you "grep"ed out
> the ActiveMQ related parts?
>
> The delay appears to occur between AMQ starting up and adding destinations
> ("localhost adding destination: topic://fedora.apim.update") which makes me
> wonder if something other than ActiveMQ startup code is causing a delay
> (e.g. if there is some kind of obscure condition between GSearch and
> Fedora).
>
> - ---
> A. Soroka
> Software & Systems Engineering :: Online Library Environment
> the University of Virginia Library
>
> On Aug 13, 2012, at 12:38 PM, André Lapointe wrote:
>
> > Here is a ActiveMQ log at startup (almost 5 minutes of hang this
> morning) :
> >
> > INFO 2012-08-13 11:23:50.545 [main] (org.fcrepo.server.Server) Loading
> bean definitions for org.fcrepo.server.storage.DefaultExternalContentManager
> > DEBUG 2012-08-13 11:23:51.033 [main]
> (org.apache.activemq.broker.jmx.ManagementContext) Probably not using JRE
> 1.4: mx4j.tools.naming.NamingService
> > DEBUG 2012-08-13 11:23:51.037 [JMX connector]
> (org.apache.activemq.broker.jmx.ManagementContext) Starting
> JMXConnectorServer...
> > INFO 2012-08-13 11:23:51.088 [JMX connector]
> (org.apache.activemq.broker.jmx.ManagementContext) JMX consoles can connect
> to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
> > INFO 2012-08-13 11:23:51.125 [main]
> (org.apache.activemq.store.kahadb.plist.PListStore)
> PListStore:/usr/local/fedora/data/activemq-data/localhost/tmp_storage
> started
> > INFO 2012-08-13 11:23:51.127 [main]
> (org.apache.activemq.broker.BrokerService) Using Persistence Adapter:
> KahaDBPersistenceAdapter[/usr/local/fedora/data/activemq-data/localhost/KahaDB]
> > INFO 2012-08-13 11:23:51.689 [main]
> (org.apache.activemq.store.kahadb.MessageDatabase) KahaDB is version 3
> > INFO 2012-08-13 11:23:51.709 [main]
> (org.apache.activemq.store.kahadb.MessageDatabase) Recovering from the
> journal ...
> > INFO 2012-08-13 11:23:51.709 [main]
> (org.apache.activemq.store.kahadb.MessageDatabase) Recovery replayed 2
> operations from the journal in 0.011 seconds.
> > INFO 2012-08-13 11:23:51.717 [main]
> (org.apache.activemq.broker.BrokerService) ActiveMQ 5.5.0 JMS Message
> Broker (localhost) is starting
> > INFO 2012-08-13 11:23:51.717 [main]
> (org.apache.activemq.broker.BrokerService) For help or more information
> please see: http://activemq.apache.org/
> > DEBUG 2012-08-13 11:28:39.938 [main]
> (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> destination: topic://fedora.apim.update
> > DEBUG 2012-08-13 11:28:39.976 [main]
> (org.apache.activemq.broker.region.TopicRegion) Restoring durable
> subscription: SubscriptionInfo {subscribedDestination =
> topic://fedora.apim.update, destination = topic://fedora.apim.update,
> clientId = fedoragsearch0, subscriptionName = fedora.apim.update, selector
> = null}
> > DEBUG 2012-08-13 11:28:40.017 [main]
> (org.apache.activemq.broker.TransportConnector) Publishing:
> tcp://localhost:61616 for broker transport URI: tcp://localhost:61616
> > DEBUG 2012-08-13 11:28:40.017 [main]
> (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> destination: topic://ActiveMQ.Advisory.Topic
> > TRACE 2012-08-13 11:28:40.025 [main]
> (org.apache.activemq.broker.jmx.ManagedRegionBroker) Sub with key:
> fedoragsearch0:fedora.apim.update, already registered
> > INFO 2012-08-13 11:28:40.030 [main]
> (org.apache.activemq.transport.TransportServerThreadSupport) Listening for
> connections at: tcp://localhost:61616
> > INFO 2012-08-13 11:28:40.030 [main]
> (org.apache.activemq.broker.TransportConnector) Connector
> tcp://localhost:61616 Started
> > INFO 2012-08-13 11:28:40.032 [main]
> (org.apache.activemq.broker.BrokerService) ActiveMQ JMS Message Broker
> (localhost, ID:corpus.ato.uqam.ca-42176-1344871431734-0:1) started
> > DEBUG 2012-08-13 11:28:40.033 [main]
> (org.apache.activemq.transport.vm.VMTransportFactory) binding to broker:
> localhost
> > INFO 2012-08-13 11:28:40.035 [main]
> (org.apache.activemq.broker.TransportConnector) Connector vm://localhost
> Started
> > DEBUG 2012-08-13 11:28:40.075 [ActiveMQ Task-2]
> (org.apache.activemq.broker.TransportConnection) Setting up new connection
> id: ID:corpus.ato.uqam.ca-42176-1344871431734-2:1, address: vm://localhost#0
> > DEBUG 2012-08-13 11:28:40.075 [ActiveMQ Task-2]
> (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> destination: topic://ActiveMQ.Advisory.Connection
> > DEBUG 2012-08-13 11:28:40.089 [ActiveMQ Task-3]
> (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> consumer: ID:corpus.ato.uqam.ca-42176-1344871431734-2:1:-1:1 for
> destination: ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic
> >
> > André
> >
> > On Mon, Aug 13, 2012 at 12:27 PM, aj...@virginia.edu <aj...@virginia.edu>
> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Is there any logging coming out of ActiveMQ right before the section
> you've shown us? I'm wondering if it is having any difficulty rebuilding
> the KahaDB store on which it relies...
> >
> > - ---
> > A. Soroka
> > Software & Systems Engineering :: Online Library Environment
> > the University of Virginia Library
> >
> > On Aug 13, 2012, at 12:18 PM, André Lapointe wrote:
> >
> > > Hi,
> > >
> > > Thanks Scott.
> > >
> > > My Fedora installation does not use queues it uses topics for
> messaging (default installation). I don't have anything stuck anywhere, I
> have check with a JMX console.
> > >
> > > GSearch is subscribed to these topics and it receives messages
> correctly. The topics are empty afterwards.
> > >
> > > Any other idea ?
> > >
> > > André
> > >
> > > On Fri, Aug 10, 2012 at 4:06 PM, Scott Prater <pra...@wisc.edu> wrote:
> > > Hello, André,
> > >
> > > The fact that the amount of time that Fedora takes to start up is
> > > variable suggests to me that ActiveMQ is trying to resend messages it
> > > has stuck in its queue. Few messages, quick startup, lots of messages,
> > > slow startup. You could verify that by deleting the messages in the
> > > queue, shutting down Fedora, starting it back up again, and seeing if
> it
> > > starts immediately; then perform a number of API-M actions (ingests,
> > > updates, etc.), examining the state of the queue, shutting down Fedora,
> > > starting it up again, and see if it takes a while.
> > >
> > > See http://activemq.apache.org/how-do-i-purge-a-queue.html for
> > > information on purging a queue.
> > >
> > > Once you've verified that is, in fact, the problem, then you'll need to
> > > see why messages are getting stuck in the queue: perhaps no configured
> > > message consumer?
> > >
> > > -- Scott
> > >
> > > On 08/10/2012 02:15 PM, André Lapointe wrote:
> > > > Hi,
> > > >
> > > > On Tomcat startup, our fedora installation hangs for a few minutes,
> > > > sometimes for as long as an hour. It seems that ActiveMQ is the
> problem
> > > > because when its turned off, the startup executes in a few second.
> > > >
> > > > I have turn on TRACE logging for ActiveMQ and here is the log traces
> > > > before and after the hangup :
> > > >
> > > > INFO 2012-08-10 10:34:44.059 [main]
> > > > (org.apache.activemq.store.kahadb.MessageDatabase) Recovery replayed
> 2
> > > > operations from the journal in 0.017 seconds.
> > > > INFO 2012-08-10 10:34:44.065 [main]
> > > > (org.apache.activemq.broker.BrokerService) ActiveMQ 5.5.0 JMS Message
> > > > Broker (localhost) is starting
> > > > INFO 2012-08-10 10:34:44.065 [main]
> > > > (org.apache.activemq.broker.BrokerService) For help or more
> information
> > > > please see: http://activemq.apache.org/
> > > > DEBUG 2012-08-10 10:45:28.699 [main]
> > > > (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> > > > destination: topic://fedora.apim.update
> > > > DEBUG 2012-08-10 10:45:28.736 [main]
> > > > (org.apache.activemq.broker.region.TopicRegion) Restoring durable
> > > > subscription: SubscriptionInfo {subscribedDestination =
> > > > topic://fedora.apim.update, destination = topic://fedora.apim.update,
> > > > clientId = fedoragsearch0, subscriptionName = fedora.apim.update,
> > > > selector = null}
> > > > DEBUG 2012-08-10 10:45:28.776 [main]
> > > > (org.apache.activemq.broker.TransportConnector) Publishing:
> > > > tcp://localhost:61616 for broker transport URI: tcp://localhost:61616
> > > > DEBUG 2012-08-10 10:45:28.777 [main]
> > > > (org.apache.activemq.broker.region.AbstractRegion) localhost adding
> > > > destination: topic://ActiveMQ.Advisory.Topic
> > > > TRACE 2012-08-10 10:45:28.785 [main]
> > > > (org.apache.activemq.broker.jmx.ManagedRegionBroker) Sub with key:
> > > > fedoragsearch0:fedora.apim.update, already registered
> > > > INFO 2012-08-10 10:45:28.790 [main]
> > > > (org.apache.activemq.transport.TransportServerThreadSupport)
> Listening
> > > > for connections at: tcp://localhost:61616
> > > >
> > > > In this case it hung for a little bit more than 10 minutes. I've been
> > > > looking for a solution all day long and I can't find anything. Any
> idea ?
> > > >
> > > > Also, I have tried to remove everything in
> > > > $FEDORA_HOME/data/activemq-data/localhost/KahaDB/ and it rebuilds
> itself
> > > > at startup fast. Where does the info is taken to rebuild the
> activemq
> > > > data ? In the FOXML ?
> > > >
> > > > Thanks,
> > > >
> > > > André
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> ------------------------------------------------------------------------------
> > > > Live Security Virtual Conference
> > > > Exclusive live event will cover all the ways today's security and
> > > > threat landscape has changed and how IT managers can respond.
> Discussions
> > > > will include endpoint security, mobile security and the latest in
> malware
> > > > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Fedora-commons-users mailing list
> > > > Fedora-commons-users@lists.sourceforge.net
> > > > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> > >
> > >
> > > --
> > > Scott Prater
> > > Shared Development Group
> > > General Library System
> > > University of Wisconsin - Madison
> > > pra...@wisc.edu
> > > 5-5415
> > >
> > >
> > >
> ------------------------------------------------------------------------------
> > > Live Security Virtual Conference
> > > Exclusive live event will cover all the ways today's security and
> > > threat landscape has changed and how IT managers can respond.
> Discussions
> > > will include endpoint security, mobile security and the latest in
> malware
> > > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > > _______________________________________________
> > > Fedora-commons-users mailing list
> > > Fedora-commons-users@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> > >
> > >
> ------------------------------------------------------------------------------
> > > Live Security Virtual Conference
> > > Exclusive live event will cover all the ways today's security and
> > > threat landscape has changed and how IT managers can respond.
> Discussions
> > > will include endpoint security, mobile security and the latest in
> malware
> > > threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
> > > Fedora-commons-users mailing list
> > > Fedora-commons-users@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> > Comment: GPGTools - http://gpgtools.org
> >
> > iQEcBAEBAgAGBQJQKSrzAAoJEATpPYSyaoIks1YH+wWyUWTcBiYYA3Hita4s1PV9
> > 4mZatEonfa4jfZp0WtNVZwsDRmiAcgTbenLJwTkx6BbLHt2C+ufM5o2IgHygkhKZ
> > Wl8ZB75f5AMI7idngD4YXA3k3J95NUmwvtrRxMmcaivV3j5i8fYHkbol3xA4QAM+
> > UQywoKeDP4wZolpGZrIRWUu39UiaEvQvbBqaHCLk9koiB2dpRwTLvYq5NAYW3s00
> > uenSKJW+znL7aOI4Tb251zmFmTCk1TkTQI6oeypmlTNm4vvr5tsS8B9MRT+HxMTB
> > EtM2FT3PGuLBwFqzmos4UCNoMYcTPv/atvjOt09vcjeIM4qezbz76yiaYAWhy3M=
> > =flZW
> > -----END PGP SIGNATURE-----
> >
> >
> ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Fedora-commons-users mailing list
> > Fedora-commons-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >
> >
> ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
> > Fedora-commons-users mailing list
> > Fedora-commons-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
>
> iQEcBAEBAgAGBQJQKTn1AAoJEATpPYSyaoIk0moIAIVczAvOaqDFihrciTfhavHk
> p5X2zdOD50EpUnEqMl4MGt+Dl3kJuRj3I2kMYH7fmMaiK1KZME2NiEVCSiM9MuQm
> paB5OlBUyPSxggcqrTbucVJKlPK9iQbbJHpUt7Df7Rj+rnqGn9hm0ji79WXa80tu
> Tj19BCaWjl8MfGkT/IglRbe/Qn5aE2T5A1UGIVoMP0jxs2J1Jy9rDtf0nK7k1aPz
> LFvBooAGG2eNCZjH/4gGzrfwDS91FdHaLuF4d4ozOaGwY98ycE1vA+QnbT/oD/Ig
> 2815iLBoMRmyZ7s1VW+ToTi62qcYaJfvZjx8OhRrWBzaU4MAFde2CRoUfTGr/80=
> =ZTxi
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
