Re: Occasional freeze on startup

Jean-Baptiste Onofré Wed, 11 Sep 2013 06:42:51 -0700

Thanks for the update Minto.

Could you check that blueprint.synchronous=true in etc/config.propertiesand you have respectStartLvl in etc/org.apache.karaf.features.cfg ?


Regards
JB

On 09/11/2013 03:35 PM, Minto van der Sluis wrote:

Hi JB,

Thanks for your response.

Sorry, I should have mentioned the Karaf version that I use. Initially I
used Karaf 2.3.0. First thing I did when I ran into these issues was
upgrading both Karaf and CXF to their latest released versions. For
Karaf this is 2.3.2 and for CXF it is 2.7.6. Since then I have NOT
noticed the mentioned behaviour on my development machine.

Unfortunately the server we intend to use for production has since
refused to start up properly due to freeze 1. The attached log in my
previous message was from that server.

Is there some additional logging that I can turn on to see what is
happening under the hood?

Regards,

Minto


Op 11-9-2013 13:21, Jean-Baptiste Onofré schreef:

Hi Minto,

I guess that you use Karaf 2.2.x or 2.3.0 ?

Previously, the blueprint containers and boot features were
asynchronous: it means that we didn't have guarantee on the boot
features order and blueprint bundles order.

In 2.3.1, I changed that:
- in etc/config.properties, you have
org.apache.aries.blueprint.synchronous=true. Aries blueprint now waits
that a blueprint container is completely started before starting
another one.
- the boot feature are now synchronous. Previously each boot features
installation was in its own thread. So for instance, in your case,
myfeature could start before cxf. Now, it's in the same thread in
order to guarantee the startup order.

Regards
JB

On 09/11/2013 12:03 PM, Minto van der Sluis wrote:

Hi Folks,

In my custom karaf based distribution I occasionally experience freezes
during start up. So far I have seen 2 different occurrence:

1) A complete freeze during start up (see log below)

In this situation services like the webconsole and my own CXF based
services are not available. Looking at the log this is not surprising
since these services have not yet started. However I am clueless why
they didn't start.

Maybe this is related to the order of boot features. In my configuration
I have the following configured:

     featuresBoot=config,management,http,webconsole,cxf,myfeature


2) Bundles depending on Apache CXF stay in creating phase.

Some of my custom bundles stay in creating phase with the log showing
the following messages:

... | INFO  | ... | Bundle tmvplus-proxy is waiting for namespace
handlers [http://cxf.apache.org/blueprint/core]
... | INFO  | ... | Bundle tmvplus-remote is waiting for namespace
handlers [http://cxf.apache.org/blueprint/jaxrs,
http://cxf.apache.org/blueprint/jaxws,
http://cxf.apache.org/blueprint/core]

When I try to stop these manually they keep hanging in stopping phase.
When done from the SSH console then prompt never returns keeping the SSH
console in a hanging state as well. When stopped from the webconsole
they also hang in the stopping phase. Only after restarting the karaf
container they are fully stopped. After the restart I can start them
again manually without any problems.

Initially I suspected start-level issue, but CXF start <= 40 and my
stuff starts >= 70.


Let me stress again that these freeze ups happen occasionally. Most of
the time the karaf container starts just fine. But sometimes it doesn't.

Unfortunately the logs are in both cases not very informative. Just
information log messages.

Has anyone seen this before? Does anyone know what needs to be done to
get around these issues?

Hope someone knows how I can move forward, since I am sort of stuck.

Regards,

Minto van der Sluis

Log of freeze occurence 1:
=========================
Sep 09, 2013 10:21:47 AM org.apache.karaf.main.SimpleFileLock lock
INFO: locking
2013-09-09 10:21:48,541 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.apache.karaf.management.cfg

2013-09-09 10:21:48,567 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.ops4j.pax.logging.cfg

2013-09-09 10:21:48,575 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/tmvplus.sync.cfg

2013-09-09 10:21:48,583 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.apache.karaf.jaas.cfg

2013-09-09 10:21:48,590 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.ops4j.pax.url.mvn.cfg

2013-09-09 10:21:48,598 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.apache.karaf.features.obr.cfg

2013-09-09 10:21:48,605 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/astore.storage.clerezza.cfg

2013-09-09 10:21:48,613 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.apache.karaf.features.repos.cfg

2013-09-09 10:21:48,617 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.apache.clerezza.rdf.jena.tdb.storage.ScalableSingleTdbDatasetTcProvider.cfg

2013-09-09 10:21:48,623 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/tmvplus.auth.cfg

2013-09-09 10:21:48,626 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/tmvplus.proxy.cfg

2013-09-09 10:21:48,632 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/astore.namedqueries.filesystem.cfg

2013-09-09 10:21:48,640 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.apache.karaf.log.cfg

2013-09-09 10:21:48,646 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.apache.cxf.osgi.cfg

2013-09-09 10:21:48,664 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.apache.felix.fileinstall-deploy.cfg

2013-09-09 10:21:48,668 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.apache.karaf.features.cfg

2013-09-09 10:21:48,674 | INFO  | FelixStartLevel  | fileinstall
                | ?                                   ? | 6 -
org.apache.felix.fileinstall - 3.2.6 | Installed
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/etc/org.apache.karaf.shell.cfg

2013-09-09 10:21:48,762 | INFO  | FelixStartLevel  | BlueprintExtender
                | rint.container.BlueprintExtender  138 | 7 -
org.apache.aries.blueprint.core - 1.1.0 | No quiesce support is
available, so blueprint components will not participate in quiesce
operations
2013-09-09 10:21:49,592 | INFO  | FelixStartLevel  |
BlueprintContainerImpl           | container.BlueprintContainerImpl  303
| 7 - org.apache.aries.blueprint.core - 1.1.0 | Bundle
org.apache.karaf.jaas.modules is waiting for namespace handlers
[http://karaf.apache.org/xmlns/jaas/v1.0.0]
2013-09-09 10:21:49,799 | INFO  | FelixStartLevel  |
BlueprintContainerImpl           | container.BlueprintContainerImpl  344
| 7 - org.apache.aries.blueprint.core - 1.1.0 | Bundle
org.apache.karaf.management.server is waiting for dependencies
[(objectClass=org.apache.karaf.jaas.config.KeystoreManager)]
2013-09-09 10:21:49,819 | INFO  | FelixStartLevel  |
BlueprintContainerImpl           | container.BlueprintContainerImpl  344
| 7 - org.apache.aries.blueprint.core - 1.1.0 | Bundle
org.apache.karaf.features.management is waiting for dependencies
[(objectClass=javax.management.MBeanServer),
(objectClass=org.apache.karaf.features.FeaturesService)]
2013-09-09 10:21:49,896 | INFO  | rint Extender: 1 |
BlueprintContainerImpl           | container.BlueprintContainerImpl  344
| 7 - org.apache.aries.blueprint.core - 1.1.0 | Bundle
org.apache.karaf.features.management is waiting for dependencies
[(objectClass=org.apache.karaf.features.FeaturesService)]
2013-09-09 10:21:49,964 | INFO  | FelixStartLevel  | core
                | ?                                   ? | 25 -
org.apache.aries.jmx.core - 1.1.1 | Starting JMX OSGi agent
2013-09-09 10:21:50,041 | INFO  | FelixStartLevel  | core
                | ?                                   ? | 25 -
org.apache.aries.jmx.core - 1.1.1 | Registering MBean with ObjectName
[osgi.compendium:service=cm,version=1.3,framework=org.apache.felix.framework,uuid=ae51c7ab-e8d1-4ffa-bcce-cff4f3ac2bcf]

for service with service.id [10]
2013-09-09 10:21:50,083 | INFO  | JMX OSGi Agent   | core
                | ?                                   ? | 25 -
org.apache.aries.jmx.core - 1.1.1 | Registering
org.osgi.jmx.service.cm.ConfigurationAdminMBean to MBeanServer
com.sun.jmx.mbeanserver.JmxMBeanServer@9eae470 with name
osgi.compendium:service=cm,version=1.3,framework=org.apache.felix.framework,uuid=ae51c7ab-e8d1-4ffa-bcce-cff4f3ac2bcf

2013-09-09 10:21:50,085 | INFO  | JMX OSGi Agent   | core
                | ?                                   ? | 25 -
org.apache.aries.jmx.core - 1.1.1 | Registering
org.osgi.jmx.framework.PackageStateMBean to MBeanServer
com.sun.jmx.mbeanserver.JmxMBeanServer@9eae470 with name
osgi.core:type=packageState,version=1.5,framework=org.apache.felix.framework,uuid=ae51c7ab-e8d1-4ffa-bcce-cff4f3ac2bcf

2013-09-09 10:21:50,086 | INFO  | JMX OSGi Agent   | core
                | ?                                   ? | 25 -
org.apache.aries.jmx.core - 1.1.1 | Registering
org.osgi.jmx.framework.ServiceStateMBean to MBeanServer
com.sun.jmx.mbeanserver.JmxMBeanServer@9eae470 with name
osgi.core:type=serviceState,version=1.7,framework=org.apache.felix.framework,uuid=ae51c7ab-e8d1-4ffa-bcce-cff4f3ac2bcf

2013-09-09 10:21:50,089 | INFO  | JMX OSGi Agent   | core
                | ?                                   ? | 25 -
org.apache.aries.jmx.core - 1.1.1 | Registering
org.osgi.jmx.framework.FrameworkMBean to MBeanServer
com.sun.jmx.mbeanserver.JmxMBeanServer@9eae470 with name
osgi.core:type=framework,version=1.7,framework=org.apache.felix.framework,uuid=ae51c7ab-e8d1-4ffa-bcce-cff4f3ac2bcf

2013-09-09 10:21:50,090 | INFO  | JMX OSGi Agent   | core
                | ?                                   ? | 25 -
org.apache.aries.jmx.core - 1.1.1 | Registering
org.osgi.jmx.framework.BundleStateMBean to MBeanServer
com.sun.jmx.mbeanserver.JmxMBeanServer@9eae470 with name
osgi.core:type=bundleState,version=1.7,framework=org.apache.felix.framework,uuid=ae51c7ab-e8d1-4ffa-bcce-cff4f3ac2bcf

2013-09-09 10:21:50,093 | INFO  | JMX OSGi Agent   | core
                | ?                                   ? | 25 -
org.apache.aries.jmx.core - 1.1.1 | Registering
org.osgi.jmx.framework.wiring.BundleWiringStateMBean to MBeanServer
com.sun.jmx.mbeanserver.JmxMBeanServer@9eae470 with name
osgi.core:type=wiringState,version=1.1,framework=org.apache.felix.framework,uuid=ae51c7ab-e8d1-4ffa-bcce-cff4f3ac2bcf

2013-09-09 10:21:50,253 | INFO  | rint Extender: 3 |
KarArtifactInstaller             | eployer.kar.KarArtifactInstaller   75
| 24 - org.apache.karaf.deployer.kar - 2.3.2 | Karaf archives will be
extracted to
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/system
2013-09-09 10:21:50,258 | INFO  | rint Extender: 3 |
KarArtifactInstaller             | eployer.kar.KarArtifactInstaller   76
| 24 - org.apache.karaf.deployer.kar - 2.3.2 | Timestamps for Karaf
archives will be extracted to
/usr/local/spg/versions/tmvplus-distribution-0.6.6.53/system/.timestamps
2013-09-09 10:21:51,237 | INFO  | FelixStartLevel  |
BlueprintContainerImpl           | container.BlueprintContainerImpl  344
| 7 - org.apache.aries.blueprint.core - 1.1.0 | Bundle
org.apache.karaf.admin.command is waiting for dependencies
[(objectClass=org.apache.karaf.admin.AdminService)]
2013-09-09 10:21:51,704 | INFO  | FelixStartLevel  |
BlueprintContainerImpl           | container.BlueprintContainerImpl  344
| 7 - org.apache.aries.blueprint.core - 1.1.0 | Bundle
org.apache.karaf.admin.management is waiting for dependencies
[(objectClass=org.apache.karaf.admin.AdminService)]


--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Occasional freeze on startup

Reply via email to