[
https://issues.apache.org/jira/browse/BROOKLYN-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752255#comment-16752255
]
Aled Sage commented on BROOKLYN-608:
------------------------------------
For future reference, here is how I identified the problem and the fix.
+*Log file points at javax.annotation-api*+
the log file showed it had hung, with last lines about stopping bundles. It's
bad that it stops + replaces bundles - we should try to get versions + ordering
right from the start, so that startup is faster. The log told us of two
problems:
{noformat}
2019-01-25T10:10:40,561 - INFO 11 o.a.k.f.i.s.FeaturesServiceImpl
[tures-3-thread-1] Stopping bundles:
2019-01-25T10:10:40,562 - INFO 11 o.a.k.f.i.s.FeaturesServiceImpl
[tures-3-thread-1] javax.annotation-api/1.2.0
2019-01-25T10:10:40,563 - INFO 11 o.a.k.f.i.s.FeaturesServiceImpl
[tures-3-thread-1] org.ops4j.pax.logging.pax-logging-log4j2/1.10.1
{noformat}
+*Check for conflicting versions*+
{noformat}
find . -name "*javax.annotation-*"
./system/javax/annotation/javax.annotation-api
./system/javax/annotation/javax.annotation-api/1.3/javax.annotation-api-1.3.jar
./system/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar
./lib/jdk9plus/javax.annotation-api-1.2.jar{noformat}
+*Check what brings in /javax.annotation-api-1.3.jar*+
Run {{mvn dependency:tree}}, which shows:
{noformat}
[INFO] org.apache.brooklyn.camp:camp-server:jar:1.0.0-SNAPSHOT
[INFO] +- org.apache.cxf:cxf-rt-frontend-jaxrs:jar:3.2.7:compile
[INFO] | +- org.apache.cxf:cxf-core:jar:3.2.7:compile
[INFO] | | \- org.apache.ws.xmlschema:xmlschema-core:jar:2.2.3:compile
[INFO] | +- javax.annotation:javax.annotation-api:jar:1.3:compile
[INFO] | \- org.apache.cxf:cxf-rt-transports-http:jar:3.2.7:compile
{noformat}
We know CXF version was recently bumped.
Check the \{{cxf-rt-frontend-jaxrs}} dependencies, via {{bundle:headers}},
which showed:
{noformat}
javax.annotation;version="[1.3,2)"{noformat}
And out of curiosity, check what feature is pulling this in:
{noformat}
karaf@amp()> feature:info cxf-jaxrs
Feature cxf-jaxrs 3.2.7
Feature has no configuration
Feature has no configuration files
Feature depends on:
cxf-core 3.2.7
cxf-http 3.2.7
cxf-jackson 3.2.7
Feature contains followed bundles:
mvn:org.codehaus.jettison/jettison/1.4.0 start-level=30
mvn:org.apache.cxf/cxf-rt-rs-extension-providers/3.2.7 start-level=40
mvn:org.apache.cxf/cxf-rt-rs-extension-search/3.2.7 start-level=40
mvn:org.apache.cxf/cxf-rt-rs-json-basic/3.2.7 start-level=40
mvn:org.apache.cxf/cxf-rt-rs-service-description/3.2.7 start-level=40
mvn:org.apache.cxf/cxf-rt-frontend-jaxrs/3.2.7 start-level=40
mvn:org.apache.cxf/cxf-rt-rs-client/3.2.7 start-level=40
Feature has no conditionals.
karaf@amp()> feature:info cxf-core
Feature cxf-core 3.2.7
Feature has no configuration
Feature has no configuration files
Feature depends on:
cxf-specs 3.2.7
Feature contains followed bundles:
mvn:org.apache.ws.xmlschema/xmlschema-core/2.2.3 start-level=30
mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.xmlresolver/1.2_5
start-level=25
mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.fastinfoset/1.2.13_1
start-level=30
mvn:org.apache.cxf/cxf-core/3.2.7 start-level=40
mvn:org.apache.cxf/cxf-rt-management/3.2.7 start-level=40
Feature contains followed conditionals:
Conditional(shell) has no configuration
Conditional(shell) has no configuration files
Conditional(shell) depends on:
cxf-commands 3.2.7
Conditional(shell) has no bundles.
karaf@amp()> feature:info cxf-specs
Feature cxf-specs 3.2.7
Feature has no configuration
Feature has no configuration files
Feature has no dependencies.
Feature contains followed bundles:
mvn:org.apache.geronimo.specs/geronimo-osgi-registry/1.1 start-level=9
mvn:org.apache.servicemix.specs/org.apache.servicemix.specs.activation-api-1.1/2.9.0
start-level=10
mvn:javax.annotation/javax.annotation-api/1.3 start-level=10
mvn:org.apache.geronimo.specs/geronimo-ws-metadata_2.0_spec/1.1.3 start-level=10
mvn:org.apache.servicemix.specs/org.apache.servicemix.specs.stax-api-1.0/2.9.0
start-level=10
mvn:org.apache.servicemix.specs/org.apache.servicemix.specs.jaxb-api-2.2/2.9.0
start-level=10
mvn:org.apache.servicemix.specs/org.apache.servicemix.specs.jaxws-api-2.2/2.9.0
start-level=10
mvn:org.apache.servicemix.specs/org.apache.servicemix.specs.saaj-api-1.3/2.9.0
start-level=10
mvn:org.apache.servicemix.specs/org.apache.servicemix.specs.jaxrs-api-2.1/2.9.1
start-level=10
mvn:javax.mail/mail/1.4.4 start-level=10
mvn:org.codehaus.woodstox/stax2-api/3.1.4 start-level=20
mvn:com.fasterxml.woodstox/woodstox-core/5.0.3 start-level=20
mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.jaxb-impl/2.2.11_1
start-level=20
mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.jaxb-xjc/2.2.11_1
start-level=20
Feature has no conditionals.{noformat}
And check the version that karaf brings in (it's 1.2):
https://github.com/apache/karaf/blob/karaf-4.2.2/pom.xml#L150
+*Decide on Solution*+
We now know that we really do need 1.3 for {{cxf-rt-frontend-jaxrs:3.2.7}}. But
1.2 is being installed first (because that ships with karaf, presumably), and
then being replaced when we go to install the CXF features. This causes lots of
rewiring.
Options include:
1. install CXF features (before all the brooklyn bundles), which will thus
install javax.annotation-api:1.3 earlier.
2. just install javax.annotation-api:1.3 earlier.
Looking in feature.xml of {{brooklyn-dist}}, we see that it already installs
the javax.annotation-api bundle early - it does it before guava, to prevent
guava from rewiring later.
This shows us there is a property in the pom.xml telling it to install v1.2 of
that bundle. If we change the version number, then we'd install 1.3 earlier.
+*Test the solution*+
Change the version number and rebuild Brooklyn.
Start it (from brooklyn-dist/karaf/apache-brooklyn/target/assembly, running
./bin/start).
Look at the log (data/log/brooklyn.info.log) to see what bundles are being
stopped and restarted.
There are now a lot fewer of them, and it does them faster.
We've sufficiently improved the situation to avoid the major problem. However,
there is still room for improvement.
> karaf startup: javax.annotation-api stopped + restarted (can cause startup to
> fail)
> -----------------------------------------------------------------------------------
>
> Key: BROOKLYN-608
> URL: https://issues.apache.org/jira/browse/BROOKLYN-608
> Project: Brooklyn
> Issue Type: Bug
> Reporter: Aled Sage
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When starting Brooklyn karaf 1.0.0-SNAPSHOT, we see in the log that
> \{{javax.annotation-api/1.2.0}} is started and then stopped again (to be
> replaced with v1.3)
> {noformat}
> 2019-01-25T10:10:40,561 - INFO 11 o.a.k.f.i.s.FeaturesServiceImpl
> [tures-3-thread-1] Stopping bundles:
> 2019-01-25T10:10:40,562 - INFO 11 o.a.k.f.i.s.FeaturesServiceImpl
> [tures-3-thread-1] javax.annotation-api/1.2.0
> 2019-01-25T10:10:40,563 - INFO 11 o.a.k.f.i.s.FeaturesServiceImpl
> [tures-3-thread-1] org.ops4j.pax.logging.pax-logging-log4j2/1.10.1{noformat}
> This causes many other bundles to be reloaded. In Brooklyn, this takes a few
> seconds. But in downstream projects that embed or build on Brooklyn, the
> consequence can be much worse - unloading javax.annotation-api causes so many
> bundles to be stopped that Karaf really struggles to figure out the
> dependencies - CPU usage maxes out for several minutes, and then it seemingly
> hangs (cpu goes to zero, and nothing else appears in the logs).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
