[
https://issues.apache.org/jira/browse/BROOKLYN-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105490#comment-17105490
]
Aled Sage commented on BROOKLYN-626:
------------------------------------
For future reference, here's how I investigated and diagnosed the problem:For
future reference, here's how I investigated and diagnosed the problem:
1. `ClassCastException` means one of two things: the class
`HandshakeBeginMessage` does not implement `Message`, or the two classes are
coming from different class loaders (established it is definitely the latter by
using `javap`)
2. Ran the unit tests outside of karaf; they work fine (unsurprisingly -
problem is with class loading in karaf).
3. Looked at normal karaf stuff via `./bin/client`, but *nothing* to suggest a
problem (e.g. `package:exports | grep javax.management.remote.message`,
`package:imports`, etc).
4. Tested with an old version of Brooklyn, connecting to the same Tomcat server
- established that it was definitely an issue with the Brooklyn version rather
than an environment issue (by using the same persisted state directory for each
Brooklyn version, and calling `start` effector on the pre-existing entity so it
would try to connect again over jmxmp).
5. Attached a remote java debugger to Brooklyn (did the same for each version),
and looked at the ClassLoaders involved when creating the `JMXConnector` and
when deserializing the message when connecting.
With Karaf 4.2.8:
{noformat}
javax.management.remote.jmxmp.JMXMPConnector.class.getClassLoader()
(org.apache.felix.framework.BundleWiringImpl$BundleClassLoader)
wrap_file__Users_aledsage_amp_cloudsoft-amp-karaf-5.5.0-20200510.223817-2142_system_org_glassfish_external_opendmk_jmxremote_optional_jar_1.0-b01-ea_opendmk_jmxremote_optional_jar-1.0-b01-ea.jar_Import-Package_javax.management.openmbean__
[292]
Message.class.getClassLoader()
(sun.misc.Launcher$AppClassLoader)
sun.misc.Launcher$AppClassLoader@18b4aac2
((java.net.URLClassLoader)Message.class.getClassLoader()).getURLs()
(java.net.URL[])
[file:/Users/aledsage/amp/cloudsoft-amp-karaf-5.5.0-20200510.223817-2142/lib/boot/opendmk_jmxremote_optional_jar-1.0-b01-ea.jar,
file:/Users/aledsage/amp/cloudsoft-amp-karaf-5.5.0-20200510.223817-2142/lib/boot/org.apache.karaf.diagnostic.boot-4.2.8.jar,
file:/Users/aledsage/amp/cloudsoft-amp-karaf-5.5.0-20200510.223817-2142/lib/boot/org.apache.karaf.jaas.boot-4.2.8.jar,
file:/Users/aledsage/amp/cloudsoft-amp-karaf-5.5.0-20200510.223817-2142/lib/boot/org.apache.karaf.main-4.2.8.jar,
file:/Users/aledsage/amp/cloudsoft-amp-karaf-5.5.0-20200510.223817-2142/lib/boot/org.apache.karaf.specs.activator-4.2.8.jar,
file:/Users/aledsage/amp/cloudsoft-amp-karaf-5.5.0-20200510.223817-2142/lib/boot/osgi.core-6.0.0.jar]
{noformat}
This shows `lib/boot/opendmk_jmxremote_optional_jar-1.0-b01-ea.jar`.
6. Checked if this is in vanilla Karaf (rather than just the Brooklyn distro -
confirmed). Found the relevant section of the Karaf 4.2.8 release notes!
[https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311140&version=12346100]
https://issues.apache.org/jira/browse/KARAF-2925
> jmxmp broken: ClassCastException:
> javax.management.remote.message.HandshakeBeginMessage cannot be cast to
> javax.management.remote.message.Message
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: BROOKLYN-626
> URL: https://issues.apache.org/jira/browse/BROOKLYN-626
> Project: Brooklyn
> Issue Type: Bug
> Affects Versions: 1.1.0-SNAPSHOT
> Reporter: Aled Sage
> Priority: Major
>
> (Since upgrading to Karaf 4.2.8; bug will hopefully never be in a GA release
> of Brooklyn!).
> When using jmxmp (e.g. in `Tomcat8Server` entity), Brooklyn fails with the
> exception like that below:When using jmxmp (e.g. in `Tomcat8Server` entity),
> Brooklyn fails with the exception like that below:
>
> {noformat}
> unable to re-connect to JMX url (repeated failure):
> service:jmx:jmxmp://34.78.32.213:31001: {}unable to re-connect to JMX url
> (repeated failure): service:jmx:jmxmp://34.78.32.213:31001: {}
> java.lang.ClassCastException:
> javax.management.remote.message.HandshakeBeginMessage cannot be cast to
> javax.management.remote.message.Message
> at
> com.sun.jmx.remote.socket.SocketConnection.readMessage(SocketConnection.java:251)
> ~[opendmk_jmxremote_optional_jar-1.0-b01-ea.jar:?]
> at
> com.sun.jmx.remote.opt.security.AdminClient.connectionOpen(AdminClient.java:97)
> ~[opendmk_jmxremote_optional_jar-1.0-b01-ea.jar:?]
> at
> com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.connect(ClientSynchroMessageConnectionImpl.java:112)
> ~[opendmk_jmxremote_optional_jar-1.0-b01-ea.jar:?]
> at
> javax.management.remote.generic.GenericConnector.connect(GenericConnector.java:217)
> ~[!/:?]
> at
> javax.management.remote.jmxmp.JMXMPConnector.connect(JMXMPConnector.java:159)
> ~[!/:?]
> at
> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
> ~[?:1.8.0_232]
> at
> org.apache.brooklyn.feed.jmx.JmxHelper.newConnector(JmxHelper.java:318)
> ~[!/:1.1.0-SNAPSHOT]
> at org.apache.brooklyn.feed.jmx.JmxHelper.connect(JmxHelper.java:277)
> ~[!/:1.1.0-SNAPSHOT]
> at org.apache.brooklyn.feed.jmx.JmxHelper.reconnect(JmxHelper.java:252)
> [!/:1.1.0-SNAPSHOT]
> at
> org.apache.brooklyn.feed.jmx.JmxHelper.reconnectWithRetryDampened(JmxHelper.java:245)
> [!/:1.1.0-SNAPSHOT]
> at
> org.apache.brooklyn.feed.jmx.JmxHelper.invokeWithReconnect(JmxHelper.java:494)
> [!/:1.1.0-SNAPSHOT]
> at org.apache.brooklyn.feed.jmx.JmxHelper.findMBeans(JmxHelper.java:522)
> [!/:1.1.0-SNAPSHOT]
> at org.apache.brooklyn.feed.jmx.JmxHelper.findMBean(JmxHelper.java:530)
> [!/:1.1.0-SNAPSHOT]
> at
> org.apache.brooklyn.feed.jmx.JmxHelper.toLiteralObjectName(JmxHelper.java:514)
> [!/:1.1.0-SNAPSHOT]
> at
> org.apache.brooklyn.feed.jmx.JmxHelper.getAttribute(JmxHelper.java:639)
> [!/:1.1.0-SNAPSHOT]
> at org.apache.brooklyn.feed.jmx.JmxFeed$7.call(JmxFeed.java:373)
> [!/:1.1.0-SNAPSHOT] at
> org.apache.brooklyn.core.feed.Poller$PollJob$1.run(Poller.java:76)
> [!/:1.1.0-SNAPSHOT]
> at org.apache.brooklyn.core.feed.Poller$1.call(Poller.java:158)
> [!/:1.1.0-SNAPSHOT] at
> org.apache.brooklyn.core.feed.Poller$1.call(Poller.java:151)
> [!/:1.1.0-SNAPSHOT]
> at
> org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:364)
> [!/:1.1.0-SNAPSHOT]
> at
> org.apache.brooklyn.util.core.task.BasicExecutionManager$ScheduledTaskCallable$1.call(BasicExecutionManager.java:476)
> [!/:1.1.0-SNAPSHOT]
> at
> org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:565)
> [!/:1.1.0-SNAPSHOT]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_232]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_232]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]{noformat}
>
> All comms over jmxmp fail, so the `JmxFeed` doesn't get its sensor values.
> This can cause entities (such as `Tomcat8Server`) to show as failing to start.
> Here's a simple blueprint to reproduce:
> {noformat}
> location: centos7_aws_useast1
> services:
> - type: org.apache.brooklyn.entity.webapp.tomcat.Tomcat8Server
> id: tomcat
> brooklyn.config:
> webapp.enabledProtocols:
> - http
> - https
> jmx.enabled: true
> jmx.password: <snip>
> webapp.https.ssl:
> keystorePassword: password
> keyAlias: tomcat
> keystoreUrl:
> https://github.com/brooklyncentral/brooklyn-tomcat-8-server/raw/master/sample-keystore{noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
