Aled Sage created BROOKLYN-314:
----------------------------------
Summary: Rebind failed: calling getOsDetails failed in entity
driver's constructor
Key: BROOKLYN-314
URL: https://issues.apache.org/jira/browse/BROOKLYN-314
Project: Brooklyn
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Aled Sage
A customer hit a rebind error. Brooklyn failed to instantiate the entity
because the driver's constructor called {{SshMachineLocation.getOsDetails}},
which tried to call the vCloudDirector API. This call failed because the VM no
longer exists. The exception was propagated, causing the entity to fail to be
re-created.
We should avoid propagating the exception, e.g. by returning an OsDetails
saying "unknown".
Even better, we shouldn't be calling the cloud api at this point. But the call
to getOsDetails is happening from the customer code.
Should we store the OsDetails in the machine persisted state, so we don't need
to look it up again?
{noformat}
2016-07-11 12:46:13,258 ERROR o.a.b.l.common.BasicLauncher [main]: Subsystem
for persistence had startup error (continuing with startup):
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Failure
rebinding; 2 errors including: problem rebinding entity DKtbsydb
(JenkinsNodeImpl{id=DKtbsydb}): InvalidCacheLoadException: CacheLoader returned
null for key urn:vcloud:vm:43ebf981-3900-491c-bfd1-68ca1ab3a9ea.
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException:
at
org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:128)
~[brooklyn-utils-common-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebind(RebindManagerImpl.java:513)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.ha.HighAvailabilityManagerImpl.promoteToMaster(HighAvailabilityManagerImpl.java:836)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.ha.HighAvailabilityManagerImpl.checkMaster(HighAvailabilityManagerImpl.java:810)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.ha.HighAvailabilityManagerImpl.publishAndCheck(HighAvailabilityManagerImpl.java:592)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.ha.HighAvailabilityManagerImpl.changeMode(HighAvailabilityManagerImpl.java:315)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.ha.HighAvailabilityManagerImpl.start(HighAvailabilityManagerImpl.java:255)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.launcher.common.BasicLauncher.startPersistence(BasicLauncher.java:642)
[brooklyn-launcher-common-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.launcher.common.BasicLauncher.handlePersistence(BasicLauncher.java:506)
[brooklyn-launcher-common-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.launcher.common.BasicLauncher.start(BasicLauncher.java:424)
[brooklyn-launcher-common-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at org.apache.brooklyn.cli.Main$LaunchCommand.call(Main.java:461)
[brooklyn-cli-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at org.apache.brooklyn.cli.Main$LaunchCommand.call(Main.java:196)
[brooklyn-cli-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at org.apache.brooklyn.cli.AbstractMain.execCli(AbstractMain.java:252)
[brooklyn-cli-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at org.apache.brooklyn.cli.AbstractMain.execCli(AbstractMain.java:244)
[brooklyn-cli-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at io.cloudsoft.amp.Main.main(Main.java:36) [amp-launcher-3.1.1.jar:na]
Caused by: java.util.concurrent.ExecutionException:
org.apache.brooklyn.util.exceptions.CompoundRuntimeException: Failure
rebinding; 2 errors including: problem rebinding entity DKtbsydb
(JenkinsNodeImpl{id=DKtbsydb}): InvalidCacheLoadException: CacheLoader returned
null for key urn:vcloud:vm:43ebf981-3900-491c-bfd1-68ca1ab3a9ea.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
~[na:1.7.0_101]
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
~[na:1.7.0_101]
at
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
~[guava-17.0.jar:na]
at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:361)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebind(RebindManagerImpl.java:511)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
... 13 common frames omitted
org.apache.brooklyn.util.exceptions.CompoundRuntimeException: Failure
rebinding; 2 errors including: problem rebinding entity DKtbsydb
(JenkinsNodeImpl{id=DKtbsydb}): InvalidCacheLoadException: CacheLoader returned
null for key urn:vcloud:vm:43ebf981-3900-491c-bfd1-68ca1ab3a9ea.
at
org.apache.brooklyn.util.exceptions.Exceptions.create(Exceptions.java:393)
~[brooklyn-utils-common-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindExceptionHandlerImpl.onDoneImpl(RebindExceptionHandlerImpl.java:497)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindExceptionHandlerImpl.onDone(RebindExceptionHandlerImpl.java:413)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindIteration.run(RebindIteration.java:267)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebindImpl(RebindManagerImpl.java:558)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl$3.call(RebindManagerImpl.java:508)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl$3.call(RebindManagerImpl.java:506)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_101]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_101]
Caused by: java.lang.IllegalStateException: problem rebinding entity DKtbsydb
(JenkinsNodeImpl{id=DKtbsydb})
at
org.apache.brooklyn.core.mgmt.rebind.RebindExceptionHandlerImpl.onRebindFailed(RebindExceptionHandlerImpl.java:304)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindIteration.reconstructEverything(RebindIteration.java:625)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindIteration.doRun(RebindIteration.java:242)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.InitialFullRebindIteration.doRun(InitialFullRebindIteration.java:69)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindIteration.run(RebindIteration.java:265)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
... 8 common frames omitted
Caused by: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException:
at
org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:128)
~[brooklyn-utils-common-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.entity.drivers.ReflectiveEntityDriverFactory.build(ReflectiveEntityDriverFactory.java:266)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.entity.drivers.BasicEntityDriverManager.build(BasicEntityDriverManager.java:53)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.entity.software.base.SoftwareProcessImpl.newDriver(SoftwareProcessImpl.java:126)
~[brooklyn-software-base-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.entity.software.base.SoftwareProcessImpl.doInitDriver(SoftwareProcessImpl.java:539)
~[brooklyn-software-base-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.entity.software.base.SoftwareProcessImpl.initDriver(SoftwareProcessImpl.java:519)
~[brooklyn-software-base-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.entity.software.base.SoftwareProcessImpl.rebind(SoftwareProcessImpl.java:458)
~[brooklyn-software-base-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.AbstractBrooklynObjectRebindSupport.reconstruct(AbstractBrooklynObjectRebindSupport.java:69)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.core.mgmt.rebind.RebindIteration.reconstructEverything(RebindIteration.java:623)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
... 11 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[na:1.7.0_101]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
~[na:1.7.0_101]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[na:1.7.0_101]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
~[na:1.7.0_101]
at
org.apache.brooklyn.core.entity.drivers.ReflectiveEntityDriverFactory.build(ReflectiveEntityDriverFactory.java:263)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
... 18 common frames omitted
Caused by: com.google.common.cache.CacheLoader$InvalidCacheLoadException:
CacheLoader returned null for key
urn:vcloud:vm:43ebf981-3900-491c-bfd1-68ca1ab3a9ea.
at
com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348)
~[guava-17.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318)
~[guava-17.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
~[guava-17.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
~[guava-17.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
~[guava-17.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
~[guava-17.0.jar:na]
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
~[guava-17.0.jar:na]
at
com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4827)
~[guava-17.0.jar:na]
at
org.jclouds.vcloud.director.v1_5.functions.URNToHref.apply(URNToHref.java:49)
~[vcloud-director-1.9.2-cloudsoft.20160428.1116.jar:1.9.2-cloudsoft.20160428.1116]
at
org.jclouds.vcloud.director.v1_5.functions.URNToHref.apply(URNToHref.java:37)
~[vcloud-director-1.9.2-cloudsoft.20160428.1116.jar:1.9.2-cloudsoft.20160428.1116]
at
org.jclouds.rest.internal.RestAnnotationProcessor.getEndpointInParametersOrNull(RestAnnotationProcessor.java:495)
~[jclouds-core-1.9.2.jar:1.9.2]
at
org.jclouds.rest.internal.RestAnnotationProcessor.getEndpointFor(RestAnnotationProcessor.java:518)
~[jclouds-core-1.9.2.jar:1.9.2]
at
org.jclouds.rest.internal.RestAnnotationProcessor.findEndpoint(RestAnnotationProcessor.java:370)
~[jclouds-core-1.9.2.jar:1.9.2]
at
org.jclouds.rest.internal.RestAnnotationProcessor.apply(RestAnnotationProcessor.java:192)
~[jclouds-core-1.9.2.jar:1.9.2]
at
org.jclouds.rest.internal.RestAnnotationProcessor.apply(RestAnnotationProcessor.java:129)
~[jclouds-core-1.9.2.jar:1.9.2]
at
org.jclouds.rest.internal.InvokeHttpMethod.toCommand(InvokeHttpMethod.java:188)
~[jclouds-core-1.9.2.jar:1.9.2]
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:84)
~[jclouds-core-1.9.2.jar:1.9.2]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
~[jclouds-core-1.9.2.jar:1.9.2]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
~[jclouds-core-1.9.2.jar:1.9.2]
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
~[jclouds-core-1.9.2.jar:1.9.2]
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
~[guava-17.0.jar:na]
at com.sun.proxy.$Proxy201.get(Unknown Source) ~[na:na]
at
org.jclouds.vcloud.director.v1_5.compute.strategy.VCloudDirectorComputeServiceAdapter.getNode(VCloudDirectorComputeServiceAdapter.java:597)
~[vcloud-director-1.9.2-cloudsoft.20160428.1116.jar:1.9.2-cloudsoft.20160428.1116]
at
org.jclouds.vcloud.director.v1_5.compute.strategy.VCloudDirectorComputeServiceAdapter.getNode(VCloudDirectorComputeServiceAdapter.java:106)
~[vcloud-director-1.9.2-cloudsoft.20160428.1116.jar:1.9.2-cloudsoft.20160428.1116]
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.getNode(AdaptingComputeServiceStrategies.java:137)
~[jclouds-compute-1.9.2.jar:1.9.2]
at
org.jclouds.compute.internal.BaseComputeService.getNodeMetadata(BaseComputeService.java:402)
~[jclouds-compute-1.9.2.jar:1.9.2]
at
org.apache.brooklyn.location.jclouds.JcloudsSshMachineLocation.getOptionalNode(JcloudsSshMachineLocation.java:219)
~[brooklyn-locations-jclouds-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.location.jclouds.JcloudsSshMachineLocation.getOptionalOperatingSystem(JcloudsSshMachineLocation.java:498)
~[brooklyn-locations-jclouds-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.location.jclouds.JcloudsSshMachineLocation.inferMachineDetails(JcloudsSshMachineLocation.java:522)
~[brooklyn-locations-jclouds-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.location.ssh.SshMachineLocation.getMachineDetails(SshMachineLocation.java:1056)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at
org.apache.brooklyn.location.ssh.SshMachineLocation.getOsDetails(SshMachineLocation.java:1037)
~[brooklyn-core-0.9.1-20160429.1930.jar:0.9.1-20160429.1930]
at com.acme.JenkinsSshDriver.<init>(JenkinsSshDriver.java:53)
~[acme-amp-jenkins-1.3.1-Final.jar:1.3-Final]
... 23 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
