Hi,
I could reproduce this in Openstack. The region and image id of the
iaasProvider is null at the time of IP releasing. When I set the region in
cloud-controller.xml (which is not a solution, just for testing) it works
without the issue.
[2015-03-23 15:25:23,067] INFO
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member
terminated: [member-id]
single-cartridge-app.my-php.php.domaine4fb4a32-64b1-4804-877f-2e93748f6a06
[2015-03-23 15:25:23,076] INFO
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
- Publishing member terminated event: [service-name] php [cluster-id]
single-cartridge-app.my-php.php.domain [cluster-instance-id]
single-cartridge-app-1 [member-id]
single-cartridge-app.my-php.php.domaine4fb4a32-64b1-4804-877f-2e93748f6a06
[network-partition-id] network-partition-1 [partition-id] partition-1
[group-id] null
[2015-03-23 15:25:23,084] INFO {org.apache.
On Mon, Mar 23, 2015 at 11:07 AM, Udara Liyanage <[email protected]> wrote:
> Hi,
>
> I will have a look.
>
> On Mon, Mar 23, 2015 at 3:38 AM, Vanson Lim <[email protected]> wrote:
>
>> Devs,
>>
>> We are continuing to work on testing the latest stratos 4.1.0 codebase.
>>
>> This problem is seen only for VM that have floating ip. I've tested
>> with the non floating ip case and don't see issues.
>>
>> The error return code from jcloud api call is preventing stratos from
>> cleaning up its state.
>>
>> Stratos seems to forever throw tracebacks as it repeatedly tries to
>> terminate the faulty instance.
>>
>> Meanwhile, the "down" VM is still being reported as active in the
>> topology events, which seems wrong. If stratos detects that the VM is
>> faulty, shouldn't it report it immediately in the topology events? Stratos
>> currently has the following states define and none of them seem to be
>> appropriate.
>>
>> Created
>> Initialized
>> Starting
>> Active
>> In_Maintenance
>> ReadyToShutdown
>> Suspended
>> Terminated
>>
>>
>> Do we need new state TIMED-OUT state that stratos reports for VM as
>> stratos works to terminate it?
>>
>> How to reproduce this issue:
>>
>> 1) Start a sample cartridge instance that has a floating ip.
>>
>> 2) wait for sample cartridge to become active
>> 3) terminate sample vm via openstack horizon interface, and wait for
>> stratos to detect VM the error.
>>
>>
>> Testing using a version of stratos built off the following commit id:
>>
>> commit 01dd9e491ad3acf7cc4e0f2895aaba336b82539d
>> Author: R-Rajkumar <[email protected]> <[email protected]>
>> Date: Fri Mar 20 19:51:06 2015 +0530
>>
>> fixing an NPE in AS
>>
>>
>> I've attached the full wso2carbon.log Included below is the observed
>> traceback:
>>
>> -Vanson
>>
>>
>> TID: [0] [STRATOS] [2015-03-22 20:53:21,554] INFO
>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} -
>> Publishing member fault event for [member-id]
>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd
>> TID: [0] [STRATOS] [2015-03-22 20:54:06,386] INFO
>> {org.apache.stratos.common.client.CloudControllerServiceClient} -
>> Terminating instance via cloud controller: [member]
>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd
>> TID: [0] [STRATOS] [2015-03-22 20:54:06,399] INFO
>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Starting to
>> terminate member: [cartridge-type] cisco-sample-vm [member-id]
>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd
>> TID: [0] [STRATOS] [2015-03-22 20:54:06,450] ERROR
>> {org.apache.stratos.cloud.controller.services.impl.InstanceTerminator} -
>> Instance termination failed! MemberContext [applicationId=cisco-sample-vm,
>> cartridgeType=cisco-sample-vm,
>> clusterId=cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain,
>> memberId=cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd,
>> instanceId=RegionOne/83751110-4e5b-4aef-b6a3-c291c9eaad3d,
>> partition=Partition [id=whole-region, description=null, isPublic=false,
>> provider=Core, properties=Properties [properties=[Property [name=region,
>> value=RegionOne]]]], defaultPrivateIP=172.16.2.17,
>> defaultPublicIP=10.0.0.102, allocatedIPs=[10.0.0.102],
>> publicIPs=[10.0.0.102], privateIPs=[172.16.2.17], initTime=1427057106433,
>> lbClusterId=null, networkPartitionId=RegionOne, kubernetesPodId=null,
>> kubernetesPodLabel=null, loadBalancingIPType=Private,
>> instanceMetadata=org.apache.stratos.cloud.controller.domain.InstanceMetadata@5b176e44,
>> properties=Properties [properties=[Property [name=PRIMARY, value=false],
>> Property [name=MIN_COUNT, value=1]]]]
>> java.lang.NullPointerException: arg[0] in
>> {invocation=org.jclouds.openstack.nova.v2_0.NovaApi.public abstract
>> com.google.common.base.Optional
>> org.jclouds.openstack.nova.v2_0.NovaApi.getFloatingIPExtensionForZone(java.lang.String)[null],
>> result={annotationParser={caller=NovaApi.getFloatingIPExtensionForZone[null]}}}
>> at
>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:253)
>> at
>> org.jclouds.openstack.v2_0.functions.PresentWhenExtensionAnnotationNamespaceEqualsAnyNamespaceInExtensionsSet.apply(PresentWhenExtensionAnnotationNamespaceEqualsAnyNamespaceInExtensionsSet.java:67)
>> at
>> org.jclouds.openstack.v2_0.functions.PresentWhenExtensionAnnotationNamespaceEqualsAnyNamespaceInExtensionsSet.apply(PresentWhenExtensionAnnotationNamespaceEqualsAnyNamespaceInExtensionsSet.java:43)
>> at
>> org.jclouds.rest.internal.DelegatesToInvocationFunction.propagateContextToDelegate(DelegatesToInvocationFunction.java:205)
>> at
>> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:154)
>> at
>> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> at com.sun.proxy.$Proxy119.getFloatingIPExtensionForZone(Unknown
>> Source)
>> at
>> org.apache.stratos.cloud.controller.iaases.openstack.networking.NovaNetworkingApi.releaseAddress(NovaNetworkingApi.java:239)
>> at
>> org.apache.stratos.cloud.controller.iaases.openstack.OpenstackIaas.releaseAddress(OpenstackIaas.java:239)
>> at
>> org.apache.stratos.cloud.controller.iaases.JcloudsIaas.destroyNode(JcloudsIaas.java:334)
>> at
>> org.apache.stratos.cloud.controller.iaases.JcloudsIaas.terminateInstance(JcloudsIaas.java:314)
>> at
>> org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:56)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> TID: [0] [STRATOS] [2015-03-22 20:54:21,563] INFO
>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
>> member detected [member-id]
>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd
>> with [last time-stamp] 1427057336960 [time-out] 60000 milliseconds
>> TID: [0] [STRATOS] [2015-03-22 20:54:21,563] INFO
>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} -
>> Publishing member fault event for [member-id]
>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain85d6eda0-1df5-4be2-b846-4817cc5292cd
>>
>>
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>
--
Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware
web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897
