Hi Raj, It seems like [1] get executed again. memberTimeStampMap is a concurrent hash map. Seems that CEPTopologyEventReceiver unable to remove the element in the memberTimeStampMap.
[1] https://github.com/apache/stratos/blob/master/extensions/cep/stratos-cep-extension/src/main/java/org/apache/stratos/cep/extension/FaultHandlingWindowProcessor.java#L220 Thanks ! On Mon, Oct 13, 2014 at 11:48 AM, Rajkumar Rajaratnam <[email protected]> wrote: > Hi, > > CEP extension FaultHandlingWindowProcessor removing a terminated member > from the map. But after a while, it is complaining it as a faulty member. > Here is the log. MemberId highlighted. > > TID: [0] [STRATOS] [2014-10-13 11:15:22,409] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time > intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 41.63388 , [gradient]: -0.013520682 , [second > derivative]: 0.0013238431 , [time intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 99.9 , [gradient]: -2.0885546 , [second derivative]: > 0.03460533 , [time intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] minReplicas : 2 > TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] maxReplicas : 20 > TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] nonTerminated : 5 > TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] activeReplicas : 5 > TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] resetted ? : false > TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] predicted value : 0.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] upper limit : 80.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] lower limit : 5.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] resetted ? : true > TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] predicted value : 41.62102324969601 > TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] upper limit : 80.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] lower limit : 15.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,415] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] resetted ? : true > TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] predicted value : 97.82874980941415 > TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] upper limit : 180.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] lower limit : 20.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain scale-up action : false > TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain scale-down action : true > TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to > scale-down [cluster] : myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-down] [cluster] : > myphp.php.domain shrink the cluster to minReplicas : 2 > TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO > {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} > - Updating kubernetes replication controller via cloud controller: > [cluster] myphp.php.domain [replicas] 2 > TID: [0] [STRATOS] [2014-10-13 11:15:22,425] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > CloudControllerServiceImpl:updateContainers for cluster : myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:22,448] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Cloud Controller is delegating request to update a replication controller > myphp.php.domain to Kubernetes layer. > TID: [0] [STRATOS] [2014-10-13 11:15:22,506] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Cloud Controller successfully updated the controller myphp.php.domain via > Kubernetes layer. > TID: [0] [STRATOS] [2014-10-13 11:15:22,531] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Pods Count: 5 for cluster: myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:22,548] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:22,561] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:22,563] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:22,655] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:22,657] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:27,538] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,539] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: > Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,540] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,540] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: > Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,628] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: > Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,630] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,542] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: > Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Pods Count: 2 for cluster: myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Pods created : 2 for cluster : myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock acquired > TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating > topology > TID: [0] [STRATOS] [2014-10-13 11:15:32,553] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,555] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: > Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology > updated: > {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":[" > myphp.apachestratos.org > "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b446c0dd-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b446c0dd-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false} > TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock released > TID: [0] [STRATOS] [2014-10-13 11:15:32,564] INFO > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing > member terminated event: [service] php [cluster] myphp.php.domain > [network-partition] null [partition] null [member] > b4468fb6-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,575] INFO > {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} > - [MemberTerminatedEventListener] Received: class > org.apache.stratos.messaging.event.topology.MemberTerminatedEvent > TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG > {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} > - Removed the terminated member with id > b4468fb6-529b-11e4-acbb-08002794b041 from the cluster > TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG > {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel} > - Adding cluster [myphp.php.domain] > TID: [0] [STRATOS] [2014-10-13 11:15:32,580] INFO > {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} > - Member terminated: [service] php [cluster] myphp.php.domain [member] > b4468fb6-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > MemberContext [id]: b4468fb6-529b-11e4-acbb-08002794b041 removed from > information model. > TID: [0] [STRATOS] [2014-10-13 11:15:32,581] INFO > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > Scheduled Pod Activation Watcher task canceled. > TID: [0] [STRATOS] [2014-10-13 11:15:32,581] WARN > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member > is in the wrong list and it is removed from active members list > TID: [0] [STRATOS] [2014-10-13 11:15:32,582] INFO > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member > stat context has been removed successfully: [member] > b4468fb6-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,582] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] b4468fb6-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,583] INFO > {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member > [member id] b4468fb6-529b-11e4-acbb-08002794b041 was removed from the time > stamp map. > TID: [0] [STRATOS] [2014-10-13 11:15:32,583] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] b4468fb6-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock acquired > TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating > topology > TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology > updated: > {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":[" > myphp.apachestratos.org > "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false} > TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock released > TID: [0] [STRATOS] [2014-10-13 11:15:32,603] INFO > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing member terminated event: [service] php [cluster] > myphp.php.domain [network-partition] null [partition] null [member] > b446c0dd-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,607] INFO > {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member > [member id] b446c0dd-529b-11e4-acbb-08002794b041 was removed from the time > stamp map. > TID: [0] [STRATOS] [2014-10-13 11:15:32,608] INFO > {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} > - Member terminated: [service] php [cluster] myphp.php.domain [member] > b446c0dd-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO > {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} > - [MemberTerminatedEventListener] Received: class > org.apache.stratos.messaging.event.topology.MemberTerminatedEvent > TID: [0] [STRATOS] [2014-10-13 11:15:32,609] DEBUG > {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel} > - Adding cluster [myphp.php.domain] > TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] b446c0dd-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member > is in the wrong list and it is removed from active members list > TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member > stat context has been removed successfully: [member] > b446c0dd-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,610] DEBUG > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > MemberContext [id]: b446c0dd-529b-11e4-acbb-08002794b041 removed from > information model. > TID: [0] [STRATOS] [2014-10-13 11:15:32,610] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] b446c0dd-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,610] INFO > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > Scheduled Pod Activation Watcher task canceled. > TID: [0] [STRATOS] [2014-10-13 11:15:32,623] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock acquired > TID: [0] [STRATOS] [2014-10-13 11:15:32,624] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating > topology > TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,628] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: > Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,633] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology > updated: > {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":[" > myphp.apachestratos.org > "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false} > TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock released > TID: [0] [STRATOS] [2014-10-13 11:15:32,642] INFO > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing member terminated event: [service] php [cluster] > myphp.php.domain [network-partition] null [partition] null [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,650] INFO > {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member > [member id] 3ccf4e4e-529a-11e4-acbb-08002794b041 was removed from the time > stamp map. > TID: [0] [STRATOS] [2014-10-13 11:15:32,652] INFO > {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} > - Member terminated: [service] php [cluster] myphp.php.domain [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,653] INFO > {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} > - [MemberTerminatedEventListener] Received: class > org.apache.stratos.messaging.event.topology.MemberTerminatedEvent > TID: [0] [STRATOS] [2014-10-13 11:15:32,653] DEBUG > {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel} > - Adding cluster [myphp.php.domain] > TID: [0] [STRATOS] [2014-10-13 11:15:32,653] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,654] WARN > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member > is in the wrong list and it is removed from active members list > TID: [0] [STRATOS] [2014-10-13 11:15:32,654] DEBUG > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > MemberContext [id]: 3ccf4e4e-529a-11e4-acbb-08002794b041 removed from > information model. > TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member > stat context has been removed successfully: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,655] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > Scheduled Pod Activation Watcher task canceled. > TID: [0] [STRATOS] [2014-10-13 11:15:32,696] INFO > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Kubernetes entities are successfully starting up. [MemberContext > [memberId=b4468fb6-529b-11e4-acbb-08002794b041, nodeId=null, > instanceId=null, clusterId=myphp.php.domain, partition=null, > cartridgeType=php, privateIpAddress=172.17.8.100, > publicIpAddress=172.17.8.100, allocatedIpAddress=null, > initTime=1413178952532, lbClusterId=null, networkPartitionId=null, > properties=Properties [properties=[Property > [name=ALLOCATED_SERVICE_HOST_PORT, value=null]]]], MemberContext > [memberId=b446c0dd-529b-11e4-acbb-08002794b041, nodeId=null, > instanceId=null, clusterId=myphp.php.domain, partition=null, > cartridgeType=php, privateIpAddress=172.17.8.101, > publicIpAddress=172.17.8.101, allocatedIpAddress=null, > initTime=1413178952532, lbClusterId=null, networkPartitionId=null, > properties=Properties [properties=[Property > [name=ALLOCATED_SERVICE_HOST_PORT, value=null]]]], MemberContext > [memberId=3ccf4e4e-529a-11e4-acbb-08002794b041, nodeId=null, > instanceId=null, clusterId=myphp.php.domain, partition=null, > cartridgeType=php, privateIpAddress=172.17.8.101, > publicIpAddress=172.17.8.101, allocatedIpAddress=null, > initTime=1413178322623, lbClusterId=null, networkPartitionId=null, > properties=Properties [properties=[Property > [name=ALLOCATED_SERVICE_HOST_PORT, value=4500]]]]] > TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down, > terminated the member with id b4468fb6-529b-11e4-acbb-08002794b041 in > cluster myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down, > terminated the member with id b446c0dd-529b-11e4-acbb-08002794b041 in > cluster myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down, > terminated the member with id 3ccf4e4e-529a-11e4-acbb-08002794b041 in > cluster myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG > {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check > executed for : > org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84 > TID: [0] [STRATOS] [2014-10-13 11:16:01,002] DEBUG > {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing > complete tenant event > TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - > Executing topology synchronization task > TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing complete topology event > TID: [0] [STRATOS] [2014-10-13 11:16:22,230] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of > load avg event: [cluster] myphp.php.domain [value] -0.70397747 > TID: [0] [STRATOS] [2014-10-13 11:16:22,232] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of > Memory Consumption event: [cluster] myphp.php.domain [value] -0.46528494 > TID: [0] [STRATOS] [2014-10-13 11:16:22,234] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg > Memory Consumption event: [cluster] myphp.php.domain [value] 27.538944 > TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second > Derivation of load avg event: [cluster] myphp.php.domain [value] > -0.0012131054 > TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second > Derivation of Memory Consumption event: [cluster] myphp.php.domain [value] > 0.021650476 > TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG > {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory > consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 > TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg > load avg event: [cluster] myphp.php.domain [value] 53.7 > TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG > {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average > stats are reset, ready to do scale check [kub cluster] KubGrp1 > TID: [0] [STRATOS] [2014-10-13 11:16:22,425] DEBUG > {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} > - Member not found in the Topology: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG > {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} > - Member not found in the Topology: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG > {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} > - Member not found in the Topology: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG > {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} > - Member not found in the Topology: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG > {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver} > - Member not found in the Topology: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:16:52,406] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - > KubernetesServiceClusterMonitor is running.. > KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1, > clusterId=myphp.php.domain, serviceId=php] > TID: [0] [STRATOS] [2014-10-13 11:16:52,407] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - > Running min check for [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: > [kub-cluster] KubGrp1 [cluster] myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : > myphp.php.domain [Replicas] nonTerminated : 2 > TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : > myphp.php.domain [Replicas] minReplicas : 2 > TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers > rule [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:16:52,410] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster] > : myphp.php.domain [Replicas] obsoleteReplicas : 0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG > {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum > check executed for : > org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84 > TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - > flag of rifReset : false flag of memoryConsumptionReset : true flag of > loadAverageReset : true > TID: [0] [STRATOS] [2014-10-13 11:16:52,412] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - > Running scale check for [kub-cluster] : KubGrp1 [cluster] : > myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:16:52,412] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule > [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:16:52,414] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time > intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:16:52,415] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 27.538944 , [gradient]: -0.46528494 , [second > derivative]: 0.021650476 , [time intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:16:52,416] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 53.7 , [gradient]: -0.70397747 , [second derivative]: > -0.0012131054 , [time intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] minReplicas : 2 > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] maxReplicas : 20 > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] nonTerminated : 2 > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] activeReplicas : 2 > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] resetted ? : false > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] predicted value : 0.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] upper limit : 80.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] lower limit : 5.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] resetted ? : true > TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] predicted value : 27.084485135041177 > TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] upper limit : 80.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] lower limit : 15.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,419] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] resetted ? : true > TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] predicted value : 52.9954185327515 > TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] upper limit : 180.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] lower limit : 20.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain scale-up action : false > TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain scale-down action : true > TID: [0] [STRATOS] [2014-10-13 11:16:52,421] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain Even if all predicted stats < upperLimit, still > activeReplicas <= minReplicas ; hence not scaling down > TID: [0] [STRATOS] [2014-10-13 11:16:52,421] DEBUG > {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check > executed for : > org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84 > TID: [0] [STRATOS] [2014-10-13 11:17:01,001] DEBUG > {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing > complete tenant event > TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - > Executing topology synchronization task > TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing complete topology event > TID: [0] [STRATOS] [2014-10-13 11:17:16,141] INFO > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty > member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last > time-stamp] 1413179137678 [time-out] 60000 milliseconds > TID: [0] [STRATOS] [2014-10-13 11:17:16,141] ERROR > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed > to publish member fault event. Member having [member-id] > 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology > TID: [0] [STRATOS] [2014-10-13 11:17:22,231] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg > load avg event: [cluster] myphp.php.domain [value] 23.75 > TID: [0] [STRATOS] [2014-10-13 11:17:22,232] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of > load avg event: [cluster] myphp.php.domain [value] -0.2682943 > TID: [0] [STRATOS] [2014-10-13 11:17:22,236] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg > Memory Consumption event: [cluster] myphp.php.domain [value] 17.602432 > TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second > Derivation of load avg event: [cluster] myphp.php.domain [value] > -0.0045045386 > TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG > {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average > stats are reset, ready to do scale check [kub cluster] KubGrp1 > TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of > Memory Consumption event: [cluster] myphp.php.domain [value] 0.0016795773 > TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second > Derivation of Memory Consumption event: [cluster] myphp.php.domain [value] > 2.3589536E-4 > TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG > {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory > consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 > TID: [0] [STRATOS] [2014-10-13 11:18:01,000] DEBUG > {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing > complete tenant event > TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - > Executing topology synchronization task > TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing complete topology event > TID: [0] [STRATOS] [2014-10-13 11:18:16,142] INFO > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty > member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last > time-stamp] 1413179137678 [time-out] 60000 milliseconds > TID: [0] [STRATOS] [2014-10-13 11:18:16,144] ERROR > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed > to publish member fault event. Member having [member-id] > 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology > > Thanks. > > -- > Rajkumar Rajaratnam > Software Engineer | WSO2, Inc. > Mobile +94777568639 | +94783498120 > -- Regards, Manula Chathurika Thantriwatte Software Engineer WSO2 Inc. : http://wso2.com lean . enterprise . middleware email : [email protected] / [email protected] phone : +94 772492511 blog : http://manulachathurika.blogspot.com/
