Hi,
I am doing a test round with M3 with python agent. And faced this very
strange scenario, $Subject.
TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] -2000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 1.9090909
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.718182
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of Memory Consumption event: [cluster] php.php.domain [value]
200000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 3.3636363
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 5.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] 2000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] 13500.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 50.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.738462
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 2.3333333
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 31.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 39.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 39.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] -11000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.8
TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
KubernetesServiceClusterMonitor is running..
KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
clusterId=php.php.domain, serviceId=php]
TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
[kub-cluster] KubGrp1 [cluster] php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
php.php.domain [Replicas] nonTerminated : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
php.php.domain [Replicas] minReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
: php.php.domain [Replicas] obsoleteReplicas : 0
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
{org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
check executed for :
org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
flag of rifReset : true flag of memoryConsumptionReset : true flag of
loadAverageReset : true
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
[kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
intervals]: 1
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
intervals]: 1
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
2000000.0 , [time intervals]: 1
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [Replicas] minReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [Replicas] maxReplicas : 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [Replicas] nonTerminated : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [Replicas] activeReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [RequestInFlight] predicted value : 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [RequestInFlight] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [RequestInFlight] lower limit : 5.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [MemoryConsumption] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [MemoryConsumption] lower limit : 15.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [LoadAverage] predicted value : 989039.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [LoadAverage] upper limit : 180.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [LoadAverage] lower limit : 20.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain scale-up action : true
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain scale-down action : false
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [LoadAverage] predicted replicas : 16484
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain predicted replicas > max replicas :
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
scale-up : [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
php.php.domain valid number of replicas to expand : 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
{org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
- Updating kubernetes replication controller via cloud controller:
[cluster] php.php.domain [replicas] 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
Cloud Controller is delegating request to update a replication controller
php.php.domain to Kubernetes layer.
I couldn't investigate the agent side, since containers are destroyed.
Any idea?
Thanks.
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
