On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte < [email protected]> wrote:
> Hi Raj, > > Shall we enable the cartridge agent debug logs. Then we can compare both > CEP and agent logs with the time stamp. Then we can narrow down where the > actual problem is. > Manula, cep-trace log have the values sent by cartridge agent also right? > > Thanks ! > > On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <[email protected]> > wrote: > >> >> >> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte < >> [email protected]> wrote: >> >>> Hi Raj, >>> >>> Is this happen frequently or intermittently ? Once I go though the logs >>> I'm unable to find the high input values for load average. But output have >>> high values. >>> >>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO >>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor : >>> JMSOutputAdaptor, sent >>> >>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}} >>> >> >> This is happening intermittently. Experienced now too. This is a serious >> issue. We need to somehow sort this out. >> >> >>> >>> Also networkPartitionId is empty. In the current execution plans it took >>> networkPartitionId for the calculation. IMO we need to update the CEP >>> execution plans for 4.1.0. >>> >>> Thanks ! >>> >>> >>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> Thanks Raj. I'll go through them. >>>> >>>> Thanks ! >>>> >>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam < >>>> [email protected]> wrote: >>>> >>>>> Hi Manula, >>>>> >>>>> You can see those values in the same log I attached in the previous >>>>> mail. >>>>> >>>>> Thanks. >>>>> >>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Chamila, >>>>>> >>>>>> I couldn't find any large values sent by agent. There are traces for >>>>>> CEP sending large values. Find the cep trace log here [1] >>>>>> >>>>>> 1. >>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing >>>>>> >>>>>> Thanks. >>>>>> >>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Shall we enable the cartridge agent debug logs and see what are the >>>>>>> load average values sent from the agent. If those values are normal >>>>>>> then we >>>>>>> can narrow down to the CEP. >>>>>>> >>>>>>> Thanks ! >>>>>>> >>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Raj, >>>>>>>> >>>>>>>> Can you inspect the CEP trace logs for the same time period? It >>>>>>>> will have the events published from the agent. >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Chamila de Alwis >>>>>>>> Software Engineer | WSO2 | +94772207163 >>>>>>>> Blog: code.chamiladealwis.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I am doing a test round with M3 with python agent. And faced this >>>>>>>>> very strange scenario, $Subject. >>>>>>>>> >>>>>>>>> >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Second >>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain >>>>>>>>> [value] >>>>>>>>> 200000.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Second >>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Second >>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Second >>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain >>>>>>>>> [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Second >>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain >>>>>>>>> [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Second >>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] >>>>>>>>> 2000000.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load >>>>>>>>> average >>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Grad of >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - >>>>>>>>> Avg >>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] >>>>>>>>> KubGrp1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG >>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >>>>>>>>> PodActivationWatcher running : Running >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG >>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >>>>>>>>> PodActivationWatcher running : Running >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG >>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >>>>>>>>> PodActivationWatcher running : Running >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} >>>>>>>>> - >>>>>>>>> KubernetesServiceClusterMonitor is running.. >>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1, >>>>>>>>> clusterId=php.php.domain, serviceId=php] >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} >>>>>>>>> - >>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : >>>>>>>>> php.php.domain >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: >>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] >>>>>>>>> [cluster] : >>>>>>>>> php.php.domain [Replicas] nonTerminated : 3 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] >>>>>>>>> [cluster] : >>>>>>>>> php.php.domain [Replicas] minReplicas : 3 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete >>>>>>>>> containers >>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] >>>>>>>>> [cluster] >>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - >>>>>>>>> Minimum >>>>>>>>> check executed for : >>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} >>>>>>>>> - >>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of >>>>>>>>> loadAverageReset : true >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} >>>>>>>>> - >>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : >>>>>>>>> php.php.domain >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule >>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting >>>>>>>>> the >>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , >>>>>>>>> [time >>>>>>>>> intervals]: 1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting >>>>>>>>> the >>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , >>>>>>>>> [time >>>>>>>>> intervals]: 1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting >>>>>>>>> the >>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]: >>>>>>>>> 2000000.0 , [time intervals]: 1 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [Replicas] minReplicas : 3 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [Replicas] maxReplicas : 10 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [Replicas] nonTerminated : 3 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [Replicas] activeReplicas : 3 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [MemoryConsumption] predicted value : >>>>>>>>> 28.799999237060547 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain scale-up action : true >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain scale-down action : false >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>> php.php.domain predicted replicas > max replicas : >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to >>>>>>>>> scale-up : [cluster] : php.php.domain >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] >>>>>>>>> [cluster] : >>>>>>>>> php.php.domain valid number of replicas to expand : 10 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>>>> - Updating kubernetes replication controller via cloud controller: >>>>>>>>> [cluster] php.php.domain [replicas] 10 >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG >>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} >>>>>>>>> - >>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : >>>>>>>>> php.php.domain >>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG >>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} >>>>>>>>> - >>>>>>>>> Cloud Controller is delegating request to update a replication >>>>>>>>> controller >>>>>>>>> php.php.domain to Kubernetes layer. >>>>>>>>> >>>>>>>>> >>>>>>>>> I couldn't investigate the agent side, since containers are >>>>>>>>> destroyed. >>>>>>>>> >>>>>>>>> Any idea? >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Rajkumar Rajaratnam >>>>>>>>> Software Engineer | WSO2, Inc. >>>>>>>>> Mobile +94777568639 | +94783498120 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Manula Chathurika Thantriwatte >>>>>>> Software Engineer >>>>>>> WSO2 Inc. : http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>>> email : [email protected] / [email protected] >>>>>>> phone : +94 772492511 >>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Rajkumar Rajaratnam >>>>>> Software Engineer | WSO2, Inc. >>>>>> Mobile +94777568639 | +94783498120 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Rajkumar Rajaratnam >>>>> Software Engineer | WSO2, Inc. >>>>> Mobile +94777568639 | +94783498120 >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Manula Chathurika Thantriwatte >>>> Software Engineer >>>> WSO2 Inc. : http://wso2.com >>>> lean . enterprise . middleware >>>> >>>> email : [email protected] / [email protected] >>>> phone : +94 772492511 >>>> blog : http://manulachathurika.blogspot.com/ >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> Regards, >>> Manula Chathurika Thantriwatte >>> Software Engineer >>> WSO2 Inc. : http://wso2.com >>> lean . enterprise . middleware >>> >>> email : [email protected] / [email protected] >>> phone : +94 772492511 >>> blog : http://manulachathurika.blogspot.com/ >>> >>> >>> >>> >> >> >> -- >> Rajkumar Rajaratnam >> Software Engineer | WSO2, Inc. >> Mobile +94777568639 | +94783498120 >> > > > > -- > Regards, > Manula Chathurika Thantriwatte > Software Engineer > WSO2 Inc. : http://wso2.com > lean . enterprise . middleware > > email : [email protected] / [email protected] > phone : +94 772492511 > blog : http://manulachathurika.blogspot.com/ > > > > -- Rajkumar Rajaratnam Software Engineer | WSO2, Inc. Mobile +94777568639 | +94783498120
