Hi Lahiru, are MemberActivatedEvents sent by the Stratos agent, or are they sent by the another component (e.g. CloudController)?
On Thu, May 29, 2014 at 7:49 AM, Lahiru Sandaruwan <[email protected]> wrote: > Hi Chris, > > Can you confirm that the Autoscaler received Member activated event for > these particular instances? Logs should help identify that. > > Thanks. > > > > > On Thu, May 29, 2014 at 12:15 PM, chris snow <[email protected]> wrote: >> >> Thanks Nirmal. >> >> Non terminated member count doesn't seem to detect the two active >> instances that have been started: >> >> [2014-05-29 06:06:23,900] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: >> [network-partition] G1 [partition] P1 >> [2014-05-29 06:06:23,906] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] >> [network-partition] G1 [partition] P1 Non terminated member count: 0 >> [2014-05-29 06:06:23,908] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] >> [network-partition] G1 [partition] P1 Minimum member count: 1 >> [2014-05-29 06:06:23,925] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete >> instances rule: [partition] P1 [network-partition] G1 >> [2014-05-29 06:06:23,927] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] >> [network-partition] G1 [partition] P1 Obsolete member count: 0 >> [2014-05-29 06:06:23,930] INFO >> >> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >> - Trying to spawn an instance via cloud controller: [cluster] >> php111.php.domain [partition] P1 [lb-cluster] null >> [network-partition-id] G1 >> >> >> vagrant@puppet:~$ nova list >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+ >> | ID | Name | Status | Task >> State | Power State | Networks | >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+ >> | a3ce6f52-27bf-4343-a05d-d7c9da9a1cb2 | php111php-3cb | ERROR | - >> | NOSTATE | | >> | 82434b60-6c9d-40a3-8191-2d466cde725e | php111php-47e | ERROR | - >> | NOSTATE | | >> | 4aa942ff-a722-4ff4-b604-abaa2f44c1a7 | php111php-499 | ERROR | - >> | NOSTATE | | >> | 1548915b-98f6-48fc-b70a-b0acc74610a7 | php111php-5e0 | ERROR | - >> | NOSTATE | | >> | f313c93d-7de1-4ccb-8c0d-f4104e5d837f | php111php-701 | ERROR | - >> | NOSTATE | | >> | a49598bb-0d4a-4064-af16-5b4b91057fd8 | php111php-719 | ERROR | - >> | NOSTATE | | >> | f8376091-99f6-4d17-8ec6-3935291c3e40 | php111php-841 | ERROR | - >> | NOSTATE | | >> | cf3298da-eecc-448a-bcea-2296eeed23dc | php111php-ad | ACTIVE | - >> | Running | private=10.11.12.2 | >> | 1d8832d5-39e4-476c-9c2b-1b1f5ffe3b41 | php111php-d81 | ACTIVE | - >> | Running | private=10.11.12.3 | >> | 4f6d51fb-fd27-4be1-94be-e9b5200fb6ba | php111php-e9c | ERROR | - >> | NOSTATE | | >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+ >> >> I'll investigate some more... >> >> On Thu, May 29, 2014 at 7:42 AM, Nirmal Fernando <[email protected]> >> wrote: >> > Ah.. probably Chris. So, in min check what happens is it always make >> > sure >> > that you have min instance count in the system. >> > >> > >> > On Thu, May 29, 2014 at 11:42 AM, chris snow <[email protected]> >> > wrote: >> >> >> >> Hi Nirmal, >> >> >> >> I've just see that it is the "Minimal Rule" in mincheck.drl that is >> >> trying to spawn the instance, and not scaling.drl. >> >> >> >> This is why AutoscaleAlgorithm.getNextScaleUpPartition() is not >> >> getting >> >> called? >> >> >> >> Cheers, >> >> >> >> Chris >> >> >> >> >> >> On Wed, May 28, 2014 at 11:24 PM, Nirmal Fernando >> >> <[email protected]> wrote: >> >> > No, Chris. This happens via Drools rules located in scaling.drl file >> >> > of >> >> > conf >> >> > directory. >> >> > >> >> > >> >> > On Thu, May 29, 2014 at 3:39 AM, chris snow <[email protected]> >> >> > wrote: >> >> >> >> >> >> Hi Nirmal, >> >> >> >> >> >> For some reason, that method is not getting called. I've pasted my >> >> >> stacktrace again with the drools methods removed: >> >> >> >> >> >> --- >> >> >> Daemon Thread [Thread-56] (Suspended (breakpoint at line 137 in >> >> >> CloudControllerClient)) >> >> >> owns: CloudControllerClient (id=2433) >> >> >> owns: DefaultAgenda (id=2434) >> >> >> CloudControllerClient.spawnAnInstance(Partition, String, String, >> >> >> String) line: 137 >> >> >> RuleTasksDelegator.delegateSpawn(PartitionContext, String, >> >> >> String) >> >> >> line: 92 >> >> >> // >> >> >> // drools methods hidden >> >> >> // >> >> >> >> >> >> AutoscalerRuleEvaluator.evaluateMinCheck(StatefulKnowledgeSession, >> >> >> FactHandle, Object) line: 86 >> >> >> ClusterMonitor.monitor() line: 99 >> >> >> ClusterMonitor.run() line: 72 >> >> >> Thread.run() line: 744 >> >> >> --- >> >> >> >> >> >> Would you expect to see AutoscaleAlgorithm.getNextScaleUpPartition >> >> >> executed somewhere in the stack of calls? >> >> >> >> >> >> Many thanks, >> >> >> >> >> >> Chris >> >> >> >> >> >> On Wed, May 28, 2014 at 11:03 PM, Nirmal Fernando >> >> >> <[email protected]> wrote: >> >> >> > Hi Chris, >> >> >> > >> >> >> > PartitionMax is considered in >> >> >> > AutoscaleAlgorithm.getNextScaleUpPartition >> >> >> > method. >> >> >> > >> >> >> > >> >> >> > On Thu, May 29, 2014 at 3:09 AM, chris snow <[email protected]> >> >> >> > wrote: >> >> >> >> >> >> >> >> I setup a breakpoint in eclipse at >> >> >> >> CloudControllerClient.spawnAnInstance() and dumped the output >> >> >> >> (below). >> >> >> >> >> >> >> >> I've scanned through the code in the stratos methods in the >> >> >> >> stacktrace, but couldn't see any code that checks the >> >> >> >> partitionMax >> >> >> >> value. >> >> >> >> >> >> >> >> Any ideas? >> >> >> >> >> >> >> >> Daemon Thread [Thread-56] (Suspended (breakpoint at line 137 in >> >> >> >> CloudControllerClient)) >> >> >> >> owns: CloudControllerClient (id=2433) >> >> >> >> owns: DefaultAgenda (id=2434) >> >> >> >> CloudControllerClient.spawnAnInstance(Partition, String, >> >> >> >> String, >> >> >> >> String) line: 137 >> >> >> >> RuleTasksDelegator.delegateSpawn(PartitionContext, String, >> >> >> >> String) >> >> >> >> line: 92 >> >> >> >> NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) >> >> >> >> line: >> >> >> >> not available [native method] >> >> >> >> NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57 >> >> >> >> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43 >> >> >> >> Method.invoke(Object, Object...) line: 606 >> >> >> >> ReflectiveAccessorOptimizer.getMethod(Object, String, >> >> >> >> Object[], >> >> >> >> Class[], ExecutableStatement[]) line: 1104 >> >> >> >> ReflectiveAccessorOptimizer.getMethod(Object, String) line: >> >> >> >> 987 >> >> >> >> ReflectiveAccessorOptimizer.compileGetChain() line: 377 >> >> >> >> ReflectiveAccessorOptimizer.optimizeAccessor(ParserContext, >> >> >> >> char[], >> >> >> >> int, int, Object, Object, VariableResolverFactory, boolean, >> >> >> >> Class) >> >> >> >> line: 143 >> >> >> >> ASTNode.optimize(Object, Object, VariableResolverFactory) >> >> >> >> line: >> >> >> >> 159 >> >> >> >> ASTNode.getReducedValueAccelerated(Object, Object, >> >> >> >> VariableResolverFactory) line: 115 >> >> >> >> MVELRuntime.execute(boolean, CompiledExpression, Object, >> >> >> >> VariableResolverFactory) line: 85 >> >> >> >> CompiledExpression.getDirectValue(Object, >> >> >> >> VariableResolverFactory) >> >> >> >> line: 123 >> >> >> >> CompiledExpression.getValue(Object, VariableResolverFactory) >> >> >> >> line: >> >> >> >> 119 >> >> >> >> MVEL.executeExpression(Object, Object, >> >> >> >> VariableResolverFactory) >> >> >> >> line: >> >> >> >> 930 >> >> >> >> MVELConsequence.evaluate(KnowledgeHelper, WorkingMemory) line: >> >> >> >> 104 >> >> >> >> DefaultAgenda.fireActivation(Activation) line: 1287 >> >> >> >> DefaultAgenda.fireNextItem(AgendaFilter) line: 1221 >> >> >> >> DefaultAgenda.fireAllRules(AgendaFilter, int) line: 1456 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ReteooStatefulSession(AbstractWorkingMemory).fireAllRules(AgendaFilter, >> >> >> >> int) line: 710 >> >> >> >> ReteooStatefulSession(AbstractWorkingMemory).fireAllRules() >> >> >> >> line: >> >> >> >> 674 >> >> >> >> StatefulKnowledgeSessionImpl.fireAllRules() line: 230 >> >> >> >> >> >> >> >> >> >> >> >> AutoscalerRuleEvaluator.evaluateMinCheck(StatefulKnowledgeSession, >> >> >> >> FactHandle, Object) line: 86 >> >> >> >> ClusterMonitor.monitor() line: 99 >> >> >> >> ClusterMonitor.run() line: 72 >> >> >> >> Thread.run() line: 744 >> >> >> >> >> >> >> >> On Wed, May 28, 2014 at 3:37 AM, Lahiru Sandaruwan >> >> >> >> <[email protected]> >> >> >> >> wrote: >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > On Tue, May 27, 2014 at 11:54 PM, chris snow >> >> >> >> > <[email protected]> >> >> >> >> > wrote: >> >> >> >> >> >> >> >> >> >> Hi Lahiru, >> >> >> >> >> >> >> >> >> >> I'll have a poke around - it will give me a good opportunity >> >> >> >> >> to >> >> >> >> >> learn >> >> >> >> >> some more Stratos code :) >> >> >> >> >> >> >> >> >> > >> >> >> >> > Cool :) >> >> >> >> > >> >> >> >> > Debug logs should be able to show us where it went wrong. >> >> >> >> > >> >> >> >> > Thanks. >> >> >> >> >> >> >> >> >> >> Many thanks, >> >> >> >> >> >> >> >> >> >> Chris >> >> >> >> >> >> >> >> >> >> On Tue, May 27, 2014 at 7:45 AM, Lahiru Sandaruwan >> >> >> >> >> <[email protected]> >> >> >> >> >> wrote: >> >> >> >> >> > Hi Chris, >> >> >> >> >> > >> >> >> >> >> > Can you enable debug logs and try out? >> >> >> >> >> > If so please send them over. >> >> >> >> >> > >> >> >> >> >> > Thanks. >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > On Tue, May 27, 2014 at 12:05 PM, chris snow >> >> >> >> >> > <[email protected]> >> >> >> >> >> > wrote: >> >> >> >> >> >> >> >> >> >> >> >> Deployment Policy partitionMax seems to be ignored. My >> >> >> >> >> >> deployment >> >> >> >> >> >> policy looks like this: >> >> >> >> >> >> >> >> >> >> >> >> { >> >> >> >> >> >> "id": "economyDeployment", >> >> >> >> >> >> "partitionGroup": [ >> >> >> >> >> >> { >> >> >> >> >> >> "id": "G1", >> >> >> >> >> >> "partitionAlgo": "one-after-another", >> >> >> >> >> >> "partition": [ >> >> >> >> >> >> { >> >> >> >> >> >> "id": "P1", >> >> >> >> >> >> "partitionMax": "1", >> >> >> >> >> >> "partitionMin": "1" >> >> >> >> >> >> } >> >> >> >> >> >> ] >> >> >> >> >> >> } >> >> >> >> >> >> ] >> >> >> >> >> >> } >> >> >> >> >> >> >> >> >> >> >> >> Stratos initially starts two instances: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+ >> >> >> >> >> >> | ID | Name | >> >> >> >> >> >> Status | >> >> >> >> >> >> Task >> >> >> >> >> >> State | Power State | Networks | >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+ >> >> >> >> >> >> | c91eddd7-f2d5-49e5-b602-fde5df38e16d | php111php-9f6 | >> >> >> >> >> >> ACTIVE | >> >> >> >> >> >> - >> >> >> >> >> >> | Running | private=10.11.12.2 | >> >> >> >> >> >> | b38536d2-bc9e-4ab2-984c-bc3f2091947f | php111php-a05 | >> >> >> >> >> >> ACTIVE | >> >> >> >> >> >> - >> >> >> >> >> >> | Running | private=10.11.12.3 | >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+ >> >> >> >> >> >> Then attempts to start more: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+ >> >> >> >> >> >> | ID | Name | >> >> >> >> >> >> Status | >> >> >> >> >> >> Task >> >> >> >> >> >> State | Power State | Networks | >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+ >> >> >> >> >> >> | 91029130-37b5-4df6-ad95-1d9f609a22c3 | php111php-13e | >> >> >> >> >> >> ERROR >> >> >> >> >> >> | >> >> >> >> >> >> - >> >> >> >> >> >> | NOSTATE | | >> >> >> >> >> >> | 71c1e361-df55-4f61-84c5-79774fef2beb | php111php-3d7 | >> >> >> >> >> >> ERROR >> >> >> >> >> >> | >> >> >> >> >> >> - >> >> >> >> >> >> | NOSTATE | | >> >> >> >> >> >> | b0d72602-cdb8-4621-ad12-911ade4d89b1 | php111php-412 | >> >> >> >> >> >> ERROR >> >> >> >> >> >> | >> >> >> >> >> >> - >> >> >> >> >> >> | NOSTATE | | >> >> >> >> >> >> | d4eac55d-bb7c-48d6-83f2-cb9a11581a18 | php111php-6f5 | >> >> >> >> >> >> ERROR >> >> >> >> >> >> | >> >> >> >> >> >> - >> >> >> >> >> >> | NOSTATE | | >> >> >> >> >> >> | 738a9534-2b95-4984-a51b-748bc7169b9a | php111php-73e | >> >> >> >> >> >> ERROR >> >> >> >> >> >> | >> >> >> >> >> >> - >> >> >> >> >> >> | NOSTATE | | >> >> >> >> >> >> | c91eddd7-f2d5-49e5-b602-fde5df38e16d | php111php-9f6 | >> >> >> >> >> >> ACTIVE | >> >> >> >> >> >> - >> >> >> >> >> >> | Running | private=10.11.12.2 | >> >> >> >> >> >> | b38536d2-bc9e-4ab2-984c-bc3f2091947f | php111php-a05 | >> >> >> >> >> >> ACTIVE | >> >> >> >> >> >> - >> >> >> >> >> >> | Running | private=10.11.12.3 | >> >> >> >> >> >> | f570e654-024b-4240-bad2-8a11f04f23f0 | php111php-a30 | >> >> >> >> >> >> ERROR >> >> >> >> >> >> | >> >> >> >> >> >> - >> >> >> >> >> >> | NOSTATE | | >> >> >> >> >> >> | 0ab36824-9bea-4a95-aa22-7f935406d83d | php111php-bca | >> >> >> >> >> >> ERROR >> >> >> >> >> >> | >> >> >> >> >> >> - >> >> >> >> >> >> | NOSTATE | | >> >> >> >> >> >> | 02ed97df-02bc-4bff-b42d-9f2bcbd4f569 | php111php-f6f | >> >> >> >> >> >> ERROR >> >> >> >> >> >> | >> >> >> >> >> >> - >> >> >> >> >> >> | NOSTATE | | >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+ >> >> >> >> >> >> >> >> >> >> >> >> I'm on branch: 4.0.0-incubating, using Openstack Havana + >> >> >> >> >> >> Docker. >> >> >> >> >> >> >> >> >> >> >> >> Any ideas? >> >> >> >> >> >> >> >> >> >> >> >> Many thanks, >> >> >> >> >> >> >> >> >> >> >> >> Chris >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > -- >> >> >> >> >> > -- >> >> >> >> >> > Lahiru Sandaruwan >> >> >> >> >> > Committer and PMC member, Apache Stratos, >> >> >> >> >> > Senior Software Engineer, >> >> >> >> >> > WSO2 Inc., http://wso2.com >> >> >> >> >> > lean.enterprise.middleware >> >> >> >> >> > >> >> >> >> >> > email: [email protected] cell: (+94) 773 325 954 >> >> >> >> >> > blog: http://lahiruwrites.blogspot.com/ >> >> >> >> >> > twitter: http://twitter.com/lahirus >> >> >> >> >> > linked-in: >> >> >> >> >> > http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> Check out my professional profile and connect with me on >> >> >> >> >> LinkedIn. >> >> >> >> >> http://lnkd.in/cw5k69 >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > -- >> >> >> >> > -- >> >> >> >> > Lahiru Sandaruwan >> >> >> >> > Committer and PMC member, Apache Stratos, >> >> >> >> > Senior Software Engineer, >> >> >> >> > WSO2 Inc., http://wso2.com >> >> >> >> > lean.enterprise.middleware >> >> >> >> > >> >> >> >> > email: [email protected] cell: (+94) 773 325 954 >> >> >> >> > blog: http://lahiruwrites.blogspot.com/ >> >> >> >> > twitter: http://twitter.com/lahirus >> >> >> >> > linked-in: >> >> >> >> > http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Check out my professional profile and connect with me on >> >> >> >> LinkedIn. >> >> >> >> http://lnkd.in/cw5k69 >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > Best Regards, >> >> >> > Nirmal >> >> >> > >> >> >> > Nirmal Fernando. >> >> >> > PPMC Member & Committer of Apache Stratos, >> >> >> > >> >> >> > Senior Software Engineer, WSO2 Inc. >> >> >> > >> >> >> > Blog: http://nirmalfdo.blogspot.com/ >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Check out my professional profile and connect with me on LinkedIn. >> >> >> http://lnkd.in/cw5k69 >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > Best Regards, >> >> > Nirmal >> >> > >> >> > Nirmal Fernando. >> >> > PPMC Member & Committer of Apache Stratos, >> >> > Senior Software Engineer, WSO2 Inc. >> >> > >> >> > Blog: http://nirmalfdo.blogspot.com/ >> >> >> >> >> >> >> >> -- >> >> Check out my professional profile and connect with me on LinkedIn. >> >> http://lnkd.in/cw5k69 >> > >> > >> > >> > >> > -- >> > Best Regards, >> > Nirmal >> > >> > Nirmal Fernando. >> > PPMC Member & Committer of Apache Stratos, >> > Senior Software Engineer, WSO2 Inc. >> > >> > Blog: http://nirmalfdo.blogspot.com/ >> >> >> >> -- >> Check out my professional profile and connect with me on LinkedIn. >> http://lnkd.in/cw5k69 > > > > > -- > -- > Lahiru Sandaruwan > Committer and PMC member, Apache Stratos, > Senior Software Engineer, > WSO2 Inc., http://wso2.com > lean.enterprise.middleware > > email: [email protected] cell: (+94) 773 325 954 > blog: http://lahiruwrites.blogspot.com/ > twitter: http://twitter.com/lahirus > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > -- Check out my professional profile and connect with me on LinkedIn. http://lnkd.in/cw5k69
