Hi Chris, Can you confirm that the Autoscaler received Member activated event for these particular instances? Logs should help identify that.
Thanks. On Thu, May 29, 2014 at 12:15 PM, chris snow <[email protected]> wrote: > Thanks Nirmal. > > Non terminated member count doesn't seem to detect the two active > instances that have been started: > > [2014-05-29 06:06:23,900] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: > [network-partition] G1 [partition] P1 > [2014-05-29 06:06:23,906] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] > [network-partition] G1 [partition] P1 Non terminated member count: 0 > [2014-05-29 06:06:23,908] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] > [network-partition] G1 [partition] P1 Minimum member count: 1 > [2014-05-29 06:06:23,925] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete > instances rule: [partition] P1 [network-partition] G1 > [2014-05-29 06:06:23,927] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] > [network-partition] G1 [partition] P1 Obsolete member count: 0 > [2014-05-29 06:06:23,930] INFO > > {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} > - Trying to spawn an instance via cloud controller: [cluster] > php111.php.domain [partition] P1 [lb-cluster] null > [network-partition-id] G1 > > > vagrant@puppet:~$ nova list > > +--------------------------------------+---------------+--------+------------+-------------+--------------------+ > | ID | Name | Status | Task > State | Power State | Networks | > > +--------------------------------------+---------------+--------+------------+-------------+--------------------+ > | a3ce6f52-27bf-4343-a05d-d7c9da9a1cb2 | php111php-3cb | ERROR | - > | NOSTATE | | > | 82434b60-6c9d-40a3-8191-2d466cde725e | php111php-47e | ERROR | - > | NOSTATE | | > | 4aa942ff-a722-4ff4-b604-abaa2f44c1a7 | php111php-499 | ERROR | - > | NOSTATE | | > | 1548915b-98f6-48fc-b70a-b0acc74610a7 | php111php-5e0 | ERROR | - > | NOSTATE | | > | f313c93d-7de1-4ccb-8c0d-f4104e5d837f | php111php-701 | ERROR | - > | NOSTATE | | > | a49598bb-0d4a-4064-af16-5b4b91057fd8 | php111php-719 | ERROR | - > | NOSTATE | | > | f8376091-99f6-4d17-8ec6-3935291c3e40 | php111php-841 | ERROR | - > | NOSTATE | | > | cf3298da-eecc-448a-bcea-2296eeed23dc | php111php-ad | ACTIVE | - > | Running | private=10.11.12.2 | > | 1d8832d5-39e4-476c-9c2b-1b1f5ffe3b41 | php111php-d81 | ACTIVE | - > | Running | private=10.11.12.3 | > | 4f6d51fb-fd27-4be1-94be-e9b5200fb6ba | php111php-e9c | ERROR | - > | NOSTATE | | > > +--------------------------------------+---------------+--------+------------+-------------+--------------------+ > > I'll investigate some more... > > On Thu, May 29, 2014 at 7:42 AM, Nirmal Fernando <[email protected]> > wrote: > > Ah.. probably Chris. So, in min check what happens is it always make sure > > that you have min instance count in the system. > > > > > > On Thu, May 29, 2014 at 11:42 AM, chris snow <[email protected]> > wrote: > >> > >> Hi Nirmal, > >> > >> I've just see that it is the "Minimal Rule" in mincheck.drl that is > >> trying to spawn the instance, and not scaling.drl. > >> > >> This is why AutoscaleAlgorithm.getNextScaleUpPartition() is not getting > >> called? > >> > >> Cheers, > >> > >> Chris > >> > >> > >> On Wed, May 28, 2014 at 11:24 PM, Nirmal Fernando > >> <[email protected]> wrote: > >> > No, Chris. This happens via Drools rules located in scaling.drl file > of > >> > conf > >> > directory. > >> > > >> > > >> > On Thu, May 29, 2014 at 3:39 AM, chris snow <[email protected]> > wrote: > >> >> > >> >> Hi Nirmal, > >> >> > >> >> For some reason, that method is not getting called. I've pasted my > >> >> stacktrace again with the drools methods removed: > >> >> > >> >> --- > >> >> Daemon Thread [Thread-56] (Suspended (breakpoint at line 137 in > >> >> CloudControllerClient)) > >> >> owns: CloudControllerClient (id=2433) > >> >> owns: DefaultAgenda (id=2434) > >> >> CloudControllerClient.spawnAnInstance(Partition, String, String, > >> >> String) line: 137 > >> >> RuleTasksDelegator.delegateSpawn(PartitionContext, String, String) > >> >> line: 92 > >> >> // > >> >> // drools methods hidden > >> >> // > >> >> AutoscalerRuleEvaluator.evaluateMinCheck(StatefulKnowledgeSession, > >> >> FactHandle, Object) line: 86 > >> >> ClusterMonitor.monitor() line: 99 > >> >> ClusterMonitor.run() line: 72 > >> >> Thread.run() line: 744 > >> >> --- > >> >> > >> >> Would you expect to see AutoscaleAlgorithm.getNextScaleUpPartition > >> >> executed somewhere in the stack of calls? > >> >> > >> >> Many thanks, > >> >> > >> >> Chris > >> >> > >> >> On Wed, May 28, 2014 at 11:03 PM, Nirmal Fernando > >> >> <[email protected]> wrote: > >> >> > Hi Chris, > >> >> > > >> >> > PartitionMax is considered in > >> >> > AutoscaleAlgorithm.getNextScaleUpPartition > >> >> > method. > >> >> > > >> >> > > >> >> > On Thu, May 29, 2014 at 3:09 AM, chris snow <[email protected]> > >> >> > wrote: > >> >> >> > >> >> >> I setup a breakpoint in eclipse at > >> >> >> CloudControllerClient.spawnAnInstance() and dumped the output > >> >> >> (below). > >> >> >> > >> >> >> I've scanned through the code in the stratos methods in the > >> >> >> stacktrace, but couldn't see any code that checks the partitionMax > >> >> >> value. > >> >> >> > >> >> >> Any ideas? > >> >> >> > >> >> >> Daemon Thread [Thread-56] (Suspended (breakpoint at line 137 in > >> >> >> CloudControllerClient)) > >> >> >> owns: CloudControllerClient (id=2433) > >> >> >> owns: DefaultAgenda (id=2434) > >> >> >> CloudControllerClient.spawnAnInstance(Partition, String, > String, > >> >> >> String) line: 137 > >> >> >> RuleTasksDelegator.delegateSpawn(PartitionContext, String, > >> >> >> String) > >> >> >> line: 92 > >> >> >> NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) > line: > >> >> >> not available [native method] > >> >> >> NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57 > >> >> >> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43 > >> >> >> Method.invoke(Object, Object...) line: 606 > >> >> >> ReflectiveAccessorOptimizer.getMethod(Object, String, Object[], > >> >> >> Class[], ExecutableStatement[]) line: 1104 > >> >> >> ReflectiveAccessorOptimizer.getMethod(Object, String) line: 987 > >> >> >> ReflectiveAccessorOptimizer.compileGetChain() line: 377 > >> >> >> ReflectiveAccessorOptimizer.optimizeAccessor(ParserContext, > >> >> >> char[], > >> >> >> int, int, Object, Object, VariableResolverFactory, boolean, Class) > >> >> >> line: 143 > >> >> >> ASTNode.optimize(Object, Object, VariableResolverFactory) line: > >> >> >> 159 > >> >> >> ASTNode.getReducedValueAccelerated(Object, Object, > >> >> >> VariableResolverFactory) line: 115 > >> >> >> MVELRuntime.execute(boolean, CompiledExpression, Object, > >> >> >> VariableResolverFactory) line: 85 > >> >> >> CompiledExpression.getDirectValue(Object, > >> >> >> VariableResolverFactory) > >> >> >> line: 123 > >> >> >> CompiledExpression.getValue(Object, VariableResolverFactory) > >> >> >> line: > >> >> >> 119 > >> >> >> MVEL.executeExpression(Object, Object, VariableResolverFactory) > >> >> >> line: > >> >> >> 930 > >> >> >> MVELConsequence.evaluate(KnowledgeHelper, WorkingMemory) line: > >> >> >> 104 > >> >> >> DefaultAgenda.fireActivation(Activation) line: 1287 > >> >> >> DefaultAgenda.fireNextItem(AgendaFilter) line: 1221 > >> >> >> DefaultAgenda.fireAllRules(AgendaFilter, int) line: 1456 > >> >> >> > >> >> >> > >> >> >> > ReteooStatefulSession(AbstractWorkingMemory).fireAllRules(AgendaFilter, > >> >> >> int) line: 710 > >> >> >> ReteooStatefulSession(AbstractWorkingMemory).fireAllRules() > line: > >> >> >> 674 > >> >> >> StatefulKnowledgeSessionImpl.fireAllRules() line: 230 > >> >> >> > >> >> >> AutoscalerRuleEvaluator.evaluateMinCheck(StatefulKnowledgeSession, > >> >> >> FactHandle, Object) line: 86 > >> >> >> ClusterMonitor.monitor() line: 99 > >> >> >> ClusterMonitor.run() line: 72 > >> >> >> Thread.run() line: 744 > >> >> >> > >> >> >> On Wed, May 28, 2014 at 3:37 AM, Lahiru Sandaruwan > >> >> >> <[email protected]> > >> >> >> wrote: > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > On Tue, May 27, 2014 at 11:54 PM, chris snow < > [email protected]> > >> >> >> > wrote: > >> >> >> >> > >> >> >> >> Hi Lahiru, > >> >> >> >> > >> >> >> >> I'll have a poke around - it will give me a good opportunity to > >> >> >> >> learn > >> >> >> >> some more Stratos code :) > >> >> >> >> > >> >> >> > > >> >> >> > Cool :) > >> >> >> > > >> >> >> > Debug logs should be able to show us where it went wrong. > >> >> >> > > >> >> >> > Thanks. > >> >> >> >> > >> >> >> >> Many thanks, > >> >> >> >> > >> >> >> >> Chris > >> >> >> >> > >> >> >> >> On Tue, May 27, 2014 at 7:45 AM, Lahiru Sandaruwan > >> >> >> >> <[email protected]> > >> >> >> >> wrote: > >> >> >> >> > Hi Chris, > >> >> >> >> > > >> >> >> >> > Can you enable debug logs and try out? > >> >> >> >> > If so please send them over. > >> >> >> >> > > >> >> >> >> > Thanks. > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > On Tue, May 27, 2014 at 12:05 PM, chris snow > >> >> >> >> > <[email protected]> > >> >> >> >> > wrote: > >> >> >> >> >> > >> >> >> >> >> Deployment Policy partitionMax seems to be ignored. My > >> >> >> >> >> deployment > >> >> >> >> >> policy looks like this: > >> >> >> >> >> > >> >> >> >> >> { > >> >> >> >> >> "id": "economyDeployment", > >> >> >> >> >> "partitionGroup": [ > >> >> >> >> >> { > >> >> >> >> >> "id": "G1", > >> >> >> >> >> "partitionAlgo": "one-after-another", > >> >> >> >> >> "partition": [ > >> >> >> >> >> { > >> >> >> >> >> "id": "P1", > >> >> >> >> >> "partitionMax": "1", > >> >> >> >> >> "partitionMin": "1" > >> >> >> >> >> } > >> >> >> >> >> ] > >> >> >> >> >> } > >> >> >> >> >> ] > >> >> >> >> >> } > >> >> >> >> >> > >> >> >> >> >> Stratos initially starts two instances: > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > +--------------------------------------+---------------+--------+------------+-------------+--------------------+ > >> >> >> >> >> | ID | Name | > >> >> >> >> >> Status | > >> >> >> >> >> Task > >> >> >> >> >> State | Power State | Networks | > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > +--------------------------------------+---------------+--------+------------+-------------+--------------------+ > >> >> >> >> >> | c91eddd7-f2d5-49e5-b602-fde5df38e16d | php111php-9f6 | > >> >> >> >> >> ACTIVE | > >> >> >> >> >> - > >> >> >> >> >> | Running | private=10.11.12.2 | > >> >> >> >> >> | b38536d2-bc9e-4ab2-984c-bc3f2091947f | php111php-a05 | > >> >> >> >> >> ACTIVE | > >> >> >> >> >> - > >> >> >> >> >> | Running | private=10.11.12.3 | > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > +--------------------------------------+---------------+--------+------------+-------------+--------------------+ > >> >> >> >> >> Then attempts to start more: > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > +--------------------------------------+---------------+--------+------------+-------------+--------------------+ > >> >> >> >> >> | ID | Name | > >> >> >> >> >> Status | > >> >> >> >> >> Task > >> >> >> >> >> State | Power State | Networks | > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > +--------------------------------------+---------------+--------+------------+-------------+--------------------+ > >> >> >> >> >> | 91029130-37b5-4df6-ad95-1d9f609a22c3 | php111php-13e | > ERROR > >> >> >> >> >> | > >> >> >> >> >> - > >> >> >> >> >> | NOSTATE | | > >> >> >> >> >> | 71c1e361-df55-4f61-84c5-79774fef2beb | php111php-3d7 | > ERROR > >> >> >> >> >> | > >> >> >> >> >> - > >> >> >> >> >> | NOSTATE | | > >> >> >> >> >> | b0d72602-cdb8-4621-ad12-911ade4d89b1 | php111php-412 | > ERROR > >> >> >> >> >> | > >> >> >> >> >> - > >> >> >> >> >> | NOSTATE | | > >> >> >> >> >> | d4eac55d-bb7c-48d6-83f2-cb9a11581a18 | php111php-6f5 | > ERROR > >> >> >> >> >> | > >> >> >> >> >> - > >> >> >> >> >> | NOSTATE | | > >> >> >> >> >> | 738a9534-2b95-4984-a51b-748bc7169b9a | php111php-73e | > ERROR > >> >> >> >> >> | > >> >> >> >> >> - > >> >> >> >> >> | NOSTATE | | > >> >> >> >> >> | c91eddd7-f2d5-49e5-b602-fde5df38e16d | php111php-9f6 | > >> >> >> >> >> ACTIVE | > >> >> >> >> >> - > >> >> >> >> >> | Running | private=10.11.12.2 | > >> >> >> >> >> | b38536d2-bc9e-4ab2-984c-bc3f2091947f | php111php-a05 | > >> >> >> >> >> ACTIVE | > >> >> >> >> >> - > >> >> >> >> >> | Running | private=10.11.12.3 | > >> >> >> >> >> | f570e654-024b-4240-bad2-8a11f04f23f0 | php111php-a30 | > ERROR > >> >> >> >> >> | > >> >> >> >> >> - > >> >> >> >> >> | NOSTATE | | > >> >> >> >> >> | 0ab36824-9bea-4a95-aa22-7f935406d83d | php111php-bca | > ERROR > >> >> >> >> >> | > >> >> >> >> >> - > >> >> >> >> >> | NOSTATE | | > >> >> >> >> >> | 02ed97df-02bc-4bff-b42d-9f2bcbd4f569 | php111php-f6f | > ERROR > >> >> >> >> >> | > >> >> >> >> >> - > >> >> >> >> >> | NOSTATE | | > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > +--------------------------------------+---------------+--------+------------+-------------+--------------------+ > >> >> >> >> >> > >> >> >> >> >> I'm on branch: 4.0.0-incubating, using Openstack Havana + > >> >> >> >> >> Docker. > >> >> >> >> >> > >> >> >> >> >> Any ideas? > >> >> >> >> >> > >> >> >> >> >> Many thanks, > >> >> >> >> >> > >> >> >> >> >> Chris > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > -- > >> >> >> >> > -- > >> >> >> >> > Lahiru Sandaruwan > >> >> >> >> > Committer and PMC member, Apache Stratos, > >> >> >> >> > Senior Software Engineer, > >> >> >> >> > WSO2 Inc., http://wso2.com > >> >> >> >> > lean.enterprise.middleware > >> >> >> >> > > >> >> >> >> > email: [email protected] cell: (+94) 773 325 954 > >> >> >> >> > blog: http://lahiruwrites.blogspot.com/ > >> >> >> >> > twitter: http://twitter.com/lahirus > >> >> >> >> > linked-in: > >> >> >> >> > http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > >> >> >> >> > > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> -- > >> >> >> >> Check out my professional profile and connect with me on > >> >> >> >> LinkedIn. > >> >> >> >> http://lnkd.in/cw5k69 > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > -- > >> >> >> > Lahiru Sandaruwan > >> >> >> > Committer and PMC member, Apache Stratos, > >> >> >> > Senior Software Engineer, > >> >> >> > WSO2 Inc., http://wso2.com > >> >> >> > lean.enterprise.middleware > >> >> >> > > >> >> >> > email: [email protected] cell: (+94) 773 325 954 > >> >> >> > blog: http://lahiruwrites.blogspot.com/ > >> >> >> > twitter: http://twitter.com/lahirus > >> >> >> > linked-in: > http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > >> >> >> > > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> Check out my professional profile and connect with me on LinkedIn. > >> >> >> http://lnkd.in/cw5k69 > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > Best Regards, > >> >> > Nirmal > >> >> > > >> >> > Nirmal Fernando. > >> >> > PPMC Member & Committer of Apache Stratos, > >> >> > > >> >> > Senior Software Engineer, WSO2 Inc. > >> >> > > >> >> > Blog: http://nirmalfdo.blogspot.com/ > >> >> > >> >> > >> >> > >> >> -- > >> >> Check out my professional profile and connect with me on LinkedIn. > >> >> http://lnkd.in/cw5k69 > >> > > >> > > >> > > >> > > >> > -- > >> > Best Regards, > >> > Nirmal > >> > > >> > Nirmal Fernando. > >> > PPMC Member & Committer of Apache Stratos, > >> > Senior Software Engineer, WSO2 Inc. > >> > > >> > Blog: http://nirmalfdo.blogspot.com/ > >> > >> > >> > >> -- > >> Check out my professional profile and connect with me on LinkedIn. > >> http://lnkd.in/cw5k69 > > > > > > > > > > -- > > Best Regards, > > Nirmal > > > > Nirmal Fernando. > > PPMC Member & Committer of Apache Stratos, > > Senior Software Engineer, WSO2 Inc. > > > > Blog: http://nirmalfdo.blogspot.com/ > > > > -- > Check out my professional profile and connect with me on LinkedIn. > http://lnkd.in/cw5k69 > -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware email: [email protected] cell: (+94) 773 325 954 blog: http://lahiruwrites.blogspot.com/ twitter: http://twitter.com/lahirus linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
