Thanks Nirmal.
Non terminated member count doesn't seem to detect the two active
instances that have been started:
[2014-05-29 06:06:23,900] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
[network-partition] G1 [partition] P1
[2014-05-29 06:06:23,906] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
[network-partition] G1 [partition] P1 Non terminated member count: 0
[2014-05-29 06:06:23,908] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
[network-partition] G1 [partition] P1 Minimum member count: 1
[2014-05-29 06:06:23,925] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete
instances rule: [partition] P1 [network-partition] G1
[2014-05-29 06:06:23,927] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check]
[network-partition] G1 [partition] P1 Obsolete member count: 0
[2014-05-29 06:06:23,930] INFO
{org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
- Trying to spawn an instance via cloud controller: [cluster]
php111.php.domain [partition] P1 [lb-cluster] null
[network-partition-id] G1
vagrant@puppet:~$ nova list
+--------------------------------------+---------------+--------+------------+-------------+--------------------+
| ID | Name | Status | Task
State | Power State | Networks |
+--------------------------------------+---------------+--------+------------+-------------+--------------------+
| a3ce6f52-27bf-4343-a05d-d7c9da9a1cb2 | php111php-3cb | ERROR | -
| NOSTATE | |
| 82434b60-6c9d-40a3-8191-2d466cde725e | php111php-47e | ERROR | -
| NOSTATE | |
| 4aa942ff-a722-4ff4-b604-abaa2f44c1a7 | php111php-499 | ERROR | -
| NOSTATE | |
| 1548915b-98f6-48fc-b70a-b0acc74610a7 | php111php-5e0 | ERROR | -
| NOSTATE | |
| f313c93d-7de1-4ccb-8c0d-f4104e5d837f | php111php-701 | ERROR | -
| NOSTATE | |
| a49598bb-0d4a-4064-af16-5b4b91057fd8 | php111php-719 | ERROR | -
| NOSTATE | |
| f8376091-99f6-4d17-8ec6-3935291c3e40 | php111php-841 | ERROR | -
| NOSTATE | |
| cf3298da-eecc-448a-bcea-2296eeed23dc | php111php-ad | ACTIVE | -
| Running | private=10.11.12.2 |
| 1d8832d5-39e4-476c-9c2b-1b1f5ffe3b41 | php111php-d81 | ACTIVE | -
| Running | private=10.11.12.3 |
| 4f6d51fb-fd27-4be1-94be-e9b5200fb6ba | php111php-e9c | ERROR | -
| NOSTATE | |
+--------------------------------------+---------------+--------+------------+-------------+--------------------+
I'll investigate some more...
On Thu, May 29, 2014 at 7:42 AM, Nirmal Fernando <[email protected]> wrote:
> Ah.. probably Chris. So, in min check what happens is it always make sure
> that you have min instance count in the system.
>
>
> On Thu, May 29, 2014 at 11:42 AM, chris snow <[email protected]> wrote:
>>
>> Hi Nirmal,
>>
>> I've just see that it is the "Minimal Rule" in mincheck.drl that is
>> trying to spawn the instance, and not scaling.drl.
>>
>> This is why AutoscaleAlgorithm.getNextScaleUpPartition() is not getting
>> called?
>>
>> Cheers,
>>
>> Chris
>>
>>
>> On Wed, May 28, 2014 at 11:24 PM, Nirmal Fernando
>> <[email protected]> wrote:
>> > No, Chris. This happens via Drools rules located in scaling.drl file of
>> > conf
>> > directory.
>> >
>> >
>> > On Thu, May 29, 2014 at 3:39 AM, chris snow <[email protected]> wrote:
>> >>
>> >> Hi Nirmal,
>> >>
>> >> For some reason, that method is not getting called. I've pasted my
>> >> stacktrace again with the drools methods removed:
>> >>
>> >> ---
>> >> Daemon Thread [Thread-56] (Suspended (breakpoint at line 137 in
>> >> CloudControllerClient))
>> >> owns: CloudControllerClient (id=2433)
>> >> owns: DefaultAgenda (id=2434)
>> >> CloudControllerClient.spawnAnInstance(Partition, String, String,
>> >> String) line: 137
>> >> RuleTasksDelegator.delegateSpawn(PartitionContext, String, String)
>> >> line: 92
>> >> //
>> >> // drools methods hidden
>> >> //
>> >> AutoscalerRuleEvaluator.evaluateMinCheck(StatefulKnowledgeSession,
>> >> FactHandle, Object) line: 86
>> >> ClusterMonitor.monitor() line: 99
>> >> ClusterMonitor.run() line: 72
>> >> Thread.run() line: 744
>> >> ---
>> >>
>> >> Would you expect to see AutoscaleAlgorithm.getNextScaleUpPartition
>> >> executed somewhere in the stack of calls?
>> >>
>> >> Many thanks,
>> >>
>> >> Chris
>> >>
>> >> On Wed, May 28, 2014 at 11:03 PM, Nirmal Fernando
>> >> <[email protected]> wrote:
>> >> > Hi Chris,
>> >> >
>> >> > PartitionMax is considered in
>> >> > AutoscaleAlgorithm.getNextScaleUpPartition
>> >> > method.
>> >> >
>> >> >
>> >> > On Thu, May 29, 2014 at 3:09 AM, chris snow <[email protected]>
>> >> > wrote:
>> >> >>
>> >> >> I setup a breakpoint in eclipse at
>> >> >> CloudControllerClient.spawnAnInstance() and dumped the output
>> >> >> (below).
>> >> >>
>> >> >> I've scanned through the code in the stratos methods in the
>> >> >> stacktrace, but couldn't see any code that checks the partitionMax
>> >> >> value.
>> >> >>
>> >> >> Any ideas?
>> >> >>
>> >> >> Daemon Thread [Thread-56] (Suspended (breakpoint at line 137 in
>> >> >> CloudControllerClient))
>> >> >> owns: CloudControllerClient (id=2433)
>> >> >> owns: DefaultAgenda (id=2434)
>> >> >> CloudControllerClient.spawnAnInstance(Partition, String, String,
>> >> >> String) line: 137
>> >> >> RuleTasksDelegator.delegateSpawn(PartitionContext, String,
>> >> >> String)
>> >> >> line: 92
>> >> >> NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line:
>> >> >> not available [native method]
>> >> >> NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57
>> >> >> DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
>> >> >> Method.invoke(Object, Object...) line: 606
>> >> >> ReflectiveAccessorOptimizer.getMethod(Object, String, Object[],
>> >> >> Class[], ExecutableStatement[]) line: 1104
>> >> >> ReflectiveAccessorOptimizer.getMethod(Object, String) line: 987
>> >> >> ReflectiveAccessorOptimizer.compileGetChain() line: 377
>> >> >> ReflectiveAccessorOptimizer.optimizeAccessor(ParserContext,
>> >> >> char[],
>> >> >> int, int, Object, Object, VariableResolverFactory, boolean, Class)
>> >> >> line: 143
>> >> >> ASTNode.optimize(Object, Object, VariableResolverFactory) line:
>> >> >> 159
>> >> >> ASTNode.getReducedValueAccelerated(Object, Object,
>> >> >> VariableResolverFactory) line: 115
>> >> >> MVELRuntime.execute(boolean, CompiledExpression, Object,
>> >> >> VariableResolverFactory) line: 85
>> >> >> CompiledExpression.getDirectValue(Object,
>> >> >> VariableResolverFactory)
>> >> >> line: 123
>> >> >> CompiledExpression.getValue(Object, VariableResolverFactory)
>> >> >> line:
>> >> >> 119
>> >> >> MVEL.executeExpression(Object, Object, VariableResolverFactory)
>> >> >> line:
>> >> >> 930
>> >> >> MVELConsequence.evaluate(KnowledgeHelper, WorkingMemory) line:
>> >> >> 104
>> >> >> DefaultAgenda.fireActivation(Activation) line: 1287
>> >> >> DefaultAgenda.fireNextItem(AgendaFilter) line: 1221
>> >> >> DefaultAgenda.fireAllRules(AgendaFilter, int) line: 1456
>> >> >>
>> >> >>
>> >> >> ReteooStatefulSession(AbstractWorkingMemory).fireAllRules(AgendaFilter,
>> >> >> int) line: 710
>> >> >> ReteooStatefulSession(AbstractWorkingMemory).fireAllRules() line:
>> >> >> 674
>> >> >> StatefulKnowledgeSessionImpl.fireAllRules() line: 230
>> >> >>
>> >> >> AutoscalerRuleEvaluator.evaluateMinCheck(StatefulKnowledgeSession,
>> >> >> FactHandle, Object) line: 86
>> >> >> ClusterMonitor.monitor() line: 99
>> >> >> ClusterMonitor.run() line: 72
>> >> >> Thread.run() line: 744
>> >> >>
>> >> >> On Wed, May 28, 2014 at 3:37 AM, Lahiru Sandaruwan
>> >> >> <[email protected]>
>> >> >> wrote:
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Tue, May 27, 2014 at 11:54 PM, chris snow <[email protected]>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hi Lahiru,
>> >> >> >>
>> >> >> >> I'll have a poke around - it will give me a good opportunity to
>> >> >> >> learn
>> >> >> >> some more Stratos code :)
>> >> >> >>
>> >> >> >
>> >> >> > Cool :)
>> >> >> >
>> >> >> > Debug logs should be able to show us where it went wrong.
>> >> >> >
>> >> >> > Thanks.
>> >> >> >>
>> >> >> >> Many thanks,
>> >> >> >>
>> >> >> >> Chris
>> >> >> >>
>> >> >> >> On Tue, May 27, 2014 at 7:45 AM, Lahiru Sandaruwan
>> >> >> >> <[email protected]>
>> >> >> >> wrote:
>> >> >> >> > Hi Chris,
>> >> >> >> >
>> >> >> >> > Can you enable debug logs and try out?
>> >> >> >> > If so please send them over.
>> >> >> >> >
>> >> >> >> > Thanks.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Tue, May 27, 2014 at 12:05 PM, chris snow
>> >> >> >> > <[email protected]>
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> Deployment Policy partitionMax seems to be ignored. My
>> >> >> >> >> deployment
>> >> >> >> >> policy looks like this:
>> >> >> >> >>
>> >> >> >> >> {
>> >> >> >> >> "id": "economyDeployment",
>> >> >> >> >> "partitionGroup": [
>> >> >> >> >> {
>> >> >> >> >> "id": "G1",
>> >> >> >> >> "partitionAlgo": "one-after-another",
>> >> >> >> >> "partition": [
>> >> >> >> >> {
>> >> >> >> >> "id": "P1",
>> >> >> >> >> "partitionMax": "1",
>> >> >> >> >> "partitionMin": "1"
>> >> >> >> >> }
>> >> >> >> >> ]
>> >> >> >> >> }
>> >> >> >> >> ]
>> >> >> >> >> }
>> >> >> >> >>
>> >> >> >> >> Stratos initially starts two instances:
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+
>> >> >> >> >> | ID | Name |
>> >> >> >> >> Status |
>> >> >> >> >> Task
>> >> >> >> >> State | Power State | Networks |
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+
>> >> >> >> >> | c91eddd7-f2d5-49e5-b602-fde5df38e16d | php111php-9f6 |
>> >> >> >> >> ACTIVE |
>> >> >> >> >> -
>> >> >> >> >> | Running | private=10.11.12.2 |
>> >> >> >> >> | b38536d2-bc9e-4ab2-984c-bc3f2091947f | php111php-a05 |
>> >> >> >> >> ACTIVE |
>> >> >> >> >> -
>> >> >> >> >> | Running | private=10.11.12.3 |
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+
>> >> >> >> >> Then attempts to start more:
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+
>> >> >> >> >> | ID | Name |
>> >> >> >> >> Status |
>> >> >> >> >> Task
>> >> >> >> >> State | Power State | Networks |
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+
>> >> >> >> >> | 91029130-37b5-4df6-ad95-1d9f609a22c3 | php111php-13e | ERROR
>> >> >> >> >> |
>> >> >> >> >> -
>> >> >> >> >> | NOSTATE | |
>> >> >> >> >> | 71c1e361-df55-4f61-84c5-79774fef2beb | php111php-3d7 | ERROR
>> >> >> >> >> |
>> >> >> >> >> -
>> >> >> >> >> | NOSTATE | |
>> >> >> >> >> | b0d72602-cdb8-4621-ad12-911ade4d89b1 | php111php-412 | ERROR
>> >> >> >> >> |
>> >> >> >> >> -
>> >> >> >> >> | NOSTATE | |
>> >> >> >> >> | d4eac55d-bb7c-48d6-83f2-cb9a11581a18 | php111php-6f5 | ERROR
>> >> >> >> >> |
>> >> >> >> >> -
>> >> >> >> >> | NOSTATE | |
>> >> >> >> >> | 738a9534-2b95-4984-a51b-748bc7169b9a | php111php-73e | ERROR
>> >> >> >> >> |
>> >> >> >> >> -
>> >> >> >> >> | NOSTATE | |
>> >> >> >> >> | c91eddd7-f2d5-49e5-b602-fde5df38e16d | php111php-9f6 |
>> >> >> >> >> ACTIVE |
>> >> >> >> >> -
>> >> >> >> >> | Running | private=10.11.12.2 |
>> >> >> >> >> | b38536d2-bc9e-4ab2-984c-bc3f2091947f | php111php-a05 |
>> >> >> >> >> ACTIVE |
>> >> >> >> >> -
>> >> >> >> >> | Running | private=10.11.12.3 |
>> >> >> >> >> | f570e654-024b-4240-bad2-8a11f04f23f0 | php111php-a30 | ERROR
>> >> >> >> >> |
>> >> >> >> >> -
>> >> >> >> >> | NOSTATE | |
>> >> >> >> >> | 0ab36824-9bea-4a95-aa22-7f935406d83d | php111php-bca | ERROR
>> >> >> >> >> |
>> >> >> >> >> -
>> >> >> >> >> | NOSTATE | |
>> >> >> >> >> | 02ed97df-02bc-4bff-b42d-9f2bcbd4f569 | php111php-f6f | ERROR
>> >> >> >> >> |
>> >> >> >> >> -
>> >> >> >> >> | NOSTATE | |
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> +--------------------------------------+---------------+--------+------------+-------------+--------------------+
>> >> >> >> >>
>> >> >> >> >> I'm on branch: 4.0.0-incubating, using Openstack Havana +
>> >> >> >> >> Docker.
>> >> >> >> >>
>> >> >> >> >> Any ideas?
>> >> >> >> >>
>> >> >> >> >> Many thanks,
>> >> >> >> >>
>> >> >> >> >> Chris
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > --
>> >> >> >> > Lahiru Sandaruwan
>> >> >> >> > Committer and PMC member, Apache Stratos,
>> >> >> >> > Senior Software Engineer,
>> >> >> >> > WSO2 Inc., http://wso2.com
>> >> >> >> > lean.enterprise.middleware
>> >> >> >> >
>> >> >> >> > email: [email protected] cell: (+94) 773 325 954
>> >> >> >> > blog: http://lahiruwrites.blogspot.com/
>> >> >> >> > twitter: http://twitter.com/lahirus
>> >> >> >> > linked-in:
>> >> >> >> > http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>> >> >> >> >
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Check out my professional profile and connect with me on
>> >> >> >> LinkedIn.
>> >> >> >> http://lnkd.in/cw5k69
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > --
>> >> >> > Lahiru Sandaruwan
>> >> >> > Committer and PMC member, Apache Stratos,
>> >> >> > Senior Software Engineer,
>> >> >> > WSO2 Inc., http://wso2.com
>> >> >> > lean.enterprise.middleware
>> >> >> >
>> >> >> > email: [email protected] cell: (+94) 773 325 954
>> >> >> > blog: http://lahiruwrites.blogspot.com/
>> >> >> > twitter: http://twitter.com/lahirus
>> >> >> > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Check out my professional profile and connect with me on LinkedIn.
>> >> >> http://lnkd.in/cw5k69
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Best Regards,
>> >> > Nirmal
>> >> >
>> >> > Nirmal Fernando.
>> >> > PPMC Member & Committer of Apache Stratos,
>> >> >
>> >> > Senior Software Engineer, WSO2 Inc.
>> >> >
>> >> > Blog: http://nirmalfdo.blogspot.com/
>> >>
>> >>
>> >>
>> >> --
>> >> Check out my professional profile and connect with me on LinkedIn.
>> >> http://lnkd.in/cw5k69
>> >
>> >
>> >
>> >
>> > --
>> > Best Regards,
>> > Nirmal
>> >
>> > Nirmal Fernando.
>> > PPMC Member & Committer of Apache Stratos,
>> > Senior Software Engineer, WSO2 Inc.
>> >
>> > Blog: http://nirmalfdo.blogspot.com/
>>
>>
>>
>> --
>> Check out my professional profile and connect with me on LinkedIn.
>> http://lnkd.in/cw5k69
>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
--
Check out my professional profile and connect with me on LinkedIn.
http://lnkd.in/cw5k69
