Hi Manas,
I quickly went through the logs and mincheck.drl. And these are my
observations.
- Nothing wrong with mincheck.drl
- Cluster monitor is created successfully
- Cluster monitor is executing monitor() method
- monitor() executes mincheck drool, since min rule is not satisfied, it
delegates request to RuleTaskDelegator to spawn an instance.
- RuleTaskDelegator calls CC to create an instance which is failing[1]
- So mincheck rule is failing, implies that monitor is failing
So the root cause is instance creation is failing.
1 = >
TID: [0] [STRATOS] [2014-11-10 07:24:01,648] INFO
{org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
- Trying to spawn an instance via cloud controller: [cluster]
imc-dm.imc-dm.domain [partition] RegionOne-Core [lb-cluster] null
[network-partition-id] N1
{org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
TID: [0] [STRATOS] [2014-11-10 07:24:01,651] ERROR
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor:
Monitor failed.ClusterMonitor [clusterId=imc-ce.imc-ce.domain,
serviceId=imc-ce, deploymentPolicy=Deployment Policy [id]static-1-Core
[partitions]
[org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@5126e236],
autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null,
description=null], lbReferenceType=null, hasPrimary=false ]
{org.apache.stratos.autoscaler.monitor.ClusterMonitor}
Exception executing consequence for rule "Minimum Rule" in
org.apache.stratos.autoscaler.rule: [Error: $delegator.delegateSpawn($ctxt,
clusterId, lbRef, isPrimary): Cannot spawn an instance]
[Near : {... if (isPrimary){ ....}]
^
[Line: 1, Column: 1]
at
org.drools.runtime.rule.impl.DefaultConsequenceExceptionHandler.handleException(DefaultConsequenceExceptionHandler.java:39)
at
org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1297)
at org.drools.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:1221)
at org.drools.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1456)
at
org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:710)
at
org.drools.common.AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:674)
at
org.drools.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:230)
at
org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator.evaluateMinCheck(AutoscalerRuleEvaluator.java:94)
at
org.apache.stratos.autoscaler.monitor.ClusterMonitor.monitor(ClusterMonitor.java:157)
at
org.apache.stratos.autoscaler.monitor.ClusterMonitor.run(ClusterMonitor.java:86)
at java.lang.Thread.run(Thread.java:745)
Caused by: [Error: $delegator.delegateSpawn($ctxt, clusterId, lbRef,
isPrimary): Cannot spawn an instance]
[Near : {... if (isPrimary){ ....}]
^
[Line: 1, Column: 1]
at
org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.compileGetChain(ReflectiveAccessorOptimizer.java:416)
at
org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.optimizeAccessor(ReflectiveAccessorOptimizer.java:143)
at org.mvel2.ast.ASTNode.optimize(ASTNode.java:159)
at org.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:115)
at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85)
at
org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123)
at
org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119)
at
org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:113)
at org.mvel2.MVEL.executeExpression(MVEL.java:930)
at
org.drools.base.mvel.MVELConsequence.evaluate(MVELConsequence.java:104)
at
org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1287)
... 9 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.getMethod(ReflectiveAccessorOptimizer.java:1104)
at
org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.getMethod(ReflectiveAccessorOptimizer.java:987)
at
org.mvel2.optimizers.impl.refl.ReflectiveAccessorOptimizer.compileGetChain(ReflectiveAccessorOptimizer.java:377)
... 19 more
Caused by: java.lang.RuntimeException: Cannot spawn an instance
at
org.apache.stratos.autoscaler.rule.RuleTasksDelegator.delegateSpawn(RuleTasksDelegator.java:107)
... 26 more
Caused by: org.apache.stratos.autoscaler.exception.SpawningException:
javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1791]
Message: elementGetText() function expects text only elment but
START_ELEMENT was encountered.
at
org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient.spawnAnInstance(CloudControllerClient.java:174)
at
org.apache.stratos.autoscaler.rule.RuleTasksDelegator.delegateSpawn(RuleTasksDelegator.java:87)
... 26 more
Caused by: org.apache.axis2.AxisFault: javax.xml.stream.XMLStreamException:
ParseError at [row,col]:[1,1791]
Message: elementGetText() function expects text only elment but
START_ELEMENT was encountered.
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at
org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.fromOM(CloudControllerServiceStub.java:4312)
at
org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.startInstance(CloudControllerServiceStub.java:1415)
at
org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient.spawnAnInstance(CloudControllerClient.java:162)
... 27 more
Caused by: org.apache.axiom.om.impl.exception.OMStreamingException:
javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1791]
Message: elementGetText() function expects text only elment but
START_ELEMENT was encountered.
at
org.apache.axiom.om.impl.SwitchingWrapper.getElementText(SwitchingWrapper.java:966)
at
javax.xml.stream.util.StreamReaderDelegate.getElementText(StreamReaderDelegate.java:100)
at
org.apache.stratos.cloud.controller.stub.pojo.MemberContext$Factory.parse(MemberContext.java:1470)
at
org.apache.stratos.cloud.controller.stub.impl.StartInstanceResponse$Factory.parse(StartInstanceResponse.java:417)
at
org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.fromOM(CloudControllerServiceStub.java:4159)
... 29 more
Caused by: javax.xml.stream.XMLStreamException: ParseError at
[row,col]:[1,1791]
Message: elementGetText() function expects text only elment but
START_ELEMENT was encountered.
at
com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.getElementText(XMLStreamReaderImpl.java:856)
at
org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.getElementText(XMLStreamReaderWrapper.java:100)
at
org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.getElementText(XMLStreamReaderWrapper.java:100)
at
org.apache.axiom.om.impl.SwitchingWrapper.getElementText(SwitchingWrapper.java:962)
We need to look at why instance creation is failing. Do you have any idea
why instance creation is failing?
I am looking into it further.
Thanks.
On Mon, Nov 10, 2014 at 3:31 PM, Manas Padegaonkar -T (mpadegao - ARICENT
TECHNOLOGIES HOLDINGS LIMITED at Cisco) <[email protected]> wrote:
> Hi Raj,
>
>
>
> Mincheck.drl is attached herewith.
>
>
>
> Regards
>
> Manas
>
>
>
> *From:* Rajkumar Rajaratnam [mailto:[email protected]]
> *Sent:* Monday, November 10, 2014 3:18 PM
>
> *To:* [email protected]
> *Subject:* Re: Getting Cluster Monitor Failure Messages in wso2carbon.log
>
>
>
> Great Manas.
>
> I will go through the log and get back to you. It would be great if you
> can add the mincheck.drl file too.
>
> Thanks.
>
>
>
> On Mon, Nov 10, 2014 at 3:10 PM, Manas Padegaonkar -T (mpadegao - ARICENT
> TECHNOLOGIES HOLDINGS LIMITED at Cisco) <[email protected]> wrote:
>
> Hi Raj,
>
>
>
> Attached is the wso2carbon.log as requested by you.
>
>
>
> Regards
>
> Manas
>
>
>
> *From:* Rajkumar Rajaratnam [mailto:[email protected]]
> *Sent:* Monday, October 27, 2014 5:52 PM
>
>
> *To:* [email protected]
> *Subject:* Re: Getting Cluster Monitor Failure Messages in wso2carbon.log
>
>
>
> Will you be able to share the complete wso2carbon.log file? (at least from
> the autoscale policy deployment upto this error message)
>
>
>
> Thanks.
>
>
>
> On Mon, Oct 27, 2014 at 5:41 PM, Manas Padegaonkar -T (mpadegao - ARICENT
> TECHNOLOGIES HOLDINGS LIMITED at Cisco) <[email protected]> wrote:
>
> Hi Raj,
>
>
>
> I am using the autoscale policy name correctly during subscription &
> cartridge gets spawned, fetches catalog from puppet-master & becoming
> Active.
>
> After this I am getting Cluster monitor: Monitor Failed.
>
>
>
> If I will use the wrong autoscale-policy name for cartridge subscription,
> it will fail there only.
>
>
>
> Regards
>
> Manas
>
>
>
> *From:* Rajkumar Rajaratnam [mailto:[email protected]]
> *Sent:* Monday, October 27, 2014 5:20 PM
>
>
> *To:* [email protected]
> *Subject:* Re: Getting Cluster Monitor Failure Messages in wso2carbon.log
>
>
>
> oops. so we don't need subscription json to subscribe. I confused with
> kubernetes stuff.
>
>
>
> AFAIK, this error happens when the relevant autoscale policy is not found
> in Policy Manager. When we pass autoscale policy name, AS is loading the
> autoscale policy from Policy Manager. If the policy with the name you
> provided is not there, it will give this error. Please make sure that you
> have given the correct autoscale policy name.
>
>
>
> Thanks.
>
>
>
> On Mon, Oct 27, 2014 at 5:00 PM, Rajkumar Rajaratnam <[email protected]>
> wrote:
>
> Hi Manas,
>
>
>
> If possible, can you share your subscription json and autoscale policy
> json?
>
>
>
> Thanks.
>
>
>
> On Mon, Oct 27, 2014 at 4:56 PM, Manas Padegaonkar -T (mpadegao - ARICENT
> TECHNOLOGIES HOLDINGS LIMITED at Cisco) <[email protected]> wrote:
>
> Hi Raj,
>
>
>
> I am not using Stratos-GUI & working from CLI only.
>
>
>
> Regards
>
> Manas
>
>
>
> *From:* Rajkumar Rajaratnam [mailto:[email protected]]
> *Sent:* Monday, October 27, 2014 4:53 PM
> *To:* [email protected]
> *Subject:* Re: Getting Cluster Monitor Failure Messages in wso2carbon.log
>
>
>
> I assumed you were using CLI or Curl commands. If you are using management
> console, did you select the autoscale policy from autoscale policies drop
> down list? And could you please tell me whether you are using master branch
> or some other branch?
>
>
>
> Thanks.
>
>
>
> On Mon, Oct 27, 2014 at 4:42 PM, Rajkumar Rajaratnam <[email protected]>
> wrote:
>
> Hi Manas,
>
>
>
> Reason is your autoscale policy is null. You need to give a deployed
> autoscale policy name along with subscription json.
>
>
>
> Thanks.
>
>
>
> On Mon, Oct 27, 2014 at 4:36 PM, Manas Padegaonkar -T (mpadegao - ARICENT
> TECHNOLOGIES HOLDINGS LIMITED at Cisco) <[email protected]> wrote:
>
> Hi,
>
>
>
> I am getting following messages periodically in wso2carbon.log for all of
> the subscribed cartridges which came Active.
>
> Want to know what is triggering this error to come & what is its impact if
> any.
>
>
>
>
>
>
>
> TID: [0] [STRATOS] [2014-10-13 13:04:59,436] ERROR
> {org.apache.stratos.autoscaler.monitor.ClusterMonitor} - Cluster monitor:
> Monitor failed.ClusterMonitor [clusterId=1-14.1-14.domain, serviceId=1-14,
> deploymentPolicy=Deployment Policy [id]static-1-Core [partitions]
> [org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@118c06f9],
> autoscalePolicy=null, lbReferenceType=null, hasPrimary=false ]
> {org.apache.stratos.autoscaler.monitor.ClusterMonitor}
>
> [Error: null pointer: autoscalePolicy.getLoadThresholds()]
>
> [Near : {... autoscalePolicy.getLoadThresho ....}]
>
>
>
>
>
> Regards
>
> Manas
>
>
>
>
>
>
>
> --
>
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
>
> Mobile +94777568639 | +94783498120
>
>
>
>
>
> --
>
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
>
> Mobile +94777568639 | +94783498120
>
>
>
>
>
> --
>
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
>
> Mobile +94777568639 | +94783498120
>
>
>
>
>
> --
>
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
>
> Mobile +94777568639 | +94783498120
>
>
>
>
>
> --
>
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
>
> Mobile +94777568639 | +94783498120
>
>
>
>
> --
>
> Raj
>
--
Raj
