One more observation:
When removing the application I see the exception below ( although all members
are terminated and removed),
TID: [0] [STRATOS] [2015-03-31 19:32:57,142] INFO
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
- Publishing Cluster terminated event: [application-id] subscription-G1-G2-G3
[cluster id] subscription-G1-G2-G3.c1-1x0.c1.domain [instance-id]
subscription-G1-G2-G3-1
TID: [0] [STRATOS] [2015-03-31 19:32:57,155] INFO
{org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
- [ClusterTerminatedEvent] Received: class
org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatedEvent
TID: [0] [STRATOS] [2015-03-31 19:32:57,155] INFO
{org.apache.stratos.autoscaler.status.processor.group.GroupStatusTerminatedProcessor}
- sending group instance terminated for [group] multiple-groups-no-startup-G1
[instance] subscription-G1-G2-G3-1
TID: [0] [STRATOS] [2015-03-31 19:32:57,156] ERROR
{org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator}
- Failed to retrieve topology event message
java.lang.NullPointerException
at
org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver$8.onEvent(AutoscalerTopologyEventReceiver.java:350)
at
org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
at java.util.Observable.notifyObservers(Observable.java:159)
at
org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
at
org.apache.stratos.messaging.message.processor.topology.ClusterInstanceTerminatedProcessor.doProcess(ClusterInstanceTerminatedProcessor.java:132)
at
org.apache.stratos.messaging.message.processor.topology.ClusterInstanceTerminatedProcessor.process(ClusterInstanceTerminatedProcessor.java:64)
at
org.apache.stratos.messaging.message.processor.topology.ClusterRemovedMessageProcessor.process(ClusterRemovedMessageProcessor.java:65)
at
org.apache.stratos.messaging.message.processor.topology.ClusterInstanceInactivateProcessor.process(ClusterInstanceInactivateProcessor.java:73)
at
org.apache.stratos.messaging.message.processor.topology.ClusterInstanceActivatedProcessor.process(ClusterInstanceActivatedProcessor.java:73)
at
org.apache.stratos.messaging.message.processor.topology.ClusterCreatedMessageProcessor.process(ClusterCreatedMessageProcessor.java:65)
at
org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor.process(ApplicationClustersRemovedMessageProcessor.java:63)
at
org.apache.stratos.messaging.message.processor.topology.ApplicationClustersCreatedMessageProcessor.process(ApplicationClustersCreatedMessageProcessor.java:65)
at
org.apache.stratos.messaging.message.processor.topology.ServiceRemovedMessageProcessor.process(ServiceRemovedMessageProcessor.java:64)
at
org.apache.stratos.messaging.message.processor.topology.ServiceCreatedMessageProcessor.process(ServiceCreatedMessageProcessor.java:65)
at
org.apache.stratos.messaging.message.processor.topology.CompleteTopologyMessageProcessor.process(CompleteTopologyMessageProcessor.java:74)
at
org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
at
org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator.run(TopologyEventMessageDelegator.java:73)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
From: Martin Eppel (meppel)
Sent: Tuesday, March 31, 2015 12:04 PM
To: [email protected]
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with single
group and 3 cartridges, no dependencies ...
Hi Lahiru,
I tested the fix and noticed the following issue:
From the 4 cartridges 3 instances go Active, however one instance got stuck in
INITIALIZED state.
One of the things I noticed while the member was stuck in INITIALIZED state is
that the system received 2 active events for the same membered:
TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
- Publishing member activated event: [service-name] c2 [cluster-id]
subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
subscription-G1-G2-G3-1 [member-id]
subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
[network-partition-id] RegionOne [partition-id] whole-region
TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
- Publishing member activated event: [service-name] c2 [cluster-id]
subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id]
subscription-G1-G2-G3-1 [member-id]
subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f
[network-partition-id] RegionOne [partition-id] whole-region
As another note, but I can’t be sure since the log got overwritten is that it
appeared to me that at one point (while the cartridge was stuck in INITIALIZED
state is that the cartridge alias of 2 cartridges was the same (~71, ~70: 71:
CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when I double checked a
new instance of the cartridge was created (which went ACTIVE).
Eventually the system terminated the “faulty” member and the cartridge came up
and went into active state:
“Pending state of member expired, member will be moved to obsolete list.
[pending member]
subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9”
…
“Member activated: [service] c2 [cluster]
subscription-G1-G2-G3.c2-1x0.c2.domain [member]
subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2”
Cartridge alias after all 4 cartridges went ACTIVE:
70: CARTRIDGE_ALIAS=c2-1x0
71: CARTRIDGE_ALIAS=c2-1x1
69: CARTRIDGE_ALIAS=c1-1x0
68: CARTRIDGE_ALIAS=c1-1x1
I attached the logs and artefacts
Thanks
Martin
From: Lahiru Sandaruwan [mailto:[email protected]]
Sent: Tuesday, March 31, 2015 2:24 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with single
group and 3 cartridges, no dependencies ...
Hi Martin,
I have reproduced the issue and pushed a fix. Please verify.
Thanks.
On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan
<[email protected]<mailto:[email protected]>> wrote:
Hi Martin,
On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel)
<[email protected]<mailto:[email protected]>> wrote:
Hi Lahiru,
I tested a new scenario, where I have 2 groups with cartridges and no start up
dependencies and I see again the issue with cartridge instances being stuck in
“started” state. I also noticed the following log message:
“Artifact repositories not found for application signup, artifact updated event
not sent: [application-id] subscription-G1-G2-G3”.
Will have a look.
Thanks.
Please find attached the artefacts and log messages,
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 2:36 PM
To: [email protected]<mailto:[email protected]>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with single
group and 3 cartridges, no dependencies ...
Hi Lahiru,
I tested both scenarios,
1. Single cartridge in single group and it worked, cartridge instance
starts up and goes active,
2. Multiple cartridges with startup dependencies :
All instances are started up (in right sequence) and go active
JIRA
https://issues.apache.org/jira/browse/STRATOS-1292[http://wikicentral.cisco.com/images/icons/linkext7.gif]
is being tested and verified
Removing / Un-deploying the application caused an issue as one of the VMs did
not get terminated (throwing exceptions in the log). I am not sure if this is
already a known issue related to to the ongoing discussion of application
removal / un-deployment.
Just in case, I zipped up the logs and attached it to the email thread
(single-group-multiple-cartridges-startup.zip). Let me know if the
un-deployment issue is the same already discussed or if this is a new issue ?
Same error occurred in the scenario where multiple cartridges were started up
(all go active) and then application was removed, I attached the log for this
scenario as well
(wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip)
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 30, 2015 7:47 AM
To: [email protected]<mailto:[email protected]>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with single
group and 3 cartridges, no dependencies ...
Thanks Lahiru,
Will do and let you know
Martin
From: Lahiru Sandaruwan [mailto:[email protected]]
Sent: Monday, March 30, 2015 3:32 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with single
group and 3 cartridges, no dependencies ...
Hi Martin,
I was able to reproduce the issue and committed a fix.
Could you please verify?
Thanks.
On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan
<[email protected]<mailto:[email protected]>> wrote:
Hi Martin,
According to the logs attached, it seems the cartridge agent waits for the
artifact updated event, and somehow it is not sent.
Looking into it.
Thanks.
On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne
<[email protected]<mailto:[email protected]>> wrote:
Hi Martin,
Thanks for the quick response on this!
I will use these artifacts to re-produce the problem locally and will let you
know the results soon.
Thanks
On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel)
<[email protected]<mailto:[email protected]>> wrote:
Hi Imesh,
As discussed please find attached the “real” artefacts (application, cartridge
group, cartridge):
Attached it as well to the jira
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 11:37 AM
To: [email protected]<mailto:[email protected]>
Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with single
group and 3 cartridges, no dependencies ...
Hi Imesh,
Originally I tested with 3 cartridges but then realized that it also occurs
with a single cartridge in a group – see also the JIRA for the respective notes
and artefacts (single group, single cartridge)
Thanks
Martin
From: Imesh Gunaratne [mailto:[email protected]]
Sent: Friday, March 27, 2015 11:34 AM
To: dev
Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going
active ... was RE: Stratos 4.1.0 : Issue when deploying application with single
group and 3 cartridges, no dependencies ...
Hi Martin,
Thanks for reporting this problem. You have mentioned that there are three
cartridges in the application but in the given application.json I only can see
one cartridge. Are we missing anything here? Will it be possible for you to
attach the complete set of Stratos artifacts that you are using in a zip file?
Thanks
Imesh
On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel)
<[email protected]<mailto:[email protected]>> wrote:
Forgot to mention, this is the same issue as reported in JIRA
https://issues.apache.org/jira/browse/STRATOS-1292
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Friday, March 27, 2015 10:52 AM
To: [email protected]<mailto:[email protected]>
Subject: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active
... was RE: Stratos 4.1.0 : Issue when deploying application with single group
and 3 cartridges, no dependencies ...
I retested with a similar scenario (single cartridges in group) with debug
logs turned on, please see logs attached.
Cartridge is stuck in “Starting” state
Thanks
Martin
From: Martin Eppel (meppel)
Sent: Monday, March 23, 2015 7:15 PM
To: [email protected]<mailto:[email protected]>
Subject: Stratos 4.1.0 : Issue when deploying application with single group and
3 cartridges, no dependencies ...
Hi,
I see an issue when I subscribe a single group with 3 cartridges, no
dependencies – they never go active instead they all are stuck in “Starting”.
I attached the server log and the cartridge-agent logs
Member status:
subscription-c1-c2-c3: applicationInstances 1, groupInstances 1,
clusterInstances 3, members 3 (Starting 3)
c2:
172.16.2.18/172.28.40.69<http://172.16.2.18/172.28.40.69>: status Starting
c3:
172.16.2.19/172.28.40.70<http://172.16.2.19/172.28.40.70>: status Starting
c1:
172.16.2.17/172.28.40.68<http://172.16.2.17/172.28.40.68>: status Starting
Let me know if I should go ahead and open a JIRA
Thanks
Martin
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: [email protected]<mailto:[email protected]> blog:
http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: [email protected]<mailto:[email protected]> blog:
http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954<tel:%2B94773325954>
email: [email protected]<mailto:[email protected]> blog:
http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
phone: +94773325954
email: [email protected]<mailto:[email protected]> blog:
http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
