Great, Thanks
Martin From: [email protected] [mailto:[email protected]] On Behalf Of Isuru Haththotuwa Sent: Wednesday, April 01, 2015 9:08 PM To: dev Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Lahiru, On Thu, Apr 2, 2015 at 9:11 AM, Lahiru Sandaruwan <[email protected]<mailto:[email protected]>> wrote: Thanks Martin for the logs. So the reason is clear why happens this. We keep the payload par cartridge type in iaas provider object, in cloud controller context. When second member approaches the instance started method, it rewrites the payload in iaasProvider object. But the first instance is still not really created in iaas, and it runs in a different thread. Therefore the payload belong to second instance, is passed to second instance as well as the first instance. Nice work finding this! So the resolution would be to change the cartridge type -> payload mapping to alias -> payload? I will fix it. Thanks. On Wed, Apr 1, 2015 at 11:38 PM, Martin Eppel (meppel) <[email protected]<mailto:[email protected]>> wrote: Hi Lahiru, Attached are the logs, Thanks Martin From: Lahiru Sandaruwan [mailto:[email protected]<mailto:[email protected]>] Sent: Wednesday, April 01, 2015 4:11 AM To: dev Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... HI Martin, Could you please send the debug logs enabling debug on following classes? I have improved few debug logs to track payload, therefore please take a pull before reproducing with debug logs. log4j.logger.org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser=DEBUG log4j.logger.org.apache.stratos.cloud.controller.services.impl.InstanceCreator=DEBUG log4j.logger.org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl=DEBUG Thanks. On Wed, Apr 1, 2015 at 10:50 AM, Lahiru Sandaruwan <[email protected]<mailto:[email protected]>> wrote: I will have a look Martin. Thanks. On Wed, Apr 1, 2015 at 6:54 AM, Martin Eppel (meppel) <[email protected]<mailto:[email protected]>> wrote: Hi Lahiru, I retested the scenario to make sure proper unique aliases are provided and they are, please double check the provided artifacts. Issue remains the same, one VM gets stuck in INITIALIZED until terminated and restarted. As pointed out before, the cartridge alias in the cartridge agent is the same in 2 cartridges (although provided aliases in the application json is unique). After restarting the faulty member cartridge agent alias has the proper alias and goes into active. I also provided the cartridge-agent logs for the member which is faulty, before restart and after restart (cartridge-agent-70~INITIALIZED, ACTIVE) Please see attached logs, artifacts, Thanks Martin From: Martin Eppel (meppel) Sent: Tuesday, March 31, 2015 1:22 PM To: [email protected]<mailto:[email protected]> Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Lahiru, The aliases are all unique, please check the the application json, and as I said in my analysis it I think this is what I saw but I can’t be sure. So this is not an issue of starting the application with duplicate aliases but rather something else , Thanks Martin From: Lahiru Sandaruwan [mailto:[email protected]] Sent: Tuesday, March 31, 2015 12:58 PM To: dev Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Martin, On Wed, Apr 1, 2015 at 12:34 AM, Martin Eppel (meppel) <[email protected]<mailto:[email protected]>> wrote: Hi Lahiru, I tested the fix and noticed the following issue: From the 4 cartridges 3 instances go Active, however one instance got stuck in INITIALIZED state. One of the things I noticed while the member was stuck in INITIALIZED state is that the system received 2 active events for the same membered: TID: [0] [STRATOS] [2015-03-31 18:10:27,645] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region TID: [0] [STRATOS] [2015-03-31 18:10:31,059] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing member activated event: [service-name] c2 [cluster-id] subscription-G1-G2-G3.c2-1x1.c2.domain [cluster-instance-id] subscription-G1-G2-G3-1 [member-id] subscription-G1-G2-G3.c2-1x1.c2.domainf939e8dc-e0cc-408a-8343-ac86e479549f [network-partition-id] RegionOne [partition-id] whole-region As another note, but I can’t be sure since the log got overwritten is that it appeared to me that at one point (while the cartridge was stuck in INITIALIZED state is that the cartridge alias of 2 cartridges was the same (~71, ~70: 71: CARTRIDGE_ALIAS=c2-1x1), but I can’t say for sure since when I double checked a new instance of the cartridge was created (which went ACTIVE). Can i assume this occurred when two cartridges use same alias? May be we have to put a validation on Application as we cannot use same alias in two cartridges. If so, could you please try to reproduce this with different aliases? Thanks. Eventually the system terminated the “faulty” member and the cartridge came up and went into active state: “Pending state of member expired, member will be moved to obsolete list. [pending member] subscription-G1-G2-G3.c2-1x0.c2.domainfe9a30cf-fffc-4a6c-9b3b-f419498895b9” … “Member activated: [service] c2 [cluster] subscription-G1-G2-G3.c2-1x0.c2.domain [member] subscription-G1-G2-G3.c2-1x0.c2.domain44a2ae3c-9968-41de-9e10-7212ef3788c2” Cartridge alias after all 4 cartridges went ACTIVE: 70: CARTRIDGE_ALIAS=c2-1x0 71: CARTRIDGE_ALIAS=c2-1x1 69: CARTRIDGE_ALIAS=c1-1x0 68: CARTRIDGE_ALIAS=c1-1x1 I attached the logs and artefacts Thanks Martin From: Lahiru Sandaruwan [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, March 31, 2015 2:24 AM To: dev Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Martin, I have reproduced the issue and pushed a fix. Please verify. Thanks. On Tue, Mar 31, 2015 at 8:32 AM, Lahiru Sandaruwan <[email protected]<mailto:[email protected]>> wrote: Hi Martin, On Tue, Mar 31, 2015 at 5:41 AM, Martin Eppel (meppel) <[email protected]<mailto:[email protected]>> wrote: Hi Lahiru, I tested a new scenario, where I have 2 groups with cartridges and no start up dependencies and I see again the issue with cartridge instances being stuck in “started” state. I also noticed the following log message: “Artifact repositories not found for application signup, artifact updated event not sent: [application-id] subscription-G1-G2-G3”. Will have a look. Thanks. Please find attached the artefacts and log messages, Thanks Martin From: Martin Eppel (meppel) Sent: Monday, March 30, 2015 2:36 PM To: [email protected]<mailto:[email protected]> Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Lahiru, I tested both scenarios, 1. Single cartridge in single group and it worked, cartridge instance starts up and goes active, 2. Multiple cartridges with startup dependencies : All instances are started up (in right sequence) and go active JIRA https://issues.apache.org/jira/browse/STRATOS-1292[http://wikicentral.cisco.com/images/icons/linkext7.gif] is being tested and verified Removing / Un-deploying the application caused an issue as one of the VMs did not get terminated (throwing exceptions in the log). I am not sure if this is already a known issue related to to the ongoing discussion of application removal / un-deployment. Just in case, I zipped up the logs and attached it to the email thread (single-group-multiple-cartridges-startup.zip). Let me know if the un-deployment issue is the same already discussed or if this is a new issue ? Same error occurred in the scenario where multiple cartridges were started up (all go active) and then application was removed, I attached the log for this scenario as well (wscarbon-application-single-group-multiple-cartridges-no-startup-remval-error.log.zip) Thanks Martin From: Martin Eppel (meppel) Sent: Monday, March 30, 2015 7:47 AM To: [email protected]<mailto:[email protected]> Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Thanks Lahiru, Will do and let you know Martin From: Lahiru Sandaruwan [mailto:[email protected]] Sent: Monday, March 30, 2015 3:32 AM To: dev Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Martin, I was able to reproduce the issue and committed a fix. Could you please verify? Thanks. On Sun, Mar 29, 2015 at 5:43 PM, Lahiru Sandaruwan <[email protected]<mailto:[email protected]>> wrote: Hi Martin, According to the logs attached, it seems the cartridge agent waits for the artifact updated event, and somehow it is not sent. Looking into it. Thanks. On Sat, Mar 28, 2015 at 6:48 AM, Imesh Gunaratne <[email protected]<mailto:[email protected]>> wrote: Hi Martin, Thanks for the quick response on this! I will use these artifacts to re-produce the problem locally and will let you know the results soon. Thanks On Sat, Mar 28, 2015 at 4:19 AM, Martin Eppel (meppel) <[email protected]<mailto:[email protected]>> wrote: Hi Imesh, As discussed please find attached the “real” artefacts (application, cartridge group, cartridge): Attached it as well to the jira Thanks Martin From: Martin Eppel (meppel) Sent: Friday, March 27, 2015 11:37 AM To: [email protected]<mailto:[email protected]> Subject: RE: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Imesh, Originally I tested with 3 cartridges but then realized that it also occurs with a single cartridge in a group – see also the JIRA for the respective notes and artefacts (single group, single cartridge) Thanks Martin From: Imesh Gunaratne [mailto:[email protected]] Sent: Friday, March 27, 2015 11:34 AM To: dev Subject: Re: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi Martin, Thanks for reporting this problem. You have mentioned that there are three cartridges in the application but in the given application.json I only can see one cartridge. Are we missing anything here? Will it be possible for you to attach the complete set of Stratos artifacts that you are using in a zip file? Thanks Imesh On Fri, Mar 27, 2015 at 11:51 PM, Martin Eppel (meppel) <[email protected]<mailto:[email protected]>> wrote: Forgot to mention, this is the same issue as reported in JIRA https://issues.apache.org/jira/browse/STRATOS-1292 Thanks Martin From: Martin Eppel (meppel) Sent: Friday, March 27, 2015 10:52 AM To: [email protected]<mailto:[email protected]> Subject: Issue (?): Stratos 4.1.0 - Cartridges in a group are not going active ... was RE: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... I retested with a similar scenario (single cartridges in group) with debug logs turned on, please see logs attached. Cartridge is stuck in “Starting” state Thanks Martin From: Martin Eppel (meppel) Sent: Monday, March 23, 2015 7:15 PM To: [email protected]<mailto:[email protected]> Subject: Stratos 4.1.0 : Issue when deploying application with single group and 3 cartridges, no dependencies ... Hi, I see an issue when I subscribe a single group with 3 cartridges, no dependencies – they never go active instead they all are stuck in “Starting”. I attached the server log and the cartridge-agent logs Member status: subscription-c1-c2-c3: applicationInstances 1, groupInstances 1, clusterInstances 3, members 3 (Starting 3) c2: 172.16.2.18/172.28.40.69<http://172.16.2.18/172.28.40.69>: status Starting c3: 172.16.2.19/172.28.40.70<http://172.16.2.19/172.28.40.70>: status Starting c1: 172.16.2.17/172.28.40.68<http://172.16.2.17/172.28.40.68>: status Starting Let me know if I should go ahead and open a JIRA Thanks Martin -- Imesh Gunaratne Technical Lead, WSO2 Committer & PMC Member, Apache Stratos -- Imesh Gunaratne Technical Lead, WSO2 Committer & PMC Member, Apache Stratos -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: [email protected]<mailto:[email protected]> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: [email protected]<mailto:[email protected]> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: [email protected]<mailto:[email protected]> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: [email protected]<mailto:[email protected]> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: [email protected]<mailto:[email protected]> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: [email protected]<mailto:[email protected]> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: [email protected]<mailto:[email protected]> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- -- Lahiru Sandaruwan Committer and PMC member, Apache Stratos, Senior Software Engineer, WSO2 Inc., http://wso2.com lean.enterprise.middleware phone: +94773325954<tel:%2B94773325954> email: [email protected]<mailto:[email protected]> blog: http://lahiruwrites.blogspot.com/ linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 -- <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> Thanks and Regards, Isuru H. <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> +94 716 358 048<http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
