Re: cs 4.5.1, vm stopped when putting xen hosts in maintenance

[Rafael Weingärtner]

Ok, I dug a little further into the code.

If we put a host in maintenance, it may unfortunately happen that a timeout
> occurs and CS aborts the migration. As far as we can interpret the logs, it
> seems (we speculate) that CS detects that the VM is then still running on
> the original host, which is in maintenance, and decides somehow to stop it
> because of the "host not up" status.
>

>From wat I have seen in the code, this may happen (as you speculated). If a
timeout or some other event interrupts migrations, the VM will be stopped.
This happens in the method
“com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateAway(String,
long, DeploymentPlanner)” at line 2396 and further. This was implemented on
purpose; if the migration does not happen; the VM is stopped on the source
host. Probably it was implemented this way on that philosophy of gradient
descendent of artificial intelligence:

“pick a random point in the field, update the values according to the slope
> and hopefully after successfully iterations you converge to a minimum
> point.” (the problem is that sometimes the function diverges).
>

I mean, probably this was done this way because the HA was supposed to see
that a VM with HA is stopped, and should try to start it again somewhere
else. The log you posted do not provide any other details to debug this
further because the log is after an MS restart, right?


After restarting the MSs I guess that a similar issue may happen: the hosts
> are disconnected during the early stage and then reconnects. Maybe CS then
> again stops the VMs because the "host not up" status?
> Cf. 2017-01-30 08:44:06,157 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-6:ctx-0d738a8e job-13737/job-192988 ctx-6e9c753f) The
> last host of this VM is not UP or is not enabled, host status is:
> Disconnected, host resource state is: Enabled
>

Here is something interesting that you have found. I checked the code, but
without a live debug it is pretty hard for me to get the right flow of
execution. By looking at the code and at the logs you provided, your
deduction is right. After ACS boots up, the check for hosts that are
shutdown/deactivated starts, and the method
“com.cloud.agent.manager.AgentManagerImpl.executeUserRequest(long, Event)”
is executed. The problem is that all of the hosts are considered as
disconnected (ACS may not have finished connecting to all of them yet), and
then a whole process of HA is kicked. The problem is that there is no host
with status “UP”. That is why this happen “No suitable hosts found under
this Cluster: 25”, and then the whole flow of HA, which shuts down the VM
is kicked.

In theory, the HA worker should see that a VM with HA is down, and should
try to start it. I am not understanding why this is not working. If you
decide to debug the running ACS, I may be able to help you, just keep me
posted.


On Tue, Jan 31, 2017 at 10:26 AM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> I will be able to re-check some code and answer you with new thoughts on
> this problem tonight (the log files you provided today have some very
> insightful information regarding this problem). I believe this is a bug in
> HA, a very tricky one that has not been reported yet (at least I do not
> remember seeing anything similar). And thanks to randomness/destiny now you
> have found/uncovered a way to reproduce the problem (bug?!).
>
> I do not suggest you changing any data at all directly in the data base
> (unless you know ACS code well and are 100% of what you are doing). This
> can cause even bigger and more complicated problems. If you remove the HA
> feature from this VMs, this problem will stop. However, if you can/are
> willing to maintain the HA until we fully uncover the problem, map and
> report it, this would be great.
> BTW: why are you restarting MSs?
> What are the configurations of your MSs?
>
> Thanks for your time and for sharing information with our community.
>
>
>
> On Tue, Jan 31, 2017 at 8:53 AM, Francois Scheurer <
> francois.scheu...@everyware.ch> wrote:
>
>> Dear Rafael
>>
>>
>> Many thanks for your deep analysis.
>> Nice humor with the planetsalignments BTW ;)
>>
>> We also suspected the HA Flag to be related this issue, as we never got
>> this issue in the past, until we enabled the HA on all VMs and Offerings.
>>
>> Here is also the workers list:
>>
>> mysql> select * from op_ha_work;
>> +--+-+---++-
>> -++-+-+-
>> --+-+---+-+-+
>> | id   | instance_id | type  | vm_type| state|
>> mgmt_server_id | host_id | created | tried | taken
>>  | step  | time_to_try | updated |
>> +--+-+---++-
>> -++-+-+-
>> --+-+---+-+-+
>> |  105 | 343 | 

Re: CCC Miami Update

[Wido den Hollander]

> Op 29 januari 2017 om 19:41 schreef Will Stevens :
> 
> 
> Hello Everyone,
> I am sure most of you are aware at this point that a CloudStack
> Collaboration Conference (CCC) is being hosted in Miami on May 16-18 by
> ApacheCon.
> 

Will the CloudStack conf be a full 3 days or maybe 1 or 2 days of the three 
days of Apache Con?

The reason I'm asking is that I think we will not be able to fill up three 
complete days.

Might want to go for one day of Hackathon and then 1 full day of talks? Maybe 
have the 17th a full day of talks and on the 18th only the morning?

I'd personally rather see 2 packed days then having 3 slow days.

Wido

> I urge you to consider joining us at this event.  Here are some of the
> important event details, but all the details can be found on our event
> website: *http://us.cloudstackcollab.org/ *
> 
> *Deadline for talk submissions:* *February 11th, 2017*
> *Accepted talk notifications:* *March 6th, 2017*
> *Schedule published on: **March 9th, 2017*
> 
> The event is being run as a collection of independently themed
> conferences.  Obviously, most of you will be specifically interested in the
> CloudStack Collaboration Conference, but your registration also gives you
> access to the other conference being run at the same time.  So far the
> other conferences include; 'Apache: Big Data', 'Apache: IoT', 'Flex Project
> Summit', 'Apache Traffic Server / Apache Traffic Control' and 'TomcatCon'.
> 
> The earlier you register, the more you save, so get your registration in
> early.
> 
> *Early Registration:* *until March 12, 2017*
> *Standard Registration:* *March 13, 2017 - April 16, 2017*
> *Late Registration:* *April 17, 2017 - Event Date*
> *Committer Registration:* Special pricing is available for active Apache
> Committers. Please contact the event organizers 
> for details.
> *Speaker Registration:* *One free registration is included with each
> accepted talk.*
> 
> If you submit a talk  and
> your talk is accepted, you will get one free registration per talk accepted.
> 
> We are still looking for event sponsors, so if you are interested in
> sponsoring the event, please review the sponsorship details
> .
> 
> If you have questions about anything, feel free to contact me directly and
> I will make sure you are connected with the right people.
> 
> Looking forward to seeing you all in Miami.
> 
> Cheers,
> 
> Will

Re: XenServer & Open vSwitch

[Dag Sonstebo]

Hi Eric,

My understanding as follows:

- XenServer with basic networking only works when using the “old” linux 
switching backend, not OVS, due to the requirements to manage security groups.
- OVS plugin comes into play in advanced zones where you want to use SDN/L3 
overlay networks to provide guest isolation (which you don’t do in basic zones).
- Keep in mind this only applies when you want to use SDN, if you want to use 
standard VLAN based advanced zones then XenServer will just use OVS as it’s 
standard switching backend (since XS6.2).

Hope this helps.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 31/01/2017, 14:42, "GoGoZoom PDX"  wrote:

I'm confused by the documentation that I've found:

This page states that "If the XenServer host is part of a zone that uses
basic networking, disable Open vSwitch (OVS)":

http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/hypervisor/xenserver.html#physical-networking-setup-for-xenserver

This page states that "The following hypervisors are supported by the OVS
Plugin: XenServer >= 4.0":
http://docs.cloudstack.apache.org/en/latest/networking/ovs-plugin.html

Will the OVS Plugin allow CS to manage XenServer when using the Open
vSwitch network backend (in a Basic Networking scenario and/or an Advanced
Networking scenario)?

TIA,
Eric Pretorious
Portland, Oregon



dag.sonst...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

XenServer & Open vSwitch

[GoGoZoom PDX]

I'm confused by the documentation that I've found:

This page states that "If the XenServer host is part of a zone that uses
basic networking, disable Open vSwitch (OVS)":
http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/hypervisor/xenserver.html#physical-networking-setup-for-xenserver

This page states that "The following hypervisors are supported by the OVS
Plugin: XenServer >= 4.0":
http://docs.cloudstack.apache.org/en/latest/networking/ovs-plugin.html

Will the OVS Plugin allow CS to manage XenServer when using the Open
vSwitch network backend (in a Basic Networking scenario and/or an Advanced
Networking scenario)?

TIA,
Eric Pretorious
Portland, Oregon

AW: Login hangs/freezes after update to 4.9.2.0

[Martin Emrich]

Hello Rohit,

I upgraded an existing 4.8.0 installation (reusing all local configuration 
files and the database). I never used dynamic roles, but I'll take a look at 
them if the need arises.

But I got good news anyways, By upgrading to 4.9.0 in between (and letting it 
run for a day to complete all migration and upgrade tasks) and only then 
upgrading to 4.9.2.0, I got it working. I tried several times going from 4.8.0 
straight to 4.9.2.0 with no success.

So my problem is now basically solved, although it's surely strange that 
skipping 4.9.0 made my ACS lock up.

Thanks,

Martin


-Ursprüngliche Nachricht-
Von: Rohit Yadav [mailto:rohit.ya...@shapeblue.com] 
Gesendet: Dienstag, 31. Januar 2017 07:23
An: users@cloudstack.apache.org
Betreff: Re: Login hangs/freezes after update to 4.9.2.0

Martin,


While performing upgrades, did you upgrade in-place or setup a new management 
server (installed fresh rpms)?


If you had installed mgmt server on a new server (freshly installed), did you 
copy the commands.properties and other files from old mgmt server's 
/etc/cloudstack?


I would advise enabling dynamic roles, please see this: 
http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles


Regards.


From: Martin Emrich 
Sent: 27 January 2017 14:57:50
To: users@cloudstack.apache.org
Subject: Login hangs/freezes after update to 4.9.2.0

Hi!

I just updated my test system from 4.7.0 to 4.9.2.0... Now I cannot login 
anymore: The web UI completely freezes after clicking the login button.
(It freezes so hard, I have to kill the management server process before I can 
even close the tab in the browser).

Cloudmonkey also hangs after issuing a command.

The only line I see in the log after clicking "login":

2017-01-27 10:26:14,920 DEBUG [c.c.a.ApiServlet] (catalina-exec-2:ctx-e6b0a361) 
(logid:e9cfda72) ===START===  10.14.2.6 -- POST

An Idea where I could start digging?

Thanks

Martin


rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
  
 

Re: cs 4.5.1, vm stopped when putting xen hosts in maintenance

[Francois Scheurer]

Dear Rafael


Many thanks for your deep analysis.
Nice humor with the planetsalignments BTW ;)

We also suspected the HA Flag to be related this issue, as we never got 
this issue in the past, until we enabled the HA on all VMs and Offerings.


Here is also the workers list:

mysql> select * from op_ha_work;
+--+-+---++--++-+-+---+-+---+-+-+
| id   | instance_id | type  | vm_type| state| 
mgmt_server_id | host_id | created | tried | 
taken   | step  | time_to_try | updated |

+--+-+---++--++-+-+---+-+---+-+-+
|  105 | 343 | HA| User   | Running  |   
345049098498 |  29 | 2015-03-28 03:19:03 | 2 | 2015-03-28 03:30:04 | 
Error |  1394056013 |   3 |
|  109 | 407 | HA| ConsoleProxy   | Running  |   
345049098498 |  33 | 2015-03-28 03:19:03 | 2 | 2015-03-28 03:30:04 | 
Error |  1394056013 |   3 |
|  113 | 407 | HA| ConsoleProxy   | Running  |   
345049098498 |  33 | 2015-03-28 03:19:03 | 2 | 2015-03-28 03:30:04 | 
Error |  1394056013 |   3 |
|  117 | 407 | HA| ConsoleProxy   | Running  |   
345049098498 |  33 | 2015-03-28 03:19:03 | 2 | 2015-03-28 03:30:04 | 
Error |  1394056013 |   3 |
|  121 | 343 | HA| User   | Running  |   
345049098498 |  29 | 2015-03-28 03:19:03 | 2 | 2015-03-28 03:30:04 | 
Error |  1394056013 |   3 |
|  125 | 405 | HA| SecondaryStorageVm | Running  |   
345049098498 |  29 | 2015-03-28 03:19:03 | 2 | 2015-03-28 03:30:04 | 
Error |  1394056013 |   3 |
|  129 | 393 | HA| User   | Running  |   
345049098498 |  25 | 2015-03-28 03:19:03 | 2 | 2015-03-28 03:30:04 | 
Error |  1394056014 |  30 |
|  133 | 405 | HA| SecondaryStorageVm | Running  |   
345049098498 |  29 | 2015-03-28 03:19:03 | 2 | 2015-03-28 03:30:04 | 
Error |  1394056014 |   3 |
|  137 | 402 | HA| ConsoleProxy   | Running  |   
345049098498 |  25 | 2015-03-28 03:19:03 | 2 | 2015-03-28 03:30:04 | 
Error |  1394056014 |   3 |
|  345 | 402 | HA| ConsoleProxy   | Starting |   
345049098498 |  25 | 2015-07-06 13:09:26 | 1 | 2015-07-06 13:09:26 | 
Error |  1402527506 | 770 |
|  349 | 687 | HA| SecondaryStorageVm | Starting |   
345049098498 |  25 | 2015-07-06 13:09:26 | 1 | 2015-07-06 13:09:26 | 
Error |  1402527506 |   2 |
| 2057 |1178 | Migration | User   | Running  |   
345049098498 |  33 | 2017-01-31 07:56:08 | 0 | 2017-01-31 07:56:08 | 
Done  |  1451024774 |  95 |
| 2058 |1736 | Migration | User   | Running  |   
345049098498 | 224 | 2017-01-31 07:56:08 | 0 | 2017-01-31 07:56:08 | 
Done  |  1451024774 |  45 |
| 2059 |1240 | Migration | User   | Running  |   
345049098498 |   7 | 2017-01-31 07:56:08 | 0 | 2017-01-31 07:56:08 | 
Done  |  1451024774 |  61 |
| 2060 |1178 | Migration | User   | Running  |   
345049098122 |  33 | 2017-01-31 07:56:08 | 1 | 2017-01-31 08:06:37 | 
Done  |  1451025374 |  95 |
| 2061 |1178 | Migration | User   | Stopped  
|   NULL |  33 | 2017-01-31 07:56:09 | 5 | 
NULL| Done  |  1451027900 | 109 |
| 2062 |1690 | Migration | User   | Running  |   
345049098498 | 224 | 2017-01-31 07:56:09 | 0 | 2017-01-31 07:56:09 | 
Done  |  1451024774 | 109 |
| 2063 |1177 | Migration | User   | Running  |   
345049098498 |  29 | 2017-01-31 07:56:09 | 0 | 2017-01-31 07:56:09 | 
Done  |  1451024774 |  82 |
| 2064 |1690 | Migration | User   | Running  |   
345049098122 | 224 | 2017-01-31 07:56:09 | 1 | 2017-01-31 07:58:18 | 
Done  |  1451024900 | 109 |
| 2065 |1240 | Migration | User   | Running  |   
345049098122 |   7 | 2017-01-31 07:56:09 | 1 | 2017-01-31 07:58:18 | 
Done  |  1451024900 |  61 |
| 2066 |1736 | Migration | User   | Running  |   
345049098122 | 224 | 2017-01-31 07:56:09 | 1 | 2017-01-31 07:58:18 | 
Done  |  1451024900 |  45 |
| 2067 |1240 | Migration | User   | Stopped  
|   NULL |   7 | 2017-01-31 07:56:09 | 5 | 
NULL| Done  |  1451028003 |  80 |
| 2068 |1178 | Migration | User   | Running  |   
345049098122 |  33 | 2017-01-31 07:56:16 | 0 | 2017-01-31 07:56:16 | 
Done  |  1451024781 |  95 |
| 2069 |1736 | Migration | User   | Running  |   
345049098122 | 224 | 

RE: ApacheCon Miami

[Giles Sirett]

Hi Swen
Yes, hopefully there will be lots of people from this community there. 

It would also be good if we can get users/operators of cloudstack to come along 
too

There is a special ticket price for committers and, as Dag says, speakers don't 
have to pay at all

Kind Regards
Giles


giles.sir...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: S. Brüseke - proIO GmbH [mailto:s.brues...@proio.com] 
Sent: 31 January 2017 09:15
To: users@cloudstack.apache.org
Subject: ApacheCon Miami

Hey guys,

somebody plan to visit ApacheCon (http://us.cloudstackcollab.org/) in Miami 
this May?

Is there a way for our community to get a special price for tickets?

Mit freundlichen Grüßen / With kind regards,

Swen




- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main

USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten 
haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht 
gestattet. 

This e-mail may contain confidential and/or privileged information. 
If you are not the intended recipient (or have received this e-mail in error) 
please notify the sender immediately and destroy this e-mail.  
Any unauthorized copying, disclosure or distribution of the material in this 
e-mail is strictly forbidden. 

Re: ApacheCon Miami

[Dag Sonstebo]

Hi Swen,

If you sign up to the CloudStack marketing mailing list there is more 
information over there. In short – if you get a talk submitted you get free 
access.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 31/01/2017, 09:14, "S. Brüseke - proIO GmbH"  wrote:

Hey guys,

somebody plan to visit ApacheCon (http://us.cloudstackcollab.org/) in Miami 
this May?

Is there a way for our community to get a special price for tickets?

Mit freundlichen Grüßen / With kind regards,

Swen




- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main

USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte 
Informationen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich 
erhalten haben, 
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind 
nicht gestattet. 

This e-mail may contain confidential and/or privileged information. 
If you are not the intended recipient (or have received this e-mail in 
error) please notify 
the sender immediately and destroy this e-mail.  
Any unauthorized copying, disclosure or distribution of the material in 
this e-mail is strictly forbidden. 





dag.sonst...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

ApacheCon Miami

[S . Brüseke - proIO GmbH]

Hey guys,

somebody plan to visit ApacheCon (http://us.cloudstackcollab.org/) in Miami 
this May?

Is there a way for our community to get a special price for tickets?

Mit freundlichen Grüßen / With kind regards,

Swen




- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main

USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten 
haben, 
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht 
gestattet. 

This e-mail may contain confidential and/or privileged information. 
If you are not the intended recipient (or have received this e-mail in error) 
please notify 
the sender immediately and destroy this e-mail.  
Any unauthorized copying, disclosure or distribution of the material in this 
e-mail is strictly forbidden.