Re: Web Console Timing Out

[Sergey Levitskiy]

The error is right here in the log. SSVM is supposed to have secondary storage 
mounted over NFS. 255.255.255.0 is not the correct IP. Likely, the cause is 
garbled record for secondary storage. Try to delete it and re-add.

ERROR: NFS is not currently mounted
Try manually mounting from inside the VM
NFS server is  255.255.255.0
PING 255.255.255.0 (255.255.255.0): 56 data bytes
--- 255.255.255.0 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
WARNING: cannot ping nfs server


From: Serge Byishimo 
Date: Sunday, May 23, 2021 at 5:29 AM
To: users@cloudstack.apache.org 
Subject: Web Console Timing Out
Centos8
Cloudstack 4.15

System VMs are running and their Agent is UP,

System VMs can ping the management server but the management server can not
ping any of the IPs of the System VMs, thus the web console is not
accessible,

I’m thinking it is firewall rules on the System VM side that is causing it,

Any reasons why this is happening?


root@v-2-VM:~#
root@v-2-VM:~#
root@v-2-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh

First DNS server is  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=113 time=29.581 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=113 time=30.140 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 29.581/29.861/30.140/0.280 ms
Good: Can ping DNS server

Good: DNS resolves cloudstack.apache.org

ERROR: NFS is not currently mounted
Try manually mounting from inside the VM
NFS server is  255.255.255.0
PING 255.255.255.0 (255.255.255.0): 56 data bytes
--- 255.255.255.0 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
WARNING: cannot ping nfs server
routing table follows
Kernel IP routing table
Destination Gateway Genmask Flags Metric RefUse
Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG0  00 eth2
8.8.4.4 192.168.1.1 255.255.255.255 UGH   0  00 eth1
8.8.8.8 192.168.1.1 255.255.255.255 UGH   0  00 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0  00 eth0
192.168.1.0 0.0.0.0 255.255.255.0   U 0  00 eth1
192.168.1.0 0.0.0.0 255.255.255.0   U 0  00 eth2

Management server is 192.168.1.8. Checking connectivity.
Good: Can connect to management server 192.168.1.8 port 8250

Good: Java process is running

Tests Complete. Look for ERROR or WARNING above.
root@v-2-VM:~#





root@v-2-VM:~#
root@v-2-VM:~# iptables -nL
Chain INPUT (policy DROP)
target prot opt source   destination
ACCEPT all  --  0.0.0.0/00.0.0.0/0
ACCEPT icmp --  0.0.0.0/00.0.0.0/0icmptype 8
ACCEPT all  --  0.0.0.0/00.0.0.0/0state
RELATED,ESTABLISHED
ACCEPT all  --  0.0.0.0/00.0.0.0/0state
RELATED,ESTABLISHED
ACCEPT all  --  0.0.0.0/00.0.0.0/0state
RELATED,ESTABLISHED
ACCEPT icmp --  0.0.0.0/00.0.0.0/0
ACCEPT tcp  --  0.0.0.0/00.0.0.0/0state NEW tcp
dpt:3922
ACCEPT tcp  --  0.0.0.0/00.0.0.0/0state NEW tcp
dpt:8001
ACCEPT tcp  --  0.0.0.0/00.0.0.0/0state NEW tcp
dpt:8001
ACCEPT tcp  --  0.0.0.0/00.0.0.0/0state NEW tcp
dpt:443
ACCEPT tcp  --  0.0.0.0/00.0.0.0/0state NEW tcp
dpt:80
ACCEPT tcp  --  0.0.0.0/00.0.0.0/0state NEW tcp
dpt:8080
ACCEPT icmp --  0.0.0.0/00.0.0.0/0

Chain FORWARD (policy DROP)
target prot opt source   destination

Chain OUTPUT (policy ACCEPT)
target prot opt source   destination
root@v-2-VM:~#
root@v-2-VM:~#

Thanks

Re: [VOTE] Apache Cloudstack 4.15.0.0 and UI [RC3]

[Sergey Levitskiy]

You can try this. Restore your DB backup, register SSVM template and run the 
following against your MySQL DB before starting the  upgrade.

ALTER TABLE `cloud`.`project_account`
 ADD CONSTRAINT `fk_project_account__account_id` FOREIGN KEY(`account_id`) 
REFERENCES `account`(`id`) ON DELETE CASCADE ,
 ADD CONSTRAINT `uc_project_account__project_id_account_id_user_id` UNIQUE 
(`project_id`, `account_id`, `user_id`) ;


If it still fails capture and  post full management server log.


Thanks,
Sergey

On 12/26/20, 2:27 AM, "Hean Seng"  wrote:

I restore the backup db, and reregister the system template using
cloud-install-sys-tmplt
, it sill getting error.

stemVm template not found. Ovm3 hypervisor is not used, so not failing
upgrade

2020-12-26 10:11:37,713 DEBUG [c.c.u.d.Upgrade41400to41500] (main:null)
(logid:) Updating KVM System Vms

2020-12-26 10:11:37,720 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null)
(logid:) Unable to upgrade the database

com.cloud.utils.exception.CloudRuntimeException: 4.15.0.0KVM SystemVm
template not found. Cannot upgrade system Vms

at

com.cloud.upgrade.dao.Upgrade41400to41500.updateSystemVmTemplates(Upgrade41400to41500.java:214)

at

com.cloud.upgrade.dao.Upgrade41400to41500.performDataMigration(Upgrade41400to41500.java:70)

On Sat, Dec 26, 2020 at 5:48 PM Hean Seng  wrote:

> For first time I upgrade and start the MGMT server , it show
> following error:
>
> 2020-12-26 09:02:32,499 DEBUG [c.c.u.d.Upgrade41400to41500] (main:null)
> (logid:) Updating System Vm template IDs
>
> 2020-12-26 09:02:32,503 DEBUG [c.c.u.d.Upgrade41400to41500] (main:null)
> (logid:) Updating KVM System Vms
>
> 2020-12-26 09:02:32,511 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) Unable to upgrade the database
>
> com.cloud.utils.exception.CloudRuntimeException: 4.15.0.0KVM SystemVm
> template not found. Cannot upgrade system Vms
>
> at
> 
com.cloud.upgrade.dao.Upgrade41400to41500.updateSystemVmTemplates(Upgrade41400to41500.java:214)
>
> at
> 
com.cloud.upgrade.dao.Upgrade41400to41500.performDataMigration(Upgrade41400to41500.java:70)
>
> at
> 
com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:262)
>
> at
> 
com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:342)
>
> at
> 
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:64)
>
> at
> 
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:54)
>
> at
> 
org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182)
>
> at
> 
org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53)
>
> at
> 
org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360)
>
> at
> 
org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158)
>
> at
> 
org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:122)
>
> at
> 
org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:894)
>
> at
> 
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)
>
> at
> 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144)
>
> at
> 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:121)
>
> at
> 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:244)
>
> at
> 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:249)
>
> at
> 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:232)
>
> at
> 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:116)
>
> at
> 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:78)
>
> at
> 
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
>
> at
> 
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
>
> at
> 

Re: Deploying Linux/Windows Vm's on hypervisors

[Sergey Levitskiy]

ACC can only route provisioning based on the tags ( compute and storage) 
assigned to a service offering so if you are looking at mandatory automatic 
assignment based on guest OS provisioning it is not possible without 
segregating your environment. If you are OK with soft segregation then you can 
create 2 sets of service offerings.
http://docs.cloudstack.apache.org/en/4.11.1.0/adminguide/service_offerings.html

Thanks,
Sergey

On 9/15/20, 7:36 AM, "Rakesh Venkatesh"  wrote:

Hello All


I want your opinion on how you organize deploying different types of
VM's(Linux, windows) on a specific set of hypervisors.

In our setup, we have added a tag "Linux,Unix" on few hypervisors and also
on templates so that when we create a vm using these templates then it will
be deployed on HV's matching that tag only. If there is no tag on the
template then it can be deployed on any hypervisor.

We have some windows templates which do not have a tag as they are uploaded
by customers and when VM is created from that template, it will be deployed
in any suitable hypervisor but we want it to be deployed on a hypervisor
having "Windows" tag. Is there any way to achieve this apart from using
template tag?

Another way is to use the OS type and match with hypervisor tags but that
may not be a suitable solution. So if anyone has managed to deploy windows
vm's on windows tagged hypervisor and linux vm's on linux/unix tagged
hypervisor then please let me know
-- 
Thanks and regards
Rakesh

Re: Cloudstack 4.11.3 to 4.13.1 DB Upgrade Error

[Sergey Levitskiy]

Correcting few typos:

There are 2 possibilities here 
1.  Since column reаson was added to table network_acl_item during the DB 
schema upgrade from 4.11.2 to 4.12 this likely means that  your upgrade failed 
on a later step and you've tried to re-execute upgrade script. If so this is 
not a right sequence of recovering from a failed upgrade. You need to restore 
DB to pre-upgrade state and re-execute all according to the guide. You need to 
capture full management_server.log to figure you what went wrong
2. You might have upgraded to 4.11.3 using RC or pre-released code where 
column insertion happens during 4.11.2->4.11.3 phase that later was removed 
from the ACS master. If that's the case there is way to succeed although longer 
and not as pleasant. You still need to try #1 to capture and share full 
management_server.log


On 7/19/20, 3:56 AM, "Rakesh v"  wrote:

From the log, it looks like the column "reason" already exists in 
network_acl_item table. Does this take suppose to have that field in 4.11?

Sent from my iPhone

> On 19-Jul-2020, at 9:30 AM, Ammad Syed  wrote:
> 
> Hi,
> 
> I am trying to upgrade cloudstack from 4.11.3 to 4.13.1. I am having 
below
> db upgrade error while upgrading. I think the upgrade is executing 
wrong
> script i.e from 4.11.2 to 4.12.0. Please advise.
> 
> 2020-07-13 20:29:37,650 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle]
> (main:null) (logid:) Running system integrity checker
> com.cloud.upgrade.DatabaseUpgradeChecker@8dfe921
> 2020-07-13 20:29:37,651 INFO  [c.c.u.DatabaseUpgradeChecker] 
(main:null)
> (logid:) Grabbing lock to check for database upgrade.
> 2020-07-13 20:29:37,684 DEBUG [c.c.u.d.VersionDaoImpl] (main:null) 
(logid:)
> Checking to see if the database is at a version before it was the 
version
> table is created
> 2020-07-13 20:29:37,701 INFO  [c.c.u.DatabaseUpgradeChecker] 
(main:null)
> (logid:) DB version = 4.11.3.0 Code Version = 4.13.1.0
> 2020-07-13 20:29:37,701 INFO  [c.c.u.DatabaseUpgradeChecker] 
(main:null)
> (logid:) Database upgrade must be performed from 4.11.3.0 to 4.13.1.0
> 2020-07-13 20:29:37,712 DEBUG [c.c.u.DatabaseUpgradeChecker] 
(main:null)
> (logid:) Running upgrade Upgrade41120to41200 to upgrade from
> 4.11.2.0-4.12.0.0 to 4.12.0.0
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- Licensed to the Apache Software Foundation (ASF) under one
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- or more contributor license agreements.  See the NOTICE file
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- distributed with this work for additional information
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- regarding copyright ownership.  The ASF licenses this file
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- to you under the Apache License, Version 2.0 (the
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- "License"); you may not use this file except in compliance
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- with the License.  You may obtain a copy of the License at
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:) --
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> --   http://www.apache.org/licenses/LICENSE-2.0
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:) --
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- Unless required by applicable law or agreed to in writing,
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- software distributed under the License is distributed on an
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- KIND, either express or implied.  See the License for the
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- specific language governing permissions and limitations
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- under the License.
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> --;
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) 
(logid:)
> -- Schema upgrade from 4.11.2.0 to 

Re: Cloudstack 4.11.3 to 4.13.1 DB Upgrade Error

[Sergey Levitskiy]

There are 2 possibilities here 
1.  Since column reаson was added to table network_acl_item during the DB 
schema upgrade from 4.11.12 to 4.12 this likely means that  your upgrade failed 
on a later step and you've tried to re-execute upgrade script. If so this is 
not a right sequence of recovering from a failed upgrade. You need to restore 
DB to pre-upgrade state and re-execute all according to the guide. You need to 
capture full management_server.log to figure you what went wrong
2. You might have upgraded to 4.11.3 using RC or pre-released code where column 
insertion happens during 4.11.12->4.11.13 phase that later was removed from the 
ACS master. If that's the case there is way to succeed although longer and not 
as pleasant. You still need to try #1 to capture and share full 
management_server.log


On 7/19/20, 3:56 AM, "Rakesh v"  wrote:

From the log, it looks like the column "reason" already exists in 
network_acl_item table. Does this take suppose to have that field in 4.11?

Sent from my iPhone

> On 19-Jul-2020, at 9:30 AM, Ammad Syed  wrote:
> 
> Hi,
> 
> I am trying to upgrade cloudstack from 4.11.3 to 4.13.1. I am having below
> db upgrade error while upgrading. I think the upgrade is executing wrong
> script i.e from 4.11.2 to 4.12.0. Please advise.
> 
> 2020-07-13 20:29:37,650 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle]
> (main:null) (logid:) Running system integrity checker
> com.cloud.upgrade.DatabaseUpgradeChecker@8dfe921
> 2020-07-13 20:29:37,651 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) Grabbing lock to check for database upgrade.
> 2020-07-13 20:29:37,684 DEBUG [c.c.u.d.VersionDaoImpl] (main:null) 
(logid:)
> Checking to see if the database is at a version before it was the version
> table is created
> 2020-07-13 20:29:37,701 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) DB version = 4.11.3.0 Code Version = 4.13.1.0
> 2020-07-13 20:29:37,701 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) Database upgrade must be performed from 4.11.3.0 to 4.13.1.0
> 2020-07-13 20:29:37,712 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) Running upgrade Upgrade41120to41200 to upgrade from
> 4.11.2.0-4.12.0.0 to 4.12.0.0
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- Licensed to the Apache Software Foundation (ASF) under one
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- or more contributor license agreements.  See the NOTICE file
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- distributed with this work for additional information
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- regarding copyright ownership.  The ASF licenses this file
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- to you under the Apache License, Version 2.0 (the
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- "License"); you may not use this file except in compliance
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- with the License.  You may obtain a copy of the License at
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) 
--
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> --   http://www.apache.org/licenses/LICENSE-2.0
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) 
--
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- Unless required by applicable law or agreed to in writing,
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- software distributed under the License is distributed on an
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- KIND, either express or implied.  See the License for the
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- specific language governing permissions and limitations
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- under the License.
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> --;
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- Schema upgrade from 4.11.2.0 to 4.12.0.0
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> --;
> 2020-07-13 20:29:37,715 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- [CLOUDSTACK-10314] Add reason column to ACL rule table
> 2020-07-13 20:29:37,716 DEBUG 

Re: Need Support

[Sergey Levitskiy]

You need to cleanup entry in vmware_data_center table e.g. assuming you have 
unique vmware datacenter names:
delete from vmware_data_center where name='name_of_your_datacenter'

Thanks,
Sergey

On 7/18/20, 9:22 AM, "Rakesh v"  wrote:

All the cloudstack management logs are stored in 
/var/log/cloudstack/management/management-server.log

Search for "error|fail|exception" in the log

Sent from my iPhone

> On 18-Jul-2020, at 11:21 AM, Ajit Singh  wrote:
> 
> Hi Rakesh
> 
> Can you pls guide how to get logs for the same as I not well versed with
> apache cloud stack
> 
> Regards
> Ajit Singh
> 
>> On Sat, Jul 18, 2020 at 1:58 PM Rakesh v  
wrote:
>> 
>> Providing exceptions in the logs will be more helpful
>> 
>> Sent from my iPhone
>> 
 On 18-Jul-2020, at 7:46 AM, Ajit Singh  wrote:
>>> 
>>>  Dear All
>>> 
>>> I am new to Apache Cloud Stack and we have one already running in our
>>> office infra and it has 3 zones having hypervisors as Xenservers and
>>> management servers are also installed on Xenservers.
>>> 
>>> Now, I am trying to add VMware vsphere in the existing Apache Cloud 
Stack
>>> through it's UI by creating a new zone but not able to do so. It throws
>>> below errors.
>>> 
>>> *Something went wrong; please correct the following: This DC is already
>>> part of other CloudStack zone(s). Cannot add this DC to more zones.*
>>> 
>>> and if I am able to resolve the above error, it throws another error
>>> 
>>> *Something went wrong; please correct the following: unable to add the
>>> external cluster*
>>> 
>>> It would be great if somebody can help me out with the above mentioned
>>> errors.
>>> 
>>> Regards
>>> Ajit Singh
>> 

Re: Unable to shrink a data disk

[Sergey Levitskiy]

Not a chance. QCOW volume type can be shrunk on RBD backend only. The only 
other option of shrinking with KVM is LVM backend.


On 7/12/20, 4:48 AM, "Fariborz Navidan"  wrote:

Any chance to get it working on NFS in later version?

On Sun, Jul 12, 2020 at 9:44 AM Nicolas Vazquez <
nicolas.vazq...@shapeblue.com> wrote:

> Hi Fariborz,
>
> Unless the QCOW2 volume is on an RBD storage pool then shrinking is not
> supported, only growing resize.
>
>
> Regards,
>
> Nicolas Vazquez
>
> 
> From: Fariborz Navidan 
> Sent: Thursday, July 9, 2020 6:47 AM
> To: users@cloudstack.apache.org 
> Subject: Unable to shrink a data disk
>
> Hi,
>
> I'm unable to shrink a data disk on KVM. MS log reports "Unable to shrink
> volume of type QCOW2". I'm using CS 4.14.0.0. Doesn't QCOW2 format support
> shrinking?
>
> Regards.
>
> nicolas.vazq...@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>
>
>
>

Re: help! ACS4.11.3+vSphere6.x Sendary failure,

[Sergey Levitskiy]

You can’t have attachments in emails to this group. Instead save your full log 
elsewhere e.g. pastbin and send a link.

From: hu xl 
Reply-To: "users@cloudstack.apache.org" 
Date: Sunday, June 28, 2020 at 11:25 AM
To: "users@cloudstack.apache.org" , 
"users...@cloudstack.apache.org" 
Subject: help! ACS4.11.3+vSphere6.x Sendary failure,

这个问题已经困扰了我2周时间，希望能得到帮助，我尝试使用ACS4.11.3 版本来管理vSphere6.5/6.7，但都失败了。
具体现象是：开启资源域后SSVM和CPVM不能启动，同时在vCenter 中没有看到挂载Secondary的操作，以及复制sysvm的操作，
我尝试手工的方式使用UUID 把Secodary 挂载在vCenter Cluster中，启动资源域后也同样在没有任何响应, 
ACS日志里报错：“received secondary storage vm alert/Secondary Storage Vm creation 
failure, zone:..", 完整的ACS日志在附件里。
我有成功管理ACS+KVM的经验，以及使用ACS4.3+vSphere5.5的经验。但目前的问题确实把我难到了，希望能获得大家的指导，谢谢！

This problem has been plaguing me for two weeks and I am looking for help. I 
have tried and failed to manage Vsphere6.5/6.7 using ACS4.11.3.
The specific phenomenon is that SSVM and CPVM cannot be started after the 
resource domain is turned on. Meanwhile, there is no operation to mount 
Secondary and copy SYSVM in vCenter.
I tried to mount Secodary in vCenter Cluster by using UUID manually. After 
starting the resource domain, there was no response. The ACS log reported an 
error: "Received Secondary Storage VM Alert /Secondary Storage VM Creation 
Failure, zone:..", the complete ACS log is attached.
I have experience successfully managing ACS+KVM and using ACS4.3+vSphere5.5.But 
the current problem does make me difficult, I hope to get your guidance, thank 
you!

[cid:image001.png@01D64D6C.A2FF5EE0]

[cid:image002.png@01D64D6C.A2FF5EE0]
[cid:image003.png@01D64D6C.A2FF5EE0]

Re: help! ACS4.11.3+vSphere6.x Sendary failure,

[Sergey Levitskiy]

You can’t have attachments in emails to this group. Instead save your full log 
elsewhere e.g. pastbin and send a link.

From: hu xl 
Reply-To: "us...@cloudstack.apache.org" 
Date: Sunday, June 28, 2020 at 11:25 AM
To: "us...@cloudstack.apache.org" , 
"users-cn@cloudstack.apache.org" 
Subject: help! ACS4.11.3+vSphere6.x Sendary failure,

这个问题已经困扰了我2周时间，希望能得到帮助，我尝试使用ACS4.11.3 版本来管理vSphere6.5/6.7，但都失败了。
具体现象是：开启资源域后SSVM和CPVM不能启动，同时在vCenter 中没有看到挂载Secondary的操作，以及复制sysvm的操作，
我尝试手工的方式使用UUID 把Secodary 挂载在vCenter Cluster中，启动资源域后也同样在没有任何响应, 
ACS日志里报错：“received secondary storage vm alert/Secondary Storage Vm creation 
failure, zone:..", 完整的ACS日志在附件里。
我有成功管理ACS+KVM的经验，以及使用ACS4.3+vSphere5.5的经验。但目前的问题确实把我难到了，希望能获得大家的指导，谢谢！

This problem has been plaguing me for two weeks and I am looking for help. I 
have tried and failed to manage Vsphere6.5/6.7 using ACS4.11.3.
The specific phenomenon is that SSVM and CPVM cannot be started after the 
resource domain is turned on. Meanwhile, there is no operation to mount 
Secondary and copy SYSVM in vCenter.
I tried to mount Secodary in vCenter Cluster by using UUID manually. After 
starting the resource domain, there was no response. The ACS log reported an 
error: "Received Secondary Storage VM Alert /Secondary Storage VM Creation 
Failure, zone:..", the complete ACS log is attached.
I have experience successfully managing ACS+KVM and using ACS4.3+vSphere5.5.But 
the current problem does make me difficult, I hope to get your guidance, thank 
you!

[cid:image001.png@01D64D6C.A2FF5EE0]

[cid:image002.png@01D64D6C.A2FF5EE0]
[cid:image003.png@01D64D6C.A2FF5EE0]

Re: Converting an existing management server to a region

[Sergey Levitskiy]

Regions need to share user, account and domain records so second, third etc 
regions need to be rebuild from scratch and import certain tables from region 
1. 
http://docs.cloudstack.apache.org/en/latest/installguide/configuration.html#adding-regions-optional


On 6/7/20, 4:22 AM, "Fariborz Navidan"  wrote:

Hello,

I have two stand-alone running management servers. I would like to add them
to each other as regions. Can I set region ID on an existing management
server or I will to reinstall them
Thanks

Re: Connecting to VNC server via public IP - timed out

[Sergey Levitskiy]

When ACS deploys an instance into Vmware cluster it adds VNC configuration into 
each VM using VNC port range 5900 to 5999 and then using configurable 
additional range 5-51000
(setting name vmware.additional.vnc.portrange.start).  To get available port it 
asks Vmware cluster resource for a list of port in use and find first available 
one. Later when Console session needs to be established ACS ask vCenter which 
vnc port a particular VM listens on and proxies that connection over to the 
client. Keep in mind if you have more than a 1000 VMs you will need to use 
additional VNC range that is not open on ESXi firewall. You will have to 
manually do it on each esx host or write a custom VIB that will open FW for the 
whole range ACS is configured to use.

Thanks,
Sergey



On 5/21/20, 5:54 AM, "cristian.c@istream.today"  
wrote:

Hello,



I have a simple question regarding console proxy, from where is loading
the IP for VNC, I do not understand from where.. In cloudstack I have the
management IP(10.1.1.31) why is going through ESXI public IP ? If I leave
like this it will not work because of the route.



Log :

2020-05-21 12:45:11,966 INFO  [cloud.agent.Agent]
(agentRequest-Handler-4:null) Ready command is processed for agent id = 9

2020-05-21 12:45:11,966 INFO  [resource.consoleproxy.ConsoleProxyResource]
(agentRequest-Handler-4:null) Receive ReadyCommand, response with
ReadyAnswer

2020-05-21 12:46:02,371 INFO  [cloud.consoleproxy.ConsoleProxy]
(Thread-7:null) Added viewer object
com.cloud.consoleproxy.ConsoleProxyVncClient@f0237ee

2020-05-21 12:46:02,376 INFO  [cloud.consoleproxy.ConsoleProxyVncClient]
(Thread-8:null) Connect to VNC server directly. host: 51.79.82.161, port:
5935

2020-05-21 12:46:02,384 INFO  [consoleproxy.vnc.VncClient] (Thread-8:null)
Connecting to VNC server 51.xx.xx.161:5935...

2020-05-21 12:48:12,456 ERROR [cloud.consoleproxy.ConsoleProxyVncClient]
(Thread-8:null) Unexpected exception

java.net.ConnectException: Connection timed out (Connection timed out)





root@v-34-VM:~# ip r l

default via 51.xx.xx.190 dev eth2

10.0.0.0/8 via 10.1.1.242 dev eth1

10.1.0.0/16 dev eth1 proto kernel scope link src 10.1.4.131

51.xx.xx.176/28 dev eth2 proto kernel scope link src 51.xx.xx.177

51.xx.xx.161 via 10.1.1.242 dev eth1

158.xx.xx.137 via 10.1.1.242 dev eth1

172.16.0.0/12 via 10.1.1.242 dev eth1

192.168.0.0/16 via 10.1.1.242 dev eth1





I have a different setup where I do not have this problem, I do not
understand why this route is persistent and added back immediately after I
remove ( in case I try to load the vnc console over web)   



I have CloudStack 4.13 + VMware 6.0



Thanks in advance!

Cristian

Re: Testing CS 4.13.1

[Sergey Levitskiy]

Plz share full management-server.log since your snippet doesn't have relevant 
lines.

Thanks,
Sergey


On 5/18/20, 1:25 PM, "Luis Martinez"  wrote:

This is the error I see in the logs, I am trying to ssh to the VM for 
secondary storage but i am not able to do it.


2020-05-18 16:23:26,778 DEBUG [c.c.c.ConsoleProxyManagerImpl] 
(consoleproxy-1:ctx-195a645d) (logid:e873b8f5) Zone 1 is ready to launch 
console proxy
2020-05-18 16:23:29,820 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-2:ctx-8b9729d8) (logid:d7872342) AutoScaling Monitor is 
running...
2020-05-18 16:23:30,138 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-5:ctx-a1b22863) (logid:bd79141a) HostStatsCollector is 
running...
2020-05-18 16:23:30,195 DEBUG [c.c.a.t.Request] 
(StatsCollector-5:ctx-a1b22863) (logid:bd79141a) Seq 
1-1544453197211369507: Received:  { Ans: , MgmtId: 266682308174733, via: 
1(cloudstackvm1), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2020-05-18 16:23:30,522 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-3:ctx-ef56a510) (logid:1a0c1267) StorageCollector is 
running...
2020-05-18 16:23:30,530 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-3:ctx-ef56a510) (logid:1a0c1267) *There is no secondary 
storage VM for secondary storage host nfs://10.0.8.10/mnt/CS01/secondary*
2020-05-18 16:23:30,533 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] 
(StatsCollector-3:ctx-ef56a510) (logid:1a0c1267) 
getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
2020-05-18 16:23:30,533 DEBUG [c.c.h.XenServerGuru] 
(StatsCollector-3:ctx-ef56a510) (logid:1a0c1267) We are returning the 
default host to execute commands because the command is not of Copy type.
2020-05-18 16:23:30,612 DEBUG [c.c.a.t.Request] 
(StatsCollector-3:ctx-ef56a510) (logid:1a0c1267) Seq 
1-1544453197211369508: Received:  { Ans: , MgmtId: 266682308174733, via: 
1(cloudstackvm1), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2020-05-18 16:23:33,611 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentManager-Handler-3:null) (logid:) Ping from Routing host 
1(cloudstackvm1)

On 5/18/2020 2:43 PM, Rohit Yadav wrote:
> Hi Luis,
>
> Please use the correct systemvmtemplate version/link for CloudStack 
4.13.1.0.
>
> Refer to the install/upgrade docs such as 
http://docs.cloudstack.apache.org/en/4.13.1.0/upgrading/upgrade/upgrade-4.12.html
>
> Regards.
>
> Regards,
> Rohit Yadav
>
> 
> From: Luis Martinez 
> Sent: Monday, May 18, 2020 11:07:20 PM
> To: users@cloudstack.apache.org 
> Subject: Testing CS 4.13.1
>
> Hi Group
>
> I need help, I am installing 4.13.1 for testing, installation is fine
> but secondary storage is not working, I tried to ssh to the VM and I am
> unable to do it, I used the following lines in different installations
> to see if this fixes the problem but no. how can I troubleshoot this? or
> am I using the wrong version?
>
>
> 
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
> -m /mnt/secondary -u
> 
http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2
> -h kvm -F
> 
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
> -m /mnt/secondary -u
> 
http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.1-kvm.qcow2.bz2
> -h kvm -F
> 
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
> -m /mnt/secondary -u
> 
http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.0-kvm.qcow2.bz2
> -h kvm -F
>
> Thank you in Advance.
>
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>
>   
>

Re: Unable to create VM through 100 and 200 GB temaplate.

[Sergey Levitskiy]

Most likely this timeout is set to 20 min at present 
"vmware.vcenter.session.timeout". If you use full clones on vmware set it to at 
least 1 hour. Actual cloning time depends on the size of the root disk and 
speed of your storage array.

Thanks,
Sergey

On 4/28/20, 11:29 AM, "Andrija Panic"  wrote:

Attachments are (afaik) stripped when you send email to ML (but I got it,
NFS v3, so not relevant/VMFS)

Increase all limits to "-1" as already suggested by Marc-Andre and if it
still fails, please paste full management logs on the pastebin or similar
(or attach the management-server.log on some external service)

Regards,
Andrija

On Tue, 28 Apr 2020 at 15:44, pradeep pal  wrote:

> Hi All,
>
> If we do these changes. any impact in production?
>
> @Andrija Panic , Please find the VMFS version
> screenshot.
>
> One more thing When we create VMs through the template, the template is
> successfully copied on secondary storage, but when it goes on primary
> storage the VM creation failed.
>
> Regards
> Pradeep
>
> --
> *From:* Andrija Panic 
> *Sent:* Sunday, April 26, 2020 7:32 PM
> *To:* users 
> *Subject:* Re: Unable to create VM through 100 and 200 GB temaplate.
>
> A proper log, or perhaps (if this is what you tried to share) a
> screenshot uploaded online would be better.
>
> Which VMFS version?
>
> regards,
>
> On Sat, 25 Apr 2020 at 19:01, pradeep pal  wrote:
>
> > Hi All,
> >
> > We have built a new  VMware cluster in existing CloudStack infra. But
> when
> > we create new machines with 100 and 200 GB template. always we get this
> > error.
> >
> >
> > Clone virtual machine
> > Status:
> > The task was canceled by a user.
> > Initiator: VSPHERE.LOCAL\Administrator
> > Target:
> > 271152f909563b3f954b87539960261e
> > Server: xx.xx.xx.xx
> > Related events:
> > 04/25/2020, 9:07:42 AM
> >
> > Removed ROOT-5420
> > on xx.xx.xx.xx
> > from HOSTDC1
> > 04/25/2020, 9:07:42 AM
> >
> > Cannot clone 271152f909563b3f954b87539960261e
> > :
> > 04/25/2020, 8:46:59 AM
> >
> > Assign a new instance UUID (502dd00e-baf0-9342-0e31-a635e9bacc48) to
> > ROOT-5420
> > 04/25/2020, 8:46:59 AM
> >
> > Assigned new BIOS UUID (422d4f0d-fb3d-3ae9-8710-77eac2aa06f5) to
> ROOT-5420
> > on xx.xx.xx.xx
> > in HOSTDC1
> > 04/25/2020, 8:46:59 AM
> >
> > Cloning 271152f909563b3f954b87539960261e
> > on host xx.xx.xx.xx
> > in HOSTDC1
> > to ROOT-5420 on host 10.106.13.103
> > 04/25/2020, 8:46:59 AM
> >
> > Task: Clone virtual machine
> >
> > kindly help us in this case.
> >
> > Thanks
> > Pradeep
> >
>
>
> --
>
> Andrija Panić
>


-- 

Andrija Panić

Re: Adding Regions and Zone's

[Sergey Levitskiy]

This level of HA lays beyond the scope of ACS. I advise you to explore GLB that 
runs outside of ACS that will direct traffic to the available VMs across 
regions. Alternatively if you physical network architecture can provide the 
same L2 segments across regions you can explore keepalived VRRP feature to move 
the same IP across the set of VMs in different regions. Those floating IPs will 
need to be managed outside of ACS.

Thanks,
Sergey


On 1/27/20, 11:55 PM, "Prashant Rai"  wrote:

Hi All,

   - We have installed Cloudstack on two different regions and both are
   working fine.
   - We want to add both regions and have Zone Availability, i.e. if one
   zone goes down or fail, another zone will take care, and VM's can migrate
   to that zone. Providing zone HA feature.
   - Higher availability of the services: users can deploy services
across Availability
   Zones and even if one of the Availability Zones goes down the services
   are still available to the end-user through VMs deployed in other zones.

Please guide..

Regards,
Prashant.

Re: Adding Regions and Zone's

[Sergey Levitskiy]

This level of HA lays beyond the scope of ACS. I advise you to explore GLB that 
runs outside of ACS that will direct traffic to the available VMs across 
regions. Alternatively if you physical network architecture can provide the 
same L2 segments across regions you can explore keepalived VRRP feature to move 
the same IP across the set of VMs in different regions. Those floating IPs will 
need to be managed outside of ACS.

Thanks,
Sergey


On 1/27/20, 11:55 PM, "Prashant Rai"  wrote:

Hi All,

   - We have installed Cloudstack on two different regions and both are
   working fine.
   - We want to add both regions and have Zone Availability, i.e. if one
   zone goes down or fail, another zone will take care, and VM's can migrate
   to that zone. Providing zone HA feature.
   - Higher availability of the services: users can deploy services
across Availability
   Zones and even if one of the Availability Zones goes down the services
   are still available to the end-user through VMs deployed in other zones.

Please guide..

Regards,
Prashant.

Re: Can't start systemVM in a new advanced zone deployment

[Sergey Levitskiy]

Yes snapshots are supposed to be in PS template copy. 

On 6/6/19, 9:24 AM, "Yiping Zhang"  wrote:

The nfs volume definitely allows root mount and have RW permissions, as we 
already see the volume mounted and template staged on primary storage. The 
volume is mounted as NFS3 datastore in vSphere.

Volume snapshot is enabled,  I can ask to have snapshot disabled to see if 
it makes any differentces.   I need to find out more about NFS version and 
qtree mode from our storage admin.  

One thing I noticed is that when cloudstack templates are staged on to 
primary storage, a snapshot was created, which does not exist In the original 
OVA or on secondary storage.  I suppose this is the expected behavior?

Yiping

On 6/6/19, 6:59 AM, "Sergey Levitskiy"  wrote:

This option is 'vol options name_of_volume nosnapdir on' however if I 
recall it right is supposed to work even with .snapshot directory visible
Can you find out all vol options on your netapp volume? I would be most 
concerned about:
- NFS version - NFS v4 should be disabled
- security qtree mode to be set to UNIX
- allow root mount

I am also wondering if ACS is able to create ROOT-XX folder so you 
might want to watch the content of the DS when ACS tries the operations.
 

On 6/5/19, 11:43 PM, "Paul Angus"  wrote:

Hi Yiping,

do you have snapshots enabled on the NetApp filer?  (it used to be 
seen as a  ".snapshot"  subdirectory in each directory)

If so try disabling snapshots - there used to be a bug where the 
.snapshot directory would confuse CloudStack.

paul.an...@shapeblue.com 

https://nam04.safelinks.protection.outlook.com/?url=www.shapeblue.comdata=02%7C01%7Cyipzhang%40adobe.com%7C557bf647ff13413c66b708d6ea87220d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636954263448727822sdata=NhoxwF0x4%2F8yn%2B8ck%2BCI8RUKEEDGnI73QfDDQeSmZUc%3Dreserved=0
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Yiping Zhang  
Sent: 05 June 2019 23:38
To: users@cloudstack.apache.org
Subject: Re: Can't start systemVM in a new advanced zone deployment

Hi, Sergey:

I found more logs in vpxa.log ( the esxi hosts are using UTC time 
zone,  so I was looking at wrong time periods earlier).  I have uploaded more 
logs into pastebin.

From these log entries,  it appears that when copying template to 
VM,  it tried to open destination VMDK file and got error file not found.  

In case that the CloudStack attempted to create a systemVM,  the 
destination VMDK file path it is looking for is 
"//.vmdk",  see uploaded log at 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpastebin.com%2FaFysZkTydata=02%7C01%7Cyipzhang%40adobe.com%7C557bf647ff13413c66b708d6ea87220d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636954263448727822sdata=YyB9VdghCgiBuUmDZ8gIc0jPlM8miPzemX2UEAZ3sFA%3Dreserved=0

In case when I manually created new VM from a (different) template 
in vCenter UI,   the destination VMDK file path it is looking for is 
"//.vmdk", see uploaded log at 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpastebin.com%2FyHcsD8xBdata=02%7C01%7Cyipzhang%40adobe.com%7C557bf647ff13413c66b708d6ea87220d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636954263448732817sdata=N%2BZHteGo3LDU0pvhBtzv7wcocAv35gRE9b9yKVQa6%2FQ%3Dreserved=0

So, I am confused as to how the path for destination VMDK was 
determined and by CloudStack or VMware, how did I end up with this?

Yiping
    
    
On 6/5/19, 12:32 PM, "Sergey Levitskiy"  wrote:

Some operations log get transferred to vCenter log vpxd.log. It 
is not straightforward to trace I but Vmware will be able to help should you 
open case with them. 


On 6/5/19, 11:39 AM, "Yiping Zhang" 
 wrote:

Hi, Sergey:

During the time period when I had problem cloning template, 
 there are only a few unique entries in vmkernel.log, and they were repeated 
hundreds/thousands of times by all the cpu cores:

2019-06-02T16:47:00.633Z cpu9:8491061)FSS: 6751: Failed to 
open file 'hpilo-d0ccb15'; Requested flags 0x5, world: 8491061 [ams-ahs], 
(Existing flags 0x5, world: 8491029 [ams-main]): Busy
2019-06-02T16:47:49.320Z cpu1:66415)nhpsa: 

Re: Can't start systemVM in a new advanced zone deployment

[Sergey Levitskiy]

This option is 'vol options name_of_volume nosnapdir on' however if I recall it 
right is supposed to work even with .snapshot directory visible
Can you find out all vol options on your netapp volume? I would be most 
concerned about:
- NFS version - NFS v4 should be disabled
- security qtree mode to be set to UNIX
- allow root mount

I am also wondering if ACS is able to create ROOT-XX folder so you might want 
to watch the content of the DS when ACS tries the operations.
 

On 6/5/19, 11:43 PM, "Paul Angus"  wrote:

Hi Yiping,

do you have snapshots enabled on the NetApp filer?  (it used to be seen as 
a  ".snapshot"  subdirectory in each directory)

If so try disabling snapshots - there used to be a bug where the .snapshot 
directory would confuse CloudStack.

paul.an...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Yiping Zhang  
Sent: 05 June 2019 23:38
To: users@cloudstack.apache.org
Subject: Re: Can't start systemVM in a new advanced zone deployment

Hi, Sergey:

I found more logs in vpxa.log ( the esxi hosts are using UTC time zone,  so 
I was looking at wrong time periods earlier).  I have uploaded more logs into 
pastebin.

From these log entries,  it appears that when copying template to VM,  it 
tried to open destination VMDK file and got error file not found.  

In case that the CloudStack attempted to create a systemVM,  the 
destination VMDK file path it is looking for is 
"//.vmdk",  see uploaded log at 
https://pastebin.com/aFysZkTy

In case when I manually created new VM from a (different) template in 
vCenter UI,   the destination VMDK file path it is looking for is 
"//.vmdk", see uploaded log at 
https://pastebin.com/yHcsD8xB

So, I am confused as to how the path for destination VMDK was determined 
and by CloudStack or VMware, how did I end up with this?

Yiping
    

    On 6/5/19, 12:32 PM, "Sergey Levitskiy"  wrote:

Some operations log get transferred to vCenter log vpxd.log. It is not 
straightforward to trace I but Vmware will be able to help should you open case 
with them. 


On 6/5/19, 11:39 AM, "Yiping Zhang"  wrote:

Hi, Sergey:

During the time period when I had problem cloning template,  there 
are only a few unique entries in vmkernel.log, and they were repeated 
hundreds/thousands of times by all the cpu cores:

2019-06-02T16:47:00.633Z cpu9:8491061)FSS: 6751: Failed to open 
file 'hpilo-d0ccb15'; Requested flags 0x5, world: 8491061 [ams-ahs], (Existing 
flags 0x5, world: 8491029 [ams-main]): Busy
2019-06-02T16:47:49.320Z cpu1:66415)nhpsa: 
hpsa_vmkScsiCmdDone:6384: Sense data: error code: 0x70, key: 0x5, info:00 00 00 
00 , cmdInfo:00 00 00 00 , CmdSN: 0xd5c, worldId: 0x818e8e, Cmd: 0x85, ASC: 
0x20, ASCQ: 0x0
2019-06-02T16:47:49.320Z cpu1:66415)ScsiDeviceIO: 2948: 
Cmd(0x43954115be40) 0x85, CmdSN 0xd5c from world 8490638 to dev 
"naa.600508b1001c6d77d7dd6a0cc0953df1" failed H:0x0 D:0x2 P:0x0 Valid sense 
data: 0x5 0x20 0x0.

The device " naa.600508b1001c6d77d7dd6a0cc0953df1" is the local 
disk on this host.

    Yiping


On 6/5/19, 11:15 AM, "Sergey Levitskiy"  wrote:

This must be specific to that environment.  For a full clone 
mode ACS simply calls cloneVMTask of vSphere API so basically until cloning of 
that template succeeds when attmepted in vSphere client  it would keep failing 
in ACS. Can you post vmkernel.log from your ESX host esx-0001-a-001?


On 6/5/19, 8:47 AM, "Yiping Zhang"  
wrote:

Well,  I can always reproduce it in this particular vSphere 
set up,  but in a different ACS+vSphere environment,  I don't see this problem.

Yiping

On 6/5/19, 1:00 AM, "Andrija Panic" 
 wrote:

Yiping,

if you are sure you can reproduce the issue, it would 
be good to raise a
GitHub issue and provide as much detail as possible.

Andrija

On Wed, 5 Jun 2019 at 05:29, Yiping Zhang 

wrote:

> Hi, Sergey:
>
> Thanks for the tip. After setting 
vmware.create.full.clone=false,  I was
> able

Re: Can't start systemVM in a new advanced zone deployment

[Sergey Levitskiy]

Some operations log get transferred to vCenter log vpxd.log. It is not 
straightforward to trace I but Vmware will be able to help should you open case 
with them. 


On 6/5/19, 11:39 AM, "Yiping Zhang"  wrote:

Hi, Sergey:

During the time period when I had problem cloning template,  there are only 
a few unique entries in vmkernel.log, and they were repeated hundreds/thousands 
of times by all the cpu cores:

2019-06-02T16:47:00.633Z cpu9:8491061)FSS: 6751: Failed to open file 
'hpilo-d0ccb15'; Requested flags 0x5, world: 8491061 [ams-ahs], (Existing flags 
0x5, world: 8491029 [ams-main]): Busy
2019-06-02T16:47:49.320Z cpu1:66415)nhpsa: hpsa_vmkScsiCmdDone:6384: Sense 
data: error code: 0x70, key: 0x5, info:00 00 00 00 , cmdInfo:00 00 00 00 , 
CmdSN: 0xd5c, worldId: 0x818e8e, Cmd: 0x85, ASC: 0x20, ASCQ: 0x0
2019-06-02T16:47:49.320Z cpu1:66415)ScsiDeviceIO: 2948: Cmd(0x43954115be40) 
0x85, CmdSN 0xd5c from world 8490638 to dev 
"naa.600508b1001c6d77d7dd6a0cc0953df1" failed H:0x0 D:0x2 P:0x0 Valid sense 
data: 0x5 0x20 0x0.

The device " naa.600508b1001c6d77d7dd6a0cc0953df1" is the local disk on 
this host.

Yiping

    
    On 6/5/19, 11:15 AM, "Sergey Levitskiy"  wrote:

This must be specific to that environment.  For a full clone mode ACS 
simply calls cloneVMTask of vSphere API so basically until cloning of that 
template succeeds when attmepted in vSphere client  it would keep failing in 
ACS. Can you post vmkernel.log from your ESX host esx-0001-a-001?


On 6/5/19, 8:47 AM, "Yiping Zhang"  wrote:

Well,  I can always reproduce it in this particular vSphere set up, 
 but in a different ACS+vSphere environment,  I don't see this problem.

Yiping

On 6/5/19, 1:00 AM, "Andrija Panic"  wrote:

Yiping,

if you are sure you can reproduce the issue, it would be good 
to raise a
GitHub issue and provide as much detail as possible.

Andrija

On Wed, 5 Jun 2019 at 05:29, Yiping Zhang 

wrote:

> Hi, Sergey:
>
> Thanks for the tip. After setting 
vmware.create.full.clone=false,  I was
> able to create and start system VM instances.However,  I 
feel that the
> underlying problem still exists, and I am just working around 
it instead of
> fixing it,  because in my lab CloudStack instance with the 
same version of
> ACS and vSphere,  I still have vmware.create.full.clone=true 
and all is
> working as expected.
>
> I did some reading on VMware docs regarding full clone vs. 
linked clone.
> It seems that the best practice is to use full clone for 
production,
> especially if there are high rates of changes to the disks.  
So
> eventually,  I need to understand and fix the root cause for 
this issue.
> At least for now,  I am over this hurdle and I can move on.
>
> Thanks again,
    >
> Yiping
>
> On 6/4/19, 11:13 AM, "Sergey Levitskiy"  
wrote:
>
> Everything looks good and consistent including all 
references in VMDK
> and its snapshot. I would try these 2 routes:
> 1. Figure out what vSphere error actually means from 
vmkernel log of
> ESX when ACS tries to clone the template. If the same error 
happens while
> doing it outside of ACS then a support case with VMware can 
be an option
> 2. Try using link clones. This can be done by this global 
setting and
> restarting management server
> vmware.create.full.clonefalse
>
>
> On 6/4/19, 9:57 AM, "Yiping Zhang" 
 wrote:
>
> Hi, Sergey:
>
> Thanks for the help. By now, I have dropped and 
recreated DB,
> re-deployed this zone multiple times, blow away primary and 
secondary
> storage (including all contents on them) , or just delete 
template itself
> from primary storage, multiple times.  Every time I ended up 
with the same
> error at the same place.
>
> The full management server log,  from the point I 
seeded the
 

Re: Can't start systemVM in a new advanced zone deployment

[Sergey Levitskiy]

This must be specific to that environment.  For a full clone mode ACS simply 
calls cloneVMTask of vSphere API so basically until cloning of that template 
succeeds when attmepted in vSphere client  it would keep failing in ACS. Can 
you post vmkernel.log from your ESX host esx-0001-a-001?


On 6/5/19, 8:47 AM, "Yiping Zhang"  wrote:

Well,  I can always reproduce it in this particular vSphere set up,  but in 
a different ACS+vSphere environment,  I don't see this problem.

Yiping

On 6/5/19, 1:00 AM, "Andrija Panic"  wrote:

Yiping,

if you are sure you can reproduce the issue, it would be good to raise a
GitHub issue and provide as much detail as possible.

Andrija

On Wed, 5 Jun 2019 at 05:29, Yiping Zhang 
wrote:

> Hi, Sergey:
>
> Thanks for the tip. After setting vmware.create.full.clone=false,  I 
was
> able to create and start system VM instances.However,  I feel 
that the
> underlying problem still exists, and I am just working around it 
instead of
> fixing it,  because in my lab CloudStack instance with the same 
version of
> ACS and vSphere,  I still have vmware.create.full.clone=true and all 
is
> working as expected.
>
> I did some reading on VMware docs regarding full clone vs. linked 
clone.
> It seems that the best practice is to use full clone for production,
> especially if there are high rates of changes to the disks.  So
> eventually,  I need to understand and fix the root cause for this 
issue.
> At least for now,  I am over this hurdle and I can move on.
>
> Thanks again,
    >
> Yiping
>
> On 6/4/19, 11:13 AM, "Sergey Levitskiy"  wrote:
>
> Everything looks good and consistent including all references in 
VMDK
> and its snapshot. I would try these 2 routes:
> 1. Figure out what vSphere error actually means from vmkernel log 
of
> ESX when ACS tries to clone the template. If the same error happens 
while
> doing it outside of ACS then a support case with VMware can be an 
option
> 2. Try using link clones. This can be done by this global setting 
and
> restarting management server
> vmware.create.full.clonefalse
>
>
> On 6/4/19, 9:57 AM, "Yiping Zhang"  
wrote:
>
> Hi, Sergey:
>
> Thanks for the help. By now, I have dropped and recreated DB,
> re-deployed this zone multiple times, blow away primary and secondary
> storage (including all contents on them) , or just delete template 
itself
> from primary storage, multiple times.  Every time I ended up with the 
same
> error at the same place.
>
> The full management server log,  from the point I seeded the
> systemvmtemplate for vmware, to deploying a new advanced zone and 
enable
> the zone to let CS to create system VM's and finally disable the zone 
to
> stop infinite loop of trying to recreate failed system VM's,  are 
posted
> at pastebin:
>
>
> 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpastebin.com%2Fc05wiQ3Rdata=02%7C01%7Cyipzhang%40adobe.com%7C4c2a536c47ed44e4167908d6e98be65f%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636953184420635130sdata=GFdI2M4NdVoB848FAtYgg9zzfXAjBFYm%2BwafqMTnxWY%3Dreserved=0
>
> Here are the content of relevant files for the template on 
primary
> storage:
>
> 1) /vmfsvolumes:
>
> ls -l /vmfs/volumes/
> total 2052
> drwxr-xr-x1 root root 8 Jan  1  1970
> 414f6a73-87cd6dac-9585-133ddd409762
> lrwxr-xr-x1 root root17 Jun  4 16:37
> 42054b8459633172be231d72a52d59d4 -> afc5e946-03bfe3c2  <== 
this is
> the NFS datastore for primary storage
> drwxr-xr-x1 root root 8 Jan  1  1970
> 5cd4b46b-fa4fcff0-d2a1-00215a9b31c0
> drwxr-xr-t1 root root  1400 Jun  3 22:50
> 5cd4b471-c2318b91-8fb2-00215a9b31c0
> drwxr-xr-x1 root root 8 Jan  1  1970
> 5cd4b471-da49a95b-bdb6-00215a9b31c0
> drwxr-xr-x4 root root  4096 Jun  3 23:38
> afc5e946-03bfe3c2
> drwxr-xr-x1 root root 8

Re: Can't start systemVM in a new advanced zone deployment

[Sergey Levitskiy]

b6fcf3fa6301aadcc2b168f3f999a.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
floppy0.present = "FALSE"
memSize = "256"
scsi0.virtualDev = "lsilogic"
scsi0.present = "TRUE"
ide0:0.startConnected = "FALSE"
ide0:0.deviceType = "atapi-cdrom"
ide0:0.fileName = "CD/DVD drive 0"
ide0:0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "533b6fcf3fa6301aadcc2b168f3f999a-01.vmdk"
scsi0:0.present = "TRUE"
displayName = "533b6fcf3fa6301aadcc2b168f3f999a"
annotation = "systemvmtemplate-4.11.2.0-vmware"
guestOS = "otherlinux-64"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
uuid.bios = "42 02 f1 40 33 e8 de e5-1a c5 93 2a c9 12 47 61"
vc.uuid = "50 02 5b d9 e9 c9 77 86-28 3e 84 00 22 2b eb d3"
firmware = "bios"
migrate.hostLog = "533b6fcf3fa6301aadcc2b168f3f999a-7d5d73de.hlog"


6) *.vmsd file content:

cat 
/vmfs/volumes/42054b8459633172be231d72a52d59d4/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a.vmsd
.encoding = "UTF-8"
snapshot.lastUID = "1"
snapshot.current = "1"
snapshot0.uid = "1"
snapshot0.filename = "533b6fcf3fa6301aadcc2b168f3f999a-Snapshot1.vmsn"
snapshot0.displayName = "cloud.template.base"
snapshot0.description = "Base snapshot"
snapshot0.createTimeHigh = "363123"
snapshot0.createTimeLow = "-679076964"
snapshot0.numDisks = "1"
snapshot0.disk0.fileName = "533b6fcf3fa6301aadcc2b168f3f999a.vmdk"
snapshot0.disk0.node = "scsi0:0"
snapshot.numSnapshots = "1"

7) *-Snapshot1.vmsn content:

cat 
/vmfs/volumes/42054b8459633172be231d72a52d59d4/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-Snapshot1.vmsn
 
ҾSnapshot\?%?cfgFilet%t%.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "8"
nvram = "533b6fcf3fa6301aadcc2b168f3f999a.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
floppy0.present = "FALSE"
memSize = "256"
scsi0.virtualDev = "lsilogic"
scsi0.present = "TRUE"
ide0:0.startConnected = "FALSE"
ide0:0.deviceType = "atapi-cdrom"
ide0:0.fileName = "CD/DVD drive 0"
ide0:0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "533b6fcf3fa6301aadcc2b168f3f999a.vmdk"
scsi0:0.present = "TRUE"
displayName = "533b6fcf3fa6301aadcc2b168f3f999a"
annotation = "systemvmtemplate-4.11.2.0-vmware"
guestOS = "otherlinux-64"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
uuid.bios = "42 02 f1 40 33 e8 de e5-1a c5 93 2a c9 12 47 61"
vc.uuid = "50 02 5b d9 e9 c9 77 86-28 3e 84 00 22 2b eb d3"
firmware = "bios"
migrate.hostLog = "533b6fcf3fa6301aadcc2b168f3f999a-7d5d73de.hlog"
    



That's all the data on the template VMDK.

Much appreciate your time!

  

Re: Can't start systemVM in a new advanced zone deployment

[Sergey Levitskiy]

Have you tried deleting template from PS and let ACS to recopy it again? If the 
issue is reproducible we can try to look what is wrong with VMDK. Please post 
content of 533b6fcf3fa6301aadcc2b168f3f999a.vmdk , 
533b6fcf3fa6301aadcc2b168f3f999a-01.vmdk and 
533b6fcf3fa6301aadcc2b168f3f999a.vmx (their equitant after ACS finishes copying 
template). Also from one of your ESX hosts output of this
ls -al /vmfs/volumes
ls -al /vmfs/volumes/*/533b6fcf3fa6301aadcc2b168f3f999a (their equitant after 
ACS finishes copying template)

 Can you also post management server log starting from the point you unregister 
and delete template from the vCenter.

On 6/4/19, 8:37 AM, "Yiping Zhang"  wrote:

I have manually imported the OVA to vCenter and successfully cloned a VM 
instance with it, on the same NFS datastore.


On 6/4/19, 8:25 AM, "Sergey Levitskiy"  wrote:

I would suspect the template is corrupted on the secondary storage. You 
can try disabling/enabling link clone feature and see if it works the other 
way. 
vmware.create.full.clonefalse

Also systemVM template might have been generated on a newer version of 
vSphere and not compatible with ESXi 6.5. What you can do to validate this is 
to manually deploy OVA that is in Secondary storage and try to spin up VM from 
it directly in vCenter.



On 6/3/19, 5:41 PM, "Yiping Zhang"  wrote:

Hi, list:

I am struggling with deploying a new advanced zone using ACS 
4.11.2.0 + vSphere 6.5 + NetApp volumes for primary and secondary storage 
devices. The initial setup of CS management server, seeding of systemVM 
template, and advanced zone deployment all went smoothly.

Once I enabled the zone in web UI and the systemVM template gets 
copied/staged on to primary storage device. But subsequent VM creations from 
this template would fail with errors:


2019-06-03 18:38:15,764 INFO  [c.c.h.v.m.HostMO] 
(DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) VM 533b6fcf3fa6301aadcc2b168f3f999a not found in host cache

2019-06-03 18:38:17,017 INFO  [c.c.h.v.r.VmwareResource] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) VmwareStorageProcessor and VmwareStorageSubsystemCommandHandler 
successfully reconfigured

2019-06-03 18:38:17,128 INFO  [c.c.s.r.VmwareStorageProcessor] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) creating full clone from template

2019-06-03 18:38:17,657 INFO  [c.c.h.v.u.VmwareHelper] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) [ignored]failed toi get message for exception: Error caused by 
file 
/vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-01.vmdk

2019-06-03 18:38:17,658 ERROR [c.c.s.r.VmwareStorageProcessor] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) clone volume from base image failed due to Exception: 
java.lang.RuntimeException

Message: Error caused by file 
/vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-01.vmdk



If I try to create “new VM from template” 
(533b6fcf3fa6301aadcc2b168f3f999a) on vCenter UI manually,  I will receive 
exactly the same error message. The name of the VMDK file in the error message 
is a snapshot of the base disk image, but it is not part of the original 
template OVA on the secondary storage.  So, in the process of copying the 
template from secondary to primary storage, a snapshot got created and the disk 
became corrupted/unusable.

Much later in the log file,  there is another error message “failed 
to fetch any free public IP address” (for ssvm, I think).  I don’t know if 
these two errors are related or if one is the root cause for the other error.

The full management server log is uploaded as 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpastebin.com%2Fc05wiQ3Rdata=02%7C01%7Cyipzhang%40adobe.com%7C1541cabf6c9b42873e0708d6e900da78%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636952587209705577sdata=qDALhWV4NoLWyidmyz6f1MUSERFWb9XN%2B7GtaAnExaU%3Dreserved=0

Any help or insight on what went wrong here are much appreciated.

Thanks

Yiping

Re: Can't start systemVM in a new advanced zone deployment

[Sergey Levitskiy]

I would suspect the template is corrupted on the secondary storage. You can try 
disabling/enabling link clone feature and see if it works the other way. 
vmware.create.full.clonefalse

Also systemVM template might have been generated on a newer version of vSphere 
and not compatible with ESXi 6.5. What you can do to validate this is to 
manually deploy OVA that is in Secondary storage and try to spin up VM from it 
directly in vCenter.



On 6/3/19, 5:41 PM, "Yiping Zhang"  wrote:

Hi, list:

I am struggling with deploying a new advanced zone using ACS 4.11.2.0 + 
vSphere 6.5 + NetApp volumes for primary and secondary storage devices. The 
initial setup of CS management server, seeding of systemVM template, and 
advanced zone deployment all went smoothly.

Once I enabled the zone in web UI and the systemVM template gets 
copied/staged on to primary storage device. But subsequent VM creations from 
this template would fail with errors:


2019-06-03 18:38:15,764 INFO  [c.c.h.v.m.HostMO] 
(DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) VM 533b6fcf3fa6301aadcc2b168f3f999a not found in host cache

2019-06-03 18:38:17,017 INFO  [c.c.h.v.r.VmwareResource] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) VmwareStorageProcessor and VmwareStorageSubsystemCommandHandler 
successfully reconfigured

2019-06-03 18:38:17,128 INFO  [c.c.s.r.VmwareStorageProcessor] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) creating full clone from template

2019-06-03 18:38:17,657 INFO  [c.c.h.v.u.VmwareHelper] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) [ignored]failed toi get message for exception: Error caused by 
file 
/vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-01.vmdk

2019-06-03 18:38:17,658 ERROR [c.c.s.r.VmwareStorageProcessor] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) clone volume from base image failed due to Exception: 
java.lang.RuntimeException

Message: Error caused by file 
/vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-01.vmdk



If I try to create “new VM from template” 
(533b6fcf3fa6301aadcc2b168f3f999a) on vCenter UI manually,  I will receive 
exactly the same error message. The name of the VMDK file in the error message 
is a snapshot of the base disk image, but it is not part of the original 
template OVA on the secondary storage.  So, in the process of copying the 
template from secondary to primary storage, a snapshot got created and the disk 
became corrupted/unusable.

Much later in the log file,  there is another error message “failed to 
fetch any free public IP address” (for ssvm, I think).  I don’t know if these 
two errors are related or if one is the root cause for the other error.

The full management server log is uploaded as https://pastebin.com/c05wiQ3R

Any help or insight on what went wrong here are much appreciated.

Thanks

Yiping

Re: Problem with ACS 4.11.2 and vCenter server

[Sergey Levitskiy]

 and net packets. 
vCenter fails to add the host because it can't find it and then nothing more 
happens, no more packets between CS and the vCenter...


The only errors I can find in the logs looks like this :
INFO  [c.c.u.e.CSExceptionErrorCode] (StatsCollector-4:ctx-b2532a68) 
(logid:98b87cb6) Could not find exception: 
com.cloud.exception.OperationTimedoutException in error code list for exceptions
    


Florent

- Mail original -
> De: "Sergey Levitskiy" 
> À: "users" 
> Envoyé: Mercredi 17 Avril 2019 22:45:23
> Objet: Re: Problem with ACS 4.11.2 and vCenter server

> It doesn’t look VMware context is getting initialized. This something
> fundamental e.g. API version mismatch or missing vmware API Jar file.
> You can try re-adding exising vmware DC and cluster using fake cluster 
name to
> force connectivity probing during the addition. If you do please post  
the log.
> 
>On 4/17/19, 7:48 AM, "Florent Paillot"  wrote:
> 
>The only lines I can find in the log are linked to "*VmwareContext*".
>
>I tried to activate the TRACE level log for 
com.cloud.hypervisor.vmware but it
>didn't help.
>Maybe my conf in log4j is wrong :
>
> 
> 
>   
>
>
>Thanks for your help.
>
>Florent
>
>- Mail original -
>> De: "Suresh Kumar Anaparti" 
>> À: "users" 
>> Envoyé: Mardi 16 Avril 2019 19:25:50
>> Objet: Re: Problem with ACS 4.11.2 and vCenter server
>
>> Hi Florent,
>> 
>> As soon as CS is started, it will try to connect to the ESXi host 
with the
>> vCenter details in the zone (Check details in tables 
vmware_data_center &
>> host_details). Grep the below text in the log to see the connection
>> progress or errors:
>> - *Loading directly connected host*
>> - *initialize VmwareContext*
>> - *Unable to connect to vSphere server*
>> - *can not load directly connected host*
>> 
>> CS keeps trying to connect to the host, by default every 90 secs 
(agent
>> scan interval).
>> 
>> Regards,
>> Suresh
>> 
>> On Tue, Apr 16, 2019 at 8:57 PM Sergey Levitskiy 

>> wrote:
>> 
>>> Please post a full management-server.log without parsing.
>>>
>>> On 4/16/19, 7:49 AM, "Florent Paillot"  
wrote:
>>>
>>> I did this on purpose to limit the size of the log.
>>>
>>> I let the controller run for more than 30mins. Can't find any 
revelant
>>> error... :(
>>> I put some grep results here :
>>> https://pad.inria.fr/p/np_IvcpeIGM4BncZEM1
>>>
>>> On the production controller (4.9.3), if we have connectivity 
problem
>>> with the vCenter, I see log with the "com.cloud.hypervisor.vmware.*"
>>> context.
>>> Is there something to change in the log4j config to see this 
logs in
>>> 4.11 ?
>>>
>>> The vim25.jar is the right one, after unziping
>>> "/usr/share/cloudstack-management/lib/cloudstack-4.11.2.0.jar" in
>>> com/vmware/vim25/vim.wsdl, I see " >> soapAction="urn:vim25/6.7" style="document" />".
>>>
>>>
>>> Florent
>>>
>>> - Mail original -
>>> > De: "Suresh Kumar Anaparti" 
>>> > À: "users" 
>>> > Envoyé: Mardi 16 Avril 2019 14:22:28
>>> > Objet: Re: Problem with ACS 4.11.2 and vCenter server
>>>
>>> > Florent, The log doesn't have any CS to vCenter connectivity
>>> details. The
>>> > CS management server was started and stopped in 2 mins. It 
needs
>>> some time
>>> > to load modules and connect to the hosts and storages. Wait 
for some
>>> more
>>> > time for the connectivity details in the log.
>>> >
>>> > You can install vim25.jar for 6.7 version and give a try.
   

Re: Problem with ACS 4.11.2 and vCenter server

[Sergey Levitskiy]

It doesn’t look VMware context is getting initialized. This something 
fundamental e.g. API version mismatch or missing vmware API Jar file.
You can try re-adding exising vmware DC and cluster using fake cluster name to 
force connectivity probing during the addition. If you do please post  the log.

On 4/17/19, 7:48 AM, "Florent Paillot"  wrote:

The only lines I can find in the log are linked to "*VmwareContext*".

I tried to activate the TRACE level log for com.cloud.hypervisor.vmware but 
it didn't help.
Maybe my conf in log4j is wrong :

 
 
   


Thanks for your help.

Florent

- Mail original -
> De: "Suresh Kumar Anaparti" 
> À: "users" 
> Envoyé: Mardi 16 Avril 2019 19:25:50
> Objet: Re: Problem with ACS 4.11.2 and vCenter server

> Hi Florent,
> 
> As soon as CS is started, it will try to connect to the ESXi host with the
> vCenter details in the zone (Check details in tables vmware_data_center &
> host_details). Grep the below text in the log to see the connection
> progress or errors:
> - *Loading directly connected host*
> - *initialize VmwareContext*
> - *Unable to connect to vSphere server*
> - *can not load directly connected host*
> 
> CS keeps trying to connect to the host, by default every 90 secs (agent
> scan interval).
    > 
> Regards,
> Suresh
> 
> On Tue, Apr 16, 2019 at 8:57 PM Sergey Levitskiy 
> wrote:
> 
>> Please post a full management-server.log without parsing.
>>
>> On 4/16/19, 7:49 AM, "Florent Paillot"  wrote:
>>
>> I did this on purpose to limit the size of the log.
>>
>> I let the controller run for more than 30mins. Can't find any 
revelant
>> error... :(
>> I put some grep results here :
>> https://pad.inria.fr/p/np_IvcpeIGM4BncZEM1
>>
>> On the production controller (4.9.3), if we have connectivity problem
>> with the vCenter, I see log with the "com.cloud.hypervisor.vmware.*"
>> context.
>> Is there something to change in the log4j config to see this logs in
>> 4.11 ?
>>
>> The vim25.jar is the right one, after unziping
>> "/usr/share/cloudstack-management/lib/cloudstack-4.11.2.0.jar" in
>> com/vmware/vim25/vim.wsdl, I see " > soapAction="urn:vim25/6.7" style="document" />".
>>
>>
>> Florent
>>
>> - Mail original -
>> > De: "Suresh Kumar Anaparti" 
>> > À: "users" 
>> > Envoyé: Mardi 16 Avril 2019 14:22:28
>> > Objet: Re: Problem with ACS 4.11.2 and vCenter server
>>
>> > Florent, The log doesn't have any CS to vCenter connectivity
>> details. The
>> > CS management server was started and stopped in 2 mins. It needs
>> some time
>> > to load modules and connect to the hosts and storages. Wait for 
some
>> more
>> > time for the connectivity details in the log.
>> >
>> > You can install vim25.jar for 6.7 version and give a try.
>> >
>> > - Suresh
>> >
>> > On Tue, Apr 16, 2019 at 5:34 PM Florent Paillot <
>> florent.pail...@inria.fr>
>> > wrote:
>> >
>> >> Here is the log : https://pad.inria.fr/p/np_IvcpeIGM4BncZEM1
>> >> It's a complete 3 minutes start/run/stop.
>> >>
>> >> Thanks for the info about "cs.vmware.api.version*". For 4.11.2, it
>> seems
>> >> that vSphere 6.7 is the target :
>> >> pom.xml:6.7
>> >>
>> >>
>> >> Florent
>> >>
>> >> - Mail original -
>> >> > De: "Suresh Kumar Anaparti" 
>> >> > À: "users" 
>> >> > Envoyé: Mardi 16 Avril 2019 13:05:20
>> >> > Objet: Re: Problem with ACS 4.11.2 and vCenter server
>> >>
>> >> > Hi Florent,
>> >> >
>> >> > Log is missing. I think attachments are ignored. Either provide 
a
>> link to
>> >> > access the log or paste the logs in email (few log

Re: CloudStack and SDN

[Sergey Levitskiy]

Nicira NVP/NSX is very outdated and doesn’t support latest NSX.

On 4/17/19, 1:22 AM, "Andrija Panic"  wrote:

Simplest one that actually works (call it SDN if you like) - VXLAN.

A few others with various implementation states and all hypervisor
dependent:
Nicira NVP / Vmware NSX
Midonet
Various tunneling protocols


On Wed, 17 Apr 2019 at 09:46, Haijiao <18602198...@163.com> wrote:

>
>
> Noted Nuage's decision on ceasing SDN plugin for CloudStack. Just
> wondering is there any other SDN solutoin can be integrated with 
CloudStack
> ?
>
>
> Thanks !
>
>
> Regards,
>
>

-- 

Andrija Panić

Re: Problem with ACS 4.11.2 and vCenter server

[Sergey Levitskiy]

It doesn’t look you have properly added VMware cluster & respective storage. 
Can you try again and post management server log which should have entries 
similar to below after you added cluster in UI. First you need to add VMware DC 
and then cluster. 
BTW you also have issues with one of your KVM hosts id=65. 


2017-10-31 12:17:37,116 DEBUG [c.c.a.ApiServlet]
(catalina-exec-22:ctx-f2dc50b2) (logid:799ac7c1) ===START===
172.17.101.246 -- POST  command=addVmwareDc=json
2017-10-31 12:17:37,132 INFO  [c.c.h.v.m.VmwareManagerImpl]
(catalina-exec-22:ctx-f2dc50b2 ctx-a324e697) (logid:799ac7c1) Ignoring API
call addVmwareDc, because VMware DC 172.17.101.246/WeDatacenter is already
associated with specified zone with id 1
2017-10-31 12:17:37,133 DEBUG [c.c.a.ApiServlet]
(catalina-exec-22:ctx-f2dc50b2 ctx-a324e697) (logid:799ac7c1) ===END===
172.17.101.246 -- POST  command=addVmwareDc=json
2017-10-31 12:17:37,155 DEBUG [c.c.a.ApiServlet]
(catalina-exec-20:ctx-a26bd8ef) (logid:272077ae) ===START===
172.17.101.246 -- GET
command=addCluster=f538e730-7431-4c65-bf7d-eccc154e8b48=VMware=ExternalManaged=6262dca1-59df-4a32-b80a-52435f2ec4f7=administrator%40vsphere.local=http%3A%2F%2F172.17.101.246%2FWeDatacenter%2FWeCluster=172.17.101.246%2FWeDatacenter%2FWeCluster=json&_=1509432458636
2017-10-31 12:17:37,225 INFO  [c.c.h.v.VmwareServerDiscoverer]
(catalina-exec-20:ctx-a26bd8ef ctx-06a2c997) (logid:272077ae) Discover
host. dc: 1, pod: 1, cluster: 7, uri host: 172.17.101.246
2017-10-31 12:17:37,313 INFO  [c.c.h.v.VmwareServerDiscoverer]
(catalina-exec-20:ctx-a26bd8ef ctx-06a2c997) (logid:272077ae) Detected
private network label : DSwitch,,vmwaresvs
2017-10-31 12:17:37,313 DEBUG [c.c.h.v.r.VmwareContextFactory]
(catalina-exec-20:ctx-a26bd8ef ctx-06a2c997) (logid:272077ae) initialize
VmwareContext. url: https://172.17.101.246/sdk/vimService, username:
administrator@vsphere.local, password: W*
2017-10-31 12:17:37,535 INFO  [c.c.h.v.u.VmwareContext]
(catalina-exec-20:ctx-a26bd8ef ctx-06a2c997) (logid:272077ae) New
VmwareContext object, current outstanding count: 1
2017-10-31 12:17:37,662 INFO  [c.c.h.v.m.VmwareManagerImpl]
(catalina-exec-20:ctx-a26bd8ef ctx-06a2c997) (logid:272077ae) Preparing
network on host com.cloud.hypervisor.vmware.util.VmwareContext@53bf4bc2 for
DSwitch,,vmwaresvs 






On 4/16/19, 11:48 AM, "Florent Paillot"  wrote:

This is the full log :

https://filesender.renater.fr/?s=download=6be77301-3893-46fa-912c-3912086cae5f

Florent

- Mail original -
> De: "Sergey Levitskiy" 
> À: "users" 
> Envoyé: Mardi 16 Avril 2019 17:27:37
> Objet: Re: Problem with ACS 4.11.2 and vCenter server

> Please post a full management-server.log without parsing.
> 
>On 4/16/19, 7:49 AM, "Florent Paillot"  wrote:
> 
>I did this on purpose to limit the size of the log.
>
>I let the controller run for more than 30mins. Can't find any revelant 
error...
>:(
>I put some grep results here : 
https://pad.inria.fr/p/np_IvcpeIGM4BncZEM1
>
>On the production controller (4.9.3), if we have connectivity problem 
with the
>vCenter, I see log with the "com.cloud.hypervisor.vmware.*" context.
>Is there something to change in the log4j config to see this logs in 
4.11 ?
>
>The vim25.jar is the right one, after unziping
>"/usr/share/cloudstack-management/lib/cloudstack-4.11.2.0.jar" in
>com/vmware/vim25/vim.wsdl, I see " style="document" />".
>
>
>Florent
>
>- Mail original -
>> De: "Suresh Kumar Anaparti" 
>> À: "users" 
>> Envoyé: Mardi 16 Avril 2019 14:22:28
>> Objet: Re: Problem with ACS 4.11.2 and vCenter server
>
>> Florent, The log doesn't have any CS to vCenter connectivity 
details. The
>> CS management server was started and stopped in 2 mins. It needs 
some time
>> to load modules and connect to the hosts and storages. Wait for some 
more
>> time for the connectivity details in the log.
>> 
>> You can install vim25.jar for 6.7 version and give a try.
>> 
>> - Suresh
>> 
>> On Tue, Apr 16, 2019 at 5:34 PM Florent Paillot 

>> wrote:
>> 
>>> Here is the log : https://pad.inria.fr/p/np_IvcpeIGM4BncZEM1
>>> It's a complete 3 minutes start/run/stop.
>>>
>>> Thanks for the info about "cs.vmware.api.version*". For 4.11.2, it 
seems
>>> that vSphere 6.7 is the target :
   

Re: Problem with ACS 4.11.2 and vCenter server

[Sergey Levitskiy]

Please post a full management-server.log without parsing.

On 4/16/19, 7:49 AM, "Florent Paillot"  wrote:

I did this on purpose to limit the size of the log.

I let the controller run for more than 30mins. Can't find any revelant 
error... :(
I put some grep results here : https://pad.inria.fr/p/np_IvcpeIGM4BncZEM1

On the production controller (4.9.3), if we have connectivity problem with 
the vCenter, I see log with the "com.cloud.hypervisor.vmware.*" context.
Is there something to change in the log4j config to see this logs in 4.11 ?

The vim25.jar is the right one, after unziping 
"/usr/share/cloudstack-management/lib/cloudstack-4.11.2.0.jar" in 
com/vmware/vim25/vim.wsdl, I see " ".


Florent

- Mail original -
> De: "Suresh Kumar Anaparti" 
> À: "users" 
> Envoyé: Mardi 16 Avril 2019 14:22:28
> Objet: Re: Problem with ACS 4.11.2 and vCenter server

> Florent, The log doesn't have any CS to vCenter connectivity details. The
> CS management server was started and stopped in 2 mins. It needs some time
> to load modules and connect to the hosts and storages. Wait for some more
> time for the connectivity details in the log.
> 
> You can install vim25.jar for 6.7 version and give a try.
> 
> - Suresh
> 
> On Tue, Apr 16, 2019 at 5:34 PM Florent Paillot 
> wrote:
> 
>> Here is the log : https://pad.inria.fr/p/np_IvcpeIGM4BncZEM1
>> It's a complete 3 minutes start/run/stop.
>>
>> Thanks for the info about "cs.vmware.api.version*". For 4.11.2, it seems
>> that vSphere 6.7 is the target :
>> pom.xml:6.7
>>
>>
>> Florent
>>
>> - Mail original -
>> > De: "Suresh Kumar Anaparti" 
>> > À: "users" 
>> > Envoyé: Mardi 16 Avril 2019 13:05:20
>> > Objet: Re: Problem with ACS 4.11.2 and vCenter server
>>
>> > Hi Florent,
>> >
>> > Log is missing. I think attachments are ignored. Either provide a link 
to
>> > access the log or paste the logs in email (few logs of the issue).
>> >
>> > Irrespective of the vCenter version (hard coded in the build param
>> > *cs.vmware.api.version*), CS would try to connect to the vCenter 
server.
>> >
>> > - Suresh
>> >
>> >
>> > On Tue, Apr 16, 2019 at 2:44 PM Florent Paillot <
>> florent.pail...@inria.fr>
>> > wrote:
>> >
>> >> Thanks for the answers,
>> >>
>> >> Paul : I know that the vCenter 6.7 is not officialy supported but my
>> >> colleague reported that it worked with 4.9.3 .
>> >> When you say "cloudstack won't recognise it", is it hard coded ? If 
the
>> >> the vCenter auth API doesn't change between 6.5 and 6.7 it should work
>> or
>> >> at least try to connect to the vCenter.
>> >>
>> >> I'm agree with you Suresh, even if CS can't communicate with the 
vCenter
>> >> we should see something in the logs or in the packet capture.
>> >>
>> >> I am able to ping and telnet from the CS controller to the vCenter on
>> port
>> >> 443 (and I see the packet with tcpdump).
>> >>
>> >> The log is in attachement.
>> >>
>> >> Florent
>> >>
>> >>
>> >> - Mail original -
>> >> > De: "Suresh Kumar Anaparti" 
>> >> > À: "users" 
>> >> > Envoyé: Lundi 15 Avril 2019 17:54:00
>> >> > Objet: Re: Problem with ACS 4.11.2 and vCenter server
>> >>
>> >> > Hi Florent, What was the CS behavior & vCenter/ESXi hosts 
connectivity
>> >> > status before upgrade to 4.11.2 from 4.9.3? Have you upgraded 
vCenter
>> as
>> >> > well? Are you able to ping vCenter from CS management server?
>> >> >
>> >> > Even vCenter 6.7 is not recognised, CS would try to connect with the
>> >> > details specified, which is logged. Have you enabled debug logs?
>> >> >
>> >> > - Suresh
>> >> >
>> >> > On Mon, Apr 15, 2019 at 7:38 PM Paul Angus 

>> >> wrote:
>> >> >
>> >> >> Ah. vCenter 6.7 isn't supported yet, cloudstack won't recognise it.
>> >> >>
>> >> >> paul.an...@shapeblue.com
>> >> >> www.shapeblue.com
>> >> >> Amadeus House, Floral Street, London  WC2E 9DPUK
>> >> >> @shapeblue
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> -Original Message-
>> >> >> From: Florent Paillot 
>> >> >> Sent: 15 April 2019 14:26
>> >> >> To: users 
>> >> >> Subject: Re: Problem with ACS 4.11.2 and vCenter server
>> >> >>
>> >> >> Hi Suresh,
>> >> >> The vCenter is a 6.7u2 (last build).
>> >> >>
>> >> >> There is no issue log, that the problem... I don't see any error
>> message
>> >> >> concerning a failed connection to the vCenter and CS does not seems
>> to
>> >> send
>> >> >> any packet to the vCenter.
>> >> >>
>> >> >> Florent
>> >> >>
>> >> >>
>> >> >> - Mail original -

Re: Disaster after maintenance

[Sergey Levitskiy]

; >> work ?

> > > > > >>

> > > > > >> Andrija

> > > > > >>

> > > > > >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov <

> > > > > j.zolotar...@gmail.com>

> > > > > >> wrote:

> > > > > >>

> > > > > >> > >>>Stop mgmt,

> > > > > >> > >>>Stop all agents

> > > > > >> > >>>Restart libvirtd (and check libvirt logs afterwards)

> > > > > >> > >>>Start agents

> > > > > >> > >>>Start mgmt.

> > > > > >> >

> > > > > >> > I did that numerous time. Nothing really suspicious

> > > > > >> > I can see that systems VMs are running - both in cloudstack

> > > console

> > > > > and

> > > > > >> > with virsh list -all

> > > > > >> >

> > > > > >> > It is apparently problem with network.

> > > > > >> > Is there a way to force recreation of defaultGuestNetwork? or

> > > force

> > > > > >> > recreation of Virtual Router.

> > > > > >> > I am unable to delete network, which is supposed to rebuild

> > > network

> > > > > with

> > > > > >> > its router. Thats the issue

> > > > > >> >

> > > > > >> > The issue with libvirtd was, that eventually at some point it

> > was

> > > > > >> updated

> > > > > >> > during 4 months of running, and not rebooted. It still 
worked.

> > We

> > > > had

> > > > > to

> > > > > >> > add listen_tcp = 1 for libvirtd to start working again.

> > > > > >> >

> > > > > >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic <

> > > > > andrija.pa...@gmail.com>

> > > > > >> > wrote:

> > > > > >> >

> > > > > >> > > As Sergey suggested... but i would also verify no libvirt

> > issues

> > > > or

> > > > > >> > storage

> > > > > >> > > pool issues - so perhaps:

> > > > > >> > >

> > > > > >> > > Stop mgmt,

> > > > > >> > > Stop all agents

> > > > > >> > > Restart libvirtd (and check libvirt logs afterwards)

> > > > > >> > > Start agents

> > > > > >> > > Start mgmt.

> > > > > >> > >

> > > > > >> > > What was originally issue with libvirtd ?

> > > > > >> > > That sounds fishy to me...

> > > > > >> > >

> > > > > >> > > Andrija

> > > > > >> > >

> > > > > >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy <

> > > serg...@hotmail.com

> > > > >

> > > > > >> > wrote:

> > > > > >> > >

> > > > > >> > > > select * from networks where removed is null;

> > > > > >> > > > select * from vm_instance where id=87;

> > > > > >> > > > select id,name from vm_instance where name like 'r%' and

> > > removed

> > > > > is

> > > > > >> > null;

> > > > > >> > > >

> > > > > >> > > > Basically since the network offering is not redundant 
this

> > > error

> > > > > is

> > > > > >> > only

> > > > > >> > > > thrown when there is no router associated with your

> network.

> > > > > Usually

> > > > > >> > > > management server restart tries to implement network

> again.

> > > > Please

> > > > > >> > > restart

> > > > > >> > > > management server, save and share management server log.

> > >

Re: Disaster after maintenance

[Sergey Levitskiy]

select * from networks where removed is null;
select * from vm_instance where id=87;
select id,name from vm_instance where name like 'r%' and removed is null;

Basically since the network offering is not redundant this error is only thrown 
when there is no router associated with your network. Usually management server 
restart tries to implement network again. Please restart management server, 
save and share management server log.




On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov"  wrote:

>>>>Check network_offering table for  value in column
redundant_router_service  for the network offering you use.
in table network_offering_table all records have redundant_router_service =
0

Can you also run the following:
>>>select name, state, removed  from host where name like 'r%'
returns zero rows - nothing

>>>select * from domain_router;
# id, element_id, public_mac_address, public_ip_address, public_netmask,
guest_netmask, guest_ip_address, is_redundant_router, priority,
redundant_state, stop_pending, role, template_version, scripts_version,
vpc_id, update_state
'4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
'57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL
'49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
'75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED'
'80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
'87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov 12 15:06:49 UTC
2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL


>>>select * from router_network_ref;
# id, router_id, network_id, guest_type
'1', '4', '204', 'Shared'
'2', '49', '204', 'Shared'
'3', '73', '204', 'Shared'
'4', '75', '204', 'Shared'
'5', '76', '204', 'Shared'
'6', '77', '204', 'Shared'
'7', '80', '204', 'Shared'
'8', '85', '204', 'Shared'
'9', '86', '204', 'Shared'
'10', '87', '204', 'Shared'


On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy 
wrote:

> Check network_offering table for  value in column
> redundant_router_service  for the network offering you use.
> Can you also run the following:
> select name, state, removed  from host where name like 'r%'
> select * from domain_router;
> select * from router_network_ref;
>
> Cloudstack is supposed to recreate you VR. If it is not happening there is
> something fundamentally wrong. I would advise to destroy your VR again.
> Stop you management server. Rotate management server log and start it
> again. If your VR doesn't start in few min, post your complete  management
> server log  and agent log again.
>
>
>
>
> On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov"  wrote:
>
> >>>Network offering needs to be change to non-redundant
> How do I do that?
>
> On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy  >
> wrote:
>
> > Network offering needs to be change to non-redundant. Most likely
> old bug
> > is resurfaced.
> > https://issues.apache.org/jira/browse/CLOUDSTACK-9024
>

Re: Disaster after maintenance

[Sergey Levitskiy]

Check network_offering table for  value in column redundant_router_service  for 
the network offering you use.
Can you also run the following:
select name, state, removed  from host where name like 'r%'
select * from domain_router;
select * from router_network_ref;

Cloudstack is supposed to recreate you VR. If it is not happening there is 
something fundamentally wrong. I would advise to destroy your VR again. Stop 
you management server. Rotate management server log and start it again. If your 
VR doesn't start in few min, post your complete  management server log  and 
agent log again.




On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov"  wrote:

>>>Network offering needs to be change to non-redundant
How do I do that?

On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy 
wrote:

> Network offering needs to be change to non-redundant. Most likely old bug
> is resurfaced.
> https://issues.apache.org/jira/browse/CLOUDSTACK-9024
>
>
>
> On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov"  wrote:
>
> Hello
>
> I did exactly like you suggested.
>
> After UPDATE on db, I can see the router in cloudstack console.
> But attempt to restart network fails.
> I get error:
> Resource [DataCenter:1] is unreachable: Can't find all necessary
> running
> routers!
>
> I rechecked agents on both servers. They look running OK.
>
> On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic <
> andrija.pa...@gmail.com>
> wrote:
>
> > Ok, so:
> >
> > 1.BACKUP YOUR DB - in case there is issues, you will want to restore
> > 2. I tried to reproduce your case on 4.11.2 (though didn't do any
> > maintenance etc.) - and could not - I have artificially marked
> existing VR
> > as removed in DB - and tried to restart network and it worked just
> fine.
> >
> > Let's try to "undelete" the VR for that network - though I can't be
> sure if
> > this will work or not.
> >
> > Find VR based on the network UUID:
> >   SELECT * FROM vm_instance WHERE id IN (SELECT instance_id from
> nics
> > where network_id IN (SELECT id FROM networks WHERE
> > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
> > Obviously, replace this UUID with your network UUID.
> > SQL above will return all VMs (user VMs, VRs, etc, delete or not...)
> that
> > have NIC inside the network with that UUID - look for the r-XX-VM
> being a
> > router.
> >
> > Now that you have the name of the VR (i.e. r-4-VM) - you want to set
> 2
> > fields to some other value than what it might be already, as below:
> >
> > Need to set field "state" to Stopped:
> >   UPDATE cloud.vm_instance SET state='Stopped' WHERE
> name="r-4-VM";
> >  (what state is set currently ???)
> >
> > Need to set "removed" to NULL value:
> >   UPDATE cloud.vm_instance SET removed = NULL WHERE
> NAME="r-4-VM"; (is
> > there a removed date already set ???)
> >
> > Obviously make sure that this VR is not running on any hypervisor
> and if
> > running, virsh destroy it...
> >
> > AFTER above has been done (VR is considered to exist but stopped,
> from
> > CloudStack point of view) - try to RESTART the network (don't bother
> > deleteing it, since there are VMs in that network)
> >
> > I'm not 100% positive this will fix your issue, but doesn't hurt to
> try
> >
> > If above doesn't work - I would still take a look into agents and if
> they
> > are still connected - optionally, restart agents on both hosts once
> more
> > and confirm they are connected and up.
> >
> > Let us know how it goes (and please backup DB once more before any
> actions
> > !)
> >
> > Andrija
> >
> >
> > On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> > wrote:
> >
> > > Yes.  Just a single network.
> > >
> > > On Tue, 19 Mar 2019, 21:39 Andrija Panic,  >
> > wrote:
> > >
> > 

Re: Disaster after maintenance

[Sergey Levitskiy]

Network offering needs to be change to non-redundant. Most likely old bug is 
resurfaced.
https://issues.apache.org/jira/browse/CLOUDSTACK-9024



On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov"  wrote:

Hello

I did exactly like you suggested.

After UPDATE on db, I can see the router in cloudstack console.
But attempt to restart network fails.
I get error:
Resource [DataCenter:1] is unreachable: Can't find all necessary running
routers!

I rechecked agents on both servers. They look running OK.

On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic 
wrote:

> Ok, so:
>
> 1.BACKUP YOUR DB - in case there is issues, you will want to restore
> 2. I tried to reproduce your case on 4.11.2 (though didn't do any
> maintenance etc.) - and could not - I have artificially marked existing VR
> as removed in DB - and tried to restart network and it worked just fine.
>
> Let's try to "undelete" the VR for that network - though I can't be sure 
if
> this will work or not.
>
> Find VR based on the network UUID:
>   SELECT * FROM vm_instance WHERE id IN (SELECT instance_id from nics
> where network_id IN (SELECT id FROM networks WHERE
> UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
> Obviously, replace this UUID with your network UUID.
> SQL above will return all VMs (user VMs, VRs, etc, delete or not...) that
> have NIC inside the network with that UUID - look for the r-XX-VM being a
> router.
>
> Now that you have the name of the VR (i.e. r-4-VM) - you want to set 2
> fields to some other value than what it might be already, as below:
>
> Need to set field "state" to Stopped:
>   UPDATE cloud.vm_instance SET state='Stopped' WHERE name="r-4-VM";
>  (what state is set currently ???)
>
> Need to set "removed" to NULL value:
>   UPDATE cloud.vm_instance SET removed = NULL WHERE NAME="r-4-VM"; (is
> there a removed date already set ???)
>
> Obviously make sure that this VR is not running on any hypervisor and if
> running, virsh destroy it...
>
> AFTER above has been done (VR is considered to exist but stopped, from
> CloudStack point of view) - try to RESTART the network (don't bother
> deleteing it, since there are VMs in that network)
>
> I'm not 100% positive this will fix your issue, but doesn't hurt to try
>
> If above doesn't work - I would still take a look into agents and if they
> are still connected - optionally, restart agents on both hosts once more
> and confirm they are connected and up.
>
> Let us know how it goes (and please backup DB once more before any actions
> !)
>
> Andrija
>
>
> On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov 
> wrote:
>
> > Yes.  Just a single network.
> >
> > On Tue, 19 Mar 2019, 21:39 Andrija Panic, 
> wrote:
> >
> > > Just one more clarification - this is Isolate single network (not
> Shared
> > > Network, not VPC) ?
> > >
> > >
> > >
> > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov  >
> > > wrote:
> > >
> > > > Name defaultGuestNetwork
> > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> > > > Zone PBT zone 1
> > > > Description defaultGuestNetwork
> > > > Type Shared
> > > > State Setup
> > > > VPC ID N/A
> > > > Persistent No
> > > >
> > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Share the network id and name as seen from GUI...
> > > > >
> > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > >>> 1. Confirm please rolling restart is set to false please -
> > double
> > > > > check
> > > > > > Double checked - It is set to false
> > > > > >
> > > > > > 2. If so - do you know the name of VR which you deleted ? Is
> it
> > > > last
> > > > > > one
> > > > > > >>> ever created - if so we can find it easily...
> > > > > > I dont know the name
> > > > > > Is there a way to fetch it from DB?
> > > > > >
> > > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic <
> > > andrija.pa...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Right...still complaining on missing running routers.
> > > > > > >
> > > > > > > 1. Confirm please rolling restart is set to false please -
> double
> > > > check
> > > > > > > 2. If so - do you know the name of VR which you deleted ? Is 
it
> > > last
> > > > > one
> > > > > > > ever created - if so we can find it easily...
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov <
> > > > j.zolotar...@gmail.com

Re: Upgrading from 4.9.3 to 4.11

[Sergey Levitskiy]

It seems your screenshot was lost. Can you re-send/save it somewhere and we 
will take a look?

From: "McClune, James" 
Reply-To: "users@cloudstack.apache.org" 
Date: Monday, November 5, 2018 at 7:34 AM
To: "users@cloudstack.apache.org" 
Cc: "Adams, Christopher" 
Subject: Re: Upgrading from 4.9.3 to 4.11

Hi Sergey,

Thank you for the feedback! :)

I managed to update the schema to 4.10. I'm trying to get to 4.11. However, I'm 
running into these errors now:

2018-11-05 10:23:19,896 INFO  [c.c.u.d.T.Transaction] (main:null) (logid:) Is 
Data Base High Availiability enabled? Ans : false
2018-11-05 10:23:20,233 DEBUG [c.c.u.d.DriverLoader] (main:null) (logid:) 
Successfully loaded DB driver com.mysql.jdbc.Driver
2018-11-05 10:23:20,239 DEBUG [c.c.u.d.DriverLoader] (main:null) (logid:) DB 
driver com.mysql.jdbc.Driver was already loaded.
2018-11-05 10:23:20,240 DEBUG [c.c.u.d.DriverLoader] (main:null) (logid:) DB 
driver com.mysql.jdbc.Driver was already loaded.
2018-11-05 10:23:20,567 DEBUG [c.c.u.d.ConnectionConcierge] (main:null) 
(logid:) Registering a database connection for LockMaster1
2018-11-05 10:23:20,567 INFO  [c.c.u.d.Merovingian2] (main:null) (logid:) 
Cleaning up locks for 90520739700260
2018-11-05 10:23:20,580 INFO  [c.c.u.d.Merovingian2] (main:null) (logid:) 
Released 0 locks for 90520739700260
2018-11-05 10:23:20,951 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] 
(main:null) (logid:) Running system integrity checker 
com.cloud.upgrade.DatabaseUpgradeChecker@1e6b9a95
2018-11-05 10:23:20,952 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) 
(logid:) Grabbing lock to check for database upgrade.
2018-11-05 10:23:21,042 DEBUG [c.c.u.d.VersionDaoImpl] (main:null) (logid:) 
Checking to see if the database is at a version before it was the version table 
is created
2018-11-05 10:23:21,078 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) 
(logid:) DB version = 4.10.0.0 Code Version = 4.11.0.0
2018-11-05 10:23:21,079 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) 
(logid:) Database upgrade must be performed from 4.10.0.0 to 4.11.0.0
2018-11-05 10:23:21,081 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null) 
(logid:) Running upgrade Upgrade41000to41100 to upgrade from 4.10.0.0-4.11.0.0 
to 4.11.0.0
2018-11-05 10:23:21,086 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
Licensed to the Apache Software Foundation (ASF) under one
2018-11-05 10:23:21,086 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- or 
more contributor license agreements.  See the NOTICE file
2018-11-05 10:23:21,086 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
distributed with this work for additional information
2018-11-05 10:23:21,086 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
regarding copyright ownership.  The ASF licenses this file
2018-11-05 10:23:21,087 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- to 
you under the Apache License, Version 2.0 (the
2018-11-05 10:23:21,087 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
"License"); you may not use this file except in compliance
2018-11-05 10:23:21,087 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
with the License.  You may obtain a copy of the License at
2018-11-05 10:23:21,087 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) --
2018-11-05 10:23:21,087 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) --   
http://www.apache.org/licenses/LICENSE-2.0
2018-11-05 10:23:21,087 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) --
2018-11-05 10:23:21,087 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
Unless required by applicable law or agreed to in writing,
2018-11-05 10:23:21,087 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
software distributed under the License is distributed on an
2018-11-05 10:23:21,088 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
2018-11-05 10:23:21,088 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
KIND, either express or implied.  See the License for the
2018-11-05 10:23:21,088 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
specific language governing permissions and limitations
2018-11-05 10:23:21,088 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
under the License.
2018-11-05 10:23:21,088 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) --;
2018-11-05 10:23:21,088 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
Schema upgrade from 4.10.0.0 to 4.11.0.0
2018-11-05 10:23:21,088 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) --;
2018-11-05 10:23:21,088 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- 
Add For VPC flag
2018-11-05 10:23:21,088 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) ALTER 
TABLE cloud.network_offerings ADD COLUMN for_vpc INT(1) NOT NULL DEFAULT 0
2018-11-05 10:23:21,092 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error 
executing: ALTER TABLE cloud.network_offerings ADD COLUMN for_vpc INT(1) NOT 
NULL DEFAULT 0
2018-11-05 10:23:21,093 ERROR [c.c.u.d.ScriptRunner] 

Re: cloudstack 4.11 still support PVLAN?

[Sergey Levitskiy]

PVLAN is supported although there are some prerequisites
http://docs.cloudstack.apache.org/projects/archived-cloudstack-administration/en/latest/networking_and_traffic.html#isolation-in-advanced-zone-using-private-vlan


On 11/4/18, 7:05 PM, "Ivan Kudryavtsev"  wrote:

Probably MAC-based VLANs and custom scripting will work for you.

вс, 4 нояб. 2018 г. в 22:04, li li :

> Hi Ivan
>
>
>
> my target is:
>
> 1. All the user VM cannot reach other user VM.
>
> 2. All the user VM can reach SMB server and Gateway.
>
>
>
> Any good suggestions?
>
>
>
>
>
> 
> 发件人: Ivan Kudryavtsev 
> 发送时间: Monday, November 5, 2018 10:32:23 AM
> 收件人: users
> 主题: Re: cloudstack 4.11 still support PVLAN?
>
> Hi Li,
> Doesn't look like it's supported. Actually, CloudStack supports isolated
> networks in advanced zones which support the feature above by creation of
> isolated VLAN for every network with the dedicated virtual router and 
other
> stuff.
>
> вс, 4 нояб. 2018 г. в 21:25, li li :
>
> > HI ALL
> >
> > I saw on the wiki that PVLAN for isolation is a very useful function. Is
> > CLOUDSTACK 4.11 still supported? Are there any deployment documents (KVM
> or
> > XENSERVER or XCP-NG)?
> >
> >
> >
> >
> 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/PVLAN+for+isolation+within+a+VLAN
> >
>
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks LLC
> Cell RU: +7-923-414-1515
> Cell USA: +1-201-257-1512
> WWW: http://bitworks.software/ 
>


-- 
With best regards, Ivan Kudryavtsev
Bitworks LLC
Cell RU: +7-923-414-1515
Cell USA: +1-201-257-1512
WWW: http://bitworks.software/ 

Re: Upgrading from 4.9.3 to 4.11

[Sergey Levitskiy]

Basically version of Cloudstack you are running has already some pieces of the 
code that is part of the upgrade from 4.9.3 to 4.10. This could have happened 
if you deployed a fix on the top of 4.9.3 or if a PR was backported to 4.9.3 
codebase after you install. You still can perform successful upgrade with some 
manual effort:
1. Make a copy of you CS DB.
2. Obtain the file schema-4930to41000.sql. 
https://github.com/apache/cloudstack/blob/1d05fead49f5c856257a741b07122f5633d2e359/engine/schema/src/main/resources/META-INF/db/schema-4930to41000.sql
3. Execute it on the copy of your DB. It will fail initially with the same 
error as in yourACS log
Error executing: ALTER TABLE `cloud`.`domain_router` ADD COLUMN 
update_state varchar(64) DEFAULT NULL
 2018-11-02 08:28:48,787 ERROR [c.c.u.d.ScriptRunner] 
(main:null) (logid:)
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: 
Duplicate column name 'update_state'
4. Comment line 22 in SQL file. Restore your DB copy
5. Continue steps 2-4 until you have successful processing of SQL file against 
a copy of your DB
(6 optional) Likely this will be enough to unblock your upgrade but if you want 
to ensure throughout upgrade you might need to repeat the same with 4.11 and 
4.11.1 upgrade files. You will need to do a sequence 4.9.3->4.10->4.11->4.11.1 
in each iteration.

https://github.com/apache/cloudstack/blob/1d05fead49f5c856257a741b07122f5633d2e359/engine/schema/src/main/resources/META-INF/db/schema-41000to41100.sql

https://github.com/apache/cloudstack/blob/1d05fead49f5c856257a741b07122f5633d2e359/engine/schema/src/main/resources/META-INF/db/schema-41100to41110.sql

7. Now you have adjusted DB upgrade file(s). Next step as to re-try your 
upgrade. After you update packages on the first management server but before 
your re-start the cloudstack-managment service you need to replace DB upgrade 
file in /usr/share/cloudstack-management/setup/db

Good luck.

Thanks,
Sergey



On 11/2/18, 9:19 AM, "McClune, James"  wrote:

Hello CloudStack Community,

I was wondering if you could help me a little further. I referenced the
documentation Rohit sent:


http://docs.cloudstack.apache.org/en/4.11.1.0/upgrading/upgrade/upgrade-4.9.html

I backed up my cloud database and followed the instructions verbatim. My
CloudStack management server is running on Ubuntu 14.04. When I upgrade my
packages to 4.11, the management doesn't start correctly (/client 404
error). I was getting errors along the lines of "Cannot connect to
database". I checked my db.properties file and made sure the
cluster.node.ip was set correctly, as well as database authentication (e.g.
MySQL username/password). I ran the Java command to encrypt my database
password, as described here:


http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.11/management.html#changing-the-database-password

I added the password to db.properties, hoping that my authentication was
somehow screwed up. I still got the errors. I didn't catch those errors,
but I'll retry and send them to you.

I also tried doing a manual upgrade and re-importing the database schema.
However, the 4.9.3 schema is different from 4.11 (as expected), so I
started to get errors:

root@cloudstack:~# tail -f
/var/log/cloudstack/management/management-server.log
2018-11-01 21:04:50,909 INFO  [o.e.j.s.h.ContextHandler] (main:null)
(logid:) Started o.e.j.s.h.MovedContextHandler@5cdd8682{/,null,AVAILABLE}
2018-11-02 08:28:23,152 INFO  [o.a.c.ServerDaemon] (main:null) (logid:)
Server configuration file found:
/etc/cloudstack/management/server.properties
2018-11-02 08:28:23,160 INFO  [o.a.c.ServerDaemon] (main:null) (logid:)
Initializing server daemon on :::8080, with https.enabled=false,
https.port=8443, context.path=/client
2018-11-02 08:28:23,172 INFO  [o.e.j.u.log] (main:null) (logid:) Logging
initialized @853ms to org.eclipse.jetty.util.log.Slf4jLog
2018-11-02 08:28:23,406 INFO  [o.e.j.s.Server] (main:null) (logid:)
jetty-9.4.z-SNAPSHOT, build timestamp: 2017-11-21T16:27:37-05:00, git hash:
82b8fb23f757335bb3329d540ce37a2a2615f0a8
2018-11-02 08:28:23,454 INFO  [o.e.j.s.AbstractNCSARequestLog] (main:null)
(logid:) Opened /var/log/cloudstack/management/access.log
2018-11-02 08:28:23,586 INFO  [o.e.j.w.StandardDescriptorProcessor]
(main:null) (logid:) NO JSP Support for /client, did not find
org.eclipse.jetty.jsp.JettyJspServlet
2018-11-02 08:28:23,619 INFO  [o.e.j.s.session] (main:null) (logid:)
DefaultSessionIdManager workerName=node0
2018-11-02 08:28:23,619 INFO  [o.e.j.s.session] (main:null) (logid:) No
SessionScavenger set, using defaults
2018-11-02 08:28:23,621 INFO  [o.e.j.s.session] (main:null) (logid:)
Scavenging every 66ms
2018-11-02 08:28:45,916 

Re: 4.9 to 4.11 upgrade broken

[Sergey Levitskiy]

You can either Restart the network with cleanup or simply destroy VR and let it 
be created on the next VM deployment.

On 8/21/18, 11:13 PM, "Asai"  wrote:

Thanks, nearly back up and running.  One question, what about the Virtual 
Router upgrade?  What do I do if the upgrade fails on the Virtual Router?  
Looking for docs on this, but can’t find anything.

Thanks for your assistance.

Asai


> On Aug 21, 2018, at 6:35 PM, Sergey Levitskiy  wrote:
> 
> Yes. this should bring you back.
> However if you perform what you described in your previous reply + rename 
template in CS DB to systemvm-kvm-4.11.1  from what it is now systemvm-kvm-4.11 
you should be able to bring all up as it is. Updating template image is not 
enough.
> 
> On 8/21/18, 4:04 PM, "Asai"  wrote:
> 
>OK thanks a lot, Sergey,
> 
>That helps.  What’s the best method to roll back?  Just use yum to 
roll back to 4.9 and rebuild the DB from backup?
> 
> 
>> On Aug 21, 2018, at 3:59 PM, Sergey Levitskiy  
wrote:
>> 
>> The fastest and easiest way is to rollback both DB and management server 
and start over. You need to have correct systemVM template registered before 
you initiate an upgrade.
>> 
>> Thanks,
>> Sergey
>> 
>> 
>> On 8/21/18, 2:30 PM, "Asai"  wrote:
>> 
>>   Is there anybody out there that can assist with this?  
>> 
>>   Asai
>> 
>> 
>>> On Aug 21, 2018, at 2:01 PM, Asai  wrote:
>>> 
>>> Is there any more specific instruction about this?
>>> 
>>> What is the best practice?  Should I roll back first?  Is there any 
documentation about rolling back?  Do I uninstall cloudstack management and 
re-install 4.9? 
>>> 
>>> Or is it as simple as just overwriting the file?  If so, what about the 
template.properties file and the metadata in there like qcow2.size?
>>> 
>>> filename=9cebb971-8605-3493-86f3-f5d1aef1715e.qcow2
>>> id=225
>>> qcow2.size=316310016
>>> public=true
>>> uniquename=225-2-826a2950-bb8e-34dd-9420-1eb24ea16b4a
>>> qcow2.virtualsize=2516582400
>>> virtualsize=2516582400
>>> checksum=2d8d1e4eacc976814b97f02849481433
>>> hvm=true
>>> description=systemvm-kvm-4.11
>>> qcow2=true
>>> qcow2.filename=9cebb971-8605-3493-86f3-f5d1aef1715e.qcow2
>>> size=316310016
>>> 
>>> 
>>> Asai
>>> 
>>> 
>>>> On Aug 21, 2018, at 1:56 PM, ilya musayev 
 wrote:
>>>> 
>>>> yes - please try the proper 4.11 systemvm templates.
>>>> 
>>>>> On Aug 21, 2018, at 1:54 PM, Asai  wrote:
>>>>> 
>>>>> Can I manually download the systemvm template from here? 
http://download.cloudstack.org/systemvm/4.11/ 
<http://download.cloudstack.org/systemvm/4.11/>
>>>>> 
>>>>> Then manually overwrite it in the filesystem and update it 
accordingly in the database?
>>>>> 
>>>>> Asai
>>>>> 
>>>>> 
>>>>>> On Aug 21, 2018, at 1:40 PM, Asai  wrote:
>>>>>> 
>>>>>> 4.11.0
>>>>>> 
>>>>>> As outlined in this 
http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.11.0.0/upgrade/upgrade-4.9.html
 
<http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.11.0.0/upgrade/upgrade-4.9.html>
>>>>>>> On Aug 21, 2018, at 1:37 PM, ilya musayev 
 wrote:
>>>>>>> 
>>>>>>> which template did you use? 
>>>>>>> 
>>>>>>>> On Aug 21, 2018, at 1:36 PM, Asai  
wrote:
>>>>>>>> 
>>>>>>>> Greetings,
>>>>>>>> 
>>>>>>>> I just tried to upgrade from 4.9 to 4.11, but it looks like the 
system VM template I downloaded according to the upgrade guide is the wrong 
template.  It’s 4.11, but I upgraded to 4.11.1 and I get this error message:
>>>>>>>> 
>>>>>>>> Caused by: com.cloud.utils.exception.CloudRuntimeException: 
4.11.1.0KVM SystemVm template not found. Cannot upgrade system Vms
>>>>>>>>at 
com.cloud.upgrade.dao.Upgrade41100to41110.updateSystemVmTemplates(Upgrade41100to41110.java:281)
>>>>>>>>at 
com.cloud.upgrade.dao.Upgrade41100to41110.performDataMigration(Upgrade41100to41110.java:68)
>>>>>>>>at 
com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:578)
>>>>>>>>... 53 more
>>>>>>>> 2018-08-21 13:28:25,257 INFO  [o.e.j.s.h.ContextHandler] 
(main:null) (logid:) Started 
o.e.j.s.h.MovedContextHandler@15bfd87{/,null,AVAILABLE}
>>>>>>>> 2018-08-21 13:28:25,317 INFO  [o.e.j.s.AbstractConnector] 
(main:null) (logid:) Started 
ServerConnector@4b1c1ea0{HTTP/1.1,[http/1.1]}{:::8080}
>>>>>>>> 2018-08-21 13:28:25,318 INFO  [o.e.j.s.Server] (main:null) 
(logid:) Started @20725ms
>>>>>>>> 
>>>>>>>> Can anyone assist with getting this corrected?
>>>>>>>> 
>>>>>>>> Thank you,
>>>>>>>> Asai
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>> 
> 
> 
> 

Re: 4.9 to 4.11 upgrade broken

[Sergey Levitskiy]

Yes. this should bring you back.
However if you perform what you described in your previous reply + rename 
template in CS DB to systemvm-kvm-4.11.1  from what it is now systemvm-kvm-4.11 
you should be able to bring all up as it is. Updating template image is not 
enough.

On 8/21/18, 4:04 PM, "Asai"  wrote:

OK thanks a lot, Sergey,

That helps.  What’s the best method to roll back?  Just use yum to roll 
back to 4.9 and rebuild the DB from backup?


> On Aug 21, 2018, at 3:59 PM, Sergey Levitskiy  wrote:
> 
> The fastest and easiest way is to rollback both DB and management server 
and start over. You need to have correct systemVM template registered before 
you initiate an upgrade.
> 
> Thanks,
> Sergey
> 
> 
> On 8/21/18, 2:30 PM, "Asai"  wrote:
> 
>Is there anybody out there that can assist with this?  
> 
>Asai
> 
> 
>> On Aug 21, 2018, at 2:01 PM, Asai  wrote:
>> 
>> Is there any more specific instruction about this?
>> 
>> What is the best practice?  Should I roll back first?  Is there any 
documentation about rolling back?  Do I uninstall cloudstack management and 
re-install 4.9? 
>> 
>> Or is it as simple as just overwriting the file?  If so, what about the 
template.properties file and the metadata in there like qcow2.size?
>> 
>> filename=9cebb971-8605-3493-86f3-f5d1aef1715e.qcow2
>> id=225
>> qcow2.size=316310016
>> public=true
>> uniquename=225-2-826a2950-bb8e-34dd-9420-1eb24ea16b4a
>> qcow2.virtualsize=2516582400
>> virtualsize=2516582400
>> checksum=2d8d1e4eacc976814b97f02849481433
>> hvm=true
>> description=systemvm-kvm-4.11
>> qcow2=true
>> qcow2.filename=9cebb971-8605-3493-86f3-f5d1aef1715e.qcow2
>> size=316310016
>> 
>> 
>> Asai
>> 
>> 
>>> On Aug 21, 2018, at 1:56 PM, ilya musayev 
 wrote:
>>> 
>>> yes - please try the proper 4.11 systemvm templates.
>>> 
>>>> On Aug 21, 2018, at 1:54 PM, Asai  wrote:
>>>> 
>>>> Can I manually download the systemvm template from here? 
http://download.cloudstack.org/systemvm/4.11/ 
<http://download.cloudstack.org/systemvm/4.11/>
>>>> 
>>>> Then manually overwrite it in the filesystem and update it accordingly 
in the database?
>>>> 
>>>> Asai
>>>> 
>>>> 
>>>>> On Aug 21, 2018, at 1:40 PM, Asai  wrote:
>>>>> 
>>>>> 4.11.0
>>>>> 
>>>>> As outlined in this 
http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.11.0.0/upgrade/upgrade-4.9.html
 
<http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.11.0.0/upgrade/upgrade-4.9.html>
>>>>>> On Aug 21, 2018, at 1:37 PM, ilya musayev 
 wrote:
>>>>>> 
>>>>>> which template did you use? 
>>>>>> 
>>>>>>> On Aug 21, 2018, at 1:36 PM, Asai  
wrote:
>>>>>>> 
>>>>>>> Greetings,
>>>>>>> 
>>>>>>> I just tried to upgrade from 4.9 to 4.11, but it looks like the 
system VM template I downloaded according to the upgrade guide is the wrong 
template.  It’s 4.11, but I upgraded to 4.11.1 and I get this error message:
>>>>>>> 
>>>>>>> Caused by: com.cloud.utils.exception.CloudRuntimeException: 
4.11.1.0KVM SystemVm template not found. Cannot upgrade system Vms
>>>>>>> at 
com.cloud.upgrade.dao.Upgrade41100to41110.updateSystemVmTemplates(Upgrade41100to41110.java:281)
>>>>>>> at 
com.cloud.upgrade.dao.Upgrade41100to41110.performDataMigration(Upgrade41100to41110.java:68)
>>>>>>> at 
com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:578)
>>>>>>> ... 53 more
>>>>>>> 2018-08-21 13:28:25,257 INFO  [o.e.j.s.h.ContextHandler] 
(main:null) (logid:) Started 
o.e.j.s.h.MovedContextHandler@15bfd87{/,null,AVAILABLE}
>>>>>>> 2018-08-21 13:28:25,317 INFO  [o.e.j.s.AbstractConnector] 
(main:null) (logid:) Started 
ServerConnector@4b1c1ea0{HTTP/1.1,[http/1.1]}{:::8080}
>>>>>>> 2018-08-21 13:28:25,318 INFO  [o.e.j.s.Server] (main:null) (logid:) 
Started @20725ms
>>>>>>> 
>>>>>>> Can anyone assist with getting this corrected?
>>>>>>> 
>>>>>>> Thank you,
>>>>>>> Asai
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 
> 
> 

Re: 4.9 to 4.11 upgrade broken

[Sergey Levitskiy]

The fastest and easiest way is to rollback both DB and management server and 
start over. You need to have correct systemVM template registered before you 
initiate an upgrade.

Thanks,
Sergey


On 8/21/18, 2:30 PM, "Asai"  wrote:

Is there anybody out there that can assist with this?  

Asai


> On Aug 21, 2018, at 2:01 PM, Asai  wrote:
> 
> Is there any more specific instruction about this?
> 
> What is the best practice?  Should I roll back first?  Is there any 
documentation about rolling back?  Do I uninstall cloudstack management and 
re-install 4.9? 
> 
> Or is it as simple as just overwriting the file?  If so, what about the 
template.properties file and the metadata in there like qcow2.size?
> 
> filename=9cebb971-8605-3493-86f3-f5d1aef1715e.qcow2
> id=225
> qcow2.size=316310016
> public=true
> uniquename=225-2-826a2950-bb8e-34dd-9420-1eb24ea16b4a
> qcow2.virtualsize=2516582400
> virtualsize=2516582400
> checksum=2d8d1e4eacc976814b97f02849481433
> hvm=true
> description=systemvm-kvm-4.11
> qcow2=true
> qcow2.filename=9cebb971-8605-3493-86f3-f5d1aef1715e.qcow2
> size=316310016
> 
> 
> Asai
> 
> 
>> On Aug 21, 2018, at 1:56 PM, ilya musayev  
wrote:
>> 
>> yes - please try the proper 4.11 systemvm templates.
>> 
>>> On Aug 21, 2018, at 1:54 PM, Asai  wrote:
>>> 
>>> Can I manually download the systemvm template from here? 
http://download.cloudstack.org/systemvm/4.11/ 

>>> 
>>> Then manually overwrite it in the filesystem and update it accordingly 
in the database?
>>> 
>>> Asai
>>> 
>>> 
 On Aug 21, 2018, at 1:40 PM, Asai  wrote:
 
 4.11.0
 
 As outlined in this 
http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.11.0.0/upgrade/upgrade-4.9.html
 

> On Aug 21, 2018, at 1:37 PM, ilya musayev 
 wrote:
> 
> which template did you use? 
> 
>> On Aug 21, 2018, at 1:36 PM, Asai  wrote:
>> 
>> Greetings,
>> 
>> I just tried to upgrade from 4.9 to 4.11, but it looks like the 
system VM template I downloaded according to the upgrade guide is the wrong 
template.  It’s 4.11, but I upgraded to 4.11.1 and I get this error message:
>> 
>> Caused by: com.cloud.utils.exception.CloudRuntimeException: 
4.11.1.0KVM SystemVm template not found. Cannot upgrade system Vms
>>  at 
com.cloud.upgrade.dao.Upgrade41100to41110.updateSystemVmTemplates(Upgrade41100to41110.java:281)
>>  at 
com.cloud.upgrade.dao.Upgrade41100to41110.performDataMigration(Upgrade41100to41110.java:68)
>>  at 
com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:578)
>>  ... 53 more
>> 2018-08-21 13:28:25,257 INFO  [o.e.j.s.h.ContextHandler] (main:null) 
(logid:) Started o.e.j.s.h.MovedContextHandler@15bfd87{/,null,AVAILABLE}
>> 2018-08-21 13:28:25,317 INFO  [o.e.j.s.AbstractConnector] 
(main:null) (logid:) Started 
ServerConnector@4b1c1ea0{HTTP/1.1,[http/1.1]}{:::8080}
>> 2018-08-21 13:28:25,318 INFO  [o.e.j.s.Server] (main:null) (logid:) 
Started @20725ms
>> 
>> Can anyone assist with getting this corrected?
>> 
>> Thank you,
>> Asai
>> 
>> 
> 
 
>>> 
>> 
> 

Re: VR starting problem

[Sergey Levitskiy]

Try one of these:
https://discussions.citrix.com/topic/299938-error-when-disabling-ha-the-uuid-you-supplied-was-invalid/
http://blog.branix.net/cloudplatform/when-live-storage-migration-goes-wrong


On 11/7/17, 12:58 PM, "he...@powershield.ru"  wrote:

All day we tested and have no idea where is a problem:)

1. We can create template from stopped VM (after stopping VM, we can't 
start with the same error: Insufficient capacity)
2. We can download and install ISO
3. We can create new DATADISK and attach it to online VM's
4. We can create new Disk Offering. Create new DATADISK and 
attach/detach from the VM's.

We try'ed to search VDI, because some error's with the same VDI uuid:
c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-1:ctx-bea5904c) 
(logid:1f164ea4) Catch Exception com.xensource.xenapi.Types$UuidInvalid 
:VDI getByUuid for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due 
to The uuid you supplied was invalid.

Where in database writed this uuid? We thinking, here could be a 
problem. We can not find any entries by this uuid in our storage.
  

com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getVDIbyUuid(XenServerStorageProcessor.java:743)
Caused by: The uuid you supplied was invalid.

Unsupported data object (VOLUME, 
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@4ddd9112), 
no need to delete from object in store ref table
com.cloud.exception.StorageUnavailableException: Resource 
[StoragePool:6] is unreachable: Unable to create 
Vol[5128|vm=5101|ROOT]:com.cloud.utils.exception.CloudRuntimeException: 
Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid 
for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid 
you supplied was invalid.

It's also writing, that storage is unreachable Resource [StoragePool:6] 
is unreachable: Unable to create

But as i wrote, we can create new diskdata and so on on this storage 
without problem.


https://pastebin.com/HLt40fK0

Re: Enabling local-storage for Guest VMs

[Sergey Levitskiy]

Try unmanage your vmware cluster and make it Managed again. Restarting ACS will 
also do the trick.

On 11/6/17, 5:35 AM, "Asanka Gunasekara" <asa...@nimbus.lk> wrote:

Thank you Sergey, Though I change the record to 1, I don't see storage_pool
getting updated, do I need to restart the cloudstack?

MariaDB [cloud]> select id, is_local_storage_enabled from data_center;
++--+
| id | is_local_storage_enabled |
++--+
|  1 |1 |
++--+
1 row in set (0.00 sec)

MariaDB [cloud]>

Thank you and best regards

Asanka


On 3 November 2017 at 02:12, Dag Sonstebo <dag.sonst...@shapeblue.com>
wrote:

> Thanks – will test it in the lab.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>  S: +44 20 3603 0540  | dag.sonst...@shapeblue.com |
> http://www.shapeblue.com <http://www.shapeblue.com/> | Twitter:@ShapeBlue
> <https://twitter.com/#!/shapeblue>
>
>
> On 03/11/2017, 03:47, "Sergey Levitskiy" <serg...@hotmail.com> wrote:
>
> Never tried it for other hypervisors. You can verify if entries for
> each local datastore are created in storage_pool table. One they are 
there,
> provisioning with service offering using local storage should succeed.
>
> On 11/2/17, 10:25 AM, "Dag Sonstebo" <dag.sonst...@shapeblue.com>
> wrote:
>
> Thanks for the tip Sergey – does the unmanage/manage trick work
> for KVM and XS as well after the db flip?
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>  S: +44 20 3603 0540  | dag.sonst...@shapeblue.com |
> http://www.shapeblue.com <http://www.shapeblue.com/> | Twitter:@ShapeBlue
> <https://twitter.com/#!/shapeblue>
>
>
> On 03/11/2017, 00:32, "Sergey Levitskiy" <serg...@hotmail.com>
> wrote:
>
> You can update record in table data_center by flipping flag
> ‘is_local_storage_enabled’. Then manage/unmanage ESX clusters in that zone
> that would be trigger discovery of local storage pools.
>
> Thanks,
> Sergey
>
> On 11/2/17, 5:02 AM, "Asanka Gunasekara" <asa...@nimbus.lk>
> wrote:
>
> Thank you Lvan
>
> Only way is the recreate the Zone, starting from scratch 
:)
>
> Best Regards
>
> Asanka
>
> On 2 November 2017 at 17:29, Ivan Kudryavtsev <
> kudryavtsev...@bw-sw.com>
> wrote:
>
> > Hi, Asanka. Service offering of type local doesn't mean
> you can deploy the
> > vm instance if zone doesn't support local storage
> capability.
> >
> > Dag is absolutely right about local storage. You have to
> enable it during
> > zone deployment procedure. It leads that ACS begins to
> display all
> > /var/lib/libvirt dirs for every compute node as local
> storage in
> > infrastructure storage area. Only they fit for local
> offering deployments.
> > Also, you can specify if system vms are deployed in
> local storage (which is
> > meaningful if you would like to purge shared storage at
> all).
> >
> > 2 нояб. 2017 г. 18:48 пользователь "Dag Sonstebo" <
> > dag.sonst...@shapeblue.com> написал:
> >
> > Asanka,
> >
> > As far as I remember you need to enable local storage
> for guest VMs when
> > you first create the zone.
> >
> > Regards,
> > Dag Sonstebo
> > Cloud Architect
> > ShapeBlue
> >  S: +44 20 3603 0540  | dag.sonst...@shapeblue.com |
> > http://www.shapeblue.com <http://www.shapeblue.com/> |
> Twitter:@ShapeBlue
> > <
> > https://twitter.com/#!/shapeblue>
> >
>   

Re: Enabling local-storage for Guest VMs

[Sergey Levitskiy]

Never tried it for other hypervisors. You can verify if entries for each local 
datastore are created in storage_pool table. One they are there, provisioning 
with service offering using local storage should succeed.

On 11/2/17, 10:25 AM, "Dag Sonstebo" <dag.sonst...@shapeblue.com> wrote:

Thanks for the tip Sergey – does the unmanage/manage trick work for KVM and 
XS as well after the db flip?

Regards, 
Dag Sonstebo
Cloud Architect
ShapeBlue
 S: +44 20 3603 0540  | dag.sonst...@shapeblue.com | 
http://www.shapeblue.com <http://www.shapeblue.com/> | Twitter:@ShapeBlue 
<https://twitter.com/#!/shapeblue>


On 03/11/2017, 00:32, "Sergey Levitskiy" <serg...@hotmail.com> wrote:

You can update record in table data_center by flipping flag 
‘is_local_storage_enabled’. Then manage/unmanage ESX clusters in that zone that 
would be trigger discovery of local storage pools.

Thanks,
Sergey

On 11/2/17, 5:02 AM, "Asanka Gunasekara" <asa...@nimbus.lk> wrote:

Thank you Lvan

Only way is the recreate the Zone, starting from scratch :)

Best Regards

Asanka

On 2 November 2017 at 17:29, Ivan Kudryavtsev 
<kudryavtsev...@bw-sw.com>
wrote:

> Hi, Asanka. Service offering of type local doesn't mean you can 
deploy the
> vm instance if zone doesn't support local storage capability.
>
> Dag is absolutely right about local storage. You have to enable 
it during
> zone deployment procedure. It leads that ACS begins to display all
> /var/lib/libvirt dirs for every compute node as local storage in
> infrastructure storage area. Only they fit for local offering 
deployments.
> Also, you can specify if system vms are deployed in local storage 
(which is
> meaningful if you would like to purge shared storage at all).
>
> 2 нояб. 2017 г. 18:48 пользователь "Dag Sonstebo" <
> dag.sonst...@shapeblue.com> написал:
>
> Asanka,
>
> As far as I remember you need to enable local storage for guest 
VMs when
> you first create the zone.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>  S: +44 20 3603 0540  | dag.sonst...@shapeblue.com |
> http://www.shapeblue.com <http://www.shapeblue.com/> | 
Twitter:@ShapeBlue
> <
> https://twitter.com/#!/shapeblue>
>
>
> On 02/11/2017, 20:30, "Asanka Gunasekara" <asa...@nimbus.lk> 
wrote:
>
> Hi All, when I go to
>
> Service offerings > Compute Offering > Add compute offering
>
> In the Add Computer Offering dialogue box, when I pull-down 
on the
> option
> "Storage type" it gives me two options.
>
> a) Shared
> b) Local
>
> Is this mean, by default local storage is enabled for Guest 
VMs?
>
> Thanks and Regards
>
> Asanka
>
>
> dag.sonst...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>

dag.sonst...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

> On 2 November 2017 at 13:14, Asanka Gunasekara <asa...@nimbus.lk> wrote:
>
> > Hi All, in the global setting I am unable to find this 
particular
> option
> > (use.local.storage) but system.vm.use.local.storage is set 
to fals,
> which
> > is fine.
> >
> > Thanks and Regards
> >
> > Asanka
> >
> > On 2 November 2017 at 10:40, Asanka Gunasekara 
<asa...@nimbus.lk>
> wrote:
> >
> >> Hi I need to enable local-storage for guests that dont 
need HA or VM
> >> move, if I follow below will there be any advers effect (I 
have
> already
  

Re: Enabling local-storage for Guest VMs

[Sergey Levitskiy]

You can update record in table data_center by flipping flag 
‘is_local_storage_enabled’. Then manage/unmanage ESX clusters in that zone that 
would be trigger discovery of local storage pools.

Thanks,
Sergey

On 11/2/17, 5:02 AM, "Asanka Gunasekara"  wrote:

Thank you Lvan

Only way is the recreate the Zone, starting from scratch :)

Best Regards

Asanka

On 2 November 2017 at 17:29, Ivan Kudryavtsev 
wrote:

> Hi, Asanka. Service offering of type local doesn't mean you can deploy the
> vm instance if zone doesn't support local storage capability.
>
> Dag is absolutely right about local storage. You have to enable it during
> zone deployment procedure. It leads that ACS begins to display all
> /var/lib/libvirt dirs for every compute node as local storage in
> infrastructure storage area. Only they fit for local offering deployments.
> Also, you can specify if system vms are deployed in local storage (which 
is
> meaningful if you would like to purge shared storage at all).
>
> 2 нояб. 2017 г. 18:48 пользователь "Dag Sonstebo" <
> dag.sonst...@shapeblue.com> написал:
>
> Asanka,
>
> As far as I remember you need to enable local storage for guest VMs when
> you first create the zone.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>  S: +44 20 3603 0540  | dag.sonst...@shapeblue.com |
> http://www.shapeblue.com  | Twitter:@ShapeBlue
> <
> https://twitter.com/#!/shapeblue>
>
>
> On 02/11/2017, 20:30, "Asanka Gunasekara"  wrote:
>
> Hi All, when I go to
>
> Service offerings > Compute Offering > Add compute offering
>
> In the Add Computer Offering dialogue box, when I pull-down on the
> option
> "Storage type" it gives me two options.
>
> a) Shared
> b) Local
>
> Is this mean, by default local storage is enabled for Guest VMs?
>
> Thanks and Regards
>
> Asanka
>
>
> dag.sonst...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
> On 2 November 2017 at 13:14, Asanka Gunasekara  wrote:
>
> > Hi All, in the global setting I am unable to find this particular
> option
> > (use.local.storage) but system.vm.use.local.storage is set to fals,
> which
> > is fine.
> >
> > Thanks and Regards
> >
> > Asanka
> >
> > On 2 November 2017 at 10:40, Asanka Gunasekara 
> wrote:
> >
> >> Hi I need to enable local-storage for guests that dont need HA or 
VM
> >> move, if I follow below will there be any advers effect (I have
> already
> >> created the Zone)
> >>
> >> 1. In Global Setting change use.local.storage = true and do a
> restart
> >> 2. Create a new offering with local storage
> >>
> >> Cloudstack 4.10
> >> Hypervisor - ESXi 5.5 (Vmware)
> >>
> >> Thanks and Regards
> >>
> >> Asanka
> >>
> >>
> >
>

Re: Below error pups up while creating a Zone

[Sergey Levitskiy]

You can use a custom name. It is controlled by global config 
vmware.management.portgroup. You still have to follow these requirements as per 
http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/hypervisor/vsphere.html


If the ESXi hosts have multiple VMKernel ports, and ESXi is not using the 
default value “Management Network” as the management network name, you must 
follow these guidelines to configure the management network port group so that 
CloudStack can find it:

Use one label for the management network port across all ESXi hosts.
In the CloudStack UI, go to Configuration - Global Settings and set 
vmware.management.portgroup to the management network label from the ESXi hosts.



On 10/31/17, 8:27 AM, "Dag Sonstebo"  wrote:

Just “Management Network”.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 31/10/2017, 11:38, "Asanka Gunasekara"  wrote:

Hi Dag, thanks for the reply, meaning I have to create a port group 
called
"Management Network" in vCenter DVS and Do I need to create Guests, 
Storage
etc as well

Best Regards

Asanka

On 31 October 2017 at 16:41, Dag Sonstebo 
wrote:

> Hi Asanka,
>
> Your networking doesn’t seem to be configured correctly -  “Uanble to 
find
> management port group Management Network”. This port group is 
expected on
> the vSwitch you have tagged under management network in your 
CloudStack
> networking.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 31/10/2017, 10:49, "Asanka Gunasekara"  wrote:
>
> Hi,
>
> I am not sure if this sort-out the above issue but changing 
DSwitch to
> dvSwitch from vCenter bypassed above error and I am stuck at 
below.
>
> And from the UI it is saying "Something went wrong and correct the
> folowing. "Failed to create storage pool as it is not accessible 
to
> hosts.
> Storage is on a seprate network and there is no VLANs o any thing 
and
> this
> is accessible to all host.
>
> 2017-10-31 15:47:21,216 DEBUG [c.c.h.v.m.HypervisorHostHelper]
> (catalina-exec-3:ctx-80cb7953 ctx-e052e8f7) (logid:27870103) Found
> distributed vSwitch dvS   witch0
> 2017-10-31 15:47:21,232 INFO  [c.c.h.v.m.HypervisorHostHelper]
> (catalina-exec-3:ctx-80cb7953 ctx-e052e8f7) (logid:27870103)
> Distributed
> Virtual Port grou   p 
cloud.private.untagged.0.1-dvSwitch0
> not
> found.
> 2017-10-31 15:47:22,767 INFO  [c.c.h.v.m.VmwareManagerImpl]
> (catalina-exec-3:ctx-80cb7953 ctx-e052e8f7) (logid:27870103) 
Preparing
> network on host com.cl
> oud.hypervisor.vmware.util.VmwareContext@1b16fb90 for
> dvSwitch0,,vmwaredvs
> 2017-10-31 15:47:22,816 DEBUG [c.c.h.v.m.HypervisorHostHelper]
> (catalina-exec-3:ctx-80cb7953 ctx-e052e8f7) (logid:27870103) Found
> distributed vSwitch dvS   witch0
> 2017-10-31 15:47:22,828 INFO  [c.c.h.v.m.HypervisorHostHelper]
> (catalina-exec-3:ctx-80cb7953 ctx-e052e8f7) (logid:27870103) Found
> Distributed Virtual Por   t group
> cloud.private.untagged.0.1-dvSwitch0
> 2017-10-31 15:47:22,855 DEBUG [c.c.h.v.m.HypervisorHostHelper]
> (catalina-exec-3:ctx-80cb7953 ctx-e052e8f7) (logid:27870103) 
Checking
> if
> configuration ofdvPortGroup
> [cloud.private.untagged.0.1-dvSwitch0] has changed.
> 2017-10-31 15:47:23,064 DEBUG [c.c.h.v.r.VmwareContextFactory]
> (catalina-exec-3:ctx-80cb7953 ctx-e052e8f7) (logid:27870103) 
initialize
> VmwareContext. url   : https://172.17.101.246/sdk/
> vimService,
> username: administrator@vsphere.local, password: W*
> 2017-10-31 15:47:23,165 INFO  [c.c.h.v.u.VmwareContext]
> (catalina-exec-3:ctx-80cb7953 ctx-e052e8f7) (logid:27870103) New
> VmwareContext object, current ou   tstanding count: 2
> 2017-10-31 15:47:23,603 INFO  [c.c.h.v.r.VmwareResource]
> (catalina-exec-3:ctx-80cb7953 ctx-e052e8f7) (logid:27870103)
> VmwareResource
> network configuratio   n info. private traffic over
> vSwitch:
> dvSwitch0,,vmwaredvs, public traffic over vmwaresvs : dvSwitch0, 
guest
> traffic over vmwaredvs : dvSwitch0
> 2017-10-31 15:47:23,809 INFO  

Re: listTemplates: multiple items with same ID

[Sergey Levitskiy]

As long as template is copied from zone to zone it will present with duplicate 
ID. This is by design. crossZone parameter is to indicate to auto replicate 
template across all available zone on template creation. To put it differently 
only combination of template ID and zone ID is a primary key for template 
querying.

Thanks,
Sergey

Re: Auto Generating MAC Addresses for VMs in a given pattern

[Sergey Levitskiy]

Randomization is done for each VM. The only static part is prefix - 06. Unless 
you change the ACS code it won’t support a predefined range for Mac addresses. 
If you sort you nics table by IP address you will see that octet on the rigtht 
gets incremented due to respective sequence increment. But at the same time 
second and third octet on the left change due to randomization.

On 5/11/17, 10:44 AM, "Imran Ahmed" <im...@eaxiom.net> wrote:

Hi Sergey,

I am grateful for your prompt reply. The data center provider has provided 
a starting sequence to be used. Do you think it would be another option to 
modify the sequence in db tables (data_center and user_ip_address) so that the 
VMs get an IP/MAC from that sequence? Is  the randomization function called 
during creation of each VM  or its run just once during the creation of zone 
(data_center)?

Kind regards,

Imran 

-Original Message-
    From: Sergey Levitskiy [mailto:sergey.levits...@autodesk.com] 
Sent: Thursday, May 11, 2017 10:16 PM
To: users@cloudstack.apache.org
Subject: Re: Auto Generating MAC Addresses for VMs in a given pattern

Mac address is dynamically created based on the IP address sequence number 
being allocated on your subnet with randomization algorithm.
It is done in NetUtils.java 

public static long createSequenceBasedMacAddress(final long macAddress) 
{
return macAddress | 0x0600l | (long)s_rand.nextInt(32768) 
<< 25 & 0x00fffe00l;
}

Starting sequence of your address is defined in data_center table and 
carried over to user_ip_address tablewhere it gets properly incremented.

Routers use slightly different pattern for mac addresses.

I would advise to figure out another way of addressing your issue vs 
modifying ACS code that deals with MACs.

Thanks,
Sergey




On 5/11/17, 9:27 AM, "Imran Ahmed" <im...@eaxiom.net> wrote:

Dear Team,
We are in a situation where the datacenter provider has put a 
restriction on
the MAC addresses.
We Cloudstack 4.3 with XenServers 6.5 .  We need to modify the pattern 
in
which the guest VMs get the MAC address and use the pattern provided by 
the
datacenter provider.
Please suggest what should be a good starting point ?

Best regards,

Imran 

Re: Auto Generating MAC Addresses for VMs in a given pattern

[Sergey Levitskiy]

Mac address is dynamically created based on the IP address sequence number 
being allocated on your subnet with randomization algorithm.
It is done in NetUtils.java 

public static long createSequenceBasedMacAddress(final long macAddress) {
return macAddress | 0x0600l | (long)s_rand.nextInt(32768) << 25 
& 0x00fffe00l;
}

Starting sequence of your address is defined in data_center table and carried 
over to user_ip_address tablewhere it gets properly incremented.

Routers use slightly different pattern for mac addresses.

I would advise to figure out another way of addressing your issue vs modifying 
ACS code that deals with MACs.

Thanks,
Sergey




On 5/11/17, 9:27 AM, "Imran Ahmed"  wrote:

Dear Team,
We are in a situation where the datacenter provider has put a restriction on
the MAC addresses.
We Cloudstack 4.3 with XenServers 6.5 .  We need to modify the pattern in
which the guest VMs get the MAC address and use the pattern provided by the
datacenter provider.
Please suggest what should be a good starting point ?

Best regards,

Imran 

Re: VMware workstation integration with cloudstack

[Sergey Levitskiy]

Vmware workstation is not a supported hypervisor for Cloudstack. You need 
VMware ESX that is managed by vCenter.

  You can see detailed requirements here:

http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/hypervisor/vsphere.html#system-requirements-for-vsphere-hosts

Re: Change scope of storage pool

[Sergey Levitskiy]

We made such changes once. You can run the following update statement against 
ACS DB and then un-manage/mange all clusters in the zone (replace vmware with 
your hypervisor name e.g Xen, KVM). Workjed just fine. Please test it in the 
Lab first.

update storage_pool set scope=’ ZONE’, hypervisor=’VMware’, pod_id=null, 
cludter_id=null where uuid=’ XXX’



On 4/6/17, 7:10 AM, "Rafael Weingärtner"  wrote:

Cannot you plugin the new storage and then migrate VMs volumes for this new
one?
There is a command in the API that does just that:
https://cloudstack.apache.org/api/apidocs-4.9/apis/migrateVolume.html

2017-04-06 10:02 GMT-04:00 Ugo Vasi :

> Hello guys,
> I would like to know if it's possible change the scope of a storage pool
> (primary storage server) from CLUSTER to ZONE.
> I need to migrate some vm from an old storage to new one but the scope
> differ.
>
> Regards
> Ugo
>
> --
>
>   U g o   V a s i
>   P r o c n e  s.r.l>)
>   via Cotonificio 45  33010 Tavagnacco IT
>   phone: +390432486523 fax: +390432486523
>
> Le informazioni contenute in questo messaggio sono riservate e
> confidenziali ed è vietata la diffusione in qualunque modo eseguita.
> Qualora Lei non fosse la persona a cui il presente messaggio è
> destinato, La invitiamo ad eliminarlo e a non leggerlo, dandocene
> gentilmente comunicazione.
> Per qualsiasi informazione si prega di contattare supp...@procne.it .
> Rif. D.L. 196/2003
>
>


-- 
Rafael Weingärtner

Re: [VOTE] Retirement of midonet plugin

[Sergey Levitskiy]

+1

Re: ACS 4.9.2 with VMware 5.5

[Sergey Levitskiy]

The  root cause is below which is most likely due to not specifying VLAN range 
during zone creation. 
Can you post result for the following DB queries?

select * from cloud.networsk where id=218;
select * from cloud.vlan where network_id=218;
select count(*) from cloud.user_ip_address where vlan_db_id in (select id from 
cloud.vlan where network_id=218) and state=’Free’
select id,name,data_center_id, vnet from cloud.physical_network


2017-03-15 10:41:25,855 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(Work-Job-Executor-48:ctx-9013c76a job-140/job-142 ctx-213d798a) 
(logid:918eb10d) Cleaning up because we're unable to implement the network 
Ntwk[218|Guest|11]

2017-03-15 10:41:25,881 INFO  [c.c.v.VirtualMachineManagerImpl] 
(Work-Job-Executor-48:ctx-9013c76a job-140/job-142 ctx-213d798a) 
(logid:918eb10d) Insufficient capacity 
com.cloud.exception.InsufficientVirtualNetworkCapacityException: Unable to 
allocate vnet as a part of network Ntwk[218|Guest|11] implement Scope=interface 
com.cloud.dc.DataCenter; id=3
at 
com.cloud.network.guru.GuestNetworkGuru.allocateVnet(GuestNetworkGuru.java:289)
at 
com.cloud.network.guru.GuestNetworkGuru.implement(GuestNetworkGuru.java:319)
at 
com.cloud.network.guru.ExternalGuestNetworkGuru.implement(ExternalGuestNetworkGuru.java:132)
at 
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetwork(NetworkOrchestrator.java:1011)
at 
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetwork(NetworkOrchestrator.java:950)
at 
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1314)
at 
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:997)
at 
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4605)
at sun.reflect.GeneratedMethodAccessor302.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)








On 3/15/17, 9:08 AM, "Ciobanu Cristian" <cristian.c@istream.today> wrote:

Hello ,

   Please see : 
https://drive.google.com/file/d/0B2xyGcnD30FVcFZZa1FvTmZtRWc/view?usp=sharing I 
suggest to check on the last part of this log.


Thank you
 Cristian

-Original Message-
    From: Sergey Levitskiy [mailto:sergey.levits...@autodesk.com] 
Sent: Wednesday, March 15, 2017 5:35 PM
To: users@cloudstack.apache.org
Subject: Re: ACS 4.9.2 with VMware 5.5

Can you extract full test of exception for management server logs and post 
here? Ideally please save the full management log on e.g. dropbox and add the 
link as well.

On 3/15/17, 8:21 AM, "Ciobanu Cristian" <cristian.c@istream.today> wrote:

Hello,

 

 Today I have did another test, but now with VMware 5.5  and ACS 
4.9.2
configured with advanced networking, everything looks ok, except I'm not
able to deploy any VM.

 

SSVM - OK

Proxy Console - OK

VPC - OK

Tier - OK

Templates ready - OK

 

Deploy instance - NOT OK

 

  I also have 2 environments  with ACS 4.5.2 and VMware 5.5 configured 
with
basic and advanced networking everything works perfectly there.

 

 

  Can I get confirmation if anyone tested ACS 4.9.2 with VMware 5.5 ? 

 

 Logs related to this issue - http://pastebin.com/tM6ZBfDp

===

[root@acs management]# grep -r i-2-20-VM --color

apilog.log:2017-03-15 10:41:28,423 INFO  [a.c.c.a.ApiServer]
(catalina-exec-18:ctx-7ec54fca ctx-250a55aa) (logid:76a929db) (userId=2
accountId=2 sessionId=051314C1FA5E08E86DC0373970EE42D9) 86.125.237.148 
--
GET

command=listVirtualMachines=b7cbfcea-f13e-4954-b1e1-f7d067f64293
=json&_=148956710 200

{"listvirtualmachinesresponse":{"count":1,"virtualmachine":[{"id":"b7cbfcea-

f13e-4954-b1e1-f7d067f64293","name":"VM-b7cbfcea-f13e-4954-b1e1-f7d067f64293

","displayname":"VM-b7cbfcea-f13e-4954-b1e1-f7d067f64293","account":"admin",

"userid":"46a435ea-095d-11e7-811f-000c2929469d","username":"admin","domainid

":"c932053e-095b-11e7-811f-000c2929469d","domain":"ROOT","created":"2017-03-

15T10:41:

Re: Cloudstack Vmware NSX Integration

[Sergey Levitskiy]

NSX MH is supported up to version 4.2.X for both shared an isolated networking. 
The plugin only supports virtual networking, static and source NAT as well port 
forwarding over STT. No other services are supported and little new development 
is happening with NSX plugin. Also as per NSX 4.2 only standard vSwitch can 
co-exist with NSX.
Here it is some additional details on it:
http://docs.cloudstack.apache.org/en/latest/networking/nicira-plugin.html


Thanks,
Sergey



On 3/13/17, 6:47 AM, "Özhan Rüzgar Karaman"  wrote:

Hi;
Do we have any build-in support for Vmware NSX Platform?

We have currently integrated ACS 4.9.2 to customers Vmware environment and
we also like to integrate with their NSX solution to supply firewall and
qos support for deployed vm's.

Thanks
Özhan

Re: VM in incorrect state

[Sergey Levitskiy]

I saw something like this when VM ‘lost’ a NIC somehow. Can you check if there 
is at least one NIC assigned to the VM?  Fix was to restore NIC record for that 
VM in the DB. Another option (if VM is slated for destruction) is just to 
change VM state to ‘Destroyed’ and garbage collector thread will cleanup 
volumes and other artifacts.


On 2/14/17, 9:09 AM, "Rafael Weingärtner"  wrote:

Hi Natalia,
This problem is caused at class method

“org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.releaseNic(VirtualMachineProfile,
long)” line 1631 as you can see by the stack trace.

Did you change something else in the database or ACS while these VMs were
stuck in "stopping"?

The code at that line is the following:

updateNic(nic, network.getId(), -1);

For some reason the network object is null. The line that creates the
“network” object is 1609. The code is the following:

final NetworkVO network = _networksDao.findById(nic.getNetworkId());

It seems that the network that was used to assign the IPs for the VMs NIC
was deleted/removed? Could you check that?
The network interface card table is “nics”, you can look for the NICs of
your VM by filtering for “instance_id”.
Also, if you want to check the networks table is “networks”.

On Tue, Feb 14, 2017 at 11:52 AM, Natalia Costas Lago 
wrote:

> Dear Rafael,
>
> Thre requested information:
>
> ACS version: 4.9.0
>
> hypervisors used: kvm
> storage systems used: gluster as primary
> networking: advanced
>
> And this is the error...
>
>
>
> 2017-02-14 17:48:56,112 DEBUG [c.c.u.d.T.Transaction]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329 ctx-39cd3fd6) (logid:2eed47c7)
> Rolling back the transaction: Time = 1 Name = API-Job-Executor-2; called 
by
> -TransactionLegacy.rollback:879-TransactionLegacy.removeUpTo
> :822-TransactionLegacy.close:646-Transaction.execute:43-Tra
> nsaction.execute:47-NetworkOrchestrator.releaseNic:1600-Netw
> orkOrchestrator.release:1588-UserVmManagerImpl.releaseNetwo
> rkResourcesOnExpunge:2102-UserVmManagerImpl.expunge:
> 2050-UserVmManagerImpl.destroyVm:2636-NativeMethodAccessorImpl.invok
> e0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-02-14 17:48:56,156 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329) (logid:2eed47c7) Unexpected
> exception while executing org.apache.cloudstack.api.comm
> and.user.vm.DestroyVMCmd
> java.lang.NullPointerException
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrat
> or$6.doInTransaction(NetworkOrchestrator.java:1631)
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrat
> or$6.doInTransaction(NetworkOrchestrator.java:1600)
> at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction
> .java:50)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrat
> or.releaseNic(NetworkOrchestrator.java:1600)
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrat
> or.release(NetworkOrchestrator.java:1588)
> at com.cloud.vm.UserVmManagerImpl.releaseNetworkResourcesOnExpu
> nge(UserVmManagerImpl.java:2102)
> at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.
> java:2050)
> at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.
> java:2636)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
> ssorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
> thodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsin
> gReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:150)
> at org.apache.cloudstack.network.contrail.management.EventUtils
> $EventInterceptor.invoke(EventUtils.java:106)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:161)
> at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInt
> erceptor.java:51)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:161)
> at 

Re: Adding new host with VMware hypervisor

[Sergey Levitskiy]

With vmware clusters the behavior is the following:
1. When a new cluster is added all ESX hosts in the cluster are automatically 
discovered and added to ACS with no host tag. If a tag is needed it needs to be 
added to each host individually
2.  To add an additional host to a cluster
a. Add host in vSphere matching network config of other hosts e.g. if vDs is 
used new host needs to be added to vDS before proceeded to the next step
b. All hosts in the cluster should have the same name for Management portgroup 
and match ACS global setting ‘vmware.managment.portgroup’. The default is 
‘Management Network’
c. Add host into cluster in ACS using host name as seen in vCenter client. A 
host tag can be specified at that time. To authenticate to the host ACs will 
use vSphere credentials cached for the cluster. 

Re: Adding new host with VMware hypervisor

[Sergey Levitskiy]

All should work after the addition without any further changes. 

Re: unable to take the Database dump

[Sergey Levitskiy]

You pretty much OK with the mysqldump options you use. ACS doesn't use MySQL 
events or triggers so you should be good with all-databases flag. 

Re: unable to take the Database dump

[Sergey Levitskiy]

Easiest way to first create cloud user at the destination server and then 
import the backup

GRANT ALL ON cloud.* to cloud@`localhost` identified by 'complex_password ';
GRANT ALL ON cloud.* to cloud@`%` identified by 'complex_password';
GRANT ALL ON cloud_usage.* to cloud@`localhost`;
GRANT ALL ON cloud_usage.* to cloud@`%`;
GRANT process ON *.* TO cloud@`localhost`;
GRANT process ON *.* TO cloud@`%`;
FLUSH PRIVILEGES;

Re: Volume migration unable to serialize

[Sergey Levitskiy]

You need to adjust this threshold

job.cancel.threshold.minutes



Thanks,
Sergey
 

Re: Error when create template Cloudstack 4.8 windows/xen

[Sergey Levitskiy]

Looks like you have intermittent DB communication issue. Is the issue 
reproducible every time you create a template? Do you have DB server behind 
firewall or on a different network?


... 58 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: 
Communications link failure

The last packet successfully received from the server was 203 608 milliseconds 
ago.  The last packet sent successfully to the server was 0 milliseconds ago.

Re: Disaster Recovery - VM- Cloudstack

[Sergey Levitskiy]

If you use VMware as a hypervisor then it is easily integrated with 3-d party 
solutions like Veeam and Commvault. Veeam supports backup based on VM tag so 
defining a tag for VM/template in ACS will automatically sign up a VM for the 
backup policy.
Restore over existing VM is not a problem. The only challenging part if VM is 
destroyed and expunged and needs recovering. Some DB ‘magic’ will be required 
for this part including potential IP change.

Re: Fault percentage value of CPU usage in Cloud Platform

[Sergey Levitskiy]

First of all, if you don’t have support for CCP you need to strongly consider 
migrating to ACP. This community doesn’t have access to CCP code so we have 
very limited way to assist you.
In any case looking at the log you provided it seems you have 2 cluster: one 
(cluster id =5) that crossed your allocation threshold for capacity CPU. The 
other cluster (id=1) simply has no host matching host tag of your service 
offering (“WinL”) so that cluster can’t be allocated either.
Can you execute this SQL query and post results back:

SELECT 
`c`.`uuid` AS `cluster_uuid`,
`c`.`id` AS `cluster_id`,
`c`.`name` AS `cluster_name`,
`d`.`uuid` AS `zone_uuid`,
`d`.`name` AS `zone_name`,
IF((`capacity`.`capacity_type` = 1),
'cpu',
'ram') AS `type`,
((SUM(`capacity`.`used_capacity`) + 
SUM(`capacity`.`reserved_capacity`)) + 0) AS `used_capacity`,
SUM((`capacity`.`total_capacity` * `overcommit`.`value`)) AS 
`total_capacity`,
FORMATSUM(`capacity`.`used_capacity`) + 
SUM(`capacity`.`reserved_capacity`)) + 0) / SUM((`capacity`.`total_capacity` * 
`overcommit`.`value`))),
2) AS `utilization`,
(CASE (SELECT 
COUNT(0)
FROM
`cluster_details` `details`
WHERE
((`details`.`cluster_id` = `capacity`.`cluster_id`)
AND (`details`.`name` = 
'cluster.cpu.allocated.capacity.disablethreshold')))
WHEN
1
THEN
(CASE
WHEN
ISNULL((SELECT 
`details`.`value`
FROM
`cluster_details` `details`
WHERE
((`details`.`cluster_id` = 
`capacity`.`cluster_id`)
AND (`details`.`name` = 
'cluster.cpu.allocated.capacity.disablethreshold'
THEN
(SELECT 
`config`.`value`
FROM
`configuration` `config`
WHERE
(`config`.`name` = 
'cluster.cpu.allocated.capacity.disablethreshold'))
ELSE (SELECT 
`details`.`value`
FROM
`cluster_details` `details`
WHERE
((`details`.`cluster_id` = `capacity`.`cluster_id`)
AND (`details`.`name` = 
'cluster.cpu.allocated.capacity.disablethreshold')))
END)
ELSE (SELECT 
`config`.`value`
FROM
`configuration` `config`
WHERE
(`config`.`name` = 
'cluster.cpu.allocated.capacity.disablethreshold'))
END) AS `threshold`,
(CASE (SELECT 
COUNT(0)
FROM
`cluster_details` `details`
WHERE
((`details`.`cluster_id` = `capacity`.`cluster_id`)
AND (`details`.`name` = 'cpuOvercommitRatio')))
WHEN
1
THEN
(CASE
WHEN
ISNULL((SELECT 
`details`.`value`
FROM
`cluster_details` `details`
WHERE
((`details`.`cluster_id` = 
`capacity`.`cluster_id`)
AND (`details`.`name` = 
'cpuOvercommitRatio'
THEN
(SELECT 
`config`.`value`
FROM
`configuration` `config`
WHERE
(`config`.`name` = 'cpuOvercommitRatio'))
ELSE (SELECT 
`details`.`value`
FROM
`cluster_details` `details`
WHERE
((`details`.`cluster_id` = `capacity`.`cluster_id`)
AND (`details`.`name` = 'cpuOvercommitRatio')))
END)
ELSE (SELECT 
`config`.`value`
FROM
`configuration` `config`
WHERE
(`config`.`name` = 'cpuOvercommitRatio'))
END) AS `overprovisioning`
FROM
`op_host_capacity` `capacity`
JOIN `cluster_details` `overcommit` ON ((`overcommit`.`cluster_id` = 
`capacity`.`cluster_id`)))
JOIN `cluster` `c` ON ((`c`.`id` = 

Re: Basic and Advanced Zone

[Sergey Levitskiy]

Unfortunately, this is not good enough. You need to keep all log entries in the 
interval when VM was being deployed. You can simply archive and post link to 
you full log and indicate timeframe the failure happened.   

Re: Instance migration

[Sergey Levitskiy]

I might be incorrect stating KVM live migration is not supported. Per 4.9 admin 
guide it is. There are some limitations but go ahead and try. 
http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/virtual_machines.html#moving-vms-between-hosts-manual-live-migration
 

Re: Instance migration

[Sergey Levitskiy]

Live MV migration is supported on Xen and Vmware hypervisors. PR1709 is open to 
support live migration with KVM but it is not merged. What hypervisor do you 
use?

 

Re: Basic and Advanced Zone

[Sergey Levitskiy]

You log is not complete. Can you extract the log that covers full period of job 
execution and publish it?

Re: Snapshot fails on manually expanded volume

[Sergey Levitskiy]

We run 100% on vmware so I just try few guesses. On vmware volume resize is 
allowed only for full clones so if you run Xen with link clones this might 
cause volume corruption. From mgmt. server log it looks like the Xen task to 
take an intermediate VM snapshot fails so may be you can try to manually 
execute it and see what error you get. Increasing verbosity of Xen log might 
help as well as temporary enabling TRACE log level for ACS to show additional 
info on what exactly is failing.

Re: SSVM NFS Access Denied Problems

[Sergey Levitskiy]

Export NFS share so that root can mount it. Also you can try manually mount it 
from management server and see if you can write to it

Sent from my iPhone

> On Nov 12, 2016, at 10:54 AM, Asai  wrote:
> 
> Greetings,
> 
> Going a little nuts here.  I've been attempting to create an advanced zone 
> with my Secondary Storage on a separate NFS server running NAS4Free.  The 
> problem is I keep getting an access denied while trying to mount error and I 
> cannot figure out why this is. The directory is blank on the NFS server, 
> permissions are set to 777, All Dirs option is enabled, allowed networks are 
> set to allow the same subnet, both the Cloudstack MGMT network and secondary 
> storage server are on the same subnet, but I can't seem to figure this out... 
>  does anyone have any brilliant insights into this maddening problem?
> 
> Thanks!!!
> 

Re: Unable to start management

[Sergey Levitskiy]

Can you zip and post full management server log?
Can you run cloudstack-sccs and post the value reported?
Looks like upgrade procedure doesn’t correctly determine your version because 
schema-40to410.sql shouldn’t get executed during upgrade from 4.6 to 4.9


 

Re: Unable to start management

[Sergey Levitskiy]

What version are you upgrading from ?


On 11/10/16, 6:26 PM, "Abdul Qayyum Halid"  wrote:

Hi, I am having problem to start CS management. The error show unable to
upgrade databases. MySQL version 5.7, CS 4.9. Any solution? Below are the
error.Thank you

ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: alter
table vm_template add size bigint unsigned
2016-11-11 10:18:53,229 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:)
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
name 'size'
2016-11-11 10:18:53,229 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null)
(logid:) Unable to execute upgrade script:
/usr/share/cloudstack-management/setup/db/schema-40to410.sql
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
name 'size'
2016-11-11 10:18:53,233 WARN  [o.a.c.s.m.c.ResourceApplicationContext]
(main:null) (logid:) Exception encountered during context initialization -
cancelling refresh attempt:
org.springframework.context.ApplicationContextException: Failed to start
bean 'cloudStackLifeCycle'; nested exception is
com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the
database

Re: Create VM's with 2 cores 2sockets

[Sergey Levitskiy]

I believe you don’t need to do anything on hypervisor part any more. On VMware 
it works as described below and should work the same for other hypervisors.
You can configure it on per VM or per template level by inserting the following 
configuration into user_vm_details or vm_template_details table respectively

cpuid.coresPerSocket   
   
If you do it for the template level all VMs built from it will have this 
setting and it will be passed further to new templates if they are created.

At this point this configuration can’t be edited in ACS UI and it is rather 
hard to do via API. There is Jira ticket and PR to address it in one of the 
upcoming releases. Until then you can use the following SQL (or similar) to 
manipulate cores/socket ratio. (always test your DB changes in the Lab)

set @vmcores=2;
set @vmuuid='d550e7d6-2515-4df9-9686-6daaf3c5a7d1';

delete vd from cloud.user_vm_details vd join cloud.vm_instance i on 
i.id=vd.vm_id where  i.uuid=@vmuuid and vd.name='cpuid.coresPerSocket';
insert into cloud.user_vm_details (vm_id,`name`, `value`) (select 
i.id,'cpuid.coresPerSocket',@vmcores from cloud.vm_instance i where  
i.uuid=@vmuuid)

After it is updated you would need to switch service offering with 4 CPUs and 
stop/start your VM.

Thanks,
Sergey




On 11/7/16, 8:34 AM, "Stephan Seitz"  wrote:

AFAIK and only speaking for the hypervicsor, this feature requires a
XenServer license
https://support.citrix.com/article/CTX126524


Am Montag, den 07.11.2016, 15:31 + schrieb Jeremy Peterson:
> I have a CS environment setup and I'd like to know how to call
> CloudStack to create a VM with sockets and core distinction.
> 
> Right now it seems to be created all vms with x number of sockets
> with 1 core per socket.
> 
> Hypervisor : XenServer
> CS Version : 4.5.0
> 
> Any ideas?
> 
> Jermy

Re: Long delay in listDomains API response

[Sergey Levitskiy]

What essentially happens here that for each domain, you have access with your 
key, ACS executes 12 queries to get domain limits.  Even if such query for each 
resource limit takes 0.03 sec it translates into 0.4 sec for each domain. If 
you have more than few domains, you will see rather lengthy total execution of 
listDomains in spite of some parallel execution of such queries. This is 
apparently far from optimal but I don’t see if it can be improved by optimizing 
MySQL alone.   I looked through the ACS code and there is a way to improve this 
big time since all resource limits are already in domain_view . We will try to 
fix it soon for 4.9-4.10 versions. It usually takes at least 2-3 weeks before 
the code will be reviewed and committed. Until then I suggest to increase any 
client timeouts accordingly.

Thanks,
Sergey


On 11/6/16, 5:52 PM, "Cloud List" <cloud-l...@sg.or.id> wrote:

Hi Wei,

In addition, can you also advice any specific tweak or tuning of the
CloudStack management server and MySQL database's configurations to improve
the time it takes to perform the listDomains API query and result? We have
tried CloudStack and MySQL tunings as per below documentations and it
doesn't seem to improve the listDomains API query time.


http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.8/tuning.html

http://events.linuxfoundation.org/sites/events/files/slides/CloudSTack-Tuning-Collab-2015.pdf

https://www.percona.com/blog/2014/01/28/10-mysql-performance-tuning-settings-after-installation/

https://www.percona.com/blog/2012/01/06/mysql-high-number-connections-per-secon/

Any advice on this is greatly appreciated.

Looking forward to your reply, thank you.

Cheers.



On Mon, Nov 7, 2016 at 2:35 AM, Cloud List <cloud-l...@sg.or.id> wrote:

> Hi Wei,
>
> Thanks for your reply. Is it possible for us to remove the resource
> limitation feature -- any global settings that we can set to turn this 
off?
> If not, would you be able to let us know where on the source code this is
> being set, so that we can try changing it and recompile ACS to turn off 
the
> feature?
>
> If it's possible, how would be the impact to CloudStack operations in
> general? May I know what was the original purpose the limitation is being
> put in the first place?
>
> Thanks for the workaround recommendation -- will forward to our
> application vendor to see if the app can be fine-tuned following your
> recommendation.
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
> On Mon, Nov 7, 2016 at 12:42 AM, Wei ZHOU <ustcweiz...@gmail.com> wrote:
>
>> I cannot say it is unnecessary or not for now.
>> There are indeed many db connections when get resource limtiation/count
>> of domains.
>>
>> A workaound is (1) listDomainChildren gets id of domain and sub-domains,
>> then (2) listDomains with specified id.
>>
>> -Wei
>>
>>
>>
>> 2016-11-06 17:23 GMT+01:00 Cloud List <cloud-l...@sg.or.id>:
>>
>>> Hi Sergey,
>>>
>>> Thanks for your reply. The grep result is a bit long, total 64K lines,
>>> so I have uploaded it here:
>>>
>>> https://dl.dropboxusercontent.com/u/24788636/ctx-3f3ff343.txt
>>>
>>> It seems there are a lot of unnecessary creating and closing of
>>> connections to the DB, causing the delay?
>>>
>>> Looking forward to your reply, thank you.
>>>
>>> Cheers.
>>>
>>>
>>> On Sun, Nov 6, 2016 at 11:30 PM, Sergey Levitskiy <
>>> sergey.levits...@autodesk.com> wrote:
>>>
>>>> Can you grep for this line ‘ctx-3f3ff343’ in your log and post it here?
>>>>
>>>>
>>>> On 11/5/16, 11:01 PM, "Cloud List" <cloud-l...@sg.or.id> wrote:
>>>>
>>>> Dear all,
>>>>
>>>> After researching further, I found out that our problem is exactly
>>>> the one
>>>> reported on this JIRA bug:
>>>>
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9405
>>>>
>>>> I am cc-ing dsclose and Wei Zhou on this email. Anyone know if
>>>> there's any
>>>> resolution to the above problem/bug?
>>>>
>>>> On the bug report, Wei Zhou mentioned that he added domain level
>>&

Re: Long delay in listDomains API response

[Sergey Levitskiy]

ring_id,
> vm_instance.reservation_id, vm_instance.hypervisor_type,
> vm_instance.dynamically_scalable, vm_instance.uuid,
> vm_instance.disk_offering_id, vm_instance.power_state,
> vm_instance.power_state_update_time, vm_instance.power_state_update_count,
> vm_instance.power_host, user_vm.iso_id, user_vm.user_data,
> user_vm.display_name, user_vm.update_parameters FROM user_vm INNER JOIN
> vm_instance ON user_vm.id=vm_instance.id  WHERE vm_instance.type='User'
> AND user_vm.id = ?  AND vm_instance.removed IS NULL
> 2016-11-06 09:36:04,902 TRACE [c.c.u.d.T.Transaction]
> (http-8080-10:ctx-3f3ff343 ctx-e486fd2b ctx-f5ec429d) (logid:0429eaf0)
> Using current transaction: http-8080-10 : http-8080-10,
> 2016-11-06 09:36:04,903 TRACE [c.c.u.d.T.Statement] 
(StatsCollector-2:ctx-293a2a9e)
> (logid:725b6dd7) Closing: com.mysql.jdbc.JDBC4PreparedStatement@107795cb:
> SELECT vm_instance.id, vm_instance.name, vm_instance.vnc_password,
> vm_instance.proxy_id, vm_instance.proxy_assign_time, vm_instance.state,
> vm_instance.private_ip_address, vm_instance.instance_name,
> vm_instance.vm_template_id, vm_instance.guest_os_id, vm_instance.host_id,
> vm_instance.last_host_id, vm_instance.pod_id, 
vm_instance.private_mac_address,
> vm_instance.data_center_id, vm_instance.vm_type, vm_instance.ha_enabled,
> vm_instance.display_vm, vm_instance.limit_cpu_use,
> vm_instance.update_count, vm_instance.created, vm_instance.removed,
> vm_instance.update_time, vm_instance.domain_id, vm_instance.account_id,
> vm_instance.user_id, vm_instance.service_offering_id,
> vm_instance.reservation_id, vm_instance.hypervisor_type,
> vm_instance.dynamically_scalable, vm_instance.uuid,
> vm_instance.disk_offering_id, vm_instance.power_state,
> vm_instance.power_state_update_time, vm_instance.power_state_update_count,
> vm_instance.power_host, user_vm.iso_id, user_vm.user_data,
> user_vm.display_name, user_vm.update_parameters FROM user_vm INNER JOIN
> vm_instance ON user_vm.id=vm_instance.id  WHERE vm_instance.type='User'
> AND user_vm.id = 4221  AND vm_instance.removed IS NULL
> 2016-11-06 09:36:04,904 TRACE [c.c.u.d.T.Connection] 
(StatsCollector-2:ctx-293a2a9e)
> (logid:725b6dd7) Closing DB connection: dbconn1632185236
>
>
>
> On Sun, Nov 6, 2016 at 9:37 AM, Sergey Levitskiy <
> sergey.levits...@autodesk.com> wrote:
>
>> Please change DEBUG to TRACE for the following classes so they look like
>> below. You don’t need to restart Cloudstack since it will dynamically
>> switch log level. Once switched it will start generating enormous amount 
of
>> logging so once you capture the slowness switch it back to DEBUG.
>>
>>   
>>  
>>
>>
>>
>>   
>>
>>
>>
>> Thanks,
>> Sergey
>>
>>
>>
>>
>> On 11/5/16, 6:11 PM, "Cloud List" <cloud-l...@sg.or.id> wrote:
>>
>> Hi Sergey,
>>
>> Thanks for your reply. May I know logging for which class I need to
>> modify
>> to TRACE? Is it for the c.c.a.ApiServelet class? Not too sure what
>> does the
>> c.c.a stands for -- how do I check that? So I can put the correct
>> full name
>> of the class category on /etc/cloudstack/management/log4j-cloud.xml.
>>
>> Looking forward to your reply, thank you.
>>
>> Cheers.
>>
>>
>> On Sun, Nov 6, 2016 at 5:50 AM, Sergey Levitskiy <
>> sergey.levits...@autodesk.com> wrote:
>>
>> > You will probably need to enable TRACE logging and see what SQL
>> query took
>> > long time to execute.
>> >
>> > On 11/5/16, 1:21 PM, "Cloud List" <cloud-l...@sg.or.id> wrote:
>> >
>> > Hi,
>> >
>> > We have upgraded ACS from 4.2 to 4.8.1.1. We noted that a
>> > "listDomains" API
>> > call from our billing system took 21 seconds for our CloudStack
>> to
>> > reply,
>> > resulting in time-out.
>> >
>> > 
>> > 2016-11-06 03:40:42,891 DEBUG [c.c.a.ApiServlet]
>> > (http-8080-2:ctx-faa0bcbd)
>> > (logid:86b581fc) ===START===  X.X.X.X -- GET
>> > apikey=XX=
>> > listDomains=json=YYY
>> > 2016-11-06 03:41:03,850 DEBUG [c.c.a.ApiServlet]
>> > (http-8080-2:ctx-faa0bcbd
>> > ctx-5653f761 ctx-cb1e1f59) (logid:86b581fc) ===END===  X.X.X.X
>> -- GET
>> > apikey=XX=
>> > listDomains=json=YYY
>> > 
>> >
>> > Any advice on what caused the delay in the API reply and how we
>> can
>> > expedite?
>> >
>> > Thank you.
>> >
>> >
>> >
>>
>>
>>
>

Re: Long delay in listDomains API response

[Sergey Levitskiy]

Please change DEBUG to TRACE for the following classes so they look like below. 
You don’t need to restart Cloudstack since it will dynamically switch log 
level. Once switched it will start generating enormous amount of logging so 
once you capture the slowness switch it back to DEBUG.

  
 
   
   
   
  
   


Thanks,
Sergey




On 11/5/16, 6:11 PM, "Cloud List" <cloud-l...@sg.or.id> wrote:

Hi Sergey,

Thanks for your reply. May I know logging for which class I need to modify
to TRACE? Is it for the c.c.a.ApiServelet class? Not too sure what does the
c.c.a stands for -- how do I check that? So I can put the correct full name
of the class category on /etc/cloudstack/management/log4j-cloud.xml.

Looking forward to your reply, thank you.

Cheers.


On Sun, Nov 6, 2016 at 5:50 AM, Sergey Levitskiy <
sergey.levits...@autodesk.com> wrote:

> You will probably need to enable TRACE logging and see what SQL query took
> long time to execute.
>
> On 11/5/16, 1:21 PM, "Cloud List" <cloud-l...@sg.or.id> wrote:
>
> Hi,
>
> We have upgraded ACS from 4.2 to 4.8.1.1. We noted that a
> "listDomains" API
> call from our billing system took 21 seconds for our CloudStack to
> reply,
> resulting in time-out.
>
> 
> 2016-11-06 03:40:42,891 DEBUG [c.c.a.ApiServlet]
> (http-8080-2:ctx-faa0bcbd)
> (logid:86b581fc) ===START===  X.X.X.X -- GET
> apikey=XX=
> listDomains=json=YYY
> 2016-11-06 03:41:03,850 DEBUG [c.c.a.ApiServlet]
> (http-8080-2:ctx-faa0bcbd
> ctx-5653f761 ctx-cb1e1f59) (logid:86b581fc) ===END===  X.X.X.X -- GET
> apikey=XX=
> listDomains=json=YYY
> 
>
> Any advice on what caused the delay in the API reply and how we can
> expedite?
>
> Thank you.
>
>
>

Re: Long delay in listDomains API response

[Sergey Levitskiy]

You will probably need to enable TRACE logging and see what SQL query took long 
time to execute.

On 11/5/16, 1:21 PM, "Cloud List"  wrote:

Hi,

We have upgraded ACS from 4.2 to 4.8.1.1. We noted that a "listDomains" API
call from our billing system took 21 seconds for our CloudStack to reply,
resulting in time-out.


2016-11-06 03:40:42,891 DEBUG [c.c.a.ApiServlet] (http-8080-2:ctx-faa0bcbd)
(logid:86b581fc) ===START===  X.X.X.X -- GET

apikey=XX=listDomains=json=YYY
2016-11-06 03:41:03,850 DEBUG [c.c.a.ApiServlet] (http-8080-2:ctx-faa0bcbd
ctx-5653f761 ctx-cb1e1f59) (logid:86b581fc) ===END===  X.X.X.X -- GET

apikey=XX=listDomains=json=YYY


Any advice on what caused the delay in the API reply and how we can
expedite?

Thank you.

Re: Good backup solutions for Cloudstack

[Sergey Levitskiy]

Veeam works OK for VMware based implementations. You can tag VMs  and based on 
vsphere tag Veeam will automatically pick them up for the backup processing.

On 11/2/16, 4:21 PM, "Asai"  wrote:

Hello,

Can anyone recommend a good backup solution for a Cloudstack deployment?  
What’s the best way of backing up VMs and snapshots?  I have experience with 
XenServer, but I’m moving into a CS deployment now and am looking for 
recommendations on best practices.

Thanks
Asai
Network and Systems Administrator
GLOBAL CHANGE MEDIA
http://globalchange.media
Tucson, AZ

Re: Error while decrypting after reinstall mgmt server

[Sergey Levitskiy]

Actually both file /etc/cloudstack/management/key  as well as value of 
db.cloud.encrypt.secret=  from /etc/cloudstack/management/db.properties  
are used in different areas.
If you changed any of them from default it  means certain values in DB are 
encrypted and key is lost

One option is this
Install brand new ACS
Copy /etc/coudstack/managment/key and value of of db.cloud.encrypt.secret and 
db.cloud.encrypt.type from db.properties  is original installation

Copy values of encrypted fields as well.

All possible values are here

SELECT name, value FROM cloud.configuration where (category = 'hidden' or  
category = 'secure' ) and value is not null
But you can gnore  some e.g. ldap if you don’t use it.





On 10/18/16, 1:47 AM, "Lotic Lists"  wrote:

The LUN was lost with all data :(

I have database backup but key file not

Re: replace secondary storage

[Sergey Levitskiy]

Easiest way is rsync the whole folder structure to new secondary storage, then 
change the URL of SS in image_store table, and then restart all SSVMs, proxies 
and vrouters in this zone

Sent from my iPhone

Re: Public Templates

[Sergey Levitskiy]

Public and featured attribute doesn't affect template lifecycle. In short a 
template will reside on a secondary storage until explicitly deleted.

Sent from my iPhone

Re: Adding an ISO to Cloudstack but missing download ISO button

[Sergey Levitskiy]

You need to check route from SSVM to the website hosting the ISO over ACS 
public network. SSH to SSVM using instructions here 
http://stackoverflow.com/questions/19812463/how-to-ssh-into-cloudstack-ssvm-through-public-ip-address
and try to:
1. Check name resolution
2. Check basic connectivity e.g. PING
3. Try to curl to ISO to see if you can establish HTTP session

Re: HA-enabled VM

[Sergey Levitskiy]

As it turned out on existing VMs it is not enough to change it service offering 
since the setting persists during the VM creation and remains the same.
You need to update it for each VM
You can use this sample to update a given service offering and all active VMs 
that were started up from it to be HA enabled (rerplace X with a name of 
your service offering).  Then stop/start all VMs to activate this:

update cloud.vm_instance i  
join cloud.disk_offering do on do.id=i.service_offering_id 
join service_offering so on so.id=do.id
set i.ha_enabled=1, so.ha_enabled=1
where i.removed is null and do.name = 'X'




On 10/8/16, 7:19 AM, "Gian Paolo Buono" <gianpaolo.bu...@gesca.it> wrote:

I've  tried it but not work..any other suggestions?
 bye
On 10/07/2016 02:46 PM, Sergey Levitskiy wrote:

You can find all service offerings used by those VM and change in 
service_offereing table a field ha_enabled to ’1’. After that Stop/Start VMs 
should make them HA.


On 10/7/16, 5:40 AM, "Gian Paolo Buono" 
<gianpaolo.bu...@gesca.it><mailto:gianpaolo.bu...@gesca.it> wrote:

Thanks Boris,

It works.. The problem is that I  have 100 vm without HA, there is a
charm to enable the HA to all vm  ?

Thanks

On 10/07/2016 02:18 PM, Boris Stoyanov wrote:
> Hi Gian Paolo,
>
> In order to have a VM in HA Enabled state you need to create that VM 
with Compute Offering that has “Offer HA” checked.
>
> Thanks,
> Boris Stoyanov
> boris.stoya...@shapeblue.com<mailto:boris.stoya...@shapeblue.com>
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
> On Oct 7, 2016, at 3:01 PM, Gian Paolo Buono 
<gianpaolo.bu...@gesca.it><mailto:gianpaolo.bu...@gesca.it> wrote:
>> Hi all,
>>
>> how can i set HA Enabled to yes, in the istance of Vm ? If can't 
modify
>> that parameter.
>>
>> Thanks

Re: HA-enabled VM

[Sergey Levitskiy]

You can find all service offerings used by those VM and change in 
service_offereing table a field ha_enabled to ’1’. After that Stop/Start VMs 
should make them HA.


On 10/7/16, 5:40 AM, "Gian Paolo Buono"  wrote:

Thanks Boris,

It works.. The problem is that I  have 100 vm without HA, there is a 
charm to enable the HA to all vm  ?

Thanks

On 10/07/2016 02:18 PM, Boris Stoyanov wrote:
> Hi Gian Paolo,
>
> In order to have a VM in HA Enabled state you need to create that VM with 
Compute Offering that has “Offer HA” checked.
>
> Thanks,
> Boris Stoyanov
> boris.stoya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>   
>
> On Oct 7, 2016, at 3:01 PM, Gian Paolo Buono  
wrote:
>> Hi all,
>>
>> how can i set HA Enabled to yes, in the istance of Vm ? If can't modify
>> that parameter.
>>
>> Thanks

Re: Implicit dedication vs Explicit dedication

[Sergey Levitskiy]

To do so you can use a specific service offerring.

Sent from my iPhone

> On Oct 5, 2016, at 7:14 AM, cs user  wrote:
> 
> Hi Rafael,
> 
> Thanks for responding. It is possible to force them to only be able to use
> that pod however?
> 
> So that when the create a machine, these machines are forced to launch in
> that pod.
> 
> Cheers
> 
> On Wed, Oct 5, 2016 at 3:11 PM, Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
> 
>> You can dedicate a pod to a user account or domain.
>> Have you tried that?
>> 
>>> On Wed, Oct 5, 2016 at 11:09 AM, cs user  wrote:
>>> 
>>> Hi All,
>>> 
>>> I asked this a while back and never got a response. With the latest
>> version
>>> of Cloudstack, is it possible to force an account to launch in a
>> particular
>>> pod?
>>> 
>>> Cheers!
>>> 
 On Mon, Mar 21, 2016 at 12:59 PM, cs user  wrote:
 
 Hi All,
 
 Question regarding dedicating a Pod to an account..
 
 If I dedicate a Pod to an account, login as that account and then
>> create
 an instance, when it comes to the affinity screen, I get the following:
 
 "Please select any affinity groups you want this VM to belong to"
 
 However, what I really want to do is force this account to only be able
>>> to
 create hosts in this pod. If this is not possible then it should
>> default
>>> to
 this being ticked. Is there a way to accomplish this?
 
 I've also tried creating a new service offering and creating it with
>>> ImplicitDedicationPlanner,
 with planner mode strict. However, all this appears to do is the
 following..
 
 When the account selects this service offering, the instance is created
>>> on
 an empty host. Addition instances created with this service offering
>> then
 get put onto the same host, rather than looking for other free hosts
>>> within
 the same pod.
 
 So to summarize I want to either force an account to only launch
 instances in one Pod, or for this option to be the default. Is there a
>>> way ?
 
 Thanks!
 
>>> 
>> 
>> 
>> 
>> --
>> Rafael Weingärtner
>> 

Re: CloudStack Anti-Affinity

[Sergey Levitskiy]

Hi Ilya,
We tried it once. If I am not mistaken it is strict. The job will be accepted 
but end up in error.
Thanks,
Sergey



On 10/4/16, 3:43 PM, "ilya"  wrote:

I'm curious if anyone uses Anti-Affinity and how restrictive is it?
Meaning what happens if there are no nodes to satisfy the anti-affinity
rule? Would it still place it or deny and throw 530?

Any help is appreciated

Thanks,
ilya

Re: Issue with template

[Sergey Levitskiy]

With centos 6 you need to remove line for respective interface from
/etc/udev/rules.d/70-persistent-net.rules
Centos/rhel6 auto detects interface and creates a rules to associate a 
particular MAC address with interface name. To creating clones from template 
you need to generalize setup so that detection happens for every new VM

Sent from my iPhone

On Sep 21, 2016, at 2:20 AM, Glenn Wagner 
> wrote:

Hi

Just check in your template the eth0 has the UUID and Mac addresses lines 
removed


Glenn



glenn.wag...@shapeblue.com
www.shapeblue.com
First Floor, Victoria Centre, 7 Victoria Street, Somerset West, Cape Town  
7129South Africa
@shapeblue




-Original Message-
From: Mohd Zainal Abidin Rabani [mailto:zai...@nocser.net]
Sent: Wednesday, 21 September 2016 11:02 AM
To: users@cloudstack.apache.org
Subject: RE: Issue with template

I already follow from link given. Still getting same result.

-Original Message-
From: Glenn Wagner [mailto:glenn.wag...@shapeblue.com]
Sent: Wednesday, September 21, 2016 4:31 PM
To: users@cloudstack.apache.org
Subject: RE: Issue with template

Hi,

You can have a look at this document , this will help you

http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/templates.html

You are missing the part (Remove the udev persistent device rules)

Thanks
Glenn


glenn.wag...@shapeblue.com
www.shapeblue.com
First Floor, Victoria Centre, 7 Victoria Street, Somerset West, Cape Town  
7129South Africa @shapeblue




-Original Message-
From: Mohd Zainal Abidin Rabani [mailto:zai...@nocser.net]
Sent: Wednesday, 21 September 2016 10:15 AM
To: users@cloudstack.apache.org
Subject: Issue with template

Hi,



I'm creating template from instance that install using ISO. After finish 
install and reboot, I update OS from centos 5.6 to 6.8. Then I reboot again.
After reboot I shutdown the instance and create template from disk. After 
creating template finish and available for use I try create new instance from 
template that I created. After instance up and running using new template I saw 
eth0 not available. It show only eth1. So I need to change in 
/etc/sysconfig/network-scripts. Is this normal or I miss any step?



Thanks.

Re: How to: Snapshot

[Sergey Levitskiy]

There is an article about this
http://cloud-mate.org/2015/06/enable-snapshot-kvm-cloudstack/

It seems VM snapshots are not supported in KVM. You can try tweaking a table 
hypervisor_capabilites and set  vm_snapshot_enabled to 1 for hypervisor KVM but 
I doubt this will be successful. You probably are left with disk snapshots on 
KVM.





On 9/19/16, 9:27 PM, "Mohd Zainal Abidin Rabani" <zai...@nocser.net> wrote:

I already enable snapshot and restart cloudstack management. But when I try 
to do snapshot got error (refer link: https://postimg.org/image/uiz45le7r/). 
Any idea why?

-Original Message-
From: Mohd Zainal Abidin Rabani [mailto:zai...@nocser.net] 
Sent: Tuesday, September 20, 2016 12:07 PM
To: users@cloudstack.apache.org
Subject: RE: How to: Snapshot

Thanks, Sergey. Much help.

-Original Message-
    From: Sergey Levitskiy [mailto:sergey.levits...@autodesk.com] 
Sent: Tuesday, September 20, 2016 12:00 PM
To: users@cloudstack.apache.org
Subject: Re: How to: Snapshot

You might need to change global setting kvm.snapshot.enabled to True


On 9/19/16, 8:57 PM, "Mohd Zainal Abidin Rabani" <zai...@nocser.net> wrote:

After stopping a instance/VM's, there are no icon for snapshot also 
(refer link: https://postimg.org/image/3lthssprd/). There can only do snapshot 
on volume. 

-Original Message-
From: Mohd Zainal Abidin Rabani [mailto:zai...@nocser.net] 
Sent: Tuesday, September 20, 2016 11:55 AM
To: users@cloudstack.apache.org
Subject: RE: How to: Snapshot

From what I see, it’s impossible to do snapshot on KVM while 
instance/VM’s still running. It’s need to stop and the icon will appear after 
instance/VM’s completely stop.

 

From: Vivek Kumar [mailto:vivek.ku...@indiqus.com] 
Sent: Tuesday, September 20, 2016 11:50 AM
To: users@cloudstack.apache.org
Subject: Re: How to: Snapshot

 

No there is no icon , what hyperviser and cloud stack version are you 
using now ?

 

On Tue, Sep 20, 2016 at 9:14 AM, Mohd Zainal Abidin Rabani 
<zai...@nocser.net <mailto:zai...@nocser.net> > wrote:

Do you see any icon that can do VM’s snapshot from link below:

https://postimg.org/image/5ks6qm1j9/



From: Vivek Kumar [mailto:vivek.ku...@indiqus.com 
<mailto:vivek.ku...@indiqus.com> ]
Sent: Tuesday, September 20, 2016 11:40 AM
To: users@cloudstack.apache.org <mailto:users@cloudstack.apache.org> 
Subject: Re: How to: Snapshot



yes you can take it without shutting down the vm



On Tue, Sep 20, 2016 at 9:06 AM, Mohd Zainal Abidin Rabani 
<zai...@nocser.net <mailto:zai...@nocser.net>  <mailto:zai...@nocser.net 
<mailto:zai...@nocser.net> > > wrote:

From the link I saw need to stop VM’s. Is it possible to snapshot while 
VM’s still running??



From: Vivek Kumar [mailto:vivek.ku...@indiqus.com 
<mailto:vivek.ku...@indiqus.com>  <mailto:vivek.ku...@indiqus.com 
<mailto:vivek.ku...@indiqus.com> > ]
Sent: Tuesday, September 20, 2016 11:31 AM
To: users@cloudstack.apache.org <mailto:users@cloudstack.apache.org>  
<mailto:users@cloudstack.apache.org <mailto:users@cloudstack.apache.org> >
Subject: Re: How to: Snapshot



this link will help you.



https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Snapshots







On Tue, Sep 20, 2016 at 8:56 AM, Mohd Zainal Abidin Rabani 
<zai...@nocser.net <mailto:zai...@nocser.net>  <mailto:zai...@nocser.net 
<mailto:zai...@nocser.net> >  <mailto:zai...@nocser.net 
<mailto:zai...@nocser.net>  <mailto:zai...@nocser.net 
<mailto:zai...@nocser.net> > > > wrote:

How to do VM’s snapshot?



From: Vivek Kumar [mailto:vivek.ku...@indiqus.com 
<mailto:vivek.ku...@indiqus.com>  <mailto:vivek.ku...@indiqus.com 
<mailto:vivek.ku...@indiqus.com> >  <mailto:vivek.ku...@indiqus.com 
<mailto:vivek.ku...@indiqus.com>  <mailto:vivek.ku...@indiqus.com 
<mailto:vivek.ku...@indiqus.com> > > ]
Sent: Tuesday, September 20, 2016 11:23 AM
To: users@cloudstack.apache.org <mailto:users@cloudstack.apache.org>  
<mailto:users@cloudstack.apache.org <mailto:users@cloudstack.apache.org> >  
<mailto:users@cloudstack.apache.org <mailto:users@cloudstack.apache.org>

Re: How to: Snapshot

[Sergey Levitskiy]

You might need to change global setting 
kvm.snapshot.enabled to True


On 9/19/16, 8:57 PM, "Mohd Zainal Abidin Rabani"  wrote:

After stopping a instance/VM's, there are no icon for snapshot also (refer 
link: https://postimg.org/image/3lthssprd/). There can only do snapshot on 
volume. 

-Original Message-
From: Mohd Zainal Abidin Rabani [mailto:zai...@nocser.net] 
Sent: Tuesday, September 20, 2016 11:55 AM
To: users@cloudstack.apache.org
Subject: RE: How to: Snapshot

From what I see, it’s impossible to do snapshot on KVM while instance/VM’s 
still running. It’s need to stop and the icon will appear after instance/VM’s 
completely stop.

 

From: Vivek Kumar [mailto:vivek.ku...@indiqus.com] 
Sent: Tuesday, September 20, 2016 11:50 AM
To: users@cloudstack.apache.org
Subject: Re: How to: Snapshot

 

No there is no icon , what hyperviser and cloud stack version are you using 
now ?

 

On Tue, Sep 20, 2016 at 9:14 AM, Mohd Zainal Abidin Rabani 
 > wrote:

Do you see any icon that can do VM’s snapshot from link below:

https://postimg.org/image/5ks6qm1j9/



From: Vivek Kumar [mailto:vivek.ku...@indiqus.com 
 ]
Sent: Tuesday, September 20, 2016 11:40 AM
To: users@cloudstack.apache.org  
Subject: Re: How to: Snapshot



yes you can take it without shutting down the vm



On Tue, Sep 20, 2016 at 9:06 AM, Mohd Zainal Abidin Rabani 
   > > wrote:

From the link I saw need to stop VM’s. Is it possible to snapshot while 
VM’s still running??



From: Vivek Kumar [mailto:vivek.ku...@indiqus.com 
   > ]
Sent: Tuesday, September 20, 2016 11:31 AM
To: users@cloudstack.apache.org   
 >
Subject: Re: How to: Snapshot



this link will help you.



https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Snapshots







On Tue, Sep 20, 2016 at 8:56 AM, Mohd Zainal Abidin Rabani 
   >     > > > wrote:

How to do VM’s snapshot?



From: Vivek Kumar [mailto:vivek.ku...@indiqus.com 
   >     > > ]
Sent: Tuesday, September 20, 2016 11:23 AM
To: users@cloudstack.apache.org   
 >  
  
 > >
Subject: Re: How to: Snapshot



Hello Zainal,



There are two type of snapshot ,



1- VM's snapshot

2- disk snapshot



and yes you can take either VM's snapshot or disk snapshot from running vm's



Vivek Kumar
Virtualization and Cloud Consultant

 
IndiQus Technologies Pvt Ltd
A-98, LGF, C.R.Park, New Delhi - 110019
O +91 11 4055 1411 | M +91 7503460090
  www.indiqus.com   
  







On Tue, Sep 20, 2016 at 8:40 AM, Mohd Zainal Abidin Rabani 
   >     > >     >     > > > > wrote:

Hi,



Is it possible to do snapshot while instance is running? Or need to stop
instance and do snapshot? Please advice.



Thanks,

Tech support







--

Vivek Kumar
Virtualization and Cloud Consultant

 
IndiQus Technologies Pvt Ltd
A-98, 

Re: No suitable hosts found (ACS 4.5.2 Advanced )

[Sergey Levitskiy]

Your allocator works fine. The problem is that your SSVM can’t import OVA into 
vcenter or it can’t download OVA from URL 
‘http://download.cloud.com/releases/2.2.0/CentOS5.3-x86_64.ova’
Check if SSVM can communicate to vcenter over TCP/443. Check if there is an 
error in vcenter when you submit the deploy job. Also check SSVM log by 
executing this from management server:
sudo ssh -i /var/cloudstack/management/.ssh/id_rsa  -p 3922 root@IP_OF_SSVM cat 
/var/log/cloud.log



On 9/19/16, 1:38 AM, "Cristian Ciobanu"  wrote:

Hello All,

 

  I'm working on a ACS 4.5.2 with VMware 5.5 using Advanced
networking setup ( local storage), I don't have any issue till my first VM
deploy, please see my logs.

 

-I don't see any issue for Proxy console, SSVM, Router, I also run
the test on ssvm "for check ssvm" no issue found.

 






U4IsCAAB4cAAAEO9zcQB-AFAAdwQAeHNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgA
BSgAFdmFsdWV4cQB-AFMAAXZyABdjb20uY2xvdWQuZGMuRGF0YUNlbnRlcrLxCuun53i
-AgAAeHAA

management-server.log:2016-09-19 03:53:34,771 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-64:ctx-97809d3f
job-115/job-116) Publish async job-116 complete on message bus

management-server.log:2016-09-19 03:53:34,771 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-64:ctx-97809d3f
job-115/job-116) Wake up jobs related to job-116

management-server.log:2016-09-19 03:53:34,771 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-64:ctx-97809d3f
job-115/job-116) Update db status for job-116

management-server.log:2016-09-19 03:53:34,772 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-64:ctx-97809d3f
job-115/job-116) Wake up jobs joined with job-116 and disjoin all subjobs
created from job- 116

management-server.log:2016-09-19 03:53:34,775 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-64:ctx-97809d3f
job-115/job-116) Done executing com.cloud.vm.VmWorkStart for job-116

management-server.log:2016-09-19 03:53:34,775 INFO
[o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-64:ctx-97809d3f
job-115/job-116) Remove job-116 from job monitoring

management-server.log:2016-09-19 03:53:34,783 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-45:ctx-e9ced335 job-115
ctx-77ab93c4) Sync job-118 execution on object VmWorkJobQueue.17

management-server.log:2016-09-19 03:53:34,803 WARN
[o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-137019e3) Task (job-115) has been
pending for 135 seconds

management-server.log:2016-09-19 03:53:35,940 INFO
[o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-65:ctx-5592b640
job-115/job-118) Add job-118 into job monitoring

management-server.log:2016-09-19 03:53:35,945 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-65:ctx-5592b640
job-115/job-118) Executing AsyncJobVO {id:118, userId: 2, accountId: 2,
instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart,
cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAC
AAIAEXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAHBwcHBw
cHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNo
b2xkeHA_QAAADHcIEAF0AApWbVBhc3N3b3JkdAAcck8wQUJYUUFEbk5oZG1Wa1gz
QmhjM04zYjNKa3hw, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode: 0, result: null, initMsid: 345048629065, completeMsid: null,
lastUpdated: null, lastPolled: null, created: Mon Sep 19 03:53:34 EDT 2016}

management-server.log:2016-09-19 03:53:35,946 DEBUG
[c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-65:ctx-5592b640
job-115/job-118) Run VM work job: com.cloud.vm.VmWorkStart for VM 17, job
origin: 115

management-server.log:2016-09-19 03:53:35,947 DEBUG
[c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-65:ctx-5592b640
job-115/job-118 ctx-acb9e21f) Execute VM work job:
com.cloud.vm.VmWorkStart{"dcId":0,"rawParams":{"VmPassword":"rO0ABXQADnNhdmV
kX3Bhc3N3b3Jk"},"userId":2,"accountId":2,"vmId":17,"handlerName":"VirtualMac
hineManagerImpl"}

Re: ACS 4.9 Basic Networking on XEN 6.5 - Cannot create instances anymore

[Sergey Levitskiy]

It seems the allocator can't find any hosts with enough capacity. Check if all 
your hosts are in green state , host tag in service offering matches tags of 
your hosts and if you have enough free capacity.

2016-09-18 14:55:37,279 DEBUG [c.c.d.DeploymentPlanningManagerImpl] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50) (logid:c69c1ccd) Trying 
to allocate a host and storage pools from dc:1, pod:null,cluster:null, 
requested cpu: 1000, requested ram: 1073741824
2016-09-18 14:55:37,279 DEBUG [c.c.d.DeploymentPlanningManagerImpl] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50) (logid:c69c1ccd) Is 
ROOT volume READY (pool already allocated)?: No
2016-09-18 14:55:37,279 DEBUG [c.c.d.FirstFitPlanner] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50) (logid:c69c1ccd) 
Searching all possible resources under this Zone: 1
2016-09-18 14:55:37,279 DEBUG [c.c.d.FirstFitPlanner] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50) (logid:c69c1ccd) 
Listing clusters in order of aggregate capacity, that have (atleast one host 
with) enough CPU and RAM capacity under this Zone: 1
2016-09-18 14:55:37,282 DEBUG [c.c.d.FirstFitPlanner] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50) (logid:c69c1ccd) 
Removing from the clusterId list these clusters from avoid set: []
2016-09-18 14:55:37,288 DEBUG [c.c.d.DeploymentPlanningManagerImpl] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50) (logid:c69c1ccd) 
Checking resources in Cluster: 1 under Pod: 1
2016-09-18 14:55:37,288 DEBUG [c.c.a.m.a.i.FirstFitAllocator] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50 
FirstFitRoutingAllocator) (logid:c69c1ccd) Looking for hosts in dc: 1  pod:1  
cluster:1
2016-09-18 14:55:37,290 DEBUG [c.c.a.m.a.i.FirstFitAllocator] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50 
FirstFitRoutingAllocator) (logid:c69c1ccd) FirstFitAllocator has 0 hosts to 
check for allocation: []
2016-09-18 14:55:37,292 DEBUG [c.c.a.m.a.i.FirstFitAllocator] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50 
FirstFitRoutingAllocator) (logid:c69c1ccd) Found 0 hosts for allocation after 
prioritization: []
2016-09-18 14:55:37,292 DEBUG [c.c.a.m.a.i.FirstFitAllocator] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50 
FirstFitRoutingAllocator) (logid:c69c1ccd) Looking for speed=1000Mhz, Ram=1024
2016-09-18 14:55:37,292 DEBUG [c.c.a.m.a.i.FirstFitAllocator] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50 
FirstFitRoutingAllocator) (logid:c69c1ccd) Host Allocator returning 0 suitable 
hosts
2016-09-18 14:55:37,292 DEBUG [c.c.d.DeploymentPlanningManagerImpl] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50) (logid:c69c1ccd) No 
suitable hosts found
2016-09-18 14:55:37,292 DEBUG [c.c.d.DeploymentPlanningManagerImpl] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50) (logid:c69c1ccd) No 
suitable hosts found under this Cluster: 1
2016-09-18 14:55:37,294 DEBUG [c.c.d.DeploymentPlanningManagerImpl] 
(API-Job-Executor-27:ctx-4b92f4ea job-135 ctx-fac5dc50) (logid:c69c1ccd) Could 
not find suitable Deployment Destination for this VM under any clusters, 
returning.

Sent from my iPhone

On Sep 18, 2016, at 6:08 AM, Jeroen Keerl 
> wrote:

Hi,

I can’t create new instances anymore on my ACS 4.9 on XEN, with basic 
networking.
Last Thursday, this was possible without any issues.

Since then, I have tested 2 things:

-  Creating domains and accounts

-  Adding SSH auth to one VM I had created with an ISO
(So, no real issues, I’d assume)

However, after I destroyed the one VM mentioned above, I tried to set up a new 
one, so I could create a base template.

But alas: All methods I tried fail (API and Web)
To me it looks, like it is looking “in the wrong direction”, as it can’t seem 
to see my cluster, hosts etc.
However, looking under “Infrastructure” – all seems to be fine.

First “solution”: Reboot mgmt server and system vms: No success.


2016-09-18 14:55:37,115 DEBUG [c.c.a.ApiServlet] (catalina-exec-3:ctx-a5945e12) 
(logid:3dd43961) ===START===  192.168.254.142 -- GET  
command=deployVirtualMachine=json=baee96f3-fee4-4e18-9f44-428206d1b3f3=7662eb72-791c-11e6-9aa7-00155dfa0a0e=XenServer=083d3f86-4b95-44c8-8361-0e571890ae2a=07265180-791d-11e6-9aa7-00155dfa0a0e&_=1474203217288
2016-09-18 14:55:37,132 DEBUG [c.c.u.AccountManagerImpl] 
(catalina-exec-3:ctx-a5945e12 ctx-a27aa791) (logid:3dd43961) Access granted to 
Acct[07261698-791d-11e6-9aa7-00155dfa0a0e-admin] to 
org.apache.cloudstack.quota.vo.ServiceOfferingVO$$EnhancerByCGLIB$$7f648b5a@5724d0d6
 by AffinityGroupAccessChecker
2016-09-18 14:55:37,133 DEBUG [c.c.u.AccountManagerImpl] 
(catalina-exec-3:ctx-a5945e12 ctx-a27aa791) (logid:3dd43961) Access granted to 
Acct[07261698-791d-11e6-9aa7-00155dfa0a0e-admin] to null by 
AffinityGroupAccessChecker
2016-09-18 14:55:37,158 DEBUG [c.c.v.UserVmManagerImpl] 
(catalina-exec-3:ctx-a5945e12 ctx-a27aa791) (logid:3dd43961) 

Re: configdrive in ACS

[Sergey Levitskiy]

ACS uses a different system than configdrive. Userdata and metadata is saved on 
a vrouter and can be queried from the instance with basic HTTP call. It is very 
similar to AWS.
To save userdata you would use deployVirtualMachine or updateVirtualMachine call
Use this guide for details how format encode it:
http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.8/virtual_machines/user-data.html

Here it is an example how you would get VM ID and some random property from 
metadata and userdata assuming userdata has propertyname:”propertyvalue” style 
tag inside

if [[ $OS == "RHEL" || $OS == "CentOS" ]]; then
  DHCPBASE="/var/lib/dhclient/dhclient-*.lease"
elif [ $OS == "Ubuntu" ]; then
  DHCPBASE="/var/lib/dhcp/dhclient.*.leases"
fi

DHCPSERVER=`grep dhcp-server-identifier $DHCPBASE |tail -1|awk '{print $3}'|sed 
's/;//'`
IPADDRESS=`curl -s http://$DHCPSERVER/latest/local-ipv4`
PROVIDERID=`curl -s http://$DHCPSERVER/latest/vm-id|sed 
'/^i-.*/s/i-[0-9]*-//'|sed 's/-VM$//'|sed 
'/[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/s/-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}$//'`

PROPERTYVALUE=`curl -s http://$DHCPSERVER/latest/user-data|grep 
propertyname|cut -d":" -f2|cut -d'"' -f2`






On 9/11/16, 3:36 AM, "Helge Waastad"  wrote:

Hi,
Im testing out RancherOS in ACS and have a couple of issues.

First, it seems that RancherOS does not support cloudstack datasource for 
meta/userdata but that I need to take with Rancher guys.

But, in openstack I can always use configdrive to get userdata to my vm.

Is it possible to use configdrive in acs? (i have'nt had any luck yet)

Br hw




Sendt fra Galaxy Tab

Re: Templates inside Primary storage

[Sergey Levitskiy]

Templates get deleted from primary storage during the storage cleanup activity. 
There are few global settings that manages cleanup:
storage.cleanup.enabled
storage.cleanup.interval
storage.template.cleanup.enabled

Actual deletion of the template copy on PS happens on 2d cleanup thread run. 
One the first run if template is identified to have no local volumes it is 
maked for GC (garbage collection) and on second run it is actually deleted 
assuming it is still not in use.

Thanks,
Sergey


On 9/6/16, 12:01 PM, "Ghaith Bannoura"  wrote:

Hello All,


Cloud stack copy templates from secondary storage to primary storage for 
the running instances , is there any option to remove (or not keeping it at all 
)  the template form primary storage ? if not  is it ok if I deleted the 
template from primary storage manually  ?


Best Regards,
EtQ, Inc.
Ghaith Bannoura
System Administrator
MCT, MCSE (Messaging, Server Infrastructure)
MCSA (Windows Server 2008, 2012), MCP

*  Phone: +9626-537-0923 Ext. 376
*  gbanno...@etq.com 
*   http://www.etq.com

Re: Management Server can't start after upgrade to 4.9.0

[Sergey Levitskiy]

This issue should get fixed in 4.10
https://github.com/apache/cloudstack/pull/1621



On 8/22/16, 6:51 AM, "Jan Siml"  wrote:

Hello,

I have noticed the following error in db.properties after upgrade:

db.cloud.secondsBeforeRetryMaster=3600db.cloud.driver=jdbc:mysql

It should be:

db.cloud.secondsBeforeRetryMaster=3600
db.cloud.driver=jdbc:mysql

Maybe it's the same cause for you? Afterwards cloudstack-management 
service could be restarted successfully.

Kind regards

Jan


On 08/22/2016 03:46 PM, Marty Godsey wrote:
> I noticed this step missing as well. I use Ubuntu for my management 
servers but I know I had to rerun the cloudstack-setup-databases script again 
without the deploy option to correct.
>
> Regards,
> Marty Godsey
>
> -Original Message-
> From: Joshua [mailto:joshua...@gmail.com]
> Sent: Monday, August 22, 2016 6:10 AM
> To: users@cloudstack.apache.org
> Subject: Re: Management Server can't start after upgrade to 4.9.0
>
> The settings look correct in that file. Where do I find this? "DB driver 
type null is not supported!"
>
> On Mon, Aug 22, 2016 at 6:02 PM, Glenn Wagner 
> wrote:
>
>> Hi,
>>
>> You can have a look at the db.properties file on both your Management
>> servers
>>
>> /etc/cloudstack/management/db.properties
>>
>> Thanks
>> Glenn
>>
>>
>> glenn.wag...@shapeblue.com
>> www.shapeblue.com
>> First Floor, Victoria Centre, 7 Victoria Street, Somerset West, Cape
>> Town 7129South Africa @shapeblue
>>
>>
>>
>>
>> -Original Message-
>> From: joshua...@gmail.com [mailto:joshua...@gmail.com] On Behalf Of
>> Joshua
>> Sent: Monday, 22 August 2016 11:48 AM
>> To: users@cloudstack.apache.org
>> Subject: Re: Management Server can't start after upgrade to 4.9.0
>>
>> Yes. Everything was followed.
>>
>> It seems like it's connecting to the wrong MySQL server (trying for
>> localhost but it's supposed to connect to a HAProxy balanced "remote" 
IP).
>>
>> On Mon, Aug 22, 2016 at 5:44 PM, Glenn Wagner
>> 
>> wrote:
>>
>>> Ok did you install the new mysql connector for version 4.9 ?
>>>
>>> http://docs.cloudstack.apache.org/projects/cloudstack-
>>> release-notes/en/4.9.0/upgrade/upgrade-4.8.html#
>>> install-new-mysql-connector
>>>
>>>
>>> Glenn
>>>
>>>
>>>
>>> glenn.wag...@shapeblue.com
>>> www.shapeblue.com
>>> First Floor, Victoria Centre, 7 Victoria Street, Somerset West, Cape
>>> Town 7129South Africa @shapeblue
>>>
>>>
>>>
>>>
>>> -Original Message-
>>> From: joshua...@gmail.com [mailto:joshua...@gmail.com] On Behalf Of
>>> Joshua
>>> Sent: Monday, 22 August 2016 11:40 AM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: Management Server can't start after upgrade to 4.9.0
>>>
>>> Hello Glenn,
>>>
>>> # cat /etc/*-release
>>> CentOS release 6.7 (Final)
>>> LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-n
>>> oa
>>> rch
>>> CentOS release 6.7 (Final)
>>> CentOS release 6.7 (Final)
>>>
>>> Thanks
>>> Joshua
>>>
>>> On Mon, Aug 22, 2016 at 5:07 PM, Glenn Wagner
>>> 
>>> wrote:
>>>
 Morning Joshua,

 What OS are you using on your Management server nodes?

 Thanks
 Glenn


 glenn.wag...@shapeblue.com
 www.shapeblue.com
 First Floor, Victoria Centre, 7 Victoria Street, Somerset West,
 Cape Town 7129South Africa @shapeblue




 -Original Message-
 From: joshua...@gmail.com [mailto:joshua...@gmail.com] On Behalf
 Of Joshua
 Sent: Monday, 22 August 2016 10:49 AM
 To: users@cloudstack.apache.org
 Subject: Management Server can't start after upgrade to 4.9.0

 Hi all,

 I followed the instructions under
 http://docs.cloudstack.apache.org/projects/cloudstack-
 release-notes/en/4.9.0/upgrade/upgrade-4.8.html
 to upgrade form 4.8.0 to 4.9.0 and it seems to have overwritten
 the default DB connection config.

 The setup is running two management nodes connecting to a HAProxy
 balanced Active-Passive Galera cluster. It seems that after the
 upgrade both management nodes are trying to connect to the local
 MySQL server
 (127.0.0.1) which is wrong. Where should I edit the config?

 Thanks in advance!

 Paste of log:

 2016-08-22 16:23:46,669 ERROR [c.c.u.d.DriverLoader] 

Re: ACS 4.5.2 - VMware 5.5 VM migration

[Sergey Levitskiy]

Migration from host to host only make sense if VM is in running state. In 
VMware based infra you need to have vmotion working properly for ACS to be able 
to migrate running VM
Thanks
Sergey
Sent from my iPhone

> On Jul 27, 2016, at 5:30 AM, Cristian Ciobanu  
> wrote:
> 
> Hello,
> 
> 
> 
>I have a setup with ACS 4.5.2 with Vmware 5.5 ( Basic setup without HA
> ), my problem is : how is possibile to migrate a VM from a host to another
> Host, i tried to migrate but i see only volume migration also fail to start
> becuase i don't have HA enabled.
> 
> 
> 
> Regards,
> 
> Cristian
> 
> 
>