SAML Certificates Expired

[Sean Lair]

Hi all, how are the SAML signing and encryption certificate renewals handled in 
CloudStack?  We use SAML and the certs show using the API command below just 
expired and now SAML logins don't work.  Are those supposed to auto-renew?

http://acs-server:8080/client/api?command=getSPMetadata


Thanks
Sean

RE: [DISCUSS] Moving to OpenVPN as the remote access VPN provider

[Sean Lair]

I would love to see OpenVPN as the client VPN.  We consider the current Client 
VPN unusable.  We use OpenVPN with OPNsense firewalls and it has been 
rock-solid.


-Original Message-
From: Rohit Yadav  
Sent: Friday, June 11, 2021 12:40 PM
To: users@cloudstack.apache.org; d...@cloudstack.apache.org
Subject: [DKIM Fail] Re: [DISCUSS] Moving to OpenVPN as the remote access VPN 
provider

Hi PL,

You can check the ikev2 support in 4.15+ here: 
https://github.com/apache/cloudstack/pull/4953

I think a generic VPN framework-provider feature is probably what we need (i.e. 
to let user or admin decide what VPN provider they want, supporting 
strongswan/ipsec and openvpn) so I'm not trying to defend OpenVPN here but your 
comments on OpenVPN are incorrect. It is widely used (in many projects incl. 
pfSense) and both server/clients are opensource and not proprietary afaik (GPL 
or AGPL license, I'm not sure about platform-specific clients (the GUI ones) 
but I checked the CLI clients are opensource):
https://github.com/OpenVPN/openvpn
https://github.com/OpenVPN/openvpn3

One key requirement for whatever VPN provider we support is that it should be 
free and opensource and available on Debian (for use in the systemvmtemplate) 
and OpenVPN fits that requirement. The package is available on Debian: 
https://packages.debian.org/buster-backports/openvpn

Regards.


From: Pierre-Luc Dion 
Sent: Friday, June 11, 2021 20:10
To: users@cloudstack.apache.org 
Cc: dev 
Subject: Re: [DISCUSS] Moving to OpenVPN as the remote access VPN provider

Just to be sure, what CloudStack > v4.15 uses Strongswan/l2tp or
strongswan/ikev2 ?

Because l2tp became complicated to configure on native vpn clients on some 
OSes, kind of deprecated remote management VPN, compared to IKEv2.
I'm a bit concerned about OpenVPN for the clients, what if binaries become 
subscription based availability or become proprietary ?

For sure we need the option to select what type of VPN solution to offer when 
deploying a cloud.

>From my perspective I cannot use/offer OpenVPN as a solution to my customers 
>because it involves forcing them to download third party software on their 
>workstations and I don't want to be responsible for a security breach on their 
>workstation because of a requirement for 3rd party software that we don't 
>control.



On Fri, Jun 11, 2021 at 10:14 AM Rohit Yadav 
wrote:

> Thanks all for the feedback so far, looks like the majority of people 
> on the thread would prefer OpenVPN but for s2s they may continue to 
> prefer strongswan/ipsec for site-to-site VPC feature. If we're unable 
> to reach consensus then a general-purpose provider-framework may be 
> more flexible to the end-user or admin (to select which VPN provider 
> they want for their network, we heard in this thread - openvpn, 
> strongswan/l2tp, wireguard, and maybe other providers in future).
>
> Btw, ikev2 is supported now with strongswan with this -
> https://github.com/apache/cloudstack/pull/4953
>
> My personal opinion: As user of most of these VPN providers, I 
> personally like OpenVPN which I found to be easier to use both on 
> desktop/laptop and on phone. With openvpn as the default I imagine in 
> CloudStack I could enable VPN for a network and CloudStack gives me an 
> option to download a .ovpn file which I can import in my openvpn 
> client (desktop, phone, cli...) click connect to connect to the VPN. 
> For certificate generation/storage, the CA framework could be used so 
> the openvpn server certs are the same across network restarts (with 
> cleanup). I think a process like this could be simpler than what we've 
> right now, and the ovpn download+import workflow would be easier than 
> what we'll get from either strongswan/current or wireguard. While I 
> like the simplicity of wireguard, which is more like SSH setup I 
> wouldn't mind doing setup on individual VMs (much like setting up ssh key) or 
> use something like TailScale.
>
>
> Regards.
>
> 
> From: Gabriel Bräscher 
> Sent: Friday, June 11, 2021 19:28
> To: dev 
> Cc: users 
> Subject: Re: [DISCUSS] Moving to OpenVPN as the remote access VPN 
> provider
>
> I understand that OpenVPN is a great option and far adopted.
> I am  ++1 in allowing Users/Admins to choose which VPN provider suits 
> them best; creating an offering (or global settings) that would allow 
> setting which VPN provider will be used would be awesome.
>
> I understand that OpenVPN is a great option and far adopted; however, 
> I would be -1 if this would impact on removing support for Strongswan 
> -- which from what I understood is not the proposal, but saying anyway 
> to be sure.
>
> Thanks for raising this proposal/discussion, Rohit.
>
> Cheers,
> Gabriel.
>
>
> Em sex., 11 de jun. de 2021 às 08:46, Pierre-Luc Dion 
>  >
> escreveu:
>
> > Hello,
> >
> > Daan, I agree we should provide capability to select the vpn 
> > solution to use, the 

Primary Storage Allocation

[Sean Lair]

We have two primary storage pools setup, both are NFS.  Both are functioning, 
but CloudStack is provisioning all our VMs on one of the two pools.  Is there a 
way to have CloudStack provision new VMs on the pool with the most available 
storage?  Or how does CloudStack pick between the two pools?

Thanks
Sean

RE: VirtIO Network Adapter for system vms on KVM Hypervisor

[Sean Lair]

Just for feedback, we are 4.11.3 and run KVM on CentOS 7.  Our 4.11.3 template 
is set to Debian GNU/Linux 8 (64-bit).  Our lspci is shown below:

root@r-281-VM:~# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] 
(rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
00:04.0 Communication controller: Red Hat, Inc Virtio console
00:05.0 SCSI storage controller: Red Hat, Inc Virtio block device
00:06.0 System peripheral: Intel Corporation 6300ESB Watchdog Timer
00:07.0 Ethernet controller: Red Hat, Inc Virtio network device
00:08.0 Ethernet controller: Red Hat, Inc Virtio network device
00:09.0 Ethernet controller: Red Hat, Inc Virtio network device


-Original Message-
From: Andrija Panic  
Sent: Saturday, May 16, 2020 7:12 AM
To: users 
Subject: Re: VirtIO Network Adapter for system vms on KVM Hypervisor

Thanks for the feedback on that one, Rafal.

Regards

On Sat, 16 May 2020 at 12:58, Rafal Turkiewicz  wrote:

> Just for a record
>
> I have tested this with Debian GNU/Linux 7.0 (64-bit) OS Type and it 
> also worked. It basically breaks as soon as I pick Debian GNU/Linux 8 
> (64-bit).
>
> Thanks
>
> On 2020/05/15 14:00:53, Rafal Turkiewicz  wrote:
> > Andrija,
> >
> > You are the man! I have changed the OS Type to the default Debian 5 
> > x64
> and boom! All sorted.
> >
> > It's really odd that picking older OS Type solved the issue where in
> fact the systemVM is running Debian 9. Is this a BUG of some sort?
> >
> > I might try and experiment with other OS Type Debian version X to 
> > see
> where it falls but for now I'm all happy!
> >
> > Once again thank you very much for the pointer!
> >
> > Raf
> >
> > On 2020/05/15 13:51:01, Andrija Panic  wrote:
> > > In the upgrade guide, we always advise (when registering the new
> systeVM
> > > template) to go as:
> > >
> > >   OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest 
> > > Debian
> release
> > > number available in the dropdown)
> > >
> > > That being said, in the clean 4.13 installation, the OS type is 
> > > set to Debian 5 x64 - so try each version and in between destroy VR (i.e.
> restart
> > > the network with cleanup) and observe "lspci" if virtio or intel 
> > > NICs
> - but
> > > also make sure that each time the VR is created on KVM host (i.e. 
> > > not
> on
> > > XEN).
> > >
> > > In order to change OS type for systemVM template, you will have to 
> > > use
> DB
> > > - modify the "vm_template" table - update the "guest_os_id" field
> value for
> > > that specific template, to the ID from the "guest_os" table where 
> > > name=Debian XXX 64.
> > >
> > > Hope that solves the issue - should by all means.
> > >
> > > Regards
> > > Andrija
> > >
> > >
> > > On Fri, 15 May 2020 at 15:33, Rafal Turkiewicz 
> wrote:
> > >
> > > > Hello Andrija,
> > > >
> > > > Thanks for your input the OS Type for the systemVM template is 
> > > > set to "Debian GNU/Linux 8 (64-bit)"
> > > >
> > > > I think I forgot to mention a very important aspect of my setup. 
> > > > This Cloudstack instance is powering XenServer and KVM where KVM 
> > > > was added recently.
> > > >
> > > > Your message made me think and look at my other (test lab) setup
> where
> > > > CloudStack is only powering KVM hypervisors. I can confirm all 
> > > > VRs
> are
> > > > running with virtio which implies there got to be something on 
> > > > the
> my mixed
> > > > HV CloudStack.
> > > >
> > > > I will keep looking into this but if you have any further 
> > > > thoughts
> on this
> > > > please let me know.
> > > >
> > > > Raf
> > > >
> > > > On 2020/05/15 11:14:37, Andrija Panic 
> wrote:
> > > > > Rafal,
> > > > >
> > > > > what is the OS type you defined for the systemVM template?
> > > > >
> > > > > In my env, VR (VPC) - all interfaces are VirtIO.
> > > > >
> > > > > Best
> > > > > Andrija
> > > > >
> > > > > On Fri, 15 May 2020 at 12:14, Rafal Turkiewicz 
> > > > > 
> > > > wrote:
> > > > >
> > > > > > Platform:
> > > > > > CloudStack 4.11.2 on CentOS 7 KVM Hypervisor on CentOS 7
> > > > > >
> > > > > > I have found some throughput issues on our VirtualRuters and 
> > > > > > I've
> > > > tracked
> > > > > > it down to CPU IRQ hitting 99% on the VR which was related 
> > > > > > to NIC interrupts.
> > > > > >
> > > > > > I decided to lookup what NIC is being emulated on the VRs; 
> > > > > > lsmod
> listed
> > > > > > three Intel NICs:
> > > > > >
> > > > > > 00:03.0 Ethernet controller: Intel Corporation 82540EM 
> > > > > > Gigabit
> Ethernet
> > > > > > Controller (rev 03)
> > > > > > 00:04.0 Ethernet controller: Intel Corporation 82540EM 
> > > > > > 

RE: Virtual machines volume lock manager

[Sean Lair]

Are you using NFS?

Yea, we implmented locking because of that problem:

https://libvirt.org/locking-lockd.html

echo lock_manager = \"lockd\" >> /etc/libvirt/qemu.conf

-Original Message-
From: Andrija Panic  
Sent: Wednesday, October 30, 2019 6:55 AM
To: dev 
Cc: users 
Subject: Re: Virtual machines volume lock manager

I would advise trying to reproduce.

start migration, then either:
- configure timeout so that it''s way too low, so that migration fails due to 
timeouts.
- restart mgmt server in the middle of migrations This should cause migration 
to fail - and you can observe if you have reproduced the problem.
keep in mind, that there might be some garbage left, due to not-properly 
handling the failed migration But from QEMU point of view - if migration fails, 
by all means the new VM should be destroyed...



On Wed, 30 Oct 2019 at 11:31, Rakesh Venkatesh 

wrote:

> Hi Andrija
>
>
> Sorry for the late reply.
>
> Im using 4.7 version of ACS. Qemu version 1:2.5+dfsg-5ubuntu10.40
>
> Im not sure if ACS job failed or libvirt job as I didnt see into logs.
> Yes the vm will be in paused state during migration but after the 
> failed migration, the same vm was in "running" state on two different 
> hypervisors.
> We wrote a script to find out how duplicated vm's are running and 
> found out that more than 5 vm's had this issue.
>
>
> On Mon, Oct 28, 2019 at 2:42 PM Andrija Panic 
> 
> wrote:
>
> > I've been running KVM public cloud up to recently and have never 
> > seen
> such
> > behaviour.
> >
> > What versions (ACS, qemu, libvrit) are you running?
> >
> > How does the migration fail - ACS job - or libvirt job?
> > destination VM is by default always in PAUSED state, until the 
> > migration
> is
> > finished - only then the destination VM (on the new host) will get
> RUNNING,
> > while previously pausing the original VM (on the old host).
> >
> > i,e.
> > phase1  source vm RUNNING, destination vm PAUSED (RAM content being
> > copied over... takes time...)
> > phase2  source vm PAUSED, destination vm PAUSED (last bits of RAM
> > content are migrated)
> > phase3  source vm destroyed, destination VM RUNNING.
> >
> > Andrija
> >
> > On Mon, 28 Oct 2019 at 14:26, Rakesh Venkatesh <
> http://sea.ippathways.com:32224/?dmVyPTEuMDAxJiYzM2ZmODRmOWFhMzdmZmQ1O
> T01REI5N0ExQV84NTE5N18yMDM4OV8xJiZjZjE2YzBlNTI0N2VmMjM9MTIzMyYmdXJsPXd
> 3dyUyRXJha2VzaHYlMkVjb20=@gmail.com>
> > wrote:
> >
> > > Hello Users
> > >
> > >
> > > Recently we have seen cases where when the Vm migration fails,
> cloudstack
> > > ends up running two instances of the same VM on different hypervisors.
> > The
> > > state will be "running" and not any other transition state. This 
> > > will
> of
> > > course lead to corruption of disk. Does CloudStack has any option 
> > > of
> > volume
> > > locking so that two instances of the same VM wont be running?
> > > Anyone else has faced this issue and found some solution to fix it?
> > >
> > > We are thinking of using "virtlockd" of libvirt or implementing 
> > > custom
> > lock
> > > mechanisms. There are some pros and cons of the both the solutions 
> > > and
> i
> > > want your feedback before proceeding further.
> > >
> > > --
> > > Thanks and regards
> > > Rakesh venkatesh
> > >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
> --
> Thanks and regards
> Rakesh venkatesh
>


-- 

Andrija Panić

VM NIC changes on host move

[Sean Lair]

Hi all,

We are seeing a strange behavior and are still troubleshooting... but what 
seems to be happening is when we shutdown a VM, and then power it on, on a 
different host, the VM’s operating systems sees a new NIC and the old NIC is 
gone.  We aren’t 100% sure this is the scenario yet to reproduce... but the 
below issue has happened multiple times now on different VMs and the latest 
time we did purposely power it back up on a different host.

The VMs have been Windows Servers and in device manager the new NIC shows up as 
NIC #2 and the original NIC is no longer present (it is greyed out indicating 
it isn’t present).  NIC2 then of course doesn’t have NIC1’s static IP so it 
causes all kinds of issues...  Any ideas?

CloudStack 4.11.3
CentOS7 / KVM
RedHat VirtIO

[cid:7F130E37-0C79-4BC3-9116-17D644843909-L0-001]

Thanks
Sean

RE: Issue adding a second zone to Cloudstack

[Sean Lair]

Thank you for the reply Vivek!  I was wondering if that was the case but just 
couldn't find any documentation to verify.  We've done that and are back on the 
right path!


-Original Message-
From: Vivek Kumar  
Sent: Saturday, March 28, 2020 4:22 AM
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Re: Issue adding a second zone to Cloudstack

Hello Sean,

You need to again seed the template to the secondary storage of your new zone 
just like you did for the first zone.

i.e
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt 
-m /mnt/secondary -u 
http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2
 
<http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2>
 -h kvm -s  -F

Vivek Kumar
Manager - Cloud & DevOps 
IndiQus Technologies
24*7  O +91 11 4055 1411  |   M +91 7503460090 
www.indiqus.com <http://indiqus.com/>

This message is intended only for the use of the individual or entity to which 
it is addressed and may contain information that is confidential and/or 
privileged. If you are not the intended recipient please delete the original 
message and any copy of it from your computer system. You are hereby notified 
that any dissemination, distribution or copying of this communication is 
strictly prohibited unless proper authorization has been obtained for such 
action. If you have received this communication in error, please notify the 
sender immediately. Although IndiQus attempts to sweep e-mail and attachments 
for viruses, it does not guarantee that both are virus-free and accepts no 
liability for any damage sustained as a result of viruses.

> On 28-Mar-2020, at 4:08 AM, Sean Lair  wrote:
> 
> Hi all,
> 
> We are running 4.11.3 with a single zone, that zone is working without issue. 
>  We are trying to add a second zone to the installation, and everything seems 
> to go well, except we are confused on how the SystemVM templates should be 
> handled for the new zone.  The new zone has its own secondary storage (NFS).  
> When Cloudstack sees the new Zone, it attempts to provision a Secondary 
> Storage VM.  However, it is unable to because the SystemVM Template doesn't 
> exist on the new secondary storage (NFS).
> 
> Are we supposed to pre-populate another copy of the SystemVM Template in the 
> additional zone and secondary storage?  Or should cloudstack copy the 
> existing SystemVM Template (which is set as cross-zone) to the new zone for 
> us?  Here is some detailed information:
> 
> MariaDB [cloud]> SELECT id,name,type,cross_zones,state FROM cloud.vm_template 
> WHERE name like '%systemvm-kvm%' AND removed IS NULL;
> +-+--+-+-+--+
> | id  | name | type| cross_zones | state|
> +-+--+-+-+--+
> | 344 | systemvm-kvm-4.11.3  | SYSTEM  |   1 | Active   |
> +-+--+-+-+--+
> 
> MariaDB [cloud]> select id,store_id,template_id,install_path, download_state 
> from template_store_ref;
> +-+--+-+++
> | id  | store_id | template_id | install_path 
>   | download_state |
> +-+--+-+++
> | 666 |1 | 344 | 
> template/tmpl/2/344/182f0a79-1e16-3e53-a6e9-fcffe5f11c3e.qcow2 | 
> DOWNLOADED |
> | 756 |   16 | 344 | template/tmpl/1/344/ 
>   | DOWNLOADED |
> +-+--+-+++
> 
> Why in the template_store_ref table did cloudstack add a new row with 
> "downloaded" and missing a filename in the "install_path"?
> 
> 
> The KVM host cannot mount the template on the new secondary storage, because 
> it isn't there yet (should cloudstack be copying that template from the 
> existing zone to the new one for us?):
> --
> 2020-03-27 18:51:40,626 ERROR [kvm.storage.LibvirtStorageAdaptor] 
> (agentRequest-Handler-2:null) (logid:6b50f03a) Failed to create netfs mount: 
> 10.102.33.5:/zone2_secondary/template/tmpl/1/344
> org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount 
> 10.10.33.5:/zone2_secondary/template/tmpl/1/344 
> /mnt/b69caab0-4c1e-34b6-94b8-2617ba561e9a -o nodev,nosuid,noexec) unexpected 
> exit status 32: mount.nfs: mounting 
> 10.10.33.5:/zone2__secondary/template/tmpl/1/344 failed, reason given by 
> server: No such file or directory
> -
> 
> 
> Thanks!
> Sean

Issue adding a second zone to Cloudstack

[Sean Lair]

Hi all,

We are running 4.11.3 with a single zone, that zone is working without issue.  
We are trying to add a second zone to the installation, and everything seems to 
go well, except we are confused on how the SystemVM templates should be handled 
for the new zone.  The new zone has its own secondary storage (NFS).  When 
Cloudstack sees the new Zone, it attempts to provision a Secondary Storage VM.  
However, it is unable to because the SystemVM Template doesn't exist on the new 
secondary storage (NFS).

Are we supposed to pre-populate another copy of the SystemVM Template in the 
additional zone and secondary storage?  Or should cloudstack copy the existing 
SystemVM Template (which is set as cross-zone) to the new zone for us?  Here is 
some detailed information:

MariaDB [cloud]> SELECT id,name,type,cross_zones,state FROM cloud.vm_template 
WHERE name like '%systemvm-kvm%' AND removed IS NULL;
+-+--+-+-+--+
| id  | name | type| cross_zones | state|
+-+--+-+-+--+
| 344 | systemvm-kvm-4.11.3  | SYSTEM  |   1 | Active   |
+-+--+-+-+--+

MariaDB [cloud]> select id,store_id,template_id,install_path, download_state 
from template_store_ref;
+-+--+-+++
| id  | store_id | template_id | install_path   
| download_state |
+-+--+-+++
| 666 |1 | 344 | 
template/tmpl/2/344/182f0a79-1e16-3e53-a6e9-fcffe5f11c3e.qcow2 | 
DOWNLOADED |
| 756 |   16 | 344 | template/tmpl/1/344/   
| DOWNLOADED |
+-+--+-+++

Why in the template_store_ref table did cloudstack add a new row with 
"downloaded" and missing a filename in the "install_path"?


The KVM host cannot mount the template on the new secondary storage, because it 
isn't there yet (should cloudstack be copying that template from the existing 
zone to the new one for us?):
--
2020-03-27 18:51:40,626 ERROR [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:6b50f03a) Failed to create netfs mount: 
10.102.33.5:/zone2_secondary/template/tmpl/1/344
org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount 
10.10.33.5:/zone2_secondary/template/tmpl/1/344 
/mnt/b69caab0-4c1e-34b6-94b8-2617ba561e9a -o nodev,nosuid,noexec) unexpected 
exit status 32: mount.nfs: mounting 
10.10.33.5:/zone2__secondary/template/tmpl/1/344 failed, reason given by 
server: No such file or directory
-


Thanks!
Sean

RE: Update for those having database upgrade issues recently

[Sean Lair]

Yep, see this Issue:

https://github.com/apache/cloudstack/issues/3826


-Original Message-
From: Robert Ward  
Sent: Wednesday, January 22, 2020 9:42 PM
To: users@cloudstack.apache.org
Subject: Update for those having database upgrade issues recently

Hello,

If you have been one of the unfortunate ones with the recent database upgrade 
problem I have stumbled across a possible solution...at least in my case.

Looking a little closer at the logs it seems there was some sort of java 
disconnect triggering the DB upgrade issue so I decided to downgrade the 
mysql-connector-java module. In my case that did the trick! Everything seems to 
working ok so far.

BTW - I am running mysql-connector-python-8.0.19 so that may have not been the 
culprit after all.

Robert

RE: RE: Unable to authenticate in UI

[Sean Lair]

We tried 8.0.16 through 8.0.19 and it had the same error regarding Long and 
BigInt.

For some reason when we try 8.0.11 through 8.0.15, we have some other issue 
(shown below) but did not have time to troubleshoot that unless  you know a 
quick fix:

"Failed to load DB driver com.mysql.jdbc.Driver"

-Original Message-
From: Andrija Panic  
Sent: Tuesday, January 21, 2020 12:22 PM
To: users 
Subject: Re: RE: Unable to authenticate in UI

Alright, my bad - so this is a different issue then...but the issue described 
is true with mysql-connector-python (issue with
cloudstack-setup-databases)

Either way, something is broken/different with those 2 in version
8.0.19-1.el7

@Sean did you test with mysql-connector-java 8.0.18 maybe? Is the whole 8.x 
series broken?


Thx

On Tue, 21 Jan 2020 at 19:08, Sean Lair  wrote:

> We actually had to downgrade the mysql-connector-java one - not the 
> mysql-connector-python one
>
> -Original Message-
> From: Andrija Panic 
> Sent: Tuesday, January 21, 2020 11:59 AM
> To: users 
> Subject: Re: RE: Unable to authenticate in UI
>
> broken version - go to the older version.
> yum localinstall
>
> http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/mysql
> -connector-python-8.0.18-1.el7.x86_64.rpm
>
> 1. Did you update your system - can you check yum/apt logs if the 
> connector was upgraded, please?
> 2. When was your system installed - i.e. can you confirm  you have NOT 
> installed it with that version of the connector, since I've seen 
> cloudstack-setup-database failing with that connector version?
>
> If you system got broken due to updating the connector on the existing 
> installation -  then this warrants an email to everyone if that is the 
> case
>
> Please do check and let me know.
>
> Thanks
> Andrija
>
> On Tue, 21 Jan 2020 at 18:54, Ioan Marginean 
> 
> wrote:
>
> > Hi Sean,
> >
> > I have the following output:
> >
> > mysql-connector-java.noarch  1:8.0.19-1.el7 
> > @mysql-connectors-community
> >
> > On 2020/01/21 17:41:44, Sean Lair  wrote:
> > > We are hitting a similar error message when doing other functions.
> > > What
> > version of the mysql-connector-java are you running?
> > >
> > > yum list installed mysql-connector-java
> > >
> > >
> > > -Original Message-
> > > From: Ioan Marginean 
> > > Sent: Tuesday, January 21, 2020 7:46 AM
> > > To: users@cloudstack.apache.org
> > > Subject: Re: Unable to authenticate in UI
> > >
> > > Is it possible to reset password directly into the database?
> > > Thank you.
> > >
> > > On 2020/01/21 11:55:18, Ioan Marginean 
> > > 
> > wrote:
> > > > Hello,
> > > >
> > > > I facing a strange situation related to UI authentication. After
> > restart cloudstack-management I can't authenticate. The restart was 
> > needed because I change settings related to e-mail notifications.
> > > > My configuration is: CS version = 4.13, KVM. On error log I got:
> > > >
> > > > 2020-01-21 13:47:04,111 DEBUG [c.c.a.ApiServlet]
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) ===START===
> > 172.16.1.106
> > -- POST
> > > > 2020-01-21 13:47:04,117 DEBUG [c.c.u.AccountManagerImpl]
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Attempting to log in
> user:
> > alex in domain 1
> > > > 2020-01-21 13:47:04,119 DEBUG 
> > > > [o.a.c.s.a.PBKDF2UserAuthenticator]
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Retrieving user: 
> > alex
> > > > 2020-01-21 13:47:04,556 DEBUG [c.c.u.AccountManagerImpl]
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) CIDRs from which 
> > account 'Acct[95411505-a071-419e-bf6d-65c94971c1ad-alex]' is allowed 
> > to perform API
> > calls: 0.0.0.0/0,::/0
> > > > 2020-01-21 13:47:04,556 DEBUG [c.c.u.AccountManagerImpl]
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) User: alex in domain 
> > 1 has successfully logged in
> > > > 2020-01-21 13:47:04,557 DEBUG [c.c.u.d.T.Transaction]
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Rolling back the
> > transaction: Time = 1 Name =  qtp504527234-32; called by 
> > -TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-Tra
> > ns 
> > actionLegacy.close:657-TransactionContextInterceptor.invoke:36-Refle
> > ct
> > iveMethodInvocation.proceed:174-ExposeInvocationInterceptor.invoke:9
> > 2- 
> > ReflectiveMethodInvocation.proceed:185-JdkDynamicAopProxy.invoke:212
> > -$ 
> > Proxy129

RE: RE: Unable to authenticate in UI

[Sean Lair]

We actually had to downgrade the mysql-connector-java one - not the 
mysql-connector-python one

-Original Message-
From: Andrija Panic  
Sent: Tuesday, January 21, 2020 11:59 AM
To: users 
Subject: Re: RE: Unable to authenticate in UI

broken version - go to the older version.
yum localinstall
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/mysql-connector-python-8.0.18-1.el7.x86_64.rpm

1. Did you update your system - can you check yum/apt logs if the connector was 
upgraded, please?
2. When was your system installed - i.e. can you confirm  you have NOT 
installed it with that version of the connector, since I've seen 
cloudstack-setup-database failing with that connector version?

If you system got broken due to updating the connector on the existing 
installation -  then this warrants an email to everyone if that is the case

Please do check and let me know.

Thanks
Andrija

On Tue, 21 Jan 2020 at 18:54, Ioan Marginean 
wrote:

> Hi Sean,
>
> I have the following output:
>
> mysql-connector-java.noarch  1:8.0.19-1.el7 
> @mysql-connectors-community
>
> On 2020/01/21 17:41:44, Sean Lair  wrote:
> > We are hitting a similar error message when doing other functions.  
> > What
> version of the mysql-connector-java are you running?
> >
> > yum list installed mysql-connector-java
> >
> >
> > -Original Message-
> > From: Ioan Marginean 
> > Sent: Tuesday, January 21, 2020 7:46 AM
> > To: users@cloudstack.apache.org
> > Subject: Re: Unable to authenticate in UI
> >
> > Is it possible to reset password directly into the database?
> > Thank you.
> >
> > On 2020/01/21 11:55:18, Ioan Marginean 
> wrote:
> > > Hello,
> > >
> > > I facing a strange situation related to UI authentication. After
> restart cloudstack-management I can't authenticate. The restart was 
> needed because I change settings related to e-mail notifications.
> > > My configuration is: CS version = 4.13, KVM. On error log I got:
> > >
> > > 2020-01-21 13:47:04,111 DEBUG [c.c.a.ApiServlet]
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) ===START===  
> 172.16.1.106
> -- POST
> > > 2020-01-21 13:47:04,117 DEBUG [c.c.u.AccountManagerImpl]
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Attempting to log in user:
> alex in domain 1
> > > 2020-01-21 13:47:04,119 DEBUG [o.a.c.s.a.PBKDF2UserAuthenticator]
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Retrieving user: alex
> > > 2020-01-21 13:47:04,556 DEBUG [c.c.u.AccountManagerImpl]
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) CIDRs from which 
> account 'Acct[95411505-a071-419e-bf6d-65c94971c1ad-alex]' is allowed 
> to perform API
> calls: 0.0.0.0/0,::/0
> > > 2020-01-21 13:47:04,556 DEBUG [c.c.u.AccountManagerImpl]
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) User: alex in domain 1 
> has successfully logged in
> > > 2020-01-21 13:47:04,557 DEBUG [c.c.u.d.T.Transaction]
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Rolling back the
> transaction: Time = 1 Name =  qtp504527234-32; called by
> -TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-Trans
> actionLegacy.close:657-TransactionContextInterceptor.invoke:36-Reflect
> iveMethodInvocation.proceed:174-ExposeInvocationInterceptor.invoke:92-
> ReflectiveMethodInvocation.proceed:185-JdkDynamicAopProxy.invoke:212-$
> Proxy129.persist:-1-ActionEventUtils.persistActionEvent:186-ActionEven
> tUtils.onActionEvent:98-AccountManagerImpl.authenticateUser:2342
> > > 2020-01-21 13:47:04,559 ERROR [c.c.a.ApiServlet]
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) unknown exception 
> writing api response
> > > com.cloud.utils.exception.CloudRuntimeException: Problem with 
> > > getting
> the ec attribute
> > > at
> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1454)
> > > at sun.reflect.GeneratedMethodAccessor145.invoke(Unknown Source)
> > > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> n(AopUtils.java:338)
> > > at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:197)
> > > at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:163)
> > > at
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionCon
> textInterceptor.java:34)
> > > at
> org.springframework.aop.framework.ReflectiveMeth

RE: RE: Unable to authenticate in UI

[Sean Lair]

Yea- that's the issue.  We hit it also, you need to downgrade to the non-8.x 
version...  run this command to downgrade, note, you need to use "yum upgrade" 
since it has a different epoc:

yum upgrade mysql-connector-java-5.1.25

That will put you back on version 5.1.25

If this helps, could you please comment on this Issue ID?  It is good to track 
the details in the Issue in GitHub:

https://github.com/apache/cloudstack/issues/3826



-Original Message-
From: Ioan Marginean  
Sent: Tuesday, January 21, 2020 11:55 AM
To: users@cloudstack.apache.org
Subject: Re: RE: Unable to authenticate in UI 

Hi Sean,

I have the following output:

mysql-connector-java.noarch  1:8.0.19-1.el7
@mysql-connectors-community

On 2020/01/21 17:41:44, Sean Lair  wrote: 
> We are hitting a similar error message when doing other functions.  What 
> version of the mysql-connector-java are you running?
> 
> yum list installed mysql-connector-java
> 
> 
> -Original Message-
> From: Ioan Marginean  
> Sent: Tuesday, January 21, 2020 7:46 AM
> To: users@cloudstack.apache.org
> Subject: Re: Unable to authenticate in UI 
> 
> Is it possible to reset password directly into the database?
> Thank you.
> 
> On 2020/01/21 11:55:18, Ioan Marginean  wrote: 
> > Hello,
> > 
> > I facing a strange situation related to UI authentication. After restart 
> > cloudstack-management I can't authenticate. The restart was needed because 
> > I change settings related to e-mail notifications.
> > My configuration is: CS version = 4.13, KVM. On error log I got:
> > 
> > 2020-01-21 13:47:04,111 DEBUG [c.c.a.ApiServlet] 
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) ===START===  172.16.1.106 
> > -- POST  
> > 2020-01-21 13:47:04,117 DEBUG [c.c.u.AccountManagerImpl] 
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Attempting to log in user: 
> > alex in domain 1
> > 2020-01-21 13:47:04,119 DEBUG [o.a.c.s.a.PBKDF2UserAuthenticator] 
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Retrieving user: alex
> > 2020-01-21 13:47:04,556 DEBUG [c.c.u.AccountManagerImpl] 
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) CIDRs from which account 
> > 'Acct[95411505-a071-419e-bf6d-65c94971c1ad-alex]' is allowed to perform API 
> > calls: 0.0.0.0/0,::/0
> > 2020-01-21 13:47:04,556 DEBUG [c.c.u.AccountManagerImpl] 
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) User: alex in domain 1 has 
> > successfully logged in
> > 2020-01-21 13:47:04,557 DEBUG [c.c.u.d.T.Transaction] 
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Rolling back the 
> > transaction: Time = 1 Name =  qtp504527234-32; called by 
> > -TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor.invoke:92-ReflectiveMethodInvocation.proceed:185-JdkDynamicAopProxy.invoke:212-$Proxy129.persist:-1-ActionEventUtils.persistActionEvent:186-ActionEventUtils.onActionEvent:98-AccountManagerImpl.authenticateUser:2342
> > 2020-01-21 13:47:04,559 ERROR [c.c.a.ApiServlet] 
> > (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) unknown exception writing 
> > api response
> > com.cloud.utils.exception.CloudRuntimeException: Problem with getting the 
> > ec attribute 
> > at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1454)
> > at sun.reflect.GeneratedMethodAccessor145.invoke(Unknown Source)
> > at 
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at 
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > at 
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > at 
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at 
> > com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> > at 
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > at 
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > at 
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > at 
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > at com.sun.proxy.$Proxy129.

RE: Cloudstack - Unable to Upgrade database

[Sean Lair]

We are hitting a similar error message.  What version of the 
mysql-connector-java are you running?

yum list installed mysql-connector-java


-Original Message-
From: Cloud Udupi  
Sent: Friday, January 17, 2020 4:36 AM
To: users@cloudstack.apache.org
Subject: Re: Cloudstack - Unable to Upgrade database

Hi,

*You have done yum install mysql-connector-python ?*
Yes

We followed the below link for the installation as given in the Apache 
CloudStack link 
http://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html#mysql-connector-installation

*Which version is installed? (rpm -qa | grep ...)*

*mysql-connector-python*-8.0.19-1.el7.x86_64


*(I've installed it 4-5 days ago and it was fine)*


Yes this *(I've installed it 4-5 days ago and it was fine)* worked for me too.  
Our setup was working fine when we installed on 13th Jan 2020.


Since we are testing for HA we wanted to reinstall and check for final setup 
and we got the error as mentioned in the mail when we tried to install on 15th 
Jan 2020.

Please help,
Thanks,
Mark.

On Fri, Jan 17, 2020 at 2:55 PM Andrija Panic 
wrote:

> You have done yum install mysql-connector-python ?
>
> Which version is installed? (rpm -qa | grep ...) (I've installed it 
> 4-5 days ago and it was fine)
>
> On Fri, 17 Jan 2020 at 07:57, Cloud Udupi  wrote:
>
> > Hi,
> >
> > I am installing Cloudstack-4.13 on 3 node setup, earlier It used to 
> > work without any errors. But from 15th of Jan 2020, I'm facing some 
> > errors
> while
> > setting up Apache Cloudstack-management server.
> >
> > *Using this guide:*
> >
> >
> http://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig
> .html#mysql-connector-installation
> >
> > Following are the errors;
> > *I got this error while running 'cloudstack-setup-management'*
> >
> > [root@cloud1 ~]# *cloudstack-setup-management*
> >
> > Traceback (most recent call last):
> >
> >   File "/usr/bin/cloudstack-setup-management", line 24, in 
> >
> > from cloudutils.serviceConfigServer import cloudManagementConfig
> >
> >   File
> > "/usr/lib64/python2.7/site-packages/cloudutils/serviceConfigServer.p
> > y",
> > line 17, in 
> >
> > from db import Database
> >
> >   File "/usr/lib64/python2.7/site-packages/cloudutils/db.py", line 
> > 20, in 
> >
> > import mysql.connector
> >
> >   File 
> > "/usr/lib64/python2.7/site-packages/mysql/connector/__init__.py",
> > line 41, in 
> >
> > import dns.resolver
> >
> > ImportError: No module named dns.resolver
> >
> >
> >
> > After this I tried Installing python-dns, I was able to run 
> > "cloudstack-setup-management", but had issue with the dashboard not
> coming
> > up. When I checked the log file, I found the below errors.
> >
> >
> > *Error Message from the Apache CloudStack Management server.*
> >
> >
> > 2020-01-17 11:35:05,275 ERROR [c.c.u.DatabaseUpgradeChecker] 
> > (main:null)
> > (logid:) Unable to upgrade the database
> >
> > com.cloud.utils.exception.CloudRuntimeException: Problem with 
> > getting the ec attribute
> >
> > Caused by: java.lang.IllegalArgumentException: Can not set long 
> > field com.cloud.upgrade.dao.VersionVO.id to java.math.BigInteger
> >
> > at 
> > sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException
> > (UnsafeFieldAccessorImpl.java:167)
> >
> > at 
> > sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException
> > (UnsafeFieldAccessorImpl.java:171)
> >
> > 2020-01-17 11:35:05,283 WARN  
> > [o.a.c.s.m.c.ResourceApplicationContext]
> > (main:null) (logid:) Exception encountered during context 
> > initialization
> -
> > cancelling refresh attempt:
> org.springframework.context.ApplicationContext
> > Exception: Failed to start bean 'cloudStackLifeCycle'; nested 
> > exception
> is
> > com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade 
> > the database
> >
> > 2020-01-17 11:35:05,285 WARN  [o.e.j.w.WebAppContext] (main:null)
> (logid:)
> > Failed startup of context o.e.j.w.WebAppContext@6e38921c
> >
> >
> {/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{
> /usr/share/cloudstack-management/webapp}
> >
> > org.springframework.context.ApplicationContextException: Failed to 
> > start bean 'cloudStackLifeCycle'; nested exception is 
> > com.cloud.utils.exception
> > .CloudRuntimeException: Unable to upgrade the database
> >
> > Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable 
> > to upgrade the database
> >
> > Caused by: com.cloud.utils.exception.CloudRuntimeException: Problem 
> > with getting the ec attribute
> >
> > Caused by: java.lang.IllegalArgumentException: Can not set long 
> > field com.cloud.upgrade.dao.VersionVO.id to java.math.BigInteger
> >
> > at 
> > sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException
> > (UnsafeFieldAccessorImpl.java:167)
> >
> > at 
> > sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException
> > (UnsafeFieldAccessorImpl.java:171)
> >
> >
> >
> > *This is 

RE: Unable to authenticate in UI

[Sean Lair]

We are hitting a similar error message when doing other functions.  What 
version of the mysql-connector-java are you running?

yum list installed mysql-connector-java


-Original Message-
From: Ioan Marginean  
Sent: Tuesday, January 21, 2020 7:46 AM
To: users@cloudstack.apache.org
Subject: Re: Unable to authenticate in UI 

Is it possible to reset password directly into the database?
Thank you.

On 2020/01/21 11:55:18, Ioan Marginean  wrote: 
> Hello,
> 
> I facing a strange situation related to UI authentication. After restart 
> cloudstack-management I can't authenticate. The restart was needed because I 
> change settings related to e-mail notifications.
> My configuration is: CS version = 4.13, KVM. On error log I got:
> 
> 2020-01-21 13:47:04,111 DEBUG [c.c.a.ApiServlet] 
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) ===START===  172.16.1.106 -- 
> POST  
> 2020-01-21 13:47:04,117 DEBUG [c.c.u.AccountManagerImpl] 
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Attempting to log in user: 
> alex in domain 1
> 2020-01-21 13:47:04,119 DEBUG [o.a.c.s.a.PBKDF2UserAuthenticator] 
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Retrieving user: alex
> 2020-01-21 13:47:04,556 DEBUG [c.c.u.AccountManagerImpl] 
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) CIDRs from which account 
> 'Acct[95411505-a071-419e-bf6d-65c94971c1ad-alex]' is allowed to perform API 
> calls: 0.0.0.0/0,::/0
> 2020-01-21 13:47:04,556 DEBUG [c.c.u.AccountManagerImpl] 
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) User: alex in domain 1 has 
> successfully logged in
> 2020-01-21 13:47:04,557 DEBUG [c.c.u.d.T.Transaction] 
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) Rolling back the transaction: 
> Time = 1 Name =  qtp504527234-32; called by 
> -TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor.invoke:92-ReflectiveMethodInvocation.proceed:185-JdkDynamicAopProxy.invoke:212-$Proxy129.persist:-1-ActionEventUtils.persistActionEvent:186-ActionEventUtils.onActionEvent:98-AccountManagerImpl.authenticateUser:2342
> 2020-01-21 13:47:04,559 ERROR [c.c.a.ApiServlet] 
> (qtp504527234-32:ctx-d36239ff) (logid:b2b6e874) unknown exception writing api 
> response
> com.cloud.utils.exception.CloudRuntimeException: Problem with getting the ec 
> attribute 
>   at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1454)
>   at sun.reflect.GeneratedMethodAccessor145.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>   at 
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>   at 
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>   at com.sun.proxy.$Proxy129.persist(Unknown Source)
>   at 
> com.cloud.event.ActionEventUtils.persistActionEvent(ActionEventUtils.java:186)
>   at 
> com.cloud.event.ActionEventUtils.onActionEvent(ActionEventUtils.java:98)
>   at 
> com.cloud.user.AccountManagerImpl.authenticateUser(AccountManagerImpl.java:2342)
>   at sun.reflect.GeneratedMethodAccessor200.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>   at 
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>   at com.sun.proxy.$Proxy125.authenticateUser(Unknown Source)

RE: Snapshots on KVM corrupting disk images

[Sean Lair]

Hi Ivan, I wanted to respond here and see if you published a PR yet on this.

This is a very scary issue for us as customer can snapshot their volumes and 
end up causing corruption - and they blame us.  It's already happened - luckily 
we had Storage Array level snapshots in place as a safety net...

Thanks!!
Sean

-Original Message-
From: Ivan Kudryavtsev [mailto:kudryavtsev...@bw-sw.com] 
Sent: Sunday, January 27, 2019 7:29 PM
To: users ; cloudstack-fan 

Cc: dev 
Subject: Re: Snapshots on KVM corrupting disk images

Well, guys. I dived into CS agent scripts, which make volume snapshots and 
found there are no code for suspend/resume and also no code for qemu-agent call 
fsfreeze/fsthaw. I don't see any blockers adding that code yet and try to add 
it in nearest days. If tests go well, I'll publish the PR, which I suppose 
could be integrated into 4.11.3.

пн, 28 янв. 2019 г., 2:45 cloudstack-fan
cloudstack-...@protonmail.com.invalid:

> Hello Sean,
>
> It seems that you've encountered the same issue that I've been facing 
> during the last 5-6 years of using ACS with KVM hosts (see this 
> thread, if you're interested in additional details:
> https://mail-archives.apache.org/mod_mbox/cloudstack-users/201807.mbox
> /browser
> ).
>
> I'd like to state that creating snapshots of a running virtual machine 
> is a bit risky. I've implemented some workarounds in my environment, 
> but I'm still not sure that they are 100% effective.
>
> I have a couple of questions, if you don't mind. What kind of storage 
> do you use, if it's not a secret? Does you storage use XFS as a filesystem?
> Did you see something like this in your log-files?
> [***.***] XFS: qemu-kvm(***) possible memory allocation deadlock size
> 65552 in kmem_realloc (mode:0x250)
> [***.***] XFS: qemu-kvm(***) possible memory allocation deadlock size
> 65552 in kmem_realloc (mode:0x250)
> [***.***] XFS: qemu-kvm(***) possible memory allocation deadlock size
> 65552 in kmem_realloc (mode:0x250)
> Did you see any unusual messages in your log-file when the disaster 
> happened?
>
> I hope, things will be well. Wish you good luck and all the best!
>
>
> ‐‐‐ Original Message ‐‐‐
> On Tuesday, 22 January 2019 18:30, Sean Lair  wrote:
>
> > Hi all,
> >
> > We had some instances where VM disks are becoming corrupted when 
> > using
> KVM snapshots. We are running CloudStack 4.9.3 with KVM on CentOS 7.
> >
> > The first time was when someone mass-enabled scheduled snapshots on 
> > a
> lot of large number VMs and secondary storage filled up. We had to 
> restore all those VM disks... But believed it was just our fault with 
> letting secondary storage fill up.
> >
> > Today we had an instance where a snapshot failed and now the disk 
> > image
> is corrupted and the VM can't boot. here is the output of some commands:
> >
> >
> --
> --
> --
> --
> --
> --
> --
> 
> >
> > [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img 
> > check
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> > qemu-img: Could not open './184aa458-9d4b-4c1b-a3c6-23d28ea28e80': 
> > Could
> not read snapshots: File too large
> >
> > [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img 
> > info
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> > qemu-img: Could not open './184aa458-9d4b-4c1b-a3c6-23d28ea28e80': 
> > Could
> not read snapshots: File too large
> >
> > [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# ls -lh
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> > -rw-r--r--. 1 root root 73G Jan 22 11:04
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> >
> >
> --
> --
> --
> --
> --
> --
> --
> 

RE: Snapshots on KVM corrupting disk images

[Sean Lair]

Sounds good, I think something needs to be done.  Very scary that users can 
corrupt their VMs if they are doing volume snapshots


-Original Message-
From: Ivan Kudryavtsev [mailto:kudryavtsev...@bw-sw.com] 
Sent: Sunday, January 27, 2019 7:29 PM
To: users ; cloudstack-fan 

Cc: dev 
Subject: Re: Snapshots on KVM corrupting disk images

Well, guys. I dived into CS agent scripts, which make volume snapshots and 
found there are no code for suspend/resume and also no code for qemu-agent call 
fsfreeze/fsthaw. I don't see any blockers adding that code yet and try to add 
it in nearest days. If tests go well, I'll publish the PR, which I suppose 
could be integrated into 4.11.3.

пн, 28 янв. 2019 г., 2:45 cloudstack-fan
cloudstack-...@protonmail.com.invalid:

> Hello Sean,
>
> It seems that you've encountered the same issue that I've been facing 
> during the last 5-6 years of using ACS with KVM hosts (see this 
> thread, if you're interested in additional details:
> https://mail-archives.apache.org/mod_mbox/cloudstack-users/201807.mbox
> /browser
> ).
>
> I'd like to state that creating snapshots of a running virtual machine 
> is a bit risky. I've implemented some workarounds in my environment, 
> but I'm still not sure that they are 100% effective.
>
> I have a couple of questions, if you don't mind. What kind of storage 
> do you use, if it's not a secret? Does you storage use XFS as a filesystem?
> Did you see something like this in your log-files?
> [***.***] XFS: qemu-kvm(***) possible memory allocation deadlock size
> 65552 in kmem_realloc (mode:0x250)
> [***.***] XFS: qemu-kvm(***) possible memory allocation deadlock size
> 65552 in kmem_realloc (mode:0x250)
> [***.***] XFS: qemu-kvm(***) possible memory allocation deadlock size
> 65552 in kmem_realloc (mode:0x250)
> Did you see any unusual messages in your log-file when the disaster 
> happened?
>
> I hope, things will be well. Wish you good luck and all the best!
>
>
> ‐‐‐ Original Message ‐‐‐
> On Tuesday, 22 January 2019 18:30, Sean Lair  wrote:
>
> > Hi all,
> >
> > We had some instances where VM disks are becoming corrupted when 
> > using
> KVM snapshots. We are running CloudStack 4.9.3 with KVM on CentOS 7.
> >
> > The first time was when someone mass-enabled scheduled snapshots on 
> > a
> lot of large number VMs and secondary storage filled up. We had to 
> restore all those VM disks... But believed it was just our fault with 
> letting secondary storage fill up.
> >
> > Today we had an instance where a snapshot failed and now the disk 
> > image
> is corrupted and the VM can't boot. here is the output of some commands:
> >
> >
> --
> --
> --
> --
> --
> --
> --
> 
> >
> > [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img 
> > check
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> > qemu-img: Could not open './184aa458-9d4b-4c1b-a3c6-23d28ea28e80': 
> > Could
> not read snapshots: File too large
> >
> > [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img 
> > info
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> > qemu-img: Could not open './184aa458-9d4b-4c1b-a3c6-23d28ea28e80': 
> > Could
> not read snapshots: File too large
> >
> > [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# ls -lh
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> > -rw-r--r--. 1 root root 73G Jan 22 11:04
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> >
> >
> --
> --
> --
> --
> --
> --
> --
> --
> ---
> >
> > We tried restoring to before the snapshot failure, but still have
> strang

RE: Snapshots on KVM corrupting disk images

[Sean Lair]

Hello,

We are using NFS storage.  It is actually native NFS mounts on a NetApp storage 
system.  We haven't seen those log entries, but we also don't always know when 
a VM gets corrupted...  When we finally get a call that a VM is having issues, 
we've found that it was corrupted a while ago.


-Original Message-
From: cloudstack-fan [mailto:cloudstack-...@protonmail.com.INVALID] 
Sent: Sunday, January 27, 2019 1:45 PM
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Re: Snapshots on KVM corrupting disk images

Hello Sean,

It seems that you've encountered the same issue that I've been facing during 
the last 5-6 years of using ACS with KVM hosts (see this thread, if you're 
interested in additional details: 
https://mail-archives.apache.org/mod_mbox/cloudstack-users/201807.mbox/browser).

I'd like to state that creating snapshots of a running virtual machine is a bit 
risky. I've implemented some workarounds in my environment, but I'm still not 
sure that they are 100% effective.

I have a couple of questions, if you don't mind. What kind of storage do you 
use, if it's not a secret? Does you storage use XFS as a filesystem? Did you 
see something like this in your log-files?
[***.***] XFS: qemu-kvm(***) possible memory allocation deadlock size 65552 in 
kmem_realloc (mode:0x250) [***.***] XFS: qemu-kvm(***) possible memory 
allocation deadlock size 65552 in kmem_realloc (mode:0x250) [***.***] XFS: 
qemu-kvm(***) possible memory allocation deadlock size 65552 in kmem_realloc 
(mode:0x250) Did you see any unusual messages in your log-file when the 
disaster happened?

I hope, things will be well. Wish you good luck and all the best!


‐‐‐ Original Message ‐‐‐
On Tuesday, 22 January 2019 18:30, Sean Lair  wrote:

> Hi all,
>
> We had some instances where VM disks are becoming corrupted when using KVM 
> snapshots. We are running CloudStack 4.9.3 with KVM on CentOS 7.
>
> The first time was when someone mass-enabled scheduled snapshots on a lot of 
> large number VMs and secondary storage filled up. We had to restore all those 
> VM disks... But believed it was just our fault with letting secondary storage 
> fill up.
>
> Today we had an instance where a snapshot failed and now the disk image is 
> corrupted and the VM can't boot. here is the output of some commands:
>
> --
> --
> --
> --
> --
> --
> --
> 
>
> [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img check 
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> qemu-img: Could not open './184aa458-9d4b-4c1b-a3c6-23d28ea28e80': 
> Could not read snapshots: File too large
>
> [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img info 
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> qemu-img: Could not open './184aa458-9d4b-4c1b-a3c6-23d28ea28e80': 
> Could not read snapshots: File too large
>
> [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# ls -lh 
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> -rw-r--r--. 1 root root 73G Jan 22 11:04 
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
>
> --
> --
> --
> --
> --
> --
> --
> --
> ---
>
> We tried restoring to before the snapshot failure, but still have strange 
> errors:
>
> --
> --
>
> [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# ls -lh 
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> -rw-r--r--. 1 root root 73G Jan 22 11:04 
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
>
> [root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img info 
> ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> image: ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
> file format: qcow2
> virtual size: 50G (53687091200 b

RE: CloudStack 4.11.2 Snapshot Revert fail

[Sean Lair]

Luckily it was for a VM that is never touched in CloudStack.  The snaps were 
scheduled ones.  No, no changes to VM or template.

We are due to upgrade from 4.9.3 but we have not yet.

-Original Message-
From: Andrija Panic [mailto:andrija.pa...@gmail.com] 
Sent: Tuesday, January 22, 2019 11:05 AM
To: dev 
Cc: users@cloudstack.apache.org
Subject: Re: CloudStack 4.11.2 Snapshot Revert fail

Hi there,

after VM was deployed and snapshots created - was there any changes to VM or 
template from which VM was created - did ACS version get upgraded ?

Best

On Tue, 22 Jan 2019 at 17:52, li jerry  wrote:

> HI ALL
>
> I use CloudStack 4.11.2 to manage Xenserver 7.1.2 (XenServer CU2).
>
> VM snapshot for revert failure (snapshot does not contain memorysnapshot).
>
> 2019-01-23 00:06:54,210 DEBUG [c.c.a.m.ClusteredAgentAttache] 
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Seq 5-6201456686889173919: Forwarding Seq
> 5-6201456686889173919:  { Cmd , MgmtId: 240661250348494, via: 
> 5(wxac6001),
> Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.RevertToVMSnapshotCommand":{"reloadVm":false,"vmUuid":"b2a78e9c-06ab-4200-ad6d-fe095f622502","volumeTOs":[{"uuid":"7a58ffdc-b02c-41bf-963c-be56c2da0e9b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"WXACP01CL01_LUN10","id":19,"poolType":"PreSetup","host":"localhost","path":"/WXACP01CL01_LUN10","port":0,"url":"PreSetup://localhost/WXACP01CL01_LUN10/?ROLE=Primary=WXACP01CL01_LUN10","isManaged":false}},"name":"ROOT-33","size":21474836480,"path":"dd1cf43d-d5a4-4633-9c3e-8f73d1ccc484","volumeId":93,"vmName":"i-2-33-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":93,"deviceId":0,"hypervisorType":"XenServer"},{"uuid":"74268aa2-b4e5-4574-a981-027e55b5383f","volumeType":"DATADISK","dataStore":{"org.apache.
> cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"WXACP01CL01_LUN01",
> "id":1,"poolType":"PreSetup","host":"localhost","path":"/WXACP01CL01_L
> UN01","port":0,"url":"PreSetup://localhost/WXACP01CL01_LUN01/?ROLE=Pri
> mary=WXACP01CL01_LUN01","isManaged":false}},"name":"DATA-33"
> ,"size":1099511627776,"path":"e2ead686-d0bb-49f2-b656-77c2bf497990","v
> olumeId":95,"vmName":"i-2-33-VM","accountId":2,"format":"VHD","provisi
> oningType":"THIN","id":95,"deviceId":1,"hypervisorType":"XenServer"}],
> "target":{"id":27,"snapshotName":"i-2-33-VM_VS_20190122155503","type":
> "Disk","createTime":1548172503000,"current":true,"description":"asdfas
> df","quiescevm":true},"vmName":"i-2-33-VM","guestOSType":"CentOS
> 7","wait":0}}] } to 55935224135780
>
> 2019-01-23 00:06:54,222 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-14:null) (logid:) Seq 5-6201456686889173919:
> Processing:  { Ans: , MgmtId: 240661250348494, via: 5, Ver: v1, Flags: 
> 10, 
> [{"com.cloud.agent.api.RevertToVMSnapshotAnswer":{"result":false,"details":"
> Hypervisor 
> com.cloud.hypervisor.xenserver.resource.XenServer650Resource
> doesn't support guest OS type CentOS 7. you can choose 'Other install 
> media' to run it as HVM","wait":0}}] }
> 2019-01-23 00:06:54,223 DEBUG [c.c.a.t.Request] 
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Seq 5-6201456686889173919: Received:  { Ans: , MgmtId:
> 240661250348494, via: 5(wxac6001), Ver: v1, Flags: 10, { 
> RevertToVMSnapshotAnswer } }
> 2019-01-23 00:06:54,223 ERROR [o.a.c.s.v.DefaultVMSnapshotStrategy]
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Revert VM: i-2-33-VM to snapshot:
> i-2-33-VM_VS_20190122155503 failed due to  Hypervisor 
> com.cloud.hypervisor.xenserver.resource.XenServer650Resource doesn't 
> support guest OS type CentOS 7. you can choose 'Other install media' 
> to run it as HVM
> 2019-01-23 00:06:54,226 DEBUG [c.c.v.s.VMSnapshotManagerImpl] 
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Failed to revert vmsnapshot: 27
> com.cloud.utils.exception.CloudRuntimeException: Revert VM: i-2-33-VM 
> to
> snapshot: i-2-33-VM_VS_20190122155503 failed due to  Hypervisor 
> com.cloud.hypervisor.xenserver.resource.XenServer650Resource doesn't 
> support guest OS type CentOS 7. you can choose 'Other install media' 
> to run it as HVM
>   at
> org.apache.cloudstack.storage.vmsnapshot.DefaultVMSnapshotStrategy.revertVMSnapshot(DefaultVMSnapshotStrategy.java:393)
>   at
> com.cloud.vm.snapshot.VMSnapshotManagerImpl.orchestrateRevertToVMSnapshot(VMSnapshotManagerImpl.java:846)
>   at
> com.cloud.vm.snapshot.VMSnapshotManagerImpl.orchestrateRevertToVMSnapshot(VMSnapshotManagerImpl.java:1211)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at
> 

RE: CloudStack 4.11.2 Snapshot Revert fail

[Sean Lair]

Sorry, replied to wrong snapshot thread..


-Original Message-
From: Sean Lair 
Sent: Tuesday, January 22, 2019 11:48 AM
To: dev 
Cc: users@cloudstack.apache.org
Subject: RE: CloudStack 4.11.2 Snapshot Revert fail

Luckily it was for a VM that is never touched in CloudStack.  The snaps were 
scheduled ones.  No, no changes to VM or template.

We are due to upgrade from 4.9.3 but we have not yet.

-Original Message-
From: Andrija Panic [mailto:andrija.pa...@gmail.com]
Sent: Tuesday, January 22, 2019 11:05 AM
To: dev 
Cc: users@cloudstack.apache.org
Subject: Re: CloudStack 4.11.2 Snapshot Revert fail

Hi there,

after VM was deployed and snapshots created - was there any changes to VM or 
template from which VM was created - did ACS version get upgraded ?

Best

On Tue, 22 Jan 2019 at 17:52, li jerry  wrote:

> HI ALL
>
> I use CloudStack 4.11.2 to manage Xenserver 7.1.2 (XenServer CU2).
>
> VM snapshot for revert failure (snapshot does not contain memorysnapshot).
>
> 2019-01-23 00:06:54,210 DEBUG [c.c.a.m.ClusteredAgentAttache] 
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Seq 5-6201456686889173919: Forwarding Seq
> 5-6201456686889173919:  { Cmd , MgmtId: 240661250348494, via: 
> 5(wxac6001),
> Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.RevertToVMSnapshotCommand":{"reloadVm":false,"vmUuid":"b2a78e9c-06ab-4200-ad6d-fe095f622502","volumeTOs":[{"uuid":"7a58ffdc-b02c-41bf-963c-be56c2da0e9b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"WXACP01CL01_LUN10","id":19,"poolType":"PreSetup","host":"localhost","path":"/WXACP01CL01_LUN10","port":0,"url":"PreSetup://localhost/WXACP01CL01_LUN10/?ROLE=Primary=WXACP01CL01_LUN10","isManaged":false}},"name":"ROOT-33","size":21474836480,"path":"dd1cf43d-d5a4-4633-9c3e-8f73d1ccc484","volumeId":93,"vmName":"i-2-33-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":93,"deviceId":0,"hypervisorType":"XenServer"},{"uuid":"74268aa2-b4e5-4574-a981-027e55b5383f","volumeType":"DATADISK","dataStore":{"org.apache.
> cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"WXACP01CL01_LUN01",
> "id":1,"poolType":"PreSetup","host":"localhost","path":"/WXACP01CL01_L
> UN01","port":0,"url":"PreSetup://localhost/WXACP01CL01_LUN01/?ROLE=Pri
> mary=WXACP01CL01_LUN01","isManaged":false}},"name":"DATA-33"
> ,"size":1099511627776,"path":"e2ead686-d0bb-49f2-b656-77c2bf497990","v
> olumeId":95,"vmName":"i-2-33-VM","accountId":2,"format":"VHD","provisi
> oningType":"THIN","id":95,"deviceId":1,"hypervisorType":"XenServer"}],
> "target":{"id":27,"snapshotName":"i-2-33-VM_VS_20190122155503","type":
> "Disk","createTime":1548172503000,"current":true,"description":"asdfas
> df","quiescevm":true},"vmName":"i-2-33-VM","guestOSType":"CentOS
> 7","wait":0}}] } to 55935224135780
>
> 2019-01-23 00:06:54,222 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-14:null) (logid:) Seq 5-6201456686889173919:
> Processing:  { Ans: , MgmtId: 240661250348494, via: 5, Ver: v1, Flags: 
> 10, 
> [{"com.cloud.agent.api.RevertToVMSnapshotAnswer":{"result":false,"details":"
> Hypervisor
> com.cloud.hypervisor.xenserver.resource.XenServer650Resource
> doesn't support guest OS type CentOS 7. you can choose 'Other install 
> media' to run it as HVM","wait":0}}] }
> 2019-01-23 00:06:54,223 DEBUG [c.c.a.t.Request] 
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Seq 5-6201456686889173919: Received:  { Ans: , MgmtId:
> 240661250348494, via: 5(wxac6001), Ver: v1, Flags: 10, { 
> RevertToVMSnapshotAnswer } }
> 2019-01-23 00:06:54,223 ERROR [o.a.c.s.v.DefaultVMSnapshotStrategy]
> (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9)
> (logid:a9ef7fe7) Revert VM: i-2-33-VM to snapshot:
> i-2-33-VM_VS_20190122155503 failed due to 

RE: Snapshots on KVM corrupting disk images

[Sean Lair]

Hi Simon

It is NFS mount.  The underlying storage is NetApp that we run a lot of 
different environments on, it is rock-solid, the only issues we've had are with 
KVM snapshots.

Thanks
Sean

-Original Message-
From: Simon Weller [mailto:swel...@ena.com.INVALID] 
Sent: Tuesday, January 22, 2019 10:42 AM
To: users@cloudstack.apache.org; d...@cloudstack.apache.org
Subject: Re: Snapshots on KVM corrupting disk images

Sean,


What underlying primary storage are you using and how is it being utilized by 
ACS (e.g. NFS, shared mount et al)?



- Si



From: Sean Lair 
Sent: Tuesday, January 22, 2019 10:30 AM
To: users@cloudstack.apache.org; d...@cloudstack.apache.org
Subject: Snapshots on KVM corrupting disk images

Hi all,

We had some instances where VM disks are becoming corrupted when using KVM 
snapshots.  We are running CloudStack 4.9.3 with KVM on CentOS 7.

The first time was when someone mass-enabled scheduled snapshots on a lot of 
large number VMs and secondary storage filled up.  We had to restore all those 
VM disks...  But believed it was just our fault with letting secondary storage 
fill up.

Today we had an instance where a snapshot failed and now the disk image is 
corrupted and the VM can't boot.  here is the output of some commands:

---
[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img check 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
qemu-img: Could not open './184aa458-9d4b-4c1b-a3c6-23d28ea28e80': Could not 
read snapshots: File too large

[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img info 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
qemu-img: Could not open './184aa458-9d4b-4c1b-a3c6-23d28ea28e80': Could not 
read snapshots: File too large

[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# ls -lh 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
-rw-r--r--. 1 root root 73G Jan 22 11:04 ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
---

We tried restoring to before the snapshot failure, but still have strange 
errors:

--
[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# ls -lh 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
-rw-r--r--. 1 root root 73G Jan 22 11:04 ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80

[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img info 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
image: ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 73G
cluster_size: 65536
Snapshot list:
IDTAG VM SIZEDATE   VM CLOCK
1 a8fdf99f-8219-4032-a9c8-87a6e09e7f95   3.7G 2018-12-23 11:01:43 
3099:35:55.242
2 b4d74338-b0e3-4eeb-8bf8-41f6f75d9abd   3.8G 2019-01-06 11:03:16 
3431:52:23.942
Format specific information:
compat: 1.1
lazy refcounts: false

[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img check 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
tcmalloc: large alloc 1539750010880 bytes == (nil) @  0x7fb9cbbf7bf3 
0x7fb9cbc19488 0x7fb9cb71dc56 0x55d16ddf1c77 0x55d16ddf1edc 0x55d16ddf2541 
0x55d16ddf465e 0x55d16ddf8ad1 0x55d16de336db 0x55d16de373e6 0x7fb9c63a3c05 
0x55d16ddd9f7d No errors were found on the image.

[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img snapshot -l 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
Snapshot list:
IDTAG VM SIZEDATE   VM CLOCK
1 a8fdf99f-8219-4032-a9c8-87a6e09e7f95   3.7G 2018-12-23 11:01:43 
3099:35:55.242
2 b4d74338-b0e3-4eeb-8bf8-41f6f75d9abd   3.8G 2019-01-06 11:03:16 
3431:52:23.942
--

Everyone is now extremely hesitant to use snapshots in KVM  We tried 
deleting the snapshots in the restored disk image, but it errors out...


Does anyone else have issues with KVM snapshots?  We are considering just 
disabling this functionality now...

Thanks
Sean

Snapshots on KVM corrupting disk images

[Sean Lair]

Hi all,

We had some instances where VM disks are becoming corrupted when using KVM 
snapshots.  We are running CloudStack 4.9.3 with KVM on CentOS 7.

The first time was when someone mass-enabled scheduled snapshots on a lot of 
large number VMs and secondary storage filled up.  We had to restore all those 
VM disks...  But believed it was just our fault with letting secondary storage 
fill up.

Today we had an instance where a snapshot failed and now the disk image is 
corrupted and the VM can't boot.  here is the output of some commands:

---
[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img check 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
qemu-img: Could not open './184aa458-9d4b-4c1b-a3c6-23d28ea28e80': Could not 
read snapshots: File too large

[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img info 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
qemu-img: Could not open './184aa458-9d4b-4c1b-a3c6-23d28ea28e80': Could not 
read snapshots: File too large

[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# ls -lh 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
-rw-r--r--. 1 root root 73G Jan 22 11:04 ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
---

We tried restoring to before the snapshot failure, but still have strange 
errors:

--
[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# ls -lh 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
-rw-r--r--. 1 root root 73G Jan 22 11:04 ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80

[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img info 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
image: ./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 73G
cluster_size: 65536
Snapshot list:
IDTAG VM SIZEDATE   VM CLOCK
1 a8fdf99f-8219-4032-a9c8-87a6e09e7f95   3.7G 2018-12-23 11:01:43 
3099:35:55.242
2 b4d74338-b0e3-4eeb-8bf8-41f6f75d9abd   3.8G 2019-01-06 11:03:16 
3431:52:23.942
Format specific information:
compat: 1.1
lazy refcounts: false

[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img check 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
tcmalloc: large alloc 1539750010880 bytes == (nil) @  0x7fb9cbbf7bf3 
0x7fb9cbc19488 0x7fb9cb71dc56 0x55d16ddf1c77 0x55d16ddf1edc 0x55d16ddf2541 
0x55d16ddf465e 0x55d16ddf8ad1 0x55d16de336db 0x55d16de373e6 0x7fb9c63a3c05 
0x55d16ddd9f7d
No errors were found on the image.

[root@cloudkvm02 c3be0ae5-2248-3ed6-a0c7-acffe25cc8d3]# qemu-img snapshot -l 
./184aa458-9d4b-4c1b-a3c6-23d28ea28e80
Snapshot list:
IDTAG VM SIZEDATE   VM CLOCK
1 a8fdf99f-8219-4032-a9c8-87a6e09e7f95   3.7G 2018-12-23 11:01:43 
3099:35:55.242
2 b4d74338-b0e3-4eeb-8bf8-41f6f75d9abd   3.8G 2019-01-06 11:03:16 
3431:52:23.942
--

Everyone is now extremely hesitant to use snapshots in KVM  We tried 
deleting the snapshots in the restored disk image, but it errors out...


Does anyone else have issues with KVM snapshots?  We are considering just 
disabling this functionality now...

Thanks
Sean

RE: [VOTE] Apache CloudStack 4.11.1.0 LTS [RC3]

[Sean Lair]

Would someone mind testing testing a Restart VPC w/ Cleanup on a VPC that has a 
private gateway configured?  The test 
"test_03_vpc_privategw_restart_vpc_cleanup" is failing due to the following 
(according to logs).  My test environment is not available right now so I can't 
check myself.  I don't have this problem in my 4.9.3 prod environment. 


Java.lang.NullPointerException
at 
com.cloud.network.router.NicProfileHelperImpl.createPrivateNicProfileForGateway(NicProfileHelperImpl.java:95)



NicProfileHelperImpl.java (Lines 93 - 95)

final PrivateIpAddress ip =
new PrivateIpAddress(ipVO, 
privateNetwork.getBroadcastUri().toString(), privateNetwork.getGateway(), 
netmask,

NetUtils.long2Mac(NetUtils.createSequenceBasedMacAddress(ipVO.getMacAddress(), 
NetworkModel.MACIdentifier.value(;


Thanks
Sean

-Original Message-
From: Paul Angus [mailto:paul.an...@shapeblue.com] 
Sent: Thursday, June 21, 2018 11:00 AM
To: d...@cloudstack.apache.org; users@cloudstack.apache.org
Subject: [VOTE] Apache CloudStack 4.11.1.0 LTS [RC3]

Hi All,



I've created a 4.11.1.0 release (RC3), with the following artefacts up for 
testing and a vote:
The changes since RC2 are listed at the end of this email.



Git Branch and Commit SH:

https://gitbox.apache.org/repos/asf?p=cloudstack.git;a=shortlog;h=refs/heads/4.11.1.0-RC20180621T1552

Commit: 2cb2dacbe75a23f5068b80f6ea45031c29052c31



Source release (checksums and signatures are available at the same

location):

https://dist.apache.org/repos/dist/dev/cloudstack/4.11.1.0/



PGP release keys (signed using 8B309F7251EE0BC8):

https://dist.apache.org/repos/dist/release/cloudstack/KEYS



The vote will be open for at least 72hrs.



For sanity in tallying the vote, can PMC members please be sure to indicate 
"(binding)" with their vote?



[ ] +1  approve

[ ] +0  no opinion

[ ] -1  disapprove (and reason why)





Additional information:



For users' convenience, I've built packages from 
5f48487dc62fd1decaabc4ab2a10f549d6c82400 and published RC1 repository here:

http://packages.shapeblue.com/testing/4111rc3/



The release notes are still work-in-progress, but the systemvm template upgrade 
section has been updated. You may refer the following for systemvm template 
upgrade testing:

http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/latest/index.html



4.11.1 systemvm templates are available from here:

http://packages.shapeblue.com/systemvmtemplate/4.11.1-rc1/




Changes Since RC2:

Merged #2712 reuse ip for non redundant VPC 6 hours ago Merged #2714 send 
unsupported answer only when applicable 10 hours ago Merged #2715 smoketest: 
Fix test_vm_life_cycle secure migration tests a day ago Merged #2493 
CLOUDSTACK-10326: Prevent hosts fall into Maintenance when there are running 
VMs on it a day ago Merged #2716 configdrive: make fewer mountpoints on hosts a 
day ago Merged #2681 Source NAT option on Private Gateway 2 days ago Merged 
#2710 comply with api key constraint 2 days ago Merged #2706 packaging: use 
libuuid x86_64 package for cloudstack-common 2 days ago

Kind regards,

Paul Angus


paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
  
 

RE: HA-enabled VM not starting or migrating to another host if current host goes down

[Sean Lair]

Yea, we had the exact same problem.  VM HA is broken in 4.9 - at least under KVM

We use this Pull Request in our environment to fix it.

https://github.com/apache/cloudstack/pull/2474

However, as stated in the PR, enable libvirt (edit /etc/libvirt/qemu.conf and 
change and comment: lock_manager = "lockd")

https://libvirt.org/locking-lockd.html



-Original Message-
From: Parth Patel [mailto:parthpatel2...@gmail.com] 
Sent: Saturday, February 24, 2018 10:54 PM
To: users@cloudstack.apache.org
Subject: Re: HA-enabled VM not starting or migrating to another host if current 
host goes down

Hi Simon,

I'm using KVM hypervisor.

On Sun 25 Feb, 2018, 01:41 Simon Weller,  wrote:

> Which hypervisor are you using?
>
> Simon Weller/615-312-6068
>
> -Original Message-
> From: Parth Patel [parthpatel2...@gmail.com]
> Received: Saturday, 24 Feb 2018, 11:33AM
> To: users@cloudstack.apache.org [users@cloudstack.apache.org]
> Subject: HA-enabled VM not starting or migrating to another host if 
> current host goes down
>
> Hi,
>
> I am developing an enterprise-level cloud infrastructure currently 
> using Cloudstack 4.9. My head of department wished to check a specific 
> failsafe scenario. It is as follows:
>
> A highly available VM whose continuous execution even in event of some 
> storage or connection error, should remain running or migrate to 
> another host automatically if anything were to happen to the current 
> host. I tried removing the LAN cable from the current host after an 
> HA-enabled VM was executing on it, but the management server would not 
> auto-start the VM on another host. It kept printing error messages 
> such as "Communication failure. Host 5 timed out due to even 
> PingTimeout" in management server logs. I have manually set the ping 
> timeout duration to 30 seconds and its multiplier value to 1.
>
> Any suggestions as to what extra configuration is needed to make 
> Cloudstack start the VM on another host or migrate it?
>
> Just to be clear, the host on which the HA-enabled VM is running does 
> not have its primary and secondary storage added to the management 
> server, so the primary and secondary storage disks on NFS shares are 
> already available to the management server if it decides to start the 
> instance on another suitable host (of which there are 2).
>
> I tried searching some feature like this in Cloudstack administration 
> docs but could not find anything fitting to this scenario.
>
> Regards,
> Parth Patel
>

Traffic being tagged with VLAN 0 w/ KVM

[Sean Lair]

Hi all,

We are seeing some strange behavior with our KVM guests.  When the guest VM is 
on the same KVM host as the vRouter, traffic to the guest VM is being tagged 
with VLAN 0 (it should just be untagged traffic).  This breaks connectivity for 
some operating systems that aren't expecting packets to be tagged with a VLAN.  
Technically, the guest VLAN is 306, and we see Cloudstack creating the 306 VLAN 
sub-interface on the host and the corresponding bridge interface - that all 
looks good.

This only occurs when the guest VM is on the same host as the vRouter.  
However, it is likely that the physical switch connecting the KVM hosts is 
being smart and stripping the VLAN 0 tag... and that is why we don't have the 
problem with the traffic flows between hosts.

Cloudstack 4.9.3 w/ Advanced Networking
CentOS 7.4 KVM host (fully up-to-date)
Bridge networking within the host

Here is a tcpdump on the guest VM, showing the dot1q "VLAN 0" tag on the 
response packets only.  The guest doesn't support VLAN tagging, and just says 
request timed out to the ping.

01:18:22.723029 02:00:0c:a3:00:05 > 02:00:56:46:00:04, ethertype IPv4 (0x0800), 
length 98: 10.1.1.246 > 8.8.8.8: ICMP echo request, id 1375, seq 150, length 64
01:18:22.738067 02:00:56:46:00:04 > 02:00:0c:a3:00:05, ethertype 802.1Q 
(0x8100), length 102: vlan 0, p 0, ethertype IPv4, 8.8.8.8 > 10.1.1.246: ICMP 
echo reply, id 1375, seq 150, length 64
01:18:23.724363 02:00:0c:a3:00:05 > 02:00:56:46:00:04, ethertype IPv4 (0x0800), 
length 98: 10.1.1.246 > 8.8.8.8: ICMP echo request, id 1375, seq 151, length 64
01:18:23.739301 02:00:56:46:00:04 > 02:00:0c:a3:00:05, ethertype 802.1Q 
(0x8100), length 102: vlan 0, p 0, ethertype IPv4, 8.8.8.8 > 10.1.1.246: ICMP 
echo reply, id 1375, seq 151, length 64
01:18:24.725480 02:00:0c:a3:00:05 > 02:00:56:46:00:04, ethertype IPv4 (0x0800), 
length 98: 10.1.1.246 > 8.8.8.8: ICMP echo request, id 1375, seq 152, length 64
01:18:24.740498 02:00:56:46:00:04 > 02:00:0c:a3:00:05, ethertype 802.1Q 
(0x8100), length 102: vlan 0, p 0, ethertype IPv4, 8.8.8.8 > 10.1.1.246: ICMP 
echo reply, id 1375, seq 152, length 64
01:18:25.726752 02:00:0c:a3:00:05 > 02:00:56:46:00:04, ethertype IPv4 (0x0800), 
length 98: 10.1.1.246 > 8.8.8.8: ICMP echo request, id 1375, seq 153, length 64
01:18:25.741749 02:00:56:46:00:04 > 02:00:0c:a3:00:05, ethertype 802.1Q 
(0x8100), length 102: vlan 0, p 0, ethertype IPv4, 8.8.8.8 > 10.1.1.246: ICMP 
echo reply, id 1375, seq 153, length 64

Any idea how to stop KVM from adding that VLAN 0 dot1q tag?  KVM is a new add 
to our environment.

Thanks
Sean

RE: XenServer Licensing Change - Switch Hypervisors?

[Sean Lair]

Thanks for the reply Nux, yea we originally chose XenServer over KVM because 
KVM didn't support all of the VM snapshot functionality of XenServer.

We are evaluating switching to KVM now...  But don't have a good way of moving 
customers over from XenServer host to KVM hosts...

We are on XenServer 6.5 and with the new Spectre and Meltdown vulnerabilities 
not being patched in 6.5...  We may accelerate the move to KVM.


-Original Message-
From: Nux! [mailto:n...@li.nux.ro] 
Sent: Friday, January 5, 2018 4:22 PM
To: users <users@cloudstack.apache.org>
Subject: Re: XenServer Licensing Change - Switch Hypervisors?

If you have expertise with XenServer and don't mind paying, then it's not a bad 
direction to follow. It's a nice HV.
On the long term I think KVM will be a much better solution though.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
> From: "Sean Lair" <sl...@ippathways.com>
> To: "users" <users@cloudstack.apache.org>
> Sent: Wednesday, 3 January, 2018 00:53:32
> Subject: XenServer Licensing Change - Switch Hypervisors?

> It looks like XenServer 7.3 will no longer have the following features 
> in the Free Edition.  Is anyone considering moving from Free to 
> Standard Edition or possibly to another hyper-visor (like KVM) for their 
> CloudStack environment?
> 
> Thoughts?  We are looking more at KVM at this point, any feature gaps 
> we should be aware of?
> 
> Free Edition Changes
> 
> -  Limited to up to 3 hosts per clusters
> 
> -  No Pool High-Availability
> 
> -  No Dynamic Memory Control (DMC)
> 
> https://www.citrix.com/content/dam/citrix/en_us/documents/product-over
> view/citrix-xenserver-feature-matrix.pdf
> 
> Thanks
> Sean

XenServer Licensing Change - Switch Hypervisors?

[Sean Lair]

It looks like XenServer 7.3 will no longer have the following features in the 
Free Edition.  Is anyone considering moving from Free to Standard Edition or 
possibly to another hyper-visor (like KVM) for their CloudStack environment?

Thoughts?  We are looking more at KVM at this point, any feature gaps we should 
be aware of?

Free Edition Changes

-  Limited to up to 3 hosts per clusters

-  No Pool High-Availability

-  No Dynamic Memory Control (DMC)

https://www.citrix.com/content/dam/citrix/en_us/documents/product-overview/citrix-xenserver-feature-matrix.pdf

Thanks
Sean

RE: [restart network services]

[Sean Lair]

For these false-positive error messages:

site-to-site Vpn Connection to x on router r-15854-VM(id: 15854)  just 
switch from Connected to Disconnected

We also had that issue, it was a bug and is fixed in 4.9.3 by this pull request:

https://github.com/apache/cloudstack/pull/2040

-Original Message-
From: Gian Paolo Buono [mailto:gianpaolo.bu...@gesca.it] 
Sent: Tuesday, September 26, 2017 2:05 AM
To: nitinkumar.mahar...@accelerite.com
Cc: users@cloudstack.apache.org
Subject: Re: [restart network services]

Hi Nitin,
thanks but i have a big problem on this vpc, because have a site to stite vpn 
and often i receive this message:

site-to-site Vpn Connection to x on router r-15854-VM(id: 15854)  just 
switch from Connected to Disconnected

The vpn is always UP, infact does not lose ping packages, but on the same vpc i 
have a forwarding rules on port https 443 that hangs and I must restart the 
firewall rules
Furthermore i have  found in  /var/log/cloud.log csnetfilter.py, but what is 
this scripts ? Can i Block it ?

Please, help me...thanks
Gian Paolo


On 09/26/2017 08:55 AM, Nitin Kumar Maharana wrote:

Hi Gianpaolo,

Actually, it runs a couple of scripts based on the number of NICs and rules 
configured etc.. present in the router.
There is no specific command run on it but I can say a couple of commands run 
with the help of existing scripts.

To restart the network on the router VM. The below command can be used.
"sudo /etc/init.d/networking restart”

Hope this will help!!


Thanks,
Nitin



On 26-Sep-2017, at 11:34 AM, Gian Paolo Buono 
>
 wrote:

Hi all,

which command on router is run  when click on restart network services (home 
--> network --> 10.1.1.0)

Thanks bye

DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

RE: Quick 1 Question Survey

[Sean Lair]

CloudStack Mgmt: CentOS Linux release 7.3.1611
Hypervisor: XenServer

-Original Message-
From: Rene Moser [mailto:m...@renemoser.net] 
Sent: Tuesday, September 12, 2017 7:13 AM
To: users@cloudstack.apache.org
Subject: Quick 1 Question Survey

What Linux OS and release are you running below your:

* CloudStack/Cloudplatform Management
* KVM/XEN Hypvervisor Host

Possible answer example

Cloudstack Management = centos6
KVM/XEN = None, No KVM/XEN

Thanks in advance

Regards
René

RE: [ANNOUNCE][CLOUDSTACK] Apache CloudStack 4.9.3.0 (LTS)

[Sean Lair]

FYI, small thing, some of the fixed issues aren't showing up in the release 
notes.  They were merged, but just not in the release notes.  Here are a few I 
noticed:

https://issues.apache.org/jira/browse/CLOUDSTACK-9872
https://issues.apache.org/jira/browse/CLOUDSTACK-9873
https://issues.apache.org/jira/browse/CLOUDSTACK-9165

Thanks
Sean


-Original Message-
From: Rohit Yadav [mailto:bhais...@apache.org] 
Sent: Tuesday, September 12, 2017 1:40 AM
To: d...@cloudstack.apache.org; users@cloudstack.apache.org
Subject: [ANNOUNCE][CLOUDSTACK] Apache CloudStack 4.9.3.0 (LTS)

# Apache CloudStack LTS Maintenance Release 4.9.3.0

The Apache CloudStack project is pleased to announce the release of CloudStack 
4.9.3.0 as part of its LTS 4.9.x releases. The CloudStack
4.9.3.0 release contains more than 180 fixes since the CloudStack 4.9.2.0 
release. Cloudstack LTS branches are supported for 20 months, and will receive 
updates for first 14 months and only security updates in its last 6 months. The 
4.9 LTS branch is supported until  1 June 2018.

Apache CloudStack is an integrated Infrastructure-as-a-Service (IaaS) software 
platform that allows users to build feature-rich public and private cloud 
environments. CloudStack includes an intuitive user interface and rich API for 
managing the compute, networking, software, and storage resources. The project 
became an Apache top level project in March, 2013.

More information about Apache CloudStack can be found at:
http://cloudstack.apache.org/

# Documentation

What's new in CloudStack 4.9.3.0:
http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.9.3.0/about.html

The 4.9.3.0 release notes include a full list of issues fixed, as well as 
upgrade instructions from previous versions of Apache CloudStack, and can be 
found at:
http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.9.3.0

The official installation, administration and API documentation for each of the 
releases are available on our documentation page:
http://docs.cloudstack.apache.org/

# Downloads

The official source code for the 4.9.3.0 release can be downloaded from our 
downloads page:
http://cloudstack.apache.org/downloads.html

In addition to the official source code release, individual contributors have 
also made convenience binaries available on the Apache CloudStack download 
page, and can be found at:

http://www.shapeblue.com/packages/
http://cloudstack.apt-get.eu/ubuntu/dists/ (packages to be published soon) 
http://cloudstack.apt-get.eu/centos/6/ (packages to be published soon) 
http://cloudstack.apt-get.eu/centos/7/ (packages to be published soon)

Regards,
Rohit Yadav

RE: [DISCUSS] CloudStack 4.9.3.0 (LTS)

[Sean Lair]

Hi Rohit

I previous suggested these for 4.9.3.0

https://github.com/apache/cloudstack/pull/2041 (VR related jobs scheduled and 
run twice on mgmt servers)
https://github.com/apache/cloudstack/pull/2040 (Bug in monitoring of S2S VPNs - 
also exists in 4.10)
https://github.com/apache/cloudstack/pull/1966 (IPSEC VPNs do not work after 
vRouter reboot)

I'd also like to suggest these:
https://github.com/apache/cloudstack/pull/1246 (unable to use reserved IP range 
in a network)
https://github.com/apache/cloudstack/pull/2201 (VPC VR doesn't respond to DNS 
requests from remote access vpn clients)


Thanks
Sean

-Original Message-
From: Rohit Yadav [mailto:rohit.ya...@shapeblue.com] 
Sent: Monday, July 24, 2017 5:56 AM
To: d...@cloudstack.apache.org; users@cloudstack.apache.org
Subject: Re: [DISCUSS] CloudStack 4.9.3.0 (LTS)

All,


We'll accept bugfixes on 4.9 branch till end of next week, following which I'll 
start release work towards 4.9.3.0 (LTS) release. Please help review 
outstanding PRs, share PRs that we should consider and advise/suggest issues 
that need to be reverted/backported, for example see: 
https://github.com/apache/cloudstack/pull/2052


Thank you for your support and co-operation.


- Rohit


From: Rohit Yadav 
Sent: Sunday, July 23, 2017 1:26:48 PM
To: d...@cloudstack.apache.org; users@cloudstack.apache.org
Subject: Re: [DISCUSS] CloudStack 4.9.3.0 (LTS)

All,


I've started looking into reviewing/testing/merging of the PRs targeting 4.9+, 
I'll share some plans around 4.9.3.0 soon. Meanwhile, help in reporting any 
major/critical bugs and PRs we should consider reviewing/testing/merging. 
Thanks.


- Rohit

rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue




rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
  
 

RE: [DISCUSS] CloudStack 4.9.3.0 (LTS)

[Sean Lair]

Here are three issues we ran into in 4.9.2.0.  We have been running all of 
these fixes for several months without issues.  The code changes are all very 
easy/small, but had a big impact for us.

I'd respectfully suggest they go into 4.9.3.0:

https://github.com/apache/cloudstack/pull/2041 (VR related jobs scheduled and 
run twice on mgmt servers)
https://github.com/apache/cloudstack/pull/2040 (Bug in monitoring of S2S VPNs - 
also exists in 4.10)
https://github.com/apache/cloudstack/pull/1966 (IPSEC VPNs do not work after 
vRouter reboot)

Thanks
Sean

-Original Message-
From: Rohit Yadav [mailto:rohit.ya...@shapeblue.com] 
Sent: Friday, July 7, 2017 1:14 AM
To: d...@cloudstack.apache.org
Cc: users@cloudstack.apache.org
Subject: [DISCUSS] CloudStack 4.9.3.0 (LTS)

All,


With 4.10.0.0 voted, I would like to start some initial discussion around the 
next minor LTS release 4.9.3.0. At the moment I don't have a timeline, plans or 
dates to share but I would like to engage with the community to gather list of 
issues, commits, PRs that we should consider for the next LTS release 4.9.3.0.


To reduce our test and QA scope, we don't want to consider changes that are new 
feature, or enhancements but strictly blockers/critical/major bugfixes and 
security related fixes, and we can consider reverting any already 
committed/merged PR(s) on 4.9 branch (committed since 4.9.2.0).


Please go through list of commits since 4.9.2.0 (you can also run, git log 
4.9.2.0..4.9) and let us know if there is any change we should consider 
reverting:

https://github.com/apache/cloudstack/commits/4.9


I started backporting some 
fixes on the 4.9 branch, please go through the following PR and raise 
objections on changes/commits that we should not backport or revert:

https://github.com/apache/cloudstack/pull/2052


Lastly, please also share any PRs that we should consider reviewing+merging on 
4.9 branch for the 4.9.3.0 release effort.


- Rohit

rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
  
 

RE: Remote VPN not working on 4.7.1

[Sean Lair]

Hello, I remember that issue.  It was fixed by this pull request:

https://github.com/apache/cloudstack/pull/1423

This is a related bug that hasn't been committed yet:

https://github.com/apache/cloudstack/pull/1966


From: Vijay Sachdeva [mailto:vijay.sachd...@indiqus.com]
Sent: Monday, April 10, 2017 3:56 AM
To: users@cloudstack.apache.org
Subject: Remote VPN not working on 4.7.1

Dear Team,

Trying to enable to Remote VPN in cloudstack 4.7.1, it gets create but in VR 
IPsec service shows stopped. Once you manually start the Ipsec service in VR, 
remote VPN gets connected. But there is also one thing when you disable VPN 
from UI and again re-enable it, remote VPN stops working.

Any help or suggestion is highly appreciated..!!

Thanks

Vijay Sachdeva
Consultant - Cloud & Virtualisation

[cid:image001.png@01D2B367.75D914D0]
A-98, LGF, C.R.Park, New Delhi - 110019
24x7 011-40551409 | M +918826699409
www.indiqus.com

RE: Bug in Snapshot Retention?

[Sean Lair]

Thanks!  That PR does look like it will fix our issue and allow the retention 
thread to delete the allocated snapshot.

A bigger question is do you think the management server should be checking for 
these orphaned allocated snapshots periodically or at startup?



-Original Message-
From: Simon Weller [mailto:swel...@ena.com] 
Sent: Monday, April 25, 2016 10:01 AM
To: users@cloudstack.apache.org
Subject: Re: Bug in Snapshot Retention?

There is an open issue (and associated PR) that covers this I think

https://issues.apache.org/jira/browse/CLOUDSTACK-9200
and PR: https://github.com/apache/cloudstack/pull/1282

- Si



From: Anshul Gangwar <anshul.gang...@accelerite.com>
Sent: Sunday, April 24, 2016 11:48 PM
To: users@cloudstack.apache.org
Subject: Re: Bug in Snapshot Retention?

They can stuck in Allocated state if the management server is restarted just 
before the snapshot transitioned to new state. Time period when this can happen 
will depend on the jobs scheduled on VM to which this volume is attached.

This sounds like a bug that snapshot which is stuck in Allocated state is 
failing to delete. Can you create a ticket to track this?

Regards,
Anshul



> On 20-Apr-2016, at 8:00 PM, Sean Lair <sl...@ippathways.com> wrote:
>
> Thanks for the responses all.  The "Removed Field" for the snapshots with the 
> status of "BackedUp" is NULL.
>
> I combed the logs and found the exception below.  It was successfully 
> deleting snapshots before that log entry, then errored on the "Allocated" 
> snapshot and stopped any further deletions.  I'm not sure what allocated 
> mean, but will start researching.
>
> 2016-04-20 00:44:38,830 DEBUG [c.c.s.s.SnapshotManagerImpl] 
> (Work-Job-Executor-1:ctx-954cbd99 job-7396/job-7401 ctx-d0261c6c) 
> (logid:bafb5d42) post process snapshot failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to delete 
> snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete 
> snapshotshot 1351 due to it is in Allocated Status
>at 
> com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:478)
>at 
> com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateRecurringSnapshotForPolicy(SnapshotManagerImpl.java:420)
>at 
> com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateSnapshot(SnapshotManagerImpl.java:399)
>at 
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1010)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>at java.lang.reflect.Method.invoke(Method.java:497)
>at 
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>at 
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>at com.sun.proxy.$Proxy191.takeSnapshot(Unknown Source)
>at 
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591)
>at 
> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107)
>at 
> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>at java.lang.reflect.Method.invoke(Method.java:497)
>at 
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>at 
> com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>at 
> sun.reflect.Deleg

RE: Bug in Snapshot Retention?

[Sean Lair]

Hi all,

Any insight in why some snapshots could be stuck in "allocated" state?

Thanks
Sean
-Original Message-----
From: Sean Lair 
Sent: Wednesday, April 20, 2016 9:31 AM
To: users@cloudstack.apache.org
Subject: RE: Bug in Snapshot Retention?

Thanks for the responses all.  The "Removed Field" for the snapshots with the 
status of "BackedUp" is NULL.

I combed the logs and found the exception below.  It was successfully deleting 
snapshots before that log entry, then errored on the "Allocated" snapshot and 
stopped any further deletions.  I'm not sure what allocated mean, but will 
start researching.

2016-04-20 00:44:38,830 DEBUG [c.c.s.s.SnapshotManagerImpl] 
(Work-Job-Executor-1:ctx-954cbd99 job-7396/job-7401 ctx-d0261c6c) 
(logid:bafb5d42) post process snapshot failed
com.cloud.utils.exception.CloudRuntimeException: Failed to delete 
snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete 
snapshotshot 1351 due to it is in Allocated Status
at 
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:478)
at 
com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateRecurringSnapshotForPolicy(SnapshotManagerImpl.java:420)
at 
com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateSnapshot(SnapshotManagerImpl.java:399)
at 
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1010)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy191.takeSnapshot(Unknown Source)
at 
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591)
at 
com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107)
at 
com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at 
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi

RE: Bug in Snapshot Retention?

[Sean Lair]

ManagedContextRunnable.java:46)
at 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


-Original Message-
From: Anshul Gangwar [mailto:anshul.gang...@accelerite.com] 
Sent: Wednesday, April 20, 2016 12:05 AM
To: users@cloudstack.apache.org
Subject: Re: Bug in Snapshot Retention?

Are you getting any exception in logs after the completion of snapshot for this 
volume?

 If so try changing “job.cancel.threshold” global setting to appropriate 
timeout. This will make sure that job is not completed before the snapshot 
process is finished and will also make sure that cleanup process is not 
failing. This setting will affect all the jobs.

Regards,
Anshul



> On 20-Apr-2016, at 6:24 AM, Sean Lair <sl...@ippathways.com> wrote:
> 
> Hi all,
> 
> I'm running Cloudstack 4.8 on XenServer 6.5.   I have one volume that I'm 
> taking snapshots of, it is set to keep a total of 29 snapshots, but I have 
> close to 100 snapshots in the state of "BackedUp".  Am I misinterpreting the 
> scheduled snapshot screen or am I running into a bug?  Please see the output 
> below for more detail:
> 
> [cid:image004.jpg@01D19A75.2310A6C0]
> 
> MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and 
> status = 'backedup';
> +---+
> | count(status) |
> +---+
> |98 |
> +---+
> 1 row in set (0.00 sec)
> 
> Thanks
> Sean
> 




DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

Bug in Snapshot Retention?

[Sean Lair]

Hi all,

I'm running Cloudstack 4.8 on XenServer 6.5.   I have one volume that I'm 
taking snapshots of, it is set to keep a total of 29 snapshots, but I have 
close to 100 snapshots in the state of "BackedUp".  Am I misinterpreting the 
scheduled snapshot screen or am I running into a bug?  Please see the output 
below for more detail:

[cid:image004.jpg@01D19A75.2310A6C0]

MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and 
status = 'backedup';
+---+
| count(status) |
+---+
|98 |
+---+
1 row in set (0.00 sec)

Thanks
Sean

Re: Best way to update Templates

[Sean Lair]

Thanks for the reply Kirk.  Will it auto-delete templates if there are not any 
VMs cloned from them?  Is there a better way to keep templates up-to-date with 
patches?

We will be patching these templates fairly often as Microsoft releases new 
patches.  It sounds like our secondary storage will grow quite quick if it 
doesn't delete old templates, or if we have long-lived VMs that are linked to 
these old templates.

This does explain why our secondary storage has grown more than we expected 
lately.

Thanks for any help or guidance you can provide!

Sean
 

> On Apr 18, 2016, at 7:24 PM, Kirk Kosinski <kirk.kosin...@shapeblue.com> 
> wrote:
> 
> Hi, yes that is a good way to do it.  Note that if you do delete a template 
> that is in use, it won't actually be deleted on the back-end (i.e. the 
> database entries will remain, as will the files on secondary storage), rather 
> it is just hidden from the end-users in the UI and API.  Giving a new 
> template the same name as a deleted one might be a bit confusing so you could 
> consider adding a version number or date code.
> 
> Best regards,
> 
> Regards,
> 
> Kirk Kosinski
> 
> kirk.kosin...@shapeblue.com 
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HS
> @shapeblue
> 
> -Original Message-
> From: Sean Lair [mailto:sl...@ippathways.com] 
> Sent: Monday, April 18, 2016 3:09 PM
> To: users@cloudstack.apache.org
> Subject: Best way to update Templates
> 
> Hello,
> 
> We need to periodically update our templates, for example installing the 
> latest Windows updates in our Windows Server templates  Is there a way to 
> update existing templates or do we just need to delete the old template and 
> create new one with the same name?
> 
> The only issue I see with that so far is that we lose the linkage between 
> instaces (VMs) and templates.
> 
> Thanks
> Sean
> 

Best way to update Templates

[Sean Lair]

Hello,

We need to periodically update our templates, for example installing the latest 
Windows updates in our Windows Server templates  Is there a way to update 
existing templates or do we just need to delete the old template and create new 
one with the same name?

The only issue I see with that so far is that we lose the linkage between 
instaces (VMs) and templates.

Thanks
Sean

GRE with CloudStack 4.8

[Sean Lair]

Hi all,

I'm trying to configure GRE isolation for the Guest networks in CloudStack 4.8 
using XenServer 6.5.  I have the isolation method set to GRE "CloudGuest", but 
it is still creating VLAN interfaces in my XenServer Pod.

I tried following the link below as it looks like what I want, but it isn't 
working.

http://shankerbalan.net/blog/cloudstack-advanced-networking-with-gre-sdn-tunnels/

It says I need to set these global settings:
sdn.ovs.controller=  true
sdn.ovs.controller.default.label = XenServers physical guest interface label

However, I don't see sdn.ovs.controller as a global setting in CloudStack 4.8.  
Can anyone help point me in the right direction?

Thanks!
Sean

Guest VMs cannot access Internet

[Sean Lair]

Hi all,

I'm having an issue I'm hoping you can assist with.  Brand new Cloudstack 4.8 
deployment running on CentOS7 and KVM hypervisors.  Using advanced networking 
with VLAN isolation.

Deploying new VMs using the default CentOS5.5 instance works great.  The 
virtual router is deployed as expected to perform source NAT.  If I log into 
the virtual router, it can ping the Internet and the guest VMs.  The guest VMs 
can ping each other as they are on the same subnet.  The virtual router has an 
Internet public IP it is using for Source NAT.

The guest VMs however cannot access the Internet.  Under the public IP address 
[Source NAT] -> Firewall, I'm allowing 0.0.0.0/0 ICMP with "-1" for ICMP Type 
and code.  For the Egress rules for the guest network, I have 0.0.0.0/0 All 
protocols and All ports.  I can ping the outside of the virtual router (public 
IP) from the Internet.

>From my troubleshooting above I'm guessing it is something to do with the 
>virtual router, but am not sure how to troubleshoot next.

Thanks in advance for any assistance.

Thanks
Sean

RE: Guest VMs cannot access Internet

[Sean Lair]

Thanks for the response!  the iptables service is currently stopped:

# systemctl stop iptables
Failed to stop iptables.service: Unit iptables.service not loaded.

-Original Message-
From: Nux! [mailto:n...@li.nux.ro] 
Sent: Saturday, February 6, 2016 4:13 PM
To: users@cloudstack.apache.org
Subject: Re: Guest VMs cannot access Internet

Hi Sean,

Have you double checked iptables rules are correct (or disabled) on the 
underlying KVM hypervisor?

Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
> From: "Sean Lair" <sl...@ippathways.com>
> To: users@cloudstack.apache.org
> Sent: Saturday, 6 February, 2016 21:47:19
> Subject: Guest VMs cannot access Internet

> Hi all,
> 
> I'm having an issue I'm hoping you can assist with.  Brand new 
> Cloudstack 4.8 deployment running on CentOS7 and KVM hypervisors.  
> Using advanced networking with VLAN isolation.
> 
> Deploying new VMs using the default CentOS5.5 instance works great.  
> The virtual router is deployed as expected to perform source NAT.  If 
> I log into the virtual router, it can ping the Internet and the guest 
> VMs.  The guest VMs can ping each other as they are on the same 
> subnet.  The virtual router has an Internet public IP it is using for Source 
> NAT.
> 
> The guest VMs however cannot access the Internet.  Under the public IP 
> address [Source NAT] -> Firewall, I'm allowing 0.0.0.0/0 ICMP with 
> "-1" for ICMP Type and code.  For the Egress rules for the guest 
> network, I have 0.0.0.0/0 All protocols and All ports.  I can ping the 
> outside of the virtual router (public
> IP) from the Internet.
> 
> From my troubleshooting above I'm guessing it is something to do with 
> the virtual router, but am not sure how to troubleshoot next.
> 
> Thanks in advance for any assistance.
> 
> Thanks
> Sean

RE: Guest VMs cannot access Internet

[Sean Lair]

Here is the output:

-
[root@dc01cloudkvm01 ~]# systemctl status firewalld
â firewalld.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

-

[root@dc01cloudkvm01 ~]# iptables-save
# Generated by iptables-save v1.4.21 on Sat Feb  6 23:46:44 2016
*mangle
:PREROUTING ACCEPT [1306448:4376908074]
:INPUT ACCEPT [1185701:4364833786]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1294026:2863147676]
:POSTROUTING ACCEPT [1294026:2863147676]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Sat Feb  6 23:46:44 2016
# Generated by iptables-save v1.4.21 on Sat Feb  6 23:46:44 2016
*nat
:PREROUTING ACCEPT [120793:12078892]
:INPUT ACCEPT [46:4604]
:OUTPUT ACCEPT [1446:103514]
:POSTROUTING ACCEPT [1446:103514]
-A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE 
--to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE 
--to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
COMMIT
# Completed on Sat Feb  6 23:46:44 2016
# Generated by iptables-save v1.4.21 on Sat Feb  6 23:46:44 2016
*filter
:INPUT ACCEPT [1185701:4364833786]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1294026:2863147676]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate 
RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
COMMIT
# Completed on Sat Feb  6 23:46:44 2016
---

-Original Message-
From: Nux! [mailto:n...@li.nux.ro] 
Sent: Saturday, February 6, 2016 5:38 PM
To: users@cloudstack.apache.org
Subject: Re: Guest VMs cannot access Internet

That's not you check it, CentOS 7 now comes with firewalld and the 
iptables-services are not installed by defaut.
"iptables-save" will output the current state of the firewall

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
> From: "Sean Lair" <sl...@ippathways.com>
> To: users@cloudstack.apache.org
> Sent: Saturday, 6 February, 2016 22:56:23
> Subject: RE: Guest VMs cannot access Internet

> Thanks for the response!  the iptables service is currently stopped:
> 
> # systemctl stop iptables
> Failed to stop iptables.service: Unit iptables.service not loaded.
> 
> -Original Message-
> From: Nux! [mailto:n...@li.nux.ro]
> Sent: Saturday, February 6, 2016 4:13 PM
> To: users@cloudstack.apache.org
> Subject: Re: Guest VMs cannot access Internet
> 
> Hi Sean,
> 
> Have you double checked iptables rules are correct (or disabled) on 
> the underlying KVM hypervisor?
> 
> Lucian
> 
> --
> Sent from the Delta quadrant using Borg technology!
> 
> Nux!
> www.nux.ro
> 
> - Original Message -
>> From: "Sean Lair" <sl...@ippathways.com>
>> To: users@cloudstack.apache.org
>> Sent: Saturday, 6 February, 2016 21:47:19
>> Subject: Guest VMs cannot access Internet
> 
>> Hi all,
>> 
>> I'm having an issue I'm hoping you can assist with.  Brand new 
>> Cloudstack 4.8 deployment running on CentOS7 and KVM hypervisors.
>> Using advanced networking with VLAN isolation.
>> 
>> Deploying new VMs using the default CentOS5.5 instance works great.
>> The virtual router is deployed as expected to perform source NAT.  If 
>> I log into the virtual router, it can ping the Internet and the guest 
>> VMs.  The guest VMs can ping each other as they are on the same 
>> subnet.  The virtual router has an Internet public IP it is using for 
>> Source NAT.
>> 
>> The guest VMs however cannot access the Internet.  Under the public 
>> IP address [Source NAT] -> Firewall, I'm allowing 0.0.0.0/0 ICMP with 
>> "-1" for ICMP Type and code.  For the Egress rules for the guest 
>> network, I have 0.0.0.0/0 All protocols and All ports.  I can ping 
>> the outside of the virtual router (public
>> IP) from the Internet.
>> 
>> From my troubleshooting above I'm guessing it is something to do with 
>> the virtual router, but am not sure how to troubleshoot next.
>> 
>> Thanks in advance for any assistance.
>> 
>> Thanks
> > Sean