Re: IPv6 Support

[Eric Lee Green]

On 11/11/2020 2:01 AM, Hean Seng wrote:

IPv6 do not have NAT , each VM suppose to have indiviual Ipv6 Address.


NAT66 does in fact exist, and the virtual routers used for VLANs could 
in fact be configured with RADV to provide an IETF RFC4193 SLAAC prefix 
to private VPC networks then use NAT66 to communicate with the rest of 
the IPv6 Internet via a SLAAC-configured IPv6 address on the virtual 
router's public interface. They are not currently so configured, but all 
the stuff to do it is already there in the base Debian distribution used 
for the virtual routers.


Port forwarding would require changes to the virtual router to allow 
IPv6 port forwarding (as well as likely allowing a fixed IPv6 address 
for the virtual router rather than SLAAC).


DHCPv6 to advertise IPv6 DNS servers would be the other part of that 
equation.


Routing public subnets would require significant work, since the virtual 
routers would need to advertise routes upstream to whatever layer 3 
switch or router routes things to and from the Internet. In addition 
security would require disabling incoming IPv6 connections to the 
advertised subnet except to specific instances that have a hole poked in 
the firewall allowing incoming IPv6. It is unlikely that anybody is 
going to bother implementing this anytime soon, since NAT66 works fine 
for Cloudstack's purposes and is significantly easier to implement since 
it doesn't require upstream routers to accept route advertisements from 
virtual routers.




For NAT zone,  is that any way to allocate IPv6 subnet ?







On Tue, Nov 10, 2020 at 3:51 PM Andrija Panic 
wrote:


If not mistaken, ipv6 is only supported for Shared Networks, and not for
Isolated/VPC networks.

On Tue, 3 Nov 2020 at 04:31, Hean Seng  wrote:


Hi

Is that anyone have a idea of best way implementing ipv6 in cloudstack ?

I saw the doc, and mentioned create another SharedGuestNework in
AdvanceZone, and assigned ipv6 /64 network there.

However, I not quite understand is in Advancezone with NAT (public ip,
isolated vlan), the network of  the VM is  their own LAN IP and isolated

by

VLAN or VXLAN.   How can we assign Ipv6 over there? Or shall we

create

another SharedGuestNetwork with another VLAN , and assign another
GuestNetwork manually to the VM ?  But then, the VM become 2 network.  Is
that the way to do ?


--
Regards,
Hean Seng



--

Andrija Panić

Re: Compute Offering - Network Rate

[Eric Lee Green]

On 8/13/2020 10:22 AM, Hean Seng wrote:

Hi

Cloudstack 4.14 ,  Advance Network with GiuestNetwork have Public IP

Create a compute offering with 10Mbps, but vm created stil able to burst to
200Mps

Any one know how to solve this


The virtualization hypervisor is responsible for enforcing network 
limits. Which virtualization hypervisor are you using? Xen? KVM? VMware?


For KVM hosts, libvirt has the capability to throttle bandwidth. It has 
three knobs for inbound and outbound bandwidth -- average bandwidth, 
peak bandwidth, and burst number of bytes. To see the XML being 
generated for your virtual machine, turn on debugging on the agent on 
one of your hosts  in /etc/cloudstack/agent/log4j-cloud.xml (change all 
INFO to DEBUG then restart the agent), then start the instance on that 
specific host. Then look for the XML in the log file in 
/var/log/cloudstack/agent/agent.log which should have a segment in it 
like this for the network interface: [note -- all file locations are for 
CentOS 7]



...
  *  
  ... *


For information on what that means:

https://libvirt.org/formatnetwork.html#elementQoS

Hmm, yeah, just checked the source.

if ((s_libvirtVersion >= 9004) && (_networkRateKBps > 0)) {// supported from libvirt 0.9.4 
netBuilder.append("\n");
netBuilder.append("\n");
netBuilder.append("\n");
netBuilder.append("\n");
}

According to the source code, average and peak for both inbound and 
outbound are set to the max bandwidth in the offering, while 'burst' is 
not defined. Note the limitation documented in the above libvirt 
document: the 'peak' is ignored in the outbound setting, because of 
Linux OS network layer limitations. Cloudstack properly produces the 
tag, but the KVM hypervisor ignores it. If the hypervisor ever actually 
respects the limit Cloudstack is sending it the right data, but 
Cloudstack can't force the hypervisor to do something it can't currently do.


In other words, the behavior you're seeing appears to be a result of 
limitations of KVM on Linux if that's the hypervisor you're using. 
Apparently only the 'average' is respected for outbound data traffic, 
meaning you'll be able to burst on outbound traffic and it'll be 
throttled only once the average exceeds the limit.

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

[Eric Lee Green]

Correct, 4.11.3 template is used for 4.11.3, 4.12, and 4.13. 4.14 moves 
to the 4.14.0 template.


There seems to be something odd happening key-wise sometimes with 
upgrades from 4.11.3 to 4.13.1 or 4.14.0.   I managed an upgrade from 
4.11.3 to 4.13.1 that *almost* worked, but the secondary storage VM 
wouldn't work and thus I couldn't spawn new virtual machines. Same 
symptom -- key error when the agent tried to ssh into it. And deleting 
it and making it respawn didn't help. Then I tried 4.11.3 to 4.14.0 and 
*all* the VM's failed at that point (of course, that was with the new 
template).


Right now I'm back at 4.11.3 until this can be figured out.

On 8/11/2020 5:53 AM, Ammad Syed wrote:

Hi,

I think 4.12 and 4.13 uses same systemVM template i.e 4.11.3 version, which
I already have registered. Currently I am running 4.11.3 version of ACS.

MariaDB [cloud]> SELECT id,name,type,cross_zones,state FROM
cloud.vm_template WHERE name like '%systemvm-xenserver%' AND removed IS
NULL;
+--+-++-+--+
| id   | name| type   | cross_zones | state|
+--+-++-+--+
|  337 | systemvm-xenserver-3.0.0| SYSTEM |   0 | Inactive |
|  418 | systemvm-xenserver-4.2  | SYSTEM |   0 | Active   |
|  472 | systemvm-xenserver-4.3  | USER   |   1 | Inactive |
|  473 | systemvm-xenserver-4.3  | USER   |   1 | Inactive |
|  474 | systemvm-xenserver-4.3  | USER   |   1 | Inactive |
|  475 | systemvm-xenserver-4.3  | USER   |   1 | Inactive |
|  476 | systemvm-xenserver-4.3  | USER   |   0 | Inactive |
|  479 | systemvm-xenserver-4.3-2| USER   |   1 | Inactive |
|  480 | systemvm-xenserver-4.3  | SYSTEM |   0 | Active   |
|  549 | systemvm-xenserver-4.5.1| USER   |   0 | Active   |
|  550 | systemvm-xenserver-4.5.1| SYSTEM |   0 | Active   |
|  651 | systemvm-xenserver-4.7.0| USER   |   0 | Inactive |
|  652 | systemvm-xenserver-4.7.0| USER   |   0 | Inactive |
|  653 | systemvm-xenserver-4.7.0| SYSTEM |   0 | Inactive |
|  737 | systemvm-xenserver-4.9.2| SYSTEM |   1 | Inactive |
|  739 | systemvm-xenserver-4.9.2-sb | SYSTEM |   1 | Active   |
| 1245 | systemvm-xenserver-4.11.1   | SYSTEM |   1 | Active   |
| 1584 | systemvm-xenserver-4.11.2   | SYSTEM |   1 | Active   |
| 1677 | systemvm-xenserver-4.11.3   | SYSTEM |   1 | Active   |
+--+-++-+--+

- Ammad

On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion  wrote:
db.

Hi Syed,
 From 4.12, the systemvm template had to be upgraded because of OS change in
the template, moved to a latest version of Debian. Because of that, some VR
scripts have changed and make obsolete older version of VRs, so you will
most likely have to register an updated systemvm templates and upgrade your
system VMs and VRs.

Regards,

On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed  wrote:


Hi Guys,

I was previously on 4.9.3 cloudstack and upgraded to 4.11.1 then 4.11.3.
The version 4.11.3 is working fine since six months.

Now I have tried to upgrade my system from 4.11.3 to 4.13.1. The upgrade
goes successful. I didn't uploaded any system VM template. However the
problem occured when I recreated my systemVM of POD, the VM recreated and
its state was running but agent state was not getting up, its showing

blank

in column.

Digging further via job logs, the job is failed with error that unable to
execute command via ssh. Below are the logs.

2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper]
(DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH execution of command
/opt/cloud/bin/router_proxy.sh keystore-s
etup 169.254.2.199 /usr/local/cloud/systemvm/conf/agent.properties
/usr/local/cloud/systemvm/conf/cloud.jks TJaQYChYBwKh7Cx9 365
/usr/local/cloud/systemvm/conf/clou
d.csr has an error status code in return. Result output:
2020-07-25 02:30:48,127 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
906-3195585410596077730: Response Received:
2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
(DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
906-3195585410596077730: Processing:  { Ans: , MgmtId: 779271079
43497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10,
[{"org.apache.cloudstack.ca
.SetupKeystoreAnswer":{"result":false,"wait":0}}]
}
2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258 ctx-df740f75)
(logid:9fa7dece) Seq 906-319558541059607773
0: Received:  { Ans: , MgmtId: 77927107943497, via:
906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, { SetupKeystoreAnswer } }
2020-07-25 02:30:48,127 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258 ctx-df740f75)
(logid:9fa7dece) 

Re: Windows 10 KVM becoming inaccessible

[Eric Lee Green]

I will note that we have several Windows 10 / Windows 2018 genre VM's on 
Centos 7.8.2003 and are not seeing any guest freezes. But:


1) We are not using CPU host-passthrough.

2) We do have the QEMU Windows guest drivers installed in the VM's.

3) I am running Cloudstack 4.11.3. I tried upgrading to Cloudstack 
4.14.0 and the upgrade failed, the agent was unable to ssh into virtual 
routers to configure them.


That said, the version of Cloudstack should not matter, since all 
Cloudstack does is tell libvirtd to launch the VM. libvirtd handles 
everything from there. We are literally using the stock Centos 7.8 
libvirtd.conf as modified by the agent configurator.  I wonder if you 
are having a hardware problem on your new server, or if there is a Linux 
OS problem handling your new server. Use journalctl and see if you're 
seeing anything at the Linux OS level when a freeze happens, and check 
the qemu log files for the instance also, because normally libvirtd is 
quite reliable and Centos 7.8.2003 supports Windows 10 VM's just fine.


On 8/9/2020 8:48 AM, Rohit Yadav wrote:

Hi Dave,

I've not specifically worked with Windows guest VMs on KVM, but what you've 
described is largely subject to a guest OS support by the hypervisor, in this 
case the specific libvirt/qemu/linux-kernel version and any guest tools/drivers 
that must be installed inside the Windows guest VM. You've yourself 
acknowledged that the issue is not seen in your older CentOS6 based environment.

To rule out CloudStack (a) you may add or upgrade hosts in a cluster to use 
qemu-ev (enterprise qemu release by CentOS SIG 
https://wiki.centos.org/SpecialInterestGroup/Virtualization, i.e. install the 
centos-release-qemu-ev pkg) or (b) you may add a new cluster with Ubuntu 18.04 
KVM hosts and recreate your Windows VM setup. Or, it could be a specific 
Windows build or requires additional drivers (such as 
https://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers).


Hope this helps.


Regards.


From: Dave Lapointe 
Sent: Saturday, August 8, 2020 00:48
To: users@cloudstack.apache.org 
Subject: Windows 10 KVM becoming inaccessible

Hi,

I've been lurking on this group for a few years now and have found the 
information that people have posted here to be quite helpful for me to 
understand CloudStack better.  I've never replied here because there are always 
far more knowledgeable people on this list who can offer much better insight 
than I ever could.

An issue has arisen recently that I can't find any solution for.  I apologize 
ahead of time if this is the wrong list to post to.

I recently configured a new server to run CloudStack using Centos 7.8.2003 and CloudStack 
4.14, and configured some Windows 10 LTSB KVM guests.  This is a fairly specialized 
server, so the configuration is a little unusual.  It's configured to use the 
"cloudbr0" software bridge for the guest network which is then routed 
externally through a single NIC.  Also, because the VM's will never be migrated, I've set 
guest.cpu.mode=host-passthrough.

What's been happening though is that the VM's will just freeze sometimes, 
apparently randomly.  Sometimes it will happen during boot, or a couple minutes 
after connecting by RDP.  And sometimes the VM won't freeze at all.  I haven't 
been able to determine a pattern as to when this will happen.  And I haven't 
found anything in the logs that might help me understand what's happening 
(/var/log/messages and /var/log/cloudstack/management/management-server.log).  
I've checked on the QEMU and Linux forums, but have only found a bit of 
information about VM's freezing for people using specific graphics drivers with 
passthrough for their graphics cards.  I tried removing 
guest.cpu.mode=host-passthrough but that made no difference.

What's especially odd to me is that this didn't happen with older systems I've 
created (eg. CentOS 6 and CloudStack 4.9).  I've setup half a dozen or so using 
the same configuration as this system, just older software.

I can't tell if this is related to CloudStack (maybe there is something in the 
guest parameters that is causing this), or if this is strictly a KVM issue.  
And since I can't find anything in the logs I don't know where else to look.  
I'm hoping to get some suggestions from this list so that I can do some more 
digging.

Thanks,
Dave

rohit.ya...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
   
  

Re: Has anybody gotten Cloudstack 4.13.1.0 or 4.14.0.0 working on Centos 7.7 or 7.8?

[Eric Lee Green]

After turning on debugs, I found that it was detecting that the routing 
VM was started, and it was pinging the routing VM, but failing to ssh 
into the routing VM to get its information and configure it. My guess is 
that for some reason the template has the wrong ssh key and it is a key 
issue between what I was updating from ( 4.11.3) and 4.14.0. I 
downgraded back to 4.11.3 and all is working fine, but of course that 
leaves me on 4.11.3 (still). But at least 4.11.3 is working fine with 
Centos 7.8, so I have verified that the problem isn't Centos 7.8.


I think I'm going to wait a bit before attempting to upgrade to 4.14.x 
again. Maybe wait for 4.14.1 or 4.14.2. Upgrading to a .0 always seems 
to be problematic, even when the .0 would normally be a .2 if not for 
the transition from Java 8 to Java 11. I was doing the upgrade primarily 
for Primate, which doesn't run well with 4.11.3 and which I hoped would 
make my users quit complaining about the primitive Cloudstack UI, but 
(shrug). So it goes. They'll just have to suffer until the end of the 
year.  I have very spoiled users lol, they don't know just how hard it 
is to get new VMs commissioned in most companies (self service? ROFL!). 
Despite its quirks, Cloudstack has been a huge boon to us.


On 8/9/2020 8:42 AM, Rohit Yadav wrote:

Hi Eric,

This looks like a Libvirt specific error [1] outside of the scope of 
CloudStack. Can you try the CentOS SIG Virtualisation repos (the qemu-ev 
releases) [2] which usually requires installing centos-release-qemu-ev on the 
CentOS KVM hosts and doing an upgrade (or installation of the qemu* packages).

[1] Similar issue: https://bugzilla.redhat.com/show_bug.cgi?id=1090551

[2] 
https://wiki.centos.org/SpecialInterestGroup/Virtualizatio<https://wiki.centos.org/SpecialInterestGroup/Virtualization>



Regards.


From: Eric Lee Green 
Sent: Saturday, August 8, 2020 08:44
To: users@cloudstack.apache.org 
Subject: Has anybody gotten Cloudstack 4.13.1.0 or 4.14.0.0 working on Centos 
7.7 or 7.8?

I've tried multiple times now and it's refusing to start the virtual
routers and thus refusing to start anything else. Or rather, it's
starting them, I see the VM appear in the process list (ps -ax | grep
kvm) and I see log messages in /var/log/messages saying it's started and
I see the log,  but then the agent is refusing to connect and then
shutting them down. This happened with both 4.13 and 4.14.  4.13 worked
with an earlier version of Centos, but had issues with creating new
virtual machines (as versus starting them) so I am trying to get things
running on a modern Centos (i.e., one that's not a security breach just
by existing).

Any ideas?

Example:

/var/log/messages:

-snip--
Aug  7 19:57:49 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:) (logid:35b3a97e) Trying to fetch storage pool
f7c824aa-e6e4-3c40-b890-36a459245385 from libvirt
Aug  7 19:57:49 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:) (logid:35b3a97e) Asking libvirt to refresh
storage pool f7c824aa-e6e4-3c40-b890-36a459245385
Aug  7 19:57:50 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:) (logid:35b3a97e) Trying to fetch storage pool
0ab13de9-2310-334c-b438-94dfb0b8ec84 from libvirt
Aug  7 19:57:50 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:) (logid:35b3a97e) Asking libvirt to refresh
storage pool 0ab13de9-2310-334c-b438-94dfb0b8ec84
Aug  7 19:58:29 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:) (logid:1051f59c) Trying to fetch storage pool
0ab13de9-2310-334c-b438-94dfb0b8ec84 from libvirt
Aug  7 19:58:29 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:) (logid:1051f59c) Trying to fetch storage pool
0ab13de9-2310-334c-b438-94dfb0b8ec84 from libvirt
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 1(bond0.102) entered
blocking state
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 1(bond0.102) entered
disabled state
Aug  7 19:58:29 cloud1 kernel: device bond0.102 entered promiscuous mode
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 1(bond0.102) entered
blocking state
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 1(bond0.102) entered
forwarding state
Aug  7 19:58:29 cloud1 java: WARNING: An illegal reflective access
operation has occurred
Aug  7 19:58:29 cloud1 java: WARNING: Illegal reflective access by
org.codehaus.groovy.reflection.CachedClass
(file:/usr/share/cloudstack-agent/lib/groovy-all-2.4.17.jar) to method
java.lang.Object.finalize()
Aug  7 19:58:29 cloud1 java: WARNING: Please consider reporting this to
the maintainers of org.codehaus.groovy.reflection.CachedClass
Aug  7 19:58:29 cloud1 java: WARNING: Use --illegal-access=warn to
enable warnings of further illegal reflective access operations
Aug  7 19:58:29 cloud

Has anybody gotten Cloudstack 4.13.1.0 or 4.14.0.0 working on Centos 7.7 or 7.8?

[Eric Lee Green]

I've tried multiple times now and it's refusing to start the virtual 
routers and thus refusing to start anything else. Or rather, it's 
starting them, I see the VM appear in the process list (ps -ax | grep 
kvm) and I see log messages in /var/log/messages saying it's started and 
I see the log,  but then the agent is refusing to connect and then 
shutting them down. This happened with both 4.13 and 4.14.  4.13 worked 
with an earlier version of Centos, but had issues with creating new 
virtual machines (as versus starting them) so I am trying to get things 
running on a modern Centos (i.e., one that's not a security breach just 
by existing).


Any ideas?

Example:

/var/log/messages:

-snip--
Aug  7 19:57:49 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-1:) (logid:35b3a97e) Trying to fetch storage pool 
f7c824aa-e6e4-3c40-b890-36a459245385 from libvirt
Aug  7 19:57:49 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-1:) (logid:35b3a97e) Asking libvirt to refresh 
storage pool f7c824aa-e6e4-3c40-b890-36a459245385
Aug  7 19:57:50 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:) (logid:35b3a97e) Trying to fetch storage pool 
0ab13de9-2310-334c-b438-94dfb0b8ec84 from libvirt
Aug  7 19:57:50 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:) (logid:35b3a97e) Asking libvirt to refresh 
storage pool 0ab13de9-2310-334c-b438-94dfb0b8ec84
Aug  7 19:58:29 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-4:) (logid:1051f59c) Trying to fetch storage pool 
0ab13de9-2310-334c-b438-94dfb0b8ec84 from libvirt
Aug  7 19:58:29 cloud1 java: INFO [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-4:) (logid:1051f59c) Trying to fetch storage pool 
0ab13de9-2310-334c-b438-94dfb0b8ec84 from libvirt
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 1(bond0.102) entered 
blocking state
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 1(bond0.102) entered 
disabled state

Aug  7 19:58:29 cloud1 kernel: device bond0.102 entered promiscuous mode
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 1(bond0.102) entered 
blocking state
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 1(bond0.102) entered 
forwarding state
Aug  7 19:58:29 cloud1 java: WARNING: An illegal reflective access 
operation has occurred
Aug  7 19:58:29 cloud1 java: WARNING: Illegal reflective access by 
org.codehaus.groovy.reflection.CachedClass 
(file:/usr/share/cloudstack-agent/lib/groovy-all-2.4.17.jar) to method 
java.lang.Object.finalize()
Aug  7 19:58:29 cloud1 java: WARNING: Please consider reporting this to 
the maintainers of org.codehaus.groovy.reflection.CachedClass
Aug  7 19:58:29 cloud1 java: WARNING: Use --illegal-access=warn to 
enable warnings of further illegal reflective access operations
Aug  7 19:58:29 cloud1 java: WARNING: All illegal access operations will 
be denied in a future release
Aug  7 19:58:29 cloud1 java: WARN [kvm.resource.LibvirtKvmAgentHook] 
(agentRequest-Handler-4:) (logid:1051f59c) Groovy script 
'/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not 
available. Transformations will not be applied.
Aug  7 19:58:29 cloud1 java: WARN [kvm.resource.LibvirtKvmAgentHook] 
(agentRequest-Handler-4:) (logid:1051f59c) Groovy scripting engine is 
not initialized. Data transformation skipped.
Aug  7 19:58:29 cloud1 java: libvirt: QEMU Driver error : Domain not 
found: no domain with matching name 'r-739-VM'
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 2(vnet0) entered 
blocking state
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 2(vnet0) entered 
disabled state

Aug  7 19:58:29 cloud1 kernel: device vnet0 entered promiscuous mode
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 2(vnet0) entered 
blocking state
Aug  7 19:58:29 cloud1 kernel: brbond0-102: port 2(vnet0) entered 
forwarding state

Aug  7 19:58:29 cloud1 kernel: cloud0: port 1(vnet1) entered blocking state
Aug  7 19:58:29 cloud1 kernel: cloud0: port 1(vnet1) entered disabled state
Aug  7 19:58:29 cloud1 kernel: device vnet1 entered promiscuous mode
Aug  7 19:58:29 cloud1 kernel: cloud0: port 1(vnet1) entered blocking state
Aug  7 19:58:29 cloud1 kernel: cloud0: port 1(vnet1) entered forwarding 
state
Aug  7 19:58:29 cloud1 libvirtd: 2020-08-08 02:58:29.657+: 8966: 
warning : qemuDomainObjTaint:7530 : Domain id=1 name='r-739-VM' 
uuid=7f5bcd06-3c91-4b6c-a67f-f031a18db38a is tainted: high-privileges
Aug  7 19:58:29 cloud1 dbus[1143]: [system] Activating via systemd: 
service name='org.freedesktop.machine1' 
unit='dbus-org.freedesktop.machine1.service'
Aug  7 19:58:29 cloud1 systemd: Starting Virtual Machine and Container 
Registration Service...
Aug  7 19:58:29 cloud1 dbus[1143]: [system] Successfully activated 
service 'org.freedesktop.machine1'
Aug  7 19:58:29 cloud1 systemd: Started Virtual Machine 

Re: Can't ping from VR to a vm on different host on anIsolated guest network

[Eric Lee Green]

On 10/10/19 5:19 AM, Ioan Marginean wrote:

Hi users,

I installed CS 4.13 on 3 KVM hypervisors. Every host has 2 interfaces, eno1 end 
eno2. I had defined cloudbr0 on eno1 end here goes Management, Public and 
storage traffic. The guest traffic goes on eno2. All seems perfect until I 
started to create instances on a guest isolated network. If the instance VM is 
on the same host as the VR, all is fine and dandy but if the VM instance is 
created on a different host than VR's the outcome for ping is Destination Host 
Unreachable.
The network service on that instance appears as failed and ifconfig shows eth0 
with no ipv4 address as it should...
I suspect that I didn't configure something wrong but I can't figure out what 
is wrong... Googleing and searching on user mailing list didn't helped  Can 
anyone point me to the right direction?



Isolated guest networks by default operate as a VLAN. Your switch will 
need to pass tagged VLAN traffic for those ports. If your switch is not 
configured properly then VLAN traffic won't pass and you won't be able 
to have isolated networks that extend outside of a single host.


I used advanced networking and VLANs. Here is how my 
/etc/sysconfig/network-scripts looks:


ifcfg-bond0   ifcfg-bond0.102   ifcfg-enp1s0f1
ifcfg-bond0.100  ifcfg-br100  ifcfg-enp5s0
ifcfg-bond0.101   ifcfg-enp1s0f0

'bond0' is my bonded network ports, defined as the Physical Network in 
my zone, which actually just happens to have a single port right now on 
this particular host, enp5s0 (it can be different ports on different 
hosts, my hosts aren't completely homogenous, so keeping it as bond0 
means it doesn't matter if it's enp4s0 on some other host). 100 is my 
management network, the one used to get everything else up and running, 
and is the only one that has a bridge predefined for it, br100. This is 
tagged in the KVM traffic label of the details in the management 
network. (Which you get to by clicking the 'bond0' in the network). I 
then defined 101 as 'public', 102 as 'Guest', and for storage it's 100 
again with KVM traffic label br100.


In my Dell managed switch, for the 10 gigabit Ethernet ports connected 
to the compute and storage hosts, they look like this:


interface ethernet 1/xg2
spanning-tree portfast
mtu 9216
switchport mode general
switchport general pvid 100
switchport general acceptable-frame-type tagged-only
switchport general allowed vlan add 1000-2000
switchport general allowed vlan add 100-104,120,192,200 tagged
exit

And of course for the ports that lead off to other switches in untagged 
mode, like this port that hooks to my main infrastructure switch from 
whence it is routed to the Internet, they look like this:


interface ethernet 1/g4
spanning-tree portfast
switchport access vlan 101
exit

Note that by default all managed switches have all ports defined as 
access ports for VLAN 1 and will not accept tagged traffic for any other 
VLAN. Now, you can define a port as *BOTH* an access port (i.e., 
untagged packets get tagged to a VLAN, outgoing traffic on that VLAN 
gets untagged and sent out as plain packets) *and* as a tagged port for 
other VLANs in most switches, but it's a hinky way of doing things and 
subject to many issues. In general, make a port either an access port 
(i.e., untagged, on a single VLAN) or a tagged port (multiple VLANs) or 
you are asking for a world of trouble.

Re: DNS quit working when I updated to 4.11.2

[Eric Lee Green]

On 5/24/19 12:21 PM, Andrija Panic wrote:

In other words - you are hitting an internal interface of a VR?


The VR has two NIC's. I presume that the Guest NIC as vs the Control NIC 
is the "internal" NIC?


TypeShared
Traffic TypeGuest
Network NameShared
Netmask 255.255.0.0
IP Address  10.102.199.148
ID  7f59d904-cdc0-43eb-b679-0721077f5bb1
Network ID  924eda5f-9a1f-4a8e-9423-18000dc92093
Isolation URI   vlan://102
Broadcast URI   vlan://102


Type
Traffic TypeControl
Network Name
Netmask 255.255.0.0
IP Address  169.254.2.203
ID  9c3676bc-23e6-48e3-baca-b8cce6511092
Network ID  6eff5bd9-4f4d-48fe-b6ed-f50fc115947b
Isolation URI   
Broadcast URI   



I would replace (for a test) bind9 with just the default setup of 
DNSmasq, while specifying it's uper/ROOT DNS servers to be the VR IP - 
i.e. client --> DNSmasq (internal server) --> DNSmasq (VR).

See if that work - so you can draw possibly some conclusions.


That gives me room for some more experiments. I am fairly sure that I am 
running into recent changes to bind9 / dnsmasq intended to prevent DNS 
amplification and spoofing attacks, but the question of which one 
changed and how to work around it is still a question I'm trying to answer.





Andrija

On Fri, 24 May 2019 at 21:12, Eric Lee Green <mailto:eric.lee.gr...@gmail.com>> wrote:


On 5/24/19 10:16 AM, Andrija Panic wrote:
> Eric,
>
> your BIND9 servers is on a "Public" network (trying to talk to
the Public
> IP of the VR during forwarding DNS requests) or a VM inside an
Isolated
> network behind VR)?

It's on *a* public network, but not *the* public network. I don't
have
any Isolated networks, though I have them enabled from VLAN
1000-2000. I
am using "Advanced Networking" but for my own purposes -- I have one
"Shared" guest network at VLAN 102, and then several isolated
specialty
physical "Shared" networks like "Security Cameras" (VLAN 103) and
"Storage Network" (VLAN 200) that are attached to virtual machines
that
need access to those things. The "Shared" guest network (VLAN 102) is
routed by my layer 3 switch with the rest of my network's public
VLANs
so if I am on e.g. 10.31.1.2 (VLAN 31), which is similarly a routed
public VLAN (but not one that Cloudstack is allowed to directly
talk to
or manage, it has to go thru the layer 3 switch) or 10.120.0.5 (VLAN
120), I can talk directly to 10.102.199.148 since all are routed into
the common fabric via the layer 3 switch.  I only care about the VM's
that are VLAN 102, which are supposed to be publicly available to my
users, thus why my quicky script hack to generate a zone file out
of the
database does

select v.name <http://v.name>, n.ip4_address from vm_instance as
v, nics as n where v.removed is null and n.instance_id = v.id
<http://v.id> and n.ip4_address like '10.102.%'  and type =
'User'  order by n.ip4_address;

in order to select out the name and IP address of virtual machines
with
NIC's on that VLAN. (Which, if it's a different list from the last
list
that was queried, then gets massaged into a zone file for
name.cloud.mydomain.com <http://name.cloud.mydomain.com> by the
script, which then scp's to my master
domain server and does a reload to reload the zone file from the new
version).

Both of my BIND9 servers can talk directly to 10.102.199.148 (the
IP of
the virtual router for the 10.102.xxx.xxx network, VLAN 102) if I use
'host' to directly query 10.102.199.148 for an API address like, say,
'api-default1.cloud.mydomain.com
<http://api-default1.cloud.mydomain.com>' but when I try to put a
forward domain
there, nope. This was working, but now is not. I suspect it's got
to do
with the recent changes in DNS software, both bind9 and dnsmasq,  to
deal with multiple attacks on the domain name system, but I'm having
trouble figuring out why, or what my solution should be.

Note that it's quite reasonable / feasible / viable to put a DNS
server
actually inside the Cloudstack constellation if that's necessary and
then do a two-stage hop if necessary. I'm just trying to figure
out the
"right" way to do this right now so I can retire my hack script.

> On Fri, 24 May 2019 at 02:15, Eric Lee Green
mailto:eric.lee.gr...@gmail.com>>
> wrote:
>
>> I had this working under 4.9. All I did was, on my main BIND9
servers,
>> point a forward zone at 'cloud..com' to the virtual
router
>> associated with all VM's that were publicly available. I could then
>> resolve all foo.cloud..com names on my global network.
>

Re: DNS quit working when I updated to 4.11.2

[Eric Lee Green]

On 5/24/19 10:16 AM, Andrija Panic wrote:

Eric,

your BIND9 servers is on a "Public" network (trying to talk to the Public
IP of the VR during forwarding DNS requests) or a VM inside an Isolated
network behind VR)?


It's on *a* public network, but not *the* public network. I don't have 
any Isolated networks, though I have them enabled from VLAN 1000-2000. I 
am using "Advanced Networking" but for my own purposes -- I have one 
"Shared" guest network at VLAN 102, and then several isolated specialty 
physical "Shared" networks like "Security Cameras" (VLAN 103) and 
"Storage Network" (VLAN 200) that are attached to virtual machines that 
need access to those things. The "Shared" guest network (VLAN 102) is 
routed by my layer 3 switch with the rest of my network's public VLANs 
so if I am on e.g. 10.31.1.2 (VLAN 31), which is similarly a routed 
public VLAN (but not one that Cloudstack is allowed to directly talk to 
or manage, it has to go thru the layer 3 switch) or 10.120.0.5 (VLAN 
120), I can talk directly to 10.102.199.148 since all are routed into 
the common fabric via the layer 3 switch.  I only care about the VM's 
that are VLAN 102, which are supposed to be publicly available to my 
users, thus why my quicky script hack to generate a zone file out of the 
database does


select v.name, n.ip4_address from vm_instance as v, nics as n where v.removed 
is null and n.instance_id = v.id and n.ip4_address like '10.102.%'  and type = 
'User'  order by n.ip4_address;

in order to select out the name and IP address of virtual machines with 
NIC's on that VLAN. (Which, if it's a different list from the last list 
that was queried, then gets massaged into a zone file for 
name.cloud.mydomain.com by the script, which then scp's to my master 
domain server and does a reload to reload the zone file from the new 
version).


Both of my BIND9 servers can talk directly to 10.102.199.148 (the IP of 
the virtual router for the 10.102.xxx.xxx network, VLAN 102) if I use 
'host' to directly query 10.102.199.148 for an API address like, say, 
'api-default1.cloud.mydomain.com' but when I try to put a forward domain 
there, nope. This was working, but now is not. I suspect it's got to do 
with the recent changes in DNS software, both bind9 and dnsmasq,  to 
deal with multiple attacks on the domain name system, but I'm having 
trouble figuring out why, or what my solution should be.


Note that it's quite reasonable / feasible / viable to put a DNS server 
actually inside the Cloudstack constellation if that's necessary and 
then do a two-stage hop if necessary. I'm just trying to figure out the 
"right" way to do this right now so I can retire my hack script.



On Fri, 24 May 2019 at 02:15, Eric Lee Green 
wrote:


I had this working under 4.9. All I did was, on my main BIND9 servers,
point a forward zone at 'cloud..com' to the virtual router
associated with all VM's that were publicly available. I could then
resolve all foo.cloud..com names on my global network.

Somehow, though, this quit working after I updated to 4.11. I'm not
quite sure why.

The 'Guest Network' is defined with domain 'cloud.mydomain.com'.

Okay, so my router for the 'Guest Network' advanced networking is
located at 10.102.199.148. In my master BIND9 DNS server at 10.31.1.2 I
have this:
zone "cloud.mydomain.com" IN {
 type forward;
 forward only;
 forwarders {
  10.102.199.148;
  };
};

If I send a NAMED request directly to the virtual router while logged
into my main name server, it works:

[root@ypbind ~]# host eric-gui.cloud.mydomain.com 10.102.199.148
Using domain server:
Name: 10.102.199.148
Address: 10.102.199.148#53
Aliases:

eric-gui.cloud.mydomain.com has address 10.102.199.234

If I try to use the name server however, it doesn't work:

[root@ypbind logs]# host eric-gui.cloud.mydomain.com
Host eric-gui.cloud.viakoo.com not found: 3(NXDOMAIN)

I'm baffled, because this *was* working.

So I disabled any dnssec in the {options} on bind9  and gave all
permissions to see if that was the problem (note that this is internal
to my infrastructure, so DNS amplification isn't an issue):

  dnssec-enable no;
  dnssec-validation no;
  dnssec-lookaside auto;
  recursion yes;
  allow-recursion { any; };
  allow-query { any; };
  allow-query-cache { any; };user

Still nope. Still baffled.

Anybody got any clues as to what I may be doing wrong? I'm thinking it
has to be on the BIND9 side, because I can resolve the host name if I
talk to the virtual router directly, but for some reason it's not
allowing me to get any records from the router.

Right now I've temporarily worked around this with a script that
directly queries the MySQL database every few minutes and generates a
revised zone file on my master DNS server when the list of virtual
machines queried out of the database change

DNS quit working when I updated to 4.11.2

[Eric Lee Green]

I had this working under 4.9. All I did was, on my main BIND9 servers, 
point a forward zone at 'cloud..com' to the virtual router 
associated with all VM's that were publicly available. I could then 
resolve all foo.cloud..com names on my global network.


Somehow, though, this quit working after I updated to 4.11. I'm not 
quite sure why.


The 'Guest Network' is defined with domain 'cloud.mydomain.com'.

Okay, so my router for the 'Guest Network' advanced networking is 
located at 10.102.199.148. In my master BIND9 DNS server at 10.31.1.2 I 
have this:

zone "cloud.mydomain.com" IN {
   type forward;
   forward only;
   forwarders {
    10.102.199.148;
    };
};

If I send a NAMED request directly to the virtual router while logged 
into my main name server, it works:


[root@ypbind ~]# host eric-gui.cloud.mydomain.com 10.102.199.148
Using domain server:
Name: 10.102.199.148
Address: 10.102.199.148#53
Aliases:

eric-gui.cloud.mydomain.com has address 10.102.199.234

If I try to use the name server however, it doesn't work:

[root@ypbind logs]# host eric-gui.cloud.mydomain.com
Host eric-gui.cloud.viakoo.com not found: 3(NXDOMAIN)

I'm baffled, because this *was* working.

So I disabled any dnssec in the {options} on bind9  and gave all 
permissions to see if that was the problem (note that this is internal 
to my infrastructure, so DNS amplification isn't an issue):


    dnssec-enable no;
    dnssec-validation no;
    dnssec-lookaside auto;
    recursion yes;
    allow-recursion { any; };
    allow-query { any; };
    allow-query-cache { any; };user

Still nope. Still baffled.

Anybody got any clues as to what I may be doing wrong? I'm thinking it 
has to be on the BIND9 side, because I can resolve the host name if I 
talk to the virtual router directly, but for some reason it's not 
allowing me to get any records from the router.


Right now I've temporarily worked around this with a script that 
directly queries the MySQL database every few minutes and generates a 
revised zone file on my master DNS server when the list of virtual 
machines queried out of the database changes, but that's clearly not the 
right way to do it. The question is, what *is* the right way to do it?


-Eric

Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*

[Eric Lee Green]

EXCELLENT!

That did it. I downgraded to the libvirt and openssh rpms from Centos 
7.5, which was the latest DVD I had hanging around (likely the one I 
used to install the systems with in the first place). Turns out it was 
the Red Hat updates that came in when I did 'rpm update' that broke 
things, not the Cloudstack upgrade. I am now happily running on 
Cloudstack 4.11.2. Got rid of the qemu-kvm-ev stuff, btw, I had 
installed that at one point trying to resolve the problems, but have now 
downgraded it back to the release stuff.


In case it is not obvious, I am the person who is serious about security 
policies. The company itself doesn't care as long as we have one to show 
to investors and customers.


Regarding hardware, security policies don't cost a lot of money given 
all the open source tools we have available now. New hardware does cost 
money. Old commercial grade hardware is ridiculously cheap from Fleabay 
right now as big corporations dump "obsolete" gear.  I bought three used 
commercial-grade 24-port 10 gigabit switches for the price of a single 
new 1 gigabit switch, for example. I won't have cheap white box junk in 
the back room, it's three racks of commercial-quality gear with dual 
processors, ECC, 10gbit Ethernet, IPMI for remote management, etc., 
reliable as a brick -- just old and a bit power hungry. Power is the 
biggest issue that will lead to an upgrade, not reliability or 
performance -- I only have power budget for three racks, and new 
hardware is much more dense for a specific power consumption (modern 16 
core processors use about the same power as the 6 core processors I have 
in my nodes). When I have the budget to upgrade the hardware, I will, 
but what's there works, is reliable as a brick and not an issue, unlike 
Red Hat breaking my infrastructure with incompatible updates GRR!


On 5/22/19 9:20 AM, Andrija Panic wrote:

+1 on what Nux said - simply downgrade the packages and check if that works
- no need to reinstall everything !
As for the $ from github (CentOS 7.6 issue) - simple new line at the
end of id_rsa file and/or downgrading the ssh-keygen will work.

Don't want to annoy you, but if the company is so serious about security
policies ask them to be serious about HW as well (regular white trash PC
will do...so you can test software itself) - otherwise, you personally
suffer the consequences and look bad  (had the same pain many years ago...).

On Wed, 22 May 2019 at 17:44, Nux!  wrote:


Eric,

I think you can get away with just downgrading to stock qemu-kvm packages
(or patch your cloudstack installation).
BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages
don't get the same level of testing and QA.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -

From: "Eric Lee Green" 
To: "users" 
Sent: Wednesday, 22 May, 2019 15:33:15
Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Okay. This makes sense.

And people wonder why Amazon decided to make their own Linux rather than
use Centos and why Ubuntu has seized huge market share from Red Hat in
the past few years. SIGH.

Downgrading my CentOS is not going to be easy. There were security
patches in the latest CentOS that I am required to have. It would
actually be easier to just switch to Ubuntu at this point since we're
talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software.
I hope IBM fixes them, but I have my doubts.

On 5/22/2019 1:05 AM, Andrija Panic wrote:

Hi Eric, all,

I believe you might be hitting this one - issues on latest CentOS 7.6:
https://github.com/apache/cloudstack/pull/ due to changes in the OS
itself...

If you believe that is the case, please try with CentOS 7.4 (can confirm
works fine) and/or CentOS 7.5.

Best,
Andrija



On Wed, 22 May 2019 at 09:04, Eric Lee Green 
wrote:


On 5/21/2019 11:10 PM, Thomas Joseph wrote:

Hello Eric,

  If the router version is displayed as UNKNOWN in the portal, it

would be

a connectivity issue. Check your connections related to firewall rules
between the ACP management hosts, hypervisor and VR.  Is your VR

management

network setup correctly?

Communications between the hosts is working fine and I have not changed
the VR management network between the running 4.9 and the not-running
4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
should properly forward VR traffic.

More tomorrow when I have some sleep since it is now midnight here in
the SF Bay area.



On Wed, 22 May 2019, 6:10 am Eric Lee Green, <

eric.lee.gr...@gmail.com>

wrote:


Thanks for the response, sorry if I sound frustrated, but this is
supposed to be a simple easy process and it's been horrible all the

way

through. 4.11.1 failed so I had to downgrade back to 4.9, and now

4.11.2

has failed to upgrade too. I've thus far spent around 16 hours of my
time on what should have taken an hour at most. I'm frustrated and

bum

Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*

[Eric Lee Green]

I don't have spare hardware. I work for a small business, i.e., poor, 
which is why the hardware is the antiques I listed, it's all basically 
other company's garbage sourced from eBay. I tested it on a small 
cluster used by QA to test software deployments on various versions of 
Windows and Ubuntu Linux, our infrastructure is Centos for historical 
reasons, not because there's any real reason to run Centos other than 
standardization. We don't release our software until we make sure it 
doesn't break anything, that's why we have a QA department (which is 
peeved at me now but what else is new). Apparently Red Hat Software 
thinks one of them (a QA department) is something that doesn't matter. 
(Sorry if I sound annoyed, other people breaking my stuff annoys me). I 
can roll things back to an earlier Centos, but it's going to be a major 
effort, since I have to reinstall the OS on each of the servers one by 
one. (Data lives elsewhere and configuration info is backed up there 
nightly, that's not a problem, but we're still talking hours 
reinstalling then restoring things from backups where I'd prefer to be 
doing something that makes money for the company). Frankly, I'd prefer 
to not do that, but if it's the only way to make things work, 
downgrading to a insecure version of the OS with multiple known security 
vulnerabilities that don't pass our corporate security policy, that's 
what I'll do (I wrote the bloody policy, I'll just shut up the security 
scanner when it yells and screams at me) . Or just switch to a real 
Linux that doesn't break things when security fixes are released, if 
such a thing exists, but that's a huge effort too, especially since my 
configuration backups wouldn't just slide right in.


SIGH.

Ah well, off to work.

On 5/22/2019 7:56 AM, Andrija Panic wrote:

Eric,

did you actually test this in production?

Andrija

On Wed, 22 May 2019 at 16:33, Eric Lee Green 
wrote:


Okay. This makes sense.

And people wonder why Amazon decided to make their own Linux rather than
use Centos and why Ubuntu has seized huge market share from Red Hat in
the past few years. SIGH.

Downgrading my CentOS is not going to be easy. There were security
patches in the latest CentOS that I am required to have. It would
actually be easier to just switch to Ubuntu at this point since we're
talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software.
I hope IBM fixes them, but I have my doubts.

On 5/22/2019 1:05 AM, Andrija Panic wrote:

Hi Eric, all,

I believe you might be hitting this one - issues on latest CentOS 7.6:
https://github.com/apache/cloudstack/pull/ due to changes in the OS
itself...

If you believe that is the case, please try with CentOS 7.4 (can confirm
works fine) and/or CentOS 7.5.

Best,
Andrija



On Wed, 22 May 2019 at 09:04, Eric Lee Green 
wrote:


On 5/21/2019 11:10 PM, Thomas Joseph wrote:

Hello Eric,

  If the router version is displayed as UNKNOWN in the portal, it

would be

a connectivity issue. Check your connections related to firewall rules
between the ACP management hosts, hypervisor and VR.  Is your VR

management

network setup correctly?

Communications between the hosts is working fine and I have not changed
the VR management network between the running 4.9 and the not-running
4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
should properly forward VR traffic.

More tomorrow when I have some sleep since it is now midnight here in
the SF Bay area.



On Wed, 22 May 2019, 6:10 am Eric Lee Green, 
Thanks for the response, sorry if I sound frustrated, but this is
supposed to be a simple easy process and it's been horrible all the

way

through. 4.11.1 failed so I had to downgrade back to 4.9, and now

4.11.2

has failed to upgrade too. I've thus far spent around 16 hours of my
time on what should have taken an hour at most. I'm frustrated and

bummed.

[root@cloud2 ~]# rpm -q centos-release
centos-release-7-6.1810.2.el7.centos.x86_64
[root@cloud2 ~]# rpm -q libvirt
libvirt-4.5.0-10.el7_6.9.x86_64
[root@cloud1 ~]# rpm -qa | grep kvm
qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
[root@cloud2 ~]# cat /proc/version
Linux version 3.10.0-862.6.3.el7.x86_64
(buil...@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red

Hat

4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
[root@cloud2 ~]# cat /proc/cpuinfo
...
processor   : 23
vendor_id   : GenuineIntel
cpu family  : 6
model   : 44
model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
stepping: 2
microcode   : 0x1f
cpu MHz : 3068.000
cache size  : 12288 KB
physical id : 1
siblings: 12
core id : 10
cpu cores   : 6
apicid  : 53
initial apicid  : 53
fpu : yes
fpu_exception   : yes
cpuid level : 11
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse

Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*

[Eric Lee Green]

Okay. This makes sense.

And people wonder why Amazon decided to make their own Linux rather than 
use Centos and why Ubuntu has seized huge market share from Red Hat in 
the past few years. SIGH.


Downgrading my CentOS is not going to be easy. There were security 
patches in the latest CentOS that I am required to have. It would 
actually be easier to just switch to Ubuntu at this point since we're 
talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software. 
I hope IBM fixes them, but I have my doubts.


On 5/22/2019 1:05 AM, Andrija Panic wrote:

Hi Eric, all,

I believe you might be hitting this one - issues on latest CentOS 7.6:
https://github.com/apache/cloudstack/pull/ due to changes in the OS
itself...

If you believe that is the case, please try with CentOS 7.4 (can confirm
works fine) and/or CentOS 7.5.

Best,
Andrija



On Wed, 22 May 2019 at 09:04, Eric Lee Green 
wrote:


On 5/21/2019 11:10 PM, Thomas Joseph wrote:

Hello Eric,

 If the router version is displayed as UNKNOWN in the portal, it

would be

a connectivity issue. Check your connections related to firewall rules
between the ACP management hosts, hypervisor and VR.  Is your VR

management

network setup correctly?

Communications between the hosts is working fine and I have not changed
the VR management network between the running 4.9 and the not-running
4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
should properly forward VR traffic.

More tomorrow when I have some sleep since it is now midnight here in
the SF Bay area.



On Wed, 22 May 2019, 6:10 am Eric Lee Green, 
wrote:


Thanks for the response, sorry if I sound frustrated, but this is
supposed to be a simple easy process and it's been horrible all the way
through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2
has failed to upgrade too. I've thus far spent around 16 hours of my
time on what should have taken an hour at most. I'm frustrated and

bummed.

[root@cloud2 ~]# rpm -q centos-release
centos-release-7-6.1810.2.el7.centos.x86_64
[root@cloud2 ~]# rpm -q libvirt
libvirt-4.5.0-10.el7_6.9.x86_64
[root@cloud1 ~]# rpm -qa | grep kvm
qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
[root@cloud2 ~]# cat /proc/version
Linux version 3.10.0-862.6.3.el7.x86_64
(buil...@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
[root@cloud2 ~]# cat /proc/cpuinfo
...
processor   : 23
vendor_id   : GenuineIntel
cpu family  : 6
model   : 44
model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
stepping: 2
microcode   : 0x1f
cpu MHz : 3068.000
cache size  : 12288 KB
physical id : 1
siblings: 12
core id : 10
cpu cores   : 6
apicid  : 53
initial apicid  : 53
fpu : yes
fpu_exception   : yes
cpuid level : 11
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl
vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm
ida arat
bogomips: 6133.21
clflush size: 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
[root@cloud1 ~]# free
 totalusedfree  shared buff/cache
available
Mem:  1235963887979579234047696  632116 9752900

3332

Swap:   4194300 1956824 2237476
[root@cloud1 ~]# yum update



=

Package Arch Version Repository Size



=

Updating:
cloudstack-agent x86_64 4.11.2.0-1.el7.centos
cloudstack411  47 M
cloudstack-common x86_64 4.11.2.0-1.el7.centos
cloudstack411  58 M
cloudstack-management x86_64 4.11.2.0-1.el7.centos
cloudstack411  82 M

Three compute servers running the above processor averaging 128gb of
memory apiece, two data servers running NFS for primary and secondary
storage. Running the 4.11.2 community RPM's above.

Yes, I did register the 4.11.2 systemvmtemplate before updating. The
routers in fact started with the template, it said 4.11.2 when I opened
up the console window and looked at the login prompt. But they never got
configured, when I logged in with root/password at the console prompt
they had no networking set up

Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*

[Eric Lee Green]

On 5/21/2019 11:10 PM, Thomas Joseph wrote:

Hello Eric,

If the router version is displayed as UNKNOWN in the portal, it would be
a connectivity issue. Check your connections related to firewall rules
between the ACP management hosts, hypervisor and VR.  Is your VR management
network setup correctly?


Communications between the hosts is working fine and I have not changed 
the VR management network between the running 4.9 and the not-running 
4.11. FIrewall rules appear to be the defaults set up by Cloudstack and 
should properly forward VR traffic.


More tomorrow when I have some sleep since it is now midnight here in 
the SF Bay area.





On Wed, 22 May 2019, 6:10 am Eric Lee Green, 
wrote:


Thanks for the response, sorry if I sound frustrated, but this is
supposed to be a simple easy process and it's been horrible all the way
through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2
has failed to upgrade too. I've thus far spent around 16 hours of my
time on what should have taken an hour at most. I'm frustrated and bummed.

[root@cloud2 ~]# rpm -q centos-release
centos-release-7-6.1810.2.el7.centos.x86_64
[root@cloud2 ~]# rpm -q libvirt
libvirt-4.5.0-10.el7_6.9.x86_64
[root@cloud1 ~]# rpm -qa | grep kvm
qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
[root@cloud2 ~]# cat /proc/version
Linux version 3.10.0-862.6.3.el7.x86_64
(buil...@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
[root@cloud2 ~]# cat /proc/cpuinfo
...
processor   : 23
vendor_id   : GenuineIntel
cpu family  : 6
model   : 44
model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
stepping: 2
microcode   : 0x1f
cpu MHz : 3068.000
cache size  : 12288 KB
physical id : 1
siblings: 12
core id : 10
cpu cores   : 6
apicid  : 53
initial apicid  : 53
fpu : yes
fpu_exception   : yes
cpuid level : 11
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl
vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm
ida arat
bogomips: 6133.21
clflush size: 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
[root@cloud1 ~]# free
totalusedfree  shared buff/cache
available
Mem:  1235963887979579234047696  632116 97529003332
Swap:   4194300 1956824 2237476
[root@cloud1 ~]# yum update

=
   Package Arch Version Repository Size

=
Updating:
   cloudstack-agent x86_64 4.11.2.0-1.el7.centos
cloudstack411  47 M
   cloudstack-common x86_64 4.11.2.0-1.el7.centos
cloudstack411  58 M
   cloudstack-management x86_64 4.11.2.0-1.el7.centos
cloudstack411  82 M

Three compute servers running the above processor averaging 128gb of
memory apiece, two data servers running NFS for primary and secondary
storage. Running the 4.11.2 community RPM's above.

Yes, I did register the 4.11.2 systemvmtemplate before updating. The
routers in fact started with the template, it said 4.11.2 when I opened
up the console window and looked at the login prompt. But they never got
configured, when I logged in with root/password at the console prompt
they had no networking set up and no configuration provided, the
cloud-init output in /var/log was essentially blank.

Do I recall correctly that there is an issue with a particular version
of the hypervisor on the latest Centos 7? Any other information that you
need? I think I provided the complete log file for the cloud-init of the
router in another email...

On 5/21/2019 9:38 PM, Rohit Yadav wrote:

Hi Eric,

Can you describe your environment in detail such as management server

host distro, hypervisor version, current CloudStack version, did you
register the 4.11.2 systemvmtemplate beforw upgrading etc.

Regards.

Regards,
Rohit Yadav


From: Eric Lee Green 
Sent: Wednesday, May 22, 2019 6:21:16 AM
To: users@cloudstack.apache.org
Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*

You may remember me as the person who had to roll back to Cloudstack
4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once

Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*

[Eric Lee Green]

On 5/21/2019 11:14 PM, Rene Moser wrote:

Just a suspicion: have you checked, the system VMs actually have
allocated 512 MB RAM? I remember the systemvm templates had the setting
to only use 256 MB RAM in previous versions which is too low. IMHO this
setting must be adjusted manually in the database before the upgrade
(you can do it while running cloudstack 4.9)


Hmm... during the process of when it was trying to start up the virtual 
router, I see this:


> Trying to allocate a host and storage pools from dc:1, 
pod:null,cluster:null, requested cpu: 4000, requested ram: 4294967296


That is not 512 MB of RAM, that is 400MB of ram, but that is still 
enough to fire up the VM and the VM did in fact start up. It just didn't 
do anything once it started up, it was as if cloud-init was not getting 
any input telling it what to do (that big JSON blob that I see flowing 
from the management server to the agent to feed to the instance).





I would appreciate if one of the list could assist to check and change
if necessary.

Regards
René



On 5/22/19 2:51 AM, Eric Lee Green wrote:

You may remember me as the person who had to roll back to Cloudstack
4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once
I upgraded to it, claiming that there were inadequate resources even
though I had over 150 gigabytes of memory free in my cluster and oodles
of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb
router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
*again* it's misbehaving.

The symptom is that my virtual routers when I log into their console
show 4.11.2 but when I look at them in the console they say 'Version:
UNKNOWN'. Also when I try to ssh into their guest IP address or link
local IP address it fails. And when I try to start up a virtual machine
that uses that virtual network, it says "Network unavailable', even
though the router for that network is showing up and running.

Clearly something's broken in the virtual routers but I don't know what
because I can't get into the router virtual machines. How do I get the
console password to get into the router virtual machines? It's encrypted
in the database (duh), how do I decrypt it?

Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*

[Eric Lee Green]

Thanks for the response, sorry if I sound frustrated, but this is 
supposed to be a simple easy process and it's been horrible all the way 
through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2 
has failed to upgrade too. I've thus far spent around 16 hours of my 
time on what should have taken an hour at most. I'm frustrated and bummed.


[root@cloud2 ~]# rpm -q centos-release
centos-release-7-6.1810.2.el7.centos.x86_64
[root@cloud2 ~]# rpm -q libvirt
libvirt-4.5.0-10.el7_6.9.x86_64
[root@cloud1 ~]# rpm -qa | grep kvm
qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
[root@cloud2 ~]# cat /proc/version
Linux version 3.10.0-862.6.3.el7.x86_64 
(buil...@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 
4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018

[root@cloud2 ~]# cat /proc/cpuinfo
...
processor   : 23
vendor_id   : GenuineIntel
cpu family  : 6
model   : 44
model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
stepping    : 2
microcode   : 0x1f
cpu MHz : 3068.000
cache size  : 12288 KB
physical id : 1
siblings    : 12
core id : 10
cpu cores   : 6
apicid  : 53
initial apicid  : 53
fpu : yes
fpu_exception   : yes
cpuid level : 11
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe 
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good 
nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl 
vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt 
lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm 
ida arat

bogomips    : 6133.21
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
[root@cloud1 ~]# free
  total    used    free  shared buff/cache   
available

Mem:  123596388    79795792    34047696  632116 9752900    3332
Swap:   4194300 1956824 2237476
[root@cloud1 ~]# yum update
=
 Package Arch Version Repository Size
=
Updating:
 cloudstack-agent x86_64 4.11.2.0-1.el7.centos 
cloudstack411  47 M
 cloudstack-common x86_64 4.11.2.0-1.el7.centos 
cloudstack411  58 M
 cloudstack-management x86_64 4.11.2.0-1.el7.centos 
cloudstack411  82 M


Three compute servers running the above processor averaging 128gb of 
memory apiece, two data servers running NFS for primary and secondary 
storage. Running the 4.11.2 community RPM's above.


Yes, I did register the 4.11.2 systemvmtemplate before updating. The 
routers in fact started with the template, it said 4.11.2 when I opened 
up the console window and looked at the login prompt. But they never got 
configured, when I logged in with root/password at the console prompt 
they had no networking set up and no configuration provided, the 
cloud-init output in /var/log was essentially blank.


Do I recall correctly that there is an issue with a particular version 
of the hypervisor on the latest Centos 7? Any other information that you 
need? I think I provided the complete log file for the cloud-init of the 
router in another email...


On 5/21/2019 9:38 PM, Rohit Yadav wrote:

Hi Eric,

Can you describe your environment in detail such as management server host 
distro, hypervisor version, current CloudStack version, did you register the 
4.11.2 systemvmtemplate beforw upgrading etc.

Regards.

Regards,
Rohit Yadav


From: Eric Lee Green 
Sent: Wednesday, May 22, 2019 6:21:16 AM
To: users@cloudstack.apache.org
Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*

You may remember me as the person who had to roll back to Cloudstack
4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once
I upgraded to it, claiming that there were inadequate resources even
though I had over 150 gigabytes of memory free in my cluster and oodles
of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb
router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
*again* it's misbehaving.

The symptom is that my virtual routers when I log into their console
show 4.11.2 but when I look at them in the console they say 'Version:
UNKNOWN'. Also when I try to ssh into their guest IP address or link
local IP address it fails. And when I try to start up a virtual machine
that uses that virtual network, it says "Network unavailable', even
though the router for that ne

Upgrade to Cloudstack 4.11.2 fails *AGAIN*

[Eric Lee Green]

You may remember me as the person who had to roll back to Cloudstack 
4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once 
I upgraded to it, claiming that there were inadequate resources even 
though I had over 150 gigabytes of memory free in my cluster and oodles 
of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb 
router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and 
*again* it's misbehaving.


The symptom is that my virtual routers when I log into their console 
show 4.11.2 but when I look at them in the console they say 'Version: 
UNKNOWN'. Also when I try to ssh into their guest IP address or link 
local IP address it fails. And when I try to start up a virtual machine 
that uses that virtual network, it says "Network unavailable', even 
though the router for that network is showing up and running.


Clearly something's broken in the virtual routers but I don't know what 
because I can't get into the router virtual machines. How do I get the 
console password to get into the router virtual machines? It's encrypted 
in the database (duh), how do I decrypt it?

Re: System VMs cannot reach internet

[Eric Lee Green]

On 2/28/19 7:19 AM, Fariborz Navidan wrote:

Hello,

It seems cloudstack does not configure system vms correctly. Even if I
delete them and cloudstack recreates, they cannot reach internet however
they can be reached from outside. When I check, I see no gateway is set on
them. If I set manually, it willbe removed on next reboot. Please help!

Version of Cloudstack? Compute host OS / hypervisor type? Cloudstack 
system VM image ID?

Re: Adding template from management server's local storage

[Eric Lee Green]

On 2/27/19 2:58 PM, Fariborz Navidan wrote:

Hello All,

I have used qemu-img tool to convert a vmdk to qcow2 image. I want to add
the image as template to ACS so I can deploy from it and get the VM
migrated to ACS. I have installed httpd on management server and I am able
to start the file at url such as http://hostname/images/test.qcow2 from my
browser. However when I add the template to ACS, under Zone tab, it show
status as Active and Ready as No. It seems, it does not start to download
and install the template. I don't see any error.


You can log into the secondary storage VM and see the error, but I can 
tell you what it most probably is.


Look for

secstorage.allowed.internal.sites

in your Global Settings tab.

And see what's there. This is a list of cidrs that are allowed as 
sources for downloading images. For example, if you have an FTP server 
set up at 185.27.174.5 and you also have an internal network of 
10.120.0.0/24, you'd put this string in there:



10.120.0.0/24,185.27.174.5/32

Re: number of cores

[Eric Lee Green]

On 11/19/18 3:47 PM, Yiping Zhang wrote:

Eric:

What's your value for global setting cpu.overprovisioning.factor?

I have this value set to 3.0. Right now, one of my servers with 32 cores @ 2.0 
GHz (with HT enabled), I can allocate a total of 79 vCPU and 139 GHz to 26 VM 
instances.  That's over 200% over provisioning!


I changed it to 4.0 and restarted the management server first thing. It 
started out at 1.5. At 1.5, my zone shows 41% usage with the typical 
workload -- 169.60Ghz / 409.78Ghz.  I have 2x24x3.03ghz and 1x24x2.40ghz 
servers for a total of 203.04Ghz actual, so even without the multiplier 
I'm not over provisioning my CPU Mhz.



On 11/19/18, 6:43 AM, "Andrija Panic"  wrote:

 Unless someone gives you better answer, I guess it's for fun - to have more
 detailed numbers in dashboard (may be it's related to other hypervisor
 types, just assuming... or not...)
 
 Cheers
 
 On Mon, 19 Nov 2018 at 14:11, Ugo Vasi  wrote:
 
 > Hi Andrija,

 > not having noticed this new voice before I wondered if it is limiting
 > the fact of reaching or exceeding the number of physical cores.
 >
 > What is the purpose of this dashboard pane?
 >
 >
 > Il 19/11/18 12:56, Andrija Panic ha scritto:
 > > Hi Ugo,
 > >
 > > Why would you want to do this, just curious ?
 > >
 > > I believe it's not possible, but anyway (at least with KVM, probably 
same
 > > for other hypervisors) it doesn't even makes sense/use, since when
 > > deploying a VM, ACS query host free/unused number of MHz (GHz), so it's
 > not
 > > even relevant for ACS - number of cores in not relevant in ACS
 > calculations
 > > during VM deployment.
 > >
 > >
 > > Cheers,
 > > Andrija
 > >
 > > On Mon, Nov 19, 2018, 11:31 Ugo Vasi  >
 > >> Hi all,
 > >> in the dashboard of an ACS installation vesion 4.11.1.0 (Ubuntu 16.04
 > >> with KVM hypervisor), the new entry "# of CPU Cores" appears.
 > >> Is it possible to over-provision like for MHz or storage?
 > >>
 > >> Thanks
 > >>
 > >>
 > >> --
 > >>
 > >> *Ugo Vasi* / System Administrator
 > >> ugo.v...@procne.it 
 > >>
 > >>
 > >>
 > >>
 > >> *Procne S.r.l.*
 > >> +39 0432 486 523
 > >> via Cotonificio, 45
 > >> 33010 Tavagnacco (UD)
 > >> www.procne.it 
 > >>
 > >>
 > >> Le informazioni contenute nella presente comunicazione ed i relativi
 > >> allegati possono essere riservate e sono, comunque, destinate
 > >> esclusivamente alle persone od alla Società sopraindicati. La
 > >> diffusione, distribuzione e/o copiatura del documento trasmesso da 
parte
 > >> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
 > >> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
 > >> "Codice in materia di protezione dei dati personali". Se avete 
ricevuto
 > >> questo messaggio per errore, vi preghiamo di distruggerlo e di 
informare
 > >> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
 > >> i...@procne.it .
 > >>
 > >>
 > >
 > >
 > >
 >
 >
 > --
 >
 > *Ugo Vasi* / System Administrator
 > ugo.v...@procne.it 
 >
 >
 >
 >
 > *Procne S.r.l.*
 > +39 0432 486 523
 > via Cotonificio, 45
 > 33010 Tavagnacco (UD)
 > www.procne.it 
 >
 >
 > Le informazioni contenute nella presente comunicazione ed i relativi
 > allegati possono essere riservate e sono, comunque, destinate
 > esclusivamente alle persone od alla Società sopraindicati. La
 > diffusione, distribuzione e/o copiatura del documento trasmesso da parte
 > di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
 > dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
 > "Codice in materia di protezione dei dati personali". Se avete ricevuto
 > questo messaggio per errore, vi preghiamo di distruggerlo e di informare
 > immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
 > i...@procne.it .
 >
 >
 
 --
 
 Andrija Panić
 

Re: number of cores

[Eric Lee Green]

On 11/19/18 03:56, Andrija Panic wrote:

Hi Ugo,

Why would you want to do this, just curious ?

I believe it's not possible, but anyway (at least with KVM, probably same
for other hypervisors) it doesn't even makes sense/use, since when
deploying a VM, ACS query host free/unused number of MHz (GHz), so it's not
even relevant for ACS - number of cores in not relevant in ACS calculations
during VM deployment.



I think you are misunderstanding the question. I have 72 cores in my 
cluster. Each of my hosts has 24 cores. With 4.9.2, I can provision 10 
virtual machines, each of which is programmed with 8 cores, meaning 80 
cores total. They on average are using only 25% of the CPU available to 
them (they need to be able to burst) and my compute servers on average 
are only 40% CPU used so that is not a problem.


When I tried upgrading to 4.11.1,  the dashboard showed a new value "# 
of CPU Cores" in red and showed that I had more cores provisioned for 
virtual machines than available in the cluster (80 versus 72 available). 
Cloudstack would not launch new virtual machines. I shut down two 
virtual machines, and now I can launch one, but not the second because I 
would need 80 cores total in my cluster. I cannot launch all 10 virtual 
machines because I would need 80 cores total. I know this because I 
tried it. I then used MySQL to tell Cloudstack that each of my hosts has 
48 cores (144 total), and suddenly I can launch all of my virtual machines.


Is this a bug in 4.11.1? Or is this expected behavior? If expected 
behavior, is there a way to over-provision "total # of cores used" other 
than to go into MySQL and tell it that my hosts have more cores than 
they in fact have? (Note that my service offerings are limited to 8 
cores max, so there's no way to launch a single VM with more cores than 
exists on a physical host, since all my hosts have 24 cores).




On Mon, Nov 19, 2018, 11:31 Ugo Vasi 
Hi all,
in the dashboard of an ACS installation vesion 4.11.1.0 (Ubuntu 16.04
with KVM hypervisor), the new entry "# of CPU Cores" appears.
Is it possible to over-provision like for MHz or storage?

Thanks


--

*Ugo Vasi* / System Administrator
ugo.v...@procne.it 




*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it 


Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
i...@procne.it .

Re: Problem creating networks after 4.11 upgrade

[Eric Lee Green]

Yeah, had all sorts of problems with custom network offerings after 
upgrading to 4.11.1, along with problems with launching virtual machines 
(every attempt to launch resulted in a "not enough resources" error), 
couldn't get virtual routers to come up for custom networks, etc. I 
didn't have time in my service window to do any detailed examinations of 
why they were failing, I just downgraded back to 4.9.2 before my service 
window ended. When 4.11 is stable, maybe I'll try upgrading to it again. 
(OS: Centos 7. Old version: 4.9.2. New version: 4.11.1. Hardware: Three 
compute servers with dual hexacore processors and 96gb+ of memory w/KVM. 
End result after two hours of trying to make it work: Downgrade back to 
4.9.2.)


I was thinking about migrating most of my other computer servers into 
the Cloudstack cloud because it's easier for my users to take care of 
their own resources, but I was hoping to do it after migrating to 4.11. 
I guess not.


On 11/4/18 13:14, Jean-Francois Nadeau wrote:


I all,

I was wondering if anyone else had this problem after upgrading from 4.9.
  All our networks are using a custom network offering with no services
defined since the physical network provides DHCP and DNS.   Environment is
CentOS 7, KVM with the openvswitch driver.

Now after the upgrade to 4.11,  creating a network using that same network
offering fails with an  "Unable to convert network offering with specified
id to network profile" error.

The issue is documented here:
https://github.com/apache/cloudstack/issues/2989

I hope someone can have a look at it.  This is the last issue that blocks
us from upgrading.

best,

Jean-Francois

Re: Is that safe to put public IP directly on Virtual Router/ System VMs?

[Eric Lee Green]

On 9/26/18 6:21 PM, Netlynker wrote:

Hi Eric,

Usual setup for my other infra service is that we use external firewall
doing NAT and protecting the resource behind. The public IP will stay on
that firewall and it is NATed to private IP of the service internal.

What CS document imply is to put “real” public IP address on System VMs and
VR which will leave those systems exposed directly to outside world.


That is the configuration for a public Cloudstack service. If you are a 
corporation setting up a private cloud within a corporation, the "public 
address" should be routed to your corporate network. For example, my 
corporate "floor" network that goes to the desktops on people's desks is 
10.100.0.0/16 while my corporate "wifi" is 10.101.0.0/16. I gave my 
Cloudstack cluster the public address of 10.102.0.0/16 which is routed 
to my external firewall router via a VLAN-enabled Cisco Layer 3 switch 
to the actual firewall on the external DMZ VLAN 10.2.0.0/24. (Other 
private networks such as the pool for VPC creation use additional 10.x 
subnets). The 10.102.x.y addresses are "internet" addresses as far as 
Cloudstack is concerned -- they can access the Internet. The fact that 
they're accessing it via my corporate network rather than being directly 
connected to the Internet is irrelevant.


As far as the safety of the router virtual machines, they are a small 
Linux distribution just like the one in your external firewall, that, 
like the one in your external firewall, is configured with routes and 
firewall rules to be secure. If you were doing a public-facing Internet 
service there would be no problem putting them directly on the public 
Internet. From a design perspective they're no more or less secure than 
your current NAT firewall, which is also a small Linux distribution 
unless it's a Cisco. Whether you want to put them directly onto the 
Internet or not depends on your design goals, not safety. If you want to 
be able to access your virtual machines from the public Internet without 
logging in via a VPN, such as publicly available services, then a NAT 
firewall between your Cloudstack service and your virtual machines will 
not allow for that. If you are satisfied with having your virtual 
machines only be reachable from your corporate network and comfortable 
with VPN access for those times you need to access them remotely, then 
sure, put them on your corporate network behind the NAT firewall. That's 
what I do, because my production virtual machines providing service to 
the general public are running on the Amazon cloud, my Cloudstack 
installation is for corporate-private R and support instances that the 
general public isn't supposed to access.



My question is if that architecture is recommeneded and how safe it is to
put “real” public IP on System VMs and VRs directly.

Thanks in advance,
Netlynker

On Thu, 27 Sep 2018 at 8:58 AM, Eric Lee Green 
wrote:


On 9/25/18 6:29 PM, Netlynker wrote:

Hi,

I looked at the deployment architecture from document and it said to have
public IP addresses on Virtaul Router/System VMs.

Is that recommended setup?

How safe will it be to expose Virtaul Router/ System VMs directly to
internet?


If a virtual router is not connected to the Internet, how will it route
traffic from your internal VM's in their virtual private networks to the
Internet? Magic? (This presuming you have an Internet-facing service,
but even if it's internal to your company, the virtual router is going
to need to be able to talk to the Internet via your company's "internal"
Internet network if your internal VM's on their own internal private
networks are going to get to the Internet or other corporate resources).

Re: Case of CEPH Environment Hyper-converged

[Eric Lee Green]

In particular, Ceph needs a *lot* of spindles / CPU / network interfaces 
to run with reasonable performance. I tried just a 3-system 6-spindle 
Ceph implementation, and was getting streaming write throughput of 20 
megabytes per second. Which, uhm, isn't good, in case you're wondering. 
As in, external USB 2.0 disk drive speed not good.


Use NFS if you're doing a small installation. I did a small 2 node 
installation (1 compute, 1 file server) that way, and it worked fine.


On 9/3/18 11:13, Andrija Panic wrote:
Had exactly the case (CEPH +KVM + CS), very long time ago = just dont' 
do it as simple as that :)


BTW, It's interesting ,these days it's called hyper-converged when you 
put all things (roles) on single box - back in the days (again, 
lng time ago) this was called "we don't have enough money" :)))


Best

On Mon, 3 Sep 2018 at 19:57, Felipe Rossi 
 wrote:


Hi All,

I would like know if some one have case of CEPH + CS + KVM or
other hypervisor in
environment hyper-converged using CS.


Att / Regards




--

Andrija Panić

Re: Autostarting VMs on KVM?

[Eric Lee Green]

If you set the offering to allow HA and create the instances as HA 
instances, they will autostart once the management server figures out 
they're really dead (either because it used STONITH to kill the 
unreachable node, or because that node became reachable again). When I 
had to reboot my cluster due to a massive network failure (critical 10 
gigabit switch croaked, had to slide a new one in), all the instances 
marked "HA" came back up all by themselves without me having to do 
anything about it.


On 8/15/18 09:11, Asai wrote:

Thanks, Dag,

Looks like scripting it is the way to go.
Asai



On Aug 15, 2018, at 9:06 AM, Dag Sonstebo  wrote:

Hi Asai,

In short – no that is not a use case CloudStack is designed for, the VM states 
are controlled by CloudStack management. You should however look at using HA 
service offerings and host HA (if you meet all the pre-requisites). Between 
these mechanisms VMs can be brought up on other hosts if a host goes down.

Alternatively if you are looking to trigger an automated startup of VMs I 
suggest you simply script this with e.g. cloudmonkey. Keep in mind this still 
requires a healthy management server though.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 15/08/2018, 16:47, "Asai"  wrote:

Thanks, Dag,

On boot of the server, I would like the VMs to start up automatically, 
rather than me having to go to the management console and start them manually.  
We suffered some downtime and in restarting the hardware, I had to manually get 
everything back up and running.
Asai



dag.sonst...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




On Aug 15, 2018, at 1:22 AM, Dag Sonstebo  wrote:

Hi Asai,

Can you explain a bit more what you are trying to achieve? Everything in 
CloudStack is controlled by the management server, not the KVM host, and in 
general the assumption is a KVM host is always online.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 15/08/2018, 03:38, "Asai"  wrote:

   Greetings,

   Can anyone offer advice on how to autostart VMs at boot time using KVM?  
There doesn’t seem to be any documentation for this in the CS docs.  We’re on 
CS 4.9.2.0.

   I tried doing it with virsh autostart, but it just throws an error.

   Thank you,
   Asai



dag.sonst...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

Re: [PROPOSE] EOL for supported OSes & Hypervisors

[Eric Lee Green]

This is the type of discussion that I wanted to open - the argument that I see 
for earlier dropping of v6 is that - Between May 2018 and q2 2020 RHEL/CentOS 
6.x will only receive security and mission critical updates, meanwhile packages 
on which we depend or may want to utilise in the future are been deprecated or 
not developed for v6.x
But this has always been the case for Centos 6.x. It is running antique 
versions of everything, and has been doing so for quite some time. It 
is, for example, running versions of Gnome and init that have been 
obsolete for years. Same deal with the version of MySQL that it comes with.


The reality is that Centos 6.x guest support, at the very least, needs 
to be tested with each new version of Cloudstack until final EOL of 
Centos 6 in Q2 2020. New versions of Cloudstack with new features not 
supported by Centos 6 (such as LVM support for KVM, which requires the 
LIO storage stack) can require Centos 7 or later, but the last 
Cloudstack version that supports Centos 6.x as its server host should 
continue to receive bug fixes until Centos 6.x is EOL.


Making someone's IT investment obsolete is a way to irrelevancy. 
Cloudstack is already an also-ran in the cloud marketplace. Making 
someone's IT investment obsolete before the official EOL time for their 
IT investment is a good way to have a mass migration away from your 
technology.


This doesn't particularly affect me since my Centos 6 virtualization 
hosts are not running Cloudstack and are going to be re-imaged to Centos 
7 before being added to the Cloudstack cluster, but ignoring the IT 
environment that people actually live in, as versus the one we wish 
existed, is annoying regardless. A friend of mine once said of the state 
of ERP software, "enterprise software is dog food if dog food was being 
designed by cats." I.e., the people writing the software rarely have any 
understanding of how it is actually used by real life enterprises in 
real life environments. Don't be those people.



On 01/16/2018 09:58 AM, Paul Angus wrote:

Hi Eric,

This is the type of discussion that I wanted to open - the argument that I see 
for earlier dropping of v6 is that - Between May 2018 and q2 2020 RHEL/CentOS 
6.x will only receive security and mission critical updates, meanwhile packages 
on which we depend or may want to utilise in the future are been deprecated or 
not developed for v6.x
Also the testing and development burden on the CloudStack community increases 
as we try to maintain backward compatibility while including new versions.

Needing installation documentation for centos 7 is a great point, and something 
that we need to address regardless.


Does anyone else have a view, I'd really like to here from a wide range of 
people.

Kind regards,

Paul Angus

paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
   
  



-Original Message-
From: Eric Green [mailto:eric.lee.gr...@gmail.com]
Sent: 12 January 2018 17:24
To: users@cloudstack.apache.org
Cc: d...@cloudstack.apache.org
Subject: Re: [PROPOSE] EOL for supported OSes & Hypervisors

Official EOL for Centos 6 / RHEL 6 as declared by Red Hat Software is 
11/30/2020. Jumping the gun a bit there, padme.

People on Centos 6 should certainly be working on a migration strategy right 
now, but the end is not here *yet*. Furthermore, the install documentation is 
still written for Centos 6 rather than Centos 7. That needs to be fixed before 
discontinuing support for Centos 6, eh?


On Jan 12, 2018, at 04:35, Rohit Yadav  wrote:

+1 I've updated the page with upcoming Ubuntu 18.04 LTS.


After 4.11, I think 4.12 (assuming releases by mid of 2018) should remove 
"declared" (they might still work with 4.12+ but in docs and by project we 
should officially support them) support for following:


a. Hypervisor:

XenServer - 6.2, 6.5,

KVM - CentOS6, RHEL6, Ubuntu12.04 (I think this is already removed, packages 
don't work I think?)

vSphere/Vmware - 4.x, 5.0, 5.1, 5.5


b. Remove packaging for CentOS6.x, RHEL 6.x (the el6 packages), and Ubuntu 
12.04 (any non-systemd debian distro).


Thoughts, comments?

Re: hypervisor choice

[Eric Lee Green]

On 11/10/2017 11:01 AM, Ron Wheeler wrote:
I have been using CentOS for a long time but they seem to have screwed 
up the recent updates to CentOS 7 to the point where after updating to 
the latest version (originally build 514 and now 683), the system no 
longer boots. I have to boot to build 327 which runs fine.


The idea of having a server that fails after updating is not in my 
comfort zone. 
The other popular choice if you are using KVM on Linux is Ubuntu LTS. 
The current LTS version is 16.04 which is supported until 2021. 
Cloudstack runs fine on Ubuntu LTS, but configuring the network may be a 
bit cumbersome for someone accustomed to the Centos 
/etc/sysconfig/network-scripts mechanism.


In my experience over the years Ubuntu has not been quite as stable as 
Red Hat Enterprise Linux, *but*, that may have changed with RHEL7/Centos 
7, where they appear to break things regularly between minor version 
updates in order to "improve" the system. I, too, ended up with the 
issue of one of my Centos 7 servers not rebooting after an update, and 
having to boot it back to an older kernel. I ended up re-formatting and 
re-installing that server entirely and restoring the system 
configuration from backups.


At this point I'd suggest remaining with KVM on Linux as your 
hypervisor. It appears to perform better overall than Xen or vSphere and 
the cost-effectiveness overall cannot be beat, especially if you are 
buying hardware in bulk and using an automated mechanism to deploy your 
hardware and the software load upon it so that you don't have to manage 
it individually.


If you are looking for overall reliability (at a cost), vSphere is of 
course "the" reliable choice (I have some ESXi hosts that have been up 
for over 500 days, and the last time they went down was during a planned 
outage to rearrange the racks), but it is very picky about its hardware 
and likely won't like your current hardware. It can also become somewhat 
expensive as you add hosts to your vSphere cluster, which is the basis 
of a CloudStack pod (rather than the individual hosts). It's also as 
much as 10% slower by my measurements under many workloads because they 
make numerous decisions that improve reliability at the expense of 
performance. Still, for customers that value reliability above all else, 
vSphere is a brick -- reliable and pretty much bullet-proof.

Re: Secondary storage is not secondary properly

[Eric Lee Green]

On 08/17/2017 11:17 AM, Asanka Gunasekara wrote:

Hi Dag, the ip 172.17.101.1 which it is looking for is my gateway IP. Below
are the urls for the the requested query output files

SELECT * FROM cloud.image_store;
Interesting. The only row with a NULL 'removed' column  looks good, so 
it looks like your database configuration is correct:


5  |   NFS_Secondary   |  NFS  |   nfs  | 
nfs://172.17.101.253/share_smb/export/secondary |1  |   ZONE |  
Image   |  a4e17aca-dc16-494e-b696-8f8fae58a391 | 2017-08-14 
19:12:50.0 |||


Compare with my own query of my own image store, which is basically 
identical


MariaDB [cloud]> select * from cloud.image_store;
+++-+--+-++---+---+--+--+-+-+++
| id | name   | image_provider_name | protocol | 
url | data_center_id | scope | role | 
uuid | 
parent   | created | removed | 
total_size | used_bytes |

+++-+--+-++---+---+--+--+-+-+++
|  1 | secondary1 | NFS | nfs  | 
nfs://10.100.255.1/export/secondary |  1 | ZONE  | Image | 
fdaab425-a102-484b-b746-c07c4b564edd | 
50d77b6b-4d99-3695-b830-24ed10d0155c | 2017-07-31 00:55:06 | NULL 
|   NULL |   NULL |

+++-+--+-++---+---+--+--+-+-+++
1 row in set (0.00 sec)

Note that 10.100.255.1 is on my management network, which is also my 
storage network (I have everything coming in on VLAN's on a 10Gbit bond, 
the 10.100.x.x network is on VLAN 100).  When I go into my secondary 
storage VM, here is what I see:


Now, getting into my secondary storage VM and asking it for a list of 
addresses, here is what I see:


root@s-397-VM:~# ip addr list | grep inet
inet 127.0.0.1/8 scope host lo
inet 169.254.0.95/16 brd 169.254.255.255 scope global eth0
inet 10.100.196.66/16 brd 10.100.255.255 scope global eth1
inet 10.101.199.255/16 brd 10.101.255.255 scope global eth2
inet 10.100.250.159/16 brd 10.100.255.255 scope global eth3

So as you can see, it definitely has access to the management network 
(10.100.x.x), as well as my public IP pool (10.101.x.x) and storage pool 
(the second 10.100 address).  As well as the local agent-visible IP (the 
169.254.0.95) that ssh is listening on so that the agent can configure 
the VM via its shared keys.


Here is what my ssvm-check says:

root@s-397-VM:/opt# /usr/local/cloud/systemvm/ssvm-check.sh

First DNS server is  10.100.255.2
PING 10.100.255.2 (10.100.255.2): 48 data bytes
56 bytes from 10.100.255.2: icmp_seq=0 ttl=64 time=0.189 ms
56 bytes from 10.100.255.2: icmp_seq=1 ttl=64 time=0.438 ms
--- 10.100.255.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.189/0.314/0.438/0.125 ms
Good: Can ping DNS server

Good: DNS resolves download.cloud.com

nfs is currently mounted
Mount point is /mnt/SecStorage/50d77b6b-4d99-3695-b830-24ed10d0155c
Good: Can write to mount point

Management server is 10.100.255.2. Checking connectivity.
Good: Can connect to management server port 8250

Good: Java process is running

Tests Complete. Look for ERROR or WARNING above.


Your says 'Java process not running'. I wonder if your 169 address is 
working? Let's check your cloud.log to see if you ever got a setup command:


s-397-VM:/# grep com.cloud.agent.api.SecStorageSetupCommand 
/var/log/cloud.log


Mine replies with:

2017-08-17 02:21:12,975 DEBUG [cloud.agent.Agent] 
(agentRequest-Handler-1:null) Request:Seq 9-231372430856159233:  { Cmd , 
MgmtId: 11967559506, via: 9, Ver: v1, Flags: 100111, 
[{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.100.255.1/export/secondary","_role":"Image"}},"secUrl":"nfs://10.100.255.1/export/secondary","postUploadKey":"KZQd8G06ABN3D_CGAJiKBmhLe3e5dim5hfA7ouuZnvQtZNoHxE3T4WiqTxOdVPBh5hHhNtvX8e9Gac0Tw7gM5g","wait":0}}] 
}


See if you got a similar command with your own NFS server's address.