Important to see you happy with 4.11.2 - the rest (HW) will follow ;) On Wed, 22 May 2019 at 21:49, Eric Lee Green <eric.lee.gr...@gmail.com> wrote:
> EXCELLENT! > > That did it. I downgraded to the libvirt and openssh rpms from Centos > 7.5, which was the latest DVD I had hanging around (likely the one I > used to install the systems with in the first place). Turns out it was > the Red Hat updates that came in when I did 'rpm update' that broke > things, not the Cloudstack upgrade. I am now happily running on > Cloudstack 4.11.2. Got rid of the qemu-kvm-ev stuff, btw, I had > installed that at one point trying to resolve the problems, but have now > downgraded it back to the release stuff. > > In case it is not obvious, I am the person who is serious about security > policies. The company itself doesn't care as long as we have one to show > to investors and customers. > > Regarding hardware, security policies don't cost a lot of money given > all the open source tools we have available now. New hardware does cost > money. Old commercial grade hardware is ridiculously cheap from Fleabay > right now as big corporations dump "obsolete" gear. I bought three used > commercial-grade 24-port 10 gigabit switches for the price of a single > new 1 gigabit switch, for example. I won't have cheap white box junk in > the back room, it's three racks of commercial-quality gear with dual > processors, ECC, 10gbit Ethernet, IPMI for remote management, etc., > reliable as a brick -- just old and a bit power hungry. Power is the > biggest issue that will lead to an upgrade, not reliability or > performance -- I only have power budget for three racks, and new > hardware is much more dense for a specific power consumption (modern 16 > core processors use about the same power as the 6 core processors I have > in my nodes). When I have the budget to upgrade the hardware, I will, > but what's there works, is reliable as a brick and not an issue, unlike > Red Hat breaking my infrastructure with incompatible updates GRR! > > On 5/22/19 9:20 AM, Andrija Panic wrote: > > +1 on what Nux said - simply downgrade the packages and check if that > works > > - no need to reinstall everything ! > > As for the $3333 from github (CentOS 7.6 issue) - simple new line at the > > end of id_rsa file and/or downgrading the ssh-keygen will work. > > > > Don't want to annoy you, but if the company is so serious about security > > policies ask them to be serious about HW as well (regular white trash PC > > will do...so you can test software itself) - otherwise, you personally > > suffer the consequences and look bad (had the same pain many years > ago...). > > > > On Wed, 22 May 2019 at 17:44, Nux! <n...@li.nux.ro> wrote: > > > >> Eric, > >> > >> I think you can get away with just downgrading to stock qemu-kvm > packages > >> (or patch your cloudstack installation). > >> BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages > >> don't get the same level of testing and QA. > >> > >> -- > >> Sent from the Delta quadrant using Borg technology! > >> > >> Nux! > >> www.nux.ro > >> > >> ----- Original Message ----- > >>> From: "Eric Lee Green" <eric.lee.gr...@gmail.com> > >>> To: "users" <users@cloudstack.apache.org> > >>> Sent: Wednesday, 22 May, 2019 15:33:15 > >>> Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN* > >>> Okay. This makes sense. > >>> > >>> And people wonder why Amazon decided to make their own Linux rather > than > >>> use Centos and why Ubuntu has seized huge market share from Red Hat in > >>> the past few years. SIGH. > >>> > >>> Downgrading my CentOS is not going to be easy. There were security > >>> patches in the latest CentOS that I am required to have. It would > >>> actually be easier to just switch to Ubuntu at this point since we're > >>> talking a complete re-install anyhow. SIGH. Thank you, Red Hat > Software. > >>> I hope IBM fixes them, but I have my doubts. > >>> > >>> On 5/22/2019 1:05 AM, Andrija Panic wrote: > >>>> Hi Eric, all, > >>>> > >>>> I believe you might be hitting this one - issues on latest CentOS 7.6: > >>>> https://github.com/apache/cloudstack/pull/3333 due to changes in the > OS > >>>> itself... > >>>> > >>>> If you believe that is the case, please try with CentOS 7.4 (can > confirm > >>>> works fine) and/or CentOS 7.5. > >>>> > >>>> Best, > >>>> Andrija > >>>> > >>>> > >>>> > >>>> On Wed, 22 May 2019 at 09:04, Eric Lee Green < > eric.lee.gr...@gmail.com> > >>>> wrote: > >>>> > >>>>> On 5/21/2019 11:10 PM, Thomas Joseph wrote: > >>>>>> Hello Eric, > >>>>>> > >>>>>> If the router version is displayed as UNKNOWN in the portal, > it > >>>>> would be > >>>>>> a connectivity issue. Check your connections related to firewall > rules > >>>>>> between the ACP management hosts, hypervisor and VR. Is your VR > >>>>> management > >>>>>> network setup correctly? > >>>>> Communications between the hosts is working fine and I have not > changed > >>>>> the VR management network between the running 4.9 and the not-running > >>>>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack > and > >>>>> should properly forward VR traffic. > >>>>> > >>>>> More tomorrow when I have some sleep since it is now midnight here in > >>>>> the SF Bay area. > >>>>> > >>>>> > >>>>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, < > >> eric.lee.gr...@gmail.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Thanks for the response, sorry if I sound frustrated, but this is > >>>>>>> supposed to be a simple easy process and it's been horrible all the > >> way > >>>>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now > >> 4.11.2 > >>>>>>> has failed to upgrade too. I've thus far spent around 16 hours of > my > >>>>>>> time on what should have taken an hour at most. I'm frustrated and > >>>>> bummed. > >>>>>>> [root@cloud2 ~]# rpm -q centos-release > >>>>>>> centos-release-7-6.1810.2.el7.centos.x86_64 > >>>>>>> [root@cloud2 ~]# rpm -q libvirt > >>>>>>> libvirt-4.5.0-10.el7_6.9.x86_64 > >>>>>>> [root@cloud1 ~]# rpm -qa | grep kvm > >>>>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64 > >>>>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64 > >>>>>>> [root@cloud2 ~]# cat /proc/version > >>>>>>> Linux version 3.10.0-862.6.3.el7.x86_64 > >>>>>>> (buil...@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red > >> Hat > >>>>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018 > >>>>>>> [root@cloud2 ~]# cat /proc/cpuinfo > >>>>>>> ... > >>>>>>> processor : 23 > >>>>>>> vendor_id : GenuineIntel > >>>>>>> cpu family : 6 > >>>>>>> model : 44 > >>>>>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz > >>>>>>> stepping : 2 > >>>>>>> microcode : 0x1f > >>>>>>> cpu MHz : 3068.000 > >>>>>>> cache size : 12288 KB > >>>>>>> physical id : 1 > >>>>>>> siblings : 12 > >>>>>>> core id : 10 > >>>>>>> cpu cores : 6 > >>>>>>> apicid : 53 > >>>>>>> initial apicid : 53 > >>>>>>> fpu : yes > >>>>>>> fpu_exception : yes > >>>>>>> cpuid level : 11 > >>>>>>> wp : yes > >>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr > >> pge > >>>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe > >>>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts > >> rep_good > >>>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor > >> ds_cpl > >>>>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt > >>>>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid > >> dtherm > >>>>>>> ida arat > >>>>>>> bogomips : 6133.21 > >>>>>>> clflush size : 64 > >>>>>>> cache_alignment : 64 > >>>>>>> address sizes : 40 bits physical, 48 bits virtual > >>>>>>> [root@cloud1 ~]# free > >>>>>>> total used free shared > buff/cache > >>>>>>> available > >>>>>>> Mem: 123596388 79795792 34047696 632116 9752900 > >>>>> 39999332 > >>>>>>> Swap: 4194300 1956824 2237476 > >>>>>>> [root@cloud1 ~]# yum update > >>>>>>> > >>>>>>> > >> > ========================================================================================================================================================================= > >>>>>>> Package Arch Version Repository > Size > >>>>>>> > >>>>>>> > >> > ========================================================================================================================================================================= > >>>>>>> Updating: > >>>>>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos > >>>>>>> cloudstack411 47 M > >>>>>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos > >>>>>>> cloudstack411 58 M > >>>>>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos > >>>>>>> cloudstack411 82 M > >>>>>>> > >>>>>>> Three compute servers running the above processor averaging 128gb > of > >>>>>>> memory apiece, two data servers running NFS for primary and > secondary > >>>>>>> storage. Running the 4.11.2 community RPM's above. > >>>>>>> > >>>>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating. > The > >>>>>>> routers in fact started with the template, it said 4.11.2 when I > >> opened > >>>>>>> up the console window and looked at the login prompt. But they > never > >> got > >>>>>>> configured, when I logged in with root/password at the console > prompt > >>>>>>> they had no networking set up and no configuration provided, the > >>>>>>> cloud-init output in /var/log was essentially blank. > >>>>>>> > >>>>>>> Do I recall correctly that there is an issue with a particular > >> version > >>>>>>> of the hypervisor on the latest Centos 7? Any other information > that > >> you > >>>>>>> need? I think I provided the complete log file for the cloud-init > of > >> the > >>>>>>> router in another email... > >>>>>>> > >>>>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote: > >>>>>>>> Hi Eric, > >>>>>>>> > >>>>>>>> Can you describe your environment in detail such as management > >> server > >>>>>>> host distro, hypervisor version, current CloudStack version, did > you > >>>>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc. > >>>>>>>> Regards. > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> Rohit Yadav > >>>>>>>> > >>>>>>>> ________________________________ > >>>>>>>> From: Eric Lee Green <eric.lee.gr...@gmail.com> > >>>>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM > >>>>>>>> To: users@cloudstack.apache.org > >>>>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN* > >>>>>>>> > >>>>>>>> You may remember me as the person who had to roll back to > Cloudstack > >>>>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual > machines > >>>>> once > >>>>>>>> I upgraded to it, claiming that there were inadequate resources > even > >>>>>>>> though I had over 150 gigabytes of memory free in my cluster and > >> oodles > >>>>>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a > >>>>> 512mb > >>>>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and > >>>>>>>> *again* it's misbehaving. > >>>>>>>> > >>>>>>>> The symptom is that my virtual routers when I log into their > console > >>>>>>>> show 4.11.2 but when I look at them in the console they say > >> 'Version: > >>>>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or > link > >>>>>>>> local IP address it fails. And when I try to start up a virtual > >> machine > >>>>>>>> that uses that virtual network, it says "Network unavailable', > even > >>>>>>>> though the router for that network is showing up and running. > >>>>>>>> > >>>>>>>> Clearly something's broken in the virtual routers but I don't know > >> what > >>>>>>>> because I can't get into the router virtual machines. How do I get > >> the > >>>>>>>> console password to get into the router virtual machines? It's > >>>>> encrypted > >>>>>>>> in the database (duh), how do I decrypt it? > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> rohit.ya...@shapeblue.com > >>>>>>>> www.shapeblue.com > >>>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK > >>>>>>>> @shapeblue > >>>>>>>> > >>>>>>>> > >>>>>>>> > > > > -- Andrija Panić
