It seems easier than I thought... stop mgmt server Rollback DB with your statement shared in some of the previous emails (and after that please make DB backup immediately, just for the sake of having one...) Again, perform that manual template install with: - From mgmt, mount the Secondary Storage NFS on /mnt/secondary and execute:
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /mnt/secondary -u http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2 -h kvm -F umount /mnt/secondary Update some fields for that template: update vm_template set name="systemvm-kvm-4.11.3" where id="3"; update vm_template set display_text="systemvm-kvm-4.11.3" where id="3"; update vm_template set type="USER" where id="3"; Check that ID=3 for default KVM system template - but should be identical on everyones env. CloudStack upgrade is looking for specific NAME of the template in order to proceed with the upgrade (and initially type=USER, and the upgrade will change that again to SYSTEM) After those SQL changes, start mgmt server - all should be good. If mgmt is started fine, you will have to DESTROY all system VMs and VRs, i.e. NOT reboot - to force using new template - so Destroy SSVM and CPVM (will be recreated automatically) and for the network/VPC - restart them with "Clean Up" option. My previous instructions didn't work, since I was doing more things than what I've shared with you - sorry for that - but those steps are needed anyway, just were incomplete. (We could do different fix by manually inserting DB records for a completely new template, new rows in vm_template, templat_store_ref and template_zone_ref and unpack that BZ2 file/template to specific folder on Secondary Storage - but that would require a bit of MySQL and is more error-prone. This kind of fix would simulate the step of manually registering new systemVM template as regular user template, with the specific name - which would be later recognized by it's name as the new systemVM templat for 4.11.3) Let me know how it goes. Andrija On Thu, 3 Oct 2019 at 19:07, Fariborz Navidan <mdvlinqu...@gmail.com> wrote: > MariaDB [cloud]> select max(id) from vm_template; > +---------+ > | max(id) | > +---------+ > | 390 | > +---------+ > > Can we have a chat session to investigate together? > > On Thu, Oct 3, 2019 at 8:27 PM Andrija Panic <andrija.pa...@gmail.com> > wrote: > > > do for me: > > > > select max(id) from vm_template: > > > > i might need more info later... > > > > On Thu, 3 Oct 2019 at 18:25, Andrija Panic <andrija.pa...@gmail.com> > > wrote: > > > > > Can you share logs during this last attempt? > > > > > > I'll try to craft some SQL additions for your vm_template and > > > template_store_ref tables while also manually downloading and unpacking > > the > > > file to specific folder on Secondary Storage... > > > > > > Andrija > > > > > > On Thu, 3 Oct 2019 at 16:55, Fariborz Navidan <mdvlinqu...@gmail.com> > > > wrote: > > > > > >> My cloud is down for 2 days. I would really appreciate if you can help > > me > > >> fix the issue. > > >> > > >> On Thu, Oct 3, 2019 at 4:44 PM Fariborz Navidan < > mdvlinqu...@gmail.com> > > >> wrote: > > >> > > >> > Any idea? > > >> > > > >> > On Thu, Oct 3, 2019 at 2:09 PM Fariborz Navidan < > > mdvlinqu...@gmail.com> > > >> > wrote: > > >> > > > >> >> I did the steps exactly but still same error. It first says cannot > > >> >> upgrade database with no further sql errors stated. Thn says cannot > > >> find > > >> >> System VM template for KVM. It look likes system vm template 4.11.3 > > is > > >> >> installed only for router! > > >> >> > > >> >> MariaDB [cloud]> select * from vm_template where name like > > 'SystemVM%' > > >> >> and removed is null; > > >> >> > > >> >> > > >> > > > +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+ > > >> >> | id | unique_name | name | uuid > > >> >> | public | featured | type | hvm | bits | url > > >> >> > > | > > >> >> format | created | removed | account_id | checksum > > >> >> | display_text | enable_password | > > >> >> enable_sshkey | guest_os_id | bootable | prepopulate | cross_zones > | > > >> >> extractable | hypervisor_type | source_template_id | template_tag | > > >> >> sort_key | size | state | update_count | updated | > > >> >> dynamically_scalable | parent_template_id | direct_download | > > >> >> > > >> >> > > >> > > > +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+ > > >> >> | 1 | routing-1 | SystemVM Template (XenServer) | > > >> >> c1ea2f68-ca75-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM | > > >> 0 | > > >> >> 64 | > > >> >> > > >> > > > https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-xen.vhd.bz2 > > >> >> | VHD | 2019-08-29 17:57:52 | NULL | 1 | > > >> >> 6e8b3ae84ca8145736d1d7d3f7546e65 | SystemVM Template (XenServer) | > > >> >> 0 | 0 | 184 | 1 | 0 | > > >> >> 1 | 0 | XenServer | NULL | NULL > > | > > >> >> 0 | NULL | Active | 0 | NULL | > > >> 0 | > > >> >> NULL | 0 | > > >> >> | 3 | routing-3 | SystemVM Template (KVM) | > > >> >> e4621347-39b8-427e-a55f-82cf51275b69 | 0 | 0 | SYSTEM | > > >> 0 | > > >> >> 64 | > > >> >> > > >> > > > http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2 > > >> >> | QCOW2 | 2019-08-29 17:57:52 | NULL | 1 | > > >> >> 6d12cc764cd7d64112d8c35d70923eb1 | SystemVM Template (KVM) | > > >> >> 0 | 0 | 15 | 1 | 0 | > > >> >> 1 | 0 | KVM | NULL | NULL > > | > > >> >> 0 | 385242624 | Active | 0 | NULL | > > >> 0 | > > >> >> NULL | 0 | > > >> >> | 8 | routing-8 | SystemVM Template (vSphere) | > > >> >> c1ea7e39-ca75-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM | > > >> 0 | > > >> >> 64 | > > >> >> > > >> > > > https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-vmware.ova > > >> >> | OVA | 2019-08-29 17:57:52 | NULL | 1 | > > >> >> e981f8cb951688efd93481913198c9cc | SystemVM Template (vSphere) | > > >> >> 0 | 0 | 99 | 1 | 0 | > > >> >> 1 | 0 | VMware | NULL | NULL > > | > > >> >> 0 | NULL | Active | 0 | NULL | > > >> 1 | > > >> >> NULL | 0 | > > >> >> | 9 | routing-9 | SystemVM Template (HyperV) | > > >> >> c1eaa7b0-ca75-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM | > > >> 0 | > > >> >> 64 | > > >> >> > > >> > > > https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-hyperv.vhd.zip > > >> >> | VHD | 2019-08-29 17:57:52 | NULL | 1 | > > >> >> e9032635ffba021371780307162551b9 | SystemVM Template (HyperV) | > > >> >> 0 | 0 | 15 | 1 | 0 | > > >> >> 1 | 0 | Hyperv | NULL | NULL > > | > > >> >> 0 | NULL | Active | 0 | NULL | > > >> 0 | > > >> >> NULL | 0 | > > >> >> | 10 | routing-10 | SystemVM Template (LXC) | > > >> >> 6706cad5-ca77-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM | > > >> 0 | > > >> >> 64 | > > >> >> > > >> > > > https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-kvm.qcow2.bz2 > > >> >> | QCOW2 | 2019-08-29 18:09:38 | NULL | 1 | > > >> >> 6d12cc764cd7d64112d8c35d70923eb1 | SystemVM Template (LXC) | > > >> >> 0 | 0 | 15 | 1 | 0 | > > >> >> 1 | 0 | LXC | NULL | NULL > > | > > >> >> 0 | NULL | Active | 0 | NULL | > > >> 0 | > > >> >> NULL | 0 | > > >> >> | 12 | routing-12 | SystemVM Template (Ovm3) | > > >> >> 694d5de8-ca77-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM | > > >> 0 | > > >> >> 64 | > > >> >> > > >> > > > https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-ovm.raw.bz2 > > >> >> | RAW | 2019-08-29 18:09:42 | NULL | 1 | > > >> >> c4a91f8e52e4531a1c2a9a17c530d5fe | SystemVM Template (Ovm3) | > > >> >> 0 | 0 | 183 | 1 | 0 | > > >> >> 1 | 0 | Ovm3 | NULL | NULL > > | > > >> >> 0 | NULL | Active | NULL | NULL | > > >> 0 | > > >> >> NULL | 0 | > > >> >> > > >> >> > > >> > > > +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+ > > >> >> 6 rows in set (0.00 sec) > > >> >> > > >> >> On Thu, Oct 3, 2019 at 1:52 AM Andrija Panic < > > andrija.pa...@gmail.com> > > >> >> wrote: > > >> >> > > >> >>> - ake sure mgmt is stopped (you can leave it on 4.13 if not > > >> >>> already downgraded), stop cloudstack-agents on KVM hosts. > > >> >>> - Rollback DB with your SQL statements, that you shared a few > emails > > >> ago > > >> >>> (unfortunately after this 4.12 is NOT possible to work > normally...) > > >> >>> -Download the new systemVM template - it WILL OVERWRITE your old > > one > > >> - > > >> >>> i.e.: > > >> >>> From mgmt, mount the Secondary Storage NFS on > /mnt/secondary > > >> and > > >> >>> execute: > > >> >>> > > >> >>> > > >> >>> > > >> > > > /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt > > >> >>> -m /mnt/secondary -u > > >> >>> > > >> >>> > > >> > > > http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2 > > >> >>> -h > > >> >>> < > > >> > > > http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2-h > > >> > > > >> >>> kvm -F > > >> >>> This will overwrite existing systemVM template on the > > >> >>> /mnt/secondary/templates/1/3/xxxxxx and will update the DB with > new > > >> names > > >> >>> etc... > > >> >>> - Upgrade packages to 4.13 now (if not already having 4.13 > > installed) > > >> >>> - start mgmt - upgrade path will find the new systemVM template > and > > >> mgmt > > >> >>> server should start fine. > > >> >>> - Destroy your SSVM and CPVM, new ones will be recreated from new > > >> >>> systemVM > > >> >>> template. > > >> >>> - Restart all VRs (or restart networks/VPCs with Clean Up option) > - > > >> new > > >> >>> will be created from the new systemVM template. > > >> >>> > > >> >>> I've just tested all this in my lab - let me know if any issues. > > >> >>> > > >> >>> > > >> >>> > > >> >>> On Wed, 2 Oct 2019 at 20:37, Fariborz Navidan < > > mdvlinqu...@gmail.com> > > >> >>> wrote: > > >> >>> > > >> >>> > I cannot rollback DB to 4.12. How can I download 4.11.3 and and > > >> >>> register in > > >> >>> > DB? > > >> >>> > > > >> >>> > On Wed, Oct 2, 2019 at 9:43 PM Andrija Panic < > > >> andrija.pa...@gmail.com> > > >> >>> > wrote: > > >> >>> > > > >> >>> > > There's a GitHub issue about this (perhaps you or some > > colleague?) > > >> >>> > > > > >> >>> > > Did you register 4.11.3 (not 4.13) systemVM template? > > >> Documentation > > >> >>> has > > >> >>> > > just been updated... Sorry for this. > > >> >>> > > > > >> >>> > > i..e. downgrade packages to 4.12, make sure your DB is rolled > > >> back, > > >> >>> start > > >> >>> > > 4.12 normally, register 4.11.3 systemVM template as per the > doc > > >> and > > >> >>> then > > >> >>> > > try again to upgrade. > > >> >>> > > > > >> >>> > > let me know how it goes. > > >> >>> > > > > >> >>> > > Andrija > > >> >>> > > > > >> >>> > > On Wed, 2 Oct 2019, 17:27 Fariborz Navidan, < > > >> mdvlinqu...@gmail.com> > > >> >>> > wrote: > > >> >>> > > > > >> >>> > > > Any idea? > > >> >>> > > > > > >> >>> > > > On Wed, Oct 2, 2019 at 3:59 PM Fariborz Navidan < > > >> >>> mdvlinqu...@gmail.com > > >> >>> > > > > >> >>> > > > wrote: > > >> >>> > > > > > >> >>> > > > > Yes, It was clean 4.12 installation. > > >> >>> > > > > > > >> >>> > > > > On Wed, Oct 2, 2019 at 3:52 PM Andrija Panic < > > >> >>> > andrija.pa...@gmail.com> > > >> >>> > > > > wrote: > > >> >>> > > > > > > >> >>> > > > >> "was this a clean 4.12 install, then later upgrade - or > was > > >> it > > >> >>> > > > previously > > >> >>> > > > >> 4.11.x then 4.12 etc?" > > >> >>> > > > >> > > >> >>> > > > >> On Wed, 2 Oct 2019 at 14:20, Fariborz Navidan < > > >> >>> > mdvlinqu...@gmail.com> > > >> >>> > > > >> wrote: > > >> >>> > > > >> > > >> >>> > > > >> > Which question? > > >> >>> > > > >> > > > >> >>> > > > >> > I have sent pastebin link. I lost 4.12 backup > > >> accidentally. I > > >> >>> am > > >> >>> > > > trying > > >> >>> > > > >> to > > >> >>> > > > >> > make 4.13 upgrade successfully. I do run following > > commands > > >> >>> before > > >> >>> > > > >> upgrade > > >> >>> > > > >> > retry: > > >> >>> > > > >> > > > >> >>> > > > >> > [root@fr-kvm1 ~]# cat rollback.sql > > >> >>> > > > >> > delete from guest_os_hypervisor where guest_os_id >= > 277; > > >> >>> > > > >> > delete from guest_os where id >= 277; > > >> >>> > > > >> > SET FOREIGN_KEY_CHECKS=0; > > >> >>> > > > >> > drop table direct_download_certificate; > > >> >>> > > > >> > drop table vpc_offering_details; > > >> >>> > > > >> > drop table direct_download_certificate_host_map; > > >> >>> > > > >> > drop table template_ovf_properties; > > >> >>> > > > >> > SET FOREIGN_KEY_CHECKS=1; > > >> >>> > > > >> > alter table data_center drop column sort_key; > > >> >>> > > > >> > alter table vpc_offerings drop column sort_key; > > >> >>> > > > >> > alter table disk_offering add column domain_id int(32) > > not > > >> >>> null; > > >> >>> > > > >> > alter table service_offering_details add constraint > > >> >>> > > > >> > uk_service_offering_id_name unique (id); > > >> >>> > > > >> > alter table network_offering_details DROP COLUMN > > `display`; > > >> >>> > > > >> > ALTER TABLE `cloud_usage`.`usage_vmsnapshot` DROP > COLUMN > > >> >>> > > > >> `vm_snapshot_id`; > > >> >>> > > > >> > ALTER TABLE `cloud_usage`.`usage_snapshot_on_primary` > > DROP > > >> >>> COLUMN > > >> >>> > > > >> > `vm_snapshot_id`; > > >> >>> > > > >> > [root@fr-kvm1 ~]# > > >> >>> > > > >> > > > >> >>> > > > >> > However I don't understand the error I showed you on > log > > in > > >> >>> > > pastebin. > > >> >>> > > > >> > Please check and let me know what does it mean. > > >> >>> > > > >> > > > >> >>> > > > >> > On Wed, Oct 2, 2019 at 3:27 PM Andrija Panic < > > >> >>> > > andrija.pa...@gmail.com > > >> >>> > > > > > > >> >>> > > > >> > wrote: > > >> >>> > > > >> > > > >> >>> > > > >> > > It was not designed to rollback on its own, you are > > >> >>> expected to > > >> >>> > > > >> rollback > > >> >>> > > > >> > DB > > >> >>> > > > >> > > yourself (DB restore) and try again if needed. > > >> >>> > > > >> > > > > >> >>> > > > >> > > But you have not answered my question. > > >> >>> > > > >> > > > > >> >>> > > > >> > > On Wed, 2 Oct 2019, 13:50 Fariborz Navidan, < > > >> >>> > > mdvlinqu...@gmail.com> > > >> >>> > > > >> > wrote: > > >> >>> > > > >> > > > > >> >>> > > > >> > > > In my experience, if DB upgrade fails, it does not > > >> fully > > >> >>> rolls > > >> >>> > > > back > > >> >>> > > > >> > > > remaining 4.13's schema upgraded partially and upon > > >> next > > >> >>> > restart > > >> >>> > > > it > > >> >>> > > > >> > does > > >> >>> > > > >> > > > not check existence of 4.13's table or columns > > causing > > >> SQL > > >> >>> > > errors > > >> >>> > > > >> > > > complaining about tables or columns already exists. > > >> >>> > > > >> > > > > > >> >>> > > > >> > > > On Wed, Oct 2, 2019 at 3:14 PM Andrija Panic < > > >> >>> > > > >> andrija.pa...@gmail.com> > > >> >>> > > > >> > > > wrote: > > >> >>> > > > >> > > > > > >> >>> > > > >> > > > > was this a clean 4.12 install, then later > upgrade - > > >> or > > >> >>> was > > >> >>> > it > > >> >>> > > > >> > > previously > > >> >>> > > > >> > > > > 4.11.x then 4.12 etc? > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > > > On Wed, 2 Oct 2019 at 12:21, Fariborz Navidan < > > >> >>> > > > >> mdvlinqu...@gmail.com > > >> >>> > > > >> > > > > >> >>> > > > >> > > > > wrote: > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > > > > https://pastebin.com/2EFm0jaW > > >> >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > On Wed, Oct 2, 2019 at 1:19 PM Andrija Panic < > > >> >>> > > > >> > > andrija.pa...@gmail.com> > > >> >>> > > > >> > > > > > wrote: > > >> >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > (backup, backup, backup...) > > >> >>> > > > >> > > > > > > > > >> >>> > > > >> > > > > > > Can you send the full logs (passtebin or > > similar > > >> >>> please, > > >> >>> > > not > > >> >>> > > > >> in > > >> >>> > > > >> > > > email) > > >> >>> > > > >> > > > > ? > > >> >>> > > > >> > > > > > > > > >> >>> > > > >> > > > > > > Andrija > > >> >>> > > > >> > > > > > > > > >> >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > > > -- > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > > > Andrija Panić > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > > > > >> >>> > > > >> > > > > >> >>> > > > >> > > > >> >>> > > > >> > > >> >>> > > > >> > > >> >>> > > > >> -- > > >> >>> > > > >> > > >> >>> > > > >> Andrija Panić > > >> >>> > > > >> > > >> >>> > > > > > > >> >>> > > > > > >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > > >> >>> -- > > >> >>> > > >> >>> Andrija Panić > > >> >>> > > >> >> > > >> > > > > > > > > > -- > > > > > > Andrija Panić > > > > > > -- Andrija Panić
