Cool! On Thu, 3 Oct 2019, 21:02 Fariborz Navidan, <mdvlinqu...@gmail.com> wrote:
> Thank you very much! Finally, I could start management server normally. > > Thanks. > > On Thu, Oct 3, 2019 at 9:26 PM Andrija Panic <andrija.pa...@gmail.com> > wrote: > > > It seems easier than I thought... > > > > stop mgmt server > > Rollback DB with your statement shared in some of the previous emails > (and > > after that please make DB backup immediately, just for the sake of having > > one...) > > Again, perform that manual template install with: > > - From mgmt, mount the Secondary Storage NFS on /mnt/secondary and > > execute: > > > > > > > /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt > > -m /mnt/secondary -u > > > > > http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2 > > -h > > < > http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2-h > > > > kvm -F > > umount /mnt/secondary > > > > Update some fields for that template: > > update vm_template set name="systemvm-kvm-4.11.3" where id="3"; > > update vm_template set display_text="systemvm-kvm-4.11.3" where > id="3"; > > update vm_template set type="USER" where id="3"; > > Check that ID=3 for default KVM system template - but should be identical > > on everyones env. > > > > CloudStack upgrade is looking for specific NAME of the template in order > to > > proceed with the upgrade (and initially type=USER, and the upgrade will > > change that again to SYSTEM) > > After those SQL changes, start mgmt server - all should be good. > > If mgmt is started fine, you will have to DESTROY all system VMs and VRs, > > i.e. NOT reboot - to force using new template - so Destroy SSVM and CPVM > > (will be recreated automatically) and for the network/VPC - restart them > > with "Clean Up" option. > > > > My previous instructions didn't work, since I was doing more things than > > what I've shared with you - sorry for that - but those steps are needed > > anyway, just were incomplete. > > > > (We could do different fix by manually inserting DB records for a > > completely new template, new rows in vm_template, templat_store_ref and > > template_zone_ref and unpack that BZ2 file/template to specific folder on > > Secondary Storage - but that would require a bit of MySQL and is more > > error-prone. This kind of fix would simulate the step of manually > > registering new systemVM template as regular user template, with the > > specific name - which would be later recognized by it's name as the new > > systemVM templat for 4.11.3) > > > > Let me know how it goes. > > Andrija > > > > On Thu, 3 Oct 2019 at 19:07, Fariborz Navidan <mdvlinqu...@gmail.com> > > wrote: > > > > > MariaDB [cloud]> select max(id) from vm_template; > > > +---------+ > > > | max(id) | > > > +---------+ > > > | 390 | > > > +---------+ > > > > > > Can we have a chat session to investigate together? > > > > > > On Thu, Oct 3, 2019 at 8:27 PM Andrija Panic <andrija.pa...@gmail.com> > > > wrote: > > > > > > > do for me: > > > > > > > > select max(id) from vm_template: > > > > > > > > i might need more info later... > > > > > > > > On Thu, 3 Oct 2019 at 18:25, Andrija Panic <andrija.pa...@gmail.com> > > > > wrote: > > > > > > > > > Can you share logs during this last attempt? > > > > > > > > > > I'll try to craft some SQL additions for your vm_template and > > > > > template_store_ref tables while also manually downloading and > > unpacking > > > > the > > > > > file to specific folder on Secondary Storage... > > > > > > > > > > Andrija > > > > > > > > > > On Thu, 3 Oct 2019 at 16:55, Fariborz Navidan < > mdvlinqu...@gmail.com > > > > > > > > wrote: > > > > > > > > > >> My cloud is down for 2 days. I would really appreciate if you can > > help > > > > me > > > > >> fix the issue. > > > > >> > > > > >> On Thu, Oct 3, 2019 at 4:44 PM Fariborz Navidan < > > > mdvlinqu...@gmail.com> > > > > >> wrote: > > > > >> > > > > >> > Any idea? > > > > >> > > > > > >> > On Thu, Oct 3, 2019 at 2:09 PM Fariborz Navidan < > > > > mdvlinqu...@gmail.com> > > > > >> > wrote: > > > > >> > > > > > >> >> I did the steps exactly but still same error. It first says > > cannot > > > > >> >> upgrade database with no further sql errors stated. Thn says > > cannot > > > > >> find > > > > >> >> System VM template for KVM. It look likes system vm template > > 4.11.3 > > > > is > > > > >> >> installed only for router! > > > > >> >> > > > > >> >> MariaDB [cloud]> select * from vm_template where name like > > > > 'SystemVM%' > > > > >> >> and removed is null; > > > > >> >> > > > > >> >> > > > > >> > > > > > > > > > > +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+ > > > > >> >> | id | unique_name | name | uuid > > > > >> >> | public | featured | type | hvm | bits | url > > > > >> >> > > > > | > > > > >> >> format | created | removed | account_id | checksum > > > > >> >> | display_text | enable_password | > > > > >> >> enable_sshkey | guest_os_id | bootable | prepopulate | > > cross_zones > > > | > > > > >> >> extractable | hypervisor_type | source_template_id | > > template_tag | > > > > >> >> sort_key | size | state | update_count | updated | > > > > >> >> dynamically_scalable | parent_template_id | direct_download | > > > > >> >> > > > > >> >> > > > > >> > > > > > > > > > > +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+ > > > > >> >> | 1 | routing-1 | SystemVM Template (XenServer) | > > > > >> >> c1ea2f68-ca75-11e9-8212-d05099d38681 | 0 | 0 | > > SYSTEM | > > > > >> 0 | > > > > >> >> 64 | > > > > >> >> > > > > >> > > > > > > > > > > https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-xen.vhd.bz2 > > > > >> >> | VHD | 2019-08-29 17:57:52 | NULL | 1 | > > > > >> >> 6e8b3ae84ca8145736d1d7d3f7546e65 | SystemVM Template > (XenServer) > > | > > > > >> >> 0 | 0 | 184 | 1 | 0 > | > > > > >> >> 1 | 0 | XenServer | NULL | NULL > > > > | > > > > >> >> 0 | NULL | Active | 0 | NULL | > > > > >> 0 | > > > > >> >> NULL | 0 | > > > > >> >> | 3 | routing-3 | SystemVM Template (KVM) | > > > > >> >> e4621347-39b8-427e-a55f-82cf51275b69 | 0 | 0 | > > SYSTEM | > > > > >> 0 | > > > > >> >> 64 | > > > > >> >> > > > > >> > > > > > > > > > > http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2 > > > > >> >> | QCOW2 | 2019-08-29 17:57:52 | NULL | 1 | > > > > >> >> 6d12cc764cd7d64112d8c35d70923eb1 | SystemVM Template (KVM) > > | > > > > >> >> 0 | 0 | 15 | 1 | 0 > | > > > > >> >> 1 | 0 | KVM | NULL | NULL > > > > | > > > > >> >> 0 | 385242624 | Active | 0 | NULL | > > > > >> 0 | > > > > >> >> NULL | 0 | > > > > >> >> | 8 | routing-8 | SystemVM Template (vSphere) | > > > > >> >> c1ea7e39-ca75-11e9-8212-d05099d38681 | 0 | 0 | > > SYSTEM | > > > > >> 0 | > > > > >> >> 64 | > > > > >> >> > > > > >> > > > > > > > > > > https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-vmware.ova > > > > >> >> | OVA | 2019-08-29 17:57:52 | NULL | 1 | > > > > >> >> e981f8cb951688efd93481913198c9cc | SystemVM Template (vSphere) > > | > > > > >> >> 0 | 0 | 99 | 1 | 0 > | > > > > >> >> 1 | 0 | VMware | NULL | NULL > > > > | > > > > >> >> 0 | NULL | Active | 0 | NULL | > > > > >> 1 | > > > > >> >> NULL | 0 | > > > > >> >> | 9 | routing-9 | SystemVM Template (HyperV) | > > > > >> >> c1eaa7b0-ca75-11e9-8212-d05099d38681 | 0 | 0 | > > SYSTEM | > > > > >> 0 | > > > > >> >> 64 | > > > > >> >> > > > > >> > > > > > > > > > > https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-hyperv.vhd.zip > > > > >> >> | VHD | 2019-08-29 17:57:52 | NULL | 1 | > > > > >> >> e9032635ffba021371780307162551b9 | SystemVM Template (HyperV) > > | > > > > >> >> 0 | 0 | 15 | 1 | 0 > | > > > > >> >> 1 | 0 | Hyperv | NULL | NULL > > > > | > > > > >> >> 0 | NULL | Active | 0 | NULL | > > > > >> 0 | > > > > >> >> NULL | 0 | > > > > >> >> | 10 | routing-10 | SystemVM Template (LXC) | > > > > >> >> 6706cad5-ca77-11e9-8212-d05099d38681 | 0 | 0 | > > SYSTEM | > > > > >> 0 | > > > > >> >> 64 | > > > > >> >> > > > > >> > > > > > > > > > > https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-kvm.qcow2.bz2 > > > > >> >> | QCOW2 | 2019-08-29 18:09:38 | NULL | 1 | > > > > >> >> 6d12cc764cd7d64112d8c35d70923eb1 | SystemVM Template (LXC) > > | > > > > >> >> 0 | 0 | 15 | 1 | 0 > | > > > > >> >> 1 | 0 | LXC | NULL | NULL > > > > | > > > > >> >> 0 | NULL | Active | 0 | NULL | > > > > >> 0 | > > > > >> >> NULL | 0 | > > > > >> >> | 12 | routing-12 | SystemVM Template (Ovm3) | > > > > >> >> 694d5de8-ca77-11e9-8212-d05099d38681 | 0 | 0 | > > SYSTEM | > > > > >> 0 | > > > > >> >> 64 | > > > > >> >> > > > > >> > > > > > > > > > > https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-ovm.raw.bz2 > > > > >> >> | RAW | 2019-08-29 18:09:42 | NULL | 1 | > > > > >> >> c4a91f8e52e4531a1c2a9a17c530d5fe | SystemVM Template (Ovm3) > > | > > > > >> >> 0 | 0 | 183 | 1 | 0 > | > > > > >> >> 1 | 0 | Ovm3 | NULL | NULL > > > > | > > > > >> >> 0 | NULL | Active | NULL | NULL | > > > > >> 0 | > > > > >> >> NULL | 0 | > > > > >> >> > > > > >> >> > > > > >> > > > > > > > > > > +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+ > > > > >> >> 6 rows in set (0.00 sec) > > > > >> >> > > > > >> >> On Thu, Oct 3, 2019 at 1:52 AM Andrija Panic < > > > > andrija.pa...@gmail.com> > > > > >> >> wrote: > > > > >> >> > > > > >> >>> - ake sure mgmt is stopped (you can leave it on 4.13 if not > > > > >> >>> already downgraded), stop cloudstack-agents on KVM hosts. > > > > >> >>> - Rollback DB with your SQL statements, that you shared a few > > > emails > > > > >> ago > > > > >> >>> (unfortunately after this 4.12 is NOT possible to work > > > normally...) > > > > >> >>> -Download the new systemVM template - it WILL OVERWRITE your > > old > > > > one > > > > >> - > > > > >> >>> i.e.: > > > > >> >>> From mgmt, mount the Secondary Storage NFS on > > > /mnt/secondary > > > > >> and > > > > >> >>> execute: > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt > > > > >> >>> -m /mnt/secondary -u > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2 > > > > >> >>> -h > > > > >> >>> < > > > > >> > > > > > > > > > > http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2-h > > > > >> > > > > > >> >>> kvm -F > > > > >> >>> This will overwrite existing systemVM template on the > > > > >> >>> /mnt/secondary/templates/1/3/xxxxxx and will update the DB > with > > > new > > > > >> names > > > > >> >>> etc... > > > > >> >>> - Upgrade packages to 4.13 now (if not already having 4.13 > > > > installed) > > > > >> >>> - start mgmt - upgrade path will find the new systemVM > template > > > and > > > > >> mgmt > > > > >> >>> server should start fine. > > > > >> >>> - Destroy your SSVM and CPVM, new ones will be recreated from > > new > > > > >> >>> systemVM > > > > >> >>> template. > > > > >> >>> - Restart all VRs (or restart networks/VPCs with Clean Up > > option) > > > - > > > > >> new > > > > >> >>> will be created from the new systemVM template. > > > > >> >>> > > > > >> >>> I've just tested all this in my lab - let me know if any > issues. > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >> >>> On Wed, 2 Oct 2019 at 20:37, Fariborz Navidan < > > > > mdvlinqu...@gmail.com> > > > > >> >>> wrote: > > > > >> >>> > > > > >> >>> > I cannot rollback DB to 4.12. How can I download 4.11.3 and > > and > > > > >> >>> register in > > > > >> >>> > DB? > > > > >> >>> > > > > > >> >>> > On Wed, Oct 2, 2019 at 9:43 PM Andrija Panic < > > > > >> andrija.pa...@gmail.com> > > > > >> >>> > wrote: > > > > >> >>> > > > > > >> >>> > > There's a GitHub issue about this (perhaps you or some > > > > colleague?) > > > > >> >>> > > > > > > >> >>> > > Did you register 4.11.3 (not 4.13) systemVM template? > > > > >> Documentation > > > > >> >>> has > > > > >> >>> > > just been updated... Sorry for this. > > > > >> >>> > > > > > > >> >>> > > i..e. downgrade packages to 4.12, make sure your DB is > > rolled > > > > >> back, > > > > >> >>> start > > > > >> >>> > > 4.12 normally, register 4.11.3 systemVM template as per > the > > > doc > > > > >> and > > > > >> >>> then > > > > >> >>> > > try again to upgrade. > > > > >> >>> > > > > > > >> >>> > > let me know how it goes. > > > > >> >>> > > > > > > >> >>> > > Andrija > > > > >> >>> > > > > > > >> >>> > > On Wed, 2 Oct 2019, 17:27 Fariborz Navidan, < > > > > >> mdvlinqu...@gmail.com> > > > > >> >>> > wrote: > > > > >> >>> > > > > > > >> >>> > > > Any idea? > > > > >> >>> > > > > > > > >> >>> > > > On Wed, Oct 2, 2019 at 3:59 PM Fariborz Navidan < > > > > >> >>> mdvlinqu...@gmail.com > > > > >> >>> > > > > > > >> >>> > > > wrote: > > > > >> >>> > > > > > > > >> >>> > > > > Yes, It was clean 4.12 installation. > > > > >> >>> > > > > > > > > >> >>> > > > > On Wed, Oct 2, 2019 at 3:52 PM Andrija Panic < > > > > >> >>> > andrija.pa...@gmail.com> > > > > >> >>> > > > > wrote: > > > > >> >>> > > > > > > > > >> >>> > > > >> "was this a clean 4.12 install, then later upgrade - > or > > > was > > > > >> it > > > > >> >>> > > > previously > > > > >> >>> > > > >> 4.11.x then 4.12 etc?" > > > > >> >>> > > > >> > > > > >> >>> > > > >> On Wed, 2 Oct 2019 at 14:20, Fariborz Navidan < > > > > >> >>> > mdvlinqu...@gmail.com> > > > > >> >>> > > > >> wrote: > > > > >> >>> > > > >> > > > > >> >>> > > > >> > Which question? > > > > >> >>> > > > >> > > > > > >> >>> > > > >> > I have sent pastebin link. I lost 4.12 backup > > > > >> accidentally. I > > > > >> >>> am > > > > >> >>> > > > trying > > > > >> >>> > > > >> to > > > > >> >>> > > > >> > make 4.13 upgrade successfully. I do run following > > > > commands > > > > >> >>> before > > > > >> >>> > > > >> upgrade > > > > >> >>> > > > >> > retry: > > > > >> >>> > > > >> > > > > > >> >>> > > > >> > [root@fr-kvm1 ~]# cat rollback.sql > > > > >> >>> > > > >> > delete from guest_os_hypervisor where guest_os_id > >= > > > 277; > > > > >> >>> > > > >> > delete from guest_os where id >= 277; > > > > >> >>> > > > >> > SET FOREIGN_KEY_CHECKS=0; > > > > >> >>> > > > >> > drop table direct_download_certificate; > > > > >> >>> > > > >> > drop table vpc_offering_details; > > > > >> >>> > > > >> > drop table direct_download_certificate_host_map; > > > > >> >>> > > > >> > drop table template_ovf_properties; > > > > >> >>> > > > >> > SET FOREIGN_KEY_CHECKS=1; > > > > >> >>> > > > >> > alter table data_center drop column sort_key; > > > > >> >>> > > > >> > alter table vpc_offerings drop column sort_key; > > > > >> >>> > > > >> > alter table disk_offering add column domain_id > > int(32) > > > > not > > > > >> >>> null; > > > > >> >>> > > > >> > alter table service_offering_details add constraint > > > > >> >>> > > > >> > uk_service_offering_id_name unique (id); > > > > >> >>> > > > >> > alter table network_offering_details DROP COLUMN > > > > `display`; > > > > >> >>> > > > >> > ALTER TABLE `cloud_usage`.`usage_vmsnapshot` DROP > > > COLUMN > > > > >> >>> > > > >> `vm_snapshot_id`; > > > > >> >>> > > > >> > ALTER TABLE > `cloud_usage`.`usage_snapshot_on_primary` > > > > DROP > > > > >> >>> COLUMN > > > > >> >>> > > > >> > `vm_snapshot_id`; > > > > >> >>> > > > >> > [root@fr-kvm1 ~]# > > > > >> >>> > > > >> > > > > > >> >>> > > > >> > However I don't understand the error I showed you > on > > > log > > > > in > > > > >> >>> > > pastebin. > > > > >> >>> > > > >> > Please check and let me know what does it mean. > > > > >> >>> > > > >> > > > > > >> >>> > > > >> > On Wed, Oct 2, 2019 at 3:27 PM Andrija Panic < > > > > >> >>> > > andrija.pa...@gmail.com > > > > >> >>> > > > > > > > > >> >>> > > > >> > wrote: > > > > >> >>> > > > >> > > > > > >> >>> > > > >> > > It was not designed to rollback on its own, you > are > > > > >> >>> expected to > > > > >> >>> > > > >> rollback > > > > >> >>> > > > >> > DB > > > > >> >>> > > > >> > > yourself (DB restore) and try again if needed. > > > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > But you have not answered my question. > > > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > On Wed, 2 Oct 2019, 13:50 Fariborz Navidan, < > > > > >> >>> > > mdvlinqu...@gmail.com> > > > > >> >>> > > > >> > wrote: > > > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > > In my experience, if DB upgrade fails, it does > > not > > > > >> fully > > > > >> >>> rolls > > > > >> >>> > > > back > > > > >> >>> > > > >> > > > remaining 4.13's schema upgraded partially and > > upon > > > > >> next > > > > >> >>> > restart > > > > >> >>> > > > it > > > > >> >>> > > > >> > does > > > > >> >>> > > > >> > > > not check existence of 4.13's table or columns > > > > causing > > > > >> SQL > > > > >> >>> > > errors > > > > >> >>> > > > >> > > > complaining about tables or columns already > > exists. > > > > >> >>> > > > >> > > > > > > > >> >>> > > > >> > > > On Wed, Oct 2, 2019 at 3:14 PM Andrija Panic < > > > > >> >>> > > > >> andrija.pa...@gmail.com> > > > > >> >>> > > > >> > > > wrote: > > > > >> >>> > > > >> > > > > > > > >> >>> > > > >> > > > > was this a clean 4.12 install, then later > > > upgrade - > > > > >> or > > > > >> >>> was > > > > >> >>> > it > > > > >> >>> > > > >> > > previously > > > > >> >>> > > > >> > > > > 4.11.x then 4.12 etc? > > > > >> >>> > > > >> > > > > > > > > >> >>> > > > >> > > > > On Wed, 2 Oct 2019 at 12:21, Fariborz > Navidan < > > > > >> >>> > > > >> mdvlinqu...@gmail.com > > > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > > > wrote: > > > > >> >>> > > > >> > > > > > > > > >> >>> > > > >> > > > > > https://pastebin.com/2EFm0jaW > > > > >> >>> > > > >> > > > > > > > > > >> >>> > > > >> > > > > > On Wed, Oct 2, 2019 at 1:19 PM Andrija > Panic > > < > > > > >> >>> > > > >> > > andrija.pa...@gmail.com> > > > > >> >>> > > > >> > > > > > wrote: > > > > >> >>> > > > >> > > > > > > > > > >> >>> > > > >> > > > > > > (backup, backup, backup...) > > > > >> >>> > > > >> > > > > > > > > > > >> >>> > > > >> > > > > > > Can you send the full logs (passtebin or > > > > similar > > > > >> >>> please, > > > > >> >>> > > not > > > > >> >>> > > > >> in > > > > >> >>> > > > >> > > > email) > > > > >> >>> > > > >> > > > > ? > > > > >> >>> > > > >> > > > > > > > > > > >> >>> > > > >> > > > > > > Andrija > > > > >> >>> > > > >> > > > > > > > > > > >> >>> > > > >> > > > > > > > > > >> >>> > > > >> > > > > > > > > >> >>> > > > >> > > > > > > > > >> >>> > > > >> > > > > -- > > > > >> >>> > > > >> > > > > > > > > >> >>> > > > >> > > > > Andrija Panić > > > > >> >>> > > > >> > > > > > > > > >> >>> > > > >> > > > > > > > >> >>> > > > >> > > > > > > >> >>> > > > >> > > > > > >> >>> > > > >> > > > > >> >>> > > > >> > > > > >> >>> > > > >> -- > > > > >> >>> > > > >> > > > > >> >>> > > > >> Andrija Panić > > > > >> >>> > > > >> > > > > >> >>> > > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > >> >>> > > > > > >> >>> > > > > >> >>> > > > > >> >>> -- > > > > >> >>> > > > > >> >>> Andrija Panić > > > > >> >>> > > > > >> >> > > > > >> > > > > > > > > > > > > > > > -- > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > -- > > > > Andrija Panić > > >
