On Thu, Dec 20, 2018 at 8:23 PM Torsten Stolpmann <[email protected]> wrote: > > On 20.12.2018 07:41, Yedidyah Bar David wrote: > > On Wed, Dec 19, 2018 at 1:35 PM Torsten Stolpmann > > <[email protected]> wrote: > >> > >> On 19.12.2018 11:54, Yedidyah Bar David wrote: > >>> On Wed, Dec 19, 2018 at 12:34 PM Torsten Stolpmann > >>> <[email protected]> wrote: > >>>> > >>>> On 19.12.2018 08:01, Yedidyah Bar David wrote: > >>>>> On Tue, Dec 18, 2018 at 5:20 PM Torsten Stolpmann > >>>>> <[email protected]> wrote: > >>>>>> > >>>>>> Hi Yedidyah, > >>>>>> > >>>>>> please find the logs at the following URL: > >>>>>> http://www.klaros-testmanagement.com/files/ovirt/ovirt-restore-logs.tar.gz > >>>>>> > >>>>>> Let me know once you received them safely so I can remove them again. > >>>>> > >>>>> Done. > >>>>> > >>>> > >>>> Thanks, removed. > >>>> > >>>>>> > >>>>>> I also added the restore.log containing the actual error which occured > >>>>>> during the restore. > >>>>>> > >>>>>> Since this was a clean system I restored on, the setup has been > >>>>>> executed > >>>>>> after the database restore, so the setup logs probably contain nothing > >>>>>> of interest. I added them anyway. > >>>>> > >>>>> Sorry I wasn't clear enough. I meant the setup logs on the machine used > >>>>> to create the backup. So that I can try to see why your backup contained > >>>>> this function. Do you still have these by any chance? > >>>>> > >>>>> Indeed, I can't see anything wrong in the logs above. > >>>>> > >>>> Sorry for misunderstanding, this totally makes sense. > >>>> > >>>> I added the setup logs i found at: > >>>> http://www.klaros-testmanagement.com/files/ovirt/ovirt-setup-logs.tar.gz > >>>> > >>>> Again, please let me know once I can remove them. > >>> > >>> Done. > >>> > >> Thanks, removed. > >> > >>>> > >>>>>> > >>>>>> Please let me know if there is anything I can add to this. > >>>>> > >>>>> If you do not have setup logs from the original machine, at least try > >>>>> to think about its history and tell us notable points - including entire > >>>>> version history, setup/upgrade (or similar) problems you had there (and > >>>>> perhaps worked around), etc. > >>>>> > >>>> > >>>> We started with one of the first 4.0 releases and updated the system on > >>>> a regular basis since then. We almost never skipped a release. > >>> > >>> The last log there is ovirt-engine-setup-20171231170651-lb3q89.log , > >>> meaning it was last upgraded almost a year ago. Were there indeed no > >>> more upgrades since? > >>> > >> > >> 20171231170651 is most probably the date when we installed the last > >> major release (4.2.0). We did a lot more updates since and I now have a > >> suspicion what went wrong here. > >> > >> We naively assumed that a yum update is sufficient for a minor upgrade > >> of the engine installation. > > > > :-( > > > >> Rereading the documentation I think we were > >> missing explicit engine-setup calls after each minor upgrade. > > > > Indeed > > > >> > >> It may be the case, that the extra call to engine-setup is required to > >> not be part of the yum update. > > > > engine-setup might need to ask the user questions, so we do not run it > > inside yum update. > > > As long as it clear to users that this is required this is totally ok > for me. > > >> I think in this case it would be a good > >> idea to warn users that this step has not yet been taken and the update > >> is not completed yet. > > > > Do you think you would have noticed? It's not that hard to add such a > > message during yum update, but not sure it's that helpful. > > > Yes, I would probably have noticed that. Sooner or later ... :)
OK, now finished verifying this patch in CI: https://gerrit.ovirt.org/96446 https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/3834/ https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/3834/artifact/exported-artifacts/test_logs/upgrade-from-release-suite-master/post-001_upgrade_engine.py/lago_logs/lago.log Total output of: 2018-12-27 06:58:03,392::ssh.py::ssh::58::lago.ssh::DEBUG::Running c14e276c on lago-upgrade-from-release-suite-master-engine: yum -y update ovirt-*setup* Is 220 lines. Among these, line 134-136 are: Updating : ovirt-engine-setup-4.3.0-0.4.master.20181226121322.gitfe 17/33 Updating ovirt-engine-setup. To update the engine, you need to run: engine-setup Updating : ovirt-engine-setup-plugin-websocket-proxy-4.3.0-0.4.mast 18/33 and lines 151-153 are: Erasing : ovirt-engine-lib-4.2.8.2-0.0.master.20181220151741.git31 33/33 Updated ovirt-engine-setup. To update the engine, you need to run: engine-setup Verifying : ovirt-engine-dwh-setup-4.3.0-0.4.master.20181210085817.e 1/33 I looked at yum's sources, and do not see a way to make it output something closer to the end. Comments are welcome (here or on gerrit). > > > Perhaps we can also make the engine check e.g. if the setup package is > > newer than itself, and warn the user in the admin ui. > > > I think that both is a good idea. Some applications even go the way to > lock out users from the GUI until a necessary database migration has > been executed by an administrator. Your mileage may vary here but this > is the solution I would favor. Also filed (but am not going to try working on, it's not my expertise): https://bugzilla.redhat.com/show_bug.cgi?id=1662109 Best regards, > > >> > >> Could this be the cause for the missing logs and behavior discrepancies? > >> > >> If yes, would another call to engine-setup in the current state fix this > >> and allow us to produce correct backups in the future? > > > > If you still have the source machine, then yes, it should be enough. > > Run engine-setup, then engine-backup again to backup, then restore > > on the new machine. > > > Thanks, will do. > > >> > >>> The log indicates the machine was upgraded to ovirt-engine-4.2.0.2 , > >>> which didn't remove the uuid functions. The bug I linked at before, > >>> 1515635, was fixed in 4.2.1. > >>> > >>> Based on this, I think the best solution for now would be the patch > >>> you suggested. Would you like to open a bug for this and push a patch > >>> to gerrit? I can do this as well if you prefer. Bug summary line > >>> should probably be something like: > >>> > >>> engine-setup fails after restoring a backup taken with 4.2.0 > >>> > >> I currently fear that would only cure the symptom. > >> > >>> Another solution would be to try using the same engine version during > >>> restore (4.2.0.2), and only upgrade later. This is a bit hard, because > >>> we do not have separate repos for each version, although they do > >>> include all released versions - so you can try e.g.: > >>> > >>> yum install ovirt-engine-4.2.0.2-1.el7 > >>> > >>> (meaning, after you remove existing 4.2.7, or you can try yum downgrade). > >>> > >>> I didn't try this myself, not sure how well it would work. There might > >>> be older dependencies to handle, and/or it might break due to too-new > >>> stuff. > >>> > >> I don't think this is necessary. > >> > >>> Obviously, the best solution is to upgrade more often and backup more > >>> often, and have a smaller difference between backup version and restore > >>> version, ideally no difference. But I realize this does not always > >>> work out for everyone... > >>> > >> You are right, we did this. > >> > > > > OK. I assume this discussion is theoretical, right? > > That you already patched engine-backup manually and restored. > > Yes it is. > > Thanks a lot for your patience and support. > > Torsten > > > > > Best regards, > > > >> Best regards > >> > >> Torsten > >>>> > >>>> There was indeed one glitch during updates which was connected to the > >>>> Postgres version in use: > >>>> > >>>> The initial install was (accidentally) done under Postgres 9.4 which > >>>> happened to be present on the machine at that point of time. oVirt 4.0 > >>>> was allowing this way back then. I think in 4.1 Postgres 9.2 has been > >>>> enforced, so there had been workarounds to allow updates under 9.4. This > >>>> got resolved with the migration to 4.2 which brought the Postgres 9.5 > >>>> installation (if I recall that correctly). > >>>> > >>>> Other than that I have no idea which might have introduced this. > >>> > >>> All of this sounds irrelevant to current problem. Thanks for recalling :-) > >>> > >>> Best regards, > >>> > >>>> > >>>> If you find something in the logs you need more information on, please > >>>> let me know. > >>>> > >>>> Best regards > >>>> > >>>> Torsten > >>>> > >>>> > >>>>> If you, or anyone, manages to come up with a flow resulting in a 4.2 > >>>>> engine database that contains a function uuid_generate_v1 in the engine > >>>>> database schema, I'd definitely want to know about it. > >>>>> > >>>>> Re-adding others posting in this thread, in case someone has a clue. > >>>>> Perhaps one of you has setup logs of the machine on which the backup > >>>>> was generated? If so, please share. Thanks. > >>>>> > >>>>> That said, on a second thought, the implications of this are not that > >>>>> significant - mainly somewhat lower performance - so perhaps it's more > >>>>> important to fix your bug (by adding a line to IGNORED_ERRORS, as you > >>>>> suggested)... but it's still weird. > >>>>> > >>>>> Thanks and best regards, > >>>>> > >>>>>> > >>>>>> Cheers, > >>>>>> > >>>>>> Torsten > >>>>>> > >>>>>> > >>>>>> On 18.12.2018 07:54, Yedidyah Bar David wrote: > >>>>>>> On Mon, Dec 17, 2018 at 2:26 PM Torsten Stolpmann > >>>>>>> <[email protected]> wrote: > >>>>>>>> > >>>>>>>> I experienced the same issue while restoring a full backup (engine & > >>>>>>>> dwh) on a clean machine. Both machines are running CentOS 7.6 and > >>>>>>>> oVirt 4.2.7. > >>>>>>>> > >>>>>>>> The issue went away when adding the following line to the > >>>>>>>> IGNORED_ERRORS > >>>>>>>> list starting at 1944 in engine-backup: > >>>>>>>> > >>>>>>>> must be owner of function uuid_generate_v1 > >>>>>>>> > >>>>>>>> Hope this helps, > >>>>>>> > >>>>>>> It does, and thanks for the report! > >>>>>>> > >>>>>>> Can you please share all of your setup logs > >>>>>>> (/var/log/ovirt-engine/setup/*)? > >>>>>>> > >>>>>>> Thanks! > >>>>>>> > >>>>>>> More details: > >>>>>>> > >>>>>>> This function should not normally be included in a 4.2 backup, and I > >>>>>>> do > >>>>>>> not yet understand why you get this error. > >>>>>>> > >>>>>>> If it's a clean 4.2 setup, it should never have been defined. > >>>>>>> Earlier versions did create it, and the upgrade process should have > >>>>>>> dropped it, if it's an upgraded setup. See also: > >>>>>>> > >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1515635 > >>>>>>> > >>>>>>> Best regards, > >>>>>>> > >>>>>>>> > >>>>>>>> Torsten > >>>>>>>> > >>>>>>>> On 24.05.2018 11:32, [email protected] wrote: > >>>>>>>>> hi, > >>>>>>>>> > >>>>>>>>> I have the same error when I try to restore on a new hosted VM > >>>>>>>>> (ovirt 4.2.3) > >>>>>>>>> Have you a solution ? > >>>>>>>>> > >>>>>>>>> Emmanuel > >>>>>>>>> _______________________________________________ > >>>>>>>>> Users mailing list -- [email protected] > >>>>>>>>> To unsubscribe send an email to [email protected] > >>>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> Users mailing list -- [email protected] > >>>>>>>> To unsubscribe send an email to [email protected] > >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > >>>>>>>> oVirt Code of Conduct: > >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ > >>>>>>>> List Archives: > >>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/YMMFCYUMLCM47OJSTHXS2IMOIKEEFU27/ > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> _______________________________________________ > >>>>>> Users mailing list -- [email protected] > >>>>>> To unsubscribe send an email to [email protected] > >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > >>>>>> oVirt Code of Conduct: > >>>>>> https://www.ovirt.org/community/about/community-guidelines/ > >>>>>> List Archives: > >>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/FCN5WVAJOGFBUTFV276O2RMCTMACAGTS/ > >>>>> > >>>>> > >>>>> > >>> > >>> > >>> > > > > > > -- Didi _______________________________________________ Users mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/OVXKZY2LO53XYKROH2P5QAPGTQQ73DK4/
