----- Original Message ----- > From: "Jakub Bittner" <[email protected]> > To: [email protected] > Sent: Wednesday, August 14, 2013 8:09:45 AM > Subject: Re: [Users] All VMs disappeared > > Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a): > > Ahoj Jakub, > > > > Just one more idea: could you turn statement logging on in postgresql > > before you try again? > > It is in /var/lib/pgsql/data/postgresql.conf > > log_statement = 'all' > > > > > > ----- Original Message ----- > >> From: "Jakub Bittner" <[email protected]> > >> To: "Itamar Heim" <[email protected]> > >> Cc: [email protected] > >> Sent: Tuesday, August 13, 2013 5:20:29 PM > >> Subject: Re: [Users] All VMs disappeared > >> > >> Dne 13.8.2013 17:02, Itamar Heim napsal(a): > >>> On 08/13/2013 05:31 PM, Jakub Bittner wrote: > >>>> Dne 13.8.2013 15:39, Itamar Heim napsal(a): > >>>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: > >>>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): > >>>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: > >>>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): > >>>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): > >>>>>>>>>> I looked around Noam's patch and that should not cause such > >>>>>>>>>> behavior. > >>>>>>>>>> I am wondering how that lost VM's could happen. > >>>>>>>>>> > >>>>>>>>>> Jakub, can you give a more detailed description what you were > >>>>>>>>>> doing > >>>>>>>>>> with oVirt when this happened? Maybe the bug is still there. > >>>>>>>>>> > >>>>>>>>>> Thank you, > >>>>>>>>>> Laszlo > >>>>>>>>>> > >>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>> From: "Greg Sheremeta" <[email protected]> > >>>>>>>>>>> To: "Laszlo Hornyak" <[email protected]> > >>>>>>>>>>> Cc: "Jakub Bittner" <[email protected]>, "Noam Slomianko" > >>>>>>>>>>> <[email protected]>, [email protected] > >>>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM > >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared > >>>>>>>>>>> > >>>>>>>>>>> Not the one I fixed, 987907. It was a simple UI > >>>>>>>>>>> NullPointerException. > >>>>>>>>>>> > >>>>>>>>>>> Greg > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>> From: "Laszlo Hornyak" <[email protected]> > >>>>>>>>>>>> To: "Jakub Bittner" <[email protected]>, "Noam Slomianko" > >>>>>>>>>>>> <[email protected]>, "Greg Sheremeta" > >>>>>>>>>>>> <[email protected]> > >>>>>>>>>>>> Cc: [email protected] > >>>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM > >>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared > >>>>>>>>>>>> > >>>>>>>>>>>> Well if they are no longer in DB then that explains why the > >>>>>>>>>>>> exception no > >>>>>>>>>>>> longer occurs, but at the cost of database corruption. > >>>>>>>>>>>> Noam and Greg, can these bugs cause data corruption? > >>>>>>>>>>>> > >>>>>>>>>>>> Thank you, > >>>>>>>>>>>> Laszlo > >>>>>>>>>>>> > >>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>> From: "Jakub Bittner" <[email protected]> > >>>>>>>>>>>>> To: "Greg Sheremeta" <[email protected]> > >>>>>>>>>>>>> Cc: [email protected], "Laszlo Hornyak" <[email protected]> > >>>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM > >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared > >>>>>>>>>>>>> > >>>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): > >>>>>>>>>>>>>> It could also be this bug[1], for which I just submitted a > >>>>>>>>>>>>>> fix. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> You can work around it by typing just "VMs:" (without the > >>>>>>>>>>>>>> quotes) in > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>> search bar. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Greg > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>>> From: "Laszlo Hornyak" <[email protected]> > >>>>>>>>>>>>>>> To: "Jakub Bittner" <[email protected]> > >>>>>>>>>>>>>>> Cc: [email protected] > >>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM > >>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Jakub, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's are in > >>>>>>>>>>>>>>> your > >>>>>>>>>>>>>>> DB? > >>>>>>>>>>>>>>> select * from vm_static; > >>>>>>>>>>>>>>> or > >>>>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> It seems Noam fixed this issue already in > >>>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thank you, > >>>>>>>>>>>>>>> Laszlo > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>>>> From: "Jakub Bittner" <[email protected]> > >>>>>>>>>>>>>>>> To: [email protected] > >>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM > >>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): > >>>>>>>>>>>>>>>>> Hello, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs > >>>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. > >>>>>>>>>>>>>>>>> Repeating > >>>>>>>>>>>>>>>>> problem in log is: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN > >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] > >>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , > >>>>>>>>>>>>>>>>> vds = > >>>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = > >>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: > >>>>>>>>>>>>>>>>> java.lang.NullPointerException > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown > >>>>>>>>>>>>>>>>> Source) [:1.7.0_25] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) > >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [scheduler.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) > >>>>>>>>>>>>>>>>> [quartz.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [quartz.jar:] > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN > >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] > >>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , > >>>>>>>>>>>>>>>>> vds = > >>>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = > >>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: > >>>>>>>>>>>>>>>>> java.lang.NullPointerException > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [vdsbroker.jar:] > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown > >>>>>>>>>>>>>>>>> Source) [:1.7.0_25] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) > >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [scheduler.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) > >>>>>>>>>>>>>>>>> [quartz.jar:] > >>>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [quartz.jar:] > >>>>>>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>>>>>> Users mailing list > >>>>>>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>>>>>>>>>>>> I restarted management node and both nodes and this problem > >>>>>>>>>>>>>>>> disappeared. > >>>>>>>>>>>>>>>> But I still can not see any VMs. > >>>>>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>>>>> Users mailing list > >>>>>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>>>>> Users mailing list > >>>>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> Hi, > >>>>>>>>>>>>> > >>>>>>>>>>>>> I do not have them in database. Both commands shows nothing > >>>>>>>>>>>>> > >>>>>>>>> Hello, > >>>>>>>>> > >>>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great > >>>>>>>>> (except SPICE. VNC and creating new WM due to serialization bug.). > >>>>>>>>> Because those bugs were irritating, we switched to nightly. Still > >>>>>>>>> works fine (but bug with VMs is not fixed there), than I installed > >>>>>>>>> new > >>>>>>>>> ovirt engine management server in KVM virtualization (using > >>>>>>>>> libvirtd) > >>>>>>>>> due to taking snapshots before deploying nightly builds. New > >>>>>>>>> virtualized engine works great and I delete old one. Next day I > >>>>>>>>> came > >>>>>>>>> to work and engine web page was very very slow, every operation > >>>>>>>>> takes > >>>>>>>>> ages, that time I found there were no VMs, no HDDs and so. I > >>>>>>>>> converted > >>>>>>>>> its qcow2 image to raw, because I think it has slow DB because of > >>>>>>>>> running i qcow2. Did not helped, than I take a look in nodes > >>>>>>>>> logs and > >>>>>>>>> found this problem, which I posted in first email (VMs still > >>>>>>>>> running, > >>>>>>>>> but not visible in engine VMs tab, but on host I can see number of > >>>>>>>>> running VMs). After restart bug disappeared, but VMs too. > >>>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are > >>>>>>>>> gone. Even from storage. (but disappearing from storage may be my > >>>>>>>>> fault during recovery). > >>>>>>>>> > >>>>>>>>> I can provide logs from both two nodes and (hopefully) from engine > >>>>>>>>> too. I can provide you with those logs, but not publicly. Sorry. > >>>>>>>>> > >>>>>>>>> Thank you, > >>>>>>>>> Jakub Bittner, NSA Czech > >>>>>>>>> _______________________________________________ > >>>>>>>>> Users mailing list > >>>>>>>>> [email protected] > >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>>>> > >>>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone > >>>>>>>> again. > >>>>>>>> _______________________________________________ > >>>>>>>> Users mailing list > >>>>>>>> [email protected] > >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>>>> can you verify the db from before the upgrade has the VMs? > >>>>>> I dont know what you mean by verify database. Please, could you > >>>>>> point me > >>>>>> how to do it? Thank you > >>>>>> _______________________________________________ > >>>>>> Users mailing list > >>>>>> [email protected] > >>>>>> http://lists.ovirt.org/mailman/listinfo/users > >>>>> I assume from your expectation to see the VMs from the previous > >>>>> install that you upgraded your system rather than do a clean install? > >>>>> in an upgrade, there should be a backup of the db from before the > >>>>> upgrade. > >>>>> (in a clean install, engine is not aware of VMs running on hosts from > >>>>> previous installs - we need to cover "import existing storage domain" > >>>>> in a future version to close this gap. current alternatives around > >>>>> this are to convert the storage domain to an export domain (for an nfs > >>>>> data domain) an re-import them to the system or a few other options. > >>>>> > >>>>> how many VMs are there? > >>>> They(VMs) are deleted. So, my situation is this: > >>>> > >>>> I have clean install of ovirt 3.3.0beta with imported VMs from export > >>>> domain. Everything works except snapshots. But when I restart ovirt mgmt > >>>> server, VMs will disappear. Right now I am doing hourly backup of DB. > >>> this is very fishy... > >>> can you arrange for a remote access session for someone to take a look? > >>> > >>>> I can not use workaround with converting domain to export domain, > >>>> because we use iscsi. It would be perfect if I can retake existing > >>>> domain. Another gap is if I dont detach export domain I can not attach > >>>> it to new datacenter and I have to manually edit some files. It would be > >>>> absolutely perfect to do it from ovirt web gui (for example by "force > >>>> override button") > >>> i thought 3.3 allowed for that, but i could be wrong. > >> > >> Well, I will backup all VMs to export domain, than create snapshot of > >> ovirt and I will try to restart that server if its going to delete VMs > >> from DB. Than I will report back. I think, that remote session will not > >> be possible, but I can be wrong (I hope). > >> > >> > >> _______________________________________________ > >> Users mailing list > >> [email protected] > >> http://lists.ovirt.org/mailman/listinfo/users > >> > > I change it. But I can not restart postgres.
You can restart it only when the engine is down, it will not restart with database connections open. > _______________________________________________ > Users mailing list > [email protected] > http://lists.ovirt.org/mailman/listinfo/users > _______________________________________________ Users mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/users
