[ovirt-users] Re: Upgraded to 4.3: getting SpmStart failure
On Fri, Jul 5, 2019 at 6:45 PM Richard Chan wrote: > Unfortunately, this is a separate engine running external to the oVirt > cluster. > The main error on the hosts is: any idea what this Volume is referring to? > I have 3 existings storage domains > ISO EXPORT and DATA all upgraded to 4.3 > None have this UUID. > > > VolumeDoesNotExist: Volume does not exist: > (u'1d1b6dfd-197a-4583-8398-6b941dbca854',) > > 2019-07-05 23:02:15,954+0800 ERROR (tasks/0) [storage.StoragePool] failed: > Volume does not exist: (u'1d1b6dfd-197a-4583-8398-6b941dbca854',) (sp:384) > 2019-07-05 23:02:15,954+0800 INFO (tasks/0) [storage.SANLock] Releasing > Lease(name='SDM', > path=u'/rhev/data-center/mnt/192.168.88.40:_genie_ovirt_data/f9aaa65a-0ac7-4c61-a27d-15af39016419/dom_md/leases', > offset=1048576) (clusterlock:488) > 2019-07-05 23:02:15,955+0800 INFO (tasks/0) [storage.SANLock] > Successfully released Lease(name='SDM', > path=u'/rhev/data-center/mnt/192.168.88.40:_genie_ovirt_data/f9aaa65a-0ac7-4c61-a27d-15af39016419/dom_md/leases', > offset=1048576) (clusterlock:497) > 2019-07-05 23:02:15,955+0800 ERROR (tasks/0) [storage.TaskManager.Task] > (Task='159b9026-7034-43c2-9587-8fec036e92ba') Unexpected error (task:875) > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, > in _run > return fn(*args, **kargs) > File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, > in run > return self.cmd(*self.argslist, **self.argsdict) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 333, in > startSpm > self._upgradePool(expectedDomVersion, __securityOverride=True) > File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line > 79, in wrapper > return method(self, *args, **kwargs) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 484, in > _upgradePool > str(targetDomVersion)) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1108, > in _convertDomain > targetFormat) > File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", > line 447, in convert > converter(repoPath, hostId, imageRepo, isMsd) > File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", > line 405, in v5DomainConverter > domain.convert_volumes_metadata(target_version) > File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line > 813, in convert_volumes_metadata > for vol in self.iter_volumes(): > File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 764, in > iter_volumes > yield self.produceVolume(img_id, vol_id) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 846, in > produceVolume > volUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line > 817, in __init__ > self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line > 71, in __init__ > volUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, > in __init__ > self.validate() > File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line > 112, in validate > self.validateVolumePath() > File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line > 131, in validateVolumePath > raise se.VolumeDoesNotExist(self.volUUID) > This usually means the volume does not have the right permissions. 130 if not self.oop.fileUtils.pathExists(volPath): 131 raise se.VolumeDoesNotExist(self.volUUID) oop.fileUtils.pathExists() is: 130 if not self.oop.fileUtils.pathExists(volPath): 131 raise se.VolumeDoesNotExist(self.volUUID) IOProcess.pathExists() is: 482 def pathExists(self, filename, writable=False): 483 check = os.R_OK 484 485 if writable: 486 check |= os.W_OK 487 488 if self.access(filename, check): 489 return True 490 491 return self.access(filename, check) Bad permissions would fail converting a domain of course, so this real fix is to fix the bad permissions. Nir > > On Fri, Jul 5, 2019 at 11:31 PM Strahil wrote: > >> Can you run the engine manually on another server ? >> hosted-engine --vm-start >> >> Best Regards, >> Strahil Nikolov >> On Jul 5, 2019 18:05, Richard Chan wrote: >> >> My engine is getting SpmStart failure. >> >> What is the best way to troubleshoot this? >> >> >> -- >> Richard Chan >> >> > > -- > Richard Chan > Chief Architect > > TreeBox Solutions Pte Ltd > 1 Commonwealth Lane #03-01 > Singapore 149544 > Tel: 6570 3725 > http://www.treeboxsolutions.com > > Co.Reg.No. 201100585R > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: >
[ovirt-users] Re: Upgraded to 4.3: getting SpmStart failure
This is in the engine.log [storage.StoragePool] Trying to upgrade master domain `f9aaa65a-0ac7-4c61-a27d-15af3901 6419` (sp:479) 2019-07-06 00:24:14,056+0800 INFO (tasks/5) [storage.StorageDomain] Converting domain f9aaa65a-0ac7-4c61-a27d-15af39016419 volumes metadata from version 4 to version 5 (fileSD:811) 2019-07-06 00:24:14,121+0800 INFO (tasks/5) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-07-06 00:24:14,121+0800 ERROR (tasks/5) [storage.StoragePool] Unexpected error (sp:383) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 333, in startSpm On Fri, Jul 5, 2019 at 11:31 PM Strahil wrote: > Can you run the engine manually on another server ? > hosted-engine --vm-start > > Best Regards, > Strahil Nikolov > On Jul 5, 2019 18:05, Richard Chan wrote: > > My engine is getting SpmStart failure. > > What is the best way to troubleshoot this? > > > -- > Richard Chan > > -- Richard Chan Chief Architect TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com Co.Reg.No. 201100585R ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GW7HOCWLI2OVGC2OR7A6AAAYCBXGOPLT/
[ovirt-users] Re: Upgraded to 4.3: getting SpmStart failure
Unfortunately, this is a separate engine running external to the oVirt cluster. The main error on the hosts is: any idea what this Volume is referring to? I have 3 existings storage domains ISO EXPORT and DATA all upgraded to 4.3 None have this UUID. VolumeDoesNotExist: Volume does not exist: (u'1d1b6dfd-197a-4583-8398-6b941dbca854',) 2019-07-05 23:02:15,954+0800 ERROR (tasks/0) [storage.StoragePool] failed: Volume does not exist: (u'1d1b6dfd-197a-4583-8398-6b941dbca854',) (sp:384) 2019-07-05 23:02:15,954+0800 INFO (tasks/0) [storage.SANLock] Releasing Lease(name='SDM', path=u'/rhev/data-center/mnt/192.168.88.40:_genie_ovirt_data/f9aaa65a-0ac7-4c61-a27d-15af39016419/dom_md/leases', offset=1048576) (clusterlock:488) 2019-07-05 23:02:15,955+0800 INFO (tasks/0) [storage.SANLock] Successfully released Lease(name='SDM', path=u'/rhev/data-center/mnt/192.168.88.40:_genie_ovirt_data/f9aaa65a-0ac7-4c61-a27d-15af39016419/dom_md/leases', offset=1048576) (clusterlock:497) 2019-07-05 23:02:15,955+0800 ERROR (tasks/0) [storage.TaskManager.Task] (Task='159b9026-7034-43c2-9587-8fec036e92ba') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 333, in startSpm self._upgradePool(expectedDomVersion, __securityOverride=True) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 484, in _upgradePool str(targetDomVersion)) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1108, in _convertDomain targetFormat) File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", line 447, in convert converter(repoPath, hostId, imageRepo, isMsd) File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", line 405, in v5DomainConverter domain.convert_volumes_metadata(target_version) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 813, in convert_volumes_metadata for vol in self.iter_volumes(): File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 764, in iter_volumes yield self.produceVolume(img_id, vol_id) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 846, in produceVolume volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 817, in __init__ self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 71, in __init__ volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, in __init__ self.validate() File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 112, in validate self.validateVolumePath() File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 131, in validateVolumePath raise se.VolumeDoesNotExist(self.volUUID) On Fri, Jul 5, 2019 at 11:31 PM Strahil wrote: > Can you run the engine manually on another server ? > hosted-engine --vm-start > > Best Regards, > Strahil Nikolov > On Jul 5, 2019 18:05, Richard Chan wrote: > > My engine is getting SpmStart failure. > > What is the best way to troubleshoot this? > > > -- > Richard Chan > > -- Richard Chan Chief Architect TreeBox Solutions Pte Ltd 1 Commonwealth Lane #03-01 Singapore 149544 Tel: 6570 3725 http://www.treeboxsolutions.com Co.Reg.No. 201100585R ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3A5Y2JMNHBTPWET4LOFTONYPEGORLGZK/
[ovirt-users] Re: Upgraded to 4.3: getting SpmStart failure
Can you run the engine manually on another server ? hosted-engine --vm-start Best Regards, Strahil NikolovOn Jul 5, 2019 18:05, Richard Chan wrote: > > My engine is getting SpmStart failure. > > What is the best way to troubleshoot this? > > > -- > Richard Chan > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LTYGACYXBB65FMSWNZLZZFLC2XQZU7WG/
