Re: [ovirt-users] Can not access storage domain hosted_storage
Thanks Darryl! For others having the same problem and needing more information on how to temporarily fix this: On the Engine VM you find the information to access the DB in /etc/ovirt-engine/engine.conf.d/10-setup-database.conf Then access the engine database and update the vfs_type field in the storage_server_connections table of the engine storage volume entry: psql -U engine -W -h localhost select * from storage_server_connections; update storage_server_connections set vfs_type = 'glusterfs' where id = 'THE_ID_YOU_FOUND_IN_THE_OUTPUT_ABOVE_FOR_THE_ENGINE_VOLUME'; After that adding new hosts works as expected. Cheers Richard On 04/08/2016 12:43 AM, Bond, Darryl wrote: > The workaround for this bug is here > https://bugzilla.redhat.com/show_bug.cgi?id=1317699 > > > > From: users-boun...@ovirt.org <users-boun...@ovirt.org> on behalf of Simone > Tiraboschi <stira...@redhat.com> > Sent: Friday, 8 April 2016 1:30 AM > To: Richard Neuboeck; Roy Golan > Cc: users > Subject: Re: [ovirt-users] Can not access storage domain hosted_storage > > On Thu, Apr 7, 2016 at 4:17 PM, Richard Neuboeck <h...@tbi.univie.ac.at> > wrote: >> Hi oVirt Users/Developers, >> >> I'm having trouble adding another host to a working hosted engine >> setup. Through the WebUI I try to add another host. The package >> installation and configuration processes seemingly run without >> problems. When the second host tries to mount the engine storage >> volume it halts with the WebUI showing the following message: >> >> 'Failed to connect Host cube-two to the Storage Domain hosted_engine' >> >> The mount fails which results in the host status as 'non operational'. >> >> Checking the vdsm.log on the newly added host shows that the mount >> attempt of the engine volume doesn't use -t glusterfs. On the other >> hand the VM storage volume (also a glusterfs volume) is mounted the >> right way. >> >> It seems the Engine configuration that is given to the second host >> lacks the vfs_type property. So without glusterfs as fs given the >> system assumes an NFS mount and obviously fails. > > It seams that the auto-import procedure in the engine didn't recognize > that the hosted-engine storage domain was on gluster and took it for > NFS. > > Adding Roy here to take a look. > > >> Here are the relevant log lines showing the JSON reply to the >> configuration request, the working mount of the VM storage (called >> plexus) and the failing mount of the engine storage. >> >> ... >> jsonrpc.Executor/4::INFO::2016-04-07 >> 15:45:53,043::logUtils::48::dispatcher::(wrapper) Run and protect: >> connectStorageServer(domType=7, >> spUUID=u'0001-0001-0001-0001-03ce', conList=[{u'id': >> u'981cd3aa-052b-498a-914e-5e8f314357a8', u'connection': >> u'borg-sphere-one:/plexus', u'iqn': u'', u'user': u'', u'tpgt': >> u'1', u'vfs_type': u'glusterfs', u'password': '', u'port': >> u''}, {u'id': u'cceaa988-9607-4bef-8854-0e7a585720aa', >> u'connection': u'borg-sphere-one:/engine', u'iqn': u'', u'user': >> u'', u'tpgt': u'1', u'password': '', u'port': u''}], >> options=None) >> ... >> jsonrpc.Executor/4::DEBUG::2016-04-07 >> 15:45:53,062::mount::229::Storage.Misc.excCmd::(_runcmd) >> /usr/bin/taskset --cpu-list 0-39 /usr/bin/sudo -n >> /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount >> -t glusterfs -o >> backup-volfile-servers=borg-sphere-two:borg-sphere-three >> borg-sphere-one:/plexus >> /rhev/data-center/mnt/glusterSD/borg-sphere-one:_plexus (cwd None) >> ... >> jsonrpc.Executor/4::DEBUG::2016-04-07 >> 15:45:53,380::mount::229::Storage.Misc.excCmd::(_runcmd) >> /usr/bin/taskset --cpu-list 0-39 /usr/bin/sudo -n >> /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount >> -o backup-volfile-servers=borg-sphere-two:borg-sphere-three >> borg-sphere-one:/engine >> /rhev/data-center/mnt/glusterSD/borg-sphere-one:_engine (cwd None) >> ... >> >> The problem seems to have been introduced since March 22nd. On this >> install I have added two additional hosts without problem. Three >> days ago I tried to reinstall the whole system for testing and >> documentation purposes but now am not able to add other hosts. >> >> All the installs follow the same documented procedure. I've verified >> several times that the problem exists with the components in the >> current 3.6 release repo as well as in the 3.6 snapshot repo. >> >> If I check the storage configuration of hosted_engine domain in the >> WebUI
Re: [ovirt-users] Can not access storage domain hosted_storage
The workaround for this bug is here https://bugzilla.redhat.com/show_bug.cgi?id=1317699 From: users-boun...@ovirt.org <users-boun...@ovirt.org> on behalf of Simone Tiraboschi <stira...@redhat.com> Sent: Friday, 8 April 2016 1:30 AM To: Richard Neuboeck; Roy Golan Cc: users Subject: Re: [ovirt-users] Can not access storage domain hosted_storage On Thu, Apr 7, 2016 at 4:17 PM, Richard Neuboeck <h...@tbi.univie.ac.at> wrote: > Hi oVirt Users/Developers, > > I'm having trouble adding another host to a working hosted engine > setup. Through the WebUI I try to add another host. The package > installation and configuration processes seemingly run without > problems. When the second host tries to mount the engine storage > volume it halts with the WebUI showing the following message: > > 'Failed to connect Host cube-two to the Storage Domain hosted_engine' > > The mount fails which results in the host status as 'non operational'. > > Checking the vdsm.log on the newly added host shows that the mount > attempt of the engine volume doesn't use -t glusterfs. On the other > hand the VM storage volume (also a glusterfs volume) is mounted the > right way. > > It seems the Engine configuration that is given to the second host > lacks the vfs_type property. So without glusterfs as fs given the > system assumes an NFS mount and obviously fails. It seams that the auto-import procedure in the engine didn't recognize that the hosted-engine storage domain was on gluster and took it for NFS. Adding Roy here to take a look. > Here are the relevant log lines showing the JSON reply to the > configuration request, the working mount of the VM storage (called > plexus) and the failing mount of the engine storage. > > ... > jsonrpc.Executor/4::INFO::2016-04-07 > 15:45:53,043::logUtils::48::dispatcher::(wrapper) Run and protect: > connectStorageServer(domType=7, > spUUID=u'0001-0001-0001-0001-03ce', conList=[{u'id': > u'981cd3aa-052b-498a-914e-5e8f314357a8', u'connection': > u'borg-sphere-one:/plexus', u'iqn': u'', u'user': u'', u'tpgt': > u'1', u'vfs_type': u'glusterfs', u'password': '', u'port': > u''}, {u'id': u'cceaa988-9607-4bef-8854-0e7a585720aa', > u'connection': u'borg-sphere-one:/engine', u'iqn': u'', u'user': > u'', u'tpgt': u'1', u'password': '', u'port': u''}], > options=None) > ... > jsonrpc.Executor/4::DEBUG::2016-04-07 > 15:45:53,062::mount::229::Storage.Misc.excCmd::(_runcmd) > /usr/bin/taskset --cpu-list 0-39 /usr/bin/sudo -n > /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount > -t glusterfs -o > backup-volfile-servers=borg-sphere-two:borg-sphere-three > borg-sphere-one:/plexus > /rhev/data-center/mnt/glusterSD/borg-sphere-one:_plexus (cwd None) > ... > jsonrpc.Executor/4::DEBUG::2016-04-07 > 15:45:53,380::mount::229::Storage.Misc.excCmd::(_runcmd) > /usr/bin/taskset --cpu-list 0-39 /usr/bin/sudo -n > /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount > -o backup-volfile-servers=borg-sphere-two:borg-sphere-three > borg-sphere-one:/engine > /rhev/data-center/mnt/glusterSD/borg-sphere-one:_engine (cwd None) > ... > > The problem seems to have been introduced since March 22nd. On this > install I have added two additional hosts without problem. Three > days ago I tried to reinstall the whole system for testing and > documentation purposes but now am not able to add other hosts. > > All the installs follow the same documented procedure. I've verified > several times that the problem exists with the components in the > current 3.6 release repo as well as in the 3.6 snapshot repo. > > If I check the storage configuration of hosted_engine domain in the > WebUI it shows glusterfs as VFS type. > > The initial mount during the hosted engine setup on the first host > shows the correct parameters (vfs_type) in vdsm.log: > > Thread-42::INFO::2016-04-07 > 14:56:29,464::logUtils::48::dispatcher::(wrapper) Run and protect: > connectStorageServer(domType=7, > spUUID='----', conList=[{'id': > 'b13ae31f-d66a-43a7-8aba-eaf4e62a6fb0', 'tpgt': '1', 'vfs_type': > 'glusterfs', 'connection': 'borg-sphere-one:/engine', 'user': > 'kvm'}], options=None) > Thread-42::DEBUG::2016-04-07 > 14:56:29,591::fileUtils::143::Storage.fileUtils::(createdir) > Creating directory: > /rhev/data-center/mnt/glusterSD/borg-sphere-one:_engine mode: None > Thread-42::DEBUG::2016-04-07 > 14:56:29,592::storageServer::364::Storage.StorageServer.MountConnection::(_get_backup_servers_option) > Using bricks: ['borg-sphere-one', 'borg-sphere-two', > 'borg-sphere-three'] > Thread-42::DEBUG::2016-04-07 > 14:56:29,592::mount::229::Storage.Misc.excCmd::(_ru
Re: [ovirt-users] Can not access storage domain hosted_storage
On Thu, Apr 7, 2016 at 4:17 PM, Richard Neuboeckwrote: > Hi oVirt Users/Developers, > > I'm having trouble adding another host to a working hosted engine > setup. Through the WebUI I try to add another host. The package > installation and configuration processes seemingly run without > problems. When the second host tries to mount the engine storage > volume it halts with the WebUI showing the following message: > > 'Failed to connect Host cube-two to the Storage Domain hosted_engine' > > The mount fails which results in the host status as 'non operational'. > > Checking the vdsm.log on the newly added host shows that the mount > attempt of the engine volume doesn't use -t glusterfs. On the other > hand the VM storage volume (also a glusterfs volume) is mounted the > right way. > > It seems the Engine configuration that is given to the second host > lacks the vfs_type property. So without glusterfs as fs given the > system assumes an NFS mount and obviously fails. It seams that the auto-import procedure in the engine didn't recognize that the hosted-engine storage domain was on gluster and took it for NFS. Adding Roy here to take a look. > Here are the relevant log lines showing the JSON reply to the > configuration request, the working mount of the VM storage (called > plexus) and the failing mount of the engine storage. > > ... > jsonrpc.Executor/4::INFO::2016-04-07 > 15:45:53,043::logUtils::48::dispatcher::(wrapper) Run and protect: > connectStorageServer(domType=7, > spUUID=u'0001-0001-0001-0001-03ce', conList=[{u'id': > u'981cd3aa-052b-498a-914e-5e8f314357a8', u'connection': > u'borg-sphere-one:/plexus', u'iqn': u'', u'user': u'', u'tpgt': > u'1', u'vfs_type': u'glusterfs', u'password': '', u'port': > u''}, {u'id': u'cceaa988-9607-4bef-8854-0e7a585720aa', > u'connection': u'borg-sphere-one:/engine', u'iqn': u'', u'user': > u'', u'tpgt': u'1', u'password': '', u'port': u''}], > options=None) > ... > jsonrpc.Executor/4::DEBUG::2016-04-07 > 15:45:53,062::mount::229::Storage.Misc.excCmd::(_runcmd) > /usr/bin/taskset --cpu-list 0-39 /usr/bin/sudo -n > /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount > -t glusterfs -o > backup-volfile-servers=borg-sphere-two:borg-sphere-three > borg-sphere-one:/plexus > /rhev/data-center/mnt/glusterSD/borg-sphere-one:_plexus (cwd None) > ... > jsonrpc.Executor/4::DEBUG::2016-04-07 > 15:45:53,380::mount::229::Storage.Misc.excCmd::(_runcmd) > /usr/bin/taskset --cpu-list 0-39 /usr/bin/sudo -n > /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount > -o backup-volfile-servers=borg-sphere-two:borg-sphere-three > borg-sphere-one:/engine > /rhev/data-center/mnt/glusterSD/borg-sphere-one:_engine (cwd None) > ... > > The problem seems to have been introduced since March 22nd. On this > install I have added two additional hosts without problem. Three > days ago I tried to reinstall the whole system for testing and > documentation purposes but now am not able to add other hosts. > > All the installs follow the same documented procedure. I've verified > several times that the problem exists with the components in the > current 3.6 release repo as well as in the 3.6 snapshot repo. > > If I check the storage configuration of hosted_engine domain in the > WebUI it shows glusterfs as VFS type. > > The initial mount during the hosted engine setup on the first host > shows the correct parameters (vfs_type) in vdsm.log: > > Thread-42::INFO::2016-04-07 > 14:56:29,464::logUtils::48::dispatcher::(wrapper) Run and protect: > connectStorageServer(domType=7, > spUUID='----', conList=[{'id': > 'b13ae31f-d66a-43a7-8aba-eaf4e62a6fb0', 'tpgt': '1', 'vfs_type': > 'glusterfs', 'connection': 'borg-sphere-one:/engine', 'user': > 'kvm'}], options=None) > Thread-42::DEBUG::2016-04-07 > 14:56:29,591::fileUtils::143::Storage.fileUtils::(createdir) > Creating directory: > /rhev/data-center/mnt/glusterSD/borg-sphere-one:_engine mode: None > Thread-42::DEBUG::2016-04-07 > 14:56:29,592::storageServer::364::Storage.StorageServer.MountConnection::(_get_backup_servers_option) > Using bricks: ['borg-sphere-one', 'borg-sphere-two', > 'borg-sphere-three'] > Thread-42::DEBUG::2016-04-07 > 14:56:29,592::mount::229::Storage.Misc.excCmd::(_runcmd) > /usr/bin/taskset --cpu-list 0-39 /usr/bin/sudo -n > /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount > -t glusterfs -o > backup-volfile-servers=borg-sphere-two:borg-sphere-three > borg-sphere-one:/engine > /rhev/data-center/mnt/glusterSD/borg-sphere-one:_engine (cwd None) > > > I've already created a bug report but since I didn't know where to > put it filed it as VDSM bug which it doesn't seem to be. > https://bugzilla.redhat.com/show_bug.cgi?id=1324075 > > > I would really like to help resolve this problem. If there is > anything I can test, please let me know. I appreciate any help in > this matter. > > Currently I'm running an oVirt 3.6