Re: [ovirt-users] hosted_storage locked [Solved]

[Gary Pedretty]

I tried one more thing.  Not sure if this is what fixed it, but the hosts and 
the engine were on different timezones.  The hosts were all on AKST -9 UTC and 
the engine was on the default of UTC.   I changed the timezone on the engine to 
match the hosts and destroyed the hosted_storage domain again.  This time it 
re-imported it without any problems.

Thanks

Gary


Gary Pedrettyg...@ravnalaska.net
Systems Manager  www.flyravn.com
Ravn Alaska   /\907-450-7251
5245 Airport Industrial Road /  \/\ 907-450-7238 fax
Fairbanks, Alaska  99709/\  /\ \ Second greatest commandment
Serving Alaska's Interior  /  \/  /\  \ \/\   "Love your neighbor as
Having a heatwave, its summer   yourself” Matt 22:39






> On Dec 29, 2016, at 3:13 PM, Gary Pedretty  wrote:
> 
> Ok so I tried destroying the storage domain.  The system then tried to import 
> it automatically again but it failed with the following event logged in the 
> Manager Events Log
> 
> Failed to attach Storage Domain hosted_storage to Data Center DOH. (User: 
> SYSTEM)
> 
> I tried this twice and same thing each time.
> 
> 
> If I try to mount the gluster volume on one of the hosts, it will mount just 
> fine and I can see all the files and it is read/writeable by root.
> 
> Nothing unusual in the glusterfs volume status etc.
> 
> 
> Gary
> 
> 

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] hosted_storage locked

[Gary Pedretty]

Ok so I tried destroying the storage domain.  The system then tried to import 
it automatically again but it failed with the following event logged in the 
Manager Events Log

Failed to attach Storage Domain hosted_storage to Data Center DOH. (User: 
SYSTEM)

I tried this twice and same thing each time.


If I try to mount the gluster volume on one of the hosts, it will mount just 
fine and I can see all the files and it is read/writeable by root.

Nothing unusual in the glusterfs volume status etc.


Gary



Gary Pedrettyg...@ravnalaska.net 

Systems Manager  www.flyravn.com 

Ravn Alaska   /\907-450-7251
5245 Airport Industrial Road /  \/\ 907-450-7238 fax
Fairbanks, Alaska  99709/\  /\ \ Second greatest commandment
Serving All of Alaska  /  \/  /\  \ \/\   “Love your neighbor as
Really loving the record green up date! Summmer!!   yourself” Matt 22:39













> On Dec 29, 2016, at 12:38 AM, Evgenia Tokar  wrote:
> 
> 
> I would suggest:
> 1. Verify the you can access the storage domain. 
> 2. If you can access the domain, you can try removing it from the ui (I think 
> the destroy option should be available). 
> After the removal the engine will try to reimport it, and hopefully will 
> succeed. 
> Removing the storage domain may cause the engine and the ha-agent to 
> restart. 
> 
> Jenny
> 

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] hosted_storage locked

[Gary Pedretty]

Destroying it was going to be my next move, just wanted to make sure that was 
the best option.  Yes I can mount the gluster volume “engine” directly on one 
of the hosts without issue.  The engine is obviously accessing it ok or at 
least enough that it can run as a VM image located on it.  

I will shut down any other VMs before trying this, but it makes sense that it 
should just re-import it.

Thanks

Gary



Gary Pedrettyg...@ravnalaska.net 

Systems Manager  www.flyravn.com 

Ravn Alaska   /\907-450-7251
5245 Airport Industrial Road /  \/\ 907-450-7238 fax
Fairbanks, Alaska  99709/\  /\ \ Second greatest commandment
Serving All of Alaska  /  \/  /\  \ \/\   “Love your neighbor as
Really loving the record green up date! Summmer!!   yourself” Matt 22:39





> On Dec 29, 2016, at 12:38 AM, Evgenia Tokar  wrote:
> 
> 
> I would suggest:
> 1. Verify the you can access the storage domain. 
> 2. If you can access the domain, you can try removing it from the ui (I think 
> the destroy option should be available). 
> After the removal the engine will try to reimport it, and hopefully will 
> succeed. 
> Removing the storage domain may cause the engine and the ha-agent to 
> restart. 
> 
> Jenny
> 
>  

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] hosted_storage locked

[Evgenia Tokar]

It seems that the import of the storage domain failed since it was
inaccessible:

2016-12-28 07:06:25,406 ERROR
[org.ovirt.engine.core.bll.storage.domain.GetExistingStorageDomainListQuery]
(org.ovirt.thread.pool-6-thread-46) [] Exception:
org.ovirt.engine.core.common.errors.EngineException: EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to
HSMGetStorageDomainInfoVDS, error = Domain is either partially accessible
or entirely inaccessible: (u'247b250c-ccd3-4056-853f-f03a05369da9',), code
= 379 (Failed with error StorageDomainAccessError and code 379)

2016-12-28 07:06:25,413 ERROR
[org.ovirt.engine.core.bll.storage.domain.ImportHostedEngineStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-46) [] Failed query for all Storage
Domains. The import command can not proceed without this info

Once that happened the command was logged in the db as failed, and every
engine restart the engine tries to run it again (compensate mechanism).
However, since the import storage domain command is not "compensatable"
this fails and the domain remains locked.

2016-12-28 07:09:02,643 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
(ServerService Thread Pool -- 56) [] CommandsFactory : Failed to get type
information using reflection for Class
'org.ovirt.engine.core.bll.storage.domain.ImportHostedEngineStorageDomainCommand',
Command Id '2ab30734-bbe4-4f50-897f-f1b393a31852':
org.ovirt.engine.core.bll.storage.domain.ImportHostedEngineStorageDomainCommand.(org.ovirt.engine.core.compat.Guid)
2016-12-28 07:09:02,643 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
(ServerService Thread Pool -- 56) [] Exception:
java.lang.NoSuchMethodException:
org.ovirt.engine.core.bll.storage.domain.ImportHostedEngineStorageDomainCommand.(org.ovirt.engine.core.compat.Guid)


2016-12-28 07:09:02,646 ERROR [org.ovirt.engine.core.bll.Backend]
(ServerService Thread Pool -- 56) [] Failed to run compensation on startup
for Command
'org.ovirt.engine.core.bll.storage.domain.ImportHostedEngineStorageDomainCommand',
Command Id '2ab30734-bbe4-4f50-897f-f1b393a31852'


I would suggest:
1. Verify the you can access the storage domain.
2. If you can access the domain, you can try removing it from the ui (I
think the destroy option should be available).
After the removal the engine will try to reimport it, and hopefully
will succeed.
Removing the storage domain may cause the engine and the ha-agent to
restart.

Jenny





On Wed, Dec 28, 2016 at 6:29 PM, Gary Pedretty  wrote:

> Here is the engine log from the engine vm.
>
> Gary
>
>
>
>
> 
> Gary Pedrettyg...@ravnalaska.net
> 
> Systems Manager  www.flyravn.com
> Ravn Alaska   /\907-450-7251
> <(907)%20450-7251>
> 5245 Airport Industrial Road /  \/\ 907-450-7238
> <(907)%20450-7238> fax
> Fairbanks, Alaska  99709/\  /\ \ Second greatest commandment
> Serving All of Alaska  /  \/  /\  \ \/\   “Love your neighbor as
> Really loving the record green up date! Summmer!!   yourself” Matt 22:39
> 
>
>
>
>
>
>
>
>
>
>
>
>
> On Dec 28, 2016, at 1:12 AM, Evgenia Tokar  wrote:
>
> Hi!
>
> Can you attach the engine log from the hosted engine vm?
>
> Thanks,
> Jenny
>
>
> On Wed, Dec 28, 2016 at 10:56 AM, Gary Pedretty 
> wrote:
>
>> Just setup a new hosted-engine Data Center.  With 3 hosts and gluster
>> based storage domains.  Used the latest installers.  Everything went well,
>> until I got the engine deployed and each host setup and went to the web
>> interface for the first time.
>>
>> After adding the master storage domain, the hosted_storage added itself
>> like usual, but remained locked.   Everything is working just fine with the
>> exception of no listing of the engine as a VM under that tab or the icon on
>> the current host.   The Volumes tab does not show anything on the
>> hosted_storage, but all the other volumes including the master data and a
>> second data all show normal status for bricks.  Error log is not reporting
>> anything unusual either.  All the gluster volumes including the engine
>> (hosted_storage) show good status via command line in the hosts.
>>
>> Going to let it run for a day or so before I add any VMs.
>>
>> Any ideas?
>>
>> Gary
>>
>> 
>> Gary Pedrettyg...@ravnalaska.net
>> 
>> Systems Manager  www.flyravn.com
>> Ravn Alaska   /\907-450-7251
>> <(907)%20450-7251>
>> 5245 Airport Industrial Road   

Re: [ovirt-users] hosted_storage locked

[Evgenia Tokar]

Hi!

Can you attach the engine log from the hosted engine vm?

Thanks,
Jenny


On Wed, Dec 28, 2016 at 10:56 AM, Gary Pedretty  wrote:

> Just setup a new hosted-engine Data Center.  With 3 hosts and gluster
> based storage domains.  Used the latest installers.  Everything went well,
> until I got the engine deployed and each host setup and went to the web
> interface for the first time.
>
> After adding the master storage domain, the hosted_storage added itself
> like usual, but remained locked.   Everything is working just fine with the
> exception of no listing of the engine as a VM under that tab or the icon on
> the current host.   The Volumes tab does not show anything on the
> hosted_storage, but all the other volumes including the master data and a
> second data all show normal status for bricks.  Error log is not reporting
> anything unusual either.  All the gluster volumes including the engine
> (hosted_storage) show good status via command line in the hosts.
>
> Going to let it run for a day or so before I add any VMs.
>
> Any ideas?
>
> Gary
>
> 
> Gary Pedrettyg...@ravnalaska.net
> 
> Systems Manager  www.flyravn.com
> Ravn Alaska   /\907-450-7251
> <(907)%20450-7251>
> 5245 Airport Industrial Road /  \/\ 907-450-7238
> <(907)%20450-7238> fax
> Fairbanks, Alaska  99709/\  /\ \ Second greatest commandment
> Serving All of Alaska  /  \/  /\  \ \/\   “Love your neighbor as
> Really loving the record green up date! Summmer!!   yourself” Matt 22:39
> 
>
>
>
>
>
>
>
>
>
>
>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] hosted_storage locked

[Gary Pedretty]

Just setup a new hosted-engine Data Center.  With 3 hosts and gluster based 
storage domains.  Used the latest installers.  Everything went well, until I 
got the engine deployed and each host setup and went to the web interface for 
the first time.

After adding the master storage domain, the hosted_storage added itself like 
usual, but remained locked.   Everything is working just fine with the 
exception of no listing of the engine as a VM under that tab or the icon on the 
current host.   The Volumes tab does not show anything on the hosted_storage, 
but all the other volumes including the master data and a second data all show 
normal status for bricks.  Error log is not reporting anything unusual either.  
All the gluster volumes including the engine (hosted_storage) show good status 
via command line in the hosts.

Going to let it run for a day or so before I add any VMs.

Any ideas?

Gary


Gary Pedrettyg...@ravnalaska.net 

Systems Manager  www.flyravn.com 

Ravn Alaska   /\907-450-7251
5245 Airport Industrial Road /  \/\ 907-450-7238 fax
Fairbanks, Alaska  99709/\  /\ \ Second greatest commandment
Serving All of Alaska  /  \/  /\  \ \/\   “Love your neighbor as
Really loving the record green up date! Summmer!!   yourself” Matt 22:39













___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users