[ovirt-users] Re: Remote console access for old Ovirt 3.6 install

[David Gossage]

Thanks, I'll look into it though I am hoping to migrate the whole system to
new hardware and updated version in coming weeks.
Last night I copied the engine vm image to a simple kvm host and loaded up
the image and ran the xfs_repair on /var which fixed the problems it had
then copied the image back over to ovirt and it started up.  Little lengthy
but it got me back up.

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284


On Tue, Nov 27, 2018 at 12:54 AM Yedidyah Bar David  wrote:

> On Mon, Nov 26, 2018 at 8:18 PM David Gossage
>  wrote:
> >
> > I have an older 3.6.7 setup I am still using at a location.  The
> hosted-engine is up but not responding
> >
> > {"reason": "failed liveliness check", "health": "bad", "vm": "up",
> "detail": "up"}
> >
> > I am guessing it got into some paused or halted state after a power
> outage and storage went offline for a short bit and I may need to manually
> perform some tasks.  When I attempt the suggested method of
> > hosted-engine --set-maintenance --mode=global
> > hosted-engine --console
> >
> > I get an error about
> >
> > Use the password you've set using --add-console-password for logging in
> >
> > ** (remote-viewer:11479): WARNING **: Could not open X display
> > Cannot open display:
> > Run 'remote-viewer --help' to see a full list of available command line
> options
>
> Recent versions configure the hosted-engine vm to use a serial console,
> which is much easier to use (if you have no X on the host).
>
> For older versions, you might want to check a page I wrote some time ago,
> that does not exist anymore on the website, but can be found at
> archive.org:
>
>
> http://web.archive.org/web/20150704004004/http://www.ovirt.org:80/Hosted_Engine_Console
>
> Good luck and best regards,
>
> >
> > Their are 7 other VM's running I was able to manually continue from
> vdsClient
> >
> > Is their a way to get remote access to this engine?
> >
> > David Gossage
> > Carousel Checks Inc. | System Administrator
> > Office 708.613.2284
> > ___
> > Users mailing list -- users@ovirt.org
> > To unsubscribe send an email to users-le...@ovirt.org
> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> > oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/QVO7CA4MDK7BIJJWZPEP6JLPSAJMVQCD/
>
>
>
> --
> Didi
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JH6YUZ6ONHSIIJMIW2AS6LC5QDPTGH6N/

[ovirt-users] Remote console access for old Ovirt 3.6 install

[David Gossage]

I have an older 3.6.7 setup I am still using at a location.  The
hosted-engine is up but not responding

{"reason": "failed liveliness check", "health": "bad", "vm": "up",
"detail": "up"}

I am guessing it got into some paused or halted state after a power outage
and storage went offline for a short bit and I may need to manually perform
some tasks.  When I attempt the suggested method of
hosted-engine --set-maintenance --mode=global
hosted-engine --console

I get an error about

Use the password you've set using --add-console-password for logging in

** (remote-viewer:11479): WARNING **: Could not open X display
Cannot open display:
Run 'remote-viewer --help' to see a full list of available command line
options

Their are 7 other VM's running I was able to manually continue from
vdsClient

Is their a way to get remote access to this engine?

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QVO7CA4MDK7BIJJWZPEP6JLPSAJMVQCD/

Re: [ovirt-users] Help! No VMs start after reboot of cluster

[David Gossage]

On Fri, Jun 30, 2017 at 10:34 AM, cmc  wrote:

> Hi Denis,
>
> Yes, I did check that and it said it was out of global maintenance
> ('False' I think it said).
>
>
Did you check that the storage the hostedengine VM attaches to mounted and
is in a healthy state, and that the broker and agent services are running?
Both have logs that may give some indication if it detects an issue as well.


Thanks,
>
> Cam
>
> On Fri, Jun 30, 2017 at 4:31 PM, Denis Chaplygin 
> wrote:
> > Hello!
> >
> > On Fri, Jun 30, 2017 at 4:35 PM, cmc  wrote:
> >>
> >> I restarted my 3 host cluster after setting it into global maintenance
> >> mode and then shutting down all of the nodes and then bringing them up
> >> again. I moved it out of global maintenance mode and no VM is running,
> >> including the hosted engine.
> >>
> >> Any help greatly appreciated!
> >
> >
> > Are you sure you are really out of global maintenance? Could you please
> post
> > hosted-engine --vm-status output?
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] vm has been paused due to unknown storage

[David Gossage]

On Wed, May 31, 2017 at 4:16 AM,  wrote:

> Hi,
>
> I found the cause of this problem. I had to turn off sharding.
>


Did you have sharding enabled but not have any sharded VM images or were
their shards missing on some bricks?



> --
> *De: *supo...@logicworks.pt
> *Para: *"Sahina Bose" 
> *Cc: *"ovirt users" 
> *Enviadas: *Sexta-feira, 26 De Maio de 2017 12:27:43
> *Assunto: *Re: [ovirt-users] vm has been paused due to unknown storage
>
> Hi,
>
> I updated glusterfs:
> glusterfs-client-xlators-3.8.12-1.el7.x86_64
> glusterfs-cli-3.8.12-1.el7.x86_64
> glusterfs-api-3.8.12-1.el7.x86_64
> glusterfs-fuse-3.8.12-1.el7.x86_64
> glusterfs-server-3.8.12-1.el7.x86_64
> glusterfs-libs-3.8.12-1.el7.x86_64
> glusterfs-3.8.12-1.el7.x86_64
>
> Now I cannot add a volume disk preallocated, after a while it breaks.
>
> message log:
> May 26 11:18:16 node journal: vdsm root ERROR VM metrics collection
> failed#012Traceback (most recent call last):#012  File
> "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 221, in
> send_metrics#012diskinfo['readOps']#012KeyError: 'readOps'
>
> vdsm.log
> 2017-05-26 11:18:16,715+0100 ERROR (periodic/3) [root] VM metrics
> collection failed (vmstats:264)
> 2017-05-26 11:19:39,369+0100 ERROR (tasks/5) [storage.Volume] Unexpected
> error (fileVolume:456)
> 2017-05-26 11:19:39,373+0100 ERROR (tasks/5) [storage.Volume] Unexpected
> error (volume:1107)
> 2017-05-26 11:19:39,374+0100 ERROR (tasks/5) [storage.TaskManager.Task]
> (Task='5b2adb9a-e24e-48fa-9f01-f21c23588aef') Unexpected error (task:870)
>
> glusterfs
> [2017-05-26 10:53:08.247219] W [MSGID: 114031] 
> [client-rpc-fops.c:2933:client3_3_lookup_cbk]
> 0-gv2-client-0: remote operation failed. Path: 
> /.shard/55b94942-dee5-4f69-8b0f-52e251ac6f5e.164
> (----) [No data available]
> [2017-05-26 10:53:14.899499] W [MSGID: 114031] 
> [client-rpc-fops.c:2933:client3_3_lookup_cbk]
> 0-gv2-client-0: remote operation failed. Path: 
> /.shard/55b94942-dee5-4f69-8b0f-52e251ac6f5e.167
> (----) [No data available]
> [2017-05-26 10:53:14.899526] E [MSGID: 133010] 
> [shard.c:1725:shard_common_lookup_shards_cbk]
> 0-gv2-shard: Lookup on shard 167 failed. Base file gfid =
> 55b94942-dee5-4f69-8b0f-52e251ac6f5e [No data available]
> [2017-05-26 10:53:19.712567] W [MSGID: 114031] 
> [client-rpc-fops.c:2933:client3_3_lookup_cbk]
> 0-gv2-client-0: remote operation failed. Path: 
> /.shard/55b94942-dee5-4f69-8b0f-52e251ac6f5e.169
> (----) [No data available]
> [2017-05-26 10:53:19.712614] E [MSGID: 133010] 
> [shard.c:1725:shard_common_lookup_shards_cbk]
> 0-gv2-shard: Lookup on shard 169 failed. Base file gfid =
> 55b94942-dee5-4f69-8b0f-52e251ac6f5e [No data available]
> [2017-05-26 10:53:29.419317] W [MSGID: 114031] 
> [client-rpc-fops.c:2933:client3_3_lookup_cbk]
> 0-gv2-client-0: remote operation failed. Path: 
> /.shard/55b94942-dee5-4f69-8b0f-52e251ac6f5e.173
> (----) [No data available]
> [2017-05-26 10:53:29.419369] E [MSGID: 133010] 
> [shard.c:1725:shard_common_lookup_shards_cbk]
> 0-gv2-shard: Lookup on shard 173 failed. Base file gfid =
> 55b94942-dee5-4f69-8b0f-52e251ac6f5e [No data available]
>
>
> thanks
>
> --
> *De: *"Sahina Bose" 
> *Para: *supo...@logicworks.pt, "Krutika Dhananjay" 
> *Cc: *"ovirt users" 
> *Enviadas: *Quinta-feira, 25 De Maio de 2017 7:12:40
> *Assunto: *Re: [ovirt-users] vm has been paused due to unknown storage
>
> The glusterfs logs contain below errors:
> [2017-05-22 18:12:50.941883] E [MSGID: 133010] 
> [shard.c:1725:shard_common_lookup_shards_cbk]
> 0-gv2-shard: Lookup on shard 50 failed. Base file gfid =
> 33f1fe3e-c626-49f2-861e-2259c972931d [No data available]
> [2017-05-22 18:12:50.945085] W [fuse-bridge.c:1291:fuse_err_cbk]
> 0-glusterfs-fuse: 61306713: FSYNC() ERR => -1 (No data available)
>
> Krutika, could you take a look?
>
> On Thu, May 25, 2017 at 1:02 AM,  wrote:
>
>> Hi,
>>
>> I setup an ovirt hosted enine, in only one server with local gluster
>> bricks.
>>
>> When running a MS SQL 2012 process to rebuild a data base, which take
>> around 4 hours, after a while the VM is paused with the error:
>>
>> vm has been paused due to unknown storage
>>
>> The VM disk is in Thin provision
>>
>> Ovirt and gluter versions:
>>
>> Version 4.1.1.8-1.el7.centos
>>
>> glusterfs-cli-3.8.11-1.el7.x86_64
>> glusterfs-libs-3.8.11-1.el7.x86_64
>> glusterfs-3.8.11-1.el7.x86_64
>> glusterfs-client-xlators-3.8.11-1.el7.x86_64
>> glusterfs-fuse-3.8.11-1.el7.x86_64
>> glusterfs-api-3.8.11-1.el7.x86_64
>> glusterfs-server-3.8.11-1.el7.x86_64
>>
>>
>> I can find the reason why
>> The logs are attached.
>>
>> Any idea?
>>
>> Thanks
>>
>> --
>> --
>> Jose 

Re: [ovirt-users] oVIRT / GlusterFS / Data (HA)

[David Gossage]

On Tue, Jan 24, 2017 at 4:56 PM, Devin Acosta 
wrote:

>
> I have created an oVIRT 4.0.6 Cluster, it has 2 Compute nodes, and 3
> Dedicated Gluster nodes. The Gluster nodes are configured correctly and
> they have the replica set to 3. I'm trying to figure out when I go to
> attach the Data (Master) domain to the oVIRT manager what is the best
> method to do so in the configuration?  I initially set the mount point to
> be like: gluster01-int:/data, then set in the mount options
> "backup-volfile-servers=gluster02-int:/data,gluster03-int:/data", so I
> understand that will choose another host if the 1st one is down but if i
> was to reboot the 1st Gluster node would that provide HA for my Data
> domain?
>

If you are mounting with gluster fuse then gluster itself will take care of
making sure mount is available still if one node goes down.  The
backup-volfile settings are so that if the main host in config, here being
gluster01-int, is down when it initially tries to mount on a host it has 2
other hosts in the cluster it can try to connect to which otherwise it
wouldn't know about until after the intial mount was made and it became
aware of the other nodes.


> I also configured ctdb with a floating-ip address that floats between all
> 3 Gluster nodes, and I am wondering if I should be pointing the mount to
> that VIP? What is the best solution with dealing with Gluster and keeping
> your mount HA?
>

If you use gluster-fuse I don't think that you need the floating IP.  I
think that would be more used if you mounted via nfs.


>
> --
>
> Devin Acosta
> Red Hat Certified Architect, LinuxStack
> 602-354-1220 <(602)%20354-1220> || de...@linuxguru.co
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] HostedEngine Storage

[David Gossage]

On Thu, Oct 27, 2016 at 10:05 AM, Bryan Sockel 
wrote:

> Hi,
>
> We currently have our HostedEngine VM running on a gluster Replica 3
> storage domain.  Is there a way to modify the mount options in oVirt to
> specify the Backup-Volfile servers?
>
>
I think all steps needed may be found in this thread from August.  Or at
least point you in right direction to test.

http://lists.ovirt.org/pipermail/users/2016-August/042164.html

I myself have not gotten around to trying yet as I just haven't had time to
bring my testing cluster back up.



> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Multi cluster question with regards to storage

[David Gossage]

On Thu, Oct 13, 2016 at 10:26 AM, Beckman, Daniel <
daniel.beck...@ingramcontent.com> wrote:

> Hi David,
>
>
>
> Storage is associated with a particular data center, so if they are on the
> same data center they will have access to the same storage.
>
>
>
> The Red Hat documentation on the commercial supported version RHEV (now
> RHV?) is a lot more comprehensive; the equivalent oVirt documentation is
> pretty sparse and uneven, so it’s understandable you never would have been
> introduced to that concept. You need an active subscription to access the
> KB articles but the basic documentation is freely available and it’s a good
> resource, as most things are going to be common on RHV vs. oVirt:
>
>
>
> https://access.redhat.com/documentation/en/red-hat-
> virtualization?version=4.0/
>

Thanks, I actually have RHEV licensed for some production servers and I
don't now why I didn't log back in to check that doc out as now that you
mentioned it I do recall perusing it long ago and it had decent
explanations on that.

Thanks for the reminder and the answer as well.


>
> Daniel
>
>
>
>
>
>
>
> *From: *<users-boun...@ovirt.org> on behalf of David Gossage <
> dgoss...@carouselchecks.com>
> *Date: *Thursday, October 13, 2016 at 9:08 AM
> *To: *users <users@ovirt.org>
> *Subject: *[ovirt-users] Multi cluster question with regards to storage
>
>
>
> I've not yet found a clean concise answer and so I wanted to ask before I
> start trying to play with things in my test setup and waste a bunch of time
> maybe.
>
>
>
> If I have 2 clusters in one data center each running different cpu types
> (amd/intel) can they both access the same shared storage domain(gluster in
> my case)?
>
>
>
> I'm not interested in live migration between the clusters really, though
> stopping and starting on another would be nice.  I've tried looking at
> various threads and list emails and I've found some mentions of it, but
> usually as an aside while discussing other matters so I could never quite
> say for certain it was working or not.
>
>
>
>
> *David Gossage*
>
> *Carousel Checks Inc.** | System Administrator*
> *Office* 708.613.2284
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Multi cluster question with regards to storage

[David Gossage]

I've not yet found a clean concise answer and so I wanted to ask before I
start trying to play with things in my test setup and waste a bunch of time
maybe.

If I have 2 clusters in one data center each running different cpu types
(amd/intel) can they both access the same shared storage domain(gluster in
my case)?

I'm not interested in live migration between the clusters really, though
stopping and starting on another would be nice.  I've tried looking at
various threads and list emails and I've found some mentions of it, but
usually as an aside while discussing other matters so I could never quite
say for certain it was working or not.


*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hosted engine on gluster problem

[David Gossage]

On Mon, Aug 29, 2016 at 10:47 AM, Simone Tiraboschi <stira...@redhat.com>
wrote:

>
>
> On Fri, Aug 26, 2016 at 8:54 AM, Sandro Bonazzola <sbona...@redhat.com>
> wrote:
>
>>
>>
>> On Tue, Aug 23, 2016 at 8:44 PM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>>
>>> On Fri, Apr 15, 2016 at 8:00 AM, Luiz Claudio Prazeres Goncalves <
>>> luiz...@gmail.com> wrote:
>>>
>>>> I'm not planning to move to ovirt 4 until it gets stable, so would be
>>>> great to backport to 3.6 or ,ideally, gets developed on the next release of
>>>> 3.6 branch. Considering the urgency (its a single point of failure) x
>>>> complexity wouldn't be hard to make the proposed fix.
>>>>
>>>>
>>> Bumping old email sorry.  Looks like https://bugzilla.redhat.com/sh
>>> ow_bug.cgi?id=1298693 was finished against 3.6.7 according to that RFE.
>>>
>>> So does that mean if I add appropriate lines to my
>>>  /etc/ovirt-hosted-engine/hosted-engine.conf the next time I restart
>>> engine and agent/brokers to mount that storage point it will utilize the
>>> backupvol-server features?
>>>
>>> If so are appropriate settings outlined in docs somewhere?
>>>
>>> Running ovirt 3.6.7 and gluster 3.8.2 on centos 7 nodes.
>>>
>>
>> Adding Simone
>>
>
>
> First step, you have to edit
> /etc/ovirt-hosted-engine/hosted-engine.conf on all your hosted-engine
> hosts to ensure that the storage field always point to the same entry
> point (host01 for instance)
> Then on each host you can add something like:
> mnt_options=backupvolfile-server=host02.yourdomain.com:host03.
> <http://ost03.ovirt.forest.go.th/>yourdomain.com,fetch-
> attempts=2,log-level=WARNING,log-file=/var/log/engine_domain.log
>
> Then check the representation of your storage connection in the table
> storage_server_connections of the engine DB and make sure that connection
> refers to the entry point you used in hosted-engine.conf on all your
> hosts, you have lastly to set the value of mount_options also here.
>
> Please tune also the value of network.ping-timeout for your glusterFS volume
> to avoid this:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1319657#c17
>
> You can find other information here:
> https://www.ovirt.org/develop/release-management/features/
> engine/self-hosted-engine-gluster-support/
>
>
Thanks, I'll review all that information.

>
>
>>
>>
>>>
>>>
>>> I'm using today a production environment on top of gluster replica 3 and
>>>> this is the only SPF I have.
>>>>
>>>> Thanks
>>>> Luiz
>>>>
>>>> Em sex, 15 de abr de 2016 03:05, Sandro Bonazzola <sbona...@redhat.com>
>>>> escreveu:
>>>>
>>>>> On Thu, Apr 14, 2016 at 7:35 PM, Nir Soffer <nsof...@redhat.com>
>>>>> wrote:
>>>>>
>>>>>> On Wed, Apr 13, 2016 at 4:34 PM, Luiz Claudio Prazeres Goncalves
>>>>>> <luiz...@gmail.com> wrote:
>>>>>> > Nir, here is the problem:
>>>>>> > https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>>>>>> >
>>>>>> > When you do a hosted-engine --deploy and pick "glusterfs" you don't
>>>>>> have a
>>>>>> > way to define the mount options, therefore, the use of the
>>>>>> > "backupvol-server", however when you create a storage domain from
>>>>>> the UI you
>>>>>> > can, like the attached screen shot.
>>>>>> >
>>>>>> >
>>>>>> > In the hosted-engine --deploy, I would expect a flow which includes
>>>>>> not only
>>>>>> > the "gluster" entrypoint, but also the gluster mount options which
>>>>>> is
>>>>>> > missing today. This option would be optional, but would remove the
>>>>>> single
>>>>>> > point of failure described on the Bug 1298693.
>>>>>> >
>>>>>> > for example:
>>>>>> >
>>>>>> > Existing entry point on the "hosted-engine --deploy" flow
>>>>>> > gluster1.xyz.com:/engine
>>>>>>
>>>>>> I agree, this feature must be supported.
>>>>>>
>>>>>
>>>>> It will, and it's currently targeted to 4.0.
>&g

Re: [ovirt-users] ovirt-ha-agent

[David Gossage]

On Fri, Aug 26, 2016 at 1:38 AM, Renout Gerrits <m...@renout.nl> wrote:

> Depends on your systemd configuration. ovirt-ha-agent and broker daemon's
> both log to stdout and it's own logfile. All messages to stdout will go to
> journald and be forwarded to /var/log/messages (ForwardToSyslog=yes in
> /etc/systemd/journald.conf I think).
> So the ovirt-ha-agent doesn't log to /var/log/messages, journald does. if
> it should log to stdout is another discussion, but maybe there's good
> reason for that, backwards compatibility, don't know.
>
> An easy fix is redirecting the output of the daemon to /dev/null. in
> /usr/lib/systemd/system/ovirt-ha-agent.service add StandardOutput=null to
> the [service] section.
>

This did not start occurring until after I updated ovirt on August 12th. I
jumped a couple updates I think so which update it started with not sure.

Aug 12 21:54:58 Updated: ovirt-vmconsole-1.0.2-1.el7.centos.noarch
Aug 12 21:55:23 Updated: ovirt-vmconsole-host-1.0.2-1.el7.centos.noarch
Aug 12 21:55:34 Updated: ovirt-setup-lib-1.0.1-1.el7.centos.noarch
Aug 12 21:55:42 Updated: libgovirt-0.3.3-1.el7_2.4.x86_64
Aug 12 21:56:37 Updated: ovirt-hosted-engine-ha-1.3.5.7-1.el7.centos.noarch
Aug 12 21:56:37 Updated: ovirt-engine-sdk-python-3.6.7.0-1.el7.centos.noarch
Aug 12 21:56:38 Updated:
ovirt-hosted-engine-setup-1.3.7.2-1.el7.centos.noarch
Aug 12 21:58:46 Updated: 1:ovirt-release36-3.6.7-1.noarch

So is this considered normal behavior that everyone should need to
reconfigure their logging to deal with or did someone leave it going to
stdout for debugging and not turn it back off?  I know I can manipulate
logging myself if I need to, but I'd rather not have to customize that and
then check every update that it's still applied or that I still get logs at
all.



> Renout
>
> On Thu, Aug 25, 2016 at 10:39 PM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> This service seems to be logging to both /var/log/messages
>> and /var/log/ovirt-hosted-engine-ha/agent.log
>>
>> Anything that may be causing that?  Centos7 ovirt 3.6.7
>>
>> MainThread::INFO::2016-08-25 15:38:36,912::ovf_store::109::
>> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>> Extracting Engine VM OVF from the OVF_STORE
>> MainThread::INFO::2016-08-25 15:38:36,976::ovf_store::116::
>> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>> OVF_STORE volume path: /rhev/data-center/mnt/glusterS
>> D/ccgl1.gl.local:HOST1/6a0bca4a-a1be-47d3-be51-64c627
>> 7d1f0f/images/c12c8000-0373-419b-963b-98b04adca760/fb6e250
>> 9-4786-433d-868f-a6303dd69cca
>> MainThread::INFO::2016-08-25 15:38:37,097::config::226::ovi
>> rt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
>> Found an OVF for HE VM, trying to convert
>> MainThread::INFO::2016-08-25 15:38:37,102::config::231::ovi
>> rt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
>> Got vm.conf from OVF_STORE
>> MainThread::INFO::2016-08-25 15:38:37,200::hosted_engine::4
>> 62::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
>> Current state EngineDown (score: 3400)
>> MainThread::INFO::2016-08-25 15:38:37,201::hosted_engine::4
>> 67::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
>> Best remote host ccovirt1.carouselchecks.local (id: 1, score: 3400)
>> MainThread::INFO::2016-08-25 15:38:47,346::hosted_engine::6
>> 13::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
>> Initializing VDSM
>> MainThread::INFO::2016-08-25 15:38:47,439::hosted_engine::6
>> 58::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
>> Connecting the storage
>> MainThread::INFO::2016-08-25 15:38:47,454::storage_server::218::
>> ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
>> Connecting storage server
>> MainThread::INFO::2016-08-25 15:38:47,618::storage_server::222::
>> ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
>> Connecting storage server
>> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.br
>> oker.listener.ConnectionHandler:Connection closed
>>
>> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: INFO:mem_free.MemFree:memFree:
>> 16466
>> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: 
>> INFO:engine_health.CpuLoadNoEngine:VM
>> not on this host
>> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker: INFO:mgmt_bridge.MgmtBridge:Found
>> bridge ovirtmgmt with ports
>> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker: 
>> INFO:cpu_load_no_engine.EngineHealth:VM
>&g

[ovirt-users] ovirt-ha-agent

[David Gossage]

This service seems to be logging to both /var/log/messages
and /var/log/ovirt-hosted-engine-ha/agent.log

Anything that may be causing that?  Centos7 ovirt 3.6.7

MainThread::INFO::2016-08-25
15:38:36,912::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2016-08-25
15:38:36,976::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
OVF_STORE volume path:
/rhev/data-center/mnt/glusterSD/ccgl1.gl.local:HOST1/6a0bca4a-a1be-47d3-be51-64c6277d1f0f/images/c12c8000-0373-419b-963b-98b04adca760/fb6e2509-4786-433d-868f-a6303dd69cca
MainThread::INFO::2016-08-25
15:38:37,097::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Found an OVF for HE VM, trying to convert
MainThread::INFO::2016-08-25
15:38:37,102::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Got vm.conf from OVF_STORE
MainThread::INFO::2016-08-25
15:38:37,200::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 3400)
MainThread::INFO::2016-08-25
15:38:37,201::hosted_engine::467::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host ccovirt1.carouselchecks.local (id: 1, score: 3400)
MainThread::INFO::2016-08-25
15:38:47,346::hosted_engine::613::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
Initializing VDSM
MainThread::INFO::2016-08-25
15:38:47,439::hosted_engine::658::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Connecting the storage
MainThread::INFO::2016-08-25
15:38:47,454::storage_server::218::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
MainThread::INFO::2016-08-25
15:38:47,618::storage_server::222::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
Aug 25 15:38:40 ccovirt3 ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
closed

Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: INFO:mem_free.MemFree:memFree:
16466
Aug 25 15:38:40 ccovirt3 ovirt-ha-broker:
INFO:engine_health.CpuLoadNoEngine:VM not on this host
Aug 25 15:38:45 ccovirt3 ovirt-ha-broker: INFO:mgmt_bridge.MgmtBridge:Found
bridge ovirtmgmt with ports
Aug 25 15:38:45 ccovirt3 ovirt-ha-broker:
INFO:cpu_load_no_engine.EngineHealth:VM not on this host
Aug 25 15:38:45 ccovirt3 ovirt-ha-broker:
INFO:cpu_load_no_engine.EngineHealth:System load total=0.1022,
engine=0., non-engine=0.1022
Aug 25 15:38:47 ccovirt3 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing
VDSM
Aug 25 15:38:47 ccovirt3 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Connecting the
storage
Aug 25 15:38:47 ccovirt3 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
storage server
Aug 25 15:38:47 ccovirt3 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
storage server

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hosted engine on gluster problem

[David Gossage]

On Fri, Apr 15, 2016 at 8:00 AM, Luiz Claudio Prazeres Goncalves <
luiz...@gmail.com> wrote:

> I'm not planning to move to ovirt 4 until it gets stable, so would be
> great to backport to 3.6 or ,ideally, gets developed on the next release of
> 3.6 branch. Considering the urgency (its a single point of failure) x
> complexity wouldn't be hard to make the proposed fix.
>
>
Bumping old email sorry.  Looks like
https://bugzilla.redhat.com/show_bug.cgi?id=1298693 was finished against
3.6.7 according to that RFE.

So does that mean if I add appropriate lines to my
 /etc/ovirt-hosted-engine/hosted-engine.conf the next time I restart engine
and agent/brokers to mount that storage point it will utilize the
backupvol-server features?

If so are appropriate settings outlined in docs somewhere?

Running ovirt 3.6.7 and gluster 3.8.2 on centos 7 nodes.


I'm using today a production environment on top of gluster replica 3 and
> this is the only SPF I have.
>
> Thanks
> Luiz
>
> Em sex, 15 de abr de 2016 03:05, Sandro Bonazzola 
> escreveu:
>
>> On Thu, Apr 14, 2016 at 7:35 PM, Nir Soffer  wrote:
>>
>>> On Wed, Apr 13, 2016 at 4:34 PM, Luiz Claudio Prazeres Goncalves
>>>  wrote:
>>> > Nir, here is the problem:
>>> > https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>>> >
>>> > When you do a hosted-engine --deploy and pick "glusterfs" you don't
>>> have a
>>> > way to define the mount options, therefore, the use of the
>>> > "backupvol-server", however when you create a storage domain from the
>>> UI you
>>> > can, like the attached screen shot.
>>> >
>>> >
>>> > In the hosted-engine --deploy, I would expect a flow which includes
>>> not only
>>> > the "gluster" entrypoint, but also the gluster mount options which is
>>> > missing today. This option would be optional, but would remove the
>>> single
>>> > point of failure described on the Bug 1298693.
>>> >
>>> > for example:
>>> >
>>> > Existing entry point on the "hosted-engine --deploy" flow
>>> > gluster1.xyz.com:/engine
>>>
>>> I agree, this feature must be supported.
>>>
>>
>> It will, and it's currently targeted to 4.0.
>>
>>
>>
>>>
>>> > Missing option on the "hosted-engine --deploy" flow :
>>> > backupvolfile-server=gluster2.xyz.com,fetch-attempts=3,log-
>>> level=WARNING,log-file=/var/log/glusterfs/gluster_engine_domain.log
>>> >
>>> > Sandro, it seems to me a simple solution which can be easily fixed.
>>> >
>>> > What do you think?
>>> >
>>> > Regards
>>> > -Luiz
>>> >
>>> >
>>> >
>>> > 2016-04-13 4:15 GMT-03:00 Sandro Bonazzola :
>>> >>
>>> >>
>>> >>
>>> >> On Tue, Apr 12, 2016 at 6:47 PM, Nir Soffer 
>>> wrote:
>>> >>>
>>> >>> On Tue, Apr 12, 2016 at 3:05 PM, Luiz Claudio Prazeres Goncalves
>>> >>>  wrote:
>>> >>> > Hi Sandro, I've been using gluster with 3 external hosts for a
>>> while
>>> >>> > and
>>> >>> > things are working pretty well, however this single point of
>>> failure
>>> >>> > looks
>>> >>> > like a simple feature to implement,but critical to anyone who
>>> wants to
>>> >>> > use
>>> >>> > gluster on production  . This is not hyperconvergency which has
>>> other
>>> >>> > issues/implications. So , why not have this feature out on 3.6
>>> branch?
>>> >>> > It
>>> >>> > looks like just let vdsm use the 'backupvol-server' option when
>>> >>> > mounting the
>>> >>> > engine domain and make the property tests.
>>> >>>
>>> >>> Can you explain what is the problem, and what is the suggested
>>> solution?
>>> >>>
>>> >>> Engine and vdsm already support the backupvol-server option - you can
>>> >>> define this option in the storage domain options when you create a
>>> >>> gluster
>>> >>> storage domain. With this option vdsm should be able to connect to
>>> >>> gluster
>>> >>> storage domain even if a brick is down.
>>> >>>
>>> >>> If you don't have this option in engine , you probably cannot add it
>>> with
>>> >>> hosted
>>> >>> engine setup, since for editing it you must put the storage domain in
>>> >>> maintenance
>>> >>> and if you do this the engine vm will be killed :-) This is is one of
>>> >>> the issues with
>>> >>> engine managing the storage domain it runs on.
>>> >>>
>>> >>> I think the best way to avoid this issue, is to add a DNS entry
>>> >>> providing the addresses
>>> >>> of all the gluster bricks, and use this address for the gluster
>>> >>> storage domain. This way
>>> >>> the glusterfs mount helper can mount the domain even if one of the
>>> >>> gluster bricks
>>> >>> are down.
>>> >>>
>>> >>> Again, we will need some magic from the hosted engine developers to
>>> >>> modify the
>>> >>> address of the hosted engine gluster domain on existing system.
>>> >>
>>> >>
>>> >> Magic won't happen without a bz :-) please open one describing what's
>>> >> requested.
>>> >>
>>> >>
>>> >>>
>>> >>>
>>> >>> Nir
>>> >>>
>>> >>> >
>>> >>> > Could you add this feature to the next release of 3.6 branch?
>>> >>> >
>>> >>> > 

Re: [ovirt-users] General system updates policy

[David Gossage]

On Sun, Aug 14, 2016 at 5:49 AM, Yaniv Dary <yd...@redhat.com> wrote:

> We try to keep up with latest releases, so it should be quite safe.
>

Ok, thanks


>
> Yaniv Dary
> Technical Product Manager
> Red Hat Israel Ltd.
> 34 Jerusalem Road
> Building A, 4th floor
> Ra'anana, Israel 4350109
>
> Tel : +972 (9) 7692306
> 8272306
> Email: yd...@redhat.com
> IRC : ydary
>
>
> On Fri, Aug 12, 2016 at 9:01 PM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> Is it considered safe from an ovirt stability standpoint to apply yum
>> updates to the hosts and engine as centos/redhat release them?
>>
>> For example the recent https://rhn.redhat.com/errata/RHSA-2016-1606.html
>> update for qemu.  If I run yum update and update that is their any worry
>> for me that it has not been tested with oVirt and so versions will break
>> something.  Same with kernels, libraries etc..
>>
>>
>> *David Gossage*
>> *Carousel Checks Inc. | System Administrator*
>> *Office* 708.613.2284
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

On Sat, Aug 13, 2016 at 11:00 AM, David Gossage <dgoss...@carouselchecks.com
> wrote:

> On Sat, Aug 13, 2016 at 8:19 AM, Scott <romra...@gmail.com> wrote:
>
>> Had a chance to upgrade my cluster to Gluster 3.7.14 and can confirm it
>> works for me too where 3.7.12/13 did not.
>>
>> I did find that you should NOT turn off network.remote-dio or turn
>> on performance.strict-o-direct as suggested earlier in the thread.  They
>> will prevent dd (using direct flag) and other things from working
>> properly.  I'd leave those at network.remote-dio=enabled
>> and performance.strict-o-direct=off.
>>
>
> Those were actually just suggested during a testing phase trying to trace
> down the issue.  Neither of those 2 I think have ever been suggested as
> good practice. At least not for VM storage.
>
>
>> Hopefully we can see Gluster 3.7.14 moved out of testing repo soon.
>>
>
Is it still in testing repo? I updated my production cluster I think 2
weeks ago from default repo on centos7.


>> Scott
>>
>> On Tue, Aug 2, 2016 at 9:05 AM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>> So far gluster 3.7.14 seems to have resolved issues at least on my test
>>> box.  dd commands that failed previously now work with sharding on zfs
>>> backend,
>>>
>>> Where before I couldn't even mount a new storage domain it now mounted
>>> and I have a test vm being created.
>>>
>>> Still have to let VM run for a few days and make sure no locking
>>> freezing occurs but looks hopeful so far.
>>>
>>> *David Gossage*
>>> *Carousel Checks Inc. | System Administrator*
>>> *Office* 708.613.2284
>>>
>>> On Tue, Jul 26, 2016 at 8:15 AM, David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
>>>> On Tue, Jul 26, 2016 at 4:37 AM, Krutika Dhananjay <kdhan...@redhat.com
>>>> > wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> 1.  Could you please attach the glustershd logs from all three nodes?
>>>>>
>>>>
>>>> Here are ccgl1 and ccgl2.  as previously mentioned ccgl3 third node was
>>>> down from bad nic so no relevant logs would be on that node.
>>>>
>>>>
>>>>>
>>>>> 2. Also, so far what we know is that the 'Operation not permitted'
>>>>> errors are on the main vm image itself and not its individual shards (ex
>>>>> deb61291-5176-4b81-8315-3f1cf8e3534d). Could you do the following:
>>>>> Get the inode number of 
>>>>> .glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
>>>>> (ls -li) from the first brick. I'll call this number INODE_NUMBER.
>>>>> Execute `find . -inum INODE_NUMBER` from the brick root on first brick
>>>>> to print the hard links against the file in the prev step and share the
>>>>> output.
>>>>>
>>>> [dgossage@ccgl1 ~]$ sudo ls -li /gluster1/BRICK1/1/.glusterfs/
>>>> de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
>>>> 16407 -rw-r--r--. 2 36 36 466 Jun  5 16:52
>>>> /gluster1/BRICK1/1/.glusterfs/de/b6/deb61291-5176-4b81-8315-
>>>> 3f1cf8e3534d
>>>> [dgossage@ccgl1 ~]$ cd /gluster1/BRICK1/1/
>>>> [dgossage@ccgl1 1]$ sudo find . -inum 16407
>>>> ./7c73a8dd-a72e-4556-ac88-7f6813131e64/dom_md/metadata
>>>> ./.glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
>>>>
>>>>
>>>>
>>>>>
>>>>> 3. Did you delete any vms at any point before or after the upgrade?
>>>>>
>>>>
>>>> Immediately before or after on same day pretty sure I deleted nothing.
>>>> During week prior I deleted a few dev vm's that were never setup and some
>>>> the week after upgrade as I was preparing for moving disks off and on
>>>> storage to get them sharded and felt it would be easier to just recreate
>>>> some disks that had no data yet rather than move them off and on later.
>>>>
>>>>>
>>>>> -Krutika
>>>>>
>>>>> On Mon, Jul 25, 2016 at 11:30 PM, David Gossage <
>>>>> dgoss...@carouselchecks.com> wrote:
>>>>>
>>>>>>
>>>>>> On Mon, Jul 25, 2016 at 9:58 AM, Krutika Dhananjay <
>>>>>> kdhan...@redhat.com> wrote:
>>>>>>
>>>>>>> OK, could you try the following:
>>>>>>>
>>

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

On Sat, Aug 13, 2016 at 8:19 AM, Scott <romra...@gmail.com> wrote:

> Had a chance to upgrade my cluster to Gluster 3.7.14 and can confirm it
> works for me too where 3.7.12/13 did not.
>
> I did find that you should NOT turn off network.remote-dio or turn
> on performance.strict-o-direct as suggested earlier in the thread.  They
> will prevent dd (using direct flag) and other things from working
> properly.  I'd leave those at network.remote-dio=enabled
> and performance.strict-o-direct=off.
>

Those were actually just suggested during a testing phase trying to trace
down the issue.  Neither of those 2 I think have ever been suggested as
good practice. At least not for VM storage.


> Hopefully we can see Gluster 3.7.14 moved out of testing repo soon.
>
> Scott
>
> On Tue, Aug 2, 2016 at 9:05 AM, David Gossage <dgoss...@carouselchecks.com
> > wrote:
>
>> So far gluster 3.7.14 seems to have resolved issues at least on my test
>> box.  dd commands that failed previously now work with sharding on zfs
>> backend,
>>
>> Where before I couldn't even mount a new storage domain it now mounted
>> and I have a test vm being created.
>>
>> Still have to let VM run for a few days and make sure no locking freezing
>> occurs but looks hopeful so far.
>>
>> *David Gossage*
>> *Carousel Checks Inc. | System Administrator*
>> *Office* 708.613.2284
>>
>> On Tue, Jul 26, 2016 at 8:15 AM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>> On Tue, Jul 26, 2016 at 4:37 AM, Krutika Dhananjay <kdhan...@redhat.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> 1.  Could you please attach the glustershd logs from all three nodes?
>>>>
>>>
>>> Here are ccgl1 and ccgl2.  as previously mentioned ccgl3 third node was
>>> down from bad nic so no relevant logs would be on that node.
>>>
>>>
>>>>
>>>> 2. Also, so far what we know is that the 'Operation not permitted'
>>>> errors are on the main vm image itself and not its individual shards (ex
>>>> deb61291-5176-4b81-8315-3f1cf8e3534d). Could you do the following:
>>>> Get the inode number of 
>>>> .glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
>>>> (ls -li) from the first brick. I'll call this number INODE_NUMBER.
>>>> Execute `find . -inum INODE_NUMBER` from the brick root on first brick
>>>> to print the hard links against the file in the prev step and share the
>>>> output.
>>>>
>>> [dgossage@ccgl1 ~]$ sudo ls -li /gluster1/BRICK1/1/.glusterfs/
>>> de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
>>> 16407 -rw-r--r--. 2 36 36 466 Jun  5 16:52 /gluster1/BRICK1/1/.glusterfs/
>>> de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
>>> [dgossage@ccgl1 ~]$ cd /gluster1/BRICK1/1/
>>> [dgossage@ccgl1 1]$ sudo find . -inum 16407
>>> ./7c73a8dd-a72e-4556-ac88-7f6813131e64/dom_md/metadata
>>> ./.glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
>>>
>>>
>>>
>>>>
>>>> 3. Did you delete any vms at any point before or after the upgrade?
>>>>
>>>
>>> Immediately before or after on same day pretty sure I deleted nothing.
>>> During week prior I deleted a few dev vm's that were never setup and some
>>> the week after upgrade as I was preparing for moving disks off and on
>>> storage to get them sharded and felt it would be easier to just recreate
>>> some disks that had no data yet rather than move them off and on later.
>>>
>>>>
>>>> -Krutika
>>>>
>>>> On Mon, Jul 25, 2016 at 11:30 PM, David Gossage <
>>>> dgoss...@carouselchecks.com> wrote:
>>>>
>>>>>
>>>>> On Mon, Jul 25, 2016 at 9:58 AM, Krutika Dhananjay <
>>>>> kdhan...@redhat.com> wrote:
>>>>>
>>>>>> OK, could you try the following:
>>>>>>
>>>>>> i. Set network.remote-dio to off
>>>>>> # gluster volume set  network.remote-dio off
>>>>>>
>>>>>> ii. Set performance.strict-o-direct to on
>>>>>> # gluster volume set  performance.strict-o-direct on
>>>>>>
>>>>>> iii. Stop the affected vm(s) and start again
>>>>>>
>>>>>> and tell me if you notice any improvement?
>>>>>>
>>>>>>
>>>>> Previous instll I had issue with is still on gluster 3

[ovirt-users] General system updates policy

[David Gossage]

Is it considered safe from an ovirt stability standpoint to apply yum
updates to the hosts and engine as centos/redhat release them?

For example the recent https://rhn.redhat.com/errata/RHSA-2016-1606.html
update for qemu.  If I run yum update and update that is their any worry
for me that it has not been tested with oVirt and so versions will break
something.  Same with kernels, libraries etc..


*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

So far gluster 3.7.14 seems to have resolved issues at least on my test
box.  dd commands that failed previously now work with sharding on zfs
backend,

Where before I couldn't even mount a new storage domain it now mounted and
I have a test vm being created.

Still have to let VM run for a few days and make sure no locking freezing
occurs but looks hopeful so far.

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284

On Tue, Jul 26, 2016 at 8:15 AM, David Gossage <dgoss...@carouselchecks.com>
wrote:

> On Tue, Jul 26, 2016 at 4:37 AM, Krutika Dhananjay <kdhan...@redhat.com>
> wrote:
>
>> Hi,
>>
>> 1.  Could you please attach the glustershd logs from all three nodes?
>>
>
> Here are ccgl1 and ccgl2.  as previously mentioned ccgl3 third node was
> down from bad nic so no relevant logs would be on that node.
>
>
>>
>> 2. Also, so far what we know is that the 'Operation not permitted' errors
>> are on the main vm image itself and not its individual shards (ex
>> deb61291-5176-4b81-8315-3f1cf8e3534d). Could you do the following:
>> Get the inode number of
>> .glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d (ls -li) from the
>> first brick. I'll call this number INODE_NUMBER.
>> Execute `find . -inum INODE_NUMBER` from the brick root on first brick to
>> print the hard links against the file in the prev step and share the output.
>>
> [dgossage@ccgl1 ~]$ sudo ls -li
> /gluster1/BRICK1/1/.glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
> 16407 -rw-r--r--. 2 36 36 466 Jun  5 16:52
> /gluster1/BRICK1/1/.glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
> [dgossage@ccgl1 ~]$ cd /gluster1/BRICK1/1/
> [dgossage@ccgl1 1]$ sudo find . -inum 16407
> ./7c73a8dd-a72e-4556-ac88-7f6813131e64/dom_md/metadata
> ./.glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
>
>
>
>>
>> 3. Did you delete any vms at any point before or after the upgrade?
>>
>
> Immediately before or after on same day pretty sure I deleted nothing.
> During week prior I deleted a few dev vm's that were never setup and some
> the week after upgrade as I was preparing for moving disks off and on
> storage to get them sharded and felt it would be easier to just recreate
> some disks that had no data yet rather than move them off and on later.
>
>>
>> -Krutika
>>
>> On Mon, Jul 25, 2016 at 11:30 PM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>>
>>> On Mon, Jul 25, 2016 at 9:58 AM, Krutika Dhananjay <kdhan...@redhat.com>
>>> wrote:
>>>
>>>> OK, could you try the following:
>>>>
>>>> i. Set network.remote-dio to off
>>>> # gluster volume set  network.remote-dio off
>>>>
>>>> ii. Set performance.strict-o-direct to on
>>>> # gluster volume set  performance.strict-o-direct on
>>>>
>>>> iii. Stop the affected vm(s) and start again
>>>>
>>>> and tell me if you notice any improvement?
>>>>
>>>>
>>> Previous instll I had issue with is still on gluster 3.7.11
>>>
>>> My test install of ovirt 3.6.7 and gluster 3.7.13 with 3 bricks on a
>>> locak disk right now isn't allowing me to add the gluster storage at all.
>>>
>>> Keep getting some type of UI error
>>>
>>> 2016-07-25 12:49:09,277 ERROR
>>> [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
>>> (default task-33) [] Permutation name: 430985F23DFC1C8BE1C7FDD91EDAA785
>>> 2016-07-25 12:49:09,277 ERROR
>>> [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
>>> (default task-33) [] Uncaught exception: : java.lang.ClassCastException
>>> at Unknown.ps(
>>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@3837)
>>>at Unknown.ts(
>>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@20)
>>>  at Unknown.vs(
>>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@18)
>>>  at Unknown.iJf(
>>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@19)
>>> at Unknown.Xab(
>>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@48)
>>> at Unknown.P8o(
>>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@4447)
>>&g

Re: [ovirt-users] Cannot find master domain

[David Gossage]

On Thu, Jul 28, 2016 at 11:44 AM, Siavash Safi <siavash.s...@gmail.com>
wrote:

> It seems that dir modes are wrong!?
> [root@node1 ~]# ls -ld /data/brick*/brick*
> drw---. 5 vdsm kvm 107 Jul 28 20:13 /data/brick1/brick1
> drw---. 5 vdsm kvm  82 Jul 27 23:08 /data/brick2/brick2
> [root@node2 ~]# ls -ld /data/brick*/brick*
> drwxr-xr-x. 5 vdsm kvm 107 Apr 26 19:33 /data/brick1/brick1
> drw---. 5 vdsm kvm  82 Jul 27 23:08 /data/brick2/brick2
> drw---. 5 vdsm kvm 107 Jul 28 20:13 /data/brick3/brick3
> [root@node3 ~]# ls -ld /data/brick*/brick*
> drw---. 5 vdsm kvm 107 Jul 28 20:10 /data/brick1/brick1
> drw---. 5 vdsm kvm  82 Jul 27 23:08 /data/brick2/brick2
>

That would probably do it.  kvm does read access.  plus the lack of x on
directories isnt great either.  I'd think since they are the bricks you
could maybe manually chmod them appropriately 755.  I "think" gluster is
only tracking files under /data/brick1/brick1/* not /data/brick1/brick1
itself.


/data/brick3/brick3 is the only "new" one?  Wonder if it could be a bug
from the brick move in some way.  Might be something worth posting to
gluster list about.

>
> On Thu, Jul 28, 2016 at 9:06 PM Sahina Bose <sab...@redhat.com> wrote:
>
>>
>>
>> - Original Message -
>> > From: "Siavash Safi" <siavash.s...@gmail.com>
>> > To: "Sahina Bose" <sab...@redhat.com>
>> > Cc: "David Gossage" <dgoss...@carouselchecks.com>, "users" <
>> users@ovirt.org>, "Nir Soffer" <nsof...@redhat.com>,
>> > "Allon Mureinik" <amure...@redhat.com>
>> > Sent: Thursday, July 28, 2016 9:04:32 PM
>> > Subject: Re: [ovirt-users] Cannot find master domain
>> >
>> > Please check the attachment.
>>
>> Nothing out of place in the mount logs.
>>
>> Can you ensure the brick dir permissions are vdsm:kvm - even for the
>> brick that was replaced?
>>
>> >
>> > On Thu, Jul 28, 2016 at 7:46 PM Sahina Bose <sab...@redhat.com> wrote:
>> >
>> > >
>> > >
>> > > - Original Message -
>> > > > From: "Siavash Safi" <siavash.s...@gmail.com>
>> > > > To: "Sahina Bose" <sab...@redhat.com>
>> > > > Cc: "David Gossage" <dgoss...@carouselchecks.com>, "users" <
>> > > users@ovirt.org>
>> > > > Sent: Thursday, July 28, 2016 8:35:18 PM
>> > > > Subject: Re: [ovirt-users] Cannot find master domain
>> > > >
>> > > > [root@node1 ~]# ls -ld /rhev/data-center/mnt/glusterSD/
>> > > > drwxr-xr-x. 2 vdsm kvm 6 Jul 28 19:28
>> /rhev/data-center/mnt/glusterSD/
>> > > > [root@node1 ~]# getfacl /rhev/data-center/mnt/glusterSD/
>> > > > getfacl: Removing leading '/' from absolute path names
>> > > > # file: rhev/data-center/mnt/glusterSD/
>> > > > # owner: vdsm
>> > > > # group: kvm
>> > > > user::rwx
>> > > > group::r-x
>> > > > other::r-x
>> > > >
>> > >
>> > >
>> > > The ACLs look correct to me. Adding Nir/Allon for insights.
>> > >
>> > > Can you attach the gluster mount logs from this host?
>> > >
>> > >
>> > > > And as I mentioned in another message, the directory is empty.
>> > > >
>> > > > On Thu, Jul 28, 2016 at 7:24 PM Sahina Bose <sab...@redhat.com>
>> wrote:
>> > > >
>> > > > > Error from vdsm log: Permission settings on the specified path do
>> not
>> > > > > allow access to the storage. Verify permission settings on the
>> > > specified
>> > > > > storage path.: 'path = /rhev/data-center/mnt/glusterSD/
>> 172.16.0.11:
>> > > _ovirt'
>> > > > >
>> > > > > I remember another thread about a similar issue - can you check
>> the ACL
>> > > > > settings on the storage path?
>> > > > >
>> > > > > - Original Message -
>> > > > > > From: "Siavash Safi" <siavash.s...@gmail.com>
>> > > > > > To: "David Gossage" <dgoss...@carouselchecks.com>
>> > > > > > Cc: "users" <users@ovirt.org>
>> > > > > > Sent: Thursday, July 28, 2016 7:58:29 PM
>> > > > > > Subject: Re: [ovirt-users] Cannot find maste

Re: [ovirt-users] Cannot find master domain

[David Gossage]

On Thu, Jul 28, 2016 at 10:28 AM, Siavash Safi <siavash.s...@gmail.com>
wrote:

> I created the directory with correct permissions:
> drwxr-xr-x. 2 vdsm kvm 6 Jul 28 19:51
> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt/
>
> It was removed after I tried to activate the storage from web.
>
> Is dir missing on all 3 oVirt nodes?  Did you create on all 3?

When you did test mount with oVirts mount options did permissions on files
after mount look proper?  Can you read/write to mount?


> Engine displays the master storage as inactive:
> [image: oVirt_Engine_Web_Administration.png]
>
>
> On Thu, Jul 28, 2016 at 7:40 PM David Gossage <dgoss...@carouselchecks.com>
> wrote:
>
>> On Thu, Jul 28, 2016 at 10:00 AM, Siavash Safi <siavash.s...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Jul 28, 2016 at 7:19 PM David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
>>>> On Thu, Jul 28, 2016 at 9:38 AM, Siavash Safi <siavash.s...@gmail.com>
>>>> wrote:
>>>>
>>>>> file system: xfs
>>>>> features.shard: off
>>>>>
>>>>
>>>> Ok was just seeing if matched up to the issues latest 3.7.x releases
>>>> have with zfs and sharding but doesn't look like your issue.
>>>>
>>>>  In your logs I see it mounts with thee commands.  What happens if you
>>>> use same to a test dir?
>>>>
>>>>  /usr/bin/mount -t glusterfs -o 
>>>> backup-volfile-servers=172.16.0.12:172.16.0.13
>>>> 172.16.0.11:/ovirt /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt
>>>>
>>>
>>> It mounts successfully:
>>> [root@node1 ~]# /usr/bin/mount -t glusterfs -o
>>> backup-volfile-servers=172.16.0.12:172.16.0.13 172.16.0.11:/ovirt /mnt
>>> [root@node1 ~]# ls /mnt/
>>> 4697fbde-45fb-4f91-ac4c-5516bc59f683  __DIRECT_IO_TEST__
>>>
>>>
>>>> It then umounts it and complains short while later of permissions.
>>>>
>>>> StorageServerAccessPermissionError: Permission settings on the
>>>> specified path do not allow access to the storage. Verify permission
>>>> settings on the specified storage path.: 'path =
>>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt'
>>>>
>>>> Are the permissions of dirs to
>>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt as expected?
>>>>
>>>
>>> /rhev/data-center/mnt/glusterSD/ is empty. Maybe it remove the
>>> directory after failure to cleanup?
>>>
>>
>> Maybe though I don't recall it ever being deleted unless you maybe
>> destroy detach storage. What if you create that directory and permissions
>> appropriately on any node missing then try and activate storage?
>>
>> In engine is it still displaying the master storage domain?
>>
>>
>>> How about on the bricks anything out of place?
>>>>
>>>
>>> I didn't notice anything.
>>>
>>>
>>>> Is gluster still using same options as before?  could it have reset the
>>>> user and group to not be 36?
>>>>
>>>
>>> All options seem to be correct, to make sure I ran "Optimize for Virt
>>> Store" from web.
>>>
>>> Volume Name: ovirt
>>> Type: Distributed-Replicate
>>> Volume ID: b224d9bc-d120-4fe1-b233-09089e5ca0b2
>>> Status: Started
>>> Number of Bricks: 2 x 3 = 6
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 172.16.0.11:/data/brick1/brick1
>>> Brick2: 172.16.0.12:/data/brick3/brick3
>>> Brick3: 172.16.0.13:/data/brick1/brick1
>>> Brick4: 172.16.0.11:/data/brick2/brick2
>>> Brick5: 172.16.0.12:/data/brick2/brick2
>>> Brick6: 172.16.0.13:/data/brick2/brick2
>>> Options Reconfigured:
>>> performance.readdir-ahead: on
>>> nfs.disable: off
>>> user.cifs: enable
>>> auth.allow: *
>>> performance.quick-read: off
>>> performance.read-ahead: off
>>> performance.io-cache: off
>>> performance.stat-prefetch: off
>>> cluster.eager-lock: enable
>>> network.remote-dio: enable
>>> cluster.quorum-type: auto
>>> cluster.server-quorum-type: server
>>> storage.owner-uid: 36
>>> storage.owner-gid: 36
>>> server.allow-insecure: on
>>> network.ping-timeout: 10
>>>
>>>
>>>>> On Thu, Jul 28, 2016 at 7:03 PM David Gossage <
>>>>> dgoss...@carou

Re: [ovirt-users] Cannot find master domain

[David Gossage]

On Thu, Jul 28, 2016 at 10:00 AM, Siavash Safi <siavash.s...@gmail.com>
wrote:

>
>
> On Thu, Jul 28, 2016 at 7:19 PM David Gossage <dgoss...@carouselchecks.com>
> wrote:
>
>> On Thu, Jul 28, 2016 at 9:38 AM, Siavash Safi <siavash.s...@gmail.com>
>> wrote:
>>
>>> file system: xfs
>>> features.shard: off
>>>
>>
>> Ok was just seeing if matched up to the issues latest 3.7.x releases have
>> with zfs and sharding but doesn't look like your issue.
>>
>>  In your logs I see it mounts with thee commands.  What happens if you
>> use same to a test dir?
>>
>>  /usr/bin/mount -t glusterfs -o 
>> backup-volfile-servers=172.16.0.12:172.16.0.13
>> 172.16.0.11:/ovirt /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt
>>
>
> It mounts successfully:
> [root@node1 ~]# /usr/bin/mount -t glusterfs -o
> backup-volfile-servers=172.16.0.12:172.16.0.13 172.16.0.11:/ovirt /mnt
> [root@node1 ~]# ls /mnt/
> 4697fbde-45fb-4f91-ac4c-5516bc59f683  __DIRECT_IO_TEST__
>
>
>> It then umounts it and complains short while later of permissions.
>>
>> StorageServerAccessPermissionError: Permission settings on the specified
>> path do not allow access to the storage. Verify permission settings on the
>> specified storage path.: 'path =
>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt'
>>
>> Are the permissions of dirs to
>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt as expected?
>>
>
> /rhev/data-center/mnt/glusterSD/ is empty. Maybe it remove the directory
> after failure to cleanup?
>

Maybe though I don't recall it ever being deleted unless you maybe destroy
detach storage. What if you create that directory and permissions
appropriately on any node missing then try and activate storage?

In engine is it still displaying the master storage domain?


> How about on the bricks anything out of place?
>>
>
> I didn't notice anything.
>
>
>> Is gluster still using same options as before?  could it have reset the
>> user and group to not be 36?
>>
>
> All options seem to be correct, to make sure I ran "Optimize for Virt
> Store" from web.
>
> Volume Name: ovirt
> Type: Distributed-Replicate
> Volume ID: b224d9bc-d120-4fe1-b233-09089e5ca0b2
> Status: Started
> Number of Bricks: 2 x 3 = 6
> Transport-type: tcp
> Bricks:
> Brick1: 172.16.0.11:/data/brick1/brick1
> Brick2: 172.16.0.12:/data/brick3/brick3
> Brick3: 172.16.0.13:/data/brick1/brick1
> Brick4: 172.16.0.11:/data/brick2/brick2
> Brick5: 172.16.0.12:/data/brick2/brick2
> Brick6: 172.16.0.13:/data/brick2/brick2
> Options Reconfigured:
> performance.readdir-ahead: on
> nfs.disable: off
> user.cifs: enable
> auth.allow: *
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> network.remote-dio: enable
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> storage.owner-uid: 36
> storage.owner-gid: 36
> server.allow-insecure: on
> network.ping-timeout: 10
>
>
>>> On Thu, Jul 28, 2016 at 7:03 PM David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
>>>> On Thu, Jul 28, 2016 at 9:28 AM, Siavash Safi <siavash.s...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 28, 2016 at 6:29 PM David Gossage <
>>>>> dgoss...@carouselchecks.com> wrote:
>>>>>
>>>>>> On Thu, Jul 28, 2016 at 8:52 AM, Siavash Safi <siavash.s...@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Issue: Cannot find master domain
>>>>>>> Changes applied before issue started to happen: replaced
>>>>>>> 172.16.0.12:/data/brick1/brick1 with 172.16.0.12:/data/brick3/brick3,
>>>>>>> did minor package upgrades for vdsm and glusterfs
>>>>>>>
>>>>>>> vdsm log: https://paste.fedoraproject.org/396842/
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Any errrors in glusters brick or server logs?  The client gluster
>>>>>> logs from ovirt?
>>>>>>
>>>>> Brick errors:
>>>>> [2016-07-28 14:03:25.002396] E [MSGID: 113091]
>>>>> [posix.c:178:posix_lookup] 0-ovirt-posix: null gfid for path (null)
>>>>> [2016-07-28 14:03:25.002430] E [MSGID: 113018]
>>>>> [posix.c:1

Re: [ovirt-users] Cannot find master domain

[David Gossage]

On Thu, Jul 28, 2016 at 9:38 AM, Siavash Safi <siavash.s...@gmail.com>
wrote:

> file system: xfs
> features.shard: off
>

Ok was just seeing if matched up to the issues latest 3.7.x releases have
with zfs and sharding but doesn't look like your issue.

 In your logs I see it mounts with thee commands.  What happens if you use
same to a test dir?

 /usr/bin/mount -t glusterfs -o backup-volfile-servers=172.16.0.12:172.16.0.13
172.16.0.11:/ovirt /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt

It then umounts it and complains short while later of permissions.

StorageServerAccessPermissionError: Permission settings on the specified
path do not allow access to the storage. Verify permission settings on the
specified storage path.: 'path =
/rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt'

Are the permissions of dirs to
/rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt
as expected?
How about on the bricks anything out of place?

Is gluster still using same options as before?  could it have reset the
user and group to not be 36?

>
> On Thu, Jul 28, 2016 at 7:03 PM David Gossage <dgoss...@carouselchecks.com>
> wrote:
>
>> On Thu, Jul 28, 2016 at 9:28 AM, Siavash Safi <siavash.s...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Jul 28, 2016 at 6:29 PM David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
>>>> On Thu, Jul 28, 2016 at 8:52 AM, Siavash Safi <siavash.s...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Issue: Cannot find master domain
>>>>> Changes applied before issue started to happen: replaced 
>>>>> 172.16.0.12:/data/brick1/brick1
>>>>> with 172.16.0.12:/data/brick3/brick3, did minor package upgrades for
>>>>> vdsm and glusterfs
>>>>>
>>>>> vdsm log: https://paste.fedoraproject.org/396842/
>>>>>
>>>>
>>>>
>>>> Any errrors in glusters brick or server logs?  The client gluster logs
>>>> from ovirt?
>>>>
>>> Brick errors:
>>> [2016-07-28 14:03:25.002396] E [MSGID: 113091]
>>> [posix.c:178:posix_lookup] 0-ovirt-posix: null gfid for path (null)
>>> [2016-07-28 14:03:25.002430] E [MSGID: 113018]
>>> [posix.c:196:posix_lookup] 0-ovirt-posix: lstat on null failed [Invalid
>>> argument]
>>> (Both repeated many times)
>>>
>>> Server errors:
>>> None
>>>
>>> Client errors:
>>> None
>>>
>>>
>>>>
>>>>> yum log: https://paste.fedoraproject.org/396854/
>>>>>
>>>>
>>>> What version of gluster was running prior to update to 3.7.13?
>>>>
>>> 3.7.11-1 from gluster.org repository(after update ovirt switched to
>>> centos repository)
>>>
>>
>> What file system do your bricks reside on and do you have sharding
>> enabled?
>>
>>
>>>> Did it create gluster mounts on server when attempting to start?
>>>>
>>> As I checked the master domain is not mounted on any nodes.
>>> Restarting vdsmd generated following errors:
>>>
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,661::fileUtils::143::Storage.fileUtils::(createdir) Creating
>>> directory: /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt mode: None
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,661::storageServer::364::Storage.StorageServer.MountConnection::(_get_backup_servers_option)
>>> Using bricks: ['172.16.0.11', '172.16.0.12', '172.16.0.13']
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,662::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/taskset
>>> --cpu-list 0-31 /usr/bin/sudo -n /usr/bin/systemd-run --scope
>>> --slice=vdsm-glusterfs /usr/bin/mount -t glusterfs -o
>>> backup-volfile-servers=172.16.0.12:172.16.0.13 172.16.0.11:/ovirt
>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt (cwd None)
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,789::__init__::318::IOProcessClient::(_run) Starting IOProcess...
>>> jsonrpc.Executor/5::DEBUG::2016-07-28
>>> 18:50:57,802::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/taskset
>>> --cpu-list 0-31 /usr/bin/sudo -n /usr/bin/umount -f -l
>>> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt (cwd None)
>>> jsonrpc.Executor/5::ERROR::2016-07-28
>>> 18:50:57,813::hsm::2473::Storage.HSM::(connectStorageServer) Could not
>>> connect to storageServer
>>> Traceback (most recent call last):
>>>   

Re: [ovirt-users] Cannot find master domain

[David Gossage]

On Thu, Jul 28, 2016 at 9:28 AM, Siavash Safi <siavash.s...@gmail.com>
wrote:

>
>
> On Thu, Jul 28, 2016 at 6:29 PM David Gossage <dgoss...@carouselchecks.com>
> wrote:
>
>> On Thu, Jul 28, 2016 at 8:52 AM, Siavash Safi <siavash.s...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Issue: Cannot find master domain
>>> Changes applied before issue started to happen: replaced 
>>> 172.16.0.12:/data/brick1/brick1
>>> with 172.16.0.12:/data/brick3/brick3, did minor package upgrades for
>>> vdsm and glusterfs
>>>
>>> vdsm log: https://paste.fedoraproject.org/396842/
>>>
>>
>>
>> Any errrors in glusters brick or server logs?  The client gluster logs
>> from ovirt?
>>
> Brick errors:
> [2016-07-28 14:03:25.002396] E [MSGID: 113091] [posix.c:178:posix_lookup]
> 0-ovirt-posix: null gfid for path (null)
> [2016-07-28 14:03:25.002430] E [MSGID: 113018] [posix.c:196:posix_lookup]
> 0-ovirt-posix: lstat on null failed [Invalid argument]
> (Both repeated many times)
>
> Server errors:
> None
>
> Client errors:
> None
>
>
>>
>>> yum log: https://paste.fedoraproject.org/396854/
>>>
>>
>> What version of gluster was running prior to update to 3.7.13?
>>
> 3.7.11-1 from gluster.org repository(after update ovirt switched to
> centos repository)
>

What file system do your bricks reside on and do you have sharding enabled?


>> Did it create gluster mounts on server when attempting to start?
>>
> As I checked the master domain is not mounted on any nodes.
> Restarting vdsmd generated following errors:
>
> jsonrpc.Executor/5::DEBUG::2016-07-28
> 18:50:57,661::fileUtils::143::Storage.fileUtils::(createdir) Creating
> directory: /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt mode: None
> jsonrpc.Executor/5::DEBUG::2016-07-28
> 18:50:57,661::storageServer::364::Storage.StorageServer.MountConnection::(_get_backup_servers_option)
> Using bricks: ['172.16.0.11', '172.16.0.12', '172.16.0.13']
> jsonrpc.Executor/5::DEBUG::2016-07-28
> 18:50:57,662::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/taskset
> --cpu-list 0-31 /usr/bin/sudo -n /usr/bin/systemd-run --scope
> --slice=vdsm-glusterfs /usr/bin/mount -t glusterfs -o
> backup-volfile-servers=172.16.0.12:172.16.0.13 172.16.0.11:/ovirt
> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt (cwd None)
> jsonrpc.Executor/5::DEBUG::2016-07-28
> 18:50:57,789::__init__::318::IOProcessClient::(_run) Starting IOProcess...
> jsonrpc.Executor/5::DEBUG::2016-07-28
> 18:50:57,802::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/taskset
> --cpu-list 0-31 /usr/bin/sudo -n /usr/bin/umount -f -l
> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt (cwd None)
> jsonrpc.Executor/5::ERROR::2016-07-28
> 18:50:57,813::hsm::2473::Storage.HSM::(connectStorageServer) Could not
> connect to storageServer
> Traceback (most recent call last):
>   File "/usr/share/vdsm/storage/hsm.py", line 2470, in connectStorageServer
> conObj.connect()
>   File "/usr/share/vdsm/storage/storageServer.py", line 248, in connect
> six.reraise(t, v, tb)
>   File "/usr/share/vdsm/storage/storageServer.py", line 241, in connect
> self.getMountObj().getRecord().fs_file)
>   File "/usr/share/vdsm/storage/fileSD.py", line 79, in validateDirAccess
> raise se.StorageServerAccessPermissionError(dirPath)
> StorageServerAccessPermissionError: Permission settings on the specified
> path do not allow access to the storage. Verify permission settings on the
> specified storage path.: 'path =
> /rhev/data-center/mnt/glusterSD/172.16.0.11:_ovirt'
> jsonrpc.Executor/5::DEBUG::2016-07-28
> 18:50:57,817::hsm::2497::Storage.HSM::(connectStorageServer) knownSDs: {}
> jsonrpc.Executor/5::INFO::2016-07-28
> 18:50:57,817::logUtils::51::dispatcher::(wrapper) Run and protect:
> connectStorageServer, Return response: {'statuslist': [{'status': 469,
> 'id': u'2d285de3-eede-42aa-b7d6-7b8c6e0667bc'}]}
> jsonrpc.Executor/5::DEBUG::2016-07-28
> 18:50:57,817::task::1191::Storage.TaskManager.Task::(prepare)
> Task=`21487eb4-de9b-47a3-aa37-7dce06533cc9`::finished: {'statuslist':
> [{'status': 469, 'id': u'2d285de3-eede-42aa-b7d6-7b8c6e0667bc'}]}
> jsonrpc.Executor/5::DEBUG::2016-07-28
> 18:50:57,817::task::595::Storage.TaskManager.Task::(_updateState)
> Task=`21487eb4-de9b-47a3-aa37-7dce06533cc9`::moving from state preparing ->
> state finished
>
> I can manually mount the gluster volume on the same server.
>
>
>>
>>
>>> Setup:
>>> engine running on a separate node
>>> 3 x kvm/glusterd nodes
>>>
>>> Status 

Re: [ovirt-users] Cannot find master domain

[David Gossage]

On Thu, Jul 28, 2016 at 8:52 AM, Siavash Safi 
wrote:

> Hi,
>
> Issue: Cannot find master domain
> Changes applied before issue started to happen: replaced 
> 172.16.0.12:/data/brick1/brick1
> with 172.16.0.12:/data/brick3/brick3, did minor package upgrades for vdsm
> and glusterfs
>
> vdsm log: https://paste.fedoraproject.org/396842/
>


Any errrors in glusters brick or server logs?  The client gluster logs from
ovirt?


> yum log: https://paste.fedoraproject.org/396854/
>

What version of gluster was running prior to update to 3.7.13?

Did it create gluster mounts on server when attempting to start?



> Setup:
> engine running on a separate node
> 3 x kvm/glusterd nodes
>
> Status of volume: ovirt
> Gluster process TCP Port  RDMA Port  Online
>  Pid
>
> --
> Brick 172.16.0.11:/data/brick1/brick1   49152 0  Y
> 17304
> Brick 172.16.0.12:/data/brick3/brick3   49155 0  Y
> 9363
> Brick 172.16.0.13:/data/brick1/brick1   49152 0  Y
> 23684
> Brick 172.16.0.11:/data/brick2/brick2   49153 0  Y
> 17323
> Brick 172.16.0.12:/data/brick2/brick2   49153 0  Y
> 9382
> Brick 172.16.0.13:/data/brick2/brick2   49153 0  Y
> 23703
> NFS Server on localhost 2049  0  Y
> 30508
> Self-heal Daemon on localhost   N/A   N/AY
> 30521
> NFS Server on 172.16.0.11   2049  0  Y
> 24999
> Self-heal Daemon on 172.16.0.11 N/A   N/AY
> 25016
> NFS Server on 172.16.0.13   2049  0  Y
> 25379
> Self-heal Daemon on 172.16.0.13 N/A   N/AY
> 25509
>
> Task Status of Volume ovirt
>
> --
> Task : Rebalance
> ID   : 84d5ab2a-275e-421d-842b-928a9326c19a
> Status   : completed
>
> Thanks,
> Siavash
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

On Tue, Jul 26, 2016 at 4:37 AM, Krutika Dhananjay <kdhan...@redhat.com>
wrote:

> Hi,
>
> 1.  Could you please attach the glustershd logs from all three nodes?
>

Here are ccgl1 and ccgl2.  as previously mentioned ccgl3 third node was
down from bad nic so no relevant logs would be on that node.


>
> 2. Also, so far what we know is that the 'Operation not permitted' errors
> are on the main vm image itself and not its individual shards (ex
> deb61291-5176-4b81-8315-3f1cf8e3534d). Could you do the following:
> Get the inode number of
> .glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d (ls -li) from the
> first brick. I'll call this number INODE_NUMBER.
> Execute `find . -inum INODE_NUMBER` from the brick root on first brick to
> print the hard links against the file in the prev step and share the output.
>
[dgossage@ccgl1 ~]$ sudo ls -li
/gluster1/BRICK1/1/.glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
16407 -rw-r--r--. 2 36 36 466 Jun  5 16:52
/gluster1/BRICK1/1/.glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d
[dgossage@ccgl1 ~]$ cd /gluster1/BRICK1/1/
[dgossage@ccgl1 1]$ sudo find . -inum 16407
./7c73a8dd-a72e-4556-ac88-7f6813131e64/dom_md/metadata
./.glusterfs/de/b6/deb61291-5176-4b81-8315-3f1cf8e3534d



>
> 3. Did you delete any vms at any point before or after the upgrade?
>

Immediately before or after on same day pretty sure I deleted nothing.
During week prior I deleted a few dev vm's that were never setup and some
the week after upgrade as I was preparing for moving disks off and on
storage to get them sharded and felt it would be easier to just recreate
some disks that had no data yet rather than move them off and on later.

>
> -Krutika
>
> On Mon, Jul 25, 2016 at 11:30 PM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>>
>> On Mon, Jul 25, 2016 at 9:58 AM, Krutika Dhananjay <kdhan...@redhat.com>
>> wrote:
>>
>>> OK, could you try the following:
>>>
>>> i. Set network.remote-dio to off
>>> # gluster volume set  network.remote-dio off
>>>
>>> ii. Set performance.strict-o-direct to on
>>> # gluster volume set  performance.strict-o-direct on
>>>
>>> iii. Stop the affected vm(s) and start again
>>>
>>> and tell me if you notice any improvement?
>>>
>>>
>> Previous instll I had issue with is still on gluster 3.7.11
>>
>> My test install of ovirt 3.6.7 and gluster 3.7.13 with 3 bricks on a
>> locak disk right now isn't allowing me to add the gluster storage at all.
>>
>> Keep getting some type of UI error
>>
>> 2016-07-25 12:49:09,277 ERROR
>> [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
>> (default task-33) [] Permutation name: 430985F23DFC1C8BE1C7FDD91EDAA785
>> 2016-07-25 12:49:09,277 ERROR
>> [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
>> (default task-33) [] Uncaught exception: : java.lang.ClassCastException
>> at Unknown.ps(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@3837)
>>at Unknown.ts(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@20)
>>  at Unknown.vs(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@18)
>>  at Unknown.iJf(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@19)
>> at Unknown.Xab(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@48)
>> at Unknown.P8o(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@4447)
>>   at Unknown.jQr(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@21)
>> at Unknown.A8o(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@51)
>> at Unknown.u8o(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@101)
>>at Unknown.Eap(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@10718)
>>  at Unknown.p8n(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@161)
>>at Unknown.Cao(
>> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@31)
>> at Unknown.Bap(
>> htt

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

On Mon, Jul 25, 2016 at 3:48 PM, Alexander Wels <aw...@redhat.com> wrote:

> On Monday, July 25, 2016 01:49:32 PM David Gossage wrote:
> > On Mon, Jul 25, 2016 at 1:39 PM, Alexander Wels <aw...@redhat.com>
> wrote:
> > > On Monday, July 25, 2016 01:37:47 PM David Gossage wrote:
> > > > > > My test install of ovirt 3.6.7 and gluster 3.7.13 with 3 bricks
> on a
> > > > >
> > > > > locak
> > > > >
> > > > > > disk right now isn't allowing me to add the gluster storage at
> all.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Keep getting some type of UI error
> > > > >
> > > > > Yes that is definitely a UI error. To get a better stack trace can
> you
> > > > > install the debuginfo
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > yum install ovirt-engine-webadmin-portal-debuginfo
> > > > > ovirt-engine-userportal-debuginfo
> > > > >
> > > > >
> > > > >
> > > > > And recreate the exception, that should give a better stack trace.
> > > >
> > > > Do I need to restart engine?
> > >
> > > Yes, you will need to restart engine before the log starts showing a
> > > better
> > > stack trace.
> >
> > Hopefully more informative.
> >
> >
> > 2016-07-25 13:46:54,701 ERROR
> > [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
> > (default task-33) [] Permutation name: 430985F23DFC1C8BE1C7FDD91EDAA785
> > 2016-07-25 13:46:54,702 ERROR
> > [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
> > (default task-33) [] Uncaught exception: : java.lang.ClassCastException
> > at java.lang.Throwable.fillInStackTrace(Throwable.java:114)
> > [rt.jar:1.8.0_101]
> > at java.lang.Exception.Exception(Exception.java:25)
> > [rt.jar:1.8.0_101]
> > at
> > java.lang.RuntimeException.RuntimeException(RuntimeException.java:25)
> > [rt.jar:1.8.0_101]
> > at
> >
> java.lang.ClassCastException.ClassCastException(ClassCastException.java:23)
> > [rt.jar:1.8.0_101]
> > at com.google.gwt.lang.Cast.dynamicCast(Cast.java:53)
> > at
> >
> org.ovirt.engine.ui.uicommonweb.models.datacenters.DataCenterGuideModel.run(
> > DataCenterGuideModel.java:1679) at
> > org.ovirt.engine.ui.uicompat.Task.$run(Task.java:19)
> > at
> >
> org.ovirt.engine.ui.uicommonweb.models.datacenters.DataCenterGuideModel.$sav
> > eSanStorage(DataCenterGuideModel.java:955) at
> >
> org.ovirt.engine.ui.uicommonweb.models.datacenters.DataCenterGuideModel.$pos
> > tOnAddStorage(DataCenterGuideModel.java:667) at
> >
> org.ovirt.engine.ui.uicommonweb.models.datacenters.DataCenterGuideModel$9$1.
> > onSuccess(DataCenterGuideModel.java:646) at
> >
> org.ovirt.engine.ui.uicommonweb.dataprovider.AsyncDataProvider.$getConfigFro
> > mCache(AsyncDataProvider.java:2853) at
> >
> org.ovirt.engine.ui.uicommonweb.dataprovider.AsyncDataProvider.$getStorageDo
> > mainMaxNameLength(AsyncDataProvider.java:2267) at
> >
> org.ovirt.engine.ui.uicommonweb.models.datacenters.DataCenterGuideModel$9.on
> > Success(DataCenterGuideModel.java:629) at
> > org.ovirt.engine.ui.frontend.Frontend$2.$onSuccess(Frontend.java:244)
> > [frontend.jar:]
> > at
> > org.ovirt.engine.ui.frontend.Frontend$2.onSuccess(Frontend.java:244)
> > [frontend.jar:]
> > at
> >
> org.ovirt.engine.ui.frontend.communication.OperationProcessor$2.$onSuccess(O
> > perationProcessor.java:141) [frontend.jar:]
> > at
> >
> org.ovirt.engine.ui.frontend.communication.OperationProcessor$2.onSuccess(Op
> > erationProcessor.java:141) [frontend.jar:]
> > at
> >
> org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$3$1.$
> > onSuccess(GWTRPCCommunicationProvider.java:161) [frontend.jar:]
> > at
> >
> org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$3$1.o
> > nSuccess(GWTRPCCommunicationProvider.java:161) [frontend.jar:]
> > at
> >
> com.google.gwt.rpc.client.impl.RpcCallbackAdapter.onResponseReceived(RpcCall
> > backAdapter.java:72) [gwt-servlet.jar:]
> > at
> >
> org.ovirt.engine.ui.common.gin.BaseSystemModule$1$1.onResponseReceived(BaseS
> > ystemModule.java:140) at
> >
> com.google.gwt.http.client.Request.$f

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

On Mon, Jul 25, 2016 at 1:39 PM, Alexander Wels <aw...@redhat.com> wrote:

> On Monday, July 25, 2016 01:37:47 PM David Gossage wrote:
> > > > My test install of ovirt 3.6.7 and gluster 3.7.13 with 3 bricks on a
> > >
> > > locak
> > >
> > > > disk right now isn't allowing me to add the gluster storage at all.
> > > >
> > > >
> > > >
> > > > Keep getting some type of UI error
> > >
> > > Yes that is definitely a UI error. To get a better stack trace can you
> > > install the debuginfo
> > >
> > >
> > >
> > >
> > > yum install ovirt-engine-webadmin-portal-debuginfo
> > > ovirt-engine-userportal-debuginfo
> > >
> > >
> > >
> > > And recreate the exception, that should give a better stack trace.
> >
> > Do I need to restart engine?
> >
>
> Yes, you will need to restart engine before the log starts showing a better
> stack trace.
>


Hopefully more informative.


2016-07-25 13:46:54,701 ERROR
[org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
(default task-33) [] Permutation name: 430985F23DFC1C8BE1C7FDD91EDAA785
2016-07-25 13:46:54,702 ERROR
[org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
(default task-33) [] Uncaught exception: : java.lang.ClassCastException
at java.lang.Throwable.fillInStackTrace(Throwable.java:114)
[rt.jar:1.8.0_101]
at java.lang.Exception.Exception(Exception.java:25)
[rt.jar:1.8.0_101]
at
java.lang.RuntimeException.RuntimeException(RuntimeException.java:25)
[rt.jar:1.8.0_101]
at
java.lang.ClassCastException.ClassCastException(ClassCastException.java:23)
[rt.jar:1.8.0_101]
at com.google.gwt.lang.Cast.dynamicCast(Cast.java:53)
at
org.ovirt.engine.ui.uicommonweb.models.datacenters.DataCenterGuideModel.run(DataCenterGuideModel.java:1679)
at org.ovirt.engine.ui.uicompat.Task.$run(Task.java:19)
at
org.ovirt.engine.ui.uicommonweb.models.datacenters.DataCenterGuideModel.$saveSanStorage(DataCenterGuideModel.java:955)
at
org.ovirt.engine.ui.uicommonweb.models.datacenters.DataCenterGuideModel.$postOnAddStorage(DataCenterGuideModel.java:667)
at
org.ovirt.engine.ui.uicommonweb.models.datacenters.DataCenterGuideModel$9$1.onSuccess(DataCenterGuideModel.java:646)
at
org.ovirt.engine.ui.uicommonweb.dataprovider.AsyncDataProvider.$getConfigFromCache(AsyncDataProvider.java:2853)
at
org.ovirt.engine.ui.uicommonweb.dataprovider.AsyncDataProvider.$getStorageDomainMaxNameLength(AsyncDataProvider.java:2267)
at
org.ovirt.engine.ui.uicommonweb.models.datacenters.DataCenterGuideModel$9.onSuccess(DataCenterGuideModel.java:629)
at
org.ovirt.engine.ui.frontend.Frontend$2.$onSuccess(Frontend.java:244)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.Frontend$2.onSuccess(Frontend.java:244)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.communication.OperationProcessor$2.$onSuccess(OperationProcessor.java:141)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.communication.OperationProcessor$2.onSuccess(OperationProcessor.java:141)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$3$1.$onSuccess(GWTRPCCommunicationProvider.java:161)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$3$1.onSuccess(GWTRPCCommunicationProvider.java:161)
[frontend.jar:]
at
com.google.gwt.rpc.client.impl.RpcCallbackAdapter.onResponseReceived(RpcCallbackAdapter.java:72)
[gwt-servlet.jar:]
at
org.ovirt.engine.ui.common.gin.BaseSystemModule$1$1.onResponseReceived(BaseSystemModule.java:140)
at
com.google.gwt.http.client.Request.$fireOnResponseReceived(Request.java:237)
[gwt-servlet.jar:]
at
com.google.gwt.http.client.RequestBuilder$1.onReadyStateChange(RequestBuilder.java:409)
[gwt-servlet.jar:]
at Unknown.(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@65)
at com.google.gwt.core.client.impl.Impl.apply(Impl.java:296)
[gwt-servlet.jar:]
at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:335)
[gwt-servlet.jar:]
at Unknown.(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@54
)
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

>
> >
>
> > My test install of ovirt 3.6.7 and gluster 3.7.13 with 3 bricks on a
> locak
>
> > disk right now isn't allowing me to add the gluster storage at all.
>
> >
>
> > Keep getting some type of UI error
>
> >
>
>
>
> Yes that is definitely a UI error. To get a better stack trace can you
> install the debuginfo
>
>
>
>
> yum install ovirt-engine-webadmin-portal-debuginfo 
> ovirt-engine-userportal-debuginfo
>
>
>
> And recreate the exception, that should give a better stack trace.
>
>
>
Do I need to restart engine?

Installed packages and attempted to create storage again from the guide me
of data center where I received last time and I end up with another red
banner about uncaught exception and to reload page.

ui.log seems about same

 2016-07-25 13:34:03,471 ERROR
[org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
(default task-80) [] Permutation name: 430985F23DFC1C8BE1C7FDD91EDAA785
2016-07-25 13:34:03,471 ERROR
[org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
(default task-80) [] Uncaught exception: : java.lang.ClassCastException
at Unknown.ps(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@3837)
   at Unknown.ts(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@20)
 at Unknown.vs(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@18)
 at Unknown.iJf(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@19)
at Unknown.Xab(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@48)
at Unknown.P8o(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@4447)
  at Unknown.jQr(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@21)
at Unknown.A8o(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@51)
at Unknown.u8o(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@101)
   at Unknown.Eap(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@10718)
 at Unknown.p8n(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@161)
   at Unknown.Cao(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@31)
at Unknown.Bap(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@10469)
 at Unknown.kRn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@49)
at Unknown.nRn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@438)
   at Unknown.eVn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@40)
at Unknown.hVn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@25827)
 at Unknown.MTn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@25)
at Unknown.PTn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@24052)
 at Unknown.KJe(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@21125)
 at Unknown.Izk(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@10384)
 at Unknown.P3(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@137)
at Unknown.g4(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@8271)
   at Unknown.(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@65)
at Unknown._t(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@29)
 at Unknown.du(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@57)
 at Unknown.(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@54
)
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

On Mon, Jul 25, 2016 at 1:00 PM, David Gossage <dgoss...@carouselchecks.com>
wrote:

>
> On Mon, Jul 25, 2016 at 9:58 AM, Krutika Dhananjay <kdhan...@redhat.com>
> wrote:
>
>> OK, could you try the following:
>>
>> i. Set network.remote-dio to off
>> # gluster volume set  network.remote-dio off
>>
>> ii. Set performance.strict-o-direct to on
>> # gluster volume set  performance.strict-o-direct on
>>
>> iii. Stop the affected vm(s) and start again
>>
>> and tell me if you notice any improvement?
>>
>>
> Previous instll I had issue with is still on gluster 3.7.11
>
> My test install of ovirt 3.6.7 and gluster 3.7.13 with 3 bricks on a locak
> disk right now isn't allowing me to add the gluster storage at all.
>
> Keep getting some type of UI error
>
> 2016-07-25 12:49:09,277 ERROR
> [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
> (default task-33) [] Permutation name: 430985F23DFC1C8BE1C7FDD91EDAA785
> 2016-07-25 12:49:09,277 ERROR
> [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
> (default task-33) [] Uncaught exception: : java.lang.ClassCastException
> at Unknown.ps(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@3837)
>at Unknown.ts(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@20)
>  at Unknown.vs(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@18)
>  at Unknown.iJf(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@19)
> at Unknown.Xab(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@48)
> at Unknown.P8o(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@4447)
>   at Unknown.jQr(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@21)
> at Unknown.A8o(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@51)
> at Unknown.u8o(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@101)
>at Unknown.Eap(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@10718)
>  at Unknown.p8n(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@161)
>at Unknown.Cao(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@31)
> at Unknown.Bap(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@10469)
>  at Unknown.kRn(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@49)
> at Unknown.nRn(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@438)
>at Unknown.eVn(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@40)
> at Unknown.hVn(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@25827)
>  at Unknown.MTn(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@25)
> at Unknown.PTn(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@24052)
>  at Unknown.KJe(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@21125)
>  at Unknown.Izk(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@10384)
>  at Unknown.P3(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@137)
> at Unknown.g4(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@8271)
>at Unknown.(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@65)
> at Unknown._t(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@29)
>  at Unknown.du(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@57)
>  at Unknown.(
> https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD9

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

On Mon, Jul 25, 2016 at 9:58 AM, Krutika Dhananjay <kdhan...@redhat.com>
wrote:

> OK, could you try the following:
>
> i. Set network.remote-dio to off
> # gluster volume set  network.remote-dio off
>
> ii. Set performance.strict-o-direct to on
> # gluster volume set  performance.strict-o-direct on
>
> iii. Stop the affected vm(s) and start again
>
> and tell me if you notice any improvement?
>
>
Previous instll I had issue with is still on gluster 3.7.11

My test install of ovirt 3.6.7 and gluster 3.7.13 with 3 bricks on a locak
disk right now isn't allowing me to add the gluster storage at all.

Keep getting some type of UI error

2016-07-25 12:49:09,277 ERROR
[org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
(default task-33) [] Permutation name: 430985F23DFC1C8BE1C7FDD91EDAA785
2016-07-25 12:49:09,277 ERROR
[org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService]
(default task-33) [] Uncaught exception: : java.lang.ClassCastException
at Unknown.ps(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@3837)
   at Unknown.ts(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@20)
 at Unknown.vs(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@18)
 at Unknown.iJf(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@19)
at Unknown.Xab(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@48)
at Unknown.P8o(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@4447)
  at Unknown.jQr(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@21)
at Unknown.A8o(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@51)
at Unknown.u8o(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@101)
   at Unknown.Eap(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@10718)
 at Unknown.p8n(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@161)
   at Unknown.Cao(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@31)
at Unknown.Bap(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@10469)
 at Unknown.kRn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@49)
at Unknown.nRn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@438)
   at Unknown.eVn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@40)
at Unknown.hVn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@25827)
 at Unknown.MTn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@25)
at Unknown.PTn(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@24052)
 at Unknown.KJe(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@21125)
 at Unknown.Izk(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@10384)
 at Unknown.P3(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@137)
at Unknown.g4(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@8271)
   at Unknown.(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@65)
at Unknown._t(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@29)
 at Unknown.du(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@57)
 at Unknown.(
https://ccengine2.carouselchecks.local/ovirt-engine/webadmin/430985F23DFC1C8BE1C7FDD91EDAA785.cache.html@54
)


> -Krutika
>
> On Mon, Jul 25, 2016 at 4:57 PM, Samuli Heinonen <samp...@neutraali.net>
> wrote:
>
>> Hi,
>>
>> > On 25 Jul 2016, at 12:34, David Gossage <dgoss...@carouselchecks.com>
>> wrote:
>> >
>> > On Mon, Jul 25, 2016 at 1:01 AM, Krutika Dhananjay <kdhan...@redhat.com>
>> wrote:
>> > Hi,
>> >
>> > Thanks for the logs. So I have identified one is

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

On Mon, Jul 25, 2016 at 1:01 AM, Krutika Dhananjay <kdhan...@redhat.com>
wrote:

> Hi,
>
> Thanks for the logs. So I have identified one issue from the logs for
> which the fix is this: http://review.gluster.org/#/c/14669/. Because of a
> bug in the code, ENOENT was getting converted to EPERM and being propagated
> up the stack causing the reads to bail out early with 'Operation not
> permitted' errors.
> I still need to find out two things:
> i) why there was a readv() sent on a non-existent (ENOENT) file (this is
> important since some of the other users have not faced or reported this
> issue on gluster-users with 3.7.13)
> ii) need to see if there's a way to work around this issue.
>
> Do you mind sharing the steps needed to be executed to run into this
> issue? This is so that we can apply our patches, test and ensure they fix
> the problem.
>

Well after upgrade of gluster all I did was start ovirt hosts up which
launched and started their ha-agent and broker processes.  I don't believe
I started getting any errors till it mounted GLUSTER1.  I had enabled
sharding but had no sharded disk images yet.  Not sure if the check for
shards would have caused that.  Unfortunately I can't just update this
cluster and try and see what caused it as it has sme VM's users expect to
be available in few hours.

I can see if I can get my test setup to recreate it.  I think I'll need to
de-activate data center so I can detach the storage thats on xfs and attach
the one thats over zfs with sharding enabled.  My test is 3 bricks on same
local machine, with 3 different volumes but I think im running into sanlock
issue or something as it won't mount more than one volume that was created
locally.


-Krutika
>
> On Fri, Jul 22, 2016 at 7:17 PM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> Trimmed out the logs to just about when I was shutting down ovirt servers
>> for updates which was 14:30 UTC 2016-07-09
>>
>> Pre-update settings were
>>
>> Volume Name: GLUSTER1
>> Type: Replicate
>> Volume ID: 167b8e57-28c3-447a-95cc-8410cbdf3f7f
>> Status: Started
>> Number of Bricks: 1 x 3 = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: ccgl1.gl.local:/gluster1/BRICK1/1
>> Brick2: ccgl2.gl.local:/gluster1/BRICK1/1
>> Brick3: ccgl3.gl.local:/gluster1/BRICK1/1
>> Options Reconfigured:
>> performance.readdir-ahead: on
>> storage.owner-uid: 36
>> storage.owner-gid: 36
>> performance.quick-read: off
>> performance.read-ahead: off
>> performance.io-cache: off
>> performance.stat-prefetch: off
>> cluster.eager-lock: enable
>> network.remote-dio: enable
>> cluster.quorum-type: auto
>> cluster.server-quorum-type: server
>> server.allow-insecure: on
>> cluster.self-heal-window-size: 1024
>> cluster.background-self-heal-count: 16
>> performance.strict-write-ordering: off
>> nfs.disable: on
>> nfs.addr-namelookup: off
>> nfs.enable-ino32: off
>>
>> At the time of updates ccgl3 was offline from bad nic on server but had
>> been so for about a week with no issues in volume
>>
>> Shortly after update I added these settings to enable sharding but did
>> not as of yet have any VM images sharded.
>> features.shard-block-size: 64MB
>> features.shard: on
>>
>>
>>
>>
>> *David Gossage*
>> *Carousel Checks Inc. | System Administrator*
>> *Office* 708.613.2284
>>
>> On Fri, Jul 22, 2016 at 5:00 AM, Krutika Dhananjay <kdhan...@redhat.com>
>> wrote:
>>
>>> Hi David,
>>>
>>> Could you also share the brick logs from the affected volume? They're
>>> located at
>>> /var/log/glusterfs/bricks/.log.
>>>
>>> Also, could you share the volume configuration (output of `gluster
>>> volume info `) for the affected volume(s) AND at the time you actually
>>> saw this issue?
>>>
>>> -Krutika
>>>
>>>
>>>
>>>
>>> On Thu, Jul 21, 2016 at 11:23 PM, David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
>>>> On Thu, Jul 21, 2016 at 11:47 AM, Scott <romra...@gmail.com> wrote:
>>>>
>>>>> Hi David,
>>>>>
>>>>> My backend storage is ZFS.
>>>>>
>>>>> I thought about moving from FUSE to NFS mounts for my Gluster volumes
>>>>> to help test.  But since I use hosted engine this would be a real pain.
>>>>> Its difficult to modify the storage domain type/path in the
>>>>> hosted-engine.conf.  And I don't want to go through the process of
>>>>> re-depl

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

On Thu, Jul 21, 2016 at 1:24 PM, Karli Sjöberg <karli.sjob...@slu.se> wrote:

>
> Den 21 jul 2016 7:54 em skrev David Gossage <dgoss...@carouselchecks.com>:
> >
> > On Thu, Jul 21, 2016 at 11:47 AM, Scott <romra...@gmail.com> wrote:
> >>
> >> Hi David,
> >>
> >> My backend storage is ZFS.
> >>
> >> I thought about moving from FUSE to NFS mounts for my Gluster volumes
> to help test.  But since I use hosted engine this would be a real pain.
> Its difficult to modify the storage domain type/path in the
> hosted-engine.conf.  And I don't want to go through the process of
> re-deploying hosted engine.
> >>
> >
> > I found this
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1347553
> >
> > Not sure if related.
> >
> > But I also have zfs backend, another user in gluster mailing list had
> issues and used zfs backend although she used proxmox and got it working by
> changing disk to writeback cache I think it was.
>
> David and Scott,
>
> just out of curiousity, what is the OS under ZFS?
>
> Centos 7


> /K
>
> >
> > I also use hosted engine, but I run my gluster volume for HE actually on
> a LVM separate from zfs on xfs and if i recall it did not have the issues
> my gluster on zfs did.  I'm wondering now if the issue was zfs settings.
> >
> > Hopefully should have a test machone up soon I can play around with more.
> >
> >> Scott
> >>
> >> On Thu, Jul 21, 2016 at 11:36 AM David Gossage <
> dgoss...@carouselchecks.com> wrote:
> >>>
> >>> What back end storage do you run gluster on?  xfs/zfs/ext4 etc?
> >>>
> >>> David Gossage
> >>> Carousel Checks Inc. | System Administrator
> >>> Office 708.613.2284
> >>>
> >>> On Thu, Jul 21, 2016 at 8:18 AM, Scott <romra...@gmail.com> wrote:
> >>>>
> >>>> I get similar problems with oVirt 4.0.1 and hosted engine.  After
> upgrading all my hosts to Gluster 3.7.13 (client and server), I get the
> following:
> >>>>
> >>>> $ sudo hosted-engine --set-maintenance --mode=none
> >>>> Traceback (most recent call last):
> >>>>   File "/usr/lib64/python2.7/runpy.py", line 162, in
> _run_module_as_main
> >>>> "__main__", fname, loader, pkg_name)
> >>>>   File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
> >>>> exec code in run_globals
> >>>>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/set_maintenance.py",
> line 73, in 
> >>>> if not maintenance.set_mode(sys.argv[1]):
> >>>>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/set_maintenance.py",
> line 61, in set_mode
> >>>> value=m_global,
> >>>>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 259, in set_maintenance_mode
> >>>> str(value))
> >>>>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 204, in set_global_md_flag
> >>>> all_stats = broker.get_stats_from_storage(service)
> >>>>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 232, in get_stats_from_storage
> >>>> result = self._checked_communicate(request)
> >>>>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 260, in _checked_communicate
> >>>> .format(message or response))
> >>>> ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed:
> failed to read metadata: [Errno 1] Operation not permitted
> >>>>
> >>>> If I only upgrade one host, then things will continue to work but my
> nodes are constantly healing shards.  My logs are also flooded with:
> >>>>
> >>>> [2016-07-21 13:15:14.137734] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274714: READ => -1 gfid=4
> >>>> 41f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
> permitted)
> >>>> The message "W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-0: remote
> operation failed [Operation not permitted]" repeated 6 times between
> [2016-07-21 13:13:24.134985] and [2016-07-21 13:15:04.132226]
> >>>> The message "W [MSGID: 114031]
>

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

On Thu, Jul 21, 2016 at 11:47 AM, Scott <romra...@gmail.com> wrote:

> Hi David,
>
> My backend storage is ZFS.
>
> I thought about moving from FUSE to NFS mounts for my Gluster volumes to
> help test.  But since I use hosted engine this would be a real pain.  Its
> difficult to modify the storage domain type/path in the
> hosted-engine.conf.  And I don't want to go through the process of
> re-deploying hosted engine.
>
>
I found this

https://bugzilla.redhat.com/show_bug.cgi?id=1347553

Not sure if related.

But I also have zfs backend, another user in gluster mailing list had
issues and used zfs backend although she used proxmox and got it working by
changing disk to writeback cache I think it was.

I also use hosted engine, but I run my gluster volume for HE actually on a
LVM separate from zfs on xfs and if i recall it did not have the issues my
gluster on zfs did.  I'm wondering now if the issue was zfs settings.

Hopefully should have a test machone up soon I can play around with more.

Scott
>
> On Thu, Jul 21, 2016 at 11:36 AM David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> What back end storage do you run gluster on?  xfs/zfs/ext4 etc?
>>
>> *David Gossage*
>> *Carousel Checks Inc. | System Administrator*
>> *Office* 708.613.2284
>>
>> On Thu, Jul 21, 2016 at 8:18 AM, Scott <romra...@gmail.com> wrote:
>>
>>> I get similar problems with oVirt 4.0.1 and hosted engine.  After
>>> upgrading all my hosts to Gluster 3.7.13 (client and server), I get the
>>> following:
>>>
>>> $ sudo hosted-engine --set-maintenance --mode=none
>>> Traceback (most recent call last):
>>>   File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
>>> "__main__", fname, loader, pkg_name)
>>>   File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
>>> exec code in run_globals
>>>   File
>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/set_maintenance.py",
>>> line 73, in 
>>> if not maintenance.set_mode(sys.argv[1]):
>>>   File
>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/set_maintenance.py",
>>> line 61, in set_mode
>>> value=m_global,
>>>   File
>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
>>> line 259, in set_maintenance_mode
>>> str(value))
>>>   File
>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
>>> line 204, in set_global_md_flag
>>> all_stats = broker.get_stats_from_storage(service)
>>>   File
>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>> line 232, in get_stats_from_storage
>>> result = self._checked_communicate(request)
>>>   File
>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>> line 260, in _checked_communicate
>>> .format(message or response))
>>> ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed:
>>> failed to read metadata: [Errno 1] Operation not permitted
>>>
>>> If I only upgrade one host, then things will continue to work but my
>>> nodes are constantly healing shards.  My logs are also flooded with:
>>>
>>> [2016-07-21 13:15:14.137734] W [fuse-bridge.c:2227:fuse_readv_cbk]
>>> 0-glusterfs-fuse: 274714: READ => -1 gfid=4
>>> 41f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
>>> permitted)
>>> The message "W [MSGID: 114031]
>>> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-0: remote
>>> operation failed [Operation not permitted]" repeated 6 times between
>>> [2016-07-21 13:13:24.134985] and [2016-07-21 13:15:04.132226]
>>> The message "W [MSGID: 114031]
>>> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-1: remote
>>> operation failed [Operation not permitted]" repeated 8 times between
>>> [2016-07-21 13:13:34.133116] and [2016-07-21 13:15:14.137178]
>>> The message "W [MSGID: 114031]
>>> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-2: remote
>>> operation failed [Operation not permitted]" repeated 7 times between
>>> [2016-07-21 13:13:24.135071] and [2016-07-21 13:15:14.137666]
>>> [2016-07-21 13:15:24.134647] W [MSGID: 114031]
>>> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-0: remote
>>> operation failed [Operation not permitted]
>

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

What back end storage do you run gluster on?  xfs/zfs/ext4 etc?

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284

On Thu, Jul 21, 2016 at 8:18 AM, Scott <romra...@gmail.com> wrote:

> I get similar problems with oVirt 4.0.1 and hosted engine.  After
> upgrading all my hosts to Gluster 3.7.13 (client and server), I get the
> following:
>
> $ sudo hosted-engine --set-maintenance --mode=none
> Traceback (most recent call last):
>   File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
>   File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/set_maintenance.py",
> line 73, in 
> if not maintenance.set_mode(sys.argv[1]):
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/set_maintenance.py",
> line 61, in set_mode
> value=m_global,
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 259, in set_maintenance_mode
> str(value))
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 204, in set_global_md_flag
> all_stats = broker.get_stats_from_storage(service)
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 232, in get_stats_from_storage
> result = self._checked_communicate(request)
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 260, in _checked_communicate
> .format(message or response))
> ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: failed
> to read metadata: [Errno 1] Operation not permitted
>
> If I only upgrade one host, then things will continue to work but my nodes
> are constantly healing shards.  My logs are also flooded with:
>
> [2016-07-21 13:15:14.137734] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274714: READ => -1 gfid=4
> 41f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
> permitted)
> The message "W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-0: remote
> operation failed [Operation not permitted]" repeated 6 times between
> [2016-07-21 13:13:24.134985] and [2016-07-21 13:15:04.132226]
> The message "W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-1: remote
> operation failed [Operation not permitted]" repeated 8 times between
> [2016-07-21 13:13:34.133116] and [2016-07-21 13:15:14.137178]
> The message "W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-2: remote
> operation failed [Operation not permitted]" repeated 7 times between
> [2016-07-21 13:13:24.135071] and [2016-07-21 13:15:14.137666]
> [2016-07-21 13:15:24.134647] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-0: remote
> operation failed [Operation not permitted]
> [2016-07-21 13:15:24.134764] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-2: remote
> operation failed [Operation not permitted]
> [2016-07-21 13:15:24.134793] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274741: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0038f4 (Operation not
> permitted)
> [2016-07-21 13:15:34.135413] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274756: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
> permitted)
> [2016-07-21 13:15:44.141062] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274818: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0038f4 (Operation not
> permitted)
> [2016-07-21 13:15:54.133582] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-1: remote
> operation failed [Operation not permitted]
> [2016-07-21 13:15:54.133629] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274853: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0036d8 (Operation not
> permitted)
> [2016-07-21 13:16:04.133666] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274879: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
> permitted)
> [2016-07-21 13:16:14.134954] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274894: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0036d8 (Operation not
> permitted)
>
> Scott
>
>
> On Thu, Jul 21, 2016 at 6:57 AM Frank Rothenstein 

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

I'm creating a  test box I can more thoroughly mess with so I can submit to
bugzilla something.  Since my errors all popped up while trying to get
ovirt and gluster functional again rather than thoroughly gather logs and
test my data is kinda sketchy.

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284

On Thu, Jul 21, 2016 at 8:18 AM, Scott <romra...@gmail.com> wrote:

> I get similar problems with oVirt 4.0.1 and hosted engine.  After
> upgrading all my hosts to Gluster 3.7.13 (client and server), I get the
> following:
>
> $ sudo hosted-engine --set-maintenance --mode=none
> Traceback (most recent call last):
>   File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
>   File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/set_maintenance.py",
> line 73, in 
> if not maintenance.set_mode(sys.argv[1]):
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/set_maintenance.py",
> line 61, in set_mode
> value=m_global,
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 259, in set_maintenance_mode
> str(value))
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 204, in set_global_md_flag
> all_stats = broker.get_stats_from_storage(service)
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 232, in get_stats_from_storage
> result = self._checked_communicate(request)
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 260, in _checked_communicate
> .format(message or response))
> ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: failed
> to read metadata: [Errno 1] Operation not permitted
>
> If I only upgrade one host, then things will continue to work but my nodes
> are constantly healing shards.  My logs are also flooded with:
>
> [2016-07-21 13:15:14.137734] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274714: READ => -1 gfid=4
> 41f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
> permitted)
> The message "W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-0: remote
> operation failed [Operation not permitted]" repeated 6 times between
> [2016-07-21 13:13:24.134985] and [2016-07-21 13:15:04.132226]
> The message "W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-1: remote
> operation failed [Operation not permitted]" repeated 8 times between
> [2016-07-21 13:13:34.133116] and [2016-07-21 13:15:14.137178]
> The message "W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-2: remote
> operation failed [Operation not permitted]" repeated 7 times between
> [2016-07-21 13:13:24.135071] and [2016-07-21 13:15:14.137666]
> [2016-07-21 13:15:24.134647] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-0: remote
> operation failed [Operation not permitted]
> [2016-07-21 13:15:24.134764] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-2: remote
> operation failed [Operation not permitted]
> [2016-07-21 13:15:24.134793] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274741: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0038f4 (Operation not
> permitted)
> [2016-07-21 13:15:34.135413] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274756: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
> permitted)
> [2016-07-21 13:15:44.141062] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274818: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0038f4 (Operation not
> permitted)
> [2016-07-21 13:15:54.133582] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-data-client-1: remote
> operation failed [Operation not permitted]
> [2016-07-21 13:15:54.133629] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274853: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0036d8 (Operation not
> permitted)
> [2016-07-21 13:16:04.133666] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274879: READ => -1
> gfid=441f2789-f6b1-4918-a280-1b9905a11429 fd=0x7f19bc0041d0 (Operation not
> permitted)
> [2016-07-21 13:16:14.134954] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 274894: 

[ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[David Gossage]

Anyone running one of recent 3.6.x lines and gluster using 3.7.13?  I am
looking to upgrade gluster from 3.7.11->3.7.13 for some bug fixes, but have
been told by users on gluster mail list due to some gluster changes I'd
need to change the disk parameters to use writeback cache.  Something to do
with aio support being removed.

I believe this could be done with custom parameters?  But I believe strage
tests are done using dd and would they fail with current settings then?
Last upgrade to 3.7.13 I had to rollback to 3.7.11 due to stability isues
where gluster storage would go into down state and always show N/A as space
available/used.  Even if hosts saw storage still and VM's were running on
it on all 3 hosts.

Saw a lot of messages like these that went away once gluster rollback
finished

[2016-07-09 15:27:46.935694] I [fuse-bridge.c:4083:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel
7.22
[2016-07-09 15:27:49.555466] W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote
operation failed [Operation not permitted]
[2016-07-09 15:27:49.556574] W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote
operation failed [Operation not permitted]
[2016-07-09 15:27:49.556659] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 80: READ => -1 gfid=deb61291-5176-4b81-8315-3f1cf8e3534d
fd=0x7f5224002f68 (Operation not permitted)
[2016-07-09 15:27:59.612477] W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-1: remote
operation failed [Operation not permitted]
[2016-07-09 15:27:59.613700] W [MSGID: 114031]
[client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-0: remote
operation failed [Operation not permitted]
[2016-07-09 15:27:59.613781] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 168: READ => -1 gfid=deb61291-5176-4b81-8315-3f1cf8e3534d
fd=0x7f5224002f68 (Operation not permitted)

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Gluster 3.7.12 update

[David Gossage]

I am looking to update my gluster from 3.7.11->3.7.12

Their were some bugs in 3.7.12 in relation to libgfapi, but if I recall
using oVirt 3.6 and CentOS7 traffic should still be using the fuse mount
which would avoid any issues they are having with libgfapi correct?


*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Ovirt system reboot no symlink to storage

[David Gossage]

On Sat, Jun 4, 2016 at 11:02 AM, Nir Soffer <nsof...@redhat.com> wrote:

> On Sat, Jun 4, 2016 at 5:17 PM, David Gossage
> <dgoss...@carouselchecks.com> wrote:
> > This morning I updated my gluster version to 3.7.11 and during this I
> > shutdown ovirt completely and all VM's.  On bringing them back up ovirt
> > seems to have not recreated symlinks to gluster storage
> >
> > Before VM's were created and the path they expected to find the storage
> at
> > as it was created by oVirt was
> >
> >
> /rhev/data-center/0001-0001-0001-0001-00b8/7c73a8dd-a72e-4556-ac88-7f6813131e64
> >
> > which was a symlink to what was mounted at
> > /rhev/data-center/mnt/glusterSD/ccgl1.gl.local:GLUSTER1 once engine
> > activated hosts
> >
> > It never seemed to recreate that symlink though and I ended up just
> doing so
> > manually and after that I could bring up my VM's.
> >
> > If this was caused by some error would I likely find that in the engine
> > log's on the engine vm or on one of the vdsm logs of a host?
> >
> >
> > I'm running oVirt Engine Version: 3.6.1.3-1.el7.centos
>
> Sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1271771
>
> The fix should be available in 3.6.3.
>
> As a general rule, you should run the latest 3.6 version, running
> 3.6.1 is too risky,
> almost as running 3.6.0 :-)
>

I admit it.  I have been lazy.  Been trying to catch up on my updates for
the cluster and just forced myself to come in Saturday morning and do it
starting from storage end.  I'll plan out updating ovirt itself one of
these mornings coming up and see how that goes which I am sure will fix it.

Thanks for pointing the bug report out to me.



> Nir
>
> >
> >
> >
> >
> > David Gossage
> > Carousel Checks Inc. | System Administrator
> > Office 708.613.2284
> >
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Ovirt system reboot no symlink to storage

[David Gossage]

This morning I updated my gluster version to 3.7.11 and during this I
shutdown ovirt completely and all VM's.  On bringing them back up ovirt
seems to have not recreated symlinks to gluster storage

Before VM's were created and the path they expected to find the storage at
as it was created by oVirt was

/rhev/data-center/0001-0001-0001-0001-00b8/7c73a8dd-a72e-4556-ac88-7f6813131e64

which was a symlink to what was mounted
at /rhev/data-center/mnt/glusterSD/ccgl1.gl.local:GLUSTER1 once engine
activated hosts

It never seemed to recreate that symlink though and I ended up just doing
so manually and after that I could bring up my VM's.

If this was caused by some error would I likely find that in the engine
log's on the engine vm or on one of the vdsm logs of a host?


I'm running oVirt Engine Version: 3.6.1.3-1.el7.centos




*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users