Re: [ovirt-users] [Gluster-users] HA storage based on two nodes with one point of failure

2015-06-08 Thread Ravishankar N 



On 06/08/2015 02:38 AM, Юрий Полторацкий wrote:

Hi,

I have made a lab with a config listed below and have got unexpected 
result. Someone, tell me, please, where did I go wrong?


I am testing oVirt. Data Center has two clusters: the first as a 
computing with three nodes (node1, node2, node3); the second as a 
storage (node5, node6) based on glusterfs (replica 2).


I want the storage to be HA. I have read here 
https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/sect-Managing_Split-brain.html 
next:
For a replicated volume with two nodes and one brick on each machine, 
if the server-side quorum is enabled and one of the nodes goes 
offline, the other node will also be taken offline because of the 
quorum configuration. As a result, the high availability provided by 
the replication is ineffective. To prevent this situation, a dummy 
node can be added to the trusted storage pool which does not contain 
any bricks. This ensures that even if one of the nodes which contains 
data goes offline, the other node will remain online. Note that if the 
dummy node and one of the data nodes goes offline, the brick on other 
node will be also be taken offline, and will result in data 
unavailability.


So, I have added my Engine (not self-hosted) as a dummy node without 
a brick and have configured quorum as listed below:

cluster.quorum-type: fixed
cluster.quorum-count: 1
cluster.server-quorum-type: server
cluster.server-quorum-ratio: 51%


Then, I've run a VM and have dropped the network link from node6, 
after one a hour have switched back the link and after a while have 
got a split-brain. But why? No one could write to the brick on node6: 
the VM was running on node3 and node1 was SPM.





It could have happened that after node6 came up, the client(s) saw a 
temporary disconnect of node 5 and a write happened at that time. When 
the node 5 is connected again, we have AFR xattrs on both nodes blaming 
each other, causing split-brain. For a replica 2 setup. it is best to 
set the client-quorum to auto instead of fixed. What this means is that 
the first node of the replica must always be up for writes to be 
permitted. If the first node goes down, the volume becomes read-only.  
For better availability , it would be better to use a replica 3 volume 
with (again with client-quorum set to auto). If you are using glusterfs 
3.7, you can also consider using the arbiter configuration [1] for 
replica 3.


[1] 
https://github.com/gluster/glusterfs/blob/master/doc/features/afr-arbiter-volumes.md


Thanks,
Ravi



Gluster's log from node6:
Июн 07 15:35:06 node6.virt.local etc-glusterfs-glusterd.vol[28491]: 
[2015-06-07 12:35:06.106270] C [MSGID: 106002] 
[glusterd-server-quorum.c:356:glusterd_do_volume_quorum_action] 
0-management: Server quorum lost for volume vol3. Stopping local bricks.
Июн 07 16:30:06 node6.virt.local etc-glusterfs-glusterd.vol[28491]: 
[2015-06-07 13:30:06.261505] C [MSGID: 106003] 
[glusterd-server-quorum.c:351:glusterd_do_volume_quorum_action] 
0-management: Server quorum regained for volume vol3. Starting local 
bricks.



gluster volume heal vol3 info
Brick node5.virt.local:/storage/brick12/
/5d0bb2f3-f903-4349-b6a5-25b549affe5f/dom_md/ids - Is in split-brain

Number of entries: 1

Brick node6.virt.local:/storage/brick13/
/5d0bb2f3-f903-4349-b6a5-25b549affe5f/dom_md/ids - Is in split-brain

Number of entries: 1


gluster volume info vol3

Volume Name: vol3
Type: Replicate
Volume ID: 69ba8c68-6593-41ca-b1d9-40b3be50ac80
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: node5.virt.local:/storage/brick12
Brick2: node6.virt.local:/storage/brick13
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
cluster.server-quorum-type: server
cluster.quorum-type: fixed
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
auth.allow: *
user.cifs: disable
nfs.disable: on
performance.readdir-ahead: on
cluster.quorum-count: 1
cluster.server-quorum-ratio: 51%



06.06.2015 12:09, Юрий Полторацкий пишет:

Hi,

I want to build a HA storage based on two servers. I want that if one 
goes down, my storage will be available in RW mode.


If I will use replica 2, then split-brain can occur. To avoid this I 
would use a quorum. As I understand correctly, I can use quorum on a 
client side, on a server side, or on both. I want to add a dummy node 
without a brick and make such config:


cluster.quorum-type: fixed
cluster.quorum-count: 1
cluster.server-quorum-type: server
cluster.server-quorum-ratio: 51%

I expect that client will have access in RW mode until one brick 
alive. On the other side if server's quorum will not meet, then 
bricks will be RO.


Say, HOST1 with a brick BRICK1, HOST2 with a brick BRICK2, and HOST3 
without a brick.


Once HOST1 lose a network connection, than on this node server quorum 
will not meet

Re: [ovirt-users] Ovirt/Gluster

2015-08-21 Thread Ravishankar N 



On 08/21/2015 04:32 PM, Sander Hoentjen wrote:



On 08/21/2015 11:30 AM, Ravishankar N wrote:



On 08/21/2015 01:21 PM, Sander Hoentjen wrote:



On 08/21/2015 09:28 AM, Ravishankar N wrote:



On 08/20/2015 02:14 PM, Sander Hoentjen wrote:



On 08/19/2015 09:04 AM, Ravishankar N wrote:



On 08/18/2015 04:22 PM, Ramesh Nachimuthu wrote:

+ Ravi from gluster.

Regards,
Ramesh

- Original Message -
From: Sander Hoentjen san...@hoentjen.eu
To: users@ovirt.org
Sent: Tuesday, August 18, 2015 3:30:35 PM
Subject: [ovirt-users] Ovirt/Gluster

Hi,

We are looking for some easy to manage self contained VM 
hosting. Ovirt
with GlusterFS seems to fit that bill perfectly. I installed it 
and then
starting kicking the tires. First results looked promising, but 
now I

can get a VM to pause indefinitely fairly easy:

My setup is 3 hosts that are in a Virt and Gluster cluster. 
Gluster is
setup as replica-3. The gluster export is used as the storage 
domain for

the VM's.


Hi,

What version of gluster and ovirt are you using?

glusterfs-3.7.3-1.el7.x86_64
vdsm-4.16.20-0.el7.centos.x86_64
ovirt-engine-3.5.3.1-1.el7.centos.noarch




Now when I start the VM all is good, performance is good enough 
so we

are happy. I then start bonnie++ to generate some load. I have a VM
running on host 1, host 2 is SPM and all 3 VM's are seeing some 
network

traffic courtesy of gluster.

Now, for fun, suddenly the network on host3 goes bad (iptables 
-I OUTPUT

-m statistic --mode random --probability 0.75 -j REJECT).
Some time later I see the guest has a small hickup, I'm 
guessing that
is when gluster decides host 3 is not allowed to play anymore. 
No big

deal anyway.
After a while 25% of packages just isn't good enough for Ovirt 
anymore,

so the host will be fenced.


I'm not sure what fencing means w.r.t ovirt and what it actually 
fences. As far is gluster is concerned, since only one node is 
blocked, the VM image should still be accessible by the VM 
running on host1.
Fencing means (at least in this case) that the IPMI of the server 
does a power reset.

After a reboot *sometimes* the VM will be
paused, and even after the gluster self-heal is complete it can 
not be

unpaused, has to be restarted.


Could you provide the gluster mount (fuse?) logs and the brick 
logs of all 3 nodes when the VM is paused? That should give us 
some clue.



Logs are attached. Problem was at around 8:15 - 8:20 UTC
This time however the vm stopped even without a reboot of hyp03



The mount logs  (rhev-data-center-mnt-glusterSD*) are indicating 
frequent disconnects to the bricks  with 'clnt_ping_timer_expired', 
'Client-quorum is not met' and 'Read-only file system' messages.
client-quorum is enabled by default for replica 3 volumes. So if 
the mount cannot connect to 2 bricks at least, quorum is lost and 
the gluster volume becomes read-only. That seems to be the reason 
why the VMs are pausing.
I'm not sure if the frequent disconnects are due a flaky network or 
the bricks not responding to the mount's ping timer due to it's 
epoll threads busy with I/O (unlikely). Can you also share the 
output of `gluster volume info volname` ?
The frequent disconnects are probably because I intentionally broke 
the network on hyp03 (dropped 75% of outgoing packets). In my 
opinion this should not affect the VM an hyp02. Am I wrong to think 
that?



For client-quorum, If a client (mount)  cannot connect to the number 
of bricks to achieve quorum, the client becomes read-only. So if the 
client on hyp02 can see itself and 01, it shouldn't be affected.

But it was, and I only broke hyp03.


Beats me then. I see [2015-08-18 15:15:27.922998] W [MSGID: 108001] 
[afr-common.c:4043:afr_notify] 0-VMS-replicate-0: Client-quorum is not 
met on hyp02's mount log but the time stamp is earlier than when you 
say you observed the hang (2015-08-20, around 8:15 - 8:20 UTC?).  (they 
do occur in that time on hyp03 though).






[root@hyp01 ~]# gluster volume info VMS

Volume Name: VMS
Type: Replicate
Volume ID: 9e6657e7-8520-4720-ba9d-78b14a86c8ca
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.99.50.20:/brick/VMS
Brick2: 10.99.50.21:/brick/VMS
Brick3: 10.99.50.22:/brick/VMS
Options Reconfigured:
performance.readdir-ahead: on
nfs.disable: on
user.cifs: disable
auth.allow: *
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server


I see that you have enabled server-quorum too. Since you blocked 
hyp03, the if the glusterd on that node cannot  see the other 2 nodes 
due to iptable rules, it would kill all brick processes. See the 7 
How To Test  section in 
http://www.gluster.org/community/documentation/index.php/Features/Server-quorum 
to get a better idea of server-quorum.


Yes but it should only kill the bricks on hyp03, right? So then why 
does the VM

Re: [ovirt-users] Ovirt/Gluster

2015-08-21 Thread Ravishankar N 



On 08/21/2015 07:57 PM, Sander Hoentjen wrote:

Maybe I should formulate some clear questions:
1) Am I correct in assuming that an issue on of of 3 gluster nodes 
should not cause downtime for VM's on other nodes?


From what I understand, yes. Maybe the ovirt folks can confirm. I can 
tell you this much for sure: If you create a replica 3 volume using 3 
nodes, mount the volume locally on each node, and bring down one node, 
the mounts from the other 2 nodes *must* have read+write access to the 
volume.




2) What can I/we do to fix the issue I am seeing?
3) Can anybody else reproduce my issue?

I'll try and see if I can.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Ovirt/Gluster

2015-08-21 Thread Ravishankar N 



On 08/21/2015 01:21 PM, Sander Hoentjen wrote:



On 08/21/2015 09:28 AM, Ravishankar N wrote:



On 08/20/2015 02:14 PM, Sander Hoentjen wrote:



On 08/19/2015 09:04 AM, Ravishankar N wrote:



On 08/18/2015 04:22 PM, Ramesh Nachimuthu wrote:

+ Ravi from gluster.

Regards,
Ramesh

- Original Message -
From: Sander Hoentjen san...@hoentjen.eu
To: users@ovirt.org
Sent: Tuesday, August 18, 2015 3:30:35 PM
Subject: [ovirt-users] Ovirt/Gluster

Hi,

We are looking for some easy to manage self contained VM hosting. 
Ovirt
with GlusterFS seems to fit that bill perfectly. I installed it 
and then

starting kicking the tires. First results looked promising, but now I
can get a VM to pause indefinitely fairly easy:

My setup is 3 hosts that are in a Virt and Gluster cluster. 
Gluster is
setup as replica-3. The gluster export is used as the storage 
domain for

the VM's.


Hi,

What version of gluster and ovirt are you using?

glusterfs-3.7.3-1.el7.x86_64
vdsm-4.16.20-0.el7.centos.x86_64
ovirt-engine-3.5.3.1-1.el7.centos.noarch




Now when I start the VM all is good, performance is good enough so we
are happy. I then start bonnie++ to generate some load. I have a VM
running on host 1, host 2 is SPM and all 3 VM's are seeing some 
network

traffic courtesy of gluster.

Now, for fun, suddenly the network on host3 goes bad (iptables -I 
OUTPUT

-m statistic --mode random --probability 0.75 -j REJECT).
Some time later I see the guest has a small hickup, I'm guessing 
that

is when gluster decides host 3 is not allowed to play anymore. No big
deal anyway.
After a while 25% of packages just isn't good enough for Ovirt 
anymore,

so the host will be fenced.


I'm not sure what fencing means w.r.t ovirt and what it actually 
fences. As far is gluster is concerned, since only one node is 
blocked, the VM image should still be accessible by the VM running 
on host1.
Fencing means (at least in this case) that the IPMI of the server 
does a power reset.

After a reboot *sometimes* the VM will be
paused, and even after the gluster self-heal is complete it can 
not be

unpaused, has to be restarted.


Could you provide the gluster mount (fuse?) logs and the brick logs 
of all 3 nodes when the VM is paused? That should give us some clue.



Logs are attached. Problem was at around 8:15 - 8:20 UTC
This time however the vm stopped even without a reboot of hyp03



The mount logs  (rhev-data-center-mnt-glusterSD*) are indicating 
frequent disconnects to the bricks  with 'clnt_ping_timer_expired', 
'Client-quorum is not met' and 'Read-only file system' messages.
client-quorum is enabled by default for replica 3 volumes. So if the 
mount cannot connect to 2 bricks at least, quorum is lost and the 
gluster volume becomes read-only. That seems to be the reason why the 
VMs are pausing.
I'm not sure if the frequent disconnects are due a flaky network or 
the bricks not responding to the mount's ping timer due to it's epoll 
threads busy with I/O (unlikely). Can you also share the output of 
`gluster volume info volname` ?
The frequent disconnects are probably because I intentionally broke 
the network on hyp03 (dropped 75% of outgoing packets). In my opinion 
this should not affect the VM an hyp02. Am I wrong to think that?



For client-quorum, If a client (mount)  cannot connect to the number of 
bricks to achieve quorum, the client becomes read-only. So if the client 
on hyp02 can see itself and 01, it shouldn't be affected.




[root@hyp01 ~]# gluster volume info VMS

Volume Name: VMS
Type: Replicate
Volume ID: 9e6657e7-8520-4720-ba9d-78b14a86c8ca
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.99.50.20:/brick/VMS
Brick2: 10.99.50.21:/brick/VMS
Brick3: 10.99.50.22:/brick/VMS
Options Reconfigured:
performance.readdir-ahead: on
nfs.disable: on
user.cifs: disable
auth.allow: *
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server


I see that you have enabled server-quorum too. Since you blocked hyp03, 
the if the glusterd on that node cannot  see the other 2 nodes due to 
iptable rules, it would kill all brick processes. See the 7 How To Test 
 section in 
http://www.gluster.org/community/documentation/index.php/Features/Server-quorum 
to get a better idea of server-quorum.



storage.owner-uid: 36
storage.owner-gid: 36



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Ovirt/Gluster

2015-08-21 Thread Ravishankar N 



On 08/20/2015 02:14 PM, Sander Hoentjen wrote:



On 08/19/2015 09:04 AM, Ravishankar N wrote:



On 08/18/2015 04:22 PM, Ramesh Nachimuthu wrote:

+ Ravi from gluster.

Regards,
Ramesh

- Original Message -
From: Sander Hoentjen san...@hoentjen.eu
To: users@ovirt.org
Sent: Tuesday, August 18, 2015 3:30:35 PM
Subject: [ovirt-users] Ovirt/Gluster

Hi,

We are looking for some easy to manage self contained VM hosting. Ovirt
with GlusterFS seems to fit that bill perfectly. I installed it and 
then

starting kicking the tires. First results looked promising, but now I
can get a VM to pause indefinitely fairly easy:

My setup is 3 hosts that are in a Virt and Gluster cluster. Gluster is
setup as replica-3. The gluster export is used as the storage domain 
for

the VM's.


Hi,

What version of gluster and ovirt are you using?

glusterfs-3.7.3-1.el7.x86_64
vdsm-4.16.20-0.el7.centos.x86_64
ovirt-engine-3.5.3.1-1.el7.centos.noarch




Now when I start the VM all is good, performance is good enough so we
are happy. I then start bonnie++ to generate some load. I have a VM
running on host 1, host 2 is SPM and all 3 VM's are seeing some network
traffic courtesy of gluster.

Now, for fun, suddenly the network on host3 goes bad (iptables -I 
OUTPUT

-m statistic --mode random --probability 0.75 -j REJECT).
Some time later I see the guest has a small hickup, I'm guessing that
is when gluster decides host 3 is not allowed to play anymore. No big
deal anyway.
After a while 25% of packages just isn't good enough for Ovirt anymore,
so the host will be fenced.


I'm not sure what fencing means w.r.t ovirt and what it actually 
fences. As far is gluster is concerned, since only one node is 
blocked, the VM image should still be accessible by the VM running on 
host1.
Fencing means (at least in this case) that the IPMI of the server does 
a power reset.

After a reboot *sometimes* the VM will be
paused, and even after the gluster self-heal is complete it can not be
unpaused, has to be restarted.


Could you provide the gluster mount (fuse?) logs and the brick logs 
of all 3 nodes when the VM is paused? That should give us some clue.



Logs are attached. Problem was at around 8:15 - 8:20 UTC
This time however the vm stopped even without a reboot of hyp03



The mount logs  (rhev-data-center-mnt-glusterSD*) are indicating 
frequent disconnects to the bricks  with 'clnt_ping_timer_expired', 
'Client-quorum is not met' and 'Read-only file system' messages.
client-quorum is enabled by default for replica 3 volumes. So if the 
mount cannot connect to 2 bricks at least, quorum is lost and the 
gluster volume becomes read-only. That seems to be the reason why the 
VMs are pausing.
I'm not sure if the frequent disconnects are due a flaky network or the 
bricks not responding to the mount's ping timer due to it's epoll 
threads busy with I/O (unlikely). Can you also share the output of 
`gluster volume info volname` ?


Regards,
Ravi




Regards,
Ravi


Is there anything I can do to prevent the VM from being paused?

Regards,
Sander

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users






___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Ovirt/Gluster

2015-08-19 Thread Ravishankar N 



On 08/18/2015 04:22 PM, Ramesh Nachimuthu wrote:

+ Ravi from gluster.

Regards,
Ramesh

- Original Message -
From: Sander Hoentjen san...@hoentjen.eu
To: users@ovirt.org
Sent: Tuesday, August 18, 2015 3:30:35 PM
Subject: [ovirt-users] Ovirt/Gluster

Hi,

We are looking for some easy to manage self contained VM hosting. Ovirt
with GlusterFS seems to fit that bill perfectly. I installed it and then
starting kicking the tires. First results looked promising, but now I
can get a VM to pause indefinitely fairly easy:

My setup is 3 hosts that are in a Virt and Gluster cluster. Gluster is
setup as replica-3. The gluster export is used as the storage domain for
the VM's.


Hi,

What version of gluster and ovirt are you using?



Now when I start the VM all is good, performance is good enough so we
are happy. I then start bonnie++ to generate some load. I have a VM
running on host 1, host 2 is SPM and all 3 VM's are seeing some network
traffic courtesy of gluster.

Now, for fun, suddenly the network on host3 goes bad (iptables -I OUTPUT
-m statistic --mode random --probability 0.75 -j REJECT).
Some time later I see the guest has a small hickup, I'm guessing that
is when gluster decides host 3 is not allowed to play anymore. No big
deal anyway.
After a while 25% of packages just isn't good enough for Ovirt anymore,
so the host will be fenced.


I'm not sure what fencing means w.r.t ovirt and what it actually fences. 
As far is gluster is concerned, since only one node is blocked, the VM 
image should still be accessible by the VM running on host1.

After a reboot *sometimes* the VM will be
paused, and even after the gluster self-heal is complete it can not be
unpaused, has to be restarted.


Could you provide the gluster mount (fuse?) logs and the brick logs of 
all 3 nodes when the VM is paused? That should give us some clue.


Regards,
Ravi


Is there anything I can do to prevent the VM from being paused?

Regards,
Sander

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Ha cluster 3 nodes but 1 slow

2015-10-18 Thread Ravishankar N 



On 10/18/2015 07:27 PM, Nicolas LIENARD wrote:

Hey Nil

What about 
https://gluster.readthedocs.org/en/release-3.7.0/Features/afr-arbiter-volumes/ 
?


Regards
Nico


Le 18 octobre 2015 15:12:23 GMT+02:00, Nir Soffer  
a écrit :


On Sat, Oct 17, 2015 at 12:45 PM, Nicolas LIENARD  
wrote:

Hi Currently, i ve 3 nodes, 2 in same DC and a third in
another DC. They are all bridged together through a vpn. I
know a cluster is at least 3 nodes to satisfy the quorum. 




Just adding a 3rd node (without actually using it for 3 way replication) 
might not help in preventing split-brains. gluster has client-quorum and 
server-quorum. Have a look at 
http://comments.gmane.org/gmane.comp.file-systems.gluster.user/22609 for 
some information.


If you are indeed using it as a replica-3, then it is better to have all 
3 nodes in the same DC. gluster clients sends every write() to all 
bricks of the replica (and waits for their responses too), so if one of 
them is in another DC, it might slow writes due to network latency.


My question is to know if i can have my VM balancing on the 2
fast nodes with HA and glusterfs replica 2. 




replica 2 definitely provides HA but if you have more chances of files 
ending in split-brain if you have frequent network disconnects , which 
is why replica 3 with client-quorum set to 'auto' is better for 
preventing split-brains.
arbiter-volumes are a kind of a sweet-spot between replica-2 and 
replica-3 that can prevent split-brains. The link that Nir shared 
describes it and how to create one etc.


Regards,
Ravi


gluster replica 2 is not supported.

Nir



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Cannot mount gluster storage data

2015-09-29 Thread Ravishankar N 



On 09/29/2015 01:14 PM, Jean-Michel FRANCOIS wrote:
No one has an idea how to make a 3.7.4 glusterfs client connect a 
3.6.2 server ?



Newer clients and older servers are not supported. It is advisable to 
have the same version across all machines. If you don't want to upgrade 
your production nodes to 3.7, you could perhaps manually install 3.6 
rpms [1] on your new host after uninstalling the existing one..


[1] http://download.gluster.org/pub/gluster/glusterfs/3.6/

-Ravi

It is a bit strange to get this version incompatibility in the ovirt 
rpm repository.


I found 3.6.2 rpms in another reporsitory, do you think I could try to 
install this version on current ovirt 3.4 installation ?


- Jean-Michel

Le 27/09/2015 18:26, Jean-Michel FRANCOIS a écrit :


Le 27/09/2015 16:26, Atin Mukherjee a écrit :


On 09/25/2015 01:25 PM, Ravishankar N wrote:


On 09/25/2015 12:32 PM, Jean-Michel FRANCOIS wrote:

Hi Ovirt users,

I'm running ovirt hosted 3.4 with gluster data storage.
When I add a new host (Centos 6.6) the data storage (as a glsuterfs)
cannot be mount.
I have the following errors in gluster client log file :
[2015-09-24 12:27:22.636221] I [MSGID: 101190]
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
thread with index 1
[2015-09-24 12:27:22.636588] W [socket.c:588:__socket_rwv]
0-glusterfs: readv on 172.16.0.5:24007 failed (No data available)
[2015-09-24 12:27:22.637307] E [rpc-clnt.c:362:saved_frames_unwind]
(-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7f427fb3063b]
(-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7f427f8fc1d7]
(-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f427f8fc2ee]
(-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f427f8fc3bb] 


(--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7f427f8fc9f2]
) 0-glusterfs: forced unwinding frame type(GlusterFS Handshake)
op(GETSPEC(2)) called at 2015-09-24 12:27:22.636344 (xid=0x1)
[2015-09-24 12:27:22.637333] E
[glusterfsd-mgmt.c:1604:mgmt_getspec_cbk] 0-mgmt: failed to fetch
volume file (key:/data)
[2015-09-24 12:27:22.637360] W [glusterfsd.c:1219:cleanup_and_exit]
(-->/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x20e)
[0x7f427f8fc1fe] -->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3f2)
[0x40d5d2] -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] )
0-: received signum (0), shutting down
[2015-09-24 12:27:22.637375] I [fuse-bridge.c:5595:fini] 0-fuse:
Unmounting '/rhev/data-center/mnt/glusterSD/172.16.0.5:_data'.
[2015-09-24 12:27:22.646246] W [glusterfsd.c:1219:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7a51) [0x7f427ec18a51]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-:
received signum (15), shutting down
[2015-09-24 12:27:22.646246] W [glusterfsd.c:1219:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7a51) [0x7f427ec18a51]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-:
received signum (15), shutting down
And nothing server side.

This does look like an op-version issue. Adding Atin for any 
possible help.

Yes this does look an op-version issue. The current version of the
client is not supported. What client and server version of gluster are
you using?

~Atin

Hi Atin,
the server has version 3.6.3 and client 3.7.4.
Both were provided by ovirt-3.4-glusterfs-epel rpm repository, but 
not at the same date :-)

-Ravi


I suppose it is a version issue since on server side I have
glusterfs-api-3.6.3-1.el6.x86_64
glusterfs-fuse-3.6.3-1.el6.x86_64
glusterfs-libs-3.6.3-1.el6.x86_64
glusterfs-3.6.3-1.el6.x86_64
glusterfs-cli-3.6.3-1.el6.x86_64
glusterfs-rdma-3.6.3-1.el6.x86_64
glusterfs-server-3.6.3-1.el6.x86_64

and on the new host :
glusterfs-3.7.4-2.el6.x86_64
glusterfs-api-3.7.4-2.el6.x86_64
glusterfs-libs-3.7.4-2.el6.x86_64
glusterfs-fuse-3.7.4-2.el6.x86_64
glusterfs-cli-3.7.4-2.el6.x86_64
glusterfs-server-3.7.4-2.el6.x86_64
glusterfs-client-xlators-3.7.4-2.el6.x86_64
glusterfs-rdma-3.7.4-2.el6.x86_64

But since it is a production system, i'm not confident about
performing gluster server upgrade.
Mounting a gluster volume as NFS is possible (the engine data storage
has been mounted succesfully).

I'm asking here because glusterfs comes from the ovirt3.4 rpm 
repository.


If anyone have a hint to this problem

thanks
Jean-Michel



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Fwd: Re: urgent issue

2015-09-21 Thread Ravishankar N 

Hi Chris,

Replies inline..

On 09/22/2015 09:31 AM, Sahina Bose wrote:




 Forwarded Message 
Subject:Re: [ovirt-users] urgent issue
Date:   Wed, 9 Sep 2015 08:31:07 -0700
From:   Chris Liebman 
To: users 



Ok - I think I'm going to switch to local storage - I've had way to 
many unexplainable issue with glusterfs  :-(.  Is there any reason I 
cant add local storage to the existing shared-storage cluster?  I see 
that the menu item is greyed out





What version of gluster and ovirt are you using?





On Tue, Sep 8, 2015 at 4:19 PM, Chris Liebman > wrote:


Its possible that this is specific to just one gluster volume... 
I've moved a few VM disks off of that volume and am able to start

them fine.  My recolection is that any VM started on the "bad"
volume causes it to be disconnected and forces the ovirt node to
be marked down until Maint->Activate.

On Tue, Sep 8, 2015 at 3:52 PM, Chris Liebman
 wrote:

In attempting to put an ovirt cluster in production I'm
running into some off errors with gluster it looks like.  Its
12 hosts each with one brick in distributed-replicate.
 (actually 2 bricks but they are separate volumes)



These 12 nodes in dist-rep config, are they in replica 2 or replica 3? 
The latter is what is recommended for VM use-cases. Could you give the 
output of `gluster volume info` ?


[root@ovirt-node268 glusterfs]# rpm -qa | grep vdsm

vdsm-jsonrpc-4.16.20-0.el6.noarch

vdsm-gluster-4.16.20-0.el6.noarch

vdsm-xmlrpc-4.16.20-0.el6.noarch

vdsm-yajsonrpc-4.16.20-0.el6.noarch

vdsm-4.16.20-0.el6.x86_64

vdsm-python-zombiereaper-4.16.20-0.el6.noarch

vdsm-python-4.16.20-0.el6.noarch

vdsm-cli-4.16.20-0.el6.noarch


   Everything was fine last week, however, today various
clients in the gluster cluster seem get "client quorum not
met" periodically - when they get this they take one of the
bricks offline - this causes VM's to be attempted to move -
sometimes 20 at a time.  That takes a long time :-(. I've
tried disabling automatic migration and teh VM's get paused
when this happens - resuming gets nothing at that point as the
volumes mount on the server hosting the VM is not connected:


from

rhev-data-center-mnt-glusterSD-ovirt-node268.la.taboolasyndication.com:_LADC-TBX-V02.log:

[2015-09-08 21:18:42.920771] W [MSGID: 108001]
[afr-common.c:4043:afr_notify] 2-LADC-TBX-V02-replicate-2:
Client-quorum is not met



When client-quorum is not met (due to network disconnects, or gluster 
brick processes going down etc), gluster makes the volume read-only. 
This is expected behavior and prevents split-brains. It's probably a bit 
late, but do you have the  gluster fuse mount logs to confirm this 
indeed was the issue?



[2015-09-08 21:18:42.931751] I
[fuse-bridge.c:4900:fuse_thread_proc] 0-fuse: unmounting

/rhev/data-center/mnt/glusterSD/ovirt-node268.la.taboolasyndication.com:_LADC-TBX-V02

[2015-09-08 21:18:42.931836] W
[glusterfsd.c:1219:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7a51) [0x7f1bebc84a51]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x

65) [0x4059b5] ) 0-: received signum (15), shutting down

[2015-09-08 21:18:42.931858] I [fuse-bridge.c:5595:fini]
0-fuse: Unmounting

'/rhev/data-center/mnt/glusterSD/ovirt-node268.la.taboolasyndication.com:_LADC-TBX-V02'.



The VM pause you saw could be because of the unmount.I understand that a 
fix (https://gerrit.ovirt.org/#/c/40240/)  went in for ovirt 3-.6 
(vdsm-4.17) to prevent vdsm from unmounting the gluster volume when vdsm 
exits/restarts.
Is it possible to run a test setup on 3.6 and see if this is still 
happening?




And the mount is broken at that point:

[root@ovirt-node267 ~]# df

*df:

`/rhev/data-center/mnt/glusterSD/ovirt-node268.la.taboolasyndication.com:_LADC-TBX-V02':
Transport endpoint is not connected*



Yes because it received a SIGTERM above.

Thanks,
Ravi


Filesystem           1K-blocks  Â
  Used  Available Use% Mounted on

/dev/sda3Â  Â  Â  Â  Â  Â
  51475068   1968452   46885176   5% /

tmpfs                132210244     Â
  0  132210244   0% /dev/shm

/dev/sda2Â  Â  Â  Â  Â  Â  Â Â Â 487652Â Â  Â Â 32409Â Â
  429643   8% /boot

/dev/sda1Â  Â  Â  Â  Â  Â  Â Â Â 204580Â Â  Â  Â Â 260Â Â
  204320   1% /boot/efi

/dev/sda5Â  Â  Â  Â  Â Â Â 1849960960 156714056
1599267616Â Â Â 9% /data1

Re: [ovirt-users] Cannot mount gluster storage data

2015-09-25 Thread Ravishankar N 



On 09/25/2015 12:32 PM, Jean-Michel FRANCOIS wrote:

Hi Ovirt users,

I'm running ovirt hosted 3.4 with gluster data storage.
When I add a new host (Centos 6.6) the data storage (as a glsuterfs) 
cannot be mount.

I have the following errors in gluster client log file :
[2015-09-24 12:27:22.636221] I [MSGID: 101190] 
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started 
thread with index 1
[2015-09-24 12:27:22.636588] W [socket.c:588:__socket_rwv] 
0-glusterfs: readv on 172.16.0.5:24007 failed (No data available)
[2015-09-24 12:27:22.637307] E [rpc-clnt.c:362:saved_frames_unwind] 
(--> 
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7f427fb3063b] 
(--> 
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7f427f8fc1d7] 
(--> 
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f427f8fc2ee] 
(--> 
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f427f8fc3bb] 
(--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7f427f8fc9f2] 
) 0-glusterfs: forced unwinding frame type(GlusterFS Handshake) 
op(GETSPEC(2)) called at 2015-09-24 12:27:22.636344 (xid=0x1)
[2015-09-24 12:27:22.637333] E 
[glusterfsd-mgmt.c:1604:mgmt_getspec_cbk] 0-mgmt: failed to fetch 
volume file (key:/data)
[2015-09-24 12:27:22.637360] W [glusterfsd.c:1219:cleanup_and_exit] 
(-->/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x20e) 
[0x7f427f8fc1fe] -->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3f2) 
[0x40d5d2] -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 
0-: received signum (0), shutting down
[2015-09-24 12:27:22.637375] I [fuse-bridge.c:5595:fini] 0-fuse: 
Unmounting '/rhev/data-center/mnt/glusterSD/172.16.0.5:_data'.
[2015-09-24 12:27:22.646246] W [glusterfsd.c:1219:cleanup_and_exit] 
(-->/lib64/libpthread.so.0(+0x7a51) [0x7f427ec18a51] 
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d] 
-->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-: 
received signum (15), shutting down
[2015-09-24 12:27:22.646246] W [glusterfsd.c:1219:cleanup_and_exit] 
(-->/lib64/libpthread.so.0(+0x7a51) [0x7f427ec18a51] 
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d] 
-->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-: 
received signum (15), shutting down

And nothing server side.



This does look like an op-version issue. Adding Atin for any possible help.
-Ravi


I suppose it is a version issue since on server side I have
glusterfs-api-3.6.3-1.el6.x86_64
glusterfs-fuse-3.6.3-1.el6.x86_64
glusterfs-libs-3.6.3-1.el6.x86_64
glusterfs-3.6.3-1.el6.x86_64
glusterfs-cli-3.6.3-1.el6.x86_64
glusterfs-rdma-3.6.3-1.el6.x86_64
glusterfs-server-3.6.3-1.el6.x86_64

and on the new host :
glusterfs-3.7.4-2.el6.x86_64
glusterfs-api-3.7.4-2.el6.x86_64
glusterfs-libs-3.7.4-2.el6.x86_64
glusterfs-fuse-3.7.4-2.el6.x86_64
glusterfs-cli-3.7.4-2.el6.x86_64
glusterfs-server-3.7.4-2.el6.x86_64
glusterfs-client-xlators-3.7.4-2.el6.x86_64
glusterfs-rdma-3.7.4-2.el6.x86_64

But since it is a production system, i'm not confident about 
performing gluster server upgrade.
Mounting a gluster volume as NFS is possible (the engine data storage 
has been mounted succesfully).


I'm asking here because glusterfs comes from the ovirt3.4 rpm repository.

If anyone have a hint to this problem

thanks
Jean-Michel



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Ovirt 3.5 host gluster storage connection failure

2015-12-23 Thread Ravishankar N 

On 12/23/2015 11:00 PM, Steve Dainard wrote:

I've attached the client gluster log starting at the first log of the
same day as failure.
Nothing significant in the client log after the crash and subsequent 
remount. The ENODATA warnings can be ignored. There was a patch 
(http://review.gluster.org/#/c/12015/) to change the log level, let me 
check if it has made it to a recent release version.


The brick logs are all 0 file size on all of the replica 3 nodes... I
just set 'gluster volume set vm-storage diagnostics.brick-log-level
WARNING' but I'm not immediately seeing any logging to disk.

I've also attached the compute1 vdsm.log file, which over the same
time period is able to dd successfully so perhaps this discounts a
storage side issue? I've also attached compute2 (failed node) for
comparison.


Ravi - I'm not familiar with core files, would this be in a non-devel
version of gluster? Or is this something I can enable? I don't mind
enabling it now if it could help diagnose a future issue.


You should get the core files even on the non-devel versions. I think 
the method to enable core files and its location  is distribution 
specific (depending on whether it uses abrt, systemd etc.)  but you can 
check for ulimit, and /proc/sys/kernel/core_pattern settings in general. 
On my Fedora 23 machine, I get them on /core.pid_of_the_process>.


The timestamp from the logs should also help in identifying the core file:

frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash:
2015-12-22 23:04:00



Regards,
Ravi


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Ovirt 3.5 host gluster storage connection failure

2015-12-23 Thread Ravishankar N 

On 12/23/2015 11:44 AM, Sahina Bose wrote:

signal received: 6
time of crash:
2015-12-22 23:04:00
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.6.7
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7f0d091f6392]
/lib64/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f0d0920d88d]
/lib64/libc.so.6(+0x35650)[0x7f0d0820f650]
/lib64/libc.so.6(gsignal+0x37)[0x7f0d0820f5d7]
/lib64/libc.so.6(abort+0x148)[0x7f0d08210cc8]
/lib64/libc.so.6(+0x75e07)[0x7f0d0824fe07]
/lib64/libc.so.6(+0x7d1fd)[0x7f0d082571fd]
/usr/lib64/glusterfs/3.6.7/xlator/protocol/client.so(client_local_wipe+0x39)[0x7f0cfe8acdf9] 

/usr/lib64/glusterfs/3.6.7/xlator/protocol/client.so(client3_3_readv_cbk+0x487)[0x7f0cfe8c1197] 


/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0x90)[0x7f0d08fca100]
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x174)[0x7f0d08fca374]
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f0d08fc62c3]
/usr/lib64/glusterfs/3.6.7/rpc-transport/socket.so(+0x8790)[0x7f0d047f3790] 

/usr/lib64/glusterfs/3.6.7/rpc-transport/socket.so(+0xaf84)[0x7f0d047f5f84] 


/lib64/libglusterfs.so.0(+0x767c2)[0x7f0d0924b7c2]
/usr/sbin/glusterfs(main+0x502)[0x7f0d096a0fe2]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f0d081fbaf5]
/usr/sbin/glusterfs(+0x6381)[0x7f0d096a1381]



Could you provide the gluster mount logs from the client, and the 
gluster brick logs from the gluster servers? 

Also, do you have a core file of the crash?

-Ravi

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt glusterfs performance

2016-02-12 Thread Ravishankar N 

On 02/12/2016 09:11 PM, Bill James wrote:

wow, that made a whole lot of difference!
Thank you!

[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile1 bs=1M 
count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 20.2778 s, 51.7 MB/s
That's great. It was Vijay Bellur who noticed that it was not enabled on 
your volume while we were talking on irc. So thanks to him.




___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt glusterfs performance

2016-02-11 Thread Ravishankar N 

Hi Bill,
Can you enable virt-profile setting for your volume and see if that 
helps? You need to enable this optimization when you create the volume 
using ovrit, or use the following command for an existing volume:


#gluster volume set  group virt

-Ravi


On 02/12/2016 05:22 AM, Bill James wrote:

My apologies, I'm showing how much of a noob I am.
Ignore last direct to gluster numbers, as that wasn't really glusterfs.


[root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com:/gv1 
/mnt/tmp/
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2 bs=1M 
count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s

That's more how I expected, it is pointing to glusterfs performance.



On 02/11/2016 03:27 PM, Bill James wrote:
don't know if it helps, but I ran a few more tests, all from the same 
hardware node.


The VM:
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M 
count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s

Writing directly to gluster volume:
[root@ovirt2 test ~]# time dd if=/dev/zero 
of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s


Writing to NFS volume:
[root@ovirt2 test ~]# time dd if=/dev/zero 
of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s

NFS & Gluster are using the same interface. Tests were not run at 
same time.


This would suggest my problem isn't glusterfs, but the VM performance.



On 02/11/2016 03:13 PM, Bill James wrote:

xml attached.


On 02/11/2016 12:28 PM, Nir Soffer wrote:

On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.ja...@j2.com> wrote:

thank you for the reply.

We setup gluster using the names associated with  NIC 2 IP.
  Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
  Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
  Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1

That's NIC 2's IP.
Using 'iftop -i eno2 -L 5 -t' :

dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s

Can you share the xml of this vm? You can find it in vdsm log,
at the time you start the vm.

Or you can do (on the host):

# virsh
virsh # list
(username: vdsm@ovirt password: shibboleth)
virsh # dumpxml vm-id


Peak rate (sent/received/total):  281Mb 5.36Mb
282Mb
Cumulative (sent/received/total): 1.96GB 14.6MB
1.97GB

gluster volume info gv1:
  Options Reconfigured:
  performance.write-behind-window-size: 4MB
  performance.readdir-ahead: on
  performance.cache-size: 1GB
  performance.write-behind: off

performance.write-behind: off didn't help.
Neither did any other changes I've tried.


There is no VM traffic on this VM right now except my test.



On 02/10/2016 11:55 PM, Nir Soffer wrote:
On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N 
<ravishan...@redhat.com>

wrote:

+gluster-users

Does disabling 'performance.write-behind' give a better throughput?



On 02/10/2016 11:06 PM, Bill James wrote:
I'm setting up a ovirt cluster using glusterfs and noticing not 
stellar

performance.
Maybe my setup could use some adjustments?

3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 
3.6.2.6-1.
Each node has 8 spindles configured in 1 array which is split 
using LVM

with one logical volume for system and one for gluster.
They each have 4 NICs,
   NIC1 = ovirtmgmt
   NIC2 = gluster  (1GbE)

How do you ensure that gluster trafic is using this nic?


   NIC3 = VM traffic

How do you ensure that vm trafic is using this nic?


I tried with default glusterfs settings

And did you find any difference?


and also with:
performance.cache-size: 1GB
performance.readdir-ahead: on
performance.write-behind-window-size: 4MB

[root@ovirt3 test scripts]# gluster volume info gv1

Volume Name: gv1
Type: Replicate
Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
Options Reconfigured:
performance.cache-size: 1GB
performance.readdir-ahead: on
performance.write-behind-window-size: 4MB


Using simple dd test on VM in ovirt:
dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct

block size of 1G?!

Try 1M (our default for storage operations)


1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s

Another VM not in ovirt using nfs:
 dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s


Is that expected or is there a better way to set it up to get 
better

performance?

Adding Niels for advice.


This email, its contents and 
Please avoid this, this is a public mailing list, everything you 
write

here is public

Re: [ovirt-users] Fwd: Re: ovirt - can't attach master domain II

2016-02-24 Thread Ravishankar N 
Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': 
u'0002-0002-0002-0002-021e'}
Thread-35634::DEBUG::2016-02-24 
11:18:20,860::task::595::Storage.TaskManager.Task::(_updateState) 
Task=`e887fd8b-6961-40f1-b3a0-917ffbea25c0`::moving from state init -> 
state preparing
Thread-35634::INFO::2016-02-24 
11:18:20,860::logUtils::44::dispatcher::(wrapper) Run and protect: 
getSpmStatus(spUUID=u'0002-0002-0002-0002-021e', options=None)
Thread-35634::INFO::2016-02-24 
11:18:20,867::logUtils::47::dispatcher::(wrapper) Run and protect: 
getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 
'Free', 'spmLver': -1}}
Thread-35634::DEBUG::2016-02-24 
11:18:20,867::task::1191::Storage.TaskManager.Task::(prepare) 
Task=`e887fd8b-6961-40f1-b3a0-917ffbea25c0`::finished: {'spm_st': 
{'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-35634::DEBUG::2016-02-24 
11:18:20,867::task::595::Storage.TaskManager.Task::(_updateState) 
Task=`e887fd8b-6961-40f1-b3a0-917ffbea25c0`::moving from state 
preparing -> state finished
Thread-35634::DEBUG::2016-02-24 
11:18:20,867::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}
Thread-35634::DEBUG::2016-02-24 
11:18:20,867::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-35634::DEBUG::2016-02-24 
11:18:20,867::task::993::Storage.TaskManager.Task::(_decref) 
Task=`e887fd8b-6961-40f1-b3a0-917ffbea25c0`::ref 0 aborting False

---

these blocks generated in cycle for each domain

Any IDEA ??
regs.
Pa.



On 24.2.2016 10:54, Ravishankar N wrote:

Hi,

On 02/24/2016 06:43 AM, p...@email.cz wrote:

Hi,
I found the main ( maybe ) problem with IO error ( -5 ) for "ids" 
file access

This file is not accessable via NFS, locally yes
How is NFS coming into the picture? Are you not using gluster fuse 
mount?

.
How can I fix it ??
Can you run `gluster volume heal volname info` and `gluster volume 
heal volname info split-brain` to see if the "ids" file is in 
split-brain? A file in split-brain returns EIO when accessed from the 
mount.

Regards,
Ravi



regs.
Pavel

# sanlock client log_dump

0 flags 1 timeout 0
2016-02-24 02:01:10+0100 3828 [12111]: s1316 lockspace 
88adbd49-62d6-45b1-9992-b04464a04112:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md/ids:0
2016-02-24 02:01:10+0100 3828 [12111]: cmd_add_lockspace 4,15 async 
done 0
2016-02-24 02:01:10+0100 3828 [19556]: s1316 delta_acquire begin 
88adbd49-62d6-45b1-9992-b04464a04112:1
2016-02-24 02:01:10+0100 3828 [19556]: 88adbd49 aio collect 0 
0x7fe4580008c0:0x7fe4580008d0:0x7fe458101000 result -5:0 match res
2016-02-24 02:01:10+0100 3828 [19556]: read_sectors delta_leader 
offset 0 rv -5 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md/ids
2016-02-24 02:01:10+0100 3828 [19556]: s1316 delta_acquire 
leader_read1 error -5
2016-02-24 02:01:11+0100 3829 [12111]: s1316 add_lockspace fail 
result -5
2016-02-24 02:01:12+0100 3831 [12116]: cmd_add_lockspace 4,15 
7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0 
flags 1 timeout 0
2016-02-24 02:01:12+0100 3831 [12116]: s1317 lockspace 
7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0
2016-02-24 02:01:12+0100 3831 [12116]: cmd_add_lockspace 4,15 async 
done 0
2016-02-24 02:01:12+0100 3831 [19562]: s1317 delta_acquire begin 
7f52b697-c199-4f58-89aa-102d44327124:1
2016-02-24 02:01:12+0100 3831 [19562]: 7f52b697 aio collect 0 
0x7fe4580008c0:0x7fe4580008d0:0x7fe458101000 result -5:0 match res
2016-02-24 02:01:12+0100 3831 [19562]: read_sectors delta_leader 
offset 0 rv -5 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids
2016-02-24 02:01:12+0100 3831 [19562]: s1317 delta_acquire 
leader_read1 error -5
2016-02-24 02:01:13+0100 3831 [1321]: cmd_add_lockspace 4,15 
0fcad888-d573-47be-bef3-0bc0b7a99fb7:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids:0 
flags 1 timeout 0
2016-02-24 02:01:13+0100 3831 [1321]: s1318 lockspace 
0fcad888-d573-47be-bef3-0bc0b7a99fb7:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids:0
2016-02-24 02:01:13+0100 3831 [1321]: cmd_add_lockspace 4,15 async 
done 0
2016-02-24 02:01:13+0100 3831 [19564]: s1318 delta_acquire begin 
0fcad888-d573-47be-bef3-0bc0b7a99fb7:1
2016-02-24 02:01:13+0100 3831 [19564]: 0fcad888 aio collect 0 
0x7fe4580008c0:0x7fe4580008d0:0x7fe458201000 result -5:0 match res
2016-02-24 02:01:13+0100 3831 [19564]: read_sectors delta_leader 
offset 0 rv -5 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fca

Re: [ovirt-users] ovirt - can't attach master domain II

2016-02-24 Thread Ravishankar N 

Hi,

On 02/24/2016 06:43 AM, p...@email.cz wrote:

Hi,
I found the main ( maybe ) problem with IO error ( -5 ) for "ids" file 
access

This file is not accessable via NFS, locally yes

How is NFS coming into the picture? Are you not using gluster fuse mount?

.
How can I fix it ??
Can you run `gluster volume heal volname info` and `gluster volume heal 
volname info split-brain` to see if the "ids" file is in split-brain? A 
file in split-brain returns EIO when accessed from the mount.

Regards,
Ravi



regs.
Pavel

# sanlock client log_dump

0 flags 1 timeout 0
2016-02-24 02:01:10+0100 3828 [12111]: s1316 lockspace 
88adbd49-62d6-45b1-9992-b04464a04112:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md/ids:0

2016-02-24 02:01:10+0100 3828 [12111]: cmd_add_lockspace 4,15 async done 0
2016-02-24 02:01:10+0100 3828 [19556]: s1316 delta_acquire begin 
88adbd49-62d6-45b1-9992-b04464a04112:1
2016-02-24 02:01:10+0100 3828 [19556]: 88adbd49 aio collect 0 
0x7fe4580008c0:0x7fe4580008d0:0x7fe458101000 result -5:0 match res
2016-02-24 02:01:10+0100 3828 [19556]: read_sectors delta_leader 
offset 0 rv -5 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md/ids
2016-02-24 02:01:10+0100 3828 [19556]: s1316 delta_acquire 
leader_read1 error -5

2016-02-24 02:01:11+0100 3829 [12111]: s1316 add_lockspace fail result -5
2016-02-24 02:01:12+0100 3831 [12116]: cmd_add_lockspace 4,15 
7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0 
flags 1 timeout 0
2016-02-24 02:01:12+0100 3831 [12116]: s1317 lockspace 
7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0

2016-02-24 02:01:12+0100 3831 [12116]: cmd_add_lockspace 4,15 async done 0
2016-02-24 02:01:12+0100 3831 [19562]: s1317 delta_acquire begin 
7f52b697-c199-4f58-89aa-102d44327124:1
2016-02-24 02:01:12+0100 3831 [19562]: 7f52b697 aio collect 0 
0x7fe4580008c0:0x7fe4580008d0:0x7fe458101000 result -5:0 match res
2016-02-24 02:01:12+0100 3831 [19562]: read_sectors delta_leader 
offset 0 rv -5 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids
2016-02-24 02:01:12+0100 3831 [19562]: s1317 delta_acquire 
leader_read1 error -5
2016-02-24 02:01:13+0100 3831 [1321]: cmd_add_lockspace 4,15 
0fcad888-d573-47be-bef3-0bc0b7a99fb7:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids:0 
flags 1 timeout 0
2016-02-24 02:01:13+0100 3831 [1321]: s1318 lockspace 
0fcad888-d573-47be-bef3-0bc0b7a99fb7:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids:0

2016-02-24 02:01:13+0100 3831 [1321]: cmd_add_lockspace 4,15 async done 0
2016-02-24 02:01:13+0100 3831 [19564]: s1318 delta_acquire begin 
0fcad888-d573-47be-bef3-0bc0b7a99fb7:1
2016-02-24 02:01:13+0100 3831 [19564]: 0fcad888 aio collect 0 
0x7fe4580008c0:0x7fe4580008d0:0x7fe458201000 result -5:0 match res
2016-02-24 02:01:13+0100 3831 [19564]: read_sectors delta_leader 
offset 0 rv -5 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids
2016-02-24 02:01:13+0100 3831 [19564]: s1318 delta_acquire 
leader_read1 error -5

2016-02-24 02:01:13+0100 3832 [12116]: s1317 add_lockspace fail result -5
2016-02-24 02:01:14+0100 3832 [1321]: s1318 add_lockspace fail result -5
2016-02-24 02:01:19+0100 3838 [12106]: cmd_add_lockspace 4,15 
3da46e07-d1ea-4f10-9250-6cbbb7b94d80:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P5/3da46e07-d1ea-4f10-9250-6cbbb7b94d80/dom_md/ids:0 
flags 1 timeout 0
2016-02-24 02:01:19+0100 3838 [12106]: s1319 lockspace 
3da46e07-d1ea-4f10-9250-6cbbb7b94d80:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P5/3da46e07-d1ea-4f10-9250-6cbbb7b94d80/dom_md/ids:0

2016-02-24 02:01:19+0100 3838 [12106]: cmd_add_lockspace 4,15 async done 0
2016-02-24 02:01:19+0100 3838 [19638]: s1319 delta_acquire begin 
3da46e07-d1ea-4f10-9250-6cbbb7b94d80:1
2016-02-24 02:01:19+0100 3838 [19638]: 3da46e07 aio collect 0 
0x7fe4580008c0:0x7fe4580008d0:0x7fe458101000 result -5:0 match res
2016-02-24 02:01:19+0100 3838 [19638]: read_sectors delta_leader 
offset 0 rv -5 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P5/3da46e07-d1ea-4f10-9250-6cbbb7b94d80/dom_md/ids
2016-02-24 02:01:19+0100 3838 [19638]: s1319 delta_acquire 
leader_read1 error -5

2016-02-24 02:01:20+0100 3839 [12106]: s1319 add_lockspace fail result -5
2016-02-24 02:01:20+0100 3839 [1320]: cmd_add_lockspace 4,15 
88adbd49-62d6-45b1-9992-b04464a04112:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md/ids:0 
flags 1 timeout 0
2016-02-24 02:01:20+0100 3839 [1320]: s1320 lockspace

Re: [ovirt-users] ovirt glusterfs performance

2016-04-06 Thread Ravishankar N 

On 04/06/2016 02:08 PM, Roderick Mooi wrote:

Hi Ravi and colleagues

(apologies for hijacking this thread but I’m not sure where else to 
report this (and it is related).)


With gluster 3.7.10, running
#gluster volume set  group virt
fails with:
volume set: failed: option : eager-lock does not exist
Did you mean eager-lock?

I had to remove the eager-lock setting from 
/var/lib/glusterd/groups/virt to get this to work. It seems like 
setting eager-lock has been removed from latest gluster. Is this 
correct? Either way, is there anything else I should do?


It is not removed. Can you try 'gluster volume set volname 
cluster.eager-lock enable`?
I think the disperse (EC) translator introduced a `disperse.eager-lock` 
which is why you would need to mention entire volume option name to 
avoid ambiguity.
We probably need to fix the virt profile setting to include the entire 
name. By the way 'gluster volume set help` should give you the list of 
all options.


-Ravi



Cheers,

Roderick

On 12 Feb 2016, at 6:18 AM, Ravishankar N <ravishan...@redhat.com 
<mailto:ravishan...@redhat.com>> wrote:


Hi Bill,
Can you enable virt-profile setting for your volume and see if that 
helps? You need to enable this optimization when you create the 
volume using ovrit, or use the following command for an existing volume:


#gluster volume set  group virt

-Ravi


On 02/12/2016 05:22 AM, Bill James wrote:

My apologies, I'm showing how much of a noob I am.
Ignore last direct to gluster numbers, as that wasn't really glusterfs.


[root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com 
<http://ovirt2-ks.test.j2noc.com>:/gv1 /mnt/tmp/
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2 
bs=1M count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s

That's more how I expected, it is pointing to glusterfs performance.



On 02/11/2016 03:27 PM, Bill James wrote:
don't know if it helps, but I ran a few more tests, all from the 
same hardware node.


The VM:
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M 
count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s

Writing directly to gluster volume:
[root@ovirt2 test ~]# time dd if=/dev/zero 
of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s


Writing to NFS volume:
[root@ovirt2 test ~]# time dd if=/dev/zero 
of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s

NFS & Gluster are using the same interface. Tests were not run at 
same time.


This would suggest my problem isn't glusterfs, but the VM performance.



On 02/11/2016 03:13 PM, Bill James wrote:

xml attached.


On 02/11/2016 12:28 PM, Nir Soffer wrote:
On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.ja...@j2.com> 
wrote:

thank you for the reply.

We setup gluster using the names associated with  NIC 2 IP.
  Brick1: ovirt1-ks.test.j2noc.com 
<http://ovirt1-ks.test.j2noc.com>:/gluster-store/brick1/gv1
  Brick2: ovirt2-ks.test.j2noc.com 
<http://ovirt2-ks.test.j2noc.com>:/gluster-store/brick1/gv1
  Brick3: ovirt3-ks.test.j2noc.com 
<http://ovirt3-ks.test.j2noc.com>:/gluster-store/brick1/gv1


That's NIC 2's IP.
Using 'iftop -i eno2 -L 5 -t' :

dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s

Can you share the xml of this vm? You can find it in vdsm log,
at the time you start the vm.

Or you can do (on the host):

# virsh
virsh # list
(username: vdsm@ovirt password: shibboleth)
virsh # dumpxml vm-id


Peak rate (sent/received/total): 281Mb 5.36Mb
282Mb
Cumulative (sent/received/total): 1.96GB 14.6MB
1.97GB

gluster volume info gv1:
  Options Reconfigured:
  performance.write-behind-window-size: 4MB
  performance.readdir-ahead: on
  performance.cache-size: 1GB
  performance.write-behind: off

performance.write-behind: off didn't help.
Neither did any other changes I've tried.


There is no VM traffic on this VM right now except my test.



On 02/10/2016 11:55 PM, Nir Soffer wrote:
On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N 
<ravishan...@redhat.com>

wrote:

+gluster-users

Does disabling 'performance.write-behind' give a better 
throughput?




On 02/10/2016 11:06 PM, Bill James wrote:
I'm setting up a ovirt cluster using glusterfs and noticing 
not stellar

performance.
Maybe my setup could use some adjustments?

3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 
3.6.2.6-1.
Each node has 8 spindles configured in 1 array which is split 
using LVM

with one logical volume for system and one for gluster.
They each have 4 NICs,
   NIC1 = ovirtmgmt
   NIC2 = gluster  (1GbE)

How do you ensure that gluster trafic is using this nic?


NIC3 = VM traffic

How do you ensure that vm trafic is using this nic?


I tried with default glusterfs settings

And did you find any diff

Re: [ovirt-users] ovirt glusterfs performance

2016-04-12 Thread Ravishankar N 

On 04/12/2016 02:41 PM, Roderick Mooi wrote:

Hi

It is not removed. Can you try 'gluster volume set volname 
cluster.eager-lock enable`?


This works. BTW by default this setting is “on”. What’s the difference 
between “on” and “enable”?


Both are identical. You can use any of the booleans to achieve the same 
effect. {"1", "on", "yes", "true", "enable"} or {"0", "off", "no", 
"false", "disable"}
FYI, the patch http://review.gluster.org/#/c/13958/ to fix this issue 
should make it to glusterfs-3.7.11.

-Ravi


Thanks for the clarification.

Regards,

Roderick

On 06 Apr 2016, at 10:56 AM, Ravishankar N <ravishan...@redhat.com 
<mailto:ravishan...@redhat.com>> wrote:


On 04/06/2016 02:08 PM, Roderick Mooi wrote:

Hi Ravi and colleagues

(apologies for hijacking this thread but I’m not sure where else to 
report this (and it is related).)


With gluster 3.7.10, running
#gluster volume set  group virt
fails with:
volume set: failed: option : eager-lock does not exist
Did you mean eager-lock?

I had to remove the eager-lock setting from 
/var/lib/glusterd/groups/virt to get this to work. It seems like 
setting eager-lock has been removed from latest gluster. Is this 
correct? Either way, is there anything else I should do?


It is not removed. Can you try 'gluster volume set volname 
cluster.eager-lock enable`?
I think the disperse (EC) translator introduced a 
`disperse.eager-lock` which is why you would need to mention entire 
volume option name to avoid ambiguity.
We probably need to fix the virt profile setting to include the 
entire name. By the way 'gluster volume set help` should give you the 
list of all options.


-Ravi



Cheers,

Roderick

On 12 Feb 2016, at 6:18 AM, Ravishankar N <ravishan...@redhat.com 
<mailto:ravishan...@redhat.com>> wrote:


Hi Bill,
Can you enable virt-profile setting for your volume and see if that 
helps? You need to enable this optimization when you create the 
volume using ovrit, or use the following command for an existing 
volume:


#gluster volume set  group virt

-Ravi


On 02/12/2016 05:22 AM, Bill James wrote:

My apologies, I'm showing how much of a noob I am.
Ignore last direct to gluster numbers, as that wasn't really 
glusterfs.



[root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com 
<http://ovirt2-ks.test.j2noc.com/>:/gv1 /mnt/tmp/
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2 
bs=1M count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s

That's more how I expected, it is pointing to glusterfs performance.



On 02/11/2016 03:27 PM, Bill James wrote:
don't know if it helps, but I ran a few more tests, all from the 
same hardware node.


The VM:
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M 
count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s

Writing directly to gluster volume:
[root@ovirt2 test ~]# time dd if=/dev/zero 
of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s


Writing to NFS volume:
[root@ovirt2 test ~]# time dd if=/dev/zero 
of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct

1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s

NFS & Gluster are using the same interface. Tests were not run at 
same time.


This would suggest my problem isn't glusterfs, but the VM 
performance.




On 02/11/2016 03:13 PM, Bill James wrote:

xml attached.


On 02/11/2016 12:28 PM, Nir Soffer wrote:
On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.ja...@j2.com> 
wrote:

thank you for the reply.

We setup gluster using the names associated with  NIC 2 IP.
  Brick1: ovirt1-ks.test.j2noc.com 
<http://ovirt1-ks.test.j2noc.com/>:/gluster-store/brick1/gv1
  Brick2: ovirt2-ks.test.j2noc.com 
<http://ovirt2-ks.test.j2noc.com/>:/gluster-store/brick1/gv1
  Brick3: ovirt3-ks.test.j2noc.com 
<http://ovirt3-ks.test.j2noc.com/>:/gluster-store/brick1/gv1


That's NIC 2's IP.
Using 'iftop -i eno2 -L 5 -t' :

dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s

Can you share the xml of this vm? You can find it in vdsm log,
at the time you start the vm.

Or you can do (on the host):

# virsh
virsh # list
(username: vdsm@ovirt password: shibboleth)
virsh # dumpxml vm-id


Peak rate (sent/received/total): 281Mb 5.36Mb
282Mb
Cumulative (sent/received/total): 1.96GB 14.6MB
1.97GB

gluster volume info gv1:
  Options Reconfigured:
performance.write-behind-window-size: 4MB
  performance.readdir-ahead: on
  performance.cache-size: 1GB
  performance.write-behind: off

performance.write-behind: off didn't help.
Neither did any other changes I've tried.


There is no VM traffic on this VM right now except my test.



On 02/10/2016 11:55 PM, Nir Soffer wrote:
On Thu, Feb 11, 2016 at 2:42

Re: [ovirt-users] [Gluster-users] open error -13 = sanlock

2016-03-01 Thread Ravishankar N 

On 03/02/2016 12:02 PM, Sahina Bose wrote:



On 03/02/2016 03:45 AM, Nir Soffer wrote:
On Tue, Mar 1, 2016 at 10:51 PM, p...@email.cz > wrote:

>
> HI,
> requested output:
>
> # ls -lh /rhev/data-center/mnt/glusterSD/localhost:*/*/dom_md
>
> 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md:

> total 2,1M
> -rw-rw 1 vdsm kvm 1,0M  1. bře 21.28 ids  <-- good
> -rw-rw 1 vdsm kvm  16M  7. lis 22.16 inbox
> -rw-rw 1 vdsm kvm 2,0M  7. lis 22.17 leases
> -rw-r--r-- 1 vdsm kvm  335  7. lis 22.17 metadata
> -rw-rw 1 vdsm kvm  16M  7. lis 22.16 outbox
>
> 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md:

> total 1,1M
> -rw-r--r-- 1 vdsm kvm0 24. úno 07.41 ids  <-- bad (sanlock 
cannot write, other can read)

> -rw-rw 1 vdsm kvm  16M  7. lis 00.14 inbox
> -rw-rw 1 vdsm kvm 2,0M  7. lis 03.56 leases
> -rw-r--r-- 1 vdsm kvm  333  7. lis 03.56 metadata
> -rw-rw 1 vdsm kvm  16M  7. lis 00.14 outbox
>
> 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md:

> total 1,1M
> -rw-r--r-- 1 vdsm kvm0 24. úno 07.43 ids  <-- bad (sanlock 
cannot write, other can read)

> -rw-rw 1 vdsm kvm  16M  7. lis 00.15 inbox
> -rw-rw 1 vdsm kvm 2,0M  7. lis 22.14 leases
> -rw-r--r-- 1 vdsm kvm  333  7. lis 22.14 metadata
> -rw-rw 1 vdsm kvm  16M  7. lis 00.15 outbox
>
> 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P3/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md:

> total 1,1M
> -rw-r--r-- 1 vdsm kvm0 24. úno 07.43 ids  <-- bad (sanlock 
cannot write, other can read)

> -rw-rw 1 vdsm kvm  16M 23. úno 22.51 inbox
> -rw-rw 1 vdsm kvm 2,0M 23. úno 23.12 leases
> -rw-r--r-- 1 vdsm kvm  998 25. úno 00.35 metadata
> -rw-rw 1 vdsm kvm  16M  7. lis 00.16 outbox
>
> 
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md:

> total 1,1M
> -rw-r--r-- 1 vdsm kvm0 24. úno 07.44 ids  <-- bad (sanlock 
cannot write, other can read)

> -rw-rw 1 vdsm kvm  16M  7. lis 00.17 inbox
> -rw-rw 1 vdsm kvm 2,0M  7. lis 00.18 leases
> -rw-r--r-- 1 vdsm kvm  333  7. lis 00.18 metadata
> -rw-rw 1 vdsm kvm  16M  7. lis 00.17 outbox
>
> 
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P1/42d710a9-b844-43dc-be41-77002d1cd553/dom_md:

> total 1,1M
> -rw-rw-r-- 1 vdsm kvm0 24. úno 07.32 ids  <-- bad (other can read)
> -rw-rw 1 vdsm kvm  16M  7. lis 22.18 inbox
> -rw-rw 1 vdsm kvm 2,0M  7. lis 22.18 leases
> -rw-r--r-- 1 vdsm kvm  333  7. lis 22.18 metadata
> -rw-rw 1 vdsm kvm  16M  7. lis 22.18 outbox
>
> 
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2/ff71b47b-0f72-4528-9bfe-c3da888e47f0/dom_md:

> total 3,0M
> -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids  <-- bad (other can read)
> -rw-rw 1 vdsm kvm  16M 25. úno 00.42 inbox
> -rw-rw 1 vdsm kvm 2,0M 25. úno 00.44 leases
> -rw-r--r-- 1 vdsm kvm  997 24. úno 02.46 metadata
> -rw-rw 1 vdsm kvm  16M 25. úno 00.44 outbox
>
> 
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P3/ef010d08-aed1-41c4-ba9a-e6d9bdecb4b4/dom_md:

> total 2,1M
> -rw-r--r-- 1 vdsm kvm0 24. úno 07.34 ids  <-- bad (sanlock 
cannot write, other can read)

> -rw-rw 1 vdsm kvm  16M 23. úno 22.35 inbox
> -rw-rw 1 vdsm kvm 2,0M 23. úno 22.38 leases
> -rw-r--r-- 1 vdsm kvm 1,1K 24. úno 19.07 metadata
> -rw-rw 1 vdsm kvm  16M 23. úno 22.27 outbox
>
> 
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97/dom_md:

> total 3,0M
> -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids  <-- bad (other can read)
> -rw-rw-r-- 1 vdsm kvm  16M  6. lis 23.50 inbox  <-- bad (other can 
read)
> -rw-rw-r-- 1 vdsm kvm 2,0M  6. lis 23.51 leases<-- bad (other 
can read)
> -rw-rw-r-- 1 vdsm kvm  734  7. lis 02.13 metadata  <-- bad (group 
can write, other can read)
> -rw-rw-r-- 1 vdsm kvm  16M  6. lis 16.55 outbox  <-- bad (other can 
read)

>
> 
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P5/1ca56b45-701e-4c22-9f59-3aebea4d8477/dom_md:

> total 1,1M
> -rw-rw-r-- 1 vdsm kvm0 24. úno 07.35 ids  <-- bad (other can read)
> -rw-rw-r-- 1 vdsm kvm  16M 24. úno 01.06 inbox
> -rw-rw-r-- 1 vdsm kvm 2,0M 24. úno 02.44 leases
> -rw-r--r-- 1 vdsm kvm  998 24. úno 19.07 metadata
> -rw-rw-r-- 1 vdsm kvm  16M  7. lis 22.20 outbox


It should look like this:

-rw-rw. 1 vdsm kvm 1.0M Mar  1 23:36 ids
-rw-rw. 1 vdsm kvm 2.0M Mar  1 23:35 leases
-rw-r--r--. 1 vdsm kvm  353 Mar  1 23:35 metadata
-rw-rw. 1 vdsm kvm  16M Mar  1 23:34 outbox
-rw-rw. 1 vdsm kvm  16M Mar  1 23:34 inbox

This explains the EACCES error.

You can start by fixing the permissions manually, you can do this online.

>  The ids files was generated by "touch" command after deleting them 
due "sanlock locking hang"  gluster crash & reboot
> I expected that they will be filled automaticaly after gluster 
reboot ( the

Re: [ovirt-users] [Gluster-users] open error -13 = sanlock

2016-03-02 Thread Ravishankar N 

On 03/03/2016 12:43 AM, Nir Soffer wrote:


PS:  # find /STORAGES -samefile
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
-print
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
= missing "shadowfile" in " .gluster " dir.
How can I fix it ?? - online !


Ravi?

Is this the case in all 3 bricks of the replica?
BTW, you can just stat the file on the brick and see the link count (it 
must be 2) instead of running the more expensive find command.


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] [Gluster-users] open error -13 = sanlock

2016-03-03 Thread Ravishankar N 

On 03/03/2016 02:53 PM, p...@email.cz wrote:

This is replica 2, only , with following settings

Options Reconfigured:
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: fixed

Not sure why you have set this option.
Ideally replica 3 or arbiter volumes are recommended for gluster+ovirt 
use.  (client) quorum does not make sense for a 2 node setup. I have a 
detailed write up here which explains things 
http://gluster.readthedocs.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/ 
which explains things.



cluster.server-quorum-type: none
storage.owner-uid: 36
storage.owner-gid: 36
cluster.quorum-count: 1
cluster.self-heal-daemon: enable

If I'll create "ids" file manually (  eg. " sanlock direct init -s 
3c34ad63-6c66-4e23-ab46-084f3d70b147:0:/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids:0 
" ) on both bricks,

vdsm is writing only to half of them ( that with 2 links = correct )
"ids" file has correct permittions, owner, size  on both bricks.
brick 1:  -rw-rw 1 vdsm kvm 1048576  2. bře 18.56 
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - 
not updated


Okay, so this one has link count =1 which means the .glusterfs hardlink 
is missing.  Can you try deleting this file from the brick and perform a 
stat on the file from the mount? That should heal (i.e recreate it ) on 
this brick from the other brick with the appropriate .glusterfs hard link.



brick 2:  -rw-rw 2 vdsm kvm 1048576  3. bře 10.16 
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - 
is continually updated


What happens when I'll restart vdsm ? Will oVirt storages go to 
"disable " state ??? = disconnect VMs storages ?


No idea on this one...
-Ravi


regs.Pa.

On 3.3.2016 02:02, Ravishankar N wrote:

On 03/03/2016 12:43 AM, Nir Soffer wrote:


PS:  # find /STORAGES -samefile
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
-print
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
= missing "shadowfile" in " .gluster " dir.
How can I fix it ?? - online !


Ravi?

Is this the case in all 3 bricks of the replica?
BTW, you can just stat the file on the brick and see the link count 
(it must be 2) instead of running the more expensive find command.







___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] [Attention needed] GlusterFS repository down - affects CI / Installations

2016-04-27 Thread Ravishankar N 

@gluster infra  - FYI.

On 04/27/2016 02:20 PM, Nadav Goldin wrote:

Hi,
The GlusterFS repository became unavailable this morning, as a result 
all Jenkins jobs that use the repository will fail, the common error 
would be:



http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-7/noarch/repodata/repomd.xml:
[Errno 14] HTTP Error 403 - Forbidden


Also, installations of oVirt will fail.

We are working on a solution and will update asap.

Nadav.



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Ovirt/Gluster replica 3 distributed-replicated problem

2016-09-29 Thread Ravishankar N 

On 09/29/2016 08:03 PM, Davide Ferrari wrote:
It's strange, I've tried to trigger the error again by putting vm04 in 
maintenence and stopping the gluster service (from ovirt gui) and now 
the VM starts correctly. Maybe the arbiter indeed blamed the brick 
that was still up before, but how's that possible?


A write from the client on that file (vm image) could have succeeded 
only on vm04 even before you brought it down.


The only (maybe big) difference with the previous, erroneous 
situation, is that before I did maintenence (+ reboot) of 3 of my 4 
hosts, maybe I should have left more time between one reboot and another?


If you did not do anything from the previous run other than to bring the 
node up and things worked, then the file is not in split-brain. Split 
braine'd files need to be resolved before they can be accessed again, 
which apparently did not happen in your case.


-Ravi


2016-09-29 14:16 GMT+02:00 Ravishankar N <ravishan...@redhat.com 
<mailto:ravishan...@redhat.com>>:


On 09/29/2016 05:18 PM, Sahina Bose wrote:

Yes, this is a GlusterFS problem. Adding gluster users ML

On Thu, Sep 29, 2016 at 5:11 PM, Davide Ferrari
<dav...@billymob.com <mailto:dav...@billymob.com>> wrote:

Hello

maybe this is more glustefs then ovirt related but since
OVirt integrates Gluster management and I'm experiencing the
problem in an ovirt cluster, I'm writing here.

The problem is simple: I have a data domain mappend on a
replica 3 arbiter1 Gluster volume with 6 bricks, like this:

Status of volume: data_ssd
Gluster process TCP Port  RDMA Port  Online  Pid

--
Brick vm01.storage.billy:/gluster/ssd/data/
brick 49153 0  Y   19298
Brick vm02.storage.billy:/gluster/ssd/data/
brick 49153 0  Y   6146
Brick vm03.storage.billy:/gluster/ssd/data/
arbiter_brick 49153 0  Y   6552
Brick vm03.storage.billy:/gluster/ssd/data/
brick 49154 0  Y   6559
Brick vm04.storage.billy:/gluster/ssd/data/
brick 49152 0  Y   6077
Brick vm02.storage.billy:/gluster/ssd/data/
arbiter_brick 49154 0  Y   6153
Self-heal Daemon on localhost   N/A N/A   
Y   30746
Self-heal Daemon on vm01.storage.billy  N/A N/A   
Y   196058
Self-heal Daemon on vm03.storage.billy  N/A N/A   
Y   23205
Self-heal Daemon on vm04.storage.billy  N/A N/A   
Y   8246



Now, I've put in maintenance the vm04 host, from ovirt,
ticking the "Stop gluster" checkbox, and Ovirt didn't
complain about anything. But when I tried to run a new VM it
complained about "storage I/O problem", while the storage
data status was always UP.

Looking in the gluster logs I can see this:

[2016-09-29 11:01:01.556908] I
[glusterfsd-mgmt.c:1596:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile, continuing
[2016-09-29 11:02:28.124151] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_txn_refresh_done]
0-data_ssd-replicate-1: Failing READ on gfid
bf5922b7-19f3-4ce3-98df-71e981ecca8d: split-brain observed.
[Input/output error]
[2016-09-29 11:02:28.126580] W [MSGID: 108008]
[afr-read-txn.c:244:afr_read_txn] 0-data_ssd-replicate-1:
Unreadable subvolume -1 found with event generation 6 for
gfid bf5922b7-19f3-4ce3-98df-71e981ecca8d. (Possible split-brain)
[2016-09-29 11:02:28.127374] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_txn_refresh_done]
0-data_ssd-replicate-1: Failing FGETXATTR on gfid
bf5922b7-19f3-4ce3-98df-71e981ecca8d: split-brain observed.
[Input/output error]
[2016-09-29 11:02:28.128130] W [MSGID: 108027]
[afr-common.c:2403:afr_discover_done] 0-data_ssd-replicate-1:
no read subvols for (null)
[2016-09-29 11:02:28.129890] W
[fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 8201:
READ => -1 gfid=bf5922b7-19f3-4ce3-98df-71e981ecca8d
fd=0x7f09b749d210 (Input/output error)
[2016-09-29 11:02:28.130824] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_txn_refresh_done]
0-data_ssd-replicate-1: Failing FSTAT on gfid
bf5922b7-19f3-4ce3-98df-71e981ecca8d: split-brain observed.
[Input/output error]



Does `gluster volume heal data_ssd info split-brain` report that
the file is in split-brain, with vm04 still being down?
If yes, could you provide the extended attributes of this gfid
from all 3 bricks:
getfattr -d -m . -e hex
/path/to/brick/bf/59/bf5922b7-19f3-4ce3-

Re: [ovirt-users] Ovirt/Gluster replica 3 distributed-replicated problem

2016-09-29 Thread Ravishankar N 

On 09/29/2016 05:18 PM, Sahina Bose wrote:

Yes, this is a GlusterFS problem. Adding gluster users ML

On Thu, Sep 29, 2016 at 5:11 PM, Davide Ferrari > wrote:


Hello

maybe this is more glustefs then ovirt related but since OVirt
integrates Gluster management and I'm experiencing the problem in
an ovirt cluster, I'm writing here.

The problem is simple: I have a data domain mappend on a replica 3
arbiter1 Gluster volume with 6 bricks, like this:

Status of volume: data_ssd
Gluster process TCP Port  RDMA Port  Online  Pid

--
Brick vm01.storage.billy:/gluster/ssd/data/
brick 49153 0  Y   19298
Brick vm02.storage.billy:/gluster/ssd/data/
brick 49153 0  Y   6146
Brick vm03.storage.billy:/gluster/ssd/data/
arbiter_brick 49153 0  Y   6552
Brick vm03.storage.billy:/gluster/ssd/data/
brick 49154 0  Y   6559
Brick vm04.storage.billy:/gluster/ssd/data/
brick 49152 0  Y   6077
Brick vm02.storage.billy:/gluster/ssd/data/
arbiter_brick 49154 0  Y   6153
Self-heal Daemon on localhost N/A   N/AY   30746
Self-heal Daemon on vm01.storage.billy N/A   N/A   
Y   196058
Self-heal Daemon on vm03.storage.billy N/A   N/A   
Y   23205
Self-heal Daemon on vm04.storage.billy N/A   N/A   
Y   8246



Now, I've put in maintenance the vm04 host, from ovirt, ticking
the "Stop gluster" checkbox, and Ovirt didn't complain about
anything. But when I tried to run a new VM it complained about
"storage I/O problem", while the storage data status was always UP.

Looking in the gluster logs I can see this:

[2016-09-29 11:01:01.556908] I
[glusterfsd-mgmt.c:1596:mgmt_getspec_cbk] 0-glusterfs: No change
in volfile, continuing
[2016-09-29 11:02:28.124151] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_txn_refresh_done]
0-data_ssd-replicate-1: Failing READ on gfid
bf5922b7-19f3-4ce3-98df-71e981ecca8d: split-brain observed.
[Input/output error]
[2016-09-29 11:02:28.126580] W [MSGID: 108008]
[afr-read-txn.c:244:afr_read_txn] 0-data_ssd-replicate-1:
Unreadable subvolume -1 found with event generation 6 for gfid
bf5922b7-19f3-4ce3-98df-71e981ecca8d. (Possible split-brain)
[2016-09-29 11:02:28.127374] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_txn_refresh_done]
0-data_ssd-replicate-1: Failing FGETXATTR on gfid
bf5922b7-19f3-4ce3-98df-71e981ecca8d: split-brain observed.
[Input/output error]
[2016-09-29 11:02:28.128130] W [MSGID: 108027]
[afr-common.c:2403:afr_discover_done] 0-data_ssd-replicate-1: no
read subvols for (null)
[2016-09-29 11:02:28.129890] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 8201: READ => -1
gfid=bf5922b7-19f3-4ce3-98df-71e981ecca8d fd=0x7f09b749d210
(Input/output error)
[2016-09-29 11:02:28.130824] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_txn_refresh_done]
0-data_ssd-replicate-1: Failing FSTAT on gfid
bf5922b7-19f3-4ce3-98df-71e981ecca8d: split-brain observed.
[Input/output error]



Does `gluster volume heal data_ssd info split-brain` report that the 
file is in split-brain, with vm04 still being down?
If yes, could you provide the extended attributes of this gfid from all 
3 bricks:
getfattr -d -m . -e hex 
/path/to/brick/bf/59/bf5922b7-19f3-4ce3-98df-71e981ecca8d


If no, then I'm guessing that it is not in actual split-brain (hence the 
'Possible split-brain' message). If the node you brought down contains 
the only good copy of the file (i.e the other data brick and arbiter are 
up, and the arbiter 'blames' this other brick), all I/O is failed with 
EIO to prevent file from getting into actual split-brain. The heals will 
happen when the good node comes up and I/O should be allowed again in 
that case.


-Ravi



[2016-09-29 11:02:28.133879] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 8202: FSTAT()

/ba2bd397-9222-424d-aecc-eb652c0169d9/images/f02ac1ce-52cd-4b81-8b29-f8006d0469e0/ff4e49c6-3084-4234-80a1-18a67615c527
=> -1 (Input/output error)
The message "W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn]
0-data_ssd-replicate-1: Unreadable subvolume -1 found with event
generation 6 for gfid bf5922b7-19f3-4ce3-98df-71e981ecca8d.
(Possible split-brain)" repeated 11 times between [2016-09-29
11:02:28.126580] and [2016-09-29 11:02:28.517744]
[2016-09-29 11:02:28.518607] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_txn_refresh_done]
0-data_ssd-replicate-1: Failing STAT on gfid
bf5922b7-19f3-4ce3-98df-71e981ecca8d: split-brain observed.
[Input/output error]

Now, how is it possible to have a split brain if I stopped just

Re: [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements

2017-07-20 Thread Ravishankar N 


On 07/20/2017 02:20 PM, yayo (j) wrote:

Hi,

Thank you for the answer and sorry for delay:

2017-07-19 16:55 GMT+02:00 Ravishankar N <ravishan...@redhat.com 
<mailto:ravishan...@redhat.com>>:


1. What does the glustershd.log say on all 3 nodes when you run
the command? Does it complain anything about these files?


No, glustershd.log is clean, no extra log after command on all 3 nodes


Could you check if the self-heal daemon on all nodes is connected to the 
3 bricks? You will need to check the glustershd.log for that.
If it is not connected, try restarting the shd using `gluster volume 
start engine force`, then launch the heal command like you did earlier 
and see if heals happen.


If it doesn't, please provide the getfattr outputs of the 12 files from 
all 3 nodes using `getfattr -d -m . -e hex 
//gluster/engine/brick//path-to-file` ?


Thanks,
Ravi


2. Are these 12 files also present in the 3rd data brick?


I've checked right now: all files exists in all 3 nodes

3. Can you provide the output of `gluster volume info` for the
this volume?



/Volume Name: engine/
/Type: Replicate/
/Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/
/Status: Started/
/Snapshot Count: 0/
/Number of Bricks: 1 x 3 = 3/
/Transport-type: tcp/
/Bricks:/
/Brick1: node01:/gluster/engine/brick/
/Brick2: node02:/gluster/engine/brick/
/Brick3: node04:/gluster/engine/brick/
/Options Reconfigured:/
/nfs.disable: on/
/performance.readdir-ahead: on/
/transport.address-family: inet/
/storage.owner-uid: 36/
/performance.quick-read: off/
/performance.read-ahead: off/
/performance.io-cache: off/
/performance.stat-prefetch: off/
/performance.low-prio-threads: 32/
/network.remote-dio: off/
/cluster.eager-lock: enable/
/cluster.quorum-type: auto/
/cluster.server-quorum-type: server/
/cluster.data-self-heal-algorithm: full/
/cluster.locking-scheme: granular/
/cluster.shd-max-threads: 8/
/cluster.shd-wait-qlength: 1/
/features.shard: on/
/user.cifs: off/
/storage.owner-gid: 36/
/features.shard-block-size: 512MB/
/network.ping-timeout: 30/
/performance.strict-o-direct: on/
/cluster.granular-entry-heal: on/
/auth.allow: */

  server.allow-insecure: on





Some extra info:

We have recently changed the gluster from: 2 (full
repliacated) + 1 arbiter to 3 full replicated cluster



Just curious, how did you do this? `remove-brick` of arbiter
brick  followed by an `add-brick` to increase to replica-3?


Yes


#gluster volume remove-brick engine replica 2 
node03:/gluster/data/brick force *(OK!)*


#gluster volume heal engine info *(no entries!)*

#gluster volume add-brick engine replica 3 
node04:/gluster/engine/brick *(OK!)*


*After some minutes*

[root@node01 ~]#  gluster volume heal engine info
Brick node01:/gluster/engine/brick
Status: Connected
Number of entries: 0

Brick node02:/gluster/engine/brick
Status: Connected
Number of entries: 0

Brick node04:/gluster/engine/brick
Status: Connected
Number of entries: 0

Thanks,
Ravi


Another extra info (I don't know if this can be the problem): Five 
days ago A black out has suddenly shut down the networks switch (also 
gluster network) of node 03 and 04 ... But I don't know this problem 
is in place after this black out


Thank you!



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements

2017-07-20 Thread Ravishankar N 



On 07/20/2017 03:42 PM, yayo (j) wrote:


2017-07-20 11:34 GMT+02:00 Ravishankar N <ravishan...@redhat.com 
<mailto:ravishan...@redhat.com>>:



Could you check if the self-heal daemon on all nodes is connected
to the 3 bricks? You will need to check the glustershd.log for that.
If it is not connected, try restarting the shd using `gluster
volume start engine force`, then launch the heal command like you
did earlier and see if heals happen.


I've executed the command on all 3 nodes (Know is enougth only one) , 
after that the "heal" command report elements between 6 and 10 ... 
(sometime 6, sometime 8, sometime 10)



Log on glustershd.log don't say anything :


But it does  say something. All these gfids of completed heals in the 
log below are the for the ones that you have given the getfattr output 
of. So what is likely happening is there is an intermittent connection 
problem between your mount and the brick process, leading to pending 
heals again after the heal gets completed, which is why the numbers are 
varying each time. You would need to check why that is the case.

Hope this helps,
Ravi



/[2017-07-20 09:58:46.573079] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal]
0-engine-replicate-0: Completed data selfheal on
e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1  sinks=2/
/[2017-07-20 09:59:22.995003] I [MSGID: 108026]
[afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
0-engine-replicate-0: performing metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81/
/[2017-07-20 09:59:22.999372] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal]
0-engine-replicate-0: Completed metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1  sinks=2/


If it doesn't, please provide the getfattr outputs of the 12 files
from all 3 nodes using `getfattr -d -m . -e hex
//gluster/engine/brick//path-to-file` ?


*/NODE01:/*
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68/
/trusted.afr.dirty=0x/
/trusted.afr.engine-client-1=0x/
/trusted.afr.engine-client-2=0x0012/
/trusted.bit-rot.version=0x090059647d5b000447e9/
/trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48/
/trusted.afr.dirty=0x/
/trusted.afr.engine-client-1=0x/
/trusted.afr.engine-client-2=0x000e/
/trusted.bit-rot.version=0x090059647d5b000447e9/
/trusted.gfid=0x676067891f344c1586b8c0d05b07f187/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:

gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01/
/trusted.afr.dirty=0x/
/trusted.afr.engine-client-1=0x/
/trusted.afr.engine-client-2=0x0055/
/trusted.bit-rot.version=0x090059647d5b000447e9/
/trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7/
/trusted.glusterfs.shard.block-size=0x2000/

/trusted.glusterfs.shard.file-size=0x000c80d4f229/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60/
/trusted.afr.dirty=0x/
/trusted.afr.engine-client-1=0x/
/trusted.afr.engine-client-2=0x0007/
/trusted.bit-rot.version=0x090059647d5b000447e9/
/trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids/
/trusted.afr.dirty=0x/
/trusted.afr.engine-client-1=0x/
/trusted.afr.engine-client-2=0x/
/trusted.bit-rot.version=0x0f0059647d5b000447e9/
/trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3/
/trusted.glusterfs.shard.block-size=0x2000/

/trusted.glusterfs.shard.file-size=0x00100800/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file: gluster/engine/brick/__DIRECT_IO_TEST__/
/trusted.afr.dirty=0x/
/trusted.afr.engine-client-1=0x/
/trusted.afr.engine-client-2=0x0

Re: [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements

2017-07-21 Thread Ravishankar N 


On 07/21/2017 02:55 PM, yayo (j) wrote:
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishan...@redhat.com 
<mailto:ravishan...@redhat.com>>:



But it does  say something. All these gfids of completed heals in
the log below are the for the ones that you have given the
getfattr output of. So what is likely happening is there is an
intermittent connection problem between your mount and the brick
process, leading to pending heals again after the heal gets
completed, which is why the numbers are varying each time. You
would need to check why that is the case.
Hope this helps,
Ravi




/[2017-07-20 09:58:46.573079] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal]
0-engine-replicate-0: Completed data selfheal on
e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1  sinks=2/
/[2017-07-20 09:59:22.995003] I [MSGID: 108026]
[afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
0-engine-replicate-0: performing metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81/
/[2017-07-20 09:59:22.999372] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal]
0-engine-replicate-0: Completed metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1  sinks=2/




Hi,

But we ha1e 2 gluster volume on the same network and the other one 
(the "Data" gluster) don't have any problems. Why you think there is a 
network problem?


Because pending self-heals come into the picture when I/O from the 
clients (mounts) do not succeed on some bricks. They are mostly due to

(a) the client losing connection to some bricks (likely),
(b) the I/O failing on the bricks themselves (unlikely).

If most of the i/o is also going to the 3rd brick (since you say the 
files are already present on all bricks and I/O is successful) , then it 
is likely to be (a).



How to check this on a gluster infrastructure?

In the fuse mount logs for the engine volume, check if there are any 
messages for brick disconnects. Something along the lines of 
"disconnected from volname-client-x".
Just guessing here, but maybe even the 'data' volume did experience 
disconnects and self-heals later but you did not observe it when you ran 
heal info. See the glustershd log or mount log for for self-heal 
completion messages on /0-data-replicate-0 /also.


Regards,
Ravi

Thank you





___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements

2017-07-22 Thread Ravishankar N 


On 07/21/2017 11:41 PM, yayo (j) wrote:

Hi,

Sorry for follow up again, but, checking the ovirt interface I've 
found that ovirt report the "engine" volume as an "arbiter" 
configuration and the "data" volume as full replicated volume. Check 
these screenshots:


This is probably some refresh bug in the UI, Sahina might be able to 
tell you.


https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFfVmR5aDQ?usp=sharing

But the "gluster volume info" command report that all 2 volume are 
full replicated:



/Volume Name: data/
/Type: Replicate/
/Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d/
/Status: Started/
/Snapshot Count: 0/
/Number of Bricks: 1 x 3 = 3/
/Transport-type: tcp/
/Bricks:/
/Brick1: gdnode01:/gluster/data/brick/
/Brick2: gdnode02:/gluster/data/brick/
/Brick3: gdnode04:/gluster/data/brick/
/Options Reconfigured:/
/nfs.disable: on/
/performance.readdir-ahead: on/
/transport.address-family: inet/
/storage.owner-uid: 36/
/performance.quick-read: off/
/performance.read-ahead: off/
/performance.io-cache: off/
/performance.stat-prefetch: off/
/performance.low-prio-threads: 32/
/network.remote-dio: enable/
/cluster.eager-lock: enable/
/cluster.quorum-type: auto/
/cluster.server-quorum-type: server/
/cluster.data-self-heal-algorithm: full/
/cluster.locking-scheme: granular/
/cluster.shd-max-threads: 8/
/cluster.shd-wait-qlength: 1/
/features.shard: on/
/user.cifs: off/
/storage.owner-gid: 36/
/features.shard-block-size: 512MB/
/network.ping-timeout: 30/
/performance.strict-o-direct: on/
/cluster.granular-entry-heal: on/
/auth.allow: */
/server.allow-insecure: on/





/Volume Name: engine/
/Type: Replicate/
/Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/
/Status: Started/
/Snapshot Count: 0/
/Number of Bricks: 1 x 3 = 3/
/Transport-type: tcp/
/Bricks:/
/Brick1: gdnode01:/gluster/engine/brick/
/Brick2: gdnode02:/gluster/engine/brick/
/Brick3: gdnode04:/gluster/engine/brick/
/Options Reconfigured:/
/nfs.disable: on/
/performance.readdir-ahead: on/
/transport.address-family: inet/
/storage.owner-uid: 36/
/performance.quick-read: off/
/performance.read-ahead: off/
/performance.io-cache: off/
/performance.stat-prefetch: off/
/performance.low-prio-threads: 32/
/network.remote-dio: off/
/cluster.eager-lock: enable/
/cluster.quorum-type: auto/
/cluster.server-quorum-type: server/
/cluster.data-self-heal-algorithm: full/
/cluster.locking-scheme: granular/
/cluster.shd-max-threads: 8/
/cluster.shd-wait-qlength: 1/
/features.shard: on/
/user.cifs: off/
/storage.owner-gid: 36/
/features.shard-block-size: 512MB/
/network.ping-timeout: 30/
/performance.strict-o-direct: on/
/cluster.granular-entry-heal: on/
/auth.allow: */

  server.allow-insecure: on


2017-07-21 19:13 GMT+02:00 yayo (j) <jag...@gmail.com 
<mailto:jag...@gmail.com>>:


2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishan...@redhat.com
<mailto:ravishan...@redhat.com>>:


But it does  say something. All these gfids of completed heals
in the log below are the for the ones that you have given the
getfattr output of. So what is likely happening is there is an
intermittent connection problem between your mount and the
brick process, leading to pending heals again after the heal
gets completed, which is why the numbers are varying each
time. You would need to check why that is the case.
Hope this helps,
Ravi




/[2017-07-20 09:58:46.573079] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal]
0-engine-replicate-0: Completed data selfheal on
e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1  sinks=2/
/[2017-07-20 09:59:22.995003] I [MSGID: 108026]
[afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
0-engine-replicate-0: performing metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81/
/[2017-07-20 09:59:22.999372] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal]
0-engine-replicate-0: Completed metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1  sinks=2/




Hi,

following your suggestion, I've checked the "peer" status and I
found that there is too many name for the hosts, I don't know if
this can be the problem or part of it:

/*gluster peer status on NODE01:*/
/Number of Peers: 2/
/
/
/Hostname: dnode02.localdomain.local/
/Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd/
/State: Peer in Cluster (Connected)/

Re: [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements

2017-07-19 Thread Ravishankar N 



On 07/19/2017 08:02 PM, Sahina Bose wrote:

[Adding gluster-users]

On Wed, Jul 19, 2017 at 2:52 PM, yayo (j) > wrote:


Hi all,

We have an ovirt cluster hyperconverged with hosted engine on 3
full replicated node . This cluster have 2 gluster volume:

- data: volume for the Data (Master) Domain (For vm)
- engine: volume fro the hosted_storage  Domain (for hosted engine)

We have this problem: "engine" gluster volume have always unsynced
elements and we cant' fix the problem, on command line we have
tried to use the "heal" command but elements remain always
unsynced 

Below the heal command "status":

[root@node01 ~]# gluster volume heal engine info
Brick node01:/gluster/engine/brick
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.2
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68

/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01

/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.61
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.1
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.20
/__DIRECT_IO_TEST__
Status: Connected
Number of entries: 12

Brick node02:/gluster/engine/brick

/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01

/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids



/__DIRECT_IO_TEST__



/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6


Status: Connected
Number of entries: 12

Brick node04:/gluster/engine/brick
Status: Connected
Number of entries: 0


running the "gluster volume heal engine" don't solve the problem...



1. What does the glustershd.log say on all 3 nodes when you run the 
command? Does it complain anything about these files?

2. Are these 12 files also present in the 3rd data brick?
3. Can you provide the output of `gluster volume info` for the this volume?


Some extra info:

We have recently changed the gluster from: 2 (full repliacated) +
1 arbiter to 3 full replicated cluster



Just curious, how did you do this? `remove-brick` of arbiter brick 
followed by an `add-brick` to increase to replica-3?


Thanks,
Ravi


but i don't know this is the problem...

The "data" volume is good and healty and have no unsynced entry.

Ovirt refuse to put the node02 and node01 in "maintenance mode"
and complains about "unsynced elements"

How can I fix this?
Thank you

___
Users mailing list
Users@ovirt.org 
http://lists.ovirt.org/mailman/listinfo/users





___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] can i Extend volume from replica 2 to replica 3 with arbiter

2017-05-15 Thread Ravishankar N 



-- Forwarded message --
From: *Khalid Jamal* >

Date: Sat, May 13, 2017 at 10:51 PM
Subject: [ovirt-users] can i Extend volume from replica 2 to replica 3 
with arbiter

To: users@ovirt.org 


Dear Team

i need you advice for convert our volume replica 2 to replica 3 with 
arbiter but important thing it's production environments , i try to 
convert the same name of volume in replica 2 to convert is to the 
replica 3 to avoid losing vm's or any data that's what i do it :



# gluster volume create gfs1 replica 3 arbiter 1 s1:/export/sdb1/br1 
s2:/export/sdb1/br1 s3:/export/sdb1/br1 s1:/export/sdc1/br1 
s2:/export/sdc1/br1 s3:/export/sdc1/br1 s4:/export/sdb1/br1 
s5:/export/sdb1/br1 s6:/export/sdb1/br1 s4:/export/sdc1/br1 
s5:/export/sdc1/br1 s6:/export/sdc1/br1


that's the result :


volume create: gfs1: failed: Volume gfs1 already exists


If you are converting an existing replica 2 to arbiter, you would need 
to use the `add-brick` command. The syntax is given here: 
https://review.gluster.org/#/c/14126/8//COMMIT_MSG

Hope that helps,
Ravi


i try to change the name but without any success what shall i do.


best regards



Eng khalid jamal
System Admin@IT Department
Earthlink Telecom

Email: khalid.ja...@earthlinktele.com 


No: 3355
skype: engkhalid21986
NO : 07704268321


___
Users mailing list
Users@ovirt.org 
http://lists.ovirt.org/mailman/listinfo/users 






___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] gluster replica volumes with different data usage

2017-09-18 Thread Ravishankar N 
Possibly.  I don't think changing shard size on the fly is supported, 
especially when there are files on the volume that are sharded with a 
different size.


-Ravi

On 09/18/2017 11:40 AM, Alex K wrote:
The heal status is showing that no pending files need healing (also 
shown at GUI).
When checking the bricks on the file system I see that what is 
different between the server is the .shard folder of the volume. One 
server reports 835GB while the other 1.1 TB.

I recall to have changed the shard size at some point from 4 MB to 64MB.
Could this be the cause?

Thanx,
Alex

On Mon, Sep 18, 2017 at 8:14 AM, Ravishankar N <ravishan...@redhat.com 
<mailto:ravishan...@redhat.com>> wrote:



On 09/18/2017 10:08 AM, Alex K wrote:

Hi Ravishankar,

I am not referring to the arbiter volume(which is showing 0%
usage). I am referring to the other 2 volumes which are replicas
and should have the exact same data. Checking the status of other
bricks in ovirt (bricks used from iso and export domain) I see
that they all report same usage of data on the data volumes,
except the "vms" volume used for storing vms.


Ah, okay.  Some of the things that can cause a variation in disk
usage:
- Pending self-heals in gluster (check if `gluster volume heal
 info` doesn't show any entries.  Also if there is
anything under `.glusterfs/landfill` folder of the bricks).
- XFS speculative preallocation
- Possibly some bug in self-healing of sparse files by gluster
(although we fixed known bugs a long time back in this area).

Regards
Ravi



Thanx,
Alex

On Sep 18, 2017 07:00, "Ravishankar N" <ravishan...@redhat.com
<mailto:ravishan...@redhat.com>> wrote:



On 09/17/2017 08:41 PM, Alex K wrote:

Hi all,

I have replica 3 with 1 arbiter.
When checking the gluster volume bricks they are reported as
using different space, as per attached. How come they use
different space? One would expect to use exactly the same
space since they are replica.


The 3rd brick (arbiter ) only holds meta data, so it would
not consume as much space as the other 2 data bricks. So what
you are seeing is expected behaviour.
Regards,
Ravi

Thanx,
Alex


___
Users mailing list
Users@ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>







___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] gluster replica volumes with different data usage

2017-09-17 Thread Ravishankar N 



On 09/17/2017 08:41 PM, Alex K wrote:

Hi all,

I have replica 3 with 1 arbiter.
When checking the gluster volume bricks they are reported as using 
different space, as per attached. How come they use different space? 
One would expect to use exactly the same space since they are replica.


The 3rd brick (arbiter ) only holds meta data, so it would not consume 
as much space as the other 2 data bricks. So what you are seeing is 
expected behaviour.

Regards,
Ravi

Thanx,
Alex


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] gluster replica volumes with different data usage

2017-09-17 Thread Ravishankar N 


On 09/18/2017 10:08 AM, Alex K wrote:

Hi Ravishankar,

I am not referring to the arbiter volume(which is showing 0% usage). I 
am referring to the other 2 volumes which are replicas and should have 
the exact same data. Checking the status of other bricks in ovirt 
(bricks used from iso and export domain) I see that they all report 
same usage of data on the data volumes, except the "vms" volume used 
for storing vms.


Ah, okay.  Some of the things that can cause a variation in disk usage:
- Pending self-heals in gluster (check if `gluster volume heal  
info` doesn't show any entries.  Also if there is anything under 
`.glusterfs/landfill` folder of the bricks).

- XFS speculative preallocation
- Possibly some bug in self-healing of sparse files by gluster (although 
we fixed known bugs a long time back in this area).


Regards
Ravi



Thanx,
Alex

On Sep 18, 2017 07:00, "Ravishankar N" <ravishan...@redhat.com 
<mailto:ravishan...@redhat.com>> wrote:




On 09/17/2017 08:41 PM, Alex K wrote:

Hi all,

I have replica 3 with 1 arbiter.
When checking the gluster volume bricks they are reported as
using different space, as per attached. How come they use
different space? One would expect to use exactly the same space
since they are replica.


The 3rd brick (arbiter ) only holds meta data, so it would not
consume as much space as the other 2 data bricks. So what you are
seeing is expected behaviour.
Regards,
Ravi

Thanx,
Alex


___
Users mailing list
Users@ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>




___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Re: HE + Gluster : Engine corrupted?

2018-07-02 Thread Ravishankar N 



On 07/02/2018 02:15 PM, Krutika Dhananjay wrote:

Hi,

So it seems some of the files in the volume have mismatching gfids. I 
see the following logs from 15th June, ~8pm EDT:



...
...
[2018-06-16 04:00:10.264690] E [MSGID: 108008] 
[afr-self-heal-common.c:335:afr_gfid_split_brain_source] 
0-engine-replicate-0: Gfid mismatch detected for 
/hosted-engine.lockspace>, 
6bbe6097-8520-4a61-971e-6e30c2ee0abe on engine-client-2 and 
ef21a706-41cf-4519-8659-87ecde4bbfbf on engine-client-0.


You can use 
https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/ 
(see 3. Resolution of split-brain using gluster CLI).
Nit: The doc says in the beginning that gfid split-brain cannot be fixed 
automatically but newer releases do support it, so the methods in 
section 3 should work to solve gfid split-brains.


[2018-06-16 04:00:10.265861] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4411: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:11.522600] E [MSGID: 108008] 
[afr-self-heal-common.c:212:afr_gfid_split_brain_source] 
0-engine-replicate-0: All the bricks should be up to resolve the gfid 
split barin

This is a concern. For the commands to work, all 3 bricks must be online.
Thanks,
Ravi
[2018-06-16 04:00:11.522632] E [MSGID: 108008] 
[afr-self-heal-common.c:335:afr_gfid_split_brain_source] 
0-engine-replicate-0: Gfid mismatch detected for 
/hosted-engine.lockspace>, 
6bbe6097-8520-4a61-971e-6e30c2ee0abe on engine-client-2 and 
ef21a706-41cf-4519-8659-87ecde4bbfbf on engine-client-0.
[2018-06-16 04:00:11.523750] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4493: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:12.864393] E [MSGID: 108008] 
[afr-self-heal-common.c:212:afr_gfid_split_brain_source] 
0-engine-replicate-0: All the bricks should be up to resolve the gfid 
split barin
[2018-06-16 04:00:12.864426] E [MSGID: 108008] 
[afr-self-heal-common.c:335:afr_gfid_split_brain_source] 
0-engine-replicate-0: Gfid mismatch detected for 
/hosted-engine.lockspace>, 
6bbe6097-8520-4a61-971e-6e30c2ee0abe on engine-client-2 and 
ef21a706-41cf-4519-8659-87ecde4bbfbf on engine-client-0.
[2018-06-16 04:00:12.865392] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4575: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:18.716007] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4657: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:20.553365] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4739: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:21.771698] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4821: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:23.871647] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4906: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:25.034780] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4987: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)

...
...


Adding Ravi who works on replicate component to hep resolve the 
mismatches.


-Krutika


On Mon, Jul 2, 2018 at 12:27 PM, Krutika Dhananjay 
mailto:kdhan...@redhat.com>> wrote:


Hi,

Sorry, I was out sick on Friday. I am looking into the logs. Will
get back to you in some time.

-Krutika

On Fri, Jun 29, 2018 at 7:47 PM, Hanson Turner
mailto:han...@andrewswireless.net>>
wrote:

Hi Krutika,

Did you need any other logs?


Thanks,

Hanson


On 06/27/2018 02:04 PM, Hanson Turner wrote:


Hi Krutika,

Looking at the email spams, it looks like it started at
8:04PM EDT on Jun 15 2018.

From my memory, I think the cluster was working fine until
sometime that night. Somewhere between midnight and the next
(Saturday) morning, the engine crashed and all vm's stopped.

I do have nightly backups that ran every night, using the
engine-backup command. Looks like my last valid backup was
2018-06-15.

I've included all logs I think might be of use. Please
forgive the use of 7zip, as the raw logs took 50mb which is
greater than my attachment limit.

I think the just of what happened, is we had a downed node
for a period of time. Earlier that day, the node was brought
back into service. Later that night or early the next
morning, the engine was gone and hopping from node to node.

[ovirt-users] Re: HE + Gluster : Engine corrupted?

2018-07-18 Thread Ravishankar N 

Hi,

"[2018-06-16 04:00:10.264690] E [MSGID: 108008] 
[afr-self-heal-common.c:335:afr_gfid_split_brain_source] 
0-engine-replicate-0: Gfid mismatch detected for 
/hosted-engine.lockspace>, 
6bbe6097-8520-4a61-971e-6e30c2ee0abe on engine-client-2 and 
ef21a706-41cf-4519-8659-87ecde4bbfbf on engine-client-0."


Are the gfids actually different on the bricks as this message says? If 
yes, then the commands shared earlier should have fixed it.


-Ravi


On 07/02/2018 02:15 PM, Krutika Dhananjay wrote:

Hi,

So it seems some of the files in the volume have mismatching gfids. I 
see the following logs from 15th June, ~8pm EDT:



...
...
[2018-06-16 04:00:10.264690] E [MSGID: 108008] 
[afr-self-heal-common.c:335:afr_gfid_split_brain_source] 
0-engine-replicate-0: Gfid mismatch detected for 
/hosted-engine.lockspace>, 
6bbe6097-8520-4a61-971e-6e30c2ee0abe on engine-client-2 and 
ef21a706-41cf-4519-8659-87ecde4bbfbf on engine-client-0.
[2018-06-16 04:00:10.265861] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4411: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:11.522600] E [MSGID: 108008] 
[afr-self-heal-common.c:212:afr_gfid_split_brain_source] 
0-engine-replicate-0: All the bricks should be up to resolve the gfid 
split barin
[2018-06-16 04:00:11.522632] E [MSGID: 108008] 
[afr-self-heal-common.c:335:afr_gfid_split_brain_source] 
0-engine-replicate-0: Gfid mismatch detected for 
/hosted-engine.lockspace>, 
6bbe6097-8520-4a61-971e-6e30c2ee0abe on engine-client-2 and 
ef21a706-41cf-4519-8659-87ecde4bbfbf on engine-client-0.
[2018-06-16 04:00:11.523750] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4493: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:12.864393] E [MSGID: 108008] 
[afr-self-heal-common.c:212:afr_gfid_split_brain_source] 
0-engine-replicate-0: All the bricks should be up to resolve the gfid 
split barin
[2018-06-16 04:00:12.864426] E [MSGID: 108008] 
[afr-self-heal-common.c:335:afr_gfid_split_brain_source] 
0-engine-replicate-0: Gfid mismatch detected for 
/hosted-engine.lockspace>, 
6bbe6097-8520-4a61-971e-6e30c2ee0abe on engine-client-2 and 
ef21a706-41cf-4519-8659-87ecde4bbfbf on engine-client-0.
[2018-06-16 04:00:12.865392] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4575: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:18.716007] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4657: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:20.553365] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4739: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:21.771698] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4821: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:23.871647] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4906: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)
[2018-06-16 04:00:25.034780] W [fuse-bridge.c:540:fuse_entry_cbk] 
0-glusterfs-fuse: 4987: LOOKUP() 
/c65e03f0-d553-4d5d-ba4f-9d378c153b9b/ha_agent/hosted-engine.lockspace 
=> -1 (Input/output error)

...
...


Adding Ravi who works on replicate component to hep resolve the 
mismatches.


-Krutika


On Mon, Jul 2, 2018 at 12:27 PM, Krutika Dhananjay 
mailto:kdhan...@redhat.com>> wrote:


Hi,

Sorry, I was out sick on Friday. I am looking into the logs. Will
get back to you in some time.

-Krutika

On Fri, Jun 29, 2018 at 7:47 PM, Hanson Turner
mailto:han...@andrewswireless.net>>
wrote:

Hi Krutika,

Did you need any other logs?


Thanks,

Hanson


On 06/27/2018 02:04 PM, Hanson Turner wrote:


Hi Krutika,

Looking at the email spams, it looks like it started at
8:04PM EDT on Jun 15 2018.

From my memory, I think the cluster was working fine until
sometime that night. Somewhere between midnight and the next
(Saturday) morning, the engine crashed and all vm's stopped.

I do have nightly backups that ran every night, using the
engine-backup command. Looks like my last valid backup was
2018-06-15.

I've included all logs I think might be of use. Please
forgive the use of 7zip, as the raw logs took 50mb which is
greater than my attachment limit.

I think the just of what happened, is we had a downed node
for a period of time. Earlier that day, the node was brought
back into service. Later that night or early the next
morning, the engine was gone and hopping

[ovirt-users] Re: storage healing question

2018-11-12 Thread Ravishankar N 

Hi,

Can you restart the self-heal daemon by doing a `gluster volume start 
bgl-vms-gfs force` and then launch the heal again? If you are seeing 
different entries and counts each time you run heal info, there is 
likely a network issue (disconnect) between the (gluster fuse?) mount 
and the bricks of the volume leading to pending heals.


Also, there was a bug in arbiter volumes[1] that got fixed in glusterfs 
3.12.15. It can cause VMs to pause when you reboot the arbiter node, so 
it is recommended to upgrade to this gluster version.


HTH,
Ravi

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1637989

From: Dev Ops
Date: Mon, Nov 12, 2018 at 1:09 PM
Subject: [ovirt-users] Re: storage healing question
To:


Any help would be appreciated. I have since rebooted the 3rd gluster
node which is the arbiter. This doesn't seem to want to heal.

gluster volume heal bgl-vms-gfs info |grep Number
Number of entries: 68
Number of entries: 0
Number of entries: 68
___
Users mailing list --users@ovirt.org
To unsubscribe send an email tousers-le...@ovirt.org
Privacy Statement:https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List 
Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/XNHA6WS5MGCLXJX3HCDGLZRVJ7E5Q7NX/


___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YFN4DQKKX6YQR6W4U2EC2OJQS5TOJJSW/

[ovirt-users] Re: Gluster arbiter volume storage domain - change

2019-04-16 Thread Ravishankar N 


On 16/04/19 2:20 PM, Sahina Bose wrote:

On Tue, Apr 16, 2019 at 1:39 PM Leo David  wrote:

Hi Everyone,
I have wrongly configured the main gluster volume ( 12 identical 1tb ssd disks, 
replica 3 distributed-replicated, across 6 nodes - 2 per node ) with arbiter 
one.
Oviously I am wasting storage space in this scenario with the arbiter bricks, 
and I would like to convert the volume to non-arbitrated one, so having all the 
data evenly spreaded across all the disks.
Considering the the storage is being used by about 40 vms in production, what 
would it be the steps, or is there any chance to change the volume type to 
non-arbitrated on the fly and then rebalance ?
Thank you very much !

Ravi, can you help here - to change from arbiter to replica 3?


The general steps are:

1. Ensure there are no pending heals.

2. Use the `remove-brick` command to reduce the volume to a replica 2

3. Use the `add-brick` command to convert it to a replica 3.

4. Monitor and check that the heal is eventually completed on the newly 
added bricks.


The steps are best done when the VMs are offline so that self-heal 
traffic does not eat up too much of I/O traffic.


Example:

[root@tuxpad ravi]# gluster volume info

Volume Name: testvol
Type: Distributed-Replicate
Volume ID: e3fc6ea5-a48c-4918-8a4b-0a7859f3a182
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: 127.0.0.2:/home/ravi/bricks/brick1
Brick2: 127.0.0.2:/home/ravi/bricks/brick2
Brick3: 127.0.0.2:/home/ravi/bricks/brick3 (arbiter)
Brick4: 127.0.0.2:/home/ravi/bricks/brick4
Brick5: 127.0.0.2:/home/ravi/bricks/brick5
Brick6: 127.0.0.2:/home/ravi/bricks/brick6 (arbiter)
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root@tuxpad ravi]#

[root@tuxpad ravi]# gluster volume heal testvol info
Brick 127.0.0.2:/home/ravi/bricks/brick1
Status: Connected
Number of entries: 0

Brick 127.0.0.2:/home/ravi/bricks/brick2
Status: Connected
Number of entries: 0

Brick 127.0.0.2:/home/ravi/bricks/brick3
Status: Connected
Number of entries: 0

Brick 127.0.0.2:/home/ravi/bricks/brick4
Status: Connected
Number of entries: 0

Brick 127.0.0.2:/home/ravi/bricks/brick5
Status: Connected
Number of entries: 0

Brick 127.0.0.2:/home/ravi/bricks/brick6
Status: Connected
Number of entries: 0

[root@tuxpad ravi]#
[root@tuxpad ravi]# gluster volume remove-brick testvol replica 2 
127.0.0.2:/home/ravi/bricks/brick3 127.0.0.2:/home/ravi/bricks/brick6  force

Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit force: success
[root@tuxpad ravi]#
[root@tuxpad ravi]# gluster volume info

Volume Name: testvol
Type: Distributed-Replicate
Volume ID: e3fc6ea5-a48c-4918-8a4b-0a7859f3a182
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 127.0.0.2:/home/ravi/bricks/brick1
Brick2: 127.0.0.2:/home/ravi/bricks/brick2
Brick3: 127.0.0.2:/home/ravi/bricks/brick4
Brick4: 127.0.0.2:/home/ravi/bricks/brick5
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root@tuxpad ravi]#
[root@tuxpad ravi]# gluster volume add-brick testvol replica 3 
127.0.0.2:/home/ravi/bricks/brick3_new 
127.0.0.2:/home/ravi/bricks/brick6_new

volume add-brick: success
[root@tuxpad ravi]#
[root@tuxpad ravi]#
[root@tuxpad ravi]# gluster volume info

Volume Name: testvol
Type: Distributed-Replicate
Volume ID: e3fc6ea5-a48c-4918-8a4b-0a7859f3a182
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 127.0.0.2:/home/ravi/bricks/brick1
Brick2: 127.0.0.2:/home/ravi/bricks/brick2
Brick3: 127.0.0.2:/home/ravi/bricks/brick3_new
Brick4: 127.0.0.2:/home/ravi/bricks/brick4
Brick5: 127.0.0.2:/home/ravi/bricks/brick5
Brick6: 127.0.0.2:/home/ravi/bricks/brick6_new
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root@tuxpad ravi]#
[root@tuxpad ravi]#
[root@tuxpad ravi]# gluster volume heal testvol info
Brick 127.0.0.2:/home/ravi/bricks/brick1
Status: Connected
Number of entries: 0

Brick 127.0.0.2:/home/ravi/bricks/brick2
Status: Connected
Number of entries: 0

Brick 127.0.0.2:/home/ravi/bricks/brick3_new
Status: Connected
Number of entries: 0

Brick 127.0.0.2:/home/ravi/bricks/brick4
Status: Connected
Number of entries: 0

Brick 127.0.0.2:/home/ravi/bricks/brick5
Status: Connected
Number of entries: 0

Brick 127.0.0.2:/home/ravi/bricks/brick6_new
Status: Connected
Number of entries: 0


HTH,

Ravi





___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UBEZWN35M365IKCIE3U6TRHDDX7TS75T/