Re: [Gluster-users] [EXT] Renaming a brick.

[TomK]

Thank you all for this.  Unfortunately I wasn't able to try this.  In my 
attempt to change the brick name, I noticed I'm on Gluster 3.13  (Not a 
typo) .  So I proceeded to update to 6.10 then from there, to 9.4.  To 
6.10 it went fine however from 6.10 to 9.4 everything blew up.  I got a 
torrent of error messages.  Struggled on it for several hours but no luck.


So I ended up blowing up the cluster after copying the data out to a 
single drive, then copying it back to the Gluster 9.4 cluster once the 
cluster was ready.  Would have loved to debug to see what to do about 
each error but I simply ran out of time and needed to get the cluster 
back up quickly. ( Still have most of the logs if interested but without 
some context, it would be somewhat meaningless. )


Nontheless, I'll keep the notes below. I have a few more Gluster 
clusters to work on.


Cheers



On 2022-01-04 8:55 a.m., Olaf Buitelaar wrote:

Hi Tom,

I think you could use the replace-brick sub-command; like: "gluster 
volume replace-brick  :/bricks/0/gv01 
 :/bricks/0/abc-gv01  commit force"
see; 
https://docs.gluster.org/en/latest/Administrator-Guide/Managing-Volumes/#replace-brick


Kind regards,

Olaf

Op di 4 jan. 2022 om 12:50 schreef Stefan Solbrig :

Hi,

I use the sequence below to move a brick to a different peer. I
assume it also works for renaming a brick (just put peer01 =
peer02 = whateveryourpeeris).  I used this on a distributed-only
volume (no replication) running glusterfs 9.3.
Please note that I didn't to extensive tests and I didn't find any
documentation if this is officially supported.

best wishes,
-Stefan

#--

    # stop glusterfsd for brick:

gluster v reset-brick gvol peer01:/old/path start

    # in your case,  you probably want to do something here like

mv  /old/path /new/path

    #  add the moved path as new brick:

gluster v add-brick gvol peer02:/new/path force

    # Order is important!
    # if brick is removed before other brick is added,
    # will lead to duplicate files.
    #  remove old brick (which does not exist any longer)

gluster v remove-brick gvol peer01:/old/path force

    # probably not necessary in your case:

gluster v rebalance gvol fix-layout  start

#--


> Hey All,
>
> Is there a way to rename a brick within an existing gluster volume?
>
> /bricks/0/gv01  -> /bricks/0/abc-gv01
>
>
>
> --
> Thx,
> TK.
>





Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users






Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge:https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users



--
Thx,
TK.



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Renaming a brick.

[TomK]

Hey All,

Is there a way to rename a brick within an existing gluster volume?

/bricks/0/gv01  -> /bricks/0/abc-gv01



--
Thx,
TK.




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] GlusterFS Speed

[TomK]

Wanted to run some stats by you guy's to see if this is the sort of IO 
expected off the GlusterFS w/ oVirt that I have:


[root@mdskvm-p01 master]# iperf3 -s
---
Server listening on 5201
---
Accepted connection from 192.168.0.39, port 59108
[  5] local 192.168.0.60 port 5201 connected to 192.168.0.39 port 59110
[ ID] Interval   Transfer Bandwidth
[  5]   0.00-1.00   sec   108 MBytes   905 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   939 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   939 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   939 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   939 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   938 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   938 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   939 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   940 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   939 Mbits/sec
[  5]  10.00-10.04  sec  4.25 MBytes   939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bandwidth
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec  sender
[  5]   0.00-10.04  sec  1.09 GBytes   935 Mbits/sec 
receiver

---
Server listening on 5201
---
Accepted connection from 192.168.0.39, port 59126
[  5] local 192.168.0.60 port 5201 connected to 192.168.0.39 port 59128
iperf3: the client has unexpectedly closed the connection
---
Server listening on 5201
---
Accepted connection from 192.168.0.39, port 59164
[  6] local 192.168.0.60 port 5201 connected to 192.168.0.39 port 59166
[ ID] Interval   Transfer Bandwidth
[  6]   0.00-1.00   sec   107 MBytes   898 Mbits/sec
[  6]   1.00-2.00   sec   102 MBytes   858 Mbits/sec
[  6]   2.00-3.00   sec   107 MBytes   901 Mbits/sec
[  6]   3.00-4.00   sec   107 MBytes   896 Mbits/sec
[  6]   4.00-5.00   sec  94.5 MBytes   793 Mbits/sec
[  6]   5.00-6.00   sec  93.9 MBytes   788 Mbits/sec
[  6]   6.00-7.00   sec  78.1 MBytes   655 Mbits/sec
[  6]   7.00-8.00   sec  65.8 MBytes   552 Mbits/sec
[  6]   8.00-9.00   sec  50.6 MBytes   425 Mbits/sec
[  6]   9.00-10.00  sec  52.8 MBytes   443 Mbits/sec
[  6]  10.00-10.07  sec  4.28 MBytes   491 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bandwidth
[  6]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec  sender
[  6]   0.00-10.07  sec   864 MBytes   719 Mbits/sec 
receiver

---
Server listening on 5201
---
^Ciperf3: interrupt - the server has terminated
[root@mdskvm-p01 master]#


Then the client:


[root@mdskvm-p02 master]# iperf3 -c mdskvm-p01.nix.mds.xyz
Connecting to host mdskvm-p01.nix.mds.xyz, port 5201
[  4] local 192.168.0.39 port 59110 connected to 192.168.0.60 port 5201
[ ID] Interval   Transfer Bandwidth   Retr  Cwnd
[  4]   0.00-1.00   sec   114 MBytes   955 Mbits/sec0358 KBytes
[  4]   1.00-2.00   sec   112 MBytes   938 Mbits/sec0358 KBytes
[  4]   2.00-3.00   sec   112 MBytes   939 Mbits/sec0362 KBytes
[  4]   3.00-4.00   sec   111 MBytes   935 Mbits/sec0362 KBytes
[  4]   4.00-5.00   sec   112 MBytes   940 Mbits/sec0362 KBytes
[  4]   5.00-6.00   sec   112 MBytes   942 Mbits/sec0362 KBytes
[  4]   6.00-7.00   sec   111 MBytes   935 Mbits/sec0363 KBytes
[  4]   7.00-8.00   sec   112 MBytes   941 Mbits/sec0363 KBytes
[  4]   8.00-9.00   sec   112 MBytes   937 Mbits/sec0363 KBytes
[  4]   9.00-10.00  sec   112 MBytes   942 Mbits/sec0363 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bandwidth   Retr
[  4]   0.00-10.00  sec  1.09 GBytes   940 Mbits/sec0 sender
[  4]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec 
receiver


iperf Done.
[root@mdskvm-p02 master]#



[root@mdskvm-p02 28fcc382-f469-4f8b-af0f-6b6e048dfcca]# iperf3 -c 
mdskvm-p01.nix.mds.xyz -F e13fca65-65c5-4665-8817-9668f81ad6d4

Connecting to host mdskvm-p01.nix.mds.xyz, port 5201
[  4] local 192.168.0.39 port 59166 connected to 192.168.0.60 port 5201
[ ID] Interval   Transfer Bandwidth   Retr  Cwnd
[  4]   0.00-1.00   sec   113 MBytes   945 Mbits/sec0373 KBytes
[  4]   1.00-2.00   sec   102 MBytes   859 Mbits/sec0373 KBytes
[  4]   2.00-3.00   sec   108 MBytes   905 Mbits/sec0373 KBytes
[  4]   3.00-4.00   sec   107 MBytes   896 Mbits/sec0373 KBytes
[  4]   4.00-5.00   sec  94.5 MBytes   793 Mbits/sec0373 KBytes
[  4]   5.00-6.05   sec  88.4 MBytes   710 Mbits/sec0

Re: [Gluster-users] Where does Gluster capture the hostnames from?

[TomK]

Because this was a LAB I could quickly remove the gluster setup and 
recreated it and used the FQDN's, it quickly picked up the new names. 
Exactly as expected per this thread.


[root@mdskvm-p02 network-scripts]# gluster volume status
Status of volume: mdsgv01
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/g
lusterv01   49152 0  Y 
4375

Brick mdskvm-p02.nix.mds.xyz:/mnt/p02-d01/g
lusterv02   49152 0  Y 
4376

NFS Server on localhost N/A   N/AN   N/A
Self-heal Daemon on localhost   N/A   N/AY 
4402

NFS Server on mdskvm-p01.nix.mds.xyzN/A   N/AN   N/A
Self-heal Daemon on mdskvm-p01.nix.mds.xyz  N/A   N/AY 
4384


Task Status of Volume mdsgv01
--
There are no active volume tasks

[root@mdskvm-p02 network-scripts]#

Would be handy to have a rename function in future releases.

Cheers,
TK

On 9/25/2019 7:47 AM, TomK wrote:
Thanks Thorgeir.  Since then I upgraded to Gluster 6.  Though this issue 
remaind the same, anything in the way of new options to change what's 
displayed?


Reason for the ask is that this get's inherited by oVirt when doing 
discovery of existing gluster volumes.  So now I have an IP for a host, 
a short name for a host and FQDN's for the rest.



[root@mdskvm-p02 glusterfs]# gluster volume status
Status of volume: mdsgv01
Gluster process TCP Port  RDMA Port  Online  
Pid
-- 


Brick mdskvm-p02.nix.mds.xyz:/mnt/p02-d01/g
lusterv02   49152 0  Y 22368
Brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/g
lusterv01   49152 0  Y 24487
NFS Server on localhost N/A   N/A    N   
N/A

Self-heal Daemon on localhost   N/A   N/A    Y 22406
NFS Server on 192.168.0.60  N/A   N/A    N   
N/A

Self-heal Daemon on 192.168.0.60    N/A   N/A    Y 25867

Task Status of Volume mdsgv01
-- 


There are no active volume tasks

[root@mdskvm-p02 glusterfs]#

Cheers,
TK

On 9/24/2019 2:58 AM, Thorgeir Marthinussen wrote:
In an effort to answer the actual question, in my experience the 
Gluster internals captures the address the first time you probe 
another node.
So if you're logged into the first node and probe the second using an 
IP-address, that is what will "forever" be displayed by gluster 
status, and if you use a hostname that's what will be shown.
Brick paths are captured when the brick is registered, so using a path 
with IP will always show the IP as part of the path, and hostname will 
show that, etc.


I haven't verified, but the second node I believe will attempt a 
reverse lookup of the first node (when probing first->second) and 
record that name (if any) as the "primary" name of the first node.
Also good to know, nodes can have multiple names, the primary name is 
the one "configured" during setup, and secondary names can be added by 
probing them afterwards.
All IP/hostname/FQDN parts of the brick-path has to be known to the 
cluster, by probing that IP/hostname/FQDN.



Best regards
*THORGEIR MARTHINUSSEN*

-----Original Message-
*From*: TomK <mailto:tomk%20%3ctomk...@mdevsys.com%3e>>

*Reply-To*: tomk...@mdevsys.com <mailto:tomk...@mdevsys.com>
*To*: gluster-users@gluster.org <mailto:gluster-users@gluster.org>
*Subject*: Re: [Gluster-users] Where does Gluster capture the 
hostnames from?

*Date*: Mon, 23 Sep 2019 21:31:19 -0400

Hey All,


My hosts below:


[root@mdskvm-p01 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4

localhost4.localdomain4

::1 localhost localhost.localdomain localhost6

localhost6.localdomain6

[root@mdskvm-p01 ~]# hostname

mdskvm-p01.nix.mds.xyz

[root@mdskvm-p01 ~]# hostname -f

mdskvm-p01.nix.mds.xyz

[root@mdskvm-p01 ~]#



[root@mdskvm-p02 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4

localhost4.localdomain4

::1 localhost localhost.localdomain localhost6

localhost6.localdomain6

[root@mdskvm-p02 ~]# hostname

mdskvm-p02.nix.mds.xyz

[root@mdskvm-p02 ~]# hostname -f

mdskvm-p02.nix.mds.xyz

[root@mdskvm-p02 ~]#



My take on the /etc/hosts file discussion:


1) If hostname / hostname -f returns any valid values, the software

should capture it.



2) There is no benefit or need to use /etc/hosts in a small setup.

Larger setups resolving hosts against an enterprise DNS behind many

Re: [Gluster-users] gluster volume delete mdsgv01: volume delete: mdsgv01: failed: Some of the peers are down

[TomK]

Thank you.  That worked.

On 9/29/2019 2:44 AM, Sanju Rakonde wrote:

Hi Tom,

Volume delete operation is not permitted when some of peers in cluster 
are down. Please check peer status output and make sure that all the 
nodes are up and running. and then you can go for volume delete operation.


On Sun, Sep 29, 2019 at 8:53 AM TomK <mailto:tomk...@mdevsys.com>> wrote:


Hello All,

I'm not able to remove the last brick and consequently, the volume. 
How

do I go about deleting it?

[root@mdskvm-p01 ~]# gluster volume delete mdsgv01
Deleting volume will erase all information about the volume. Do you
want
to continue? (y/n) y
volume delete: mdsgv01: failed: Some of the peers are down
[root@mdskvm-p01 ~]#

[root@mdskvm-p01 ~]# gluster volume info

Volume Name: mdsgv01
Type: Distribute
Volume ID: f5b57076-dbd4-4d77-ae13-c1f3ee3adbe0
Status: Stopped
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/glusterv01
Options Reconfigured:
diagnostics.client-log-level: DEBUG
diagnostics.brick-sys-log-level: INFO
diagnostics.brick-log-level: DEBUG
performance.readdir-ahead: on
server.allow-insecure: on
nfs.trusted-sync: on
performance.cache-size: 1GB
performance.io-thread-count: 16
performance.write-behind-window-size: 8MB
client.event-threads: 8
server.event-threads: 8
cluster.quorum-type: none
cluster.server-quorum-type: none
storage.owner-uid: 36
features.shard: on
features.shard-block-size: 512MB
performance.low-prio-threads: 32
cluster.data-self-heal-algorithm: full
storage.owner-gid: 36
[root@mdskvm-p01 ~]#


[root@mdskvm-p01 ~]# gluster volume remove-brick mdsgv01
mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/glusterv01 force
Remove-brick force will not migrate files from the removed bricks, so
they will no longer be available on the volume.
Do you want to continue? (y/n) y
volume remove-brick commit force: failed: Deleting all the bricks of
the
volume is not allowed
[root@mdskvm-p01 ~]#


[root@mdskvm-p01 ~]# rpm -aq|grep -Ei gluster
glusterfs-client-xlators-6.5-1.el7.x86_64
glusterfs-geo-replication-6.5-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-23.el7_7.1.x86_64
glusterfs-events-6.5-1.el7.x86_64
python2-gluster-6.5-1.el7.x86_64
glusterfs-server-6.5-1.el7.x86_64
glusterfs-fuse-6.5-1.el7.x86_64
glusterfs-cli-6.5-1.el7.x86_64
glusterfs-6.5-1.el7.x86_64
glusterfs-libs-6.5-1.el7.x86_64
glusterfs-api-6.5-1.el7.x86_64
glusterfs-rdma-6.5-1.el7.x86_64
[root@mdskvm-p01 ~]#





-- 
Thx,

TK.


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-users



--
Thanks,
Sanju



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users




--
Thx,
TK.


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] gluster volume delete mdsgv01: volume delete: mdsgv01: failed: Some of the peers are down

[TomK]

Hello All,

I'm not able to remove the last brick and consequently, the volume.  How 
do I go about deleting it?


[root@mdskvm-p01 ~]# gluster volume delete mdsgv01
Deleting volume will erase all information about the volume. Do you want 
to continue? (y/n) y

volume delete: mdsgv01: failed: Some of the peers are down
[root@mdskvm-p01 ~]#

[root@mdskvm-p01 ~]# gluster volume info

Volume Name: mdsgv01
Type: Distribute
Volume ID: f5b57076-dbd4-4d77-ae13-c1f3ee3adbe0
Status: Stopped
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/glusterv01
Options Reconfigured:
diagnostics.client-log-level: DEBUG
diagnostics.brick-sys-log-level: INFO
diagnostics.brick-log-level: DEBUG
performance.readdir-ahead: on
server.allow-insecure: on
nfs.trusted-sync: on
performance.cache-size: 1GB
performance.io-thread-count: 16
performance.write-behind-window-size: 8MB
client.event-threads: 8
server.event-threads: 8
cluster.quorum-type: none
cluster.server-quorum-type: none
storage.owner-uid: 36
features.shard: on
features.shard-block-size: 512MB
performance.low-prio-threads: 32
cluster.data-self-heal-algorithm: full
storage.owner-gid: 36
[root@mdskvm-p01 ~]#


[root@mdskvm-p01 ~]# gluster volume remove-brick mdsgv01 
mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/glusterv01 force
Remove-brick force will not migrate files from the removed bricks, so 
they will no longer be available on the volume.

Do you want to continue? (y/n) y
volume remove-brick commit force: failed: Deleting all the bricks of the 
volume is not allowed

[root@mdskvm-p01 ~]#


[root@mdskvm-p01 ~]# rpm -aq|grep -Ei gluster
glusterfs-client-xlators-6.5-1.el7.x86_64
glusterfs-geo-replication-6.5-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-23.el7_7.1.x86_64
glusterfs-events-6.5-1.el7.x86_64
python2-gluster-6.5-1.el7.x86_64
glusterfs-server-6.5-1.el7.x86_64
glusterfs-fuse-6.5-1.el7.x86_64
glusterfs-cli-6.5-1.el7.x86_64
glusterfs-6.5-1.el7.x86_64
glusterfs-libs-6.5-1.el7.x86_64
glusterfs-api-6.5-1.el7.x86_64
glusterfs-rdma-6.5-1.el7.x86_64
[root@mdskvm-p01 ~]#





--
Thx,
TK.


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Where does Gluster capture the hostnames from?

[TomK]

Thanks Thorgeir.  Since then I upgraded to Gluster 6.  Though this issue 
remaind the same, anything in the way of new options to change what's 
displayed?


Reason for the ask is that this get's inherited by oVirt when doing 
discovery of existing gluster volumes.  So now I have an IP for a host, 
a short name for a host and FQDN's for the rest.



[root@mdskvm-p02 glusterfs]# gluster volume status
Status of volume: mdsgv01
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick mdskvm-p02.nix.mds.xyz:/mnt/p02-d01/g
lusterv02   49152 0  Y 
22368

Brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/g
lusterv01   49152 0  Y 
24487

NFS Server on localhost N/A   N/AN   N/A
Self-heal Daemon on localhost   N/A   N/AY 
22406

NFS Server on 192.168.0.60  N/A   N/AN   N/A
Self-heal Daemon on 192.168.0.60N/A   N/AY 
25867


Task Status of Volume mdsgv01
--
There are no active volume tasks

[root@mdskvm-p02 glusterfs]#

Cheers,
TK

On 9/24/2019 2:58 AM, Thorgeir Marthinussen wrote:
In an effort to answer the actual question, in my experience the Gluster 
internals captures the address the first time you probe another node.
So if you're logged into the first node and probe the second using an 
IP-address, that is what will "forever" be displayed by gluster status, 
and if you use a hostname that's what will be shown.
Brick paths are captured when the brick is registered, so using a path 
with IP will always show the IP as part of the path, and hostname will 
show that, etc.


I haven't verified, but the second node I believe will attempt a reverse 
lookup of the first node (when probing first->second) and record that 
name (if any) as the "primary" name of the first node.
Also good to know, nodes can have multiple names, the primary name is 
the one "configured" during setup, and secondary names can be added by 
probing them afterwards.
All IP/hostname/FQDN parts of the brick-path has to be known to the 
cluster, by probing that IP/hostname/FQDN.



Best regards
*THORGEIR MARTHINUSSEN*

-----Original Message-
*From*: TomK mailto:tomk%20%3ctomk...@mdevsys.com%3e>>
*Reply-To*: tomk...@mdevsys.com <mailto:tomk...@mdevsys.com>
*To*: gluster-users@gluster.org <mailto:gluster-users@gluster.org>
*Subject*: Re: [Gluster-users] Where does Gluster capture the hostnames 
from?

*Date*: Mon, 23 Sep 2019 21:31:19 -0400

Hey All,


My hosts below:


[root@mdskvm-p01 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4

localhost4.localdomain4

::1 localhost localhost.localdomain localhost6

localhost6.localdomain6

[root@mdskvm-p01 ~]# hostname

mdskvm-p01.nix.mds.xyz

[root@mdskvm-p01 ~]# hostname -f

mdskvm-p01.nix.mds.xyz

[root@mdskvm-p01 ~]#



[root@mdskvm-p02 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4

localhost4.localdomain4

::1 localhost localhost.localdomain localhost6

localhost6.localdomain6

[root@mdskvm-p02 ~]# hostname

mdskvm-p02.nix.mds.xyz

[root@mdskvm-p02 ~]# hostname -f

mdskvm-p02.nix.mds.xyz

[root@mdskvm-p02 ~]#



My take on the /etc/hosts file discussion:


1) If hostname / hostname -f returns any valid values, the software

should capture it.



2) There is no benefit or need to use /etc/hosts in a small setup.

Larger setups resolving hosts against an enterprise DNS behind many

switches could be a problem.  Managing our /etc/hosts files using

Ansible helped to reduce some of these problems esp since lookups are

logged against the connection tracking tables, that can get full,

network response time could vary etc.  ("Semi static" I guess might

describe this approach best?)  These are populated, if changes are

needed, via an initial DNS lookup once a day. Invariably, managing

/etc/hosts is time consuming and messy.



3) Running a good DNS cluster, something like a two node IPA cluster

that I run for a small setup, prevents such outages.  This particularly

when also placing a VIP across the nodes and locating cluster nodes

across different hardware and locations.



4) Point 2) should be no reason why an application cannot obtain or

resolve proper DNS entries in 1).



Having said that, decided to check if there's any benefit to having

entries in /etc/hosts:


[root@mdskvm-p01 ~]# time $(dig  mdskvm-p01.nix.mds.xyz >/dev/null)


real0m0.092s

user0m0.087s

sys 0m0.005s

[root@mdskvm-p01 ~]# time $(dig  mdskvm-p02.nix.mds.xyz >/dev/null)


real0m0.092s

user0m0.084s

sys 0m0.008s

[root@mdskvm-p01 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.loc

[Gluster-users] 0-management: Commit failed for operation Start on local node

[TomK]

Hey All,

( sending to the right group: gluster-users@gluster.org )

I'm getting the below error when trying to start a 2 node Gluster cluster.

I had the quorum enabled when I was at version 3.12 .  However with this 
version it needed the quorum disabled.  So I did so however now see the 
subject error.


Any ideas what I could try next?

--
Thx,
TK.


[2019-09-25 05:17:26.615203] D [MSGID: 0] 
[glusterd-utils.c:1136:glusterd_resolve_brick] 0-management: Returning 0
[2019-09-25 05:17:26.61] D [MSGID: 0] 
[glusterd-mgmt.c:243:gd_mgmt_v3_pre_validate_fn] 0-management: OP = 5. 
Returning 0
[2019-09-25 05:17:26.616271] D [MSGID: 0] 
[glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
mdsgv01 found
[2019-09-25 05:17:26.616305] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
[2019-09-25 05:17:26.616327] D [MSGID: 0] 
[glusterd-utils.c:6327:glusterd_brick_start] 0-management: returning 0
[2019-09-25 05:17:26.617056] I 
[glusterd-utils.c:6312:glusterd_brick_start] 0-management: starting a 
fresh brick process for brick /mnt/p01-d01/glusterv01
[2019-09-25 05:17:26.722717] E [MSGID: 106005] 
[glusterd-utils.c:6317:glusterd_brick_start] 0-management: Unable to 
start brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/glusterv01
[2019-09-25 05:17:26.722960] D [MSGID: 0] 
[glusterd-utils.c:6327:glusterd_brick_start] 0-management: returning -107
[2019-09-25 05:17:26.723006] E [MSGID: 106122] 
[glusterd-mgmt.c:341:gd_mgmt_v3_commit_fn] 0-management: Volume start 
commit failed.
[2019-09-25 05:17:26.723027] D [MSGID: 0] 
[glusterd-mgmt.c:444:gd_mgmt_v3_commit_fn] 0-management: OP = 5. 
Returning -107
[2019-09-25 05:17:26.723045] E [MSGID: 106122] 
[glusterd-mgmt.c:1696:glusterd_mgmt_v3_commit] 0-management: Commit 
failed for operation Start on local node
[2019-09-25 05:17:26.723073] D [MSGID: 0] 
[glusterd-op-sm.c:5106:glusterd_op_modify_op_ctx] 0-management: op_ctx 
modification not required
[2019-09-25 05:17:26.723141] E [MSGID: 106122] 
[glusterd-mgmt.c:2466:glusterd_mgmt_v3_initiate_all_phases] 
0-management: Commit Op Failed
[2019-09-25 05:17:26.723204] D [MSGID: 0] 
[glusterd-locks.c:797:glusterd_mgmt_v3_unlock] 0-management: Trying to 
release lock of vol mdsgv01 for f7336db6-22b4-497d-8c2f-04c833a28546 as 
mdsgv01_vol
[2019-09-25 05:17:26.723239] D [MSGID: 0] 
[glusterd-locks.c:846:glusterd_mgmt_v3_unlock] 0-management: Lock for 
vol mdsgv01 successfully released
[2019-09-25 05:17:26.723273] D [MSGID: 0] 
[glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
mdsgv01 found
[2019-09-25 05:17:26.723326] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
[2019-09-25 05:17:26.723360] D [MSGID: 0] 
[glusterd-locks.c:464:glusterd_multiple_mgmt_v3_unlock] 0-management: 
Returning 0


==> /var/log/glusterfs/cmd_history.log <==
[2019-09-25 05:17:26.723390]  : volume start mdsgv01 : FAILED : Commit 
failed on localhost. Please check log file for details.


==> /var/log/glusterfs/glusterd.log <==
[2019-09-25 05:17:26.723479] D [MSGID: 0] 
[glusterd-rpc-ops.c:199:glusterd_op_send_cli_response] 0-management: 
Returning 0




[root@mdskvm-p01 glusterfs]# cat /etc/glusterfs/glusterd.vol
volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket,rdma
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option ping-timeout 0
option event-threads 1
option rpc-auth-allow-insecure on
# option cluster.server-quorum-type server
# option cluster.quorum-type auto
option server.event-threads 8
option client.event-threads 8
option performance.write-behind-window-size 8MB
option performance.io-thread-count 16
option performance.cache-size 1GB
option nfs.trusted-sync on
option storage.owner-uid 36
option storage.owner-uid 36
option cluster.data-self-heal-algorithm full
option performance.low-prio-threads 32
option features.shard-block-size 512MB
option features.shard on
end-volume
[root@mdskvm-p01 glusterfs]#


[root@mdskvm-p01 glusterfs]# gluster volume info

Volume Name: mdsgv01
Type: Replicate
Volume ID: f5b57076-dbd4-4d77-ae13-c1f3ee3adbe0
Status: Stopped
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: mdskvm-p02.nix.mds.xyz:/mnt/p02-d01/glusterv02
Brick2: mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/glusterv01
Options Reconfigured:
storage.owner-gid: 36
cluster.data-self-heal-algorithm: full
performance.low-prio-threads: 32
features.shard-block-size: 512MB
features.shard: on
storage.owner-uid: 36
cluster.server-quorum-type: none
cluster.quorum-type: none
server.event-threads: 8
client.event-threads: 8
performance.write-behind-window-size: 8MB
performance.io-thread-count: 16
performance.cache-size: 1GB
nfs.trusted-sync: on
server.allow-insecure: on
performance.readdir-ahead: on

Re: [Gluster-users] Where does Gluster capture the hostnames from?

[TomK]

Hey All,

My hosts below:

[root@mdskvm-p01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 
localhost6.localdomain6

[root@mdskvm-p01 ~]# hostname
mdskvm-p01.nix.mds.xyz
[root@mdskvm-p01 ~]# hostname -f
mdskvm-p01.nix.mds.xyz
[root@mdskvm-p01 ~]#


[root@mdskvm-p02 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 
localhost6.localdomain6

[root@mdskvm-p02 ~]# hostname
mdskvm-p02.nix.mds.xyz
[root@mdskvm-p02 ~]# hostname -f
mdskvm-p02.nix.mds.xyz
[root@mdskvm-p02 ~]#


My take on the /etc/hosts file discussion:

1) If hostname / hostname -f returns any valid values, the software 
should capture it.



2) There is no benefit or need to use /etc/hosts in a small setup. 
Larger setups resolving hosts against an enterprise DNS behind many 
switches could be a problem.  Managing our /etc/hosts files using 
Ansible helped to reduce some of these problems esp since lookups are 
logged against the connection tracking tables, that can get full, 
network response time could vary etc.  ("Semi static" I guess might 
describe this approach best?)  These are populated, if changes are 
needed, via an initial DNS lookup once a day. Invariably, managing 
/etc/hosts is time consuming and messy.



3) Running a good DNS cluster, something like a two node IPA cluster 
that I run for a small setup, prevents such outages.  This particularly 
when also placing a VIP across the nodes and locating cluster nodes 
across different hardware and locations.



4) Point 2) should be no reason why an application cannot obtain or 
resolve proper DNS entries in 1).



Having said that, decided to check if there's any benefit to having 
entries in /etc/hosts:


[root@mdskvm-p01 ~]# time $(dig  mdskvm-p01.nix.mds.xyz >/dev/null)

real0m0.092s
user0m0.087s
sys 0m0.005s
[root@mdskvm-p01 ~]# time $(dig  mdskvm-p02.nix.mds.xyz >/dev/null)

real0m0.092s
user0m0.084s
sys 0m0.008s
[root@mdskvm-p01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 
localhost6.localdomain6

192.168.0.60mdskvm-p01.nix.mds.xyz  mdskvm-p01
192.168.0.39mdskvm-p02.nix.mds.xyz  mdskvm-p02
[root@mdskvm-p01 ~]# vi /etc/hosts
[root@mdskvm-p01 ~]# time $(dig  mdskvm-p01.nix.mds.xyz >/dev/null)

real0m0.093s
user0m0.082s
sys 0m0.010s
[root@mdskvm-p01 ~]# time $(dig  mdskvm-p02.nix.mds.xyz >/dev/null)

real0m0.093s
user0m0.085s
sys 0m0.007s
[root@mdskvm-p01 ~]# time $(dig  mdskvm-p01.nix.mds.xyz >/dev/null)

real0m0.094s
user0m0.084s
sys 0m0.010s
[root@mdskvm-p01 ~]# time $(dig  mdskvm-p02.nix.mds.xyz >/dev/null)

real0m0.092s
user0m0.081s
sys 0m0.011s
[root@mdskvm-p01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 
localhost6.localdomain6

[root@mdskvm-p01 ~]#


So with /etc/hosts file entries present makes little difference in small 
setup when governed by /etc/nsswitch.conf .


Having entries in /etc/hosts, doesn't affect how gluster displays the 
entries when calling gluster volume status .



Cheers,
TK

On 9/23/2019 11:36 AM, Joe Julian wrote:
Perhaps I misread the intent, I apologize if I did. I read "static 
entries" as "ip addresses" which I've seen suggested (from my 
perspective) far too often. /etc/hosts is a valid solution that can 
still adapt if the network needs to evolve.


On 9/23/19 8:29 AM, ROUVRAIS Cedric wrote:

Hello,

I guess everyone sort of has his perspective on this topic.

I don't want to take this thread on an off-topic conversation 
(discussing the merits of having a local hosts file) but I do dissent, 
and therefore had to respond, on the shortcut that using a local 
etc/host file creates a fixed network configuration that can never 
adapt as business needs change. I'm running a k8s infrastructure and 
actually have local conf files, FWIW.


Regards,

Cédric


-Original Message-
From: gluster-users-boun...@gluster.org 
 On Behalf Of Joe Julian

Sent: lundi 23 septembre 2019 17:06
To: gluster-users@gluster.org
Subject: Re: [Gluster-users] Where does Gluster capture the hostnames 
from?


I disagree about it being "best practice" to lock yourself in to a 
fixed network configuration that can never adapt as business needs 
change.
There are other resilient ways of ensuring your hostnames resolve 
consistently (so that your cluster doesn't run loose ;-)).


On 9/23/19 7:38 AM, Strahil wrote:

Also,

It's more safe to have static entries for your cluster - after all if 
DNS fails for some reason - you don't want to loose  your cluster.A 
kind of 'Best Practice'.


Best Regards,
Strahil Niko

Re: [Gluster-users] Where does Gluster capture the hostnames from?

[TomK]

Do I *really* need specific /etc/hosts entries when I have IPA?

[root@mdskvm-p01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 
localhost6.localdomain6

[root@mdskvm-p01 ~]#

I really shouldn't need too.  ( Ref below, everything resolves fine. )

Cheers,
TK


On 9/23/2019 1:32 AM, Strahil wrote:

Check your /etc/hosts for an entry like:
192.168.0.60 mdskvm-p01.nix.mds.xyz mdskvm-p01

Best Regards,
Strahil NikolovOn Sep 23, 2019 06:58, TomK  wrote:


Hey All,

Take the two hosts below as example.  One host shows NFS Server on
192.168.0.60 (FQDN is mdskvm-p01.nix.mds.xyz).

The other shows mdskvm-p02 (FQDN is mdskvm-p02.nix.mds.xyz).

Why is there no consistency or correct hostname resolution?  Where does
gluster get the hostnames from?


[root@mdskvm-p02 glusterfs]# gluster volume status
Status of volume: mdsgv01
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick mdskvm-p02.nix.mds.xyz:/mnt/p02-d01/g
lusterv02   49153 0  Y
17503
Brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/g
lusterv01   49153 0  Y
15044
NFS Server on localhost N/A   N/A    N   N/A
Self-heal Daemon on localhost   N/A   N/A    Y
17531
NFS Server on 192.168.0.60  N/A   N/A    N   N/A
Self-heal Daemon on 192.168.0.60    N/A   N/A    Y
15073

Task Status of Volume mdsgv01
--
There are no active volume tasks

[root@mdskvm-p02 glusterfs]#




[root@mdskvm-p01 ~]# gluster volume status
Status of volume: mdsgv01
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick mdskvm-p02.nix.mds.xyz:/mnt/p02-d01/g
lusterv02   49153 0  Y
17503
Brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/g
lusterv01   49153 0  Y
15044
NFS Server on localhost N/A   N/A    N   N/A
Self-heal Daemon on localhost   N/A   N/A    Y
15073
NFS Server on mdskvm-p02    N/A   N/A    N   N/A
Self-heal Daemon on mdskvm-p02  N/A   N/A    Y
17531

Task Status of Volume mdsgv01
--
There are no active volume tasks

[root@mdskvm-p01 ~]#



But when verifying everything all seems fine:


(1):
[root@mdskvm-p01 glusterfs]# dig -x 192.168.0.39
;; QUESTION SECTION:
;39.0.168.192.in-addr.arpa. IN  PTR

;; ANSWER SECTION:
39.0.168.192.in-addr.arpa. 1200 IN  PTR mdskvm-p02.nix.mds.xyz.
[root@mdskvm-p01 glusterfs]# hostname -f
mdskvm-p01.nix.mds.xyz
[root@mdskvm-p01 glusterfs]# hostname -s
mdskvm-p01
[root@mdskvm-p01 glusterfs]# hostname
mdskvm-p01.nix.mds.xyz
[root@mdskvm-p01 glusterfs]#


(2):

[root@mdskvm-p02 glusterfs]# dig -x 192.168.0.60
;; QUESTION SECTION:
;60.0.168.192.in-addr.arpa. IN  PTR

;; ANSWER SECTION:
60.0.168.192.in-addr.arpa. 1200 IN  PTR mdskvm-p01.nix.mds.xyz.

[root@mdskvm-p02 glusterfs]# hostname -s
mdskvm-p02
[root@mdskvm-p02 glusterfs]# hostname -f
mdskvm-p02.nix.mds.xyz
[root@mdskvm-p02 glusterfs]# hostname
mdskvm-p02.nix.mds.xyz
[root@mdskvm-p02 glusterfs]#


Gluster version used is:

[root@mdskvm-p01 glusterfs]# rpm -aq|grep -Ei gluster
glusterfs-server-3.12.15-1.el7.x86_64
glusterfs-client-xlators-3.12.15-1.el7.x86_64
glusterfs-rdma-3.12.15-1.el7.x86_64
glusterfs-3.12.15-1.el7.x86_64
glusterfs-events-3.12.15-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-10.el7_6.12.x86_64
glusterfs-libs-3.12.15-1.el7.x86_64
glusterfs-fuse-3.12.15-1.el7.x86_64
glusterfs-geo-replication-3.12.15-1.el7.x86_64
python2-gluster-3.12.15-1.el7.x86_64
glusterfs-cli-3.12.15-1.el7.x86_64
vdsm-gluster-4.20.46-1.el7.x86_64
glusterfs-api-3.12.15-1.el7.x86_64
glusterfs-gnfs-3.12.15-1.el7.x86_64
[root@mdskvm-p01 glusterfs]#


--
Thx,
TK.


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users




--
Thx

[Gluster-users] Where does Gluster capture the hostnames from?

[TomK]

Hey All,

Take the two hosts below as example.  One host shows NFS Server on 
192.168.0.60 (FQDN is mdskvm-p01.nix.mds.xyz).


The other shows mdskvm-p02 (FQDN is mdskvm-p02.nix.mds.xyz).

Why is there no consistency or correct hostname resolution?  Where does 
gluster get the hostnames from?



[root@mdskvm-p02 glusterfs]# gluster volume status
Status of volume: mdsgv01
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick mdskvm-p02.nix.mds.xyz:/mnt/p02-d01/g
lusterv02   49153 0  Y 
17503

Brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/g
lusterv01   49153 0  Y 
15044

NFS Server on localhost N/A   N/AN   N/A
Self-heal Daemon on localhost   N/A   N/AY 
17531

NFS Server on 192.168.0.60  N/A   N/AN   N/A
Self-heal Daemon on 192.168.0.60N/A   N/AY 
15073


Task Status of Volume mdsgv01
--
There are no active volume tasks

[root@mdskvm-p02 glusterfs]#




[root@mdskvm-p01 ~]# gluster volume status
Status of volume: mdsgv01
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick mdskvm-p02.nix.mds.xyz:/mnt/p02-d01/g
lusterv02   49153 0  Y 
17503

Brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/g
lusterv01   49153 0  Y 
15044

NFS Server on localhost N/A   N/AN   N/A
Self-heal Daemon on localhost   N/A   N/AY 
15073

NFS Server on mdskvm-p02N/A   N/AN   N/A
Self-heal Daemon on mdskvm-p02  N/A   N/AY 
17531


Task Status of Volume mdsgv01
--
There are no active volume tasks

[root@mdskvm-p01 ~]#



But when verifying everything all seems fine:


(1):
[root@mdskvm-p01 glusterfs]# dig -x 192.168.0.39
;; QUESTION SECTION:
;39.0.168.192.in-addr.arpa. IN  PTR

;; ANSWER SECTION:
39.0.168.192.in-addr.arpa. 1200 IN  PTR mdskvm-p02.nix.mds.xyz.
[root@mdskvm-p01 glusterfs]# hostname -f
mdskvm-p01.nix.mds.xyz
[root@mdskvm-p01 glusterfs]# hostname -s
mdskvm-p01
[root@mdskvm-p01 glusterfs]# hostname
mdskvm-p01.nix.mds.xyz
[root@mdskvm-p01 glusterfs]#


(2):

[root@mdskvm-p02 glusterfs]# dig -x 192.168.0.60
;; QUESTION SECTION:
;60.0.168.192.in-addr.arpa. IN  PTR

;; ANSWER SECTION:
60.0.168.192.in-addr.arpa. 1200 IN  PTR mdskvm-p01.nix.mds.xyz.

[root@mdskvm-p02 glusterfs]# hostname -s
mdskvm-p02
[root@mdskvm-p02 glusterfs]# hostname -f
mdskvm-p02.nix.mds.xyz
[root@mdskvm-p02 glusterfs]# hostname
mdskvm-p02.nix.mds.xyz
[root@mdskvm-p02 glusterfs]#


Gluster version used is:

[root@mdskvm-p01 glusterfs]# rpm -aq|grep -Ei gluster
glusterfs-server-3.12.15-1.el7.x86_64
glusterfs-client-xlators-3.12.15-1.el7.x86_64
glusterfs-rdma-3.12.15-1.el7.x86_64
glusterfs-3.12.15-1.el7.x86_64
glusterfs-events-3.12.15-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-10.el7_6.12.x86_64
glusterfs-libs-3.12.15-1.el7.x86_64
glusterfs-fuse-3.12.15-1.el7.x86_64
glusterfs-geo-replication-3.12.15-1.el7.x86_64
python2-gluster-3.12.15-1.el7.x86_64
glusterfs-cli-3.12.15-1.el7.x86_64
vdsm-gluster-4.20.46-1.el7.x86_64
glusterfs-api-3.12.15-1.el7.x86_64
glusterfs-gnfs-3.12.15-1.el7.x86_64
[root@mdskvm-p01 glusterfs]#


--
Thx,
TK.


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] [SOLVED] [Nfs-ganesha-support] volume start: gv01: failed: Quorum not met. Volume operation not allowed.

[TomK]

 21 23:53:13 psql01 kernel: [] 
nfs_wait_client_init_complete+0xa1/0xe0 [nfs]
May 21 23:53:13 psql01 kernel: [] ? 
wake_up_atomic_t+0x30/0x30
May 21 23:53:13 psql01 kernel: [] 
nfs_get_client+0x22b/0x470 [nfs]
May 21 23:53:13 psql01 kernel: [] 
nfs4_set_client+0x98/0x130 [nfsv4]
May 21 23:53:13 psql01 kernel: [] 
nfs4_create_server+0x13e/0x3b0 [nfsv4]
May 21 23:53:13 psql01 kernel: [] 
nfs4_remote_mount+0x2e/0x60 [nfsv4]

May 21 23:53:13 psql01 kernel: [] mount_fs+0x3e/0x1b0
May 21 23:53:13 psql01 kernel: [] ? 
__alloc_percpu+0x15/0x20
May 21 23:53:13 psql01 kernel: [] 
vfs_kern_mount+0x67/0x110
May 21 23:53:13 psql01 kernel: [] 
nfs_do_root_mount+0x86/0xc0 [nfsv4]
May 21 23:53:13 psql01 kernel: [] 
nfs4_try_mount+0x44/0xc0 [nfsv4]
May 21 23:53:13 psql01 kernel: [] ? 
get_nfs_version+0x27/0x90 [nfs]
May 21 23:53:13 psql01 kernel: [] 
nfs_fs_mount+0x4cb/0xda0 [nfs]
May 21 23:53:13 psql01 kernel: [] ? 
nfs_clone_super+0x140/0x140 [nfs]
May 21 23:53:13 psql01 kernel: [] ? 
param_set_portnr+0x70/0x70 [nfs]

May 21 23:53:13 psql01 kernel: [] mount_fs+0x3e/0x1b0
May 21 23:53:13 psql01 kernel: [] ? 
__alloc_percpu+0x15/0x20
May 21 23:53:13 psql01 kernel: [] 
vfs_kern_mount+0x67/0x110

May 21 23:53:13 psql01 kernel: [] do_mount+0x233/0xaf0
May 21 23:53:13 psql01 kernel: [] SyS_mount+0x96/0xf0
May 21 23:53:13 psql01 kernel: [] 
system_call_fastpath+0x1c/0x21
May 21 23:53:13 psql01 kernel: [] ? 
system_call_after_swapgs+0xae/0x146





On 5/7/2018 10:28 PM, TomK wrote:
This list has been deprecated. Please subscribe to the new support list 
at lists.nfs-ganesha.org.

On 4/11/2018 11:54 AM, Alex K wrote:

Hey Guy's,

Returning to this topic, after disabling the the quorum:

cluster.quorum-type: none
cluster.server-quorum-type: none

I've ran into a number of gluster errors (see below).

I'm using gluster as the backend for my NFS storage.  I have gluster 
running on two nodes, nfs01 and nfs02.  It's mounted on /n on each host. 
  The path /n is in turn shared out by NFS Ganesha.  It's a two node 
setup with quorum disabled as noted below:


[root@nfs02 ganesha]# mount|grep gv01
nfs02:/gv01 on /n type fuse.glusterfs 
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) 



[root@nfs01 glusterfs]# mount|grep gv01
nfs01:/gv01 on /n type fuse.glusterfs 
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) 



Gluster always reports as working no matter when I type the below two 
commands:


[root@nfs01 glusterfs]# gluster volume info

Volume Name: gv01
Type: Replicate
Volume ID: e5ccc75e-5192-45ac-b410-a34ebd777666
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: nfs01:/bricks/0/gv01
Brick2: nfs02:/bricks/0/gv01
Options Reconfigured:
cluster.server-quorum-type: none
cluster.quorum-type: none
server.event-threads: 8
client.event-threads: 8
performance.readdir-ahead: on
performance.write-behind-window-size: 8MB
performance.io-thread-count: 16
performance.cache-size: 1GB
nfs.trusted-sync: on
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
[root@nfs01 glusterfs]# gluster status
unrecognized word: status (position 0)
[root@nfs01 glusterfs]# gluster volume status
Status of volume: gv01
Gluster process TCP Port  RDMA Port  Online  
Pid
-- 


Brick nfs01:/bricks/0/gv01  49152 0  Y 1422
Brick nfs02:/bricks/0/gv01  49152 0  Y 1422
Self-heal Daemon on localhost   N/A   N/A    Y 1248
Self-heal Daemon on nfs02.nix.my.dom   N/A   N/A    Y   
1251


Task Status of Volume gv01
-- 


There are no active volume tasks

[root@nfs01 glusterfs]#

[root@nfs01 glusterfs]# rpm -aq|grep -Ei gluster
glusterfs-3.13.2-2.el7.x86_64
glusterfs-devel-3.13.2-2.el7.x86_64
glusterfs-fuse-3.13.2-2.el7.x86_64
glusterfs-api-devel-3.13.2-2.el7.x86_64
centos-release-gluster313-1.0-1.el7.centos.noarch
python2-gluster-3.13.2-2.el7.x86_64
glusterfs-client-xlators-3.13.2-2.el7.x86_64
glusterfs-server-3.13.2-2.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7_4.9.x86_64
glusterfs-cli-3.13.2-2.el7.x86_64
centos-release-gluster312-1.0-1.el7.centos.noarch
python2-glusterfs-api-1.1-1.el7.noarch
glusterfs-libs-3.13.2-2.el7.x86_64
glusterfs-extra-xlators-3.13.2-2.el7.x86_64
glusterfs-api-3.13.2-2.el7.x86_64
[root@nfs01 glusterfs]#

The short of it is that everything works and mounts on guests work as 
long as I don't try to write to the NFS share from my clients.  If I try 
to write to the share, everything comes apart like this:


-sh-4.2$ pwd
/n/my.dom/tom
-sh-4.2$ ls -altri
total 6258
11715278280495367299 -rw---. 1 t...@my.dom t...@my.dom 231 Feb 17 
20:15 .bashrc
10937819299152577443 -rw---. 1 t...@my.dom t...@my.dom 193 Feb 17 

[Gluster-users] test

[TomK]

test

--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] volume start: gv01: failed: Quorum not met. Volume operation not allowed.

[TomK]

b64/glusterfs/3.13.2/xlator/protocol/client.so(fini+0x28)[0x7f32701a7c88] 
) 0-gv01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-05-05 01:38:46.965204 (xid=0x2)
ganesha-gfapi.log:[2018-05-05 01:38:50.457456] E 
[rpc-clnt.c:417:rpc_clnt_reconnect] 0-gv01-client-1: Error adding to 
timer event queue
ganesha-gfapi.log:[2018-05-07 03:05:58.522295] E 
[rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f45af6d6f0b] (--> 
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f45af49be7e] (--> 
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f45af49bf9e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7f45af49d720] 
(--> 
/usr/lib64/glusterfs/3.13.2/xlator/protocol/client.so(fini+0x28)[0x7f45a0db6c88] 
) 0-gv01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-05-07 03:05:56.080210 (xid=0x2)
ganesha-gfapi.log:[2018-05-07 03:05:59.504926] E 
[rpc-clnt.c:417:rpc_clnt_reconnect] 0-gv01-client-1: Error adding to 
timer event queue
ganesha-gfapi.log:[2018-05-07 03:05:59.505274] E 
[rpc-clnt.c:417:rpc_clnt_reconnect] 0-gv01-client-0: Error adding to 
timer event queue

[root@nfs02 ganesha]#
[root@nfs02 ganesha]#







On Wed, Apr 11, 2018 at 4:35 AM, TomK <tomk...@mdevsys.com 
<mailto:tomk...@mdevsys.com>> wrote:


On 4/9/2018 2:45 AM, Alex K wrote:
Hey Alex,

With two nodes, the setup works but both sides go down when one node
is missing.  Still I set the below two params to none and that
solved my issue:

cluster.quorum-type: none
cluster.server-quorum-type: none

yes this disables quorum so as to avoid the issue. Glad that this 
helped. Bare in in mind though that it is easier to face split-brain 
issues with quorum is disabled, that's why 3 nodes at least are 
recommended. Just to note that I have also a 2 node cluster which is 
running without issues for long time.


Thank you for that.

Cheers,
Tom

Hi,

You need 3 nodes at least to have quorum enabled. In 2 node
setup you need to disable quorum so as to be able to still use
the volume when one of the nodes go down.

On Mon, Apr 9, 2018, 09:02 TomK <tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com> <mailto:tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com>>> wrote:

     Hey All,

     In a two node glusterfs setup, with one node down, can't
use the second
     node to mount the volume.  I understand this is expected
behaviour?
     Anyway to allow the secondary node to function then
replicate what
     changed to the first (primary) when it's back online?  Or
should I just
     go for a third node to allow for this?

     Also, how safe is it to set the following to none?

     cluster.quorum-type: auto
     cluster.server-quorum-type: server


     [root@nfs01 /]# gluster volume start gv01
     volume start: gv01: failed: Quorum not met. Volume
operation not
     allowed.
     [root@nfs01 /]#


     [root@nfs01 /]# gluster volume status
     Status of volume: gv01
     Gluster process                             TCP Port  RDMA
Port     Online  Pid

--
     Brick nfs01:/bricks/0/gv01                  N/A       N/A 
       N           N/A
     Self-heal Daemon on localhost               N/A       N/A 
       Y

     25561

     Task Status of Volume gv01

--

     There are no active volume tasks

     [root@nfs01 /]#


     [root@nfs01 /]# gluster volume info

     Volume Name: gv01
     Type: Replicate
     Volume ID: e5ccc75e-5192-45ac-b410-a34ebd777666
     Status: Started
     Snapshot Count: 0
     Number of Bricks: 1 x 2 = 2
     Transport-type: tcp
     Bricks:
     Brick1: nfs01:/bricks/0/gv01
     Brick2: nfs02:/bricks/0/gv01
     Options Reconfigured:
     transport.address-family: inet
     nfs.disable: on
     performance.client-io-threads: off
     nfs.trusted-sync: on
     performance.cache-size: 1GB
     performance.io-thread-count: 16
     performance.write-behind-window-size: 8MB
     performance.readdir-ahead: on
     client.event-threads: 8
     server.event-threads: 8
     cluster.quorum-type: auto
     cluster.server-quorum-type: server
     [root@nfs01 /]#




     ==> n.log <==
     [2018-04-09 05:08:13.704156] I [MSGID: 100030]
 

Re: [Gluster-users] volume start: gv01: failed: Quorum not met. Volume operation not allowed.

[TomK]

; 
/usr/lib64/glusterfs/3.13.2/xlator/protocol/client.so(fini+0x28)[0x7f32701a7c88] 
) 0-gv01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-05-05 01:38:46.965204 (xid=0x2)
ganesha-gfapi.log:[2018-05-05 01:38:50.457456] E 
[rpc-clnt.c:417:rpc_clnt_reconnect] 0-gv01-client-1: Error adding to 
timer event queue
ganesha-gfapi.log:[2018-05-07 03:05:58.522295] E 
[rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f45af6d6f0b] (--> 
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f45af49be7e] (--> 
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f45af49bf9e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7f45af49d720] 
(--> 
/usr/lib64/glusterfs/3.13.2/xlator/protocol/client.so(fini+0x28)[0x7f45a0db6c88] 
) 0-gv01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-05-07 03:05:56.080210 (xid=0x2)
ganesha-gfapi.log:[2018-05-07 03:05:59.504926] E 
[rpc-clnt.c:417:rpc_clnt_reconnect] 0-gv01-client-1: Error adding to 
timer event queue
ganesha-gfapi.log:[2018-05-07 03:05:59.505274] E 
[rpc-clnt.c:417:rpc_clnt_reconnect] 0-gv01-client-0: Error adding to 
timer event queue

[root@nfs02 ganesha]#
[root@nfs02 ganesha]#







On Wed, Apr 11, 2018 at 4:35 AM, TomK <tomk...@mdevsys.com 
<mailto:tomk...@mdevsys.com>> wrote:


On 4/9/2018 2:45 AM, Alex K wrote:
Hey Alex,

With two nodes, the setup works but both sides go down when one node
is missing.  Still I set the below two params to none and that
solved my issue:

cluster.quorum-type: none
cluster.server-quorum-type: none

yes this disables quorum so as to avoid the issue. Glad that this 
helped. Bare in in mind though that it is easier to face split-brain 
issues with quorum is disabled, that's why 3 nodes at least are 
recommended. Just to note that I have also a 2 node cluster which is 
running without issues for long time.


Thank you for that.

Cheers,
Tom

Hi,

You need 3 nodes at least to have quorum enabled. In 2 node
setup you need to disable quorum so as to be able to still use
the volume when one of the nodes go down.

On Mon, Apr 9, 2018, 09:02 TomK <tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com> <mailto:tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com>>> wrote:

     Hey All,

     In a two node glusterfs setup, with one node down, can't
use the second
     node to mount the volume.  I understand this is expected
behaviour?
     Anyway to allow the secondary node to function then
replicate what
     changed to the first (primary) when it's back online?  Or
should I just
     go for a third node to allow for this?

     Also, how safe is it to set the following to none?

     cluster.quorum-type: auto
     cluster.server-quorum-type: server


     [root@nfs01 /]# gluster volume start gv01
     volume start: gv01: failed: Quorum not met. Volume
operation not
     allowed.
     [root@nfs01 /]#


     [root@nfs01 /]# gluster volume status
     Status of volume: gv01
     Gluster process                             TCP Port  RDMA
Port     Online  Pid

--
     Brick nfs01:/bricks/0/gv01                  N/A       N/A 
       N           N/A
     Self-heal Daemon on localhost               N/A       N/A 
       Y

     25561

     Task Status of Volume gv01

--

     There are no active volume tasks

     [root@nfs01 /]#


     [root@nfs01 /]# gluster volume info

     Volume Name: gv01
     Type: Replicate
     Volume ID: e5ccc75e-5192-45ac-b410-a34ebd777666
     Status: Started
     Snapshot Count: 0
     Number of Bricks: 1 x 2 = 2
     Transport-type: tcp
     Bricks:
     Brick1: nfs01:/bricks/0/gv01
     Brick2: nfs02:/bricks/0/gv01
     Options Reconfigured:
     transport.address-family: inet
     nfs.disable: on
     performance.client-io-threads: off
     nfs.trusted-sync: on
     performance.cache-size: 1GB
     performance.io-thread-count: 16
     performance.write-behind-window-size: 8MB
     performance.readdir-ahead: on
     client.event-threads: 8
     server.event-threads: 8
     cluster.quorum-type: auto
     cluster.server-quorum-type: server
     [root@nfs01 /]#




     ==> n.log <==
     [2018-04-09 05:08:13.704156] I [MSGID: 100030]
 

Re: [Gluster-users] volume start: gv01: failed: Quorum not met. Volume operation not allowed.

[TomK]

On 4/9/2018 2:45 AM, Alex K wrote:
Hey Alex,

With two nodes, the setup works but both sides go down when one node is 
missing.  Still I set the below two params to none and that solved my issue:


cluster.quorum-type: none
cluster.server-quorum-type: none

Thank you for that.

Cheers,
Tom


Hi,

You need 3 nodes at least to have quorum enabled. In 2 node setup you 
need to disable quorum so as to be able to still use the volume when one 
of the nodes go down.


On Mon, Apr 9, 2018, 09:02 TomK <tomk...@mdevsys.com 
<mailto:tomk...@mdevsys.com>> wrote:


Hey All,

In a two node glusterfs setup, with one node down, can't use the second
node to mount the volume.  I understand this is expected behaviour?
Anyway to allow the secondary node to function then replicate what
changed to the first (primary) when it's back online?  Or should I just
go for a third node to allow for this?

Also, how safe is it to set the following to none?

cluster.quorum-type: auto
cluster.server-quorum-type: server


[root@nfs01 /]# gluster volume start gv01
volume start: gv01: failed: Quorum not met. Volume operation not
allowed.
[root@nfs01 /]#


[root@nfs01 /]# gluster volume status
Status of volume: gv01
Gluster process                             TCP Port  RDMA Port 
Online  Pid


--
Brick nfs01:/bricks/0/gv01                  N/A       N/A        N 
      N/A

Self-heal Daemon on localhost               N/A       N/A        Y
25561

Task Status of Volume gv01

--
There are no active volume tasks

[root@nfs01 /]#


[root@nfs01 /]# gluster volume info

Volume Name: gv01
Type: Replicate
Volume ID: e5ccc75e-5192-45ac-b410-a34ebd777666
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: nfs01:/bricks/0/gv01
Brick2: nfs02:/bricks/0/gv01
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
nfs.trusted-sync: on
performance.cache-size: 1GB
performance.io-thread-count: 16
performance.write-behind-window-size: 8MB
performance.readdir-ahead: on
client.event-threads: 8
server.event-threads: 8
cluster.quorum-type: auto
cluster.server-quorum-type: server
[root@nfs01 /]#




==> n.log <==
[2018-04-09 05:08:13.704156] I [MSGID: 100030] [glusterfsd.c:2556:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version
3.13.2 (args: /usr/sbin/glusterfs --process-name fuse
--volfile-server=nfs01 --volfile-id=/gv01 /n)
[2018-04-09 05:08:13.711255] W [MSGID: 101002]
[options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is
deprecated, preferred is 'transport.address-family', continuing with
correction
[2018-04-09 05:08:13.728297] W [socket.c:3216:socket_connect]
0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Protocol not
available"
[2018-04-09 05:08:13.729025] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2018-04-09 05:08:13.737757] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 2
[2018-04-09 05:08:13.738114] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 3
[2018-04-09 05:08:13.738203] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 4
[2018-04-09 05:08:13.738324] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 5
[2018-04-09 05:08:13.738330] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 6
[2018-04-09 05:08:13.738655] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 7
[2018-04-09 05:08:13.738742] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 8
[2018-04-09 05:08:13.739460] W [MSGID: 101174]
[graph.c:363:_log_if_unknown_option] 0-gv01-readdir-ahead: option
'parallel-readdir' is not recognized
[2018-04-09 05:08:13.739787] I [MSGID: 114020] [client.c:2360:notify]
0-gv01-client-0: parent translators are ready, attempting connect on
transport
[2018-04-09 05:08:13.747040] W [socket.c:3216:socket_connect]
0-gv01-client-0: Error disabling sockopt IPV6_V6ONLY: "Protocol not
available"
[2018-04-09 05:08:13.747372] I [MSGID: 114020] [client.c:2360:notify]
0-gv01-client-1: parent translators are ready, attempting connect on
transport
   

[Gluster-users] volume start: gv01: failed: Quorum not met. Volume operation not allowed.

[TomK]

Hey All,

In a two node glusterfs setup, with one node down, can't use the second 
node to mount the volume.  I understand this is expected behaviour? 
Anyway to allow the secondary node to function then replicate what 
changed to the first (primary) when it's back online?  Or should I just 
go for a third node to allow for this?


Also, how safe is it to set the following to none?

cluster.quorum-type: auto
cluster.server-quorum-type: server


[root@nfs01 /]# gluster volume start gv01
volume start: gv01: failed: Quorum not met. Volume operation not allowed.
[root@nfs01 /]#


[root@nfs01 /]# gluster volume status
Status of volume: gv01
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick nfs01:/bricks/0/gv01  N/A   N/AN   N/A
Self-heal Daemon on localhost   N/A   N/AY 
25561


Task Status of Volume gv01
--
There are no active volume tasks

[root@nfs01 /]#


[root@nfs01 /]# gluster volume info

Volume Name: gv01
Type: Replicate
Volume ID: e5ccc75e-5192-45ac-b410-a34ebd777666
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: nfs01:/bricks/0/gv01
Brick2: nfs02:/bricks/0/gv01
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
nfs.trusted-sync: on
performance.cache-size: 1GB
performance.io-thread-count: 16
performance.write-behind-window-size: 8MB
performance.readdir-ahead: on
client.event-threads: 8
server.event-threads: 8
cluster.quorum-type: auto
cluster.server-quorum-type: server
[root@nfs01 /]#




==> n.log <==
[2018-04-09 05:08:13.704156] I [MSGID: 100030] [glusterfsd.c:2556:main] 
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 
3.13.2 (args: /usr/sbin/glusterfs --process-name fuse 
--volfile-server=nfs01 --volfile-id=/gv01 /n)
[2018-04-09 05:08:13.711255] W [MSGID: 101002] 
[options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is 
deprecated, preferred is 'transport.address-family', continuing with 
correction
[2018-04-09 05:08:13.728297] W [socket.c:3216:socket_connect] 
0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Protocol not available"
[2018-04-09 05:08:13.729025] I [MSGID: 101190] 
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 1
[2018-04-09 05:08:13.737757] I [MSGID: 101190] 
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 2
[2018-04-09 05:08:13.738114] I [MSGID: 101190] 
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 3
[2018-04-09 05:08:13.738203] I [MSGID: 101190] 
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 4
[2018-04-09 05:08:13.738324] I [MSGID: 101190] 
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 5
[2018-04-09 05:08:13.738330] I [MSGID: 101190] 
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 6
[2018-04-09 05:08:13.738655] I [MSGID: 101190] 
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 7
[2018-04-09 05:08:13.738742] I [MSGID: 101190] 
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 8
[2018-04-09 05:08:13.739460] W [MSGID: 101174] 
[graph.c:363:_log_if_unknown_option] 0-gv01-readdir-ahead: option 
'parallel-readdir' is not recognized
[2018-04-09 05:08:13.739787] I [MSGID: 114020] [client.c:2360:notify] 
0-gv01-client-0: parent translators are ready, attempting connect on 
transport
[2018-04-09 05:08:13.747040] W [socket.c:3216:socket_connect] 
0-gv01-client-0: Error disabling sockopt IPV6_V6ONLY: "Protocol not 
available"
[2018-04-09 05:08:13.747372] I [MSGID: 114020] [client.c:2360:notify] 
0-gv01-client-1: parent translators are ready, attempting connect on 
transport
[2018-04-09 05:08:13.747883] E [MSGID: 114058] 
[client-handshake.c:1571:client_query_portmap_cbk] 0-gv01-client-0: 
failed to get the port number for remote subvolume. Please run 'gluster 
volume status' on server to see if brick process is running.
[2018-04-09 05:08:13.748026] I [MSGID: 114018] 
[client.c:2285:client_rpc_notify] 0-gv01-client-0: disconnected from 
gv01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-04-09 05:08:13.748070] W [MSGID: 108001] 
[afr-common.c:5391:afr_notify] 0-gv01-replicate-0: Client-quorum is not met
[2018-04-09 05:08:13.754493] W [socket.c:3216:socket_connect] 
0-gv01-client-1: Error disabling sockopt IPV6_V6ONLY: "Protocol not 
available"

Final graph:
+--+
  1: volume gv01-client-0
  2: type protocol/client
  3: option ping-timeout 42
  4: option remote-host nfs01
  5: 

Re: [Gluster-users] Gluster very poor performance when copying small files (1x (2+1) = 3, SSD)

[TomK]

On 3/19/2018 10:52 AM, Rik Theys wrote:

Hi,

On 03/19/2018 03:42 PM, TomK wrote:

On 3/19/2018 5:42 AM, Ondrej Valousek wrote:
Removing NFS or NFS Ganesha from the equation, not very impressed on my
own setup either.  For the writes it's doing, that's alot of CPU usage
in top. Seems bottle-necked via a single execution core somewhere trying
to facilitate read / writes to the other bricks.

Writes to the gluster FS from within one of the gluster participating
bricks:

[root@nfs01 n]# dd if=/dev/zero of=./some-file.bin

393505+0 records in
393505+0 records out
201474560 bytes (201 MB) copied, 50.034 s, 4.0 MB/s


That's not really a fare comparison as you don't specify a blocksize.
What does

dd if=/dev/zero of=./some-file.bin bs=1M count=1000 oflag=direct

give?


Rik

Correct.  Higher block sizes gave me better numbers earlier.  Curious 
about improving the small file size performance though, preferrably via 
gluster tunables, if possible.


Though it could be said I guess that compressing a set of large files 
and transferring them over that way is one solution.  However needed the 
small block size on dd to perhaps quickly simulate alot of small 
requests in a somewhat ok-ish way.


Here's the numbers from the VM:

[ Via Gluster ]
[root@nfs01 n]# dd if=/dev/zero of=./some-file.bin bs=1M count=1 
oflag=direct

1+0 records in
1+0 records out
1048576 bytes (10 GB) copied, 96.3228 s, 109 MB/s
[root@nfs01 n]# rm some-file.bin
rm: remove regular file âsome-file.binâ? y

[ Via XFS ]
[root@nfs01 n]# cd /bricks/0/gv01/
[root@nfs01 gv01]# dd if=/dev/zero of=./some-file.bin bs=1M count=1 
oflag=direct

1+0 records in
1+0 records out
1048576 bytes (10 GB) copied, 44.79 s, 234 MB/s
[root@nfs01 gv01]#



top - 12:49:48 up 1 day,  9:39,  2 users,  load average: 0.66, 1.15, 1.82
Tasks: 165 total,   1 running, 164 sleeping,   0 stopped,   0 zombie
%Cpu0  : 10.3 us,  9.6 sy,  0.0 ni, 28.0 id, 50.4 wa,  0.0 hi,  1.8 si, 
0.0 st
%Cpu1  : 13.8 us, 13.8 sy,  0.0 ni, 38.6 id, 30.0 wa,  0.0 hi,  3.8 si, 
0.0 st
%Cpu2  :  8.7 us,  6.9 sy,  0.0 ni, 48.7 id, 34.9 wa,  0.0 hi,  0.7 si, 
0.0 st
%Cpu3  : 10.6 us,  7.8 sy,  0.0 ni, 57.1 id, 24.1 wa,  0.0 hi,  0.4 si, 
0.0 st

KiB Mem :  3881708 total,  3543280 free,   224008 used,   114420 buff/cache
KiB Swap:  4063228 total,  3836612 free,   226616 used.  3457708 avail Mem

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
14115 root  20   0 2504832  27640   2612 S  43.5  0.7 432:10.35 
glusterfsd
 1319 root  20   0 1269620  23780   2636 S  38.9  0.6 752:44.78 
glusterfs
 1334 root  20   0 2694264  56988   1672 S  16.3  1.5 311:20.90 
ganesha.nfsd

27458 root  20   0  108984   1404540 D   3.0  0.0   0:00.24 dd
14127 root  20   0 1164720   4860   1960 S   0.7  0.1   1:47.59 
glusterfs

  750 root  20   0  389864   5528   3988 S   0.3  0.1   0:08.77 sssd_be

--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster very poor performance when copying small files (1x (2+1) = 3, SSD)

[TomK]

On 3/19/2018 5:42 AM, Ondrej Valousek wrote:
Removing NFS or NFS Ganesha from the equation, not very impressed on my 
own setup either.  For the writes it's doing, that's alot of CPU usage 
in top. Seems bottle-necked via a single execution core somewhere trying 
to facilitate read / writes to the other bricks.


Writes to the gluster FS from within one of the gluster participating 
bricks:


[root@nfs01 n]# dd if=/dev/zero of=./some-file.bin

393505+0 records in
393505+0 records out
201474560 bytes (201 MB) copied, 50.034 s, 4.0 MB/s

[root@nfs01 n]#

Top results (10 second average)won't go over 32%:

top - 00:49:38 up 21:39,  2 users,  load average: 0.42, 0.24, 0.19
Tasks: 164 total,   1 running, 163 sleeping,   0 stopped,   0 zombie
%Cpu0  : 29.3 us, 24.7 sy,  0.0 ni, 45.1 id,  0.0 wa,  0.0 hi,  0.8 si, 
0.0 st
%Cpu1  : 27.2 us, 24.1 sy,  0.0 ni, 47.2 id,  0.0 wa,  0.0 hi,  1.5 si, 
0.0 st
%Cpu2  : 20.2 us, 13.5 sy,  0.0 ni, 64.1 id,  0.0 wa,  0.0 hi,  2.3 si, 
0.0 st
%Cpu3  : 30.0 us, 16.2 sy,  0.0 ni, 47.5 id,  0.0 wa,  0.0 hi,  6.3 si, 
0.0 st

KiB Mem :  3881708 total,  3207488 free,   346680 used,   327540 buff/cache
KiB Swap:  4063228 total,  4062828 free,  400 used.  3232208 avail Mem

   PID USER  PR  NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
  1319 root  20   0  819036  12928   4036 S 32.3  0.3   1:19.64 
glusterfs
  1310 root  20   0 1232428  25636   4364 S 12.1  0.7   0:41.25 
glusterfsd



Next, the same write but directly to the brick via XFS, which of course 
is faster:



top - 09:45:09 up 1 day,  6:34,  3 users,  load average: 0.61, 1.01, 1.04
Tasks: 171 total,   2 running, 169 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.6 us,  2.1 sy,  0.0 ni, 82.6 id, 14.5 wa,  0.0 hi,  0.2 si, 
0.0 st
%Cpu1  : 16.7 us, 83.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si, 
0.0 st
%Cpu2  :  0.4 us,  0.9 sy,  0.0 ni, 94.2 id,  4.4 wa,  0.0 hi,  0.0 si, 
0.0 st
%Cpu3  :  1.1 us,  0.6 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  0.0 si, 
0.0 st

KiB Mem :  3881708 total,   501120 free,   230704 used,  3149884 buff/cache
KiB Swap:  4063228 total,  3876896 free,   186332 used.  3343960 avail Mem

  PID USER  PR  NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
14691 root  20   0  107948608512 R 25.0  0.0   0:34.29 dd
 1334 root  20   0 2694264  61076   2228 S  2.7  1.6 283:55.96 
ganesha.nfsd



The result of a dd command directly against the brick FS itself is of 
course much better:



[root@nfs01 gv01]# dd if=/dev/zero of=./some-file.bin
5771692+0 records in
5771692+0 records out
2955106304 bytes (3.0 GB) copied, 35.3425 s, 83.6 MB/s

[root@nfs01 gv01]# pwd
/bricks/0/gv01
[root@nfs01 gv01]#

Tried a few tweak options with no effect:

[root@nfs01 glusterfs]# gluster volume info

Volume Name: gv01
Type: Replicate
Volume ID: e5ccc75e-5192-45ac-b410-a34ebd777666
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: nfs01:/bricks/0/gv01
Brick2: nfs02:/bricks/0/gv01
Options Reconfigured:
cluster.server-quorum-type: server
cluster.quorum-type: auto
server.event-threads: 8
client.event-threads: 8
performance.readdir-ahead: on
performance.write-behind-window-size: 8MB
performance.io-thread-count: 16
performance.cache-size: 1GB
nfs.trusted-sync: on
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
[root@nfs01 glusterfs]#

That's despite that I can confirm doing 90+ MB/s on my 1Gbe network. 
Thoughts?


--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.



Hi,
As I posted in my previous emails - glusterfs can never match NFS (especially 
async one) performance of small files/latency. That's given by the design.
Nothing you can do about it.
Ondrej

-Original Message-
From: gluster-users-boun...@gluster.org 
[mailto:gluster-users-boun...@gluster.org] On Behalf Of Rik Theys
Sent: Monday, March 19, 2018 10:38 AM
To: gluster-users@gluster.org; mailingli...@smcleod.net
Subject: Re: [Gluster-users] Gluster very poor performance when copying small 
files (1x (2+1) = 3, SSD)

Hi,

I've done some similar tests and experience similar performance issues (see my 
'gluster for home directories?' thread on the list).

If I read your mail correctly, you are comparing an NFS mount of the brick disk 
against a gluster mount (using the fuse client)?

Which options do you have set on the NFS export (sync or async)?

 From my tests, I concluded that the issue was not bandwidth but latency.
Gluster will only return an IO operation once all bricks have confirmed that 
the data is on disk. If you are using a fuse mount, you might compare with 
using the 'direct-io-mode=disable' option on the client might help (no 
experience with this).

In our tests, I've used NFS-ganesha to serve the gluster volume over NFS. This makes 
things even worse as NFS-ganesha has no "async" mode, 

Re: [Gluster-users] Gluster very poor performance when copying small files (1x (2+1) = 3, SSD)

[TomK]

On 3/18/2018 6:13 PM, Sam McLeod wrote:
Even your NFS transfers are 12.5 or so MB per second or less.

1) Did you use fdisk and LVM under that XFS filesystem?

2) Did you benchmark the XFS with something like bonnie++?  (There's 
probably newer benchmark suites now.)


3) Did you benchmark your Network transfer speeds?  Perhaps your NIC 
negotiated a lower speed.


3) I've done XFS tuning for another purpose but got good results.  If it 
helps, I can send you the doc.


Cheers,
Tom


Howdy all,

We're experiencing terrible small file performance when copying or 
moving files on gluster clients.


In the example below, Gluster is taking 6mins~ to copy 128MB / 21,000 
files sideways on a client, doing the same thing on NFS (which I know is 
a totally different solution etc. etc.) takes approximately 10-15 
seconds(!).


Any advice for tuning the volume or XFS settings would be greatly 
appreciated.


Hopefully I've included enough relevant information below.


## Gluster Client

root@gluster-client:/mnt/gluster_perf_test/  # du -sh .
127M    .
root@gluster-client:/mnt/gluster_perf_test/  # find . -type f | wc -l
21791
root@gluster-client:/mnt/gluster_perf_test/  # du 9584toto9584.txt
4    9584toto9584.txt


root@gluster-client:/mnt/gluster_perf_test/  # time cp -a private 
private_perf_test


real    5m51.862s
user    0m0.862s
sys    0m8.334s

root@gluster-client:/mnt/gluster_perf_test/ # time rm -rf private_perf_test/

real    0m49.702s
user    0m0.087s
sys    0m0.958s


## Hosts

- 16x Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz per Gluster host / client
- Storage: iSCSI provisioned (via 10Gbit DAC/Fibre), SSD disk, 50K R/RW 
4k IOP/s, 400MB/s per Gluster host

- Volumes are replicated across two hosts and one arbiter only host
- Networking is 10Gbit DAC/Fibre between Gluster hosts and clients
- 18GB DDR4 ECC memory

## Volume Info

root@gluster-host-01:~ # gluster pool list
UUID          Hostname                        State
ad02970b-e2aa-4ca8-998c-bd10d5970faa  gluster-host-02.fqdn Connected
ea116a94-c19e-48db-b108-0be3ae622e2e  gluster-host-03.fqdn Connected
2e855c25-e7ac-4ff6-be85-e8bcc6f45ee4  localhost   
Connected


root@gluster-host-01:~ # gluster volume info uat_storage

Volume Name: uat_storage
Type: Replicate
Volume ID: 7918f1c5-5031-47b8-b054-56f6f0c569a2
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gluster-host-01.fqdn:/mnt/gluster-storage/uat_storage
Brick2: gluster-host-02.fqdn:/mnt/gluster-storage/uat_storage
Brick3: gluster-host-03.fqdn:/mnt/gluster-storage/uat_storage (arbiter)
Options Reconfigured:
performance.rda-cache-limit: 256MB
network.inode-lru-limit: 5
server.outstanding-rpc-limit: 256
performance.client-io-threads: true
nfs.disable: on
transport.address-family: inet
client.event-threads: 8
cluster.eager-lock: true
cluster.favorite-child-policy: size
cluster.lookup-optimize: true
cluster.readdir-optimize: true
cluster.use-compound-fops: true
diagnostics.brick-log-level: ERROR
diagnostics.client-log-level: ERROR
features.cache-invalidation-timeout: 600
features.cache-invalidation: true
network.ping-timeout: 15
performance.cache-invalidation: true
performance.cache-max-file-size: 6MB
performance.cache-refresh-timeout: 60
performance.cache-size: 1024MB
performance.io -thread-count: 16
performance.md-cache-timeout: 600
performance.stat-prefetch: true
performance.write-behind-window-size: 256MB
server.event-threads: 8
transport.listen-backlog: 2048

root@gluster-host-01:~ # xfs_info /dev/mapper/gluster-storage-unlocked
meta-data=/dev/mapper/gluster-storage-unlocked isize=512    agcount=4, 
agsize=196607360 blks

          =                       sectsz=512   attr=2, projid32bit=1
          =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=786429440, imaxpct=5
          =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=383998, version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


--
Sam McLeod (protoporpoise on IRC)
https://smcleod.net
https://twitter.com/s_mcleod

Words are my own opinions and do not necessarily represent those of 
my employer or partners.




___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users




--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS Ganesha HA w/ GlusterFS

[TomK]

On 2/26/2018 7:14 AM, Kaleb S. KEITHLEY wrote:
Hey,

Yep. A blog is where I was writing it up to begin with.

Anyway, got alot of demand for it over the last one day so here it is:

http://microdevsys.com/wp/glusterfs-configuration-and-setup-w-nfs-ganesha-for-an-ha-nfs-cluster/

Skip to the SUMMARY and TESTING sections so you can just copy and paste 
the configs to get things moving very quickly.  The detailed section is 
my running log of all the troubleshooting and failed attempts.


FreeIPA is being used as the DNS / Kerberos backend to which these NFS 
servers will be configured to.  Not yet done with this piece so not yet 
including that mailing list here.


Feel free to point anything out as I would like to keep it accurate.

The post includes all work needed for firewalld and selinux on CentOS 7 
without turning off either service.


Again, thanks for all the help here.  Couldn't get this working without 
all the work you guy's do!


Cheers,
Tom



On 02/25/2018 08:29 PM, TomK wrote:

Hey Guy's,

A success story instead of a question.

With your help, managed to get the HA component working with HAPROXY and
keepalived to build a fairly resilient NFS v4 VM cluster.  ( Used
Gluster, NFS Ganesha v2.60, HAPROXY, keepalived w/ selinux enabled )

If someone needs or it could help your work, please PM me for the
written up post or I could just post here if the lists allow it.



Hi,

I strongly encourage you to write a blog post about it. And if not that
at least write about it and post it to the list(s).

I'm not sure why your post to nfs-ganesha-support was blocked. Maybe
it's waiting for some moderator attention. (But I don't believe that
list is moderated.)

Thanks for sharing.




--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS Ganesha HA w/ GlusterFS

[TomK]

Hey Guy's,

A success story instead of a question.

With your help, managed to get the HA component working with HAPROXY and 
keepalived to build a fairly resilient NFS v4 VM cluster.  ( Used 
Gluster, NFS Ganesha v2.60, HAPROXY, keepalived w/ selinux enabled )


If someone needs or it could help your work, please PM me for the 
written up post or I could just post here if the lists allow it.


Cheers,
Tom


On 2/19/2018 12:25 PM, TomK wrote:

On 2/19/2018 12:09 PM, Kaleb S. KEITHLEY wrote:
Sounds good and no problem at all.  Will look out for this update in the 
future.  In the meantime, three's a few things I'll try including your 
suggestion.


Was looking for a sense of direction with the projects and now you've 
given that.  Ty.  Appreciated!


Cheers,
Tom



On 02/19/2018 11:37 AM, TomK wrote:

On 2/19/2018 10:55 AM, Kaleb S. KEITHLEY wrote:
Yep, I noticed a couple of pages including this for 'storhaug
configuration' off google.  Adding 'mailing list' to the search didn't
help alot:

https://sourceforge.net/p/nfs-ganesha/mailman/message/35929089/

https://www.spinics.net/lists/gluster-users/msg33018.html

Hence the ask here.  storhaug feels like it's not moving with any sort
of update now.

Any plans to move back to the previous NFS Ganesha HA model with
upcoming GlusterFS versions as a result?


No.

(re)writing or finishing storhaug has been on my plate ever since the
guy who was supposed to do it didn't.

I have lots of other stuff to do too. All I can say is it'll get done
when it gets done.




In the meantime I'll look to cobble up the GlusterFS 3.10 packages and
try with those per your suggestion.

What's your thoughts on using HAPROXY / keepalived w/ NFS Ganesha and
GlusterFS?  Anyone tried this sort of combination?  I want to avoid the
situation where I have to remount clients as a result of a node failing.
  In other words, avoid this situation:

[root@yes01 ~]# cd /n
-bash: cd: /n: Stale file handle
[root@yes01 ~]#

Cheers,
Tom


On 02/19/2018 10:24 AM, TomK wrote:

On 2/19/2018 2:39 AM, TomK wrote:
+ gluster users as well.  Just read another post on the mailing lists
about a similar ask from Nov which didn't really have a clear answer.


That's funny because I've answered questions like this several times.

Gluster+Ganesha+Pacemaker-based HA is available up to GlusterFS 3.10.x.

If you need HA, that is one "out of the box" option.

There's support for using CTDB in Samba for Ganesha HA, and people have
used it successfully with Gluster+Ganesha.



Perhaps there's a way to get NFSv4 work with GlusterFS without NFS
Ganesha then?


Not that I'm aware of.



Cheers,
Tom


Hey All,

I've setup GlusterFS on two virtuals and enabled NFS Ganesha on each
node.  ATM the configs are identical between the two NFS Ganesha
hosts. (Probably shouldn't be but I'm just testing things out.)

I need HA capability and notice these instructions here:

http://aravindavkgluster.readthedocs.io/en/latest/Administrator%20Guide/Configuring%20HA%20NFS%20Server/ 





However I don't have package glusterfs-ganesha available on this
CentOS Linux release 7.4.1708 (Core) and the maintainer's of CentOS 7
haven't uploaded some of the 2.5.x packages yet so I can't use that
version.

glusterfs-api-3.12.6-1.el7.x86_64
glusterfs-libs-3.12.6-1.el7.x86_64
glusterfs-3.12.6-1.el7.x86_64
glusterfs-fuse-3.12.6-1.el7.x86_64
glusterfs-server-3.12.6-1.el7.x86_64
python2-glusterfs-api-1.1-1.el7.noarch
glusterfs-client-xlators-3.12.6-1.el7.x86_64
glusterfs-cli-3.12.6-1.el7.x86_64

nfs-ganesha-xfs-2.3.2-1.el7.x86_64
nfs-ganesha-vfs-2.3.2-1.el7.x86_64
nfs-ganesha-2.3.2-1.el7.x86_64
nfs-ganesha-gluster-2.3.2-1.el7.x86_64

The only high availability packages are the following but they don't
come with any instructions that I can find:

storhaug.noarch : High-Availability Add-on for NFS-Ganesha and Samba
storhaug-nfs.noarch : storhaug NFS-Ganesha module

Given that I'm missing that one package above, will configuring using
ganesha-ha.conf still work?  Or should I be looking at another option
alltogether?

Appreciate any help.  Ty!

















--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS Ganesha HA w/ GlusterFS

[TomK]

On 2/19/2018 12:09 PM, Kaleb S. KEITHLEY wrote:
Sounds good and no problem at all.  Will look out for this update in the 
future.  In the meantime, three's a few things I'll try including your 
suggestion.


Was looking for a sense of direction with the projects and now you've 
given that.  Ty.  Appreciated!


Cheers,
Tom



On 02/19/2018 11:37 AM, TomK wrote:

On 2/19/2018 10:55 AM, Kaleb S. KEITHLEY wrote:
Yep, I noticed a couple of pages including this for 'storhaug
configuration' off google.  Adding 'mailing list' to the search didn't
help alot:

https://sourceforge.net/p/nfs-ganesha/mailman/message/35929089/

https://www.spinics.net/lists/gluster-users/msg33018.html

Hence the ask here.  storhaug feels like it's not moving with any sort
of update now.

Any plans to move back to the previous NFS Ganesha HA model with
upcoming GlusterFS versions as a result?


No.

(re)writing or finishing storhaug has been on my plate ever since the
guy who was supposed to do it didn't.

I have lots of other stuff to do too. All I can say is it'll get done
when it gets done.




In the meantime I'll look to cobble up the GlusterFS 3.10 packages and
try with those per your suggestion.

What's your thoughts on using HAPROXY / keepalived w/ NFS Ganesha and
GlusterFS?  Anyone tried this sort of combination?  I want to avoid the
situation where I have to remount clients as a result of a node failing.
  In other words, avoid this situation:

[root@yes01 ~]# cd /n
-bash: cd: /n: Stale file handle
[root@yes01 ~]#

Cheers,
Tom


On 02/19/2018 10:24 AM, TomK wrote:

On 2/19/2018 2:39 AM, TomK wrote:
+ gluster users as well.  Just read another post on the mailing lists
about a similar ask from Nov which didn't really have a clear answer.


That's funny because I've answered questions like this several times.

Gluster+Ganesha+Pacemaker-based HA is available up to GlusterFS 3.10.x.

If you need HA, that is one "out of the box" option.

There's support for using CTDB in Samba for Ganesha HA, and people have
used it successfully with Gluster+Ganesha.



Perhaps there's a way to get NFSv4 work with GlusterFS without NFS
Ganesha then?


Not that I'm aware of.



Cheers,
Tom


Hey All,

I've setup GlusterFS on two virtuals and enabled NFS Ganesha on each
node.  ATM the configs are identical between the two NFS Ganesha
hosts. (Probably shouldn't be but I'm just testing things out.)

I need HA capability and notice these instructions here:

http://aravindavkgluster.readthedocs.io/en/latest/Administrator%20Guide/Configuring%20HA%20NFS%20Server/



However I don't have package glusterfs-ganesha available on this
CentOS Linux release 7.4.1708 (Core) and the maintainer's of CentOS 7
haven't uploaded some of the 2.5.x packages yet so I can't use that
version.

glusterfs-api-3.12.6-1.el7.x86_64
glusterfs-libs-3.12.6-1.el7.x86_64
glusterfs-3.12.6-1.el7.x86_64
glusterfs-fuse-3.12.6-1.el7.x86_64
glusterfs-server-3.12.6-1.el7.x86_64
python2-glusterfs-api-1.1-1.el7.noarch
glusterfs-client-xlators-3.12.6-1.el7.x86_64
glusterfs-cli-3.12.6-1.el7.x86_64

nfs-ganesha-xfs-2.3.2-1.el7.x86_64
nfs-ganesha-vfs-2.3.2-1.el7.x86_64
nfs-ganesha-2.3.2-1.el7.x86_64
nfs-ganesha-gluster-2.3.2-1.el7.x86_64

The only high availability packages are the following but they don't
come with any instructions that I can find:

storhaug.noarch : High-Availability Add-on for NFS-Ganesha and Samba
storhaug-nfs.noarch : storhaug NFS-Ganesha module

Given that I'm missing that one package above, will configuring using
ganesha-ha.conf still work?  Or should I be looking at another option
alltogether?

Appreciate any help.  Ty!














--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS Ganesha HA w/ GlusterFS

[TomK]

On 2/19/2018 10:55 AM, Kaleb S. KEITHLEY wrote:
Yep, I noticed a couple of pages including this for 'storhaug 
configuration' off google.  Adding 'mailing list' to the search didn't 
help alot:


https://sourceforge.net/p/nfs-ganesha/mailman/message/35929089/

https://www.spinics.net/lists/gluster-users/msg33018.html

Hence the ask here.  storhaug feels like it's not moving with any sort 
of update now.


Any plans to move back to the previous NFS Ganesha HA model with 
upcoming GlusterFS versions as a result?


In the meantime I'll look to cobble up the GlusterFS 3.10 packages and 
try with those per your suggestion.


What's your thoughts on using HAPROXY / keepalived w/ NFS Ganesha and 
GlusterFS?  Anyone tried this sort of combination?  I want to avoid the 
situation where I have to remount clients as a result of a node failing. 
 In other words, avoid this situation:


[root@yes01 ~]# cd /n
-bash: cd: /n: Stale file handle
[root@yes01 ~]#

Cheers,
Tom


On 02/19/2018 10:24 AM, TomK wrote:

On 2/19/2018 2:39 AM, TomK wrote:
+ gluster users as well.  Just read another post on the mailing lists
about a similar ask from Nov which didn't really have a clear answer.


That's funny because I've answered questions like this several times.

Gluster+Ganesha+Pacemaker-based HA is available up to GlusterFS 3.10.x.

If you need HA, that is one "out of the box" option.

There's support for using CTDB in Samba for Ganesha HA, and people have
used it successfully with Gluster+Ganesha.



Perhaps there's a way to get NFSv4 work with GlusterFS without NFS
Ganesha then?


Not that I'm aware of.



Cheers,
Tom


Hey All,

I've setup GlusterFS on two virtuals and enabled NFS Ganesha on each
node.  ATM the configs are identical between the two NFS Ganesha
hosts. (Probably shouldn't be but I'm just testing things out.)

I need HA capability and notice these instructions here:

http://aravindavkgluster.readthedocs.io/en/latest/Administrator%20Guide/Configuring%20HA%20NFS%20Server/


However I don't have package glusterfs-ganesha available on this
CentOS Linux release 7.4.1708 (Core) and the maintainer's of CentOS 7
haven't uploaded some of the 2.5.x packages yet so I can't use that
version.

glusterfs-api-3.12.6-1.el7.x86_64
glusterfs-libs-3.12.6-1.el7.x86_64
glusterfs-3.12.6-1.el7.x86_64
glusterfs-fuse-3.12.6-1.el7.x86_64
glusterfs-server-3.12.6-1.el7.x86_64
python2-glusterfs-api-1.1-1.el7.noarch
glusterfs-client-xlators-3.12.6-1.el7.x86_64
glusterfs-cli-3.12.6-1.el7.x86_64

nfs-ganesha-xfs-2.3.2-1.el7.x86_64
nfs-ganesha-vfs-2.3.2-1.el7.x86_64
nfs-ganesha-2.3.2-1.el7.x86_64
nfs-ganesha-gluster-2.3.2-1.el7.x86_64

The only high availability packages are the following but they don't
come with any instructions that I can find:

storhaug.noarch : High-Availability Add-on for NFS-Ganesha and Samba
storhaug-nfs.noarch : storhaug NFS-Ganesha module

Given that I'm missing that one package above, will configuring using
ganesha-ha.conf still work?  Or should I be looking at another option
alltogether?

Appreciate any help.  Ty!









--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS Ganesha HA w/ GlusterFS

[TomK]

On 2/19/2018 2:39 AM, TomK wrote:
+ gluster users as well.  Just read another post on the mailing lists 
about a similar ask from Nov which didn't really have a clear answer.


Perhaps there's a way to get NFSv4 work with GlusterFS without NFS 
Ganesha then?


Cheers,
Tom


Hey All,

I've setup GlusterFS on two virtuals and enabled NFS Ganesha on each 
node.  ATM the configs are identical between the two NFS Ganesha hosts. 
(Probably shouldn't be but I'm just testing things out.)


I need HA capability and notice these instructions here:

http://aravindavkgluster.readthedocs.io/en/latest/Administrator%20Guide/Configuring%20HA%20NFS%20Server/ 



However I don't have package glusterfs-ganesha available on this CentOS 
Linux release 7.4.1708 (Core) and the maintainer's of CentOS 7 haven't 
uploaded some of the 2.5.x packages yet so I can't use that version.


glusterfs-api-3.12.6-1.el7.x86_64
glusterfs-libs-3.12.6-1.el7.x86_64
glusterfs-3.12.6-1.el7.x86_64
glusterfs-fuse-3.12.6-1.el7.x86_64
glusterfs-server-3.12.6-1.el7.x86_64
python2-glusterfs-api-1.1-1.el7.noarch
glusterfs-client-xlators-3.12.6-1.el7.x86_64
glusterfs-cli-3.12.6-1.el7.x86_64

nfs-ganesha-xfs-2.3.2-1.el7.x86_64
nfs-ganesha-vfs-2.3.2-1.el7.x86_64
nfs-ganesha-2.3.2-1.el7.x86_64
nfs-ganesha-gluster-2.3.2-1.el7.x86_64

The only high availability packages are the following but they don't 
come with any instructions that I can find:


storhaug.noarch : High-Availability Add-on for NFS-Ganesha and Samba
storhaug-nfs.noarch : storhaug NFS-Ganesha module

Given that I'm missing that one package above, will configuring using 
ganesha-ha.conf still work?  Or should I be looking at another option 
alltogether?


Appreciate any help.  Ty!




--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] GlusterFS w/ NFS-Ganesha: Need working instructions.

[TomK]

On 2/18/2018 1:05 AM, TomK wrote:
Never mind.  Found what I needed.  Ty.

Cheers,
Tom


Hey All,

Trying to get GlusterFS w/ NFS-Ganesha installed but when following the 
page below, I get a few 404 errors, glusterfs-server is missing in the 
repos etc.


Is there a more recent version of the same document for CentOS7 that can 
be recommended?


Current link with issues:

http://blog.gluster.org/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/ 






--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] GlusterFS w/ NFS-Ganesha: Need working instructions.

[TomK]

Hey All,

Trying to get GlusterFS w/ NFS-Ganesha installed but when following the 
page below, I get a few 404 errors, glusterfs-server is missing in the 
repos etc.


Is there a more recent version of the same document for CentOS7 that can 
be recommended?


Current link with issues:

http://blog.gluster.org/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/

--
Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Data Replication

[TomK]

Hey All,

Still new and learning GlusterFS.  Thanks for bearing with me.

I cannot writes files from nodes to the glusterfs and see them 
replicated.  I can only write from the master node opennebula01 and see 
the file distributed and replicated.  How do I replicate from the nodes? 
Or am I just asking something that isn't part of the design:


[root@mdskvm-p01 glusterv01]# echo "from mdskvm-p01" > from.txt
[root@mdskvm-p01 glusterv01]# ls -altri
total 6792232
   146 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19 
ouch-10.iso
   145 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19 
ouch-09.iso
   144 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19 
ouch-08.iso
   143 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19 
ouch-07.iso
   142 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19 
ouch-06.iso
   141 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19 
ouch-05.iso
   140 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19 
ouch-04.iso
   136 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:19 
CentOS-7-x86_64-Minimal-1511.iso
   137 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:20 
ouch-01.iso
   139 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:21 
ouch-03.iso
   138 -rwxr--r--. 1 oneadmin oneadmin 632291328 May 19 01:21 
ouch-02.iso

2147483776 drwxr-xr-x. 3 root root24 Jun  9 01:29 .trashcan
   128 drwxr-xr-x. 3 oneadmin oneadmin23 Jun  9 01:30 ..
 536871104 drw---. 9 root root  4096 Jun  9 01:35 
.glusterfs

 402653377 -rw-r--r--. 2 root root 3 Jun  9 01:35 test.txt
 402653378 -rw-r--r--. 1 root root16 Jun  9 01:36 from.txt
 402653376 drwxr-xr-x. 4 oneadmin oneadmin  4096 Jun  9 01:36 .
[root@mdskvm-p01 glusterv01]#


[root@mdskvm-p02 glusterv02]# ls -altri
total 4
64 drwxr-xr-x. 3 root root  23 Jun  8 19:38 ..
67 drw---. 9 root root 188 Jun  8 19:45 .glusterfs
3221225600 drwxr-xr-x. 4 oneadmin oneadmin  54 Jun  8 19:45 .
3221225604 -rw-r--r--. 2 root root   3 Jun  8 19:45 test.txt
74 drwxr-xr-x. 3 root root  24 Jun  9  2016 .trashcan
[root@mdskvm-p02 glusterv02]#



[root@opennebula01 0]# ls -altri
total 8
68736015 drwxr-x---. 7 oneadmin oneadmin   60 May  9 00:19 ..
   1 drwxr-xr-x. 4 oneadmin oneadmin 4096 Jun  9 01:29 .
   5 drwxr-xr-x. 3 root root 4096 Jun  9 01:29 .trashcan
[root@opennebula01 0]#
[root@opennebula01 0]#
[root@opennebula01 0]#
[root@opennebula01 0]#
[root@opennebula01 0]# echo "yo" > test.txt
[root@opennebula01 0]# ls -altri
total 9
68736015 drwxr-x---. 7 oneadmin oneadmin   60 May  9 00:19 ..
   1 drwxr-xr-x. 4 oneadmin oneadmin 4096 Jun  8 20:45 .
10502401602053818307 -rw-r--r--. 1 root root3 Jun  8 20:45 
test.txt
   5 drwxr-xr-x. 3 root root 4096 Jun  9 01:29 
.trashcan

[root@opennebula01 0]#
[root@opennebula01 0]#



Cheers,
Tom K.
-

Living on earth is expensive, but it includes a free trip around the sun.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Hey All, Anyway to remove the last brick or change it? Added erronously.

[TomK]

Thanks very much!

Did that then recovered the ID as well using the below off of google:

vol=mdsglusterv01
brick=/mnt/p01-d01/glusterv01
setfattr -n trusted.glusterfs.volume-id \
  -v 0x$(grep volume-id /var/lib/glusterd/vols/$vol/info \
  | cut -d= -f2 | sed 's/-//g') $brick

Another question I had is, if I add a 2TB LUN brick to gluster, w/ the 
LUN existing on this physical node where I also have KVM running, will 
the KVM guests write directly to the LUN @ FC speeds or would the writes 
occur via glusterfs and the transport I selected below?   Is gluster 
intelligent enough to recognize that a particular brick exists on the 
same physical from which writes are occurring and allow direct writes to 
the brick via what ever medium it's mounted on (NFS, FC, FCoE, iSCSI etc)?


Disk /dev/sdb: 2199.0 GB, 219902322 bytes, 4294967296 sectors

[root@mdskvm-p01 ~]# gluster volume info

Volume Name: mdsglusterv01
Type: Distribute
Volume ID: 84df37b3-3c68-4cc0-9383-3ff539a4d785
Status: Started
Number of Bricks: 1
Transport-type: tcp,rdma
Bricks:
Brick1: mdskvm-p01:/mnt/p01-d01/glusterv01
Options Reconfigured:
config.transport: tcp,rdma
performance.readdir-ahead: on
[root@mdskvm-p01 ~]#

Cheers,
Tom K.
- 


Living on earth is expensive, but it includes a free trip around the sun.

On 5/2/2016 10:17 PM, Chen Chen wrote:

Hi Tom,

You may try:
gluster volume set volname config.transport tcp

ref:
http://www.gluster.org/community/documentation/index.php/RDMA_Transport#Changing_Transport_of_Volume 



Best regards,
Chen

On 5/3/2016 9:55 AM, TomK wrote:

Hey All,

New here and first time posting.  I've made a typo in configuration and
entered:

gluster volume create mdsglusterv01 transport rdma
mdskvm-p01:/mnt/p01-d01/glusterv01/

but couldn't start since rdma didn't exist:

[root@mdskvm-p01 glusterfs]# ls -altri
/usr/lib64/glusterfs/3.7.11/rpc-transport/*
135169384 -rwxr-xr-x. 1 root root 99648 Apr 18 08:21
/usr/lib64/glusterfs/3.7.11/rpc-transport/socket.so
[root@mdskvm-p01 glusterfs]#

So how do I reconfigure gluster to remove the rdma option and redefine
this brick?

[root@mdskvm-p01 glusterfs]# gluster volume info

Volume Name: mdsglusterv01
Type: Distribute
Volume ID: 84df37b3-3c68-4cc0-9383-3ff539a4d785
Status: Stopped
Number of Bricks: 1
Transport-type: rdma
Bricks:
Brick1: mdskvm-p01:/mnt/p01-d01/glusterv01
Options Reconfigured:
performance.readdir-ahead: on
[root@mdskvm-p01 glusterfs]#

I figure I can add in a dummy block device however wanted to find out if
there's anyway to change the above or redefine it before I do that.

Cheers,
Tom K.
- 



Living on earth is expensive, but it includes a free trip around the 
sun.




___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users





___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Anyway to remove the last brick or change it?

[TomK]

Hey All,

New here and first time posting.  I've made a typo and entered:

gluster volume create mdsglusterv01 transport rdma 
mdskvm-p01:/mnt/p01-d01/glusterv01/


but couldn't start since rdma didn't exist:

[root@mdskvm-p01 glusterfs]# ls -altri 
/usr/lib64/glusterfs/3.7.11/rpc-transport/*
135169384 -rwxr-xr-x. 1 root root 99648 Apr 18 08:21 
/usr/lib64/glusterfs/3.7.11/rpc-transport/socket.so

[root@mdskvm-p01 glusterfs]#

So how do I reconfigure gluster to remove the rdma option and redefine 
this brick?


[root@mdskvm-p01 glusterfs]# gluster volume info

Volume Name: mdsglusterv01
Type: Distribute
Volume ID: 84df37b3-3c68-4cc0-9383-3ff539a4d785
Status: Stopped
Number of Bricks: 1
Transport-type: rdma
Bricks:
Brick1: mdskvm-p01:/mnt/p01-d01/glusterv01
Options Reconfigured:
performance.readdir-ahead: on
[root@mdskvm-p01 glusterfs]#

Cheers,
Tom K.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Hey All, Anyway to remove the last brick or change it? Added erronously.

[TomK]

Hey All,

New here and first time posting.  I've made a typo in configuration and 
entered:


gluster volume create mdsglusterv01 transport rdma 
mdskvm-p01:/mnt/p01-d01/glusterv01/


but couldn't start since rdma didn't exist:

[root@mdskvm-p01 glusterfs]# ls -altri 
/usr/lib64/glusterfs/3.7.11/rpc-transport/*
135169384 -rwxr-xr-x. 1 root root 99648 Apr 18 08:21 
/usr/lib64/glusterfs/3.7.11/rpc-transport/socket.so

[root@mdskvm-p01 glusterfs]#

So how do I reconfigure gluster to remove the rdma option and redefine 
this brick?


[root@mdskvm-p01 glusterfs]# gluster volume info

Volume Name: mdsglusterv01
Type: Distribute
Volume ID: 84df37b3-3c68-4cc0-9383-3ff539a4d785
Status: Stopped
Number of Bricks: 1
Transport-type: rdma
Bricks:
Brick1: mdskvm-p01:/mnt/p01-d01/glusterv01
Options Reconfigured:
performance.readdir-ahead: on
[root@mdskvm-p01 glusterfs]#

I figure I can add in a dummy block device however wanted to find out if 
there's anyway to change the above or redefine it before I do that.


Cheers,
Tom K.
- 


Living on earth is expensive, but it includes a free trip around the sun.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users