Re: [Gluster-users] [Gluster-devel] Query on healing process

[ABHISHEK PALIWAL]

Hi Ravi,

3. On the rebooted node, do you have ssl enabled by any chance? There is a
bug for "Not able to fetch volfile' when ssl is enabled:
https://bugzilla.redhat.com/show_bug.cgi?id=1258931

-> I have checked but ssl is disabled but still getting these errors

# gluster volume heal c_glusterfs info
c_glusterfs: Not able to fetch volfile from glusterd
Volume heal failed.

# gluster volume heal c_glusterfs info split-brain
c_glusterfs: Not able to fetch volfile from glusterd
Volume heal failed.

And based on the your observation I understood that this is not the problem
of split-brain but *is there any way through which can find out the file
which is not in split-brain as well as not in sync?*

# getfattr -m . -d -e hex
/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
getfattr: Removing leading '/' from absolute path names
# file:
opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
trusted.afr.c_glusterfs-client-0=0x
trusted.afr.c_glusterfs-client-2=0x
trusted.afr.c_glusterfs-client-4=0x
trusted.afr.c_glusterfs-client-6=0x
trusted.afr.c_glusterfs-client-8=*0x0006** //because
client8 is the latest client in our case and starting 8 digits *

*0006are saying like there is something in changelog data.*
trusted.afr.dirty=0x
trusted.bit-rot.version=0x001356d86c0c000217fd
trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae

# lhsh 002500 getfattr -m . -d -e hex
/opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
getfattr: Removing leading '/' from absolute path names
# file:
opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
trusted.afr.c_glusterfs-client-1=*0x** // and here
we can say that there is no split brain but the file is out of sync*
trusted.afr.dirty=0x
trusted.bit-rot.version=0x001156d86c290005735c
trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae

# gluster volume info

Volume Name: c_glusterfs
Type: Replicate
Volume ID: c6a61455-d378-48bf-ad40-7a3ce897fc9c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.32.0.48:/opt/lvmdir/c2/brick
Brick2: 10.32.1.144:/opt/lvmdir/c2/brick
Options Reconfigured:
performance.readdir-ahead: on
network.ping-timeout: 4
nfs.disable: on


# gluster volume info

Volume Name: c_glusterfs
Type: Replicate
Volume ID: c6a61455-d378-48bf-ad40-7a3ce897fc9c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.32.0.48:/opt/lvmdir/c2/brick
Brick2: 10.32.1.144:/opt/lvmdir/c2/brick
Options Reconfigured:
performance.readdir-ahead: on
network.ping-timeout: 4
nfs.disable: on

# gluster --version
glusterfs 3.7.8 built on Feb 17 2016 07:49:49
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. 
>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
# gluster volume heal info heal-failed
Usage: volume heal  [enable | disable | full |statistics
[heal-count [replica ]] |info [healed | heal-failed |
split-brain] |split-brain {bigger-file  |source-brick
 []}]
# gluster volume heal c_glusterfs info heal-failed
Command not supported. Please use "gluster volume heal c_glusterfs info"
and logs to find the heal information.
# lhsh 002500
 ___  _   _  _ __   _ _ _ _ _
 |   |_] |_]  ||   | \  | | |  \___/
 |_  |   ||_ __|__ |  \_| |_| _/   \_

002500> gluster --version
glusterfs 3.7.8 built on Feb 17 2016 07:49:49
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. 
>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
002500>

Regards,
Abhishek

On Thu, Mar 3, 2016 at 4:54 PM, ABHISHEK PALIWAL 
wrote:

>
> On Thu, Mar 3, 2016 at 4:10 PM, Ravishankar N 
> wrote:
>
>> Hi,
>>
>> On 03/03/2016 11:14 AM, ABHISHEK PALIWAL wrote:
>>
>> Hi Ravi,
>>
>> As I discussed earlier this issue, I investigated this issue and find
>> that healing is not triggered because the "gluster volume heal c_glusterfs
>> info split-brain" command not showing any entries as a outcome of this
>> command even though the file in split brain 

Re: [Gluster-users] ZFS error in logs

[Kaushal M]

On Fri, Mar 4, 2016 at 10:59 AM, Gmail  wrote:
> I’m trying to use ZFS with Gluster 3.7.3 and I see the following error in
> the logs:
>
> E [MSGID: 106419] [glusterd-utils.c:4973:glusterd_add_inode_size_to_dict]
> 0-management: could not find (null) to getinode size for zpool/brick01
> (zfs): (null) package missing?
>

The function `glusterd_add_inode_size_to_dict` is called when a
`gluster volume status  detail` is issued. This command
attempts to identify information related to bricks, and one of the
info it tries to find is inode size. Fetching the inode size has been
only implemented for a handful of Linux file systems. The error is not
a ZFS error, it's a GlusterD error because it doesn't know how to get
inode size for ZFS. It should have no impact on the functioning of
your GlusterFS volumes on ZFS.

>
> did anybody notice this error before?
>
>
>
> — Bishoy
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] ZFS error in logs

[Gmail]

I’m trying to use ZFS with Gluster 3.7.3 and I see the following error in the 
logs:

E [MSGID: 106419] [glusterd-utils.c:4973:glusterd_add_inode_size_to_dict] 
0-management: could not find (null) to getinode size for zpool/brick01 (zfs): 
(null) package missing?


did anybody notice this error before?



— Bishoy

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Fwd: [Gluster-devel] proposal to remove the qemu-block xlator from master branch

[Kaleb Keithley]

- Forwarded Message -

It's not clear to some of us that anyone is using this xlator.

The associated contrib/qemu sources are very old, and there is nobody currently 
maintaining it. It would take a substantial effort to update it – to what end, 
if nobody actually uses it?

Bundled in the source, the way it is now, is strongly discouraged by at least 
one major Linux distribution.

The patch under review as an RFC at http://review.gluster.org/13473 proposes to 
remove the source from the glusterfs master branch. Thus it would not be in the 
upcoming 3.8 and later releases.

Any objections? Any comments? Please reply to the list 
mailto:gluster-de...@gluster.org 




___
Gluster-devel mailing list
gluster-de...@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [ovirt-users] open error -13 = sanlock

[Ravishankar N]

On 03/04/2016 07:53 AM, Alan Millar wrote:

On Mar 3, 2016, at 7:12 AM, p...@email.cz wrote:

will extend replica 2 to replica 3 ( arbiter )  ASAP .

Anyone know how to do that?  The add-brick command in 3.7.8 doesn’t let me 
specify “arbiter 1” after “replica 3”.  I thought I read that the ability to 
change to an arbiter volume doesn’t exist yet, but I can’t find where I read 
that.


You're right, it is not in yet. Planned for 3.8 though.



Is there a workaround with setting xattrs or something?

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [ovirt-users] open error -13 = sanlock

[Alan Millar]

> On Mar 3, 2016, at 7:12 AM, p...@email.cz wrote:
> 
> will extend replica 2 to replica 3 ( arbiter )  ASAP .

Anyone know how to do that?  The add-brick command in 3.7.8 doesn’t let me 
specify “arbiter 1” after “replica 3”.  I thought I read that the ability to 
change to an arbiter volume doesn’t exist yet, but I can’t find where I read 
that.

Is there a workaround with setting xattrs or something?

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set

[Atin Mukherjee]

-Atin
Sent from one plus one
On 04-Mar-2016 3:35 am, "Steve Dainard"  wrote:
>
> FYI Gluster storage node hostnames are gluster0[1-6].
>
> Full dump attached. I see a few clients not on 30706. Most notably the
two debian 7 servers (using packages from gluster.org) seem to be running
lower op versions than the centos7 machines (every other client in
10.0.231.0/24 subnet).
>
> glusterd.client1.identifier=10.0.231.10:1023 <-- debian 7, glusterfs
3.7.6 built on Feb  4 2016 06:25:19
> glusterd.client1.volname=storage
> glusterd.client1.max-op-version=30603
This is running with 3.6.3 and that's why volume set fails. Unmount this
client and upgrade and remount it back.
> glusterd.client1.min-op-version=1
>
> glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02
(mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
> glusterd.client2.volname=
> glusterd.client2.max-op-version=0
> glusterd.client2.min-op-version=0
>
> glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actual
mounts)
> glusterd.client3.volname=
> glusterd.client3.max-op-version=0
> glusterd.client3.min-op-version=0
>
> glusterd.client4.identifier=10.0.231.11:1022 <--- debian 7, glusterfs
3.7.6 built on Feb  4 2016 06:25:19
> glusterd.client4.volname=storage
> glusterd.client4.max-op-version=30603
> glusterd.client4.min-op-version=1
>
> glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actual
mounts)
> glusterd.client5.volname=
> glusterd.client5.max-op-version=0
> glusterd.client5.min-op-version=0
>
> glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04
(mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
> glusterd.client6.volname=
> glusterd.client6.max-op-version=0
> glusterd.client6.min-op-version=0
>
> glusterd.client7.identifier=10.0.231.50:65529
>
glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage
> glusterd.client7.max-op-version=30706
> glusterd.client7.min-op-version=1
>
> ...
>
> Debian package info:
> apt-cache policy glusterfs-client
> glusterfs-client:
>   Installed: 3.7.6-2
>   Candidate: 3.7.6-2
>   Version table:
>  *** 3.7.6-2 0
> 500
http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/
wheezy/main amd64 Packages
>
>
> Thanks
>
> On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg  wrote:
>>
>> Hi Steve,
>>
>> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof
glusterd)  command. it will create .dump file in /var/run/gluster/
directory. client-op-version information will be present in dump file.
>>
>> Thanks,
>> ~Gaurav
>>
>> - Original Message -
>> From: "Steve Dainard" 
>> To: "Gaurav Garg" 
>> Cc: "gluster-users@gluster.org List" 
>> Sent: Thursday, March 3, 2016 12:07:25 AM
>> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or
more connected clients cannot support the feature being set
>>
>> From the the client side logs I can see version info on mount:
>>
>> Final graph:
>>
+--+
>>   1: volume storage-client-0
>>   2: type protocol/client
>>   3: option clnt-lk-version 1
>>   4: option volfile-checksum 0
>>   5: option volfile-key /storage
>>   6: option client-version 3.7.6
>>   7: option process-uuid
>>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0
>>   8: option fops-version 1298437
>>   9: option ping-timeout 42
>>  10: option remote-host 10.0.231.50
>>  11: option remote-subvolume /mnt/raid6-storage/storage
>>  12: option transport-type socket
>>  13: option send-gids true
>>  14: end-volume
>>  15:
>>  16: volume storage-client-1
>>  17: type protocol/client
>>  18: option clnt-lk-version 1
>>  19: option volfile-checksum 0
>>  20: option volfile-key /storage
>>  21: option client-version 3.7.6
>>  22: option process-uuid
>>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0
>>  23: option fops-version 1298437
>>  24: option ping-timeout 42
>>  25: option remote-host 10.0.231.51
>>  26: option remote-subvolume /mnt/raid6-storage/storage
>>  27: option transport-type socket
>>  28: option send-gids true
>>  29: end-volume
>>  30:
>>  31: volume storage-client-2
>>  32: type protocol/client
>>  33: option clnt-lk-version 1
>>  34: option volfile-checksum 0
>>  35: option volfile-key /storage
>>  36: option client-version 3.7.6
>>  37: option process-uuid
>>
template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0
>>  38: option fops-version 1298437
>>  39: option ping-timeout 42
>>  40: option remote-host 10.0.231.52
>>  41: option remote-subvolume /mnt/raid6-storage/storage
>>  42:

Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more connected clients cannot support the feature being set

[Steve Dainard]

FYI Gluster storage node hostnames are gluster0[1-6].

Full dump attached. I see a few clients not on 30706. Most notably the two
debian 7 servers (using packages from gluster.org) seem to be running lower
op versions than the centos7 machines (every other client in 10.0.231.0/24
subnet).

glusterd.client1.identifier=10.0.231.10:1023 <-- *debian 7*, glusterfs
3.7.6 built on Feb  4 2016 06:25:19
glusterd.client1.volname=storage
glusterd.client1.max-op-version=30603
glusterd.client1.min-op-version=1

glusterd.client2.identifier=10.0.231.51:65515 <-- gluster02
(mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
glusterd.client2.volname=
glusterd.client2.max-op-version=0
glusterd.client2.min-op-version=0

glusterd.client3.identifier=10.0.231.54:65521 <-- gluster05 (no actual
mounts)
glusterd.client3.volname=
glusterd.client3.max-op-version=0
glusterd.client3.min-op-version=0

glusterd.client4.identifier=10.0.231.11:1022 <---* debian 7*, glusterfs
3.7.6 built on Feb  4 2016 06:25:19
glusterd.client4.volname=storage
glusterd.client4.max-op-version=30603
glusterd.client4.min-op-version=1

glusterd.client5.identifier=10.0.231.55:65530 <-- gluster06 (no actual
mounts)
glusterd.client5.volname=
glusterd.client5.max-op-version=0
glusterd.client5.min-op-version=0

glusterd.client6.identifier=10.0.231.53:65516 <-- gluster04
(mounted localhost:storage on /run/gluster/storage type fuse.glusterfs)
glusterd.client6.volname=
glusterd.client6.max-op-version=0
glusterd.client6.min-op-version=0

glusterd.client7.identifier=10.0.231.50:65529
glusterd.client7.volname=export-domain-storage.10.0.231.50.mnt-lv-export-domain-storage-export-domain-storage
glusterd.client7.max-op-version=30706
glusterd.client7.min-op-version=1

...

Debian package info:
apt-cache policy glusterfs-client
glusterfs-client:
  Installed: 3.7.6-2
  Candidate: 3.7.6-2
  Version table:
 *** 3.7.6-2 0
500
http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.6/Debian/wheezy/apt/
wheezy/main amd64 Packages


Thanks

On Wed, Mar 2, 2016 at 10:29 PM, Gaurav Garg  wrote:

> Hi Steve,
>
> As atin pointed out to take statedump by running #kill -SIGUSR1 $(pidof
> glusterd)  command. it will create .dump file in /var/run/gluster/
> directory. client-op-version information will be present in dump file.
>
> Thanks,
> ~Gaurav
>
> - Original Message -
> From: "Steve Dainard" 
> To: "Gaurav Garg" 
> Cc: "gluster-users@gluster.org List" 
> Sent: Thursday, March 3, 2016 12:07:25 AM
> Subject: Re: [Gluster-users] gluster 3.7.6 volume set: failed: One or more
> connected clients cannot support the feature being set
>
> From the the client side logs I can see version info on mount:
>
> Final graph:
>
> +--+
>   1: volume storage-client-0
>   2: type protocol/client
>   3: option clnt-lk-version 1
>   4: option volfile-checksum 0
>   5: option volfile-key /storage
>   6: option client-version 3.7.6
>   7: option process-uuid
>
> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-0-0-0
>   8: option fops-version 1298437
>   9: option ping-timeout 42
>  10: option remote-host 10.0.231.50
>  11: option remote-subvolume /mnt/raid6-storage/storage
>  12: option transport-type socket
>  13: option send-gids true
>  14: end-volume
>  15:
>  16: volume storage-client-1
>  17: type protocol/client
>  18: option clnt-lk-version 1
>  19: option volfile-checksum 0
>  20: option volfile-key /storage
>  21: option client-version 3.7.6
>  22: option process-uuid
>
> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-1-0-0
>  23: option fops-version 1298437
>  24: option ping-timeout 42
>  25: option remote-host 10.0.231.51
>  26: option remote-subvolume /mnt/raid6-storage/storage
>  27: option transport-type socket
>  28: option send-gids true
>  29: end-volume
>  30:
>  31: volume storage-client-2
>  32: type protocol/client
>  33: option clnt-lk-version 1
>  34: option volfile-checksum 0
>  35: option volfile-key /storage
>  36: option client-version 3.7.6
>  37: option process-uuid
>
> template-centos7-compute.compute.domain-2773-2016/03/02-18:28:34:328100-storage-client-2-0-0
>  38: option fops-version 1298437
>  39: option ping-timeout 42
>  40: option remote-host 10.0.231.52
>  41: option remote-subvolume /mnt/raid6-storage/storage
>  42: option transport-type socket
>  43: option send-gids true
>  44: end-volume
>  45:
>  46: volume storage-client-3
>  47: type protocol/client
>  48: option clnt-lk-version 1
>  49: option volfile-checksum 0
>  50: option volfile-key /storage
>  51: option client-version 3.7.6
>  52: option process-uuid
>
> 

Re: [Gluster-users] Moved files from one directory to another, now gone

[Andrus, Brian Contractor]

Pranith,

Sorry for taking so long. Work required my time.

So I can look at this a little more right now.
I am using glusterfs-3.7.6-1
Here is some pertinent info:
]# gluster volume info DATA

Volume Name: DATA
Type: Disperse
Volume ID: 8374314b-796a-48d0-acf7-dea51fdd83ae
Status: Started
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: server45:/zpool1/DATABRICK
Brick2: server46:/zpool1/DATABRICK
Brick3: server47:/zpool1/DATABRICK
Options Reconfigured:
performance.io-cache: off
performance.quick-read: off
performance.readdir-ahead: disable

# df -h /zpool1 /DATA
Filesystem  Size  Used Avail Use% Mounted on
zpool1  7.2T  1.2T  6.0T  17% /zpool1
server45:/DATA   15T  2.5T   12T  18% /DATA

# du -hsc /DATA/
232G/DATA/
232Gtotal

I have been cleaning up some of the stuff, but there are 2 directories I cannot 
remove:
# ls /DATA/
FINISHED_LOGS  home  JOBS  LOGS  m4p2  OLDLOGS

# ls /zpool1/DATABRICK/
FINISHED_LOGS/ .glusterfs/home/  JOBS/  LOGS/  
m4p2/  OLDLOGS/   .trashcan/

# ls /zpool1/DATABRICK/
FINISHED_LOGS  home  JOBS  LOGS  m4p2  OLDLOGS

# rm -rf /DATA/OLDLOGS/
# rm -rf /DATA/FINISHED_LOGS/
# ls /DATA/
FINISHED_LOGS  home  JOBS  LOGS  m4p2  OLDLOGS

Those two directories are empty on the glusterfs mount, but still have files 
when I look on the brick.

Strange bug. I will be just reformatting the thing when some of the work 
completes, but I can run some tests and provide info if you want to try and 
track it down.

Brian Andrus



From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Thursday, January 28, 2016 10:36 PM
To: Andrus, Brian Contractor ; gluster-users@gluster.org
Subject: Re: [Gluster-users] Moved files from one directory to another, now gone


On 01/29/2016 12:50 AM, Andrus, Brian Contractor wrote:
All,

I have a glusterfs setup with a disperse volume over 3 zfs bricks, one on each 
node.

I just did a 'mv' of some log files from one directory to another and when I 
look in the directory, they are not there at all!
Neither is any of the data I used to have. It is completely empty. I try doing 
a 'touch' of a file and nothing shows up.

I do see the files on the zpool mount, but they seem to be partially there. 
They are all just text files, but they look incomplete.
There seems to be many errors in the log:

[2016-01-28 18:57:41.353265] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 
0-DATA-disperse-0: Heal failed [Transport endpoint is not connected]

Any ideas on what this is? Other directories seem ok.

Could you give me the following information:
1) Output of: gluster volume info
2) Log files of the setup? Both client and brick logs, they should be present 
in /var/log/glusterfs
3) What is the version of glusterfs you are using?

Pranith


Thanks in advance,
Brian Andrus





___

Gluster-users mailing list

Gluster-users@gluster.org

http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] about tail command

[Alan Millar]

>>  1. the command "gluster v replace-brick " is async or sync?  The 
> replace is
>>  complete when the command quit ?
> It is a sync command, replacing the brick finishes as the command returns.


Hmm, that has not been my experience with 3.7.6 and 3.7.8.   Perhaps there is a 
question of semantics here or definition of what the command precisely does.

What I found is that the command returns fairly quickly, in a few seconds.  In 
that amount of time, the old brick is removed from the volume configuration and 
the new brick is added to the volume configuration.  As soon as the 
replace-brick command is done, the "volume info" will show the new 
configuration with the old brick gone and the new brick included.  So in the 
sense of volume configuration, it is complete.

But the data is not moved or healed at this point; that is only starting.  The 
heal process will then proceed separately after the "replace-brick" command.


I certainly think of the overall brick replacement process as including the 
full replication of data to the new brick, even if the "replace-brick" command 
does not do that.  I imagine other people might think the same way also.  A new 
empty brick isn't protecting your replicated data, so it is an incomplete 
replacement.  


Older documentation certainly refers to "replace brick start".  I couldn't find 
any 3.7 documentation that explained why that was gone, and why "commit force" 
was the only option available now.  I just got errors at the command line 
trying to do "start".  I think it would help if the new documentation was a 
little clearer about this, and how to look at heal info to find out when your 
brick replacement is fully finished.

That's my opinion :-)


- Alan
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] gluster 3.8 release date

[Dayal D Dilli]

Hey Guys,

I see from
https://www.gluster.org/pipermail/gluster-users/2016-January/024828.html
thread that gluster 3.8 is planned to be released in the end of May or
early June.

Do we have any specific/close date for the release? Also wondering if we
are still in the schedule mentioned in the thread?

Thanks!
Dayal.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [ovirt-users] open error -13 = sanlock

[p...@email.cz]

OK,
will extend replica 2 to replica 3 ( arbiter )  ASAP .

If is deleted "untouching" ids file on brick , healing of this file 
doesn't work .


regs.Pa.

On 3.3.2016 12:19, Nir Soffer wrote:
On Thu, Mar 3, 2016 at 11:23 AM, p...@email.cz  
> wrote:


This is replica 2, only , with following settings


Replica 2 is not supported. Even if you "fix" this now, you will have 
the same issue

soon.


Options Reconfigured:
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: fixed
cluster.server-quorum-type: none
storage.owner-uid: 36
storage.owner-gid: 36
cluster.quorum-count: 1
cluster.self-heal-daemon: enable

If I'll create "ids" file manually (  eg. " sanlock direct init -s

3c34ad63-6c66-4e23-ab46-084f3d70b147:0:/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids:0
" ) on both bricks,
vdsm is writing only to half of them ( that with 2 links = correct )
"ids" file has correct permittions, owner, size  on both bricks.
brick 1:  -rw-rw 1 vdsm kvm 1048576  2. bře 18.56
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
- not updated
brick 2:  -rw-rw 2 vdsm kvm 1048576  3. bře 10.16
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
- is continually updated

What happens when I'll restart vdsm ? Will oVirt storages go to
"disable " state ??? = disconnect VMs storages ?


 Nothing will happen, the vms will continue to run normally.

On block storage, stopping vdsm will prevent automatic extending of vm 
disks
when the disk become too full, but on file based storage (like 
gluster) there is no issue.



regs.Pa.


On 3.3.2016 02:02, Ravishankar N wrote:

On 03/03/2016 12:43 AM, Nir Soffer wrote:


PS:  # find /STORAGES -samefile
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
-print
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
= missing "shadowfile" in " .gluster " dir.
How can I fix it ?? - online !


Ravi?

Is this the case in all 3 bricks of the replica?
BTW, you can just stat the file on the brick and see the link
count (it must be 2) instead of running the more expensive find
command.






___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterfs NFS performance issue

[Pavel Riha]

Dne 03/03/16 08:10, Jiffin Tony Thottan napsal(a):

On 29/02/16 15:25, Pavel Riha wrote:

Hi all,

I have read some recent post about performance issues, complaining
about the fuse driver and recomended NFS..

although my final goal is replicate volume, I'm now just doing some
test for reference.

my basic benchmark is
dd if=/dev/zero of=ddtest bs=1M count=1000 conv=fsync

running directly on server xfs partition give me 120-130 MB/s .. OK

on client NFS mount using _KERNEL_ nfs server give me 100-112MB/s .. OK

but a pure "nfs replacement" gluster config (one brick distribute)
give me sometimes 112 and sometimes only 64 !!!  any idea why so
unstable?
I was watching top,iotop and iftop .. and no idea, the resources were
free

but the real problem comes with my second benchmark, untar the kernel
sources
tar xjf /usr/portage/distfiles/linux-3.2.tar.bz2

on server localy  .. 16sec
kernel nfs mount  .. 1:30
gluster nfs mount .. 23min !!! what???

is this the small files issue? so much? even on nfs?


my last quick test with gluster native (fuse) mount, give me
46MB/s for dd (really slow for non replicate) and
3:58 for the sources untar (not the best, but much much better than
gluster nfs)


I have done no tunning at all yet. But I thought I will get much
better numbers for the one brick distribute. And expected tuning with
the replica..


server is CPU E5-1650 v3 @ 3.50GHz with 4GB RAM, 7200rpm hdd
client is old i5 661 @ 3.33GHz, 4GB RAM
gigabit ethernet
glusterfs-3.7.4
XFS filesystem on brick partition



Can u provide more details about volume configuration (gluster vol info) ?


volume created just with
gluster volume create gtest 1u:/mnt/data/brick/
and run the benchmarks without any tunnig..
/*
if I remember correctly, even I haven't set anything, the gluster info shows
Options Reconfigured:
performance.readdir-ahead: on
*/


then I tried some options, someone metionet here in last posts, a then back.
(without noticable change yet)

current volume info is:

Volume Name: gtest
Type: Distribute
Volume ID: 95cd728e-0b62-4e75-bf18-a0168ec2d0c8
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 1u:/mnt/data/brick
Options Reconfigured:
performance.nfs.write-behind: On
performance.write-behind: On
performance.readdir-ahead: on
performance.nfs.io-cache: off



Pavel



--
Býváme v lásce nespravedliví, protože toho druhého pokládáme za dokonalého.
-- Jean Paul
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Per-client prefered server?

[Yannick Perret]

Le 03/03/2016 15:07, Krutika Dhananjay a écrit :

Ok, and what version of glusterfs are you using?
At this time I'm using glusterfs 3.6.7 on Debian 8 (64bit), same OS on 
servers and clients.


--
Y.


-Krutika


*From: *"Yannick Perret" 
*To: *"Krutika Dhananjay" 
*Cc: *gluster-users@gluster.org
*Sent: *Thursday, March 3, 2016 7:23:09 PM
*Subject: *Re: [Gluster-users] Per-client prefered server?

Le 03/03/2016 13:18, Krutika Dhananjay a écrit :

What does "nearest" storage server mean? Are the clients
residing in the storage pool too? Or are they external to the
cluster?

They are external.
Nearest means that they are in the same building, linked by the
same switch. Machines in the other building are rooted via some
more equipments.

--
Y.

-Krutika


*From: *"Yannick Perret" 
*To: *gluster-users@gluster.org
*Sent: *Thursday, March 3, 2016 5:38:33 PM
*Subject: *[Gluster-users] Per-client prefered server?

Hello,

I can't find if it is possible to set a prefered server on
a per-client
basis for replica volumes, so I ask the question here.

The context: we have 2 storage servers, each in one
building. We also
have several virtual machines on each building, and they
can migrate
from one building to an other (depending on load,
maintenance…).

So (for testing at this time) I setup a x2 replica volume,
one replica
on each storage server of course. As most of our volumes
are "many reads
- few writes" it would be better for bandwidth that each
client uses the
"nearest" storage server (local building switch) - for
reading, of
course. The 2 buildings have a good netlink but we prefer
to minimize -
when not needed - data transferts beetween them (this link
is shared).

Can you see a solution for this kind of tuning? As far as
I understand
geo-replica is not really what I need, no?

It exists "cluster.read-subvolume" option of course but we
can have
clients on both building so a per-volume option is not
what we need. An
per-client equivalent of this option should be nice.

I tested by myself a small patch to perform this - and it
seems to work
fine as far as I can see - but 1. before continuing in
this way I would
first check if it exists an other way and 2. I'm not
familiar with the
whole code so I'm not sure that my tests are in the
"state-of-the-art"
for glusterfs.

Thanks in advance for any help.

Regards,
--
Y.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users








smime.p7s
Description: Signature cryptographique S/MIME
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Per-client prefered server?

[Saravanakumar Arumugam]

On 03/03/2016 05:38 PM, Yannick Perret wrote:

Hello,

I can't find if it is possible to set a prefered server on a 
per-client basis for replica volumes, so I ask the question here.


The context: we have 2 storage servers, each in one building. We also 
have several virtual machines on each building, and they can migrate 
from one building to an other (depending on load, maintenance…).


So (for testing at this time) I setup a x2 replica volume, one replica 
on each storage server of course. As most of our volumes are "many 
reads - few writes" it would be better for bandwidth that each client 
uses the "nearest" storage server (local building switch) - for 
reading, of course. The 2 buildings have a good netlink but we prefer 
to minimize - when not needed - data transferts beetween them (this 
link is shared).


Can you see a solution for this kind of tuning? As far as I understand 
geo-replica is not really what I need, no?


Yes, geo-replication "cannot" be used as you wish to carry out "write" 
operation on Slave side.




It exists "cluster.read-subvolume" option of course but we can have 
clients on both building so a per-volume option is not what we need. 
An per-client equivalent of this option should be nice.


I tested by myself a small patch to perform this - and it seems to 
work fine as far as I can see - but 1. before continuing in this way I 
would first check if it exists an other way and 2. I'm not familiar 
with the whole code so I'm not sure that my tests are in the 
"state-of-the-art" for glusterfs.


maybe you should share that interesting patch :) and get better feedback 
about your test case.



Thanks in advance for any help.

Regards,
--
Y.




___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Per-client prefered server?

[Krutika Dhananjay]

Ok, and what version of glusterfs are you using? 

-Krutika 
- Original Message -

> From: "Yannick Perret" 
> To: "Krutika Dhananjay" 
> Cc: gluster-users@gluster.org
> Sent: Thursday, March 3, 2016 7:23:09 PM
> Subject: Re: [Gluster-users] Per-client prefered server?

> Le 03/03/2016 13:18, Krutika Dhananjay a écrit :

> > What does "nearest" storage server mean? Are the clients residing in the
> > storage pool too? Or are they external to the cluster?
> 

> They are external.
> Nearest means that they are in the same building, linked by the same switch.
> Machines in the other building are rooted via some more equipments.

> --
> Y.

> > -Krutika
> 
> > - Original Message -
> 

> > > From: "Yannick Perret" 
> > 
> 
> > > To: gluster-users@gluster.org
> > 
> 
> > > Sent: Thursday, March 3, 2016 5:38:33 PM
> > 
> 
> > > Subject: [Gluster-users] Per-client prefered server?
> > 
> 

> > > Hello,
> > 
> 

> > > I can't find if it is possible to set a prefered server on a per-client
> > 
> 
> > > basis for replica volumes, so I ask the question here.
> > 
> 

> > > The context: we have 2 storage servers, each in one building. We also
> > 
> 
> > > have several virtual machines on each building, and they can migrate
> > 
> 
> > > from one building to an other (depending on load, maintenance…).
> > 
> 

> > > So (for testing at this time) I setup a x2 replica volume, one replica
> > 
> 
> > > on each storage server of course. As most of our volumes are "many reads
> > 
> 
> > > - few writes" it would be better for bandwidth that each client uses the
> > 
> 
> > > "nearest" storage server (local building switch) - for reading, of
> > 
> 
> > > course. The 2 buildings have a good netlink but we prefer to minimize -
> > 
> 
> > > when not needed - data transferts beetween them (this link is shared).
> > 
> 

> > > Can you see a solution for this kind of tuning? As far as I understand
> > 
> 
> > > geo-replica is not really what I need, no?
> > 
> 

> > > It exists "cluster.read-subvolume" option of course but we can have
> > 
> 
> > > clients on both building so a per-volume option is not what we need. An
> > 
> 
> > > per-client equivalent of this option should be nice.
> > 
> 

> > > I tested by myself a small patch to perform this - and it seems to work
> > 
> 
> > > fine as far as I can see - but 1. before continuing in this way I would
> > 
> 
> > > first check if it exists an other way and 2. I'm not familiar with the
> > 
> 
> > > whole code so I'm not sure that my tests are in the "state-of-the-art"
> > 
> 
> > > for glusterfs.
> > 
> 

> > > Thanks in advance for any help.
> > 
> 

> > > Regards,
> > 
> 
> > > --
> > 
> 
> > > Y.
> > 
> 

> > > ___
> > 
> 
> > > Gluster-users mailing list
> > 
> 
> > > Gluster-users@gluster.org
> > 
> 
> > > http://www.gluster.org/mailman/listinfo/gluster-users
> > 
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Per-client prefered server?

[Yannick Perret]

Le 03/03/2016 13:18, Krutika Dhananjay a écrit :
What does "nearest" storage server mean? Are the clients residing in 
the storage pool too? Or are they external to the cluster?



They are external.
Nearest means that they are in the same building, linked by the same 
switch. Machines in the other building are rooted via some more equipments.


--
Y.

-Krutika


*From: *"Yannick Perret" 
*To: *gluster-users@gluster.org
*Sent: *Thursday, March 3, 2016 5:38:33 PM
*Subject: *[Gluster-users] Per-client prefered server?

Hello,

I can't find if it is possible to set a prefered server on a
per-client
basis for replica volumes, so I ask the question here.

The context: we have 2 storage servers, each in one building. We also
have several virtual machines on each building, and they can migrate
from one building to an other (depending on load, maintenance…).

So (for testing at this time) I setup a x2 replica volume, one
replica
on each storage server of course. As most of our volumes are "many
reads
- few writes" it would be better for bandwidth that each client
uses the
"nearest" storage server (local building switch) - for reading, of
course. The 2 buildings have a good netlink but we prefer to
minimize -
when not needed - data transferts beetween them (this link is shared).

Can you see a solution for this kind of tuning? As far as I
understand
geo-replica is not really what I need, no?

It exists "cluster.read-subvolume" option of course but we can have
clients on both building so a per-volume option is not what we
need. An
per-client equivalent of this option should be nice.

I tested by myself a small patch to perform this - and it seems to
work
fine as far as I can see - but 1. before continuing in this way I
would
first check if it exists an other way and 2. I'm not familiar with
the
whole code so I'm not sure that my tests are in the
"state-of-the-art"
for glusterfs.

Thanks in advance for any help.

Regards,
--
Y.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users






smime.p7s
Description: Signature cryptographique S/MIME
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Per-client prefered server?

[Krutika Dhananjay]

What does "nearest" storage server mean? Are the clients residing in the 
storage pool too? Or are they external to the cluster? 

-Krutika 
- Original Message -

> From: "Yannick Perret" 
> To: gluster-users@gluster.org
> Sent: Thursday, March 3, 2016 5:38:33 PM
> Subject: [Gluster-users] Per-client prefered server?

> Hello,

> I can't find if it is possible to set a prefered server on a per-client
> basis for replica volumes, so I ask the question here.

> The context: we have 2 storage servers, each in one building. We also
> have several virtual machines on each building, and they can migrate
> from one building to an other (depending on load, maintenance…).

> So (for testing at this time) I setup a x2 replica volume, one replica
> on each storage server of course. As most of our volumes are "many reads
> - few writes" it would be better for bandwidth that each client uses the
> "nearest" storage server (local building switch) - for reading, of
> course. The 2 buildings have a good netlink but we prefer to minimize -
> when not needed - data transferts beetween them (this link is shared).

> Can you see a solution for this kind of tuning? As far as I understand
> geo-replica is not really what I need, no?

> It exists "cluster.read-subvolume" option of course but we can have
> clients on both building so a per-volume option is not what we need. An
> per-client equivalent of this option should be nice.

> I tested by myself a small patch to perform this - and it seems to work
> fine as far as I can see - but 1. before continuing in this way I would
> first check if it exists an other way and 2. I'm not familiar with the
> whole code so I'm not sure that my tests are in the "state-of-the-art"
> for glusterfs.

> Thanks in advance for any help.

> Regards,
> --
> Y.

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Per-client prefered server?

[Yannick Perret]

Hello,

I can't find if it is possible to set a prefered server on a per-client 
basis for replica volumes, so I ask the question here.


The context: we have 2 storage servers, each in one building. We also 
have several virtual machines on each building, and they can migrate 
from one building to an other (depending on load, maintenance…).


So (for testing at this time) I setup a x2 replica volume, one replica 
on each storage server of course. As most of our volumes are "many reads 
- few writes" it would be better for bandwidth that each client uses the 
"nearest" storage server (local building switch) - for reading, of 
course. The 2 buildings have a good netlink but we prefer to minimize - 
when not needed - data transferts beetween them (this link is shared).


Can you see a solution for this kind of tuning? As far as I understand 
geo-replica is not really what I need, no?


It exists "cluster.read-subvolume" option of course but we can have 
clients on both building so a per-volume option is not what we need. An 
per-client equivalent of this option should be nice.


I tested by myself a small patch to perform this - and it seems to work 
fine as far as I can see - but 1. before continuing in this way I would 
first check if it exists an other way and 2. I'm not familiar with the 
whole code so I'm not sure that my tests are in the "state-of-the-art" 
for glusterfs.


Thanks in advance for any help.

Regards,
--
Y.




smime.p7s
Description: Signature cryptographique S/MIME
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain

[Anuradha Talur]

- Original Message -
> From: "ABHISHEK PALIWAL" 
> To: gluster-users@gluster.org, gluster-de...@gluster.org
> Sent: Thursday, March 3, 2016 12:10:42 PM
> Subject: [Gluster-users] gluster volume heal info split brain command not 
> showing files in split-brain
> 
> 
> Hello,
> 
> In gluster, we use the command "gluster volume heal c_glusterfs info
> split-brain" to find the files that are in split-brain scenario.
> We run heal script (developed by Windriver prime team) on the files reported
> by above command to resolve split-brain issue.
> 
> But we observed that the above command is not showing all files that are in
> split-brain,
> even though split brain scenario actually exists on the node.
> 
> Now a days this issue is seen more often and IO errors are reported when
> tried to access these files under split-brain.
> 
> Can you please check why this gluster command is not showing files under
> split-brain?
> We can provide you required logs and support to resolve this issue.
Hi,

Could you paste the output of getfattr -m. -de hex 
 from all the bricks that the files lie in?

> 
> Please reply on this because I am not getting any reply from the community.
> 
> --
> 
> Regards
> Abhishek Paliwal
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterfs NFS performance issue

[Ronny Adsetts]

Pavel Riha wrote on 29/02/2016 09:55:
> 
> I have read some recent post about performance issues, complaining
> about the fuse driver and recomended NFS..
> 
> although my final goal is replicate volume, I'm now just doing some
> test for reference.
> 
> my basic benchmark is dd if=/dev/zero of=ddtest bs=1M count=1000
> conv=fsync
> 
> running directly on server xfs partition give me 120-130 MB/s .. OK
> 
> on client NFS mount using _KERNEL_ nfs server give me 100-112MB/s ..
> OK
> 
> but a pure "nfs replacement" gluster config (one brick distribute)
> give me sometimes 112 and sometimes only 64 !!!  any idea why so
> unstable? I was watching top,iotop and iftop .. and no idea, the
> resources were free
> 
> but the real problem comes with my second benchmark, untar the kernel
> sources tar xjf /usr/portage/distfiles/linux-3.2.tar.bz2
> 
> on server localy  .. 16sec kernel nfs mount  .. 1:30 gluster nfs
> mount .. 23min !!! what???
> 
> is this the small files issue? so much? even on nfs?
> 
> my last quick test with gluster native (fuse) mount, give me 46MB/s
> for dd (really slow for non replicate) and 3:58 for the sources untar
> (not the best, but much much better than gluster nfs)
> 
> I have done no tunning at all yet. But I thought I will get much
> better numbers for the one brick distribute. And expected tuning with
> the replica..
> 
> server is CPU E5-1650 v3 @ 3.50GHz with 4GB RAM, 7200rpm hdd client
> is old i5 661 @ 3.33GHz, 4GB RAM gigabit ethernet glusterfs-3.7.4 XFS
> filesystem on brick partition

Just a quick "me too". Not sure what if 

Using a 2-brick replica over 2x bonded (round robin) 1Gb ethernet. Mount is on 
one of the brick nodes.

A kernel un-tar seems to really stress Gluster.

Extraction to Gluster NFS mount:

# time tar xf /tmp/linux-4.4.tar

real 13m46.308s
user 0m1.224s
sys 0m19.608s

Extraction to Gluster FUSE mount:

# time tar xf /tmp/linux-4.4.tar

real 9m10.413s
user 0m1.964s
sys 0m16.740s

Extraction to local hardware-RAID backed FS (where the bricks are):

# time tar xf /tmp/linux-4.4.tar

real 0m2.771s
user 0m0.284s
sys 0m2.396s

Ronny

-- 
Ronny Adsetts
Technical Director
Amazing Internet Ltd, London
t: +44 20 8977 8943
w: www.amazinginternet.com

Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ
Registered in England. Company No. 4042957




signature.asc
Description: OpenPGP digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Query on healing process

[ABHISHEK PALIWAL]

On Thu, Mar 3, 2016 at 4:10 PM, Ravishankar N 
wrote:

> Hi,
>
> On 03/03/2016 11:14 AM, ABHISHEK PALIWAL wrote:
>
> Hi Ravi,
>
> As I discussed earlier this issue, I investigated this issue and find that
> healing is not triggered because the "gluster volume heal c_glusterfs info
> split-brain" command not showing any entries as a outcome of this command
> even though the file in split brain case.
>
>
> Couple of observations from the 'commands_output' file.
>
> getfattr -d -m . -e hex
> opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
> The afr xattrs do not indicate that the file is in split brain:
> # file:
> opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
> trusted.afr.c_glusterfs-client-1=0x
> trusted.afr.dirty=0x
> trusted.bit-rot.version=0x000b56d6dd1d000ec7a9
> trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae
>
>
>
> getfattr -d -m . -e hex
> opt/lvmdir/c2/brick/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
> trusted.afr.c_glusterfs-client-0=0x0008
> trusted.afr.c_glusterfs-client-2=0x0002
> trusted.afr.c_glusterfs-client-4=0x0002
> trusted.afr.c_glusterfs-client-6=0x0002
> trusted.afr.dirty=0x
> trusted.bit-rot.version=0x000b56d6dcb7000c87e7
> trusted.gfid=0x9f5e354ecfda40149ddce7d5ffe760ae
>
> 1. There doesn't seem to be a split-brain going by the trusted.afr* xattrs.
>

if it is not the split brain problem then how can I resolve this.


> 2. You seem to have re-used the bricks from another volume/setup. For
> replica 2, only trusted.afr.c_glusterfs-client-0 and
> trusted.afr.c_glusterfs-client-1 must be present but I see 4 xattrs -
> client-0,2,4 and 6
>

could you please suggest why these entries are there because I am not able
to find out scenario. I am rebooting the one board multiple times to
reproduce the issue and after every reboot doing the remove-brick and
add-brick on the same volume for the second board.


> 3. On the rebooted node, do you have ssl enabled by any chance? There is a
> bug for "Not able to fetch volfile' when ssl is enabled:
> https://bugzilla.redhat.com/show_bug.cgi?id=1258931
>
> Btw, you for data and metadata split-brains you can use the gluster CLI
> https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md
> instead of modifying the file from the back end.
>

But you are saying it is not split brain problem and even the split-brain
command  is not showing any file so how can I find the bigger file in size.
Also in my case the file size is fix 2MB it is overwritten every time.

>
> -Ravi
>
>
> So, what I have done I manually deleted the gfid entry of that file from
> .glusterfs directory and follow the instruction mentioned in the following
> link to do heal
>
>
> https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
>
> and this works fine for me.
>
> But my question is why the split-brain command not showing any file in
> output.
>
> Here I am attaching all the log which I get from the node for you and also
> the output of commands from both of the boards
>
> In this tar file two directories are present
>
> 000300 - log for the board which is running continuously
> 002500-  log for the board which is rebooted
>
> I am waiting for your reply please help me out on this issue.
>
> Thanks in advanced.
>
> Regards,
> Abhishek
>
> On Fri, Feb 26, 2016 at 1:21 PM, ABHISHEK PALIWAL <
> abhishpali...@gmail.com> wrote:
>
>> On Fri, Feb 26, 2016 at 10:28 AM, Ravishankar N <
>> ravishan...@redhat.com> wrote:
>>
>>> On 02/26/2016 10:10 AM, ABHISHEK PALIWAL wrote:
>>>
>>> Yes correct
>>>
>>>
>>> Okay, so when you say the files are not in sync until some time, are you
>>> getting stale data when accessing from the mount?
>>> I'm not able to figure out why heal info shows zero when the files are
>>> not in sync, despite all IO happening from the mounts. Could you provide
>>> the output of getfattr -d -m . -e hex /brick/file-name from both bricks
>>> when you hit this issue?
>>>
>>> I'll provide the logs once I get. here delay means we are powering on
>>> the second board after the 10 minutes.
>>>
>>>
>>> On Feb 26, 2016 9:57 AM, "Ravishankar N" < 
>>> ravishan...@redhat.com> wrote:
>>>
 Hello,

 On 02/26/2016 08:29 AM, ABHISHEK PALIWAL wrote:

 Hi Ravi,

 Thanks for the response.

 We are using Glugsterfs-3.7.8

 Here is the use case:

 We have a logging file which saves logs of the events for every board
 of a node and these files are in sync using glusterfs. System in replica 2
 mode it means When one brick in a replicated volume goes offline, the
 glusterd daemons on the other nodes keep track of all the files that are
 

Re: [Gluster-users] [ovirt-users] open error -13 = sanlock

[Ravishankar N]

On 03/03/2016 02:53 PM, p...@email.cz wrote:

This is replica 2, only , with following settings

Options Reconfigured:
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: fixed

Not sure why you have set this option.
Ideally replica 3 or arbiter volumes are recommended for gluster+ovirt 
use.  (client) quorum does not make sense for a 2 node setup. I have a 
detailed write up here which explains things 
http://gluster.readthedocs.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/ 
which explains things.



cluster.server-quorum-type: none
storage.owner-uid: 36
storage.owner-gid: 36
cluster.quorum-count: 1
cluster.self-heal-daemon: enable

If I'll create "ids" file manually (  eg. " sanlock direct init -s 
3c34ad63-6c66-4e23-ab46-084f3d70b147:0:/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids:0 
" ) on both bricks,

vdsm is writing only to half of them ( that with 2 links = correct )
"ids" file has correct permittions, owner, size  on both bricks.
brick 1:  -rw-rw 1 vdsm kvm 1048576  2. bře 18.56 
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - 
not updated


Okay, so this one has link count =1 which means the .glusterfs hardlink 
is missing.  Can you try deleting this file from the brick and perform a 
stat on the file from the mount? That should heal (i.e recreate it ) on 
this brick from the other brick with the appropriate .glusterfs hard link.



brick 2:  -rw-rw 2 vdsm kvm 1048576  3. bře 10.16 
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - 
is continually updated


What happens when I'll restart vdsm ? Will oVirt storages go to 
"disable " state ??? = disconnect VMs storages ?


No idea on this one...
-Ravi


regs.Pa.

On 3.3.2016 02:02, Ravishankar N wrote:

On 03/03/2016 12:43 AM, Nir Soffer wrote:


PS:  # find /STORAGES -samefile
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
-print
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
= missing "shadowfile" in " .gluster " dir.
How can I fix it ?? - online !


Ravi?

Is this the case in all 3 bricks of the replica?
BTW, you can just stat the file on the brick and see the link count 
(it must be 2) instead of running the more expensive find command.







___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [ovirt-users] open error -13 = sanlock

[p...@email.cz]

This is replica 2, only , with following settings

Options Reconfigured:
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: fixed
cluster.server-quorum-type: none
storage.owner-uid: 36
storage.owner-gid: 36
cluster.quorum-count: 1
cluster.self-heal-daemon: enable

If I'll create "ids" file manually (  eg. " sanlock direct init -s 
3c34ad63-6c66-4e23-ab46-084f3d70b147:0:/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids:0 
" ) on both bricks,

vdsm is writing only to half of them ( that with 2 links = correct )
"ids" file has correct permittions, owner, size  on both bricks.
brick 1:  -rw-rw 1 vdsm kvm 1048576  2. bře 18.56 
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - 
not updated
brick 2:  -rw-rw 2 vdsm kvm 1048576  3. bře 10.16 
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - 
is continually updated


What happens when I'll restart vdsm ? Will oVirt storages go to "disable 
" state ??? = disconnect VMs storages ?


regs.Pa.

On 3.3.2016 02:02, Ravishankar N wrote:

On 03/03/2016 12:43 AM, Nir Soffer wrote:


PS:  # find /STORAGES -samefile
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
-print
/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids
= missing "shadowfile" in " .gluster " dir.
How can I fix it ?? - online !


Ravi?

Is this the case in all 3 bricks of the replica?
BTW, you can just stat the file on the brick and see the link count 
(it must be 2) instead of running the more expensive find command.




___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users