Good news, the stat on all files of my volume finished after running for over 6
hours and the 4 files (actually 2 directories and 2 files) are now finally all
healed. I checked the 3 bricks and all have the correct data. On node 1 I also
saw 4 healing log entries in glustershd.log log file. I
Thank you Ravi for your answer. I have now set the afr xattr as you suggested
and I am running the "find . | xargs -d '\n' stat" on my gluster fuse mount for
this volume.
This volume has around 3 million of files and directories so it will take a
long time to finish I suppose. Do I really need
Okay so for all files and dirs, node 2 seems to be the bad copy. Try the
following:
1. On both node 1 and node3, set theafr xattr for dir10:
setfattr -n trusted.afr.myvol-pro-client-1 -v 0x00010001
‐‐‐ Original Message ‐‐‐
On Friday, November 16, 2018 5:14 AM, Ravishankar N
wrote:
> Okay, as asked in the previous mail, please share the getfattr output
> from all bricks for these 2 files. I think once we have this, we can try
> either 'adjusting' the the gfid and symlinks on node 2
On 11/15/2018 09:17 PM, mabi wrote:
‐‐‐ Original Message ‐‐‐
On Thursday, November 15, 2018 1:41 PM, Ravishankar N
wrote:
Thanks, noted. One more query. Are there files inside each of these
directories? Or is it just empty directories?
You will find below the content of each of
‐‐‐ Original Message ‐‐‐
On Thursday, November 15, 2018 1:41 PM, Ravishankar N
wrote:
> Thanks, noted. One more query. Are there files inside each of these
> directories? Or is it just empty directories?
You will find below the content of each of these 3 directories taken the brick
on
On 11/15/2018 02:11 PM, mabi wrote:
Sure, you will find below the getfattr output of all 3 directories from all 3
nodes.
Thanks, noted. One more query. Are there files inside each of these
directories? Or is it just empty directories?
2. Do you know the file (or directory) names
‐‐‐ Original Message ‐‐‐
On Thursday, November 15, 2018 5:57 AM, Ravishankar N
wrote:
> 1.Could you provide the getfattr output of the following 3 dirs from all
> 3 nodes?
> i)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10
>
On 11/14/2018 03:19 PM, mabi wrote:
‐‐‐ Original Message ‐‐‐
On Wednesday, November 14, 2018 5:34 AM, Ravishankar N
wrote:
I thought it was missing which is why I asked you to create it. The
trusted.gfid xattr for any given file or directory must be same in all 3
bricks. But it
‐‐‐ Original Message ‐‐‐
On Wednesday, November 14, 2018 5:34 AM, Ravishankar N
wrote:
> I thought it was missing which is why I asked you to create it. The
> trusted.gfid xattr for any given file or directory must be same in all 3
> bricks. But it looks like that isn't the case. Are
On 11/13/2018 01:09 PM, mabi wrote:
‐‐‐ Original Message ‐‐‐
On Friday, November 9, 2018 2:11 AM, Ravishankar N
wrote:
Please re-create the symlink on node 2 to match how it is in the other
nodes and launch heal again. Check if this is the case for other entries
too.
-Ravi
Please
‐‐‐ Original Message ‐‐‐
On Friday, November 9, 2018 2:11 AM, Ravishankar N
wrote:
> Please re-create the symlink on node 2 to match how it is in the other
> nodes and launch heal again. Check if this is the case for other entries
> too.
> -Ravi
Please ignore my previous mail, I was
‐‐‐ Original Message ‐‐‐
On Friday, November 9, 2018 2:11 AM, Ravishankar N
wrote:
> Please re-create the symlink on node 2 to match how it is in the other
> nodes and launch heal again. Check if this is the case for other entries
> too.
> -Ravi
I can't create the missing symlink on
On 11/08/2018 06:09 PM, mabi wrote:
‐‐‐ Original Message ‐‐‐
On Thursday, November 8, 2018 11:05 AM, Ravishankar N
wrote:
It is not a split-brain. Nodes 1 and 3 have xattrs indicating a pending
entry heal on node2 , so heal must have happened ideally. Can you check
a few things?
-
‐‐‐ Original Message ‐‐‐
On Thursday, November 8, 2018 11:05 AM, Ravishankar N
wrote:
> It is not a split-brain. Nodes 1 and 3 have xattrs indicating a pending
> entry heal on node2 , so heal must have happened ideally. Can you check
> a few things?
> - Is there any disconnects
On 11/08/2018 12:58 PM, mabi wrote:
Dear Ravi,
Thank you for your answer. I will start first by sending you below the getfattr
from the first entry which does not get healed (it is in fact a directory). It
is the following path/dir from the output of one of my previous mails:
Dear Ravi,
Thank you for your answer. I will start first by sending you below the getfattr
from the first entry which does not get healed (it is in fact a directory). It
is the following path/dir from the output of one of my previous mails:
Can you share the getfattr output of all 4 entries from all 3 bricks?
Also, can you tailf glustershd.log on all nodes and see if anything is
logged for these entries when you run 'gluster volume heal $volname'?
Regards,
Ravi
On 11/07/2018 01:22 PM, mabi wrote:
To my eyes this specific
To my eyes this specific case looks like a split-brain scenario but the output
of "volume info split-brain" does not show any files. Should I still use the
process for split-brain files as documented in the glusterfs documentation? or
what do you recommend here?
‐‐‐ Original Message
Ravi, I did not yet modify the cluster.data-self-heal parameter to off because
in the mean time node2 of my cluster had a memory shortage (this node has 32 GB
of RAM) and as such I had to reboot it. After that reboot all locks got
released and there are no more files left to heal on that
On 11/03/2018 04:13 PM, mabi wrote:
Ravi (or anyone else who can help), I now have even more files which are
pending for healing.
If the count is increasing, there is likely a network (disconnect)
problem between the gluster clients and the bricks that needs fixing.
Here is the output of
Ravi (or anyone else who can help), I now have even more files which are
pending for healing. Here is the output of a "volume heal info summary":
Brick node1:/data/myvol-private/brick
Status: Connected
Total Number of entries: 49845
Number of entries in heal pending: 49845
Number of entries in
Ravi, I actually restarted glustershd by unmounting my volume on the clients,
stopping and starting the volume on the cluster and re-mounting it on the
clients yesterday evening and it managed to get around 1500~ files cleared from
the "volume heal info" output. So I am down now to around ~25k
Mabi,
If bug 1637953 is what you are experiencing, then you need to follow the
workarounds mentioned in
https://lists.gluster.org/pipermail/gluster-users/2018-October/035178.html.
Can you see if this works?
-Ravi
On 11/02/2018 11:40 PM, mabi wrote:
I tried again to manually run a heal by
I tried again to manually run a heal by using the "gluster volume heal" command
because still not files have been healed and noticed the following warning in
the glusterd.log file:
[2018-11-02 18:04:19.454702] I [MSGID: 106533]
[glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume]
Is it possible the problem I encounter described below is caused by the
following bug, introduced in 4.1.5:
Bug 1637953 - data-self-heal in arbiter volume results in stale locks.
https://bugzilla.redhat.com/show_bug.cgi?id=1637953
Regards,
M.
‐‐‐ Original Message ‐‐‐
On Wednesday,
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and
currently have a volume with around 27174 files which are not being healed. The
"volume heal info" command shows the same 27k files under the first node and
the second node but there is nothing under the 3rd node
27 matches
Mail list logo
