On 01/17/2015 05:28 AM, Kyle Harris wrote:
Hello,
I created a post a few days ago named "Turning Off Self Heal Options
Don't Appear Work?" which can be found at the following link:
http://www.gluster.org/pipermail/gluster-users/2015-January/020114.html
I never got a response so I decided to set up a test in a lab
environment. I am able to reproduce the same thing so I'm hoping
someone can help me.
I have discovered over time that if a single node in a 3-node
replicated cluster with many small files is off for any length of
time, when it comes back on-line, it does a great deal of self-healing
that can cause the glusterfs and glusterfsd processes to spike on the
machines to a degree that makes them unusable. I only have one
volume, with a client mount on each server where it hosts many
websites running PHP. All is fine until the healing process goes into
overdrive.
So, I attempted to turn off self-healing by setting the following
three settings:
gluster volume set gv0 cluster.data-self-heal off
gluster volume set gv0 cluster.entry-self-heal off
gluster volume set gv0 cluster.metadata-self-heal off
hi Kyle,
Krutika wanted to send a response to you today, but we spent the
whole day debugging a bug. Let me answer some of the things we already
discussed on behalf of Krutika.
Krutika (CCed) has found one issue where even when some of the
options are turned off, self-heal was still triggered. But if all the
options are turned off I think it wouldn't do any heals from the mount
process. But glustershd can still do heals. To disable that healing, we
need to turn off self-heal-daemon using 'gluster volume set <volname>
self-heal-daemon off'
Note that I would rather not set gv0 cluster.self-heal-daemon off as
then I can't see what needs healing such that I can do it at a later
time. Those settings appear to have no affect at all.
Ah! 3.6.2 will be able to give the output of 'gluster volume heal
<volname> info' output even when self-heal-daemon is turned off.
Here is how I reproduced this in my lab:
Output from "gluster volume info gv0":
Volume Name: gv0
Type: Replicate
Volume ID: a55f8619-0789-4a1c-9cda-a903bc908fd1
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.1.116:/export/brick1
Brick2: 192.168.1.140:/export/brick1
Brick3: 192.168.1.123:/export/brick1
Options Reconfigured:
cluster.metadata-self-heal: off
cluster.entry-self-heal: off
cluster.data-self-heal: off
This was done using the latest version of gluster as of this writing,
v3.6.1 installed on CentOS 6.6 using the rpms available from the
gluster web site.
Here is how I tested:
- With all 3 nodes up, I put 4 simple text files on the cluster
- I then turned one node off
- Next I made a change to 2 of the text files
- Then I brought the previously turned off node back up
Upon doing so, I see far more than 2 of the following message in the
glusterhd.log:
[2015-01-15 23:19:30.471384] I
[afr-self-heal-entry.c:545:afr_selfheal_entry_do] 0-gv0-replicate-0:
performing entry selfheal on 00000000-0000-0000-0000-000000000001
[2015-01-15 23:19:30.494714] I
[afr-self-heal-common.c:476:afr_log_selfheal] 0-gv0-replicate-0:
Completed entry selfheal on 00000000-0000-0000-0000-000000000001.
source=0 sinks=
Questions:
- So is this a bug?
The log seems to suggest that it didn't find any 'sinks' to heal to so
it wouldn't have done any file creation/deletions. May be we should fix
the log or see if there is more to that bug.
- Why am I seeing "entry selfheal" messaages when this feature is
supposed to be turned off?
Because glustershd can still do self-heals as wel didn't disable it?
- Also, why am I seeing far more selfheal messages than 2 when I only
changed 2 files while the single node was down?
At the moment, I believe they are just log messages and not really
heals. But we will need to look further and find if there is more to it.
- Finally, how do I really turn off these selfheals that are taking
place without completely turning off the cluster.self-heal-daemon for
reasons mentioned above?
There are 2 workarounds until 3.6.2 is released for this:
1) As a workaround may be we can turn self-heal-daemon off. When we want
to see the files that need healing, we can turn it on, see the
information and turn it off immediately. This broken functionality made
it to 3.6.1 because I couldn't re-implement the feature for afrv2 in
time for the release. Sorry about that!
2) Other way to do it is to inspect the gfids of the files that need
heal directly by looking at the directory
<brick-path>/.glusterfs/indices/xattrop. This is where self-heal-daemon
looks at and finds the files that need healing.
You were saying you know a way to make machines unusable by triggering
self-heals. It would be very good if we can replicate that test in our
labs. Wondering if you have any pointers for us to do the same.
Pranith
Thank you for any insight you may be able to provide on this.
--
Kyle
_______________________________________________
Gluster-users mailing list
[email protected]
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://www.gluster.org/mailman/listinfo/gluster-users
