Am 16.10.2015 um 18:51 schrieb Vijay Bellur:
self-healing in gluster by default syncs only modified parts of the
files from a source node. Gluster does a rolling checksum of a file
needing self-heal to identify regions of the file which need to be
synced over the network. This rolling checksum computation can
sometimes be expensive and there are plans to have a lighter
self-healing in 3.8 with more granular changelogs that can do away
with the need to do a rolling checksum.
I did some tests (see below) - could you please check this and tell me
if this is normal?
For example, I have a 200GB VM disk image in my volume (the biggest
file). About 75% of that disk is currently unused space and writes are
only about 50 kbytes/sec.
That 200GB disk image /always/ "heals" a very long time (at least 30
minutes) - even if I'm pretty sure only a few blocks could have been
changed.
Anyway, I just rebooted a node (about 2-3 minutes downtime) to collect
some information:
* In total I have about 790GB* files in that Gluster volume
* about 411GB* belong to active VM HDD images, the remaining are
backup/template files
* only VM HDD images are being healed (max 15 files)
* while healing, glusterfsd shows varying CPU usages between 70% and
650% (it's a 16 cores server); total 106 minutes CPU time once
healing completed
* once healing completes, the machine received a total of 7.0 GB and
sent 3.6 GB over the internal network (so, yes, you're right that
not all contents are transferred)
* *total heal time: whopping 58 minutes*
/* these are summed up file sizes; "du" and "df" commands show smaller usage
/Node details (all 3 nodes are identical):/
/
* DELL PowerEdge R730
* Intel Xeon E5-2600 @ 2.4GHz
* 64 GB DDR4 RAM
* the server is able to gzip-compress about 1 GB data / second (all
cores together)
* 3 TB HW-RAID10 HDD (2.7TB reserved for Gluster); minimum 500 MB/s
write speed, 350 MB/s read speed
* redundant 1GBit/s internal network
* Debian 7 Wheezy / Proxmox 3.4, Kernel 2.6.32, Gluster 3.5.2
Volume setup:/
/
# gluster volume info systems
Volume Name: systems
Type: Replicate
Volume ID: b2d72784-4b0e-4f7b-b858-4ec59979a064
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: metal1:/data/gluster/systems
Brick2: metal2:/data/gluster/systems
Brick3: metal3:/data/gluster/systems
Options Reconfigured:
cluster.server-quorum-ratio: 51%
/Note that `//gluster volume heal "systems" info//` takes 3-10 seconds
to complete during heal - I hope that doesn't slow down healing since I
tend to run that command frequently./
Would you expect these results or is something wrong?
Would upgrading to Gluster 3.6 or 3.7 improve healing performance?
Thanks,
Udo
_______________________________________________
Gluster-users mailing list
[email protected]
http://www.gluster.org/mailman/listinfo/gluster-users
