I started rebalancing my 25x2 distributed-replicate volume two days ago.
Since then, the memory usage of the rebalance processes has been
steadily climbing by 1-2 megabytes per minute. Following
http://gluster.org/community/documentation/index.php/High_Memory_Usage,
I tried "echo 2 > /proc/sys/vm/drop_caches". This had no effect on the
processes' memory usage. Some of the servers are already eating >10G of
memory. At this rate, I will have to cancel this rebalance, even though
brick usage is heavily skewed right now (most bricks are at 87%
capacity, but recently added ones are at 18-28%).
Any ideas what might be causing this? The only references to rebalance
memory leaks I could find were related to 3.2.x, not 3.3.1.
gluster volume info:
Volume Name: bigdata
Type: Distributed-Replicate
Volume ID: 56498956-7b4b-4ee3-9d2b-4c8cfce26051
Status: Started
Number of Bricks: 25 x 2 = 50
Transport-type: tcp
Bricks:
Brick1: ml43:/mnt/donottouch/localb
Brick2: ml44:/mnt/donottouch/localb
Brick3: ml43:/mnt/donottouch/localc
Brick4: ml44:/mnt/donottouch/localc
Brick5: ml45:/mnt/donottouch/localb
Brick6: ml46:/mnt/donottouch/localb
Brick7: ml45:/mnt/donottouch/localc
Brick8: ml46:/mnt/donottouch/localc
Brick9: ml47:/mnt/donottouch/localb
Brick10: ml48:/mnt/donottouch/localb
Brick11: ml47:/mnt/donottouch/localc
Brick12: ml48:/mnt/donottouch/localc
Brick13: ml45:/mnt/donottouch/locald
Brick14: ml46:/mnt/donottouch/locald
Brick15: ml47:/mnt/donottouch/locald
Brick16: ml48:/mnt/donottouch/locald
Brick17: ml51:/mnt/donottouch/localb
Brick18: ml52:/mnt/donottouch/localb
Brick19: ml51:/mnt/donottouch/localc
Brick20: ml52:/mnt/donottouch/localc
Brick21: ml51:/mnt/donottouch/locald
Brick22: ml52:/mnt/donottouch/locald
Brick23: ml59:/mnt/donottouch/locald
Brick24: ml54:/mnt/donottouch/locald
Brick25: ml59:/mnt/donottouch/localc
Brick26: ml54:/mnt/donottouch/localc
Brick27: ml59:/mnt/donottouch/localb
Brick28: ml54:/mnt/donottouch/localb
Brick29: ml55:/mnt/donottouch/localb
Brick30: ml29:/mnt/donottouch/localb
Brick31: ml55:/mnt/donottouch/localc
Brick32: ml29:/mnt/donottouch/localc
Brick33: ml30:/mnt/donottouch/localc
Brick34: ml31:/mnt/donottouch/localc
Brick35: ml30:/mnt/donottouch/localb
Brick36: ml31:/mnt/donottouch/localb
Brick37: ml40:/mnt/donottouch/localb
Brick38: ml41:/mnt/donottouch/localb
Brick39: ml40:/mnt/donottouch/localc
Brick40: ml41:/mnt/donottouch/localc
Brick41: ml56:/mnt/donottouch/localb
Brick42: ml57:/mnt/donottouch/localb
Brick43: ml56:/mnt/donottouch/localc
Brick44: ml57:/mnt/donottouch/localc
Brick45: ml25:/mnt/donottouch/localb
Brick46: ml26:/mnt/donottouch/localb
Brick47: ml01:/mnt/donottouch/localb/brick
Brick48: ml25:/mnt/donottouch/localc/brick
Brick49: ml01:/mnt/donottouch/localc/brick
Brick50: ml26:/mnt/donottouch/localc/brick
Options Reconfigured:
nfs.register-with-portmap: OFF
nfs.disable: on
performance.quick-read: on
The majority of these bricks are ext4, except for the latest 7 which are
xfs. Each is backed by a 2T hard drive.
gluster --version:
glusterfs 3.3.1 built on Oct 11 2012 21:49:37
cat /etc/system-release:
Scientific Linux release 6.1 (Carbon)
uname -a:
Linux ml01 2.6.32-131.17.1.el6.x86_64 #1 SMP Wed Oct 5 17:19:54 CDT 2011
x86_64 x86_64 x86_64 GNU/Linux
df -i /mnt/bigdata:
Filesystem Inodes IUsed IFree IUse% Mounted on
ml43:/bigdata 3272292992 114236317 3158056675 4% /mnt/bigdata
df /mnt/bigdata:
Filesystem 1K-blocks Used Available Use% Mounted on
ml43:/bigdata 48160570880 33787913600 12223793792 74% /mnt/bigdata
the process I am referring to:
/usr/sbin/glusterfs -s localhost --volfile-id bigdata --xlator-option
*dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes
--xlator-option *dht.assert-no-child-down=yes --xlator-option
*replicate*.data-self-heal=off --xlator-option
*replicate*.metadata-self-heal=off --xlator-option
*replicate*.entry-self-heal=off --xlator-option *dht.rebalance-cmd=1
--xlator-option *dht.node-uuid=5c338e03-28ff-429b-b702-0a04e25565f8
--socket-file
/var/lib/glusterd/vols/bigdata/rebalance/5c338e03-28ff-429b-b702-0a04e25565f8.sock
--pid-file
/var/lib/glusterd/vols/bigdata/rebalance/5c338e03-28ff-429b-b702-0a04e25565f8.pid
-l /var/log/glusterfs/bigdata-rebalance.log
I am seeing a lot of:
[2013-03-05 10:59:51.170051] I [dht-rebalance.c:647:dht_migrate_file]
0-bigdata-dht: /a/c/lu/lu-nod/_4.nrm: attempting to move from
bigdata-replicate-23 to bigdata-replicate-17
[2013-03-05 10:59:51.296487] W
[dht-rebalance.c:361:__dht_check_free_space] 0-bigdata-dht: data
movement attempted from node (bigdata-replicate-23) with higher disk
space to a node (bigdata-replicate-17) with lesser disk space
(/a/c/lu/lu-nod/_4.nrm)
[2013-03-05 10:59:51.296604] E
[dht-rebalance.c:1202:gf_defrag_migrate_data] 0-bigdata-dht:
migrate-data failed for /a/c/lu/lu-nod/_4.nrm
in the rebalance log, and a lot of:
[2013-03-05 11:00:02.609033] I [server3_1-fops.c:576:server_mknod_cbk]
0-bigdata-server: 586514: MKNOD (null) (--) ==> -1 (File exists)
and
[2013-03-05 10:59:58.755505] E [posix-helpers.c:701:posix_handle_pair]
0-bigdata-posix: /mnt/donottouch/localb/brick/b/somedir/foobar:
key:trusted.glusterfs.dht.linkto error:File exists
[2013-03-05 10:59:58.755576] E [posix.c:1755:posix_create]
0-bigdata-posix: setting xattrs on
/mnt/donottouch/localb/brick/b/somedir/foobar failed (File exists)
in the brick logs.
gluster volume rebalance bigdata status:
Node, Rebalanced-files, size,
scanned, failures, status
ml01 296 279.1MB
6761635 29006 in progress
ml25 1208 1.1GB
6167501 75131 in progress
ml26 0 0Bytes
7197961 0 in progress
ml29 0 0Bytes
5892140 0 in progress
ml30 118486 74.3GB
5867068 107288 in progress
ml31 0 0Bytes
6802084 0 in progress
ml40 55652 41.2GB
6100163 94717 in progress
ml41 0 0Bytes
6151999 0 in progress
ml43 124390 53.3GB
5470703 101802 in progress
ml44 0 0Bytes
6789294 0 in progress
ml45 146811 61.1GB
4829884 242848 in progress
ml46 0 0Bytes
7184315 0 in progress
ml47 82719 67.5GB
5954615 153226 in progress
ml48 0 0Bytes
7826333 0 in progress
ml51 156903 76.2GB
4681106 146970 in progress
ml52 0 0Bytes
6154015 0 in progress
ml54 0
0Bytes 42773 0 in progress
ml55 44194 33.4GB
6113086 82874 in progress
ml56 16060 16.5GB
5949356 136277 in progress
ml57 0 0Bytes
7845347 0 in progress
ml59 128070 76.7GB
5847966 145739 in progress
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
