Hello,
We have found that the memory usage of glusterfsd is increasing when
directories are created. Environment details can seen in the attached
environment.txt file, and the translator graph can be seen in the attached file
translator_graph.txt. In summary we have a setup of two gluster bricks, using
replicate mode, and one third gluster peer for quorum. We use glusterfs version
3.6.9. We build glusterfs directly from source code.
The test script creates many directories on the /mnt/export volume from the
storage client side. It creates directories of the path
"/mnt/export/dir.XXXXX", so all directories are under the same folder. In each
directory a file of a random size between 1024 and 11204 is created.
The number of directories created varied from 10000 to 200000 for different
runs. Before each run all directories were deleted.
A graph of the memory usage can be seen here: http://imgur.com/z8LOskQ
In the X-axis the number of directories is seen. The number varies because
before each run all directories are deleted. So first 80000 directories were
created, then they were deleted and 160000 were created then deleted and 200000
were created, then deleted and 40000 were created etc. Due to the amount of
data the resolution of the X-axis is not fine enough to show the exact
variation, but the general idea is that thousands of directories were created
and deleted.
The Y-axis shows the RSS memory usage of the glusterfsd process of this volume
on the two storage nodes (SN-0 and SN-1), as measured by 'pidstat -urdh -p
<pid>'.
We can see that the memory usage is continually increasing.
1) Is this increase normal expected? I can imagine that glusterfsd would
allocated more memory if more files or directories are created, but is there an
upper limit, or will it increase forever?
2) Can we control the memory allocation by any configuration parameters?
3) Is there a particular translator that controls this memory allocation?
We did another test, where only files were created. In this case the memory
usage reached an upper limit. Graph is available here: http://imgur.com/zDtkbml
So if only files are created there is an upper limit in memory allocation. Is
the case of directory creation different and why? Or is this a memory leak? Has
anyone else seen this behavior?
Best regards,
Klearchos
# gluster --version
glusterfs 3.6.9 built on Apr 14 2016 12:31:43
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
# uname -a
Linux SN-0 4.1.20-pc64-distro.git-v2.11-sctpmh #1 SMP Thu Apr 14 11:59:31 UTC
2016 x86_64 GNU/Linux
# gluster volume info export
Volume Name: export
Type: Replicate
Volume ID: f5b5173d-f7d3-434c-a9da-fff6b617e21c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: XXX.XXX.XXX.7:/mnt/bricks/export/brick
Brick2: XXX.XXX.XXX.10:/mnt/bricks/export/brick
Options Reconfigured:
cluster.server-quorum-type: server
cluster.consistent-metadata: on
server.allow-insecure: on
cluster.server-quorum-ratio: 51%
# gluster volume status export
Status of volume: export
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick XXX.XXX.XXX.7:/mnt/bricks/export/brick 49153 Y 11881
Brick XXX.XXX.XXX.10:/mnt/bricks/export/brick 49153 Y 25649
NFS Server on localhost N/A N N/A
Self-heal Daemon on localhost N/A Y 7413
NFS Server on XXX.XXX.XXX.4 N/A N N/A
Self-heal Daemon on XXX.XXX.XXX.4 N/A Y 16504
NFS Server on XXX.XXX.XXX.10 N/A N N/A
Self-heal Daemon on XXX.XXX.XXX.10 N/A N N/A
Task Status of Volume export
------------------------------------------------------------------------------
There are no active volume tasks
# gluster volume list
volume1
export
volume2
volume3
volume4
volume5
============================ /mnt/export: Client graph ==================
Final graph:
+------------------------------------------------------------------------------+
1: volume export-client-0
2: type protocol/client
3: option ping-timeout 42
4: option remote-host 169.254.0.14
5: option remote-subvolume /mnt/bricks/export/brick
6: option transport-type socket
7: option send-gids true
8: end-volume
9:
10: volume export-client-1
11: type protocol/client
12: option ping-timeout 42
13: option remote-host 169.254.0.7
14: option remote-subvolume /mnt/bricks/export/brick
15: option transport-type socket
16: option send-gids true
17: end-volume
18:
19: volume export-replicate-0
20: type cluster/replicate
21: option consistent-metadata on
22: subvolumes export-client-0 export-client-1
23: end-volume
24:
25: volume export-dht
26: type cluster/distribute
27: subvolumes export-replicate-0
28: end-volume
29:
30: volume export-write-behind
31: type performance/write-behind
32: subvolumes export-dht
33: end-volume
34:
35: volume export-read-ahead
36: type performance/read-ahead
37: subvolumes export-write-behind
38: end-volume
39:
40: volume export-io-cache
41: type performance/io-cache
42: subvolumes export-read-ahead
43: end-volume
44:
45: volume export-quick-read
46: type performance/quick-read
47: subvolumes export-io-cache
48: end-volume
49:
50: volume export-open-behind
51: type performance/open-behind
52: subvolumes export-quick-read
53: end-volume
54:
55: volume export-md-cache
56: type performance/md-cache
57: subvolumes export-open-behind
58: end-volume
59:
60: volume export
61: type debug/io-stats
62: option latency-measurement off
63: option count-fop-hits off
64: subvolumes export-md-cache
65: end-volume
66:
67: volume meta-autoload
68: type meta
69: subvolumes export
70: end-volume
71:
+------------------------------------------------------------------------------+
============================ /mnt/export: Server graph =====================
Final graph:
+------------------------------------------------------------------------------+
1: volume export-posix
2: type storage/posix
3: option glusterd-uuid 1e0b6158-ff2b-4acc-8547-547c4754b32c
4: option directory /mnt/bricks/export/brick
5: option volume-id f5b5173d-f7d3-434c-a9da-fff6b617e21c
6: end-volume
7:
8: volume export-changelog
9: type features/changelog
10: option changelog-brick /mnt/bricks/export/brick
11: option changelog-dir /mnt/bricks/export/brick/.glusterfs/changelogs
12: option changelog-barrier-timeout 120
13: subvolumes export-posix
14: end-volume
15:
16: volume export-access-control
17: type features/access-control
18: subvolumes export-changelog
19: end-volume
20:
21: volume export-locks
22: type features/locks
23: subvolumes export-access-control
24: end-volume
25:
26: volume export-io-threads
27: type performance/io-threads
28: subvolumes export-locks
29: end-volume
30:
31: volume export-barrier
32: type features/barrier
33: option barrier disable
34: option barrier-timeout 120
34: option barrier-timeout 120
35: subvolumes export-io-threads
36: end-volume
37:
38: volume export-index
39: type features/index
40: option index-base /mnt/bricks/export/brick/.glusterfs/indices
41: subvolumes export-barrier
42: end-volume
43:
44: volume export-marker
45: type features/marker
46: option volume-uuid f5b5173d-f7d3-434c-a9da-fff6b617e21c
47: option timestamp-file /var/lib/glusterd/vols/export/marker.tstamp
48: option xtime off
49: option gsync-force-xtime off
50: option quota off
51: subvolumes export-index
52: end-volume
53:
54: volume export-quota
55: type features/quota
56: option volume-uuid export
57: option server-quota off
58: option timeout 0
59: option deem-statfs off
60: subvolumes export-marker
61: end-volume
62:
63: volume /mnt/bricks/export/brick
64: type debug/io-stats
65: option latency-measurement off
66: option count-fop-hits off
67: subvolumes export-quota
68: end-volume
69:
70: volume export-server
71: type protocol/server
72: option transport.socket.listen-port 49153
73: option rpc-auth.auth-glusterfs on
74: option rpc-auth.auth-unix on
75: option rpc-auth.auth-null on
76: option transport-type tcp
77: option auth.login./mnt/bricks/export/brick.allow
aec4710a-db1a-42e6-aa4b-af6d80839ef2
78: option auth.login.aec4710a-db1a-42e6-aa4b-af6d80839ef2.password
a76c8d5a-3055-48dd-9d6a-f50f371ccf6e
79: option auth.addr./mnt/bricks/export/brick.allow *
80: option rpc-auth-allow-insecure on
81: subvolumes /mnt/bricks/export/brick
82: end-volume
83:
+------------------------------------------------------------------------------+
_______________________________________________
Gluster-devel mailing list
[email protected]
http://www.gluster.org/mailman/listinfo/gluster-devel
