Is it possible for you to send us the statedump file? It will be easier than going back and forth over emails.
Thanks, Nithya On 9 August 2018 at 09:25, huting3 <[email protected]> wrote: > Yes, I got the dump file and found there are many huge num_allocs just > like following: > > I found memusage of 4 variable types are extreamly huge. > > [protocol/client.gv0-client-0 - usage-type gf_common_mt_char memusage] > size=47202352 > num_allocs=2030212 > max_size=47203074 > max_num_allocs=2030235 > total_allocs=26892201 > > [protocol/client.gv0-client-0 - usage-type gf_common_mt_memdup memusage] > size=24362448 > num_allocs=2030204 > max_size=24367560 > max_num_allocs=2030226 > total_allocs=17830860 > > [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage] > size=2497947552 > num_allocs=4578229 > max_size=2459135680 > max_num_allocs=7123206 > total_allocs=41635232 > > [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage] > size=4038730976 > num_allocs=1 > max_size=4294962264 > max_num_allocs=37 > total_allocs=150049981 > <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png> > > > > huting3 > [email protected] > > <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png> > 签名由 网易邮箱大师 <http://mail.163.com/dashi/> 定制 > > On 08/9/2018 11:36，Raghavendra Gowdappa<[email protected]> > <[email protected]> wrote： > > > > On Thu, Aug 9, 2018 at 8:55 AM, huting3 <[email protected]> wrote: > >> Hi expert: >> >> I meet a problem when I use glusterfs. The problem is that the fuse >> client consumes huge memory when write a lot of files(>million) to the >> gluster, at last leading to killed by OS oom. The memory the fuse process >> consumes can up to 100G! I wonder if there are memory leaks in the gluster >> fuse process, or some other causes. >> > > Can you get statedump of fuse process consuming huge memory? > > >> My gluster version is 3.13.2, the gluster volume info is listed as >> following: >> >> Volume Name: gv0 >> Type: Distributed-Replicate >> Volume ID: 4a6f96f8-b3fb-4550-bd19-e1a5dffad4d0 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 19 x 3 = 57 >> Transport-type: tcp >> Bricks: >> Brick1: dl20.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick2: dl21.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick3: dl22.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick4: dl20.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick5: dl21.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick6: dl22.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick7: dl20.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick8: dl21.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick9: dl22.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick10: dl23.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick11: dl24.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick12: dl25.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick13: dl26.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick14: dl27.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick15: dl28.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick16: dl29.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick17: dl30.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick18: dl31.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick19: dl32.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick20: dl33.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick21: dl34.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick22: dl23.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick23: dl24.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick24: dl25.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick25: dl26.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick26: dl27.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick27: dl28.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick28: dl29.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick29: dl30.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick30: dl31.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick31: dl32.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick32: dl33.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick33: dl34.dg.163.org:/glusterfs_brick/brick2/gv0 >> Brick34: dl23.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick35: dl24.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick36: dl25.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick37: dl26.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick38: dl27.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick39: dl28.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick40: dl29.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick41: dl30.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick42: dl31.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick43: dl32.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick44: dl33.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick45: dl34.dg.163.org:/glusterfs_brick/brick3/gv0 >> Brick46: dl0.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick47: dl1.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick48: dl2.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick49: dl3.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick50: dl5.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick51: dl6.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick52: dl9.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick53: dl10.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick54: dl11.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick55: dl12.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick56: dl13.dg.163.org:/glusterfs_brick/brick1/gv0 >> Brick57: dl14.dg.163.org:/glusterfs_brick/brick1/gv0 >> Options Reconfigured: >> performance.cache-size: 10GB >> performance.parallel-readdir: on >> performance.readdir-ahead: on >> network.inode-lru-limit: 200000 >> performance.md-cache-timeout: 600 >> performance.cache-invalidation: on >> performance.stat-prefetch: on >> features.cache-invalidation-timeout: 600 >> features.cache-invalidation: on >> features.inode-quota: off >> features.quota: off >> cluster.quorum-reads: on >> cluster.quorum-count: 2 >> cluster.quorum-type: fixed >> transport.address-family: inet >> nfs.disable: on >> performance.client-io-threads: off >> cluster.server-quorum-ratio: 51% >> >> >> huting3 >> [email protected] >> >> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?&name=huting3&uid=huting3%40corp.netease.com&ftlId=1&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22huting3%40corp.netease.com%22%5D&logoUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyeicon%2F209a2912f40f6683af56bb7caff1cb54.png> >> 签名由 网易邮箱大师 <http://mail.163.com/dashi/> 定制 >> >> >> _______________________________________________ >> Gluster-devel mailing list >> [email protected] >> https://lists.gluster.org/mailman/listinfo/gluster-devel >> > > > _______________________________________________ > Gluster-devel mailing list > [email protected] > https://lists.gluster.org/mailman/listinfo/gluster-devel >
_______________________________________________ Gluster-devel mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-devel
