google vdsm memory leak..it's been discussed on list last year and earlier this one...
On Thu, Sep 24, 2015 at 10:14 AM, Oleksandr Natalenko < oleksa...@natalenko.name> wrote: > In our GlusterFS deployment we've encountered something like memory leak > in GlusterFS FUSE client. > > We use replicated (×2) GlusterFS volume to store mail (exim+dovecot, > maildir format). Here is inode stats for both bricks and mountpoint: > > === > Brick 1 (Server 1): > > Filesystem Inodes IUsed > IFree IUse% Mounted on > /dev/mapper/vg_vd1_misc-lv08_mail 578768144 10954918 > 567813226 2% /bricks/r6sdLV08_vd1_mail > > Brick 2 (Server 2): > > Filesystem Inodes IUsed > IFree IUse% Mounted on > /dev/mapper/vg_vd0_misc-lv07_mail 578767984 10954913 > 567813071 2% /bricks/r6sdLV07_vd0_mail > > Mountpoint (Server 3): > > Filesystem Inodes IUsed IFree IUse% > Mounted on > glusterfs.xxx:mail 578767760 10954915 567812845 2% > /var/spool/mail/virtual > === > > glusterfs.xxx domain has two A records for both Server 1 and Server 2. > > Here is volume info: > > === > Volume Name: mail > Type: Replicate > Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2 > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail > Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail > Options Reconfigured: > nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24 > features.cache-invalidation-timeout: 10 > performance.stat-prefetch: off > performance.quick-read: on > performance.read-ahead: off > performance.flush-behind: on > performance.write-behind: on > performance.io-thread-count: 4 > performance.cache-max-file-size: 1048576 > performance.cache-size: 67108864 > performance.readdir-ahead: off > === > > Soon enough after mounting and exim/dovecot start, glusterfs client > process begins to consume huge amount of RAM: > > === > user@server3 ~$ ps aux | grep glusterfs | grep mail > root 28895 14.4 15.0 15510324 14908868 ? Ssl Sep03 4310:05 > /usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable > --volfile-server=glusterfs.xxx --volfile-id=mail /var/spool/mail/virtual > === > > That is, ~15 GiB of RAM. > > Also we've tried to use mountpoint withing separate KVM VM with 2 or 3 GiB > of RAM, and soon after starting mail daemons got OOM killer for glusterfs > client process. > > Mounting same share via NFS works just fine. Also, we have much less > iowait and loadavg on client side with NFS. > > Also, we've tried to change IO threads count and cache size in order to > limit memory usage with no luck. As you can see, total cache size is > 4×64==256 MiB (compare to 15 GiB). > > Enabling-disabling stat-prefetch, read-ahead and readdir-ahead didn't help > as well. > > Here are volume memory stats: > > === > Memory status for volume : mail > ---------------------------------------------- > Brick : server1.xxx:/bricks/r6sdLV08_vd1_mail/mail > Mallinfo > -------- > Arena : 36859904 > Ordblks : 10357 > Smblks : 519 > Hblks : 21 > Hblkhd : 30515200 > Usmblks : 0 > Fsmblks : 53440 > Uordblks : 18604144 > Fordblks : 18255760 > Keepcost : 114112 > > Mempool Stats > ------------- > Name HotCount ColdCount PaddedSizeof AllocCount > MaxAlloc Misses Max-StdAlloc > ---- -------- --------- ------------ ---------- > -------- -------- ------------ > mail-server:fd_t 0 1024 108 > 30773120 137 0 0 > mail-server:dentry_t 16110 274 84 > 235676148 16384 1106499 1152 > mail-server:inode_t 16363 21 156 > 237216876 16384 1876651 1169 > mail-trash:fd_t 0 1024 108 > 0 0 0 0 > mail-trash:dentry_t 0 32768 84 > 0 0 0 0 > mail-trash:inode_t 4 32764 156 > 4 4 0 0 > mail-trash:trash_local_t 0 64 8628 > 0 0 0 0 > mail-changetimerecorder:gf_ctr_local_t 0 64 16540 > 0 0 0 0 > mail-changelog:rpcsvc_request_t 0 8 2828 > 0 0 0 0 > mail-changelog:changelog_local_t 0 64 116 > 0 0 0 0 > mail-bitrot-stub:br_stub_local_t 0 512 84 > 79204 4 0 0 > mail-locks:pl_local_t 0 32 148 > 6812757 4 0 0 > mail-upcall:upcall_local_t 0 512 108 > 0 0 0 0 > mail-marker:marker_local_t 0 128 332 > 64980 3 0 0 > mail-quota:quota_local_t 0 64 476 > 0 0 0 0 > mail-server:rpcsvc_request_t 0 512 2828 > 45462533 34 0 0 > glusterfs:struct saved_frame 0 8 124 > 2 2 0 0 > glusterfs:struct rpc_req 0 8 588 > 2 2 0 0 > glusterfs:rpcsvc_request_t 1 7 2828 > 2 1 0 0 > glusterfs:log_buf_t 5 251 140 > 3452 6 0 0 > glusterfs:data_t 242 16141 52 > 480115498 664 0 0 > glusterfs:data_pair_t 230 16153 68 > 179483528 275 0 0 > glusterfs:dict_t 23 4073 140 > 303751675 627 0 0 > glusterfs:call_stub_t 0 1024 3764 > 45290655 34 0 0 > glusterfs:call_stack_t 1 1023 1708 > 43598469 34 0 0 > glusterfs:call_frame_t 1 4095 172 > 336219655 184 0 0 > ---------------------------------------------- > Brick : server2.xxx:/bricks/r6sdLV07_vd0_mail/mail > Mallinfo > -------- > Arena : 38174720 > Ordblks : 9041 > Smblks : 507 > Hblks : 21 > Hblkhd : 30515200 > Usmblks : 0 > Fsmblks : 51712 > Uordblks : 19415008 > Fordblks : 18759712 > Keepcost : 114848 > > Mempool Stats > ------------- > Name HotCount ColdCount PaddedSizeof AllocCount > MaxAlloc Misses Max-StdAlloc > ---- -------- --------- ------------ ---------- > -------- -------- ------------ > mail-server:fd_t 0 1024 108 > 2373075 133 0 0 > mail-server:dentry_t 14114 2270 84 > 3513654 16384 2300 267 > mail-server:inode_t 16374 10 156 > 6766642 16384 194635 1279 > mail-trash:fd_t 0 1024 108 > 0 0 0 0 > mail-trash:dentry_t 0 32768 84 > 0 0 0 0 > mail-trash:inode_t 4 32764 156 > 4 4 0 0 > mail-trash:trash_local_t 0 64 8628 > 0 0 0 0 > mail-changetimerecorder:gf_ctr_local_t 0 64 16540 > 0 0 0 0 > mail-changelog:rpcsvc_request_t 0 8 2828 > 0 0 0 0 > mail-changelog:changelog_local_t 0 64 116 > 0 0 0 0 > mail-bitrot-stub:br_stub_local_t 0 512 84 > 71354 4 0 0 > mail-locks:pl_local_t 0 32 148 > 8135032 4 0 0 > mail-upcall:upcall_local_t 0 512 108 > 0 0 0 0 > mail-marker:marker_local_t 0 128 332 > 65005 3 0 0 > mail-quota:quota_local_t 0 64 476 > 0 0 0 0 > mail-server:rpcsvc_request_t 0 512 2828 > 12882393 30 0 0 > glusterfs:struct saved_frame 0 8 124 > 2 2 0 0 > glusterfs:struct rpc_req 0 8 588 > 2 2 0 0 > glusterfs:rpcsvc_request_t 1 7 2828 > 2 1 0 0 > glusterfs:log_buf_t 5 251 140 > 3443 6 0 0 > glusterfs:data_t 242 16141 52 > 138743429 290 0 0 > glusterfs:data_pair_t 230 16153 68 > 126649864 270 0 0 > glusterfs:dict_t 23 4073 140 > 20356289 63 0 0 > glusterfs:call_stub_t 0 1024 3764 > 13678560 31 0 0 > glusterfs:call_stack_t 1 1023 1708 > 11011561 30 0 0 > glusterfs:call_frame_t 1 4095 172 > 125764190 193 0 0 > ---------------------------------------------- > === > > So, my questions are: > > 1) what one should do to limit GlusterFS FUSE client memory usage? > 2) what one should do to prevent client high loadavg because of high > iowait because of multiple concurrent volume users? > > Server/client OS is CentOS 7.1, GlusterFS server version is 3.7.3, > GlusterFS client version is 3.7.4. > > Any additional info needed? > _______________________________________________ > Gluster-users mailing list > gluster-us...@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel