Could you provide this information from the server (also the client/server logs):
cat /proc/<pid>/task/*/stack ls -l /proc/<pid>/task/*/fd <pid> == process ID of glusterfsd On Fri, Feb 7, 2014 at 6:14 PM, Barry Stetler <[email protected]> wrote: > We started using gluster in a custom solution recently. Basically we > setup a gluster volume that we mount at /home. Then we give users their own > home directory with a gluster quota. They have FTP, SFTP and SSHMOUNT > access. I first used vsftpd and notice that ftp connections will not close > and the load goes up ever couple of days the funny thing is the load does > not seem to slow down the server. The load went up to 20 in the last two > months but the server was not slow at all... At first I thought it was > vsftpd so I changed to proftpd then the issue came back. If I reboot the > server it goes away for a few days depending on how many ftp connections. I > noticed in the dmesg log this making me think the issue was some kind of > leak in gluster-fuse > > This is on CentOS 6.5 64 bit. > > INFO: task proftpd:7831 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > proftpd D 0000000000000000 0 7831 4384 0x00000004 > ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439 > ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040 > ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8 > Call Trace: > [<ffffffff81051439>] ? __wake_up_common+0x59/0x90 > [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse] > [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40 > [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse] > [<ffffffff8117e0bc>] filp_close+0x3c/0x90 > [<ffffffff8117e1b5>] sys_close+0xa5/0x100 > [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b > INFO: task proftpd:7831 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > proftpd D 0000000000000000 0 7831 4384 0x00000004 > ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439 > ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040 > ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8 > Call Trace: > [<ffffffff81051439>] ? __wake_up_common+0x59/0x90 > [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse] > [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40 > [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse] > [<ffffffff8117e0bc>] filp_close+0x3c/0x90 > [<ffffffff8117e1b5>] sys_close+0xa5/0x100 > [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b > INFO: task proftpd:7831 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > proftpd D 0000000000000000 0 7831 4384 0x00000004 > ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439 > ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040 > ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8 > Call Trace: > [<ffffffff81051439>] ? __wake_up_common+0x59/0x90 > [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse] > [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40 > [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse] > [<ffffffff8117e0bc>] filp_close+0x3c/0x90 > [<ffffffff8117e1b5>] sys_close+0xa5/0x100 > [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b > INFO: task proftpd:7831 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > proftpd D 0000000000000000 0 7831 4384 0x00000004 > ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439 > ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040 > ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8 > Call Trace: > [<ffffffff81051439>] ? __wake_up_common+0x59/0x90 > [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse] > [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40 > [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse] > [<ffffffff8117e0bc>] filp_close+0x3c/0x90 > [<ffffffff8117e1b5>] sys_close+0xa5/0x100 > [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b > INFO: task proftpd:7831 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > proftpd D 0000000000000000 0 7831 4384 0x00000004 > ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439 > ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040 > ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8 > Call Trace: > [<ffffffff81051439>] ? __wake_up_common+0x59/0x90 > [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse] > [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40 > [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse] > [<ffffffff8117e0bc>] filp_close+0x3c/0x90 > [<ffffffff8117e1b5>] sys_close+0xa5/0x100 > [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b > INFO: task proftpd:7831 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > proftpd D 0000000000000000 0 7831 4384 0x00000004 > ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439 > ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040 > ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8 > Call Trace: > [<ffffffff81051439>] ? __wake_up_common+0x59/0x90 > [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse] > [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40 > [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse] > [<ffffffff8117e0bc>] filp_close+0x3c/0x90 > [<ffffffff8117e1b5>] sys_close+0xa5/0x100 > [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b > INFO: task proftpd:7831 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > proftpd D 0000000000000000 0 7831 4384 0x00000004 > ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439 > ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040 > ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8 > Call Trace: > [<ffffffff81051439>] ? __wake_up_common+0x59/0x90 > [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse] > [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40 > [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse] > [<ffffffff8117e0bc>] filp_close+0x3c/0x90 > [<ffffffff8117e1b5>] sys_close+0xa5/0x100 > [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b > INFO: task proftpd:7831 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > proftpd D 0000000000000000 0 7831 4384 0x00000004 > ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439 > ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040 > ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8 > Call Trace: > [<ffffffff81051439>] ? __wake_up_common+0x59/0x90 > [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse] > [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40 > [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse] > [<ffffffff8117e0bc>] filp_close+0x3c/0x90 > [<ffffffff8117e1b5>] sys_close+0xa5/0x100 > [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b > INFO: task proftpd:7831 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > proftpd D 0000000000000000 0 7831 4384 0x00000004 > ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439 > ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040 > ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8 > Call Trace: > [<ffffffff81051439>] ? __wake_up_common+0x59/0x90 > [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse] > [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40 > [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse] > [<ffffffff8117e0bc>] filp_close+0x3c/0x90 > [<ffffffff8117e1b5>] sys_close+0xa5/0x100 > [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b > INFO: task proftpd:7831 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > proftpd D 0000000000000000 0 7831 4384 0x00000004 > ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439 > ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040 > ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8 > Call Trace: > [<ffffffff81051439>] ? __wake_up_common+0x59/0x90 > [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse] > [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40 > [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse] > [<ffffffff8117e0bc>] filp_close+0x3c/0x90 > [<ffffffff8117e1b5>] sys_close+0xa5/0x100 > [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b > > > I appreciate any help I can get on this... > > Thank you > > -- > Barry Stetler > Hivelocity Hosting > http://hivelocity.net > _______________________________________________ > Gluster-users mailing list > [email protected] > http://supercolony.gluster.org/mailman/listinfo/gluster-users >
_______________________________________________ Gluster-users mailing list [email protected] http://supercolony.gluster.org/mailman/listinfo/gluster-users
