Sorry, I replied based on the wrong title, forget about this. From: Yong Zhang<mailto:his...@outlook.com> Sent: Saturday, April 1, 2017 1:49 AM To: Amar Tumballi<mailto:atumb...@redhat.com>; Alvin Starr<mailto:al...@netvel.net> Cc: gluster-users@gluster.org List<mailto:gluster-users@gluster.org> Subject: Re: [Gluster-users] hanging httpd processes.
Thanks Amar, I’ll consider your recommendations. But why performance is totally different on two nodes? Will data be written to both nodes at the same time? From: Amar Tumballi<mailto:atumb...@redhat.com> Sent: Friday, March 31, 2017 3:14 PM To: Alvin Starr<mailto:al...@netvel.net> Cc: gluster-users@gluster.org List<mailto:gluster-users@gluster.org> Subject: Re: [Gluster-users] hanging httpd processes. On Fri, Mar 31, 2017 at 12:29 PM, Amar Tumballi <atumb...@redhat.com<mailto:atumb...@redhat.com>> wrote: Hi Alvin, Thanks for the dump output. It helped a bit. For now, recommend turning off open-behind and read-ahead performance translators for you to get rid of this situation, As I noticed hung FLUSH operations from these translators. Looks like I gave wrong advise by looking at below snippet: [global.callpool.stack.61] stack=0x7f6c6f628f04 uid=48 gid=48 pid=11077 unique=10048797 lk-owner=a73ae5bdb5fcd0d2 op=FLUSH type=1 cnt=5 [global.callpool.stack.61.frame.1] frame=0x7f6c6f793d88 ref_count=0 translator=edocs-production-write-behind complete=0 parent=edocs-production-read-ahead wind_from=ra_flush wind_to=FIRST_CHILD (this)->fops->flush unwind_to=ra_flush_cbk [global.callpool.stack.61.frame.2] frame=0x7f6c6f796c90 ref_count=1 translator=edocs-production-read-ahead complete=0 parent=edocs-production-open-behind wind_from=default_flush_resume wind_to=FIRST_CHILD(this)->fops->flush unwind_to=default_flush_cbk [global.callpool.stack.61.frame.3] frame=0x7f6c6f79b724 ref_count=1 translator=edocs-production-open-behind complete=0 parent=edocs-production wind_from=io_stats_flush wind_to=FIRST_CHILD(this)->fops->flush unwind_to=io_stats_flush_cbk [global.callpool.stack.61.frame.4] frame=0x7f6c6f79b474 ref_count=1 translator=edocs-production complete=0 parent=fuse wind_from=fuse_flush_resume wind_to=FIRST_CHILD(this)->fops->flush unwind_to=fuse_err_cbk [global.callpool.stack.61.frame.5] frame=0x7f6c6f796684 ref_count=1 translator=fuse complete=0 Mos probably, issue is with write-behind's flush. So please turn off write-behind and test. If you don't have any hung httpd processes, please let us know. -Amar -Amar On Wed, Mar 29, 2017 at 6:56 AM, Alvin Starr <al...@netvel.net<mailto:al...@netvel.net>> wrote: We are running gluster 3.8.9-1 on Centos 7.3.1611 for the servers and on the clients 3.7.11-2 on Centos 6.8 We are seeing httpd processes hang in fuse_request_send or sync_page. These calls are from PHP 5.3.3-48 scripts I am attaching a tgz file that contains the process dump from glusterfsd and the hung pids along with the offending pid's stacks from /proc/{pid}/stack. This has been a low level annoyance for a while but it has become a much bigger issue because the number of hung processes went from a few a week to a few hundred a day. -- Alvin Starr || voice: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net<mailto:al...@netvel.net> || _______________________________________________ Gluster-users mailing list Gluster-users@gluster.org<mailto:Gluster-users@gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users -- Amar Tumballi (amarts) -- Amar Tumballi (amarts)
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users