[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 Bernard Spil changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED URL||https://svn.apache.org/view ||vc?view=revision=1 ||840028 --- Comment #15 from Bernard Spil --- (In reply to Yann Ylavic from comment #14) > Follow up in r1840002, simpler to test latest trunk anyway. Hi Yann, Tested trunk r1840028 and confirm that it's fixed. Thanks! Bernard. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #14 from Yann Ylavic --- Follow up in r1840002, simpler to test latest trunk anyway. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 Yann Ylavic changed: What|Removed |Added Status|NEEDINFO|NEW --- Comment #13 from Yann Ylavic --- Hi Bernard, should be fixed in r1839997, can you please give it a try? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #12 from Yann Ylavic --- Nevermind, I think I can reproduce with large download and some rate limit on the client side. Let me see what happens... -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #11 from Yann Ylavic --- It would be interesting to know which filters are pending in ap_filter_output_pending() too (i.e. print f->frec->name in the loop). -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #10 from Yann Ylavic --- I don't see how ap_filter_output_pending() can loop indefinitely.. Can you please step (or next) there from gdb, with "set scheduler-locking on" to stay in the same thread (and preferably symbols if possible)? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #9 from Bernard Spil --- Hi Yann, -fno-strict-aliasing is enabled by default. Build log of an earlier version here: https://brnrd.eu/poudriere/data/112libre-default/2018-08-05_22h41m54s/logs/apache25-devel-2.5.1.20180529.log poudriere (FreeBSD default ports bulk build tool) logs the MAKE_ENV in a separate section: > CFLAGS="-O2 -fno-strict-aliasing -pipe -march=native -DLIBICONV_PLUG > -fstack-protector" > CXXFLAGS="-O2 -fno-strict-aliasing -pipe -march=native -DLIBICONV_PLUG > -fstack-protector -DLIBICONV_PLUG" -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #8 from Yann Ylavic --- OK thanks, if you don't already, can you try to compile with -fno-strict-aliasing? Aliasing is known to break with APR_RING... -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #7 from Bernard Spil --- Created attachment 36129 --> https://bz.apache.org/bugzilla/attachment.cgi?id=36129=edit stepping through util_filter.c 1012-1032 Stepping through I see that I'm switching between the threads that are running at 100% CPU and never even leave that block of code (lines 1012-1032 in server/util_filter.c). https://svn.apache.org/viewvc/httpd/httpd/trunk/server/util_filter.c?revision=1837822=markup#l1012 Hope that that's what you were looking for. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #6 from Bernard Spil --- (In reply to Yann Ylavic from comment #4) > Bernard, is this with high concurrency scenario or it can be reproduced with > a single/few requests? > > Looking at the stacktrace, the most probable looping threads are numbers 60 > to 62, thus ap_filter_output_pending(). Is the endless loop internal to this > function or MPM event's process_socket() is calling it endlessly? > > If I can't reproduce with a simple scenario, could you please debug that > loop (latest trunk) and determine why ap_filter_output_pending() either > never ends or keeps returning OK? It should stop (DECLINED) when the socket > is completely flushed... Hi Yann, I have found that I can reliably reproduce it using "ANALYZE" on https://developers.google.com/speed/pagespeed/insights I have not been able to reproduce it by merely loading a page with many resources. Tried ab too (ab -n 1 -c 250 https://test.brnrd.eu/) but that doesn't trigger the issue either. Yesterday I had issues inspecting the running process, now trying again to see if I can find out more about the spinning thread. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #5 from Bernard Spil --- (In reply to Eric Covener from comment #3) > (In reply to Bernard Spil from comment #0) > > Created attachment 36126 [details] > > gdb thread apply all backtrace > > > > Somewhere between 2018-06-20 and 2018-07-17 there's been a change in trunk > > that makes multiple threads spin at 100% after requests have been processed. > > Can you try moving fwd to r1837823 or greater? r1837822+r1837823 seems > related. Hi Eric, This is reproducible with a 20180831 snapshot as well (git tag 50a5690). I pull the tarball from GitHub but that should be the same code. https://github.com/apache/httpd/commit/767deda89753286d376c5660198a0b2f7651301b https://github.com/apache/httpd/commit/10942ed3a206b83d81bbf45f91a15678fe8bcf87 -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #4 from Yann Ylavic --- Bernard, is this with high concurrency scenario or it can be reproduced with a single/few requests? Looking at the stacktrace, the most probable looping threads are numbers 60 to 62, thus ap_filter_output_pending(). Is the endless loop internal to this function or MPM event's process_socket() is calling it endlessly? If I can't reproduce with a simple scenario, could you please debug that loop (latest trunk) and determine why ap_filter_output_pending() either never ends or keeps returning OK? It should stop (DECLINED) when the socket is completely flushed... -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 Eric Covener changed: What|Removed |Added Status|NEW |NEEDINFO --- Comment #3 from Eric Covener --- (In reply to Bernard Spil from comment #0) > Created attachment 36126 [details] > gdb thread apply all backtrace > > Somewhere between 2018-06-20 and 2018-07-17 there's been a change in trunk > that makes multiple threads spin at 100% after requests have been processed. Can you try moving fwd to r1837823 or greater? r1837822+r1837823 seems related. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #2 from Bernard Spil --- The version I roll back to to make this work (build date is also trunk date): > # httpd -V > Server built: Jun 20 2018 18:28:30 > Server's Module Magic Number: 20180606:1 First version I have exhibiting the 100% CPU for multiple threads > # httpd -V > Server built: Jul 17 2018 10:28:46 > Server's Module Magic Number: 20180716:2 tyj Looking at the magic numbers I rolled back a bit further, trunk on 2018-07-11 is already problematic: > # /usr/local/sbin/httpd -V > Server's Module Magic Number: 20180711:1 Backtrace is in line with the attachment. Magic number was bumped in https://svn.apache.org/viewvc?view=revision=1835640 and the accompanying changes are the likely culprit. Going back a couple commits on trunk to 2018-07-10 results in a non-problematic version > # /usr/local/sbin/httpd -V > Server's Module Magic Number: 20180606:1 -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
[Bug 62668] httpd threads spinning at 100%
https://bz.apache.org/bugzilla/show_bug.cgi?id=62668 --- Comment #1 from Bernard Spil --- The version I roll back to to make this work: # httpd -V Server version: Apache/2.5.1-dev (FreeBSD) Server built: Jun 20 2018 18:28:30 Server's Module Magic Number: 20180606:1 First version I have exhibiting the 100% CPU for multiple threads # httpd -V Server version: Apache/2.5.1-dev (FreeBSD) Server built: Jul 17 2018 10:28:46 Server's Module Magic Number: 20180716:2 -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: bugs-unsubscr...@httpd.apache.org For additional commands, e-mail: bugs-h...@httpd.apache.org
