Christian, we are using Perforce and local git repositories. In the review board log I find that a file retrieval from Perforce took about 27sec. just before the apache threads hung. I don't know if this is related or not.
Koushik On Friday, February 5, 2016 at 4:53:03 PM UTC-5, Koushik Roy wrote: > > I have moved this discussion here as you suggested. > > Version info: > - RB 2.0.18 > - Debian 7.6 > - Apache 2.2.2 with mod_wsgi > > Yes, this is not normal. I have been administering RB for 2 yrs. > This happened for the first time. > > I enabled server-status for Apache and checked that there were > several keep-alive threads. > > There are some queries that we run against ReviewBoard using > the REST API - some of these queries cause keep-alive. > > Keep alive is ok as long as the child apache processes do not take 100% > CPU. > But, that was not the case - > > Also, at one point in time all 4 CPUs were at 100% > Review Board log, apache log & syslog do not give me any clue. > > I will investigate. In the meantime if you have any clue, that will be > helpful. > > Thanks, > > > top - 13:10:09 up 513 days, 2:39, 5 users, load average: 2.08, 2.13, > 2.01 > > Tasks: 121 total, 1 running, 120 sleeping, 0 stopped, 0 zombie > > %Cpu0 : 8.9 us, 0.3 sy, 0.0 ni, 90.1 id, 0.0 wa, 0.0 hi, 0.7 si, > 0.0 st > > %Cpu1 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > > %Cpu2 : 2.7 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > > %Cpu3 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > > KiB Mem: 8200212 total, 6805576 used, 1394636 free, 299688 buffers > > KiB Swap: 3071996 total, 16576 used, 3055420 free, 1688108 cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > > > 6874 www-data 20 0 92120 3288 1088 S 0.0 0.0 0:00.00 > apache2 > > > 6876 www-data 20 0 614m 228m 7376 S 99.8 2.9 202:46.27 > apache2 > > > > 6878 www-data 20 0 624m 231m 7380 S 99.8 2.9 221:18.05 > apache2 > > > > 9348 www-data 20 0 1395m 967m 7592 S 2.0 12.1 10:59.89 apache2 > > > > > > > > > (gdb) where > > #0 0x00007f4e64a2118d in read () at ../sysdeps/unix/syscall-template.S:82 > > #1 0x00007f4e65542cd4 in ap_mpm_pod_check () > > #2 0x00007f4e6551524b in ?? () > > #3 0x00007f4e655417a9 in ?? () > > #4 0x00007f4e65541839 in ?? () > > #5 0x00007f4e65542305 in ap_mpm_run () > > #6 0x00007f4e65515ee6 in main () > > > > > > > > > > (gdb) where > > #0 0x00007f4e64a213b0 in __accept_nocancel () at > ../sysdeps/unix/syscall-template.S:82 > > #1 0x00007f4e629f42c1 in ?? () from /usr/lib/apache2/modules/mod_cgid.so > > #2 0x00007f4e629f4d87 in ?? () from /usr/lib/apache2/modules/mod_cgid.so > > #3 0x00007f4e6552ad99 in ap_run_post_config () > > #4 0x00007f4e65515ec3 in main () > > > > > > > Hi, > > That's not normal at all for them to hang. What version of Review Board > are you running? > > Can you provide more details about your system? Apache version, Apache > configuration, mod_wsgi vs fastcgi vs mod_python? > > (Btw, this would be best on revie...@googlegroups.com, as it's not > related to the development of the product or add-ons.) > > Christian > > -- > Christian Hammond - chri...@beanbaginc.com > Review Board - https://www.reviewboard.org > Beanbag, Inc. - https://www.beanbaginc.com > > - hide quoted text - > On Fri, Feb 5, 2016 at 9:17 AM, Koushik Roy <roys...@gmail.com> wrote: > >> - hide quoted text - >> > >> At my site where Review Board is deployed, I found that 4 apache threads >> are hung. >> > These threads are running underlying Review Board code. >> > >> Any clue as to how to find out what caused these threads to hang? >> > Any method to attach a debugger to the Python code? >> > >> Thanks for the help. >> > >> Koushik >> > >> >> -- > > >> --- > > -- Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/ Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/ Happy user? Let us know! https://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups "reviewboard" group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
