Re: RB 3.0.11 causes Apache to spike to 100% and deadlock?

2018-12-18 Thread Hema 
Check your logs like httpd/error.log, ssl access log, ssl error log and 
/var/log/messages. You might get some hint on the root cause. We did the 
same when we faced the spikes.

Regards,
Hema.

On Thursday, 13 December 2018 22:05:38 UTC+5:30, pisymbol wrote:
>
> CentOS 7.6, latest everything including Djblets (1.0.8). RB was installed 
> via pip.
>
> What happens is a developer gets a RB email, clicks the link, which causes 
> several httpd processes to spike ~100% and just sit there forever:
>
> 16315 apache20   0  579684 137944  11248 R  93.8  0.8   0:30.93 httpd
> 16316 apache20   0  552804 111500  11172 R  93.8  0.7   1:22.86 httpd
>
> Box has 16GB of RAM allocated and 16GB of swap none of which is being used.
>
> On the developer's screen, I see that half of the diffs are still being 
> loaded (the spinning arrow of death?).
>
> One of our developers suggested upgraded Djblets (my guess is he read the 
> same thread I did) but it is at the latest as far as pip is concerned:
>
> [root@rboard reviewboard]# pip install --upgrade djblets
> Requirement already up-to-date: djblets in 
> /usr/lib/python2.7/site-packages (1.0.8)
> Requirement already satisfied, skipping upgrade: 
> django-pipeline<1.6.999,>=1.6.14 in /usr/lib/python2.7/site-packages (from 
> djblets) (1.6.14)
> Requirement already satisfied, skipping upgrade: Django<1.10.999,>=1.6.11 
> in /usr/lib/python2.7/site-packages (from djblets) (1.6.11)
> Requirement already satisfied, skipping upgrade: publicsuffix>=1.1 in 
> /usr/lib/python2.7/site-packages (from djblets) (1.1.0)
> Requirement already satisfied, skipping upgrade: pytz in 
> /usr/lib/python2.7/site-packages (from djblets) (2018.7)
> Requirement already satisfied, skipping upgrade: feedparser>=5.1.2 in 
> /usr/lib/python2.7/site-packages (from djblets) (5.2.1)
> Requirement already satisfied, skipping upgrade: python-dateutil>=1.5 in 
> /usr/lib/python2.7/site-packages (from djblets) (2.7.5)
> Requirement already satisfied, skipping upgrade: pillowfight in 
> /usr/lib/python2.7/site-packages (from djblets) (0.3)
> Requirement already satisfied, skipping upgrade: dnspython>=1.14.0 in 
> /usr/lib/python2.7/site-packages (from djblets) (1.16.0)
> Requirement already satisfied, skipping upgrade: futures>=2.1.3; 
> python_version < "3" in /usr/lib/python2.7/site-packages (from 
> django-pipeline<1.6.999,>=1.6.14->djblets) (3.2.0)
> Requirement already satisfied, skipping upgrade: six>=1.5 in 
> /usr/lib/python2.7/site-packages (from python-dateutil>=1.5->djblets) 
> (1.12.0)
> Requirement already satisfied, skipping upgrade: Pillow>=3.4.2 in 
> /usr/lib64/python2.7/site-packages (from pillowfight->djblets) (5.3.0)
>
> What's interesting is that this review has two pages worth of diffs and if 
> I click on the first page, all the diffs get generated. But if I click the 
> second one, it causes Apache grief.
>
> Debug logs show:
>
> 2018-12-13 16:29:29,869 - DEBUG - None - alexs - 
> /api/review-requests/229/diff-context/ - root - Generating diff viewer page 
> for filediff id 131
> 2018-12-13 16:29:29,983 - DEBUG - None - alexs - 
> /api/review-requests/229/diff-context/ - root - Begin: Generating diff file 
> info for diffset id 131
> 2018-12-13 16:29:29,988 - DEBUG - None - alexs - 
> /api/review-requests/229/diff-context/ - root - End: Generating diff file 
> info for diffset id 131
> 2018-12-13 16:29:29,988 - DEBUG - None - alexs - 
> /api/review-requests/229/diff-context/ - root - Generating diff file info 
> for diffset id 131 took 0.004751 seconds
> 2018-12-13 16:29:30,069 - DEBUG - None - alexs - 
> /api/review-requests/229/diff-context/ - root - Done generating diff viewer 
> page for filediff id 131
> 2018-12-13 16:29:30,692 - DEBUG - None - alexs - 
> /r/229/diff/1/fragment/784/ - root - Begin: Generating diff file info for 
> diffset id 131, filediff 784
> 2018-12-13 16:29:30,702 - DEBUG - None - alexs - 
> /r/229/diff/1/fragment/784/ - root - End: Generating diff file info for 
> diffset id 131, filediff 784
> 2018-12-13 16:29:30,702 - DEBUG - None - alexs - 
> /r/229/diff/1/fragment/784/ - root - Generating diff file info for diffset 
> id 131, filediff 784 took 0.010163 seconds
> 2018-12-13 16:29:30,704 - DEBUG -  - djblets.cache.backend - Cache miss 
> for key 
> diffviewer/diff_file_fragment.html-19-1-784-collapsed-highlighting-en-1544644925.
> 2018-12-13 16:29:30,705 - DEBUG -  - djblets.cache.backend - Cache miss 
> for key diff-sidebyside-hl-784-en.
> 2018-12-13 16:29:30,741 - DEBUG - None - alexs - 
> /r/229/diff/1/fragment/784/ - root - Begin: Patching file 
> /trunk/ie/include/crawl_jrpc.j
> 2018-12-13 16:29:30,756 - DEBUG - None - alexs - 
> /r/229/diff/1/fragment/784/ - root - End: Patching file 
> /trunk/ie/include/crawl_jrpc.j
> 2018-12-13 16:29:30,756 - DEBUG - None - alexs - 
> /r/229/diff/1/fragment/784/ - root - Patching file 
> /trunk/ie/include/crawl_jrpc.j took 0.014740 seconds
> 2018-12-13 16:29:30,757 - DEBUG - None - alexs - 
>

Re: RB 3.0.11 causes Apache to spike to 100% and deadlock?

2018-12-17 Thread Christian Hammond 
I've been trying to investigate any changes that could account for this,
but there's simply nothing in 3.0.11 that should have triggered any
behavior here.

We did have a bug many releases ago that would absolutely cause deadlocks,
but I'm confident we've solved that. We haven't seen any new occurrences of
that bug on any of our servers or on any of the large servers managed by
customers we work with. That was the bug resulting in an "upgrade Djblets"
recommendation, but modern versions all have the fix.

When you restart Apache and view the same diffs, do they load?

Christian

On Mon, Dec 17, 2018 at 8:13 AM pisymbol  wrote:

>
>
> On Mon, Dec 17, 2018 at 10:18 AM Koushik Roy  wrote:
>
>> Just in case this helps -
>>
>> I have experienced that memcached sometimes looses mind with inexplicable
>> behavior showing up in RB.
>> I restarted memcached and things were fine.
>> Of course this does not help debugging, but you can get going at least.
>>
>
> Thanks KR. I did of course reboot the whole machine when it moved to 7.6
> but the issue still persists.
>
> Still unclear how to debug it though since I'm not seeing anything in the
> RB log file or Apache (I will check again though).
>
> -aps
>
> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "Review Board Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RB 3.0.11 causes Apache to spike to 100% and deadlock?

2018-12-17 Thread pisymbol 
On Mon, Dec 17, 2018 at 10:18 AM Koushik Roy  wrote:

> Just in case this helps -
>
> I have experienced that memcached sometimes looses mind with inexplicable
> behavior showing up in RB.
> I restarted memcached and things were fine.
> Of course this does not help debugging, but you can get going at least.
>

Thanks KR. I did of course reboot the whole machine when it moved to 7.6
but the issue still persists.

Still unclear how to debug it though since I'm not seeing anything in the
RB log file or Apache (I will check again though).

-aps

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RB 3.0.11 causes Apache to spike to 100% and deadlock?

2018-12-17 Thread Koushik Roy 
Just in case this helps -

I have experienced that memcached sometimes looses mind with inexplicable 
behavior showing up in RB.
I restarted memcached and things were fine.
Of course this does not help debugging, but you can get going at least.

Thanks

Koushik


On Friday, December 14, 2018 at 7:12:53 PM UTC-5, pisymbol wrote:
>
>
>
> On Fri, Dec 14, 2018 at 4:04 PM Christian Hammond  > wrote:
>
>> Does this happen with any particular files in the repository? Is there 
>> anything there that may be common amongst the ones that are failing?
>>
>
> Nothing is jumping out. 
>
>>
>> There's definitely no change in 3.0.11 that impact the diff viewer code, 
>> or repository-related operations.
>>
>> Just to sanity check, was Apache restarted after upgrading?
>>
>
> Several times at this point.
>
> -aps
>

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RB 3.0.11 causes Apache to spike to 100% and deadlock?

2018-12-14 Thread pisymbol 
On Fri, Dec 14, 2018 at 4:04 PM Christian Hammond 
wrote:

> Does this happen with any particular files in the repository? Is there
> anything there that may be common amongst the ones that are failing?
>

Nothing is jumping out.

>
> There's definitely no change in 3.0.11 that impact the diff viewer code,
> or repository-related operations.
>
> Just to sanity check, was Apache restarted after upgrading?
>

Several times at this point.

-aps

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RB 3.0.11 causes Apache to spike to 100% and deadlock?

2018-12-14 Thread Christian Hammond 
Does this happen with any particular files in the repository? Is there
anything there that may be common amongst the ones that are failing?

There's definitely no change in 3.0.11 that impact the diff viewer code, or
repository-related operations.

Just to sanity check, was Apache restarted after upgrading?

Christian

On Fri, Dec 14, 2018 at 6:06 AM pisymbol  wrote:

>
>
> On Thursday, December 13, 2018 at 7:03:33 PM UTC-5, Christian Hammond
> wrote:
>>
>> Hi,
>>
>> Hmm, none of the changes we put into 3.0.11 should have any chance of
>> doing this. We didn't bump Djblets for this release, so it should be using
>> the same version as 3.0.10, so at least we can rule that out.
>>
>
> Ok.
>
>
>> Some starter questions, just for info gathering:
>>
>> 1) What version of Review Board did you upgrade from?
>>
>
> 3.0.10 -> 3.0.11
>
> 2) What database are you using?
>>
>
> postgres (CentOS 7.6 version)
>
> 3) Can you verify that memcached is being used (Admin UI -> Cache
>> Statistics)
>>
>
> It is.
>
> 4) Are you using any SCM other than Subversion?
>>
>
> I wish.
>
> -aps
>
> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "Review Board Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RB 3.0.11 causes Apache to spike to 100% and deadlock?

2018-12-14 Thread pisymbol 


On Thursday, December 13, 2018 at 7:03:33 PM UTC-5, Christian Hammond wrote:
>
> Hi,
>
> Hmm, none of the changes we put into 3.0.11 should have any chance of 
> doing this. We didn't bump Djblets for this release, so it should be using 
> the same version as 3.0.10, so at least we can rule that out.
>

Ok.


> Some starter questions, just for info gathering:
>
> 1) What version of Review Board did you upgrade from?
>

3.0.10 -> 3.0.11

2) What database are you using?
>

postgres (CentOS 7.6 version)

3) Can you verify that memcached is being used (Admin UI -> Cache 
> Statistics)
>

It is.

4) Are you using any SCM other than Subversion?
>

I wish.

-aps

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RB 3.0.11 causes Apache to spike to 100% and deadlock?

2018-12-13 Thread Christian Hammond 
Hi,

Hmm, none of the changes we put into 3.0.11 should have any chance of doing
this. We didn't bump Djblets for this release, so it should be using the
same version as 3.0.10, so at least we can rule that out.

Some starter questions, just for info gathering:

1) What version of Review Board did you upgrade from?
2) What database are you using?
3) Can you verify that memcached is being used (Admin UI -> Cache
Statistics)
4) Are you using any SCM other than Subversion?

Christian

On Thu, Dec 13, 2018 at 8:35 AM pisymbol  wrote:

> CentOS 7.6, latest everything including Djblets (1.0.8). RB was installed
> via pip.
>
> What happens is a developer gets a RB email, clicks the link, which causes
> several httpd processes to spike ~100% and just sit there forever:
>
> 16315 apache20   0  579684 137944  11248 R  93.8  0.8   0:30.93 httpd
> 16316 apache20   0  552804 111500  11172 R  93.8  0.7   1:22.86 httpd
>
> Box has 16GB of RAM allocated and 16GB of swap none of which is being used.
>
> On the developer's screen, I see that half of the diffs are still being
> loaded (the spinning arrow of death?).
>
> One of our developers suggested upgraded Djblets (my guess is he read the
> same thread I did) but it is at the latest as far as pip is concerned:
>
> [root@rboard reviewboard]# pip install --upgrade djblets
> Requirement already up-to-date: djblets in
> /usr/lib/python2.7/site-packages (1.0.8)
> Requirement already satisfied, skipping upgrade:
> django-pipeline<1.6.999,>=1.6.14 in /usr/lib/python2.7/site-packages (from
> djblets) (1.6.14)
> Requirement already satisfied, skipping upgrade: Django<1.10.999,>=1.6.11
> in /usr/lib/python2.7/site-packages (from djblets) (1.6.11)
> Requirement already satisfied, skipping upgrade: publicsuffix>=1.1 in
> /usr/lib/python2.7/site-packages (from djblets) (1.1.0)
> Requirement already satisfied, skipping upgrade: pytz in
> /usr/lib/python2.7/site-packages (from djblets) (2018.7)
> Requirement already satisfied, skipping upgrade: feedparser>=5.1.2 in
> /usr/lib/python2.7/site-packages (from djblets) (5.2.1)
> Requirement already satisfied, skipping upgrade: python-dateutil>=1.5 in
> /usr/lib/python2.7/site-packages (from djblets) (2.7.5)
> Requirement already satisfied, skipping upgrade: pillowfight in
> /usr/lib/python2.7/site-packages (from djblets) (0.3)
> Requirement already satisfied, skipping upgrade: dnspython>=1.14.0 in
> /usr/lib/python2.7/site-packages (from djblets) (1.16.0)
> Requirement already satisfied, skipping upgrade: futures>=2.1.3;
> python_version < "3" in /usr/lib/python2.7/site-packages (from
> django-pipeline<1.6.999,>=1.6.14->djblets) (3.2.0)
> Requirement already satisfied, skipping upgrade: six>=1.5 in
> /usr/lib/python2.7/site-packages (from python-dateutil>=1.5->djblets)
> (1.12.0)
> Requirement already satisfied, skipping upgrade: Pillow>=3.4.2 in
> /usr/lib64/python2.7/site-packages (from pillowfight->djblets) (5.3.0)
>
> What's interesting is that this review has two pages worth of diffs and if
> I click on the first page, all the diffs get generated. But if I click the
> second one, it causes Apache grief.
>
> Debug logs show:
>
> 2018-12-13 16:29:29,869 - DEBUG - None - alexs -
> /api/review-requests/229/diff-context/ - root - Generating diff viewer page
> for filediff id 131
> 2018-12-13 16:29:29,983 - DEBUG - None - alexs -
> /api/review-requests/229/diff-context/ - root - Begin: Generating diff file
> info for diffset id 131
> 2018-12-13 16:29:29,988 - DEBUG - None - alexs -
> /api/review-requests/229/diff-context/ - root - End: Generating diff file
> info for diffset id 131
> 2018-12-13 16:29:29,988 - DEBUG - None - alexs -
> /api/review-requests/229/diff-context/ - root - Generating diff file info
> for diffset id 131 took 0.004751 seconds
> 2018-12-13 16:29:30,069 - DEBUG - None - alexs -
> /api/review-requests/229/diff-context/ - root - Done generating diff viewer
> page for filediff id 131
> 2018-12-13 16:29:30,692 - DEBUG - None - alexs -
> /r/229/diff/1/fragment/784/ - root - Begin: Generating diff file info for
> diffset id 131, filediff 784
> 2018-12-13 16:29:30,702 - DEBUG - None - alexs -
> /r/229/diff/1/fragment/784/ - root - End: Generating diff file info for
> diffset id 131, filediff 784
> 2018-12-13 16:29:30,702 - DEBUG - None - alexs -
> /r/229/diff/1/fragment/784/ - root - Generating diff file info for diffset
> id 131, filediff 784 took 0.010163 seconds
> 2018-12-13 16:29:30,704 - DEBUG -  - djblets.cache.backend - Cache miss
> for key
> diffviewer/diff_file_fragment.html-19-1-784-collapsed-highlighting-en-1544644925.
> 2018-12-13 16:29:30,705 - DEBUG -  - djblets.cache.backend - Cache miss
> for key diff-sidebyside-hl-784-en.
> 2018-12-13 16:29:30,741 - DEBUG - None - alexs -
> /r/229/diff/1/fragment/784/ - root - Begin: Patching file
> /trunk/ie/include/crawl_jrpc.j
> 2018-12-13 16:29:30,756 - DEBUG - None - alexs -
> /r/229/diff/1/fragment/784/ - root - End: Patching file
> /trunk/ie/include/crawl_jrpc.j