Re: Review Board 3.0.3 goes down often without showing any error

[Christian Hammond]

Hi Hema,

Whoosh is good for smaller servers, but I imagine you're dealing with a
pretty substantial search index (though that kind of CPU usage doesn't
sound correct even for Whoosh). You're probably best off using a compatible
version of Elasticsearch instead, which is designed to scale.

See
https://www.reviewboard.org/docs/manual/latest/admin/sites/search-indexing/#elasticsearch-configuration

Christian

On Sun, Jul 22, 2018 at 10:23 PM Hema  wrote:

> Hello Christian,
>
> We ensured that we are running the newer Djblets. But still we face the
> server overload issue. Then we got to know that the "Full Text Search" is
> causing this. We use whoosh and we do update index every 10 minutes as
> recommended.
>
> When we hit enter in search box, it forks out a httpd process that uses
> 100% CPU and keep on running. Even when we cancel the search, the process
> don't end. Do you have any suggestions for us to get this resolved?
>
> Thanks,
> Hema.
>
> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "Review Board Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Review Board 3.0.3 goes down often without showing any error

[Hema]

Hello Christian,

We ensured that we are running the newer Djblets. But still we face the 
server overload issue. Then we got to know that the "Full Text Search" is 
causing this. We use whoosh and we do update index every 10 minutes as 
recommended. 

When we hit enter in search box, it forks out a httpd process that uses 
100% CPU and keep on running. Even when we cancel the search, the process 
don't end. Do you have any suggestions for us to get this resolved?

Thanks,
Hema.

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Review Board 3.0.3 goes down often without showing any error

[Christian Hammond]

I'd start by making 100% sure that you're running against the newer
Djblets. That is the only source of deadlocks we've encountered. Make sure
Apache has been completely killed and restarted.

I'd also advise getting the latest versions of both Review Board and
Djblets, as there may be fixes that affected this in other ways. The
Djblets release was developed alongside Review Board.

Christian

On Mon, Jul 9, 2018 at 6:44 AM Hema  wrote:

> Hello Christian,
>
> As per your suggestion on the other email thread, we upgraded Djblets from
> 1.0.3 to 1.0.4 and its dependencies. But even after upgrading it, the
> server load average often reaches 300+ and it goes unresponsive. We don't
> find any error in the logs. Could you please let us know the possible
> reasons for this?
>
> Thanks,
> Hema.
>
> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "Review Board Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Review Board 3.0.3 goes down often without showing any error

[Hema]

Hello Christian,

As per your suggestion on the other email thread, we upgraded Djblets from 
1.0.3 to 1.0.4 and its dependencies. But even after upgrading it, the 
server load average often reaches 300+ and it goes unresponsive. We don't 
find any error in the logs. Could you please let us know the possible 
reasons for this?

Thanks,
Hema.

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Review Board 3.0.3 goes down often without showing any error

[Christian Hammond]

No, I’m sorry. You’ll need to upgrade. The only other option is frequent
restarts of Apache.

The upgrade shouldn’t take more than a couple minutes, though.

Christian


On Tue, Jun 19, 2018 at 22:57 Hema  wrote:

> Hello Christian,
>
> Thanks for the reply.
>
> We cannot do another upgrade, so soon. Is there any other way to prevent
> this?
>
> -Hema.
>
>> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "Review Board Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-- 
-- 
Christian Hammond
President/CEO of Beanbag
Makers of Review Board

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Review Board 3.0.3 goes down often without showing any error

[Christian Hammond]

Hi Hema,

Yes, this is the main symptom of the deadlock bug.

Christian

On Tue, Jun 19, 2018 at 9:50 PM Hema  wrote:

> *2018-06-20 01:47:52,607 - CRITICAL -  - root - Unable to load
> SiteConfiguration: (2013, "Lost connection to MySQL server at 'reading
> authorization packet', system error: 0")*
>
>
>
> *OperationalError: (2013, "Lost connection to MySQL server at 'reading
> authorization packet', system error: 0")*
>
>
> Is this the deadlock bug, you were pointing? When we got this error, the
> server load went up too high and the RB site went inaccessible. We had to
> restart apache to bring the server load down.
>
>
> *-Hema.*
>
> On Thursday, 14 June 2018 13:26:52 UTC+5:30, Christian Hammond wrote:
>>
>> 3.0.3 contains a deadlock bug, which you *will* hit. It will bring down
>> the server periodically. This is fixed in 3.0.4. We always recommend the
>> latest 3.0.x release, though. If there's a major bug, we work to fix it
>> quickly, but I haven't seen any major issues with the most recent release
>> (3.0.7). This release also contains support for enabling GDPR-compliant
>> privacy settings.
>>
>> How is your master-slave replication configured? Can you run that SELECT
>> statement on both databases to verify that the version is 3.0.3 on both,
>> just so we can rule out an inconsistency?
>>
>> Christian
>>
>> On Thu, Jun 14, 2018 at 12:08 AM Hema  wrote:
>>
>>> Thanks for the reply.
>>>
>>> We were running 2.0 in another server. Recently, we had setup 3.0
>>> server; connected it to 2.0's database and upgraded. I'll check whether 2.0
>>> server is conflicting with the new one even though we've turned off apache.
>>>
>>> 1) Are you running Review Board on a single server, or spread across
>>> multiple servers?
>>> - Yes. We are running it on a single server.
>>>
>>> 2) Are you using any type of replication for the database?
>>> - We are using master-slave replication.
>>>
>>> 3) Are the 2.0.12 packages still installed on this server?
>>> - No.
>>>
>>> 4) How do you handle package installation? pip, easy_install, yum?
>>> - easy_install for python packages; yum for others.
>>>
>>> We recently got downtime and upgraded to 3.0.3. We won't get approval so
>>> soon to move to another new version as it requires downtime. Anyway, can
>>> you please let us know whether we have any major issues that brings down RB
>>> is in 3.0.3? Also, let us know which version in 3.0 series is the
>>> recommended or stable one.
>>>
>>> -Hema.
>>>
>>> --
>>> Supercharge your Review Board with Power Pack:
>>> https://www.reviewboard.org/powerpack/
>>> Want us to host Review Board for you? Check out RBCommons:
>>> https://rbcommons.com/
>>> Happy user? Let us know! https://www.reviewboard.org/users/
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Review Board Community" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to reviewboard...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>> --
>> Christian Hammond
>> President/CEO of Beanbag 
>> Makers of Review Board 
>>
> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "Review Board Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Review Board 3.0.3 goes down often without showing any error

[Hema]

*2018-06-20 01:47:52,607 - CRITICAL -  - root - Unable to load 
SiteConfiguration: (2013, "Lost connection to MySQL server at 'reading 
authorization packet', system error: 0")*

 

*OperationalError: (2013, "Lost connection to MySQL server at 'reading 
authorization packet', system error: 0")*


Is this the deadlock bug, you were pointing? When we got this error, the 
server load went up too high and the RB site went inaccessible. We had to 
restart apache to bring the server load down.


*-Hema.*

On Thursday, 14 June 2018 13:26:52 UTC+5:30, Christian Hammond wrote:
>
> 3.0.3 contains a deadlock bug, which you *will* hit. It will bring down 
> the server periodically. This is fixed in 3.0.4. We always recommend the 
> latest 3.0.x release, though. If there's a major bug, we work to fix it 
> quickly, but I haven't seen any major issues with the most recent release 
> (3.0.7). This release also contains support for enabling GDPR-compliant 
> privacy settings.
>
> How is your master-slave replication configured? Can you run that SELECT 
> statement on both databases to verify that the version is 3.0.3 on both, 
> just so we can rule out an inconsistency?
>
> Christian
>
> On Thu, Jun 14, 2018 at 12:08 AM Hema > 
> wrote:
>
>> Thanks for the reply.
>>
>> We were running 2.0 in another server. Recently, we had setup 3.0 server; 
>> connected it to 2.0's database and upgraded. I'll check whether 2.0 server 
>> is conflicting with the new one even though we've turned off apache.
>>
>> 1) Are you running Review Board on a single server, or spread across 
>> multiple servers?
>> - Yes. We are running it on a single server.
>>
>> 2) Are you using any type of replication for the database?
>> - We are using master-slave replication.
>>
>> 3) Are the 2.0.12 packages still installed on this server?
>> - No.
>>
>> 4) How do you handle package installation? pip, easy_install, yum?
>> - easy_install for python packages; yum for others.
>>
>> We recently got downtime and upgraded to 3.0.3. We won't get approval so 
>> soon to move to another new version as it requires downtime. Anyway, can 
>> you please let us know whether we have any major issues that brings down RB 
>> is in 3.0.3? Also, let us know which version in 3.0 series is the 
>> recommended or stable one.
>>
>> -Hema.
>>
>> -- 
>> Supercharge your Review Board with Power Pack: 
>> https://www.reviewboard.org/powerpack/
>> Want us to host Review Board for you? Check out RBCommons: 
>> https://rbcommons.com/
>> Happy user? Let us know! https://www.reviewboard.org/users/
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Review Board Community" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to reviewboard...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> -- 
> Christian Hammond
> President/CEO of Beanbag 
> Makers of Review Board 
>

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Review Board 3.0.3 goes down often without showing any error

[Christian Hammond]

3.0.3 contains a deadlock bug, which you *will* hit. It will bring down the
server periodically. This is fixed in 3.0.4. We always recommend the latest
3.0.x release, though. If there's a major bug, we work to fix it quickly,
but I haven't seen any major issues with the most recent release (3.0.7).
This release also contains support for enabling GDPR-compliant privacy
settings.

How is your master-slave replication configured? Can you run that SELECT
statement on both databases to verify that the version is 3.0.3 on both,
just so we can rule out an inconsistency?

Christian

On Thu, Jun 14, 2018 at 12:08 AM Hema  wrote:

> Thanks for the reply.
>
> We were running 2.0 in another server. Recently, we had setup 3.0 server;
> connected it to 2.0's database and upgraded. I'll check whether 2.0 server
> is conflicting with the new one even though we've turned off apache.
>
> 1) Are you running Review Board on a single server, or spread across
> multiple servers?
> - Yes. We are running it on a single server.
>
> 2) Are you using any type of replication for the database?
> - We are using master-slave replication.
>
> 3) Are the 2.0.12 packages still installed on this server?
> - No.
>
> 4) How do you handle package installation? pip, easy_install, yum?
> - easy_install for python packages; yum for others.
>
> We recently got downtime and upgraded to 3.0.3. We won't get approval so
> soon to move to another new version as it requires downtime. Anyway, can
> you please let us know whether we have any major issues that brings down RB
> is in 3.0.3? Also, let us know which version in 3.0 series is the
> recommended or stable one.
>
> -Hema.
>
> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "Review Board Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Review Board 3.0.3 goes down often without showing any error

[Hema]

Thanks for the reply.

We were running 2.0 in another server. Recently, we had setup 3.0 server; 
connected it to 2.0's database and upgraded. I'll check whether 2.0 server 
is conflicting with the new one even though we've turned off apache.

1) Are you running Review Board on a single server, or spread across 
multiple servers?
- Yes. We are running it on a single server.

2) Are you using any type of replication for the database?
- We are using master-slave replication.

3) Are the 2.0.12 packages still installed on this server?
- No.

4) How do you handle package installation? pip, easy_install, yum?
- easy_install for python packages; yum for others.

We recently got downtime and upgraded to 3.0.3. We won't get approval so 
soon to move to another new version as it requires downtime. Anyway, can 
you please let us know whether we have any major issues that brings down RB 
is in 3.0.3? Also, let us know which version in 3.0 series is the 
recommended or stable one.

-Hema.

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Review Board 3.0.3 goes down often without showing any error

[Christian Hammond]

Hi,

This is definitely not normal, and points to a conflict somewhere in the
infrastructure. Is there any other instance anywhere, any tooling, any test
server that's talking to the same database? It looks as if somewhere, a
2.0.12 instance is running and setting its version in the database.

When this happens next, can you log into the database and do:

SELECT version FROM siteconfig_siteconfiguration;

That will help to determine if it's actually being changed in the database
or if something else is going on.

A few other questions:

1) Are you running Review Board on a single server, or spread across
multiple servers?

2) Are you using any type of replication for the database?

3) Are the 2.0.12 packages still installed on this server?

4) How do you handle package installation? pip, easy_install, yum?

I'd also recommend upgrading to a newer 3.0.x release. There's some other
(unrelated but problematic) bugs in 3.0.3.

Christian

On Wed, Jun 13, 2018 at 11:25 PM Hema  wrote:

> Review Board 3.0.3 goes down often and it shows the attached error page.
> When I checked, both the web server and rb-site uses the same Python
> version.
> Error logs (httpd, ssl, reviewboard) are clean and also we don't get any
> error e-mail. And when it shows this page, just the apache restart doesn't
> work. We have to run rb-site upgrade command to get it up.
>
> Can you please provide us some pointers on what could be the underlying
> issue?
>
> Thanks,
> Hemapriya.
>
>
> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "Review Board Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.