Re: update_index always indexes the users class

[Christian Hammond]

You can apply these by going to the root of your Review Board package
(probably /usr/lib/python2.7/site-packages/ReviewBoard-2.5.9-py2.7.egg) and
running:

patch -p1 < *.patch

Probably should back up your package directory just in case first.

If the patches apply successfully, perform a complete re-index and see what
that looks like. There should be a significant performance boost.

Christian

-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

On Tue, May 2, 2017 at 6:12 AM, Daniel Kuecker  wrote:

> Sure thing.
>
> Python: 2.7.5.
> I used easy_install to install RB.
>
> On Monday, May 1, 2017 at 4:28:14 PM UTC-5, Christian Hammond wrote:
>>
>> Actually, rather than the Elasticsearch patch, I'd like you to try a
>> couple patches to the search indexing code. Turns out it was more
>> inefficient than I had thought. Alternatively, I could send you a build.
>> Just need to know the Python version and whether you used easy_install or
>> pip to install Review Board (or some other method).
>>
>> Christian
>>
>> --
>> Christian Hammond
>> President/CEO of Beanbag 
>> Makers of Review Board 
>>
>> On Mon, May 1, 2017 at 1:28 PM, Christian Hammond > > wrote:
>>
>>> Sure thing. I realized though that out of the box, this won't work
>>> without a patch to Review Board (the proper Elasticsearch support has only
>>> been thoroughly tested in 3.0). How comfortable do you feel applying
>>> patches?
>>>
>>> Christian
>>>
>>>
>>> --
>>> Christian Hammond
>>> President/CEO of Beanbag 
>>> Makers of Review Board 
>>>
>>> On Mon, May 1, 2017 at 1:17 PM, Daniel Kuecker 
>>> wrote:
>>>
 Thanks for the quick reply Christian!

 I would like the info on configuring Elasticsearch if you don't mind.

 --DK

 On Monday, May 1, 2017 at 3:12:06 PM UTC-5, Christian Hammond wrote:
>
> Hi Daniel,
>
> I believe this happens because unlike review requests, we have no way
> of determining with the search indexing stuff we use if a user's
> information is "new" to the index. Basically, we have no "last updated" 
> for
> the users.
>
> I do see an area in the search index for users that isn't as optimal,
> so we can probably shave down the number of queries a fair amount, which
> will help.
>
> I assume you're using the default search backend of Whoosh? If so, at
> your scale, it might be time to install Elasticsearch. We can give you 
> some
> info on how to configure that in settings_local.py (the upcoming Review
> Board 3.0 makes this very easy -- just select it as your search backend 
> and
> fill in fields). Indexing with Elasticsearch is more efficient for large
> amounts of data.
>
> Christian
>
> --
> Christian Hammond
> President/CEO of Beanbag 
> Makers of Review Board 
>
> On Mon, May 1, 2017 at 12:59 PM, Daniel Kuecker 
> wrote:
>
>> I have a fresh 2.5.9 install with a db taken from a backup of another
>> 2.5.9 install (basically a backup of my production instance).
>>
>> On the production instance, I am experiencing 100% cpu usage every
>> time the update_index cron job runs causing my users to complain about
>> responsiveness of the site.
>>
>> On my production system, the cron job shows it is indexing all the
>> records in the user table while only the new records in the review 
>> requests
>> table.
>>
>> For debugging, I restored the production db backup to my dev machine
>> and set up a 2.5.9 instance pointing to it.  When update_index runs on 
>> the
>> dev box (given the "-v 3" arg), I get the following warning:
>>
>>>
>>> WARNING:py.warnings:/usr/lib/python2.7/site-packages/haystack/indexes.py:167:
>>> UserWarning: No updated date field found for 'User' - not restricting by
>>> age.
>>>   warnings.warn(update_field_msg)
>>> Indexing 1971 users
>>>   indexed 1 - 1000 of 1971 (by 16819).
>>>   indexed 1001 - 1971 of 1971 (by 16819).
>>
>>
>> It then proceeds to index the entire user table and only the changed
>> review requests just as my production instance does.
>>
>> I have run the rebuild_index command on both systems.
>>
>> Am I missing something?  I have the cron job set to run ever 1/2 hour
>> and am about to have to turn search indexing off to ensure the site is
>> responsive.
>>
>> --DK
>>
>>
>>
>> --
>> Supercharge your Review Board with Power Pack:
>> https://www.reviewboard.org/powerpack/
>> Want us to host Review Board for you? Check out 

Re: update_index always indexes the users class

[Daniel Kuecker]

Sure thing. 

Python: 2.7.5.  
I used easy_install to install RB.

On Monday, May 1, 2017 at 4:28:14 PM UTC-5, Christian Hammond wrote:
>
> Actually, rather than the Elasticsearch patch, I'd like you to try a 
> couple patches to the search indexing code. Turns out it was more 
> inefficient than I had thought. Alternatively, I could send you a build. 
> Just need to know the Python version and whether you used easy_install or 
> pip to install Review Board (or some other method).
>
> Christian
>
> -- 
> Christian Hammond
> President/CEO of Beanbag 
> Makers of Review Board 
>
> On Mon, May 1, 2017 at 1:28 PM, Christian Hammond  > wrote:
>
>> Sure thing. I realized though that out of the box, this won't work 
>> without a patch to Review Board (the proper Elasticsearch support has only 
>> been thoroughly tested in 3.0). How comfortable do you feel applying 
>> patches?
>>
>> Christian
>>
>>
>> -- 
>> Christian Hammond
>> President/CEO of Beanbag 
>> Makers of Review Board 
>>
>> On Mon, May 1, 2017 at 1:17 PM, Daniel Kuecker > > wrote:
>>
>>> Thanks for the quick reply Christian!
>>>
>>> I would like the info on configuring Elasticsearch if you don't mind.  
>>>
>>> --DK
>>>
>>> On Monday, May 1, 2017 at 3:12:06 PM UTC-5, Christian Hammond wrote:

 Hi Daniel,

 I believe this happens because unlike review requests, we have no way 
 of determining with the search indexing stuff we use if a user's 
 information is "new" to the index. Basically, we have no "last updated" 
 for 
 the users.

 I do see an area in the search index for users that isn't as optimal, 
 so we can probably shave down the number of queries a fair amount, which 
 will help.

 I assume you're using the default search backend of Whoosh? If so, at 
 your scale, it might be time to install Elasticsearch. We can give you 
 some 
 info on how to configure that in settings_local.py (the upcoming Review 
 Board 3.0 makes this very easy -- just select it as your search backend 
 and 
 fill in fields). Indexing with Elasticsearch is more efficient for large 
 amounts of data.

 Christian

 -- 
 Christian Hammond
 President/CEO of Beanbag 
 Makers of Review Board 

 On Mon, May 1, 2017 at 12:59 PM, Daniel Kuecker  
 wrote:

> I have a fresh 2.5.9 install with a db taken from a backup of another 
> 2.5.9 install (basically a backup of my production instance).
>
> On the production instance, I am experiencing 100% cpu usage every 
> time the update_index cron job runs causing my users to complain about 
> responsiveness of the site.
>
> On my production system, the cron job shows it is indexing all the 
> records in the user table while only the new records in the review 
> requests 
> table.
>
> For debugging, I restored the production db backup to my dev machine 
> and set up a 2.5.9 instance pointing to it.  When update_index runs on 
> the 
> dev box (given the "-v 3" arg), I get the following warning: 
>
>>
>> WARNING:py.warnings:/usr/lib/python2.7/site-packages/haystack/indexes.py:167:
>>  
>> UserWarning: No updated date field found for 'User' - not restricting by 
>> age.
>>   warnings.warn(update_field_msg)
>> Indexing 1971 users
>>   indexed 1 - 1000 of 1971 (by 16819).
>>   indexed 1001 - 1971 of 1971 (by 16819).
>
>
> It then proceeds to index the entire user table and only the changed 
> review requests just as my production instance does.
>
> I have run the rebuild_index command on both systems.
>
> Am I missing something?  I have the cron job set to run ever 1/2 hour 
> and am about to have to turn search indexing off to ensure the site is 
> responsive.
>
> --DK
>
>
>
> -- 
> Supercharge your Review Board with Power Pack: 
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons: 
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> --- 
> You received this message because you are subscribed to the Google 
> Groups "reviewboard" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to reviewboard...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

 -- 
>>> Supercharge your Review Board with Power Pack: 
>>> https://www.reviewboard.org/powerpack/
>>> Want us to host Review Board for you? Check out RBCommons: 
>>> https://rbcommons.com/
>>> Happy user? Let us know! https://www.reviewboard.org/users/

Re: update_index always indexes the users class

[Christian Hammond]

Actually, rather than the Elasticsearch patch, I'd like you to try a couple
patches to the search indexing code. Turns out it was more inefficient than
I had thought. Alternatively, I could send you a build. Just need to know
the Python version and whether you used easy_install or pip to install
Review Board (or some other method).

Christian

-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

On Mon, May 1, 2017 at 1:28 PM, Christian Hammond 
wrote:

> Sure thing. I realized though that out of the box, this won't work without
> a patch to Review Board (the proper Elasticsearch support has only been
> thoroughly tested in 3.0). How comfortable do you feel applying patches?
>
> Christian
>
>
> --
> Christian Hammond
> President/CEO of Beanbag 
> Makers of Review Board 
>
> On Mon, May 1, 2017 at 1:17 PM, Daniel Kuecker  wrote:
>
>> Thanks for the quick reply Christian!
>>
>> I would like the info on configuring Elasticsearch if you don't mind.
>>
>> --DK
>>
>> On Monday, May 1, 2017 at 3:12:06 PM UTC-5, Christian Hammond wrote:
>>>
>>> Hi Daniel,
>>>
>>> I believe this happens because unlike review requests, we have no way of
>>> determining with the search indexing stuff we use if a user's information
>>> is "new" to the index. Basically, we have no "last updated" for the users.
>>>
>>> I do see an area in the search index for users that isn't as optimal, so
>>> we can probably shave down the number of queries a fair amount, which will
>>> help.
>>>
>>> I assume you're using the default search backend of Whoosh? If so, at
>>> your scale, it might be time to install Elasticsearch. We can give you some
>>> info on how to configure that in settings_local.py (the upcoming Review
>>> Board 3.0 makes this very easy -- just select it as your search backend and
>>> fill in fields). Indexing with Elasticsearch is more efficient for large
>>> amounts of data.
>>>
>>> Christian
>>>
>>> --
>>> Christian Hammond
>>> President/CEO of Beanbag 
>>> Makers of Review Board 
>>>
>>> On Mon, May 1, 2017 at 12:59 PM, Daniel Kuecker 
>>> wrote:
>>>
 I have a fresh 2.5.9 install with a db taken from a backup of another
 2.5.9 install (basically a backup of my production instance).

 On the production instance, I am experiencing 100% cpu usage every time
 the update_index cron job runs causing my users to complain about
 responsiveness of the site.

 On my production system, the cron job shows it is indexing all the
 records in the user table while only the new records in the review requests
 table.

 For debugging, I restored the production db backup to my dev machine
 and set up a 2.5.9 instance pointing to it.  When update_index runs on the
 dev box (given the "-v 3" arg), I get the following warning:

>
> WARNING:py.warnings:/usr/lib/python2.7/site-packages/haystack/indexes.py:167:
> UserWarning: No updated date field found for 'User' - not restricting by
> age.
>   warnings.warn(update_field_msg)
> Indexing 1971 users
>   indexed 1 - 1000 of 1971 (by 16819).
>   indexed 1001 - 1971 of 1971 (by 16819).


 It then proceeds to index the entire user table and only the changed
 review requests just as my production instance does.

 I have run the rebuild_index command on both systems.

 Am I missing something?  I have the cron job set to run ever 1/2 hour
 and am about to have to turn search indexing off to ensure the site is
 responsive.

 --DK



 --
 Supercharge your Review Board with Power Pack:
 https://www.reviewboard.org/powerpack/
 Want us to host Review Board for you? Check out RBCommons:
 https://rbcommons.com/
 Happy user? Let us know! https://www.reviewboard.org/users/
 ---
 You received this message because you are subscribed to the Google
 Groups "reviewboard" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to reviewboard...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>> --
>> Supercharge your Review Board with Power Pack:
>> https://www.reviewboard.org/powerpack/
>> Want us to host Review Board for you? Check out RBCommons:
>> https://rbcommons.com/
>> Happy user? Let us know! https://www.reviewboard.org/users/
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "reviewboard" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to reviewboard+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Supercharge your Review Board with Power Pack: 

Re: update_index always indexes the users class

[Daniel Kuecker]

Thanks for the quick reply Christian!

I would like the info on configuring Elasticsearch if you don't mind.  

--DK

On Monday, May 1, 2017 at 3:12:06 PM UTC-5, Christian Hammond wrote:
>
> Hi Daniel,
>
> I believe this happens because unlike review requests, we have no way of 
> determining with the search indexing stuff we use if a user's information 
> is "new" to the index. Basically, we have no "last updated" for the users.
>
> I do see an area in the search index for users that isn't as optimal, so 
> we can probably shave down the number of queries a fair amount, which will 
> help.
>
> I assume you're using the default search backend of Whoosh? If so, at your 
> scale, it might be time to install Elasticsearch. We can give you some info 
> on how to configure that in settings_local.py (the upcoming Review Board 
> 3.0 makes this very easy -- just select it as your search backend and fill 
> in fields). Indexing with Elasticsearch is more efficient for large amounts 
> of data.
>
> Christian
>
> -- 
> Christian Hammond
> President/CEO of Beanbag 
> Makers of Review Board 
>
> On Mon, May 1, 2017 at 12:59 PM, Daniel Kuecker  > wrote:
>
>> I have a fresh 2.5.9 install with a db taken from a backup of another 
>> 2.5.9 install (basically a backup of my production instance).
>>
>> On the production instance, I am experiencing 100% cpu usage every time 
>> the update_index cron job runs causing my users to complain about 
>> responsiveness of the site.
>>
>> On my production system, the cron job shows it is indexing all the 
>> records in the user table while only the new records in the review requests 
>> table.
>>
>> For debugging, I restored the production db backup to my dev machine and 
>> set up a 2.5.9 instance pointing to it.  When update_index runs on the dev 
>> box (given the "-v 3" arg), I get the following warning: 
>>
>>>
>>> WARNING:py.warnings:/usr/lib/python2.7/site-packages/haystack/indexes.py:167:
>>>  
>>> UserWarning: No updated date field found for 'User' - not restricting by 
>>> age.
>>>   warnings.warn(update_field_msg)
>>> Indexing 1971 users
>>>   indexed 1 - 1000 of 1971 (by 16819).
>>>   indexed 1001 - 1971 of 1971 (by 16819).
>>
>>
>> It then proceeds to index the entire user table and only the changed 
>> review requests just as my production instance does.
>>
>> I have run the rebuild_index command on both systems.
>>
>> Am I missing something?  I have the cron job set to run ever 1/2 hour and 
>> am about to have to turn search indexing off to ensure the site is 
>> responsive.
>>
>> --DK
>>
>>
>>
>> -- 
>> Supercharge your Review Board with Power Pack: 
>> https://www.reviewboard.org/powerpack/
>> Want us to host Review Board for you? Check out RBCommons: 
>> https://rbcommons.com/
>> Happy user? Let us know! https://www.reviewboard.org/users/
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "reviewboard" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to reviewboard...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: update_index always indexes the users class

[Christian Hammond]

Hi Daniel,

I believe this happens because unlike review requests, we have no way of
determining with the search indexing stuff we use if a user's information
is "new" to the index. Basically, we have no "last updated" for the users.

I do see an area in the search index for users that isn't as optimal, so we
can probably shave down the number of queries a fair amount, which will
help.

I assume you're using the default search backend of Whoosh? If so, at your
scale, it might be time to install Elasticsearch. We can give you some info
on how to configure that in settings_local.py (the upcoming Review Board
3.0 makes this very easy -- just select it as your search backend and fill
in fields). Indexing with Elasticsearch is more efficient for large amounts
of data.

Christian

-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

On Mon, May 1, 2017 at 12:59 PM, Daniel Kuecker  wrote:

> I have a fresh 2.5.9 install with a db taken from a backup of another
> 2.5.9 install (basically a backup of my production instance).
>
> On the production instance, I am experiencing 100% cpu usage every time
> the update_index cron job runs causing my users to complain about
> responsiveness of the site.
>
> On my production system, the cron job shows it is indexing all the records
> in the user table while only the new records in the review requests table.
>
> For debugging, I restored the production db backup to my dev machine and
> set up a 2.5.9 instance pointing to it.  When update_index runs on the dev
> box (given the "-v 3" arg), I get the following warning:
>
>>
>> WARNING:py.warnings:/usr/lib/python2.7/site-packages/haystack/indexes.py:167:
>> UserWarning: No updated date field found for 'User' - not restricting by
>> age.
>>   warnings.warn(update_field_msg)
>> Indexing 1971 users
>>   indexed 1 - 1000 of 1971 (by 16819).
>>   indexed 1001 - 1971 of 1971 (by 16819).
>
>
> It then proceeds to index the entire user table and only the changed
> review requests just as my production instance does.
>
> I have run the rebuild_index command on both systems.
>
> Am I missing something?  I have the cron job set to run ever 1/2 hour and
> am about to have to turn search indexing off to ensure the site is
> responsive.
>
> --DK
>
>
>
> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "reviewboard" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

update_index always indexes the users class

[Daniel Kuecker]

I have a fresh 2.5.9 install with a db taken from a backup of another 2.5.9 
install (basically a backup of my production instance).

On the production instance, I am experiencing 100% cpu usage every time the 
update_index cron job runs causing my users to complain about 
responsiveness of the site.

On my production system, the cron job shows it is indexing all the records 
in the user table while only the new records in the review requests table.

For debugging, I restored the production db backup to my dev machine and 
set up a 2.5.9 instance pointing to it.  When update_index runs on the dev 
box (given the "-v 3" arg), I get the following warning: 

>
> WARNING:py.warnings:/usr/lib/python2.7/site-packages/haystack/indexes.py:167: 
> UserWarning: No updated date field found for 'User' - not restricting by 
> age.
>   warnings.warn(update_field_msg)
> Indexing 1971 users
>   indexed 1 - 1000 of 1971 (by 16819).
>   indexed 1001 - 1971 of 1971 (by 16819).


It then proceeds to index the entire user table and only the changed review 
requests just as my production instance does.

I have run the rebuild_index command on both systems.

Am I missing something?  I have the cron job set to run ever 1/2 hour and 
am about to have to turn search indexing off to ensure the site is 
responsive.

--DK



-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.