Thanks Pat. I'll take a closer look at the logs.
I asked about the index vs rebuild because I wanted to have monit restart sphinx if it crashes (its been segfaulting lately on centOS), but I don't want monit to prematurely start it during a rebuild. I'll take a closer look. On Feb 17, 5:18 pm, "Pat Allan" <[email protected]> wrote: > On 18/02/2012, at 5:18 AM, gmoniey wrote: > > > Thanks Pat, > > > In regards to speed, is there anything I can do, or is it just a > > limitation of sphinx? Also, any idea why the db is getting hit so > > hard. I haven't got a chance to look into the logs yet to see if its > > running any slows queries, but I would imagine its just reads. > > Whenever the indexer is running, I see a lot of DB timeouts from other > > services, so I can't help but feel that something is up. > > Generally it should be just reads - the only case where it's not is when > you're using deltas, and then it's just a single UPDATE and the rest are > SELECTs. I'd recommend looking at the generated query and trying to determine > why it's slow. Also worth checking that you have database indices on any > foreign keys being used by joins in the generated SQL. > > > Also, in regards to changing the structure of my indexes, so I need to > > run rebuild, or can I just configure and restart (i.e. must I reindex > > before restarting with a new config file)? > > You could configure and index, and *then* restart, but I've heard that's not > quite reliable. So, I'd recommend rebuild when there's a structure change - > treat it the same as you would a database change, put up a temporary > maintenance page. It shouldn't be a common occurrence. > > Cheers > > -- > Pat > > > > > > > > > > > Thanks. > > > On Feb 17, 3:21 am, "Pat Allan" <[email protected]> wrote: > >> It's best to use the ts:index task in cron, because all that does is > >> reindex the data. ts:rebuild stops Sphinx, reindexes, and *then* restarts > >> Sphinx - and you really don't need to restart Sphinx except when the > >> structure of your indices change (you've added an index to another model, > >> or added a new field or attribute to an existing model, a new config > >> setting, etc). > > >> So, I'd recommend switching to ts:index - although it won't be much > >> different in speed (starting/stopping Sphinx won't take much time), but > >> it'll mean people can still search while the indexing is happening. > > >> -- > >> Pat > > >> On 17/02/2012, at 7:52 PM, gmoniey wrote: > > >>> For the past few years, I have had a cron job that wakes up around > >>> 4am, and runs the ts:rebuild rake task. Unfortunately, the size of all > >>> of my indexes has grown significantly, and this job takes about 40-60 > >>> minutes, and the load on the database is pretty significant. I'm > >>> indexing roughly 2-3 million records (some of which have pretty > >>> detailed indexes defined). > > >>> I've read on some blogs the index/reindex is significantly faster than > >>> rebuild, but haven't been able to find any formal documentation > >>> mentioning that. > > >>> Do I have any other options? > > >>> Thanks. > > >>> -- > >>> You received this message because you are subscribed to the Google Groups > >>> "Thinking Sphinx" group. > >>> To post to this group, send email to [email protected]. > >>> To unsubscribe from this group, send email to > >>> [email protected]. > >>> For more options, visit this group > >>> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Thinking Sphinx" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/thinking-sphinx?hl=en. -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
