Could be the 360K * 8K iterations youre doing.. whats that almost 3 billion? unless I read that wrong. Good call on the DB queries, always a good idea to extract out the queries youre running and run them manually - then you can start asking questions like - are my tables indexed properly, etc. generally narrow down the cause of the problem via deduction. Another question to ask is 'are all of those records getting updated between runs?' i.e. perhaps you only need to process newly created / updated records and not all of them each time. Also the problem set is really begging for a search engine, I'd investigate the many options for this from Sphinx/Solr/Ferret/etc to potentially building your own indexing schema and get your algorithm down to something more tolerable..
On Wed, Sep 3, 2008 at 1:55 AM, Dan Simpson <[EMAIL PROTECTED]> wrote: > > It's probably so slow because you are making so many queries. I bet > nearly all the time is spent waiting on the DB. > > For retrieval, find a clever way to store the information in data > structures (if you have the local resources to accomidate it). And for > persistence, perhaps organize the data and do bulk queries to the > database. > ( > http://rubypond.com/articles/2008/06/18/bulk-insertion-of-data-with-activerecord/ > ) > > Just a thought, > Dan > > Brent Hargrave wrote: > > Hi all, > > > > I've built a site that searches for local events by performer. The > > backgroundrb process that identifies the performers for each event > > consumes a lot of resources, though. Any suggestions for streamlining > > this process? > > > > Here's the gist (with embedded comments to explain the problem more > > fully): > > http://gist.github.com/8538 > > > > Feedback would be much appreciated. However, I'll be at the meetup on > > Thursday if you feel an adequate (or impolite ;) response is beyond > > the scope of this mailing list. > > > > Thanks, > > Brent > > > > > > > > > > --~--~---------~--~----~------------~-------~--~----~ SD Ruby mailing list sdruby@googlegroups.com http://groups.google.com/group/sdruby -~----------~----~----~----~------~----~------~--~---
