Have you made the suggested change to count the words in the document first then update the db, so that you are not updating each record in the database many times (presumably hundreds of times for common words)? This will change the number of db accesses from the total number of words in the document to the number of unique words in the document.
2009/5/10 Jian Lin <[email protected]> > > Phlip wrote: > > Jian Lin wrote: > > > >> so i re-ran the test and it worked quite quickly... down to 1 minute or > >> so instead of 30 minutes... (for several pages)... and the db size is > >> only 2.5MB (i created another rails project to start anew). So > >> if we are just write scripts, and running with script/runner test.rb, > >> how can we turn the "force write" feature of the sqlite3 off? > > > > Are you running inside a transaction? sqlite3 in transaction mode is > > super-fast... > > > > (Jumping into the thread late, why should a test push 6 000 records? > > Such a test > > is not exactly a "unit" test, and alternate strategies should work > > better.) > > i tried both record.save and record.save! and both needed a few > minutes at the end of the program to write the data to the db... > > i am writing a program to tally up the common word or phrases (say, > either in English or Chinese) that appears often. So for example, at > the end of the program (i used ruby script/runner check.rb to run it > using Ruby with ActiveRecored support), when the results are in, it may > be a table of 6000 keys and 6000 frequency counts. So I need to write > the data into the db. That's it. Wonder why it takes a few minutes > and how to make it just be a few seconds. > > If I just write the results to a flat text file using CSV format, it > should take no more than a few seconds, i think. > > -- > Posted via http://www.ruby-forum.com/. > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
