Jian Lin wrote: > Colin Law wrote: >> Have you made the suggested change to count the words in the document >> first >> then update the db, so that you are not updating each record in the >> database >> > yes, so right now i first tally up the count by a Hash, and then at the > end, write all the data to the table. it is strange that writing all > this data take a minute or so... because the final db is only 2.5MB, > and that if i write all the data to a flat file, i think it is done in 2 > seconds. > > so right now, i want to find out > > 1) can i tell sqlite3 not to write to the db every time i add a record > (a row), but to do it all in memory and then finally, write to the table > once at the end. > > 2) use memCache to do it. > > 3) some one at stackoverflow.com suggested using AR:Extensions > > http://stackoverflow.com/questions/843524/updating-the-db-6000-times-will-take-few-minutes > > but i just want to use the most basic way to do it... such as by turning > off the force-write of sqlite3 >
If you have made the change to count up words first and then *insert* all the (word,count) records into the database in a *single* transaction then it ought to take less than a second. I would expect that the total number of (word,count) records is in the order or hundreds or perhaps a thousand or so? Any decent DB ought to insert that in under a second as a *single* transaction. If it is still taking minutes then you are probably not doing the above somehow. I think in that case you need to post your code again so we can see what it is doing now. Cheers, Gary. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
