2009/5/10 Jian Lin <[email protected]> > > Colin Law wrote: > > Have you made the suggested change to count the words in the document > > first > > then update the db, so that you are not updating each record in the > > database > > many times (presumably hundreds of times for common words)? This will > > change the number of db accesses from the total number of words in the > > document to the number of unique words in the document. > > > yes, so right now i first tally up the count by a Hash, and then at the > end, write all the data to the table. it is strange that writing all > this data take a minute or so... because the final db is only 2.5MB, > and that if i write all the data to a flat file, i think it is done in 2
seconds. Have you checked the sql in the log to make sure that it is only rewriting each word once? Have you added the index to the word field (I think now that each record is only being written once then the index should be beneficial as it will speed up the find, someone correct me if I am wrong)? Another thing you could do is comment out the db access bit just to make sure that the time is not going somewhere else. Initially comment out the save then comment out the find also. Colin > > > so right now, i want to find out > > 1) can i tell sqlite3 not to write to the db every time i add a record > (a row), but to do it all in memory and then finally, write to the table > once at the end. > > 2) use memCache to do it. > > 3) some one at stackoverflow.com suggested using AR:Extensions > > http://stackoverflow.com/questions/843524/updating-the-db-6000-times-will-take-few-minutes > > but i just want to use the most basic way to do it... such as by turning > off the force-write of sqlite3 > > -- > Posted via http://www.ruby-forum.com/. > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
