Dude, try the Lucene plugin :) Index your records with Lucene, give your database a break :)
-----Original Message----- From: [email protected] [mailto:[EMAIL PROTECTED] On Behalf Of James Sent: 17 December 2007 09:20 To: [email protected] Subject: [symfony-users] Large Recordset running out of memory.... Hi, It's me again. v1.0.8 The whole reason I started playing with Symfony was for work. I've made a personal app to get familiar with Symfony and tonight I decided to take it to the work level. We have a database table with 1.4M records. We want to index these records in order to do a search on them. I was following the askeet example modified for our data set I got it to work adding weighs on the words in the company names, but hit an error where I ran out of memory. This was not very surprising. 1.4M records is a lot of data. I figured that the best possible way to handle this would be to go through each record on at a time. It may take a little while to run, but I knew that one record was within the memory limits. Here's my code the rest is quite literally out of the askeet example except it's not in an object it's simply functions in the batch script. $countStart=0; $countEnd=1400000; while($countStart != $countEnd){ // batch process here $c = new Criteria(); $c->setLimit(1); $c->setOffset($countStart++); $company = CompanyPeer::doSelect($c); $companyName = $company[0]->getCompany(); $companyId = $company[0]->getId(); $companyNameText = str_repeat(" ".$companyName, 2); $stemmed = stemPhrase($companyNameText); $words = array_count_values($stemmed); updateSearchIndex($companyId,$words); // Point everything at null. $c = null; $company = null; $companyName = null; $companyId = null; $componyNameText = null; $stemmed = null; $words = null; } When I run this code It does what it is suppose too up to about 19000 lines. Then it dumps and says that it ran out of memory again. My question is basically how can I work with so many records in Symfony without running out of memory, and without increasing how much memory php is allowed to use. A similar method worked for me in php, so I am wondering what I may be missing. Thanks, James --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en -~----------~----~----~----~------~----~------~--~---
