Hey Guys,
OK, so I am bringing this back up for another reason. Same data, same
problem, different situation.
I have a subset of the 1.4M records. 24,000 records roughly. One of
these fields is a 9 digit code. Which is really 3 codes in one.
111222333 would be 3 codes 111, 222, 333 I need to create a map
table between what these codes represent and the model for this data.
The problem is when I get to 9750 inserts into the table the script
errors out with a memory problem. I'm releasing all my memory in the
script and would expect symfony/propel to be able to handle this.
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted
(tried to allocate 7420 bytes) in /usr/share/pear/symfony/vendor/
propel/util/BasePeer.php on line 444
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried
to allocate 7420 bytes) in /usr/share/pear/symfony/vendor/propel/util/
BasePeer.php on line 444
Now I know I could get around this by using straight php, but I would
like to use the tools the symfony provides with the data models to
keep continuity straight. This isn't going to be the only data I need
to manipulate, so I can see this being a major pain.
I've included the code below. I am using Symfony 1.0.10
Ideas?
Thanks,
James
======================================================================
$countStart = 0;
$countEnd = 23334;
while($countStart != $countEnd)
{
query
// batch process here
$c = new Criteria();
$c->setLimit(1);
$c->setOffset($countStart++);
$dataModel = DataModelPeer::doSelect($c);
$actCodes = array();
$actCodes[] = substr($dataMode[0]->getActCode(),0,3);
$actCodes[] = substr($dataMode[0]->getActCode(),3,3);
$actCodes[] = substr($dataMode[0]->getActCode(),6,3);
foreach($actCodes as $actCode){
$c2 = new Criteria();
$c2->add(MelissaActcodePeer::CODE, $actCode);
$ntee = MelissaActcodePeer::doSelect($c2);
if(count($ntee)){
$map = new ModelDataMelissaActcodeMap();
$map->setModelDataId($dataMode[0]->getId());
$map->setMelissaActcodeId($ntee[0]->getId());
$map->save();
$map=null;
}
}
unset($c);
unset($c2);
unset($dataModel);
unset($ntee);
unset($map);
unset($actCodes);
}
On Dec 17, 2007, at 4:20 AM, James wrote:
> Hi,
>
> It's me again. v1.0.8
>
> The whole reason I started playing with Symfony was for work. I've
> made a personal app to get familiar with Symfony and tonight I
> decided to take it to the work level. We have a database table with
> 1.4M records. We want to index these records in order to do a
> search on them. I was following the askeet example modified for our
> data set I got it to work adding weighs on the words in the company
> names, but hit an error where I ran out of memory. This was not
> very surprising. 1.4M records is a lot of data.
>
> I figured that the best possible way to handle this would be to go
> through each record on at a time. It may take a little while to
> run, but I knew that one record was within the memory limits.
> Here's my code the rest is quite literally out of the askeet
> example except it's not in an object it's simply functions in the
> batch script.
>
> $countStart=0;
> $countEnd=1400000;
>
> while($countStart != $countEnd){
>
> // batch process here
> $c = new Criteria();
> $c->setLimit(1);
> $c->setOffset($countStart++);
> $company = CompanyPeer::doSelect($c);
>
> $companyName = $company[0]->getCompany();
> $companyId = $company[0]->getId();
>
> $companyNameText = str_repeat(" ".$companyName, 2);
>
> $stemmed = stemPhrase($companyNameText);
>
> $words = array_count_values($stemmed);
>
> updateSearchIndex($companyId,$words);
>
> // Point everything at null.
> $c = null;
> $company = null;
> $companyName = null;
> $companyId = null;
> $componyNameText = null;
> $stemmed = null;
> $words = null;
>
> }
>
> When I run this code It does what it is suppose too up to about
> 19000 lines. Then it dumps and says that it ran out of memory
> again. My question is basically how can I work with so many records
> in Symfony without running out of memory, and without increasing how
> much memory php is allowed to use. A similar method worked for me
> in php, so I am wondering what I may be missing.
>
> Thanks,
> James
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"symfony users" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/symfony-users?hl=en
-~----------~----~----~----~------~----~------~--~---
