I am using as I understand the doctrine equivalet of doSelectStmt (or doSelectRS) witch is much more elegant in doctrine, but i get memory leaks or memory that is not retorned to the OS, and not reused by PHP, aparently this is corrected on Doctrine 2 http://www.doctrine-project.org/blog/doctrine2-batch-processing but i can´t find an apropied solution for doctrine 1.2, Sadly it seams like is not a doctrine bug, but a PHP bug trigered by doctrine use of references.
2010/11/17 Gareth McCumskey <[email protected]>: > This is ridiculous. You mean Doctrine has no resultset function? Using > Propel you normally build your criteria for a query and then run a doSelect > call which returns your hydrated array. BUT. There is a more memory > efficient call (doSelectRS for Propel 1.2 and doSelectStmt for Propel 1.3 > and above) that essentially returns the resultset that you can while loop > through to get your data. Using these calls we have been able to load up the > results of 100's of thousands of rows from our DB at a time with minimal > memory usage. > Take a look at the doctrine documentation. There must be something! > > On Tue, Nov 16, 2010 at 10:56 PM, Luciano A. Andrade > <[email protected]> wrote: >> >> Thanks Daniel for the response, i am using PHP 5.3.2-1ubuntu4.5 with >> Suhosin-Patch (cli) (built: Sep 17 2010 13:41:55), read some bug >> reports related to circula reference and the garbage colector, but >> they suld be fixed for 5.3. >> >> OnDemand hydration did't help, i am currently doing it, i am procesing >> batches of 100 objects inside a method and i can currently hold the >> objects but i only can call this method about 18 times until it run >> out of memory. >> >> Incresing memory of the server may be a valid option but is not >> sustentable, so i like to try other options. >> >> gc_collect_cycles may help a little but didn't change the situation. >> >> The only options is to implement a Fork/Subtask (Following the sample >> provided by Rich). >> >> It's a shame since i could do this on propel with any problem :(. >> >> 2010/11/16 Daniel Lohse <[email protected]>: >> > That's unfortunate because you really have few options — I know because >> > I've been there just this week. >> > >> > So, array hydration is out of the question, fine, let's see what other >> > options you have: >> > >> > 1. Increase memory of the server (sorry, but it's one option) >> > >> > 2. Use the latest PHP 5.3.x and try to call "gc_collect_cycles();" after >> > each iteration — this will slow down your script some but could help. >> > Unfortunately, while garbage collection has certainly improved with PHP >> > 5.3, >> > you can't rely on this because PHP is not forced to garbage collect and >> > sometimes can't; tricky stuff. >> > >> > 3. Fork another process after, say, 1000 objects. Each PHP process has >> > its own memory limit. I know of one project that uses this — you'll have to >> > go look for it, though. The update-search-index task of the Apostrophe CMS >> > is using this technique to work around this problem, you can find the >> > source >> > on svn.apostrophenow.org. Sorry for not being more specific. :( >> > >> > 4. Only process a given amount of objects per call and work your way >> > through the queue, flagging objects you've finished processing along the >> > way. >> > >> > 5. Try to use the OnDemand object hydrator and see if that helps. >> > >> > If you already tried all options I listed above (and failed ;-)), you'll >> > need to tell us more about the process, about why they don't/won't work — >> > maybe we can then come up with a solution. >> > >> > >> > I feel your pain, cheers, Daniel >> > >> > Sent from my iPad >> > >> > On Nov 16, 2010, at 8:48 PM, "Luciano A. Andrade" >> > <[email protected]> wrote: >> > >> >> I am tyring to make a batch processing for synchronization of two >> >> databases, and i am running on limitation of the memory usage. I like >> >> to be able to process 30000 objects, i can't use arrays for the >> >> hydration, since i have some logic i need to execute. I only need one >> >> of this object at any time, but it looks like theres no way to >> >> liberate the used objects, so memory usage grow rapidly. Any one has >> >> be able to free some memory, i am using >> >> $object->free(true); >> >> unset($object); >> >> >> >> so i run out of ideas. Any one? >> >> >> >> -- >> >> If you want to report a vulnerability issue on symfony, please send it >> >> to security at symfony-project.com >> >> >> >> You received this message because you are subscribed to the Google >> >> Groups "symfony users" group. >> >> To post to this group, send email to [email protected] >> >> To unsubscribe from this group, send email to >> >> [email protected] >> >> For more options, visit this group at >> >> http://groups.google.com/group/symfony-users?hl=en >> > >> > -- >> > If you want to report a vulnerability issue on symfony, please send it >> > to security at symfony-project.com >> > >> > You received this message because you are subscribed to the Google >> > Groups "symfony users" group. >> > To post to this group, send email to [email protected] >> > To unsubscribe from this group, send email to >> > [email protected] >> > For more options, visit this group at >> > http://groups.google.com/group/symfony-users?hl=en >> > >> >> -- >> If you want to report a vulnerability issue on symfony, please send it to >> security at symfony-project.com >> >> You received this message because you are subscribed to the Google >> Groups "symfony users" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/symfony-users?hl=en > > > > -- > Gareth McCumskey > http://garethmccumskey.blogspot.com > twitter: @garethmcc > > -- > If you want to report a vulnerability issue on symfony, please send it to > security at symfony-project.com > > You received this message because you are subscribed to the Google > Groups "symfony users" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/symfony-users?hl=en > -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
