This is ridiculous. You mean Doctrine has no resultset function? Using Propel you normally build your criteria for a query and then run a doSelect call which returns your hydrated array. BUT. There is a more memory efficient call (doSelectRS for Propel 1.2 and doSelectStmt for Propel 1.3 and above) that essentially returns the resultset that you can while loop through to get your data. Using these calls we have been able to load up the results of 100's of thousands of rows from our DB at a time with minimal memory usage.
Take a look at the doctrine documentation. There must be something! On Tue, Nov 16, 2010 at 10:56 PM, Luciano A. Andrade < [email protected]> wrote: > Thanks Daniel for the response, i am using PHP 5.3.2-1ubuntu4.5 with > Suhosin-Patch (cli) (built: Sep 17 2010 13:41:55), read some bug > reports related to circula reference and the garbage colector, but > they suld be fixed for 5.3. > > OnDemand hydration did't help, i am currently doing it, i am procesing > batches of 100 objects inside a method and i can currently hold the > objects but i only can call this method about 18 times until it run > out of memory. > > Incresing memory of the server may be a valid option but is not > sustentable, so i like to try other options. > > gc_collect_cycles may help a little but didn't change the situation. > > The only options is to implement a Fork/Subtask (Following the sample > provided by Rich). > > It's a shame since i could do this on propel with any problem :(. > > 2010/11/16 Daniel Lohse <[email protected]>: > > That's unfortunate because you really have few options — I know because > I've been there just this week. > > > > So, array hydration is out of the question, fine, let's see what other > options you have: > > > > 1. Increase memory of the server (sorry, but it's one option) > > > > 2. Use the latest PHP 5.3.x and try to call "gc_collect_cycles();" after > each iteration — this will slow down your script some but could help. > Unfortunately, while garbage collection has certainly improved with PHP 5.3, > you can't rely on this because PHP is not forced to garbage collect and > sometimes can't; tricky stuff. > > > > 3. Fork another process after, say, 1000 objects. Each PHP process has > its own memory limit. I know of one project that uses this — you'll have to > go look for it, though. The update-search-index task of the Apostrophe CMS > is using this technique to work around this problem, you can find the source > on svn.apostrophenow.org. Sorry for not being more specific. :( > > > > 4. Only process a given amount of objects per call and work your way > through the queue, flagging objects you've finished processing along the > way. > > > > 5. Try to use the OnDemand object hydrator and see if that helps. > > > > If you already tried all options I listed above (and failed ;-)), you'll > need to tell us more about the process, about why they don't/won't work — > maybe we can then come up with a solution. > > > > > > I feel your pain, cheers, Daniel > > > > Sent from my iPad > > > > On Nov 16, 2010, at 8:48 PM, "Luciano A. Andrade" < > [email protected]> wrote: > > > >> I am tyring to make a batch processing for synchronization of two > >> databases, and i am running on limitation of the memory usage. I like > >> to be able to process 30000 objects, i can't use arrays for the > >> hydration, since i have some logic i need to execute. I only need one > >> of this object at any time, but it looks like theres no way to > >> liberate the used objects, so memory usage grow rapidly. Any one has > >> be able to free some memory, i am using > >> $object->free(true); > >> unset($object); > >> > >> so i run out of ideas. Any one? > >> > >> -- > >> If you want to report a vulnerability issue on symfony, please send it > to security at symfony-project.com > >> > >> You received this message because you are subscribed to the Google > >> Groups "symfony users" group. > >> To post to this group, send email to [email protected] > >> To unsubscribe from this group, send email to > >> [email protected]<symfony-users%[email protected]> > >> For more options, visit this group at > >> http://groups.google.com/group/symfony-users?hl=en > > > > -- > > If you want to report a vulnerability issue on symfony, please send it to > security at symfony-project.com > > > > You received this message because you are subscribed to the Google > > Groups "symfony users" group. > > To post to this group, send email to [email protected] > > To unsubscribe from this group, send email to > > [email protected]<symfony-users%[email protected]> > > For more options, visit this group at > > http://groups.google.com/group/symfony-users?hl=en > > > > -- > If you want to report a vulnerability issue on symfony, please send it to > security at symfony-project.com > > You received this message because you are subscribed to the Google > Groups "symfony users" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected]<symfony-users%[email protected]> > For more options, visit this group at > http://groups.google.com/group/symfony-users?hl=en > -- Gareth McCumskey http://garethmccumskey.blogspot.com twitter: @garethmcc -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
