I am using unset() everywhere possible - AFAIK this doesn't/won't alleviate the PDO memory leak/hog issue.
On May 7, 7:38 pm, Alexandru-Emil Lupu <[email protected]> wrote: > Or, you can try to install php 5.3. It has a better garbage collector, > so, your script will not die because of the memory issue ... > I haven't seen you code or xml format, but after you parse each game > ... unset() it (in php 5.3 works nice, and you have some nice results > ) > > > > On Fri, May 7, 2010 at 10:30 AM, Stéphane <[email protected]> wrote: > > And do || processing on multiple servers if possible/needed. > > Use stdin to retrieve to-compute xml parts for each segment. So you can pipe > > your commands. > > Cheers, > > > Before Printing, Think about Your Environmental Responsibility! > > Avant d'Imprimer, Pensez à Votre Responsabilitée Environnementale! > > > On Fri, May 7, 2010 at 4:34 AM, ken <[email protected]> wrote: > > >> Or you can add optional argument offset and limit for your task. This > >> however is not for sql but for the xml. The idea is to process only a > >> part of the xml for a given run. This will allow you to run the task > >> in many segments. > > >> On May 7, 10:17 am, dan <[email protected]> wrote: > >> > Hmm - I don't think this is going to work. > > >> > I ran the script and it got killed after 51 of 700+ games, with 21000 > >> > TyperefGame relationship records. It was consuming over 1.5GB of RAM. > >> > I understand PDO is a memory hog, so may have to refactor using > >> > mysql_query() etc. :( > > >> > Any ideas are more than welcome :) > > >> > Cheers, > > >> > Dan. > > >> > On May 7, 8:31 am, dan <[email protected]> wrote: > > >> > > Hi all, > > >> > > I have a large XML file (140+MB) which I'm parsing with an sf Task. > > >> > > It's a complicated structure, and contains some interesting > >> > > relationships. > > >> > > The approach I'm using is to open the XML file with XMLReader, and > >> > > when I find a node of interest, convert that node to a SimpleXML > >> > > object, and then build my sf Records. > > >> > > Without going into too much detail, I have a few record types that are > >> > > of primary interest; Game, Handset, Category. > > >> > > Relating Games to Categories is easy, however Games and Handsets are > >> > > related via a Typeref, i.e. Games are published for certain Typerefs, > >> > > while each handset supports one or two different Typerefs. > > >> > > So I've also created tables for Typeref, TyperefGame and > >> > > TyperefHandset in order to manage the relationships. > > >> > > Here's my real question: > > >> > > As my Task parses each Game, it looks up each of the Typerefs (approx > >> > > 450 per game) associated with the game and creates the TyperefGame > >> > > relationship record. With 700+ games, that's about 315000 records. > > >> > > As you can imagine, this takes some time, and I'm looking for every > >> > > opportunity to speed it up. > > >> > > My first attempt was simply to create a new Doctrine Record for each > >> > > Typeref, setting the necessary attributes and then saving it. This was > >> > > slow. (around 450 save()s per game) > > >> > > My second attempt was to create a Typeref Doctrine Collection for each > >> > > game, and then do one save() on the collection once all Typerefs had > >> > > been parsed. I expected this to be faster that my first method, but if > >> > > anything, it appears slower. > > >> > > What should my third attempt be? > > >> > > Should I be building a single large raw SQL statement for each game- > >> > > load of Typerefs? > > >> > > Is there an even better way to approach this whole Task? > > >> > > Your thoughts and advice would be greatly appreciated. > > >> > > Cheers, > > >> > > Dan. > > >> > > -- > >> > > If you want to report a vulnerability issue on symfony, please send it > >> > > to security at symfony-project.com > > >> > > You received this message because you are subscribed to the Google > >> > > Groups "symfony users" group. > >> > > To post to this group, send email to [email protected] > >> > > To unsubscribe from this group, send email to > >> > > [email protected] > >> > > For more options, visit this group > >> > > athttp://groups.google.com/group/symfony-users?hl=en > > >> > -- > >> > If you want to report a vulnerability issue on symfony, please send it > >> > to security at symfony-project.com > > >> > You received this message because you are subscribed to the Google > >> > Groups "symfony users" group. > >> > To post to this group, send email to [email protected] > >> > To unsubscribe from this group, send email to > >> > [email protected] > >> > For more options, visit this group > >> > athttp://groups.google.com/group/symfony-users?hl=en > > >> -- > >> If you want to report a vulnerability issue on symfony, please send it to > >> security at symfony-project.com > > >> You received this message because you are subscribed to the Google > >> Groups "symfony users" group. > >> To post to this group, send email to [email protected] > >> To unsubscribe from this group, send email to > >> [email protected] > >> For more options, visit this group at > >>http://groups.google.com/group/symfony-users?hl=en > > > -- > > If you want to report a vulnerability issue on symfony, please send it to > > security at symfony-project.com > > > You received this message because you are subscribed to the Google > > Groups "symfony users" group. > > To post to this group, send email to [email protected] > > To unsubscribe from this group, send email to > > [email protected] > > For more options, visit this group at > >http://groups.google.com/group/symfony-users?hl=en > > -- > Have a nice day! > > Alecs > Certified ScrumMaster > > P.S. If you are on a list, please don't contact me privatelly, unless > i have allowed to. Further messages will be ignored. > > There are no cannibals alive! I have ate the last one yesterday ... > I am on web: http://www.alecslupu.ro/ > I am on twitter:http://twitter.com/alecslupu > I am on linkedIn:http://www.linkedin.com/in/alecslupu > Tel: (+4)0722 621 280 > > -- > If you want to report a vulnerability issue on symfony, please send it to > security at symfony-project.com > > You received this message because you are subscribed to the Google > Groups "symfony users" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group > athttp://groups.google.com/group/symfony-users?hl=en -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
