FOr my third attempt, I used plain old mysql_connect() etc. for just the most intensive part of the script. I was able to parse the file completely, but still gobbled up just over 1GB of RAM. If memory usage is going to be come an issue going forward, I'll simply convert more of the script to plain old mysql.
I'd prefer to use Doctrine, of course, since it's simpler, more reliable, and fewer lines of code, Will using raw SQL via the Doctrine connection object still utilize PDO? Cheers, Dan On May 7, 9:20 pm, Thomas Rabaix <[email protected]> wrote: > Doctrine is not built to do such long process script : you should used raw > SQL with the doctrine connection object. > > On Fri, May 7, 2010 at 9:38 AM, Alexandru-Emil Lupu > <[email protected]>wrote: > > > > > Or, you can try to install php 5.3. It has a better garbage collector, > > so, your script will not die because of the memory issue ... > > I haven't seen you code or xml format, but after you parse each game > > ... unset() it (in php 5.3 works nice, and you have some nice results > > ) > > > On Fri, May 7, 2010 at 10:30 AM, Stéphane <[email protected]> > > wrote: > > > And do || processing on multiple servers if possible/needed. > > > Use stdin to retrieve to-compute xml parts for each segment. So you can > > pipe > > > your commands. > > > Cheers, > > > > Before Printing, Think about Your Environmental Responsibility! > > > Avant d'Imprimer, Pensez à Votre Responsabilitée Environnementale! > > > > On Fri, May 7, 2010 at 4:34 AM, ken <[email protected]> wrote: > > > >> Or you can add optional argument offset and limit for your task. This > > >> however is not for sql but for the xml. The idea is to process only a > > >> part of the xml for a given run. This will allow you to run the task > > >> in many segments. > > > >> On May 7, 10:17 am, dan <[email protected]> wrote: > > >> > Hmm - I don't think this is going to work. > > > >> > I ran the script and it got killed after 51 of 700+ games, with 21000 > > >> > TyperefGame relationship records. It was consuming over 1.5GB of RAM. > > >> > I understand PDO is a memory hog, so may have to refactor using > > >> > mysql_query() etc. :( > > > >> > Any ideas are more than welcome :) > > > >> > Cheers, > > > >> > Dan. > > > >> > On May 7, 8:31 am, dan <[email protected]> wrote: > > > >> > > Hi all, > > > >> > > I have a large XML file (140+MB) which I'm parsing with an sf Task. > > > >> > > It's a complicated structure, and contains some interesting > > >> > > relationships. > > > >> > > The approach I'm using is to open the XML file with XMLReader, and > > >> > > when I find a node of interest, convert that node to a SimpleXML > > >> > > object, and then build my sf Records. > > > >> > > Without going into too much detail, I have a few record types that > > are > > >> > > of primary interest; Game, Handset, Category. > > > >> > > Relating Games to Categories is easy, however Games and Handsets are > > >> > > related via a Typeref, i.e. Games are published for certain > > Typerefs, > > >> > > while each handset supports one or two different Typerefs. > > > >> > > So I've also created tables for Typeref, TyperefGame and > > >> > > TyperefHandset in order to manage the relationships. > > > >> > > Here's my real question: > > > >> > > As my Task parses each Game, it looks up each of the Typerefs > > (approx > > >> > > 450 per game) associated with the game and creates the TyperefGame > > >> > > relationship record. With 700+ games, that's about 315000 records. > > > >> > > As you can imagine, this takes some time, and I'm looking for every > > >> > > opportunity to speed it up. > > > >> > > My first attempt was simply to create a new Doctrine Record for each > > >> > > Typeref, setting the necessary attributes and then saving it. This > > was > > >> > > slow. (around 450 save()s per game) > > > >> > > My second attempt was to create a Typeref Doctrine Collection for > > each > > >> > > game, and then do one save() on the collection once all Typerefs had > > >> > > been parsed. I expected this to be faster that my first method, but > > if > > >> > > anything, it appears slower. > > > >> > > What should my third attempt be? > > > >> > > Should I be building a single large raw SQL statement for each game- > > >> > > load of Typerefs? > > > >> > > Is there an even better way to approach this whole Task? > > > >> > > Your thoughts and advice would be greatly appreciated. > > > >> > > Cheers, > > > >> > > Dan. > > > >> > > -- > > >> > > If you want to report a vulnerability issue on symfony, please send > > it > > >> > > to security at symfony-project.com > > > >> > > You received this message because you are subscribed to the Google > > >> > > Groups "symfony users" group. > > >> > > To post to this group, send email to [email protected] > > >> > > To unsubscribe from this group, send email to > > >> > > [email protected]<symfony-users%[email protected]> > > >> > > For more options, visit this group > > >> > > athttp://groups.google.com/group/symfony-users?hl=en > > > >> > -- > > >> > If you want to report a vulnerability issue on symfony, please send it > > >> > to security at symfony-project.com > > > >> > You received this message because you are subscribed to the Google > > >> > Groups "symfony users" group. > > >> > To post to this group, send email to [email protected] > > >> > To unsubscribe from this group, send email to > > >> > [email protected]<symfony-users%[email protected]> > > >> > For more options, visit this group > > >> > athttp://groups.google.com/group/symfony-users?hl=en > > > >> -- > > >> If you want to report a vulnerability issue on symfony, please send it > > to > > >> security at symfony-project.com > > > >> You received this message because you are subscribed to the Google > > >> Groups "symfony users" group. > > >> To post to this group, send email to [email protected] > > >> To unsubscribe from this group, send email to > > >> [email protected]<symfony-users%[email protected]> > > >> For more options, visit this group at > > >>http://groups.google.com/group/symfony-users?hl=en > > > > -- > > > If you want to report a vulnerability issue on symfony, please send it to > > > security at symfony-project.com > > > > You received this message because you are subscribed to the Google > > > Groups "symfony users" group. > > > To post to this group, send email to [email protected] > > > To unsubscribe from this group, send email to > > > [email protected]<symfony-users%[email protected]> > > > For more options, visit this group at > > >http://groups.google.com/group/symfony-users?hl=en > > > -- > > Have a nice day! > > > Alecs > > Certified ScrumMaster > > > P.S. If you are on a list, please don't contact me privatelly, unless > > i have allowed to. Further messages will be ignored. > > > There are no cannibals alive! I have ate the last one yesterday ... > > I am on web: http://www.alecslupu.ro/ > > I am on twitter:http://twitter.com/alecslupu > > I am on linkedIn:http://www.linkedin.com/in/alecslupu > > Tel: (+4)0722 621 280 > > > -- > > If you want to report a vulnerability issue on symfony, please send it to > > security at symfony-project.com > > > You received this message because you are subscribed to the Google > > Groups "symfony users" group. > > To post to this group, send email to [email protected] > > To unsubscribe from this group, send email to > > [email protected]<symfony-users%[email protected]> > > For more options, visit this group at > >http://groups.google.com/group/symfony-users?hl=en > > -- > Thomas Rabaixhttp://rabaix.net > > -- > If you want to report a vulnerability issue on symfony, please send it to > security at symfony-project.com > > You received this message because you are subscribed to the Google > Groups "symfony users" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group > athttp://groups.google.com/group/symfony-users?hl=en -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
