Or you can add optional argument offset and limit for your task. This however is not for sql but for the xml. The idea is to process only a part of the xml for a given run. This will allow you to run the task in many segments.
On May 7, 10:17 am, dan <[email protected]> wrote: > Hmm - I don't think this is going to work. > > I ran the script and it got killed after 51 of 700+ games, with 21000 > TyperefGame relationship records. It was consuming over 1.5GB of RAM. > I understand PDO is a memory hog, so may have to refactor using > mysql_query() etc. :( > > Any ideas are more than welcome :) > > Cheers, > > Dan. > > On May 7, 8:31 am, dan <[email protected]> wrote: > > > > > > > Hi all, > > > I have a large XML file (140+MB) which I'm parsing with an sf Task. > > > It's a complicated structure, and contains some interesting > > relationships. > > > The approach I'm using is to open the XML file with XMLReader, and > > when I find a node of interest, convert that node to a SimpleXML > > object, and then build my sf Records. > > > Without going into too much detail, I have a few record types that are > > of primary interest; Game, Handset, Category. > > > Relating Games to Categories is easy, however Games and Handsets are > > related via a Typeref, i.e. Games are published for certain Typerefs, > > while each handset supports one or two different Typerefs. > > > So I've also created tables for Typeref, TyperefGame and > > TyperefHandset in order to manage the relationships. > > > Here's my real question: > > > As my Task parses each Game, it looks up each of the Typerefs (approx > > 450 per game) associated with the game and creates the TyperefGame > > relationship record. With 700+ games, that's about 315000 records. > > > As you can imagine, this takes some time, and I'm looking for every > > opportunity to speed it up. > > > My first attempt was simply to create a new Doctrine Record for each > > Typeref, setting the necessary attributes and then saving it. This was > > slow. (around 450 save()s per game) > > > My second attempt was to create a Typeref Doctrine Collection for each > > game, and then do one save() on the collection once all Typerefs had > > been parsed. I expected this to be faster that my first method, but if > > anything, it appears slower. > > > What should my third attempt be? > > > Should I be building a single large raw SQL statement for each game- > > load of Typerefs? > > > Is there an even better way to approach this whole Task? > > > Your thoughts and advice would be greatly appreciated. > > > Cheers, > > > Dan. > > > -- > > If you want to report a vulnerability issue on symfony, please send it to > > security at symfony-project.com > > > You received this message because you are subscribed to the Google > > Groups "symfony users" group. > > To post to this group, send email to [email protected] > > To unsubscribe from this group, send email to > > [email protected] > > For more options, visit this group > > athttp://groups.google.com/group/symfony-users?hl=en > > -- > If you want to report a vulnerability issue on symfony, please send it to > security at symfony-project.com > > You received this message because you are subscribed to the Google > Groups "symfony users" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group > athttp://groups.google.com/group/symfony-users?hl=en -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
