And do || processing on multiple servers if possible/needed. Use stdin to retrieve to-compute xml parts for each segment. So you can pipe your commands.
Cheers, Before Printing, Think about Your Environmental Responsibility! Avant d'Imprimer, Pensez à Votre Responsabilitée Environnementale! On Fri, May 7, 2010 at 4:34 AM, ken <[email protected]> wrote: > Or you can add optional argument offset and limit for your task. This > however is not for sql but for the xml. The idea is to process only a > part of the xml for a given run. This will allow you to run the task > in many segments. > > On May 7, 10:17 am, dan <[email protected]> wrote: > > Hmm - I don't think this is going to work. > > > > I ran the script and it got killed after 51 of 700+ games, with 21000 > > TyperefGame relationship records. It was consuming over 1.5GB of RAM. > > I understand PDO is a memory hog, so may have to refactor using > > mysql_query() etc. :( > > > > Any ideas are more than welcome :) > > > > Cheers, > > > > Dan. > > > > On May 7, 8:31 am, dan <[email protected]> wrote: > > > > > > > > > > > > > Hi all, > > > > > I have a large XML file (140+MB) which I'm parsing with an sf Task. > > > > > It's a complicated structure, and contains some interesting > > > relationships. > > > > > The approach I'm using is to open the XML file with XMLReader, and > > > when I find a node of interest, convert that node to a SimpleXML > > > object, and then build my sf Records. > > > > > Without going into too much detail, I have a few record types that are > > > of primary interest; Game, Handset, Category. > > > > > Relating Games to Categories is easy, however Games and Handsets are > > > related via a Typeref, i.e. Games are published for certain Typerefs, > > > while each handset supports one or two different Typerefs. > > > > > So I've also created tables for Typeref, TyperefGame and > > > TyperefHandset in order to manage the relationships. > > > > > Here's my real question: > > > > > As my Task parses each Game, it looks up each of the Typerefs (approx > > > 450 per game) associated with the game and creates the TyperefGame > > > relationship record. With 700+ games, that's about 315000 records. > > > > > As you can imagine, this takes some time, and I'm looking for every > > > opportunity to speed it up. > > > > > My first attempt was simply to create a new Doctrine Record for each > > > Typeref, setting the necessary attributes and then saving it. This was > > > slow. (around 450 save()s per game) > > > > > My second attempt was to create a Typeref Doctrine Collection for each > > > game, and then do one save() on the collection once all Typerefs had > > > been parsed. I expected this to be faster that my first method, but if > > > anything, it appears slower. > > > > > What should my third attempt be? > > > > > Should I be building a single large raw SQL statement for each game- > > > load of Typerefs? > > > > > Is there an even better way to approach this whole Task? > > > > > Your thoughts and advice would be greatly appreciated. > > > > > Cheers, > > > > > Dan. > > > > > -- > > > If you want to report a vulnerability issue on symfony, please send it > to security at symfony-project.com > > > > > You received this message because you are subscribed to the Google > > > Groups "symfony users" group. > > > To post to this group, send email to [email protected] > > > To unsubscribe from this group, send email to > > > [email protected]<symfony-users%[email protected]> > > > For more options, visit this group athttp:// > groups.google.com/group/symfony-users?hl=en > > > > -- > > If you want to report a vulnerability issue on symfony, please send it to > security at symfony-project.com > > > > You received this message because you are subscribed to the Google > > Groups "symfony users" group. > > To post to this group, send email to [email protected] > > To unsubscribe from this group, send email to > > [email protected]<symfony-users%[email protected]> > > For more options, visit this group athttp:// > groups.google.com/group/symfony-users?hl=en > > -- > If you want to report a vulnerability issue on symfony, please send it to > security at symfony-project.com > > You received this message because you are subscribed to the Google > Groups "symfony users" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected]<symfony-users%[email protected]> > For more options, visit this group at > http://groups.google.com/group/symfony-users?hl=en > -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
