Or, you can try to install php 5.3. It has a better garbage collector, so, your script will not die because of the memory issue ... I haven't seen you code or xml format, but after you parse each game ... unset() it (in php 5.3 works nice, and you have some nice results )
On Fri, May 7, 2010 at 10:30 AM, Stéphane <[email protected]> wrote: > And do || processing on multiple servers if possible/needed. > Use stdin to retrieve to-compute xml parts for each segment. So you can pipe > your commands. > Cheers, > > Before Printing, Think about Your Environmental Responsibility! > Avant d'Imprimer, Pensez à Votre Responsabilitée Environnementale! > > > On Fri, May 7, 2010 at 4:34 AM, ken <[email protected]> wrote: >> >> Or you can add optional argument offset and limit for your task. This >> however is not for sql but for the xml. The idea is to process only a >> part of the xml for a given run. This will allow you to run the task >> in many segments. >> >> On May 7, 10:17 am, dan <[email protected]> wrote: >> > Hmm - I don't think this is going to work. >> > >> > I ran the script and it got killed after 51 of 700+ games, with 21000 >> > TyperefGame relationship records. It was consuming over 1.5GB of RAM. >> > I understand PDO is a memory hog, so may have to refactor using >> > mysql_query() etc. :( >> > >> > Any ideas are more than welcome :) >> > >> > Cheers, >> > >> > Dan. >> > >> > On May 7, 8:31 am, dan <[email protected]> wrote: >> > >> > >> > >> > >> > >> > > Hi all, >> > >> > > I have a large XML file (140+MB) which I'm parsing with an sf Task. >> > >> > > It's a complicated structure, and contains some interesting >> > > relationships. >> > >> > > The approach I'm using is to open the XML file with XMLReader, and >> > > when I find a node of interest, convert that node to a SimpleXML >> > > object, and then build my sf Records. >> > >> > > Without going into too much detail, I have a few record types that are >> > > of primary interest; Game, Handset, Category. >> > >> > > Relating Games to Categories is easy, however Games and Handsets are >> > > related via a Typeref, i.e. Games are published for certain Typerefs, >> > > while each handset supports one or two different Typerefs. >> > >> > > So I've also created tables for Typeref, TyperefGame and >> > > TyperefHandset in order to manage the relationships. >> > >> > > Here's my real question: >> > >> > > As my Task parses each Game, it looks up each of the Typerefs (approx >> > > 450 per game) associated with the game and creates the TyperefGame >> > > relationship record. With 700+ games, that's about 315000 records. >> > >> > > As you can imagine, this takes some time, and I'm looking for every >> > > opportunity to speed it up. >> > >> > > My first attempt was simply to create a new Doctrine Record for each >> > > Typeref, setting the necessary attributes and then saving it. This was >> > > slow. (around 450 save()s per game) >> > >> > > My second attempt was to create a Typeref Doctrine Collection for each >> > > game, and then do one save() on the collection once all Typerefs had >> > > been parsed. I expected this to be faster that my first method, but if >> > > anything, it appears slower. >> > >> > > What should my third attempt be? >> > >> > > Should I be building a single large raw SQL statement for each game- >> > > load of Typerefs? >> > >> > > Is there an even better way to approach this whole Task? >> > >> > > Your thoughts and advice would be greatly appreciated. >> > >> > > Cheers, >> > >> > > Dan. >> > >> > > -- >> > > If you want to report a vulnerability issue on symfony, please send it >> > > to security at symfony-project.com >> > >> > > You received this message because you are subscribed to the Google >> > > Groups "symfony users" group. >> > > To post to this group, send email to [email protected] >> > > To unsubscribe from this group, send email to >> > > [email protected] >> > > For more options, visit this group >> > > athttp://groups.google.com/group/symfony-users?hl=en >> > >> > -- >> > If you want to report a vulnerability issue on symfony, please send it >> > to security at symfony-project.com >> > >> > You received this message because you are subscribed to the Google >> > Groups "symfony users" group. >> > To post to this group, send email to [email protected] >> > To unsubscribe from this group, send email to >> > [email protected] >> > For more options, visit this group >> > athttp://groups.google.com/group/symfony-users?hl=en >> >> -- >> If you want to report a vulnerability issue on symfony, please send it to >> security at symfony-project.com >> >> You received this message because you are subscribed to the Google >> Groups "symfony users" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/symfony-users?hl=en > > -- > If you want to report a vulnerability issue on symfony, please send it to > security at symfony-project.com > > You received this message because you are subscribed to the Google > Groups "symfony users" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/symfony-users?hl=en > -- Have a nice day! Alecs Certified ScrumMaster P.S. If you are on a list, please don't contact me privatelly, unless i have allowed to. Further messages will be ignored. There are no cannibals alive! I have ate the last one yesterday ... I am on web: http://www.alecslupu.ro/ I am on twitter: http://twitter.com/alecslupu I am on linkedIn: http://www.linkedin.com/in/alecslupu Tel: (+4)0722 621 280 -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
