Kay C Lan wrote:
What I'm after is the fastest way to take a HUGE amount of data and reduce it by roughly 5-10%. The repeat for each code I supplied seems to do that, if anyone has any other code that is faster PLEASE provide.
First, I must say how delighted I am to find others fixating on performance nuances. :) This discussion has been an enjoyable read for me, and hopefully others as well, as we learn the ways various syntax options affect what the engine's doing under the hood. Fun stuff, and very useful.
FWIW, my own tests with various data over the years reflects your experience, and Raney's description: "repeat for each" is much faster than "repeat with" or anything involving "get line x" because the line counting and the parsing out of the line value happens only one per iteration.
Moreover, Raney once noted that he took the time to optimize "put...after..." specifically for cases like yours, so it could be used in conjunction with "repeat for each" to build a subset of data.
Split/combine require a lot of overhead, since under the hood they're effectively calling a form of "repeat for each" to set up the array values, with the additional overhead of creating the hash table as it goes. Useful as they are, for your purposes I'd be surprised if translating the list to an array was less than 15% slower.
Using "filter" works well in many cases, but as noted elsewhere RegEx is a complex and highly generalized subsystem, designed to optimize programmer efficiency at the expense of runtime efficiency. So while it can be handy to reduce complex filtering to a single line of code, it rarely outperforms "repeat for each".
In short, I think you're on the right track. Traversing more than a million lines will always be time-intensive, but at least using "repeat for each" with "put...after..." you can reduce that as reasonably as one can expect.
-- Richard Gaskin Fourth World Rev training and consulting: http://www.fourthworld.com Webzine for Rev developers: http://www.revjournal.com revJournal blog: http://revjournal.com/blog.irv _______________________________________________ use-revolution mailing list [email protected] Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
