On 30 January 2013 05:17, Patrick Spinler <[email protected]> wrote: > Since no one else seems to be pointing this out, i can see at least one > potential optimization: >
> This will re-open the output file and seek to the end every output. > That's a factor of 2 or 3 more syscalls every time. (possibly plus the > fork cost to call 'echo', although I seem to recall bash has an echo > builtin, so this may not be an issue) And from what I see each input line is written to exactly one output file, so that's a boatload of calls. I would not be surprised if that were the main source for cpu in this algorthm. That makes the bash/python discussion pretty moot... If the input files have a lot of 'chunks' that go to the same output file, it might be fairly easy to gobble up the ones that go together and write them in a single go. Based on more heuristics, you may be able to keep a few of those buckets to avoid appending one record at a time, disposing each bucket when it's full enough upon switch. How feasable would it be to use sed to rewrite the line with the filename isolated and sort each input file before dealing out to the output files? That takes memory, but so does an infinite number of open files. Obviously any advance knowledge about your size of infinite helps decide... Rob ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
