Since no one else seems to be pointing this out, i can see at least one
potential optimization:
On 1/29/13 4:13 PM, John McKown wrote:
>
> If you're interested, the bash script looks like:
>
> #!/bin/bash
> for i in irradu00.g*.bz2;do
> gen=${i#irradu00.}; # remove prefix
> gen=${gen%.bz2}; # remove suffix, leaving generation
> bzcat $i |\
> while read line;do
> fn=${line%% *} # remove all trailing characters after a space
> ft=${line:9:8} # get second word
> ft=${ft%% *} # and remove trailing spaces
Here:
> echo "${line}" >>${fn}.${ft}.${gen}.tx2;
This will re-open the output file and seek to the end every output.
That's a factor of 2 or 3 more syscalls every time. (possibly plus the
fork cost to call 'echo', although I seem to recall bash has an echo
builtin, so this may not be an issue)
> done;
> done
>
One possible optimization to writing this in perl / python: keep open
output file descriptors in a dictionary aka hash structure based on
filename. If the file is already open and in that hash, then it's a
single call to output the data (plus no fork overhead). If it's not,
open it and save the open file descriptor in the dict/hash.
Be interesting to measure the speed of that vs. what you have above. I
wouldn't bother for smaller datasets, but for the multi hundred gigabyte
filesets you're working with it may make an measurable difference.
-- Pat
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
