John McKown writes:
> This is more a curiosity question. I have written a bash script which
> reads a bzip2 compressed set of files. For each record in the file, it
> writes the record into a file name based on the first two "words" in
> the record and the "generation number" from the input file name. Do to
> the extreme size of the input (47 files, each of which would be around
> 120 Gb to 180 Gb expanded or 23 to 27 million lines - very large).
> Basically there are probably around 50 or so (don't know) possible
> combinations of the "words". I'm wondering if I rewriting the script
> into either Python or Perl (both basically interpreted) would be worth
> my while.
Perl (and Python) aren't simply interpreted. In the case of perl, it
compiles the source into an internal op tree (rather like bytecode)
while performing a decent amount of cheap optimisation (peephole
optimisation mostly) and then runs that internal structure. Python
will do something similar but the internal representation is
different. Most if this isn't relevant to your situation here though.
> Or should I go with a compiler such as C/C++? Or, lastly, is
> it basically irrelevant due to the extremely large number of records
> and the minimal processing; which means that I/O will dominate the
> application.
It's not I/O that dominating in your implementation below, it is
(as others have spotted) the opening and closing the relevant file
on every single line of input. Either Perl or Python will let you
remove this cost entirely. In fact, on your bash script below, bash
seems to read each character from its uncompressed input in a
separate read syscall which is dreadful but may be fixable.
> If you're interested, the bash script looks like:
>
> #!/bin/bash
> for i in irradu00.g*.bz2;do
> gen=${i#irradu00.}; # remove prefix
> gen=${gen%.bz2}; # remove suffix, leaving generation
> bzcat $i |\
> while read line;do
> fn=${line%% *} # remove all trailing characters after a space
> ft=${line:9:8} # get second word
> ft=${ft%% *} # and remove trailing spaces
> echo "${line}" >>${fn}.${ft}.${gen}.tx2;
> done;
> done
This Perl program (or analogue in Python or whatever) is likely to
give (and strace on some small test data shows) much, much better
behaviour for larger input files:
#!/usr/bin/perl
use strict;
use IO::File;
my %fhcache;
sub newfh {
my $filename = shift;
my $fh = IO::File->new($filename, "a") or die "$filename: $!\n";
$fhcache{$filename} = $fh;
return $fh;
}
sub getfh {
my $filename = shift;
return $fhcache{$filename} || newfh($filename);
}
foreach my $infile (<irradu00.g*.bz2>) {
open(IN, "bzcat $infile|") or die "bzcat $infile: $!\n";
my ($gen) = $infile =~ /\.(.*)\.bz2/;
while (<IN>) {
my ($fn, $ft) = split;
getfh("$fn.$ft.$gen.tx2")->print($_);
}
close(IN);
}
--Malcolm
--
Malcolm Beattie
Mainframe Systems and Software Business, Europe
IBM UK
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
