On Sat, Mar 08, 2014 at 10:59:22AM -0500, Steve Tolkin wrote:
> # return code: 0 == success; 1 == some warnings; 2 == some errors
> my $rc = 0;
This value never changes. I assume the larger program could change it.
> my $split_char=','; # CHANGE ME IF NEEDED (later use getopts)
>
> my @aoh; # array of hashes: key is the field value; value is the count
> my $numlines = 0;
> my $firstline;
> while (<>)
> {
> chomp;
> $firstline = $_ if $numlines == 0;
More idiomatically written as:
$firstline ||= $_;
> $numlines++;
> my @data;
>
> # Seemingly extra code below is for compatibility with older perl
> versions
> # But this might not be needed anymore.
> if ( $split_char ne '' )
> { @data = split(/$split_char/,$_); }
> else
> { @data = split; }
If you're really looking for optimizations, this test can be moved outside
the loop. Since no arg split splits on spaces, your check can be something
like:
$split_char = ' ' unless $split_char;
Then the test and two split options can be replaced with
@data = split /$split_char/, $_;
There may also be some benefit to hoisting the regular expression outside the
loop:
my $re = qr/$split_char/o;
...
@data = split $re, $_;
If there's any, it will be tiny, but may be appreciable given your input size.
> # Populate array of hashes. (This uses perl "autovivification"
> feature.)
> for (my $i = 0; $i < scalar @data; $i++) {
> $aoh[$i]{$data[$i]}++;
> }
> }
Nit: "scalar @data" can be replaced with @data.
> # print output
> print "filename: $ARGV\n"; # This writes a "-" if data is piped in
> if ($numlines >0) {
> print "firstline: $firstline\n";
> }
> print "numlines: $numlines\n";
> for (my $i = 0; $i < scalar @aoh; $i++) {
> # The number of keys in a hash is the "count distinct" number
> print "field#: ", $i, ", distinct: ", scalar keys %{$aoh[$i]}, "\n";
Nit: This reads better as a printf, I think.
printf "field#: %d, distinct: %d\n", $i, scalar keys %{$aoh[$i]};
> }
>
> exit $rc;
This is always 0, as noted above.
My initial thought at improvement was to avoid the split and walk through each
line looking for a $split_char or a "\n", but that just duplicates split in
perl instead of C. I think you've got just about the fastest program in perl.
For fun, I wrote a version in Go and it's twice as fast as the perl
version. I imagine a C version would be faster yet, but I get paid for that
kind of fun. I'd be happy to send you the Go version if you're interested.
-Gyepi
_______________________________________________
Boston-pm mailing list
[email protected]
http://mail.pm.org/mailman/listinfo/boston-pm
