I'm now using a Perl script which Malcom Beattie kindly gave me. I made some minor changes to be more generalized, but the main logic is his. I ran it and it was significantly faster than BASH. And I then managed to use up all the space in the filesystem that I had it on. OOPS. I'm going to need to process each input, then bzip2 the output. I guess that I can add that logic into the existing Perl script as well.
On Mon, Feb 4, 2013 at 3:05 AM, Pavelka, Tomas <[email protected]> wrote: > The internal bash parameter expansion functions (e.g. ${line%% *}) tend to be > quite inefficient. Here is one example, compare the performance of bash > substitution to Perl substitution: > > #!/bin/bash > comma_sep=$(perl -e 'for($i=0;$i<1000;$i++) { print("$i;") };') > time space_sep=${comma_sep//;/ } > time space_sep=$(echo $comma_sep | perl -pe 's/;/ /g') > > On my machine, the times look like this: > > Bash: > real 0m2.140s > user 0m2.049s > sys 0m0.002s > > Perl: > real 0m0.007s > user 0m0.010s > sys 0m0.002s > > I got burned more than once by unexpected inefficiencies in bash. As others > have said, Perl is very fast and reliable for large text manipulations. > > (The script is just an example to show the point. Replacing commas with > spaces through substitution is not a good idea. Transliteration will do the > job faster.) > > HTH, > > Tomas > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ -- Maranatha! <>< John McKown ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
