I think that echo will issue a openfile-appendrecord-closefile every time, and that costs. The way I fix these scripts is to code them in std c, it should only require a: gcc -o mypgm mypgm.c
I had a similar case some years ago, it also read and write files but also did calculations via expr cmd. It was daily cpu-statistics from about 120 zLinux servers going into a database. Changing from bash into c reduced time to execute from 30 min to 10 seconds. BR /Tore ________________________________________________ Tore Agblad System programmer, Volvo IT certified IT Architect Volvo Information Technology IMO Mainframe Databases & zOpen Gothenburg Dept. DE56430 SE-405 08, Gothenburg Sweden Soft Phone: +46 313086198 Telephone: +46-31-3233569 E-mail: [email protected] http://www.volvo.com/volvoit/global/en-gb/ -----Original Message----- From: Linux on 390 Port [mailto:[email protected]] On Behalf Of John McKown Sent: den 29 januari 2013 23:13 To: [email protected] Subject: Speed of BASH script vs. Python vs. Perl vs. compiled This is more a curiosity question. I have written a bash script which reads a bzip2 compressed set of files. For each record in the file, it writes the record into a file name based on the first two "words" in the record and the "generation number" from the input file name. Do to the extreme size of the input (47 files, each of which would be around 120 Gb to 180 Gb expanded or 23 to 27 million lines - very large). Basically there are probably around 50 or so (don't know) possible combinations of the "words". I'm wondering if I rewriting the script into either Python or Perl (both basically interpreted) would be worth my while. Or should I go with a compiler such as C/C++? Or, lastly, is it basically irrelevant due to the extremely large number of records and the minimal processing; which means that I/O will dominate the application. If you're interested, the bash script looks like: #!/bin/bash for i in irradu00.g*.bz2;do gen=${i#irradu00.}; # remove prefix gen=${gen%.bz2}; # remove suffix, leaving generation bzcat $i |\ while read line;do fn=${line%% *} # remove all trailing characters after a space ft=${line:9:8} # get second word ft=${ft%% *} # and remove trailing spaces echo "${line}" >>${fn}.${ft}.${gen}.tx2; done; done If you're curious about the "set ${line}", I just couldn't figure out a way to parse -- Maranatha! <>< John McKown ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/ ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
