Well, I know that downloading the 160 Gig uncompressed data takes about 8 hours on the 10 Gig/sec Ethernet connection. I then bzip2 compress that to about 50 Meg. Which I binary upload back to z/OS for safety (since it's setting on my Linux desktop) in just a few minutes. But bzgrep can scan the compressed file fairly quickly (FSVO "quickly"). So, if I use Perl or C to read the compressed file, expand it on-the-fly and process the records, then recompress the data before writing it to the proper result file (which is what I eventually want anyway), that should greatly reduce the I/O. I need to see how easy it is to do that in Perl vs Python vs C. Again, this is all for my personal use so that I can answer questions more quickly about "who did what to whom and how often? <grin/>" according to RACF.
On Wed, Jan 30, 2013 at 8:06 AM, Shane G <[email protected]> wrote: > On Thu, Jan 31st, 2013 at 12:44 AM, John McKown wrote: > >> Thanks to all for the input! I _tried_ to run the script over night. I >> added an echo to tell me which input file I was working on. I came in >> this morning. It had been running from 14:00 to 06:30 (16 1/2 hours) >> and was still on the first input file. That ain't gonna cut it. Time >> to rethink. Using a Perl hash to contain an open file handle seems >> logical. As does buffering multiple records per output file to do a >> single I/O to write them. But I may be forced into using C or C++ for >> speed. Too bad I'm not a very good C programmer. > > Interesting timing. I was about to suggest you utilize your perl skills. > Having originally ignored it, I now use awk extensively for text > parsing/reduction. But for *BIG* jobs, perl is it. > > But all that input I/O is going to be death whatever you choose. > > Shane ... > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ -- Maranatha! <>< John McKown ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
