Pawel, When sorting very large files one consideration must be whether the file is to be sorted multiple times.
Also, it can be significantly more efficient to split the file into smaller subsets, sort the subsets, then perform a merge to recombine them. What you have to remember is that the time required for a sort does not grow linearly with the number of records, but is an exponential function, when the exact equation is dependent upon the total number of records, the sorting algorithm chosen, and the ordering of the key fields in the input file. A worst case scenario can be calculated. However, the mathematics is difficult and far too few analysts these days are willing to take the time. If you are really interesting, take a look at "The Art of Computer Programming, v3, "Sorting and Searching", by Donald E. Knuth John P. Baker -----Original Message----- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of Pawel Leszczynski Sent: 03/09/2008 5:06 PM To: [email protected] Subject: how fast can I sort on mainframe (using DFSORT)? Hello everybody, I realize subject is VEEEERY broad and my question VEEEERY general, but... Recently in our shop we are reviewing our whole batch processing. Most of the time of EOD processing is consumed by sort of many big sequential files. (One such file has approximately order of 10GB, 10mln records) I listed few tens of such batch jobs (the longest-lasting ones) and computed mean sorting rate. It appeared to be about 1GB/min ~ 17MB/sec I suppose it's very poor result(???). Can you tell me how much I can improve this? These batch jobs are little CPU-consuming (~10% of one CPU), I suppose that major concern is to: -limit EXCPS (1) -increase throughput rate from DASD to central storage (2) I realize that sorting whole file in central storage (hiperspace sorting) would eliminate need to use work files and EXCPS to them. How much central storage is needed to handle in-storage sorting for let's say 10 GB file??? How much can I improve (2)? Can you tell me what is mean sorting rate in your installation? Before starting I would just like to know if I can achieve substantial improvement. TIA, Pawel Leszczynski PKO BP SA ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
