Pawel, I have been out of the sorting business for a while, but the target you can go for is what I call 2T, where T is the time to read in the data.
The 2T comes from the fact that you have to read it in and then write it out. Writing to sort work files should not be the problem, hopefully the sort is writing to several of them in parallel during the sort phase and reading from them in parallel during the merge phase. EXCP's are an accountant's measurement, not a tuner's measurement. Look at your SMF data and see what the channel connect times are for SORTIN and SORTOUT. If the elapsed time is significantly greater than those two added together then you have to start looking for the bottleneck. My first reaction to your 17M/second was a question. Is that data sorted divided by total time? I assume it is. If so then you are getting about 34M/second on input and output, and that is not a bad number. Tape and DASD inputs and output will give you different numbers. There tends to be more interference with tape files. Fewer channels, no PAV. Back when DASD was real 3390, 4M/second was the target, so you are doing pretty good. I couch that with the fact that I have not run throughput tests on the various DASD subsystems available today. 34M may be great or it may be just average. I don't know. The best thing you can do is give each sort enough memory. You will need to ask Frank Yeager what the target amounts of memory are for a 10G sort. My guess is about 17M to 24M. A 100G sort might need 50M to 75M. A 1G sort might need 5M to 7M. What you are attempting to do is make it so there is no need to do an intermediate merge, or in the case of blockset sorting, that the final merge remains efficient. I would not attempt to get a 10G sort to run as a turnaround or incore sort. It will put a large strain on RSM/ASM and not buy you that much. I hope Frank weighs in on this. Chris Blaicher BMC Software, Inc. These are my own personal comments and do not reflect the views of my employer. Please do your own evaluation of any statements or suggestions made. -----Original Message----- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of Pawel Leszczynski Sent: Sunday, March 09, 2008 4:06 PM To: [email protected] Subject: how fast can I sort on mainframe (using DFSORT)? Hello everybody, I realize subject is VEEEERY broad and my question VEEEERY general, but... Recently in our shop we are reviewing our whole batch processing. Most of the time of EOD processing is consumed by sort of many big sequential files. (One such file has approximately order of 10GB, 10mln records) I listed few tens of such batch jobs (the longest-lasting ones) and computed mean sorting rate. It appeared to be about 1GB/min ~ 17MB/sec I suppose it's very poor result(???). Can you tell me how much I can improve this? These batch jobs are little CPU-consuming (~10% of one CPU), I suppose that major concern is to: -limit EXCPS (1) -increase throughput rate from DASD to central storage (2) I realize that sorting whole file in central storage (hiperspace sorting) would eliminate need to use work files and EXCPS to them. How much central storage is needed to handle in-storage sorting for let's say 10 GB file??? How much can I improve (2)? Can you tell me what is mean sorting rate in your installation? Before starting I would just like to know if I can achieve substantial improvement. TIA, Pawel Leszczynski PKO BP SA ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
