Chris,
Since I joined the team as the performance lead a couple years ago,
Frank now defers these types of questions to me. Everything you (and all
the others that responded) have said are good points about DFSORT
performance. But the multitude of recommendations just reiterates the
point in my original response that there are so many factors, you could
spend weeks tuning things that don't improve anything. Just as I preached
during my 15+ years of batch tuning, I still say that before you start
tuning anything, you have to determine where you are spending time. Then
you start your tuning there. I always prefer to get some data (sysouts,
SMF, etc.) before I start making recommendations. That is why I asked
Pawel to send us a few sample sysouts as a start.
Have a nice day,
Dave Betten
DFSORT Development, Performance Lead
IBM Corporation
email: [EMAIL PROTECTED]
DFSORT/MVSontheweb at http://www.ibm.com/storage/dfsort/
IBM Mainframe Discussion List <[email protected]> wrote on 03/10/2008
12:48:05 AM:
> Pawel,
>
> I have been out of the sorting business for a while, but the target you
> can go for is what I call 2T, where T is the time to read in the data.
>
> The 2T comes from the fact that you have to read it in and then write it
> out.
>
> Writing to sort work files should not be the problem, hopefully the sort
> is writing to several of them in parallel during the sort phase and
> reading from them in parallel during the merge phase.
>
> EXCP's are an accountant's measurement, not a tuner's measurement. Look
> at your SMF data and see what the channel connect times are for SORTIN
> and SORTOUT. If the elapsed time is significantly greater than those
> two added together then you have to start looking for the bottleneck.
>
> My first reaction to your 17M/second was a question. Is that data
> sorted divided by total time? I assume it is. If so then you are
> getting about 34M/second on input and output, and that is not a bad
> number.
>
> Tape and DASD inputs and output will give you different numbers. There
> tends to be more interference with tape files. Fewer channels, no PAV.
>
> Back when DASD was real 3390, 4M/second was the target, so you are doing
> pretty good. I couch that with the fact that I have not run throughput
> tests on the various DASD subsystems available today. 34M may be great
> or it may be just average. I don't know.
>
> The best thing you can do is give each sort enough memory. You will
> need to ask Frank Yeager what the target amounts of memory are for a 10G
> sort. My guess is about 17M to 24M. A 100G sort might need 50M to 75M.
> A 1G sort might need 5M to 7M.
>
> What you are attempting to do is make it so there is no need to do an
> intermediate merge, or in the case of blockset sorting, that the final
> merge remains efficient.
>
> I would not attempt to get a 10G sort to run as a turnaround or incore
> sort. It will put a large strain on RSM/ASM and not buy you that much.
>
> I hope Frank weighs in on this.
>
> Chris Blaicher
> BMC Software, Inc.
> These are my own personal comments and do not reflect the views of my
> employer. Please do your own evaluation of any statements or
> suggestions made.
>
> -----Original Message-----
> From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On
> Behalf Of Pawel Leszczynski
> Sent: Sunday, March 09, 2008 4:06 PM
> To: [email protected]
> Subject: how fast can I sort on mainframe (using DFSORT)?
>
> Hello everybody,
>
> I realize subject is VEEEERY broad and my question VEEEERY general,
> but...
>
> Recently in our shop we are reviewing our whole batch processing.
> Most of the time of EOD processing is consumed by sort of many big
> sequential
> files. (One such file has approximately order of 10GB, 10mln records)
> I listed few tens of such batch jobs (the longest-lasting ones)
> and computed mean sorting rate.
> It appeared to be about 1GB/min ~ 17MB/sec
>
> I suppose it's very poor result(???).
>
> Can you tell me how much I can improve this?
>
> These batch jobs are little CPU-consuming (~10% of one CPU),
> I suppose that major concern is to:
>
> -limit EXCPS (1)
> -increase throughput rate from DASD to central storage (2)
>
> I realize that sorting whole file in central storage (hiperspace
> sorting) would
> eliminate need to use work files and EXCPS to them.
> How much central storage is needed to handle in-storage sorting for
> let's say 10 GB file???
>
> How much can I improve (2)?
>
> Can you tell me what is mean sorting rate in your installation?
> Before starting I would just like to know if I can achieve substantial
> improvement.
>
> TIA,
> Pawel Leszczynski
> PKO BP SA
>
> ----------------------------------------------------------------------
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
> Search the archives at http://bama.ua.edu/archives/ibm-main.html
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
