Re: Has anyone measured CPU savings using external SORT's vs. internal (COBOL) SORT's?
A bit late to the discussion but did the O.P. consider REPLACING the COBOL code with a pure DFSORT / ICETOOL implementation? That MIGHT remove duplicate data moves. Such a process a customer and I worked together on last week took a 200 minute job and made it run in 10 minutes. YMMV. :-) Cheers, Martin Martin Packer, zChampion, Principal Systems Investigator, Worldwide Banking Center of Excellence, IBM +44-7802-245-584 email: martin_pac...@uk.ibm.com Twitter / Facebook IDs: MartinPacker Blog: https://www.ibm.com/developerworks/mydeveloperworks/blogs/MartinPacker From: Clark Morris cfmpub...@ns.sympatico.ca To: IBM-MAIN@listserv.ua.edu Date: 04/12/2013 15:57 Subject:Re: Has anyone measured CPU savings using external SORT's vs. internal (COBOL) SORT's? Sent by:IBM Mainframe Discussion List IBM-MAIN@listserv.ua.edu On 2 Dec 2013 06:14:42 -0800, in bit.listserv.ibm-main you wrote: Sorry about the late reply. The last time I seriously looked, the COBOL sort verb invoked the installation sort (DFsort, SYNCSORT,). The COBOL program effectively became the E15/E35 sort exits. On that basis, I would not expect any significant difference in CPU time consumed, *AND* as someone previously noted, a possible significant increase in elapsed time. HTH, snip It has been suggested to management here that there could be potentially significant CPU savings from re-engineering application programs such that any SORT's are done in a separate step, so that a program with a single internal SORT would be broken up into a pre-SORT process followed by an external SORT of the massaged data followed by a post-process of the SORTed data. /snip While the sort products do more efficient I/O than the standard access methods, this advantage is lost because an extra file may be written for the sort to read. In the past the major saving by using stand alone sorts was due to main memory limitations. By giving more memory to the sort the number of intermediate passes could be reduced. In today's environment that normally is not a consideration. Clark Morris -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs. internal (COBOL) SORT's?
On 2 Dec 2013 06:14:42 -0800, in bit.listserv.ibm-main you wrote: Sorry about the late reply. The last time I seriously looked, the COBOL sort verb invoked the installation sort (DFsort, SYNCSORT,). The COBOL program effectively became the E15/E35 sort exits. On that basis, I would not expect any significant difference in CPU time consumed, *AND* as someone previously noted, a possible significant increase in elapsed time. HTH, snip It has been suggested to management here that there could be potentially significant CPU savings from re-engineering application programs such that any SORT's are done in a separate step, so that a program with a single internal SORT would be broken up into a pre-SORT process followed by an external SORT of the massaged data followed by a post-process of the SORTed data. /snip While the sort products do more efficient I/O than the standard access methods, this advantage is lost because an extra file may be written for the sort to read. In the past the major saving by using stand alone sorts was due to main memory limitations. By giving more memory to the sort the number of intermediate passes could be reduced. In today's environment that normally is not a consideration. Clark Morris -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs. internal (COBOL) SORT's?
Except that IBM COBOL only provides us with the ability to give a record to the SORT during INPUT PROCEDURE processing, requiring us to use COBOL I/O to read the records to be massaged and then sorted, whereas a true E15 exit gets the records read from SORTIN by the sort itself passed to it, and then decides what to do with each record -- use the one given, insert another in its place, or delete it. Similar logic is required in the COBOL OUTPUT PROCEDURE, which only gets records back from the SORT, and must write them out using only COBOL I/O facilities, whereas a true E35 exit can give the record back to the SORT and let SORT use its own I/O facilities to write out the sorted and massaged record. There may be some potential for decrease in CPU time for true E15/E35 exits which would come from taking advantage of the SORT's own optimized I/O facilities, but that is just a theory which I have not tested. Other replacements for COBOL SORT processing discussed in this thread do certainly seem to be almost guaranteed to increase rather than decrease CPU consumption, if only from the increase in the number of passes that must be made over the data. And paying attention to optimizing SORT's control parameters both on a shop-wide and particular-application level seems likely to produce some benefits without any program restructuring at all. Thanks to all who participated in this discussion. I appreciate the help. Peter -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Staller, Allan Sent: Monday, December 02, 2013 9:16 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs. internal (COBOL) SORT's? Sorry about the late reply. The last time I seriously looked, the COBOL sort verb invoked the installation sort (DFsort, SYNCSORT,). The COBOL program effectively became the E15/E35 sort exits. On that basis, I would not expect any significant difference in CPU time consumed, *AND* as someone previously noted, a possible significant increase in elapsed time. HTH, snip It has been suggested to management here that there could be potentially significant CPU savings from re-engineering application programs such that any SORT's are done in a separate step, so that a program with a single internal SORT would be broken up into a pre-SORT process followed by an external SORT of the massaged data followed by a post-process of the SORTed data. /snip -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs. internal (COBOL) SORT's?
Sorry about the late reply. The last time I seriously looked, the COBOL sort verb invoked the installation sort (DFsort, SYNCSORT,). The COBOL program effectively became the E15/E35 sort exits. On that basis, I would not expect any significant difference in CPU time consumed, *AND* as someone previously noted, a possible significant increase in elapsed time. HTH, snip It has been suggested to management here that there could be potentially significant CPU savings from re-engineering application programs such that any SORT's are done in a separate step, so that a program with a single internal SORT would be broken up into a pre-SORT process followed by an external SORT of the massaged data followed by a post-process of the SORTed data. /snip -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
I was delighted to learn---authoritatively from Chris Blaicher---that SYNCSORT no longer makes routine use of BSAM or QSAM to read/write its sortin/sortout datasets. His point that there are no generic, one-size-fits-all solutions is also important. It conceded, my own experience nevertheless strongly suggests that strung-out, multiple job-step schemes of the sort that the OP outlined are almost always inferior to those that attach the sort and use its exits, at least for traditional I/O-bound commercial processing, MFUs and the like. (Well used, COBOL QSAM i/o can be very efficient indeed; but it is almost always used in what I shall politely call suboptimal fashion.) The question whether the sort can itself do any required preprocessing needs always, of course, to be examined John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
Chris, Thanks for confirming my assumption that your product uses its optimized I/O facilities for SORTIN/OUT. I agree that there is no single best way to design a process. As usual, it depends Peter -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Blaicher, Christopher Y. Sent: Monday, November 25, 2013 11:10 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? John, SyncSort has for many years not used any regular access methods in the normal course of processing SORTIN or SORTOUT. There are exceptions to this such as compressed files where we have to use BSAM, but for the most part, we do not use traditional access methods. As to the original subject matter, it is impossible to make a single general statement about what way is the best way to design a process. If you are using a COBOL program or exit to transform data or select a subset of records, in general it is faster both in elapsed time and CPU time to use the many features of a sort (INCLUDE/OMIT/INREC/OUTREC/OUTFILE) rather than an exit. As with everything in computing, your mileage may vary. Chris Blaicher Principal Software Engineer, Software Development Syncsort Incorporated 50 Tice Boulevard, Woodcliff Lake, NJ 07677 P: 201-930-8260 | M: 512-627-3803 E: cblaic...@syncsort.com -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of John Gilmore Sent: Monday, November 25, 2013 11:49 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? Note that the highly efficient i/o operations of SYNCSORT and DFSORT are their internal ones. They must and do use access methods to read sortin and write sortout. They do of course use these access methods more efficiently than many/most COBOL programs, but the big i/o savings are elsewhere. John Gilmore, Ashland, MA 01721 - USA -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
Syncsort whilst performing optimally, does not work well with transparency products. Many moons ago I discovered this fact and was forced to go the E15 way. A couple of examples that Syncsort did not work with: CA-Datacom VSAM transparency and IBM - VSAM transparency. CA-Sort worked as did DFSORT. In these circumstances, one is forced to develop an E15 exit to present the rows to sort or perform the Extract/Sort multi step job. On Tue, Nov 26, 2013 at 3:09 PM, Blaicher, Christopher Y. cblaic...@syncsort.com wrote: John, SyncSort has for many years not used any regular access methods in the normal course of processing SORTIN or SORTOUT. There are exceptions to this such as compressed files where we have to use BSAM, but for the most part, we do not use traditional access methods. As to the original subject matter, it is impossible to make a single general statement about what way is the best way to design a process. If you are using a COBOL program or exit to transform data or select a subset of records, in general it is faster both in elapsed time and CPU time to use the many features of a sort (INCLUDE/OMIT/INREC/OUTREC/OUTFILE) rather than an exit. As with everything in computing, your mileage may vary. Chris Blaicher Principal Software Engineer, Software Development Syncsort Incorporated 50 Tice Boulevard, Woodcliff Lake, NJ 07677 P: 201-930-8260 | M: 512-627-3803 E: cblaic...@syncsort.com -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of John Gilmore Sent: Monday, November 25, 2013 11:49 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? Note that the highly efficient i/o operations of SYNCSORT and DFSORT are their internal ones. They must and do use access methods to read sortin and write sortout. They do of course use these access methods more efficiently than many/most COBOL programs, but the big i/o savings are elsewhere. John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN ATTENTION: - The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- Wayne V. Bickerdike -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
It has been suggested to management here that there could be potentially significant CPU savings from re-engineering application programs such that any SORT's are done in a separate step, so that a program with a single internal SORT would be broken up into a pre-SORT process followed by an external SORT of the massaged data followed by a post-process of the SORTed data. The first obvious factor is that SORT (at least Syncsort and DFSORT) are *far* more efficient at I/O than any COBOL program can be. It is also obvious that the data volume would affect the relative CPU cost of the two methods, with small volume possibly favoring an internal SORT and large(r) volume possibly favoring the external SORT process, FSVO large(r). Compressed (z/OS compression, not disk subsystem compression) vs non-compressed data files could also be another factor in CPU differences. Has anyone else been asked to measure whether this claim is true or not, and if true where the break point in volume might be? TIA for any insight you can provide. Peter -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
Peter (et al): If this is the case, and the decision is made to make these changes, contact me. I have access to a partner product that can modify your programs and batch processes automatically. With Regards, Mitch McCluhan, Legacy Modernization Consultant www.lcmg.us -Original Message- From: Farley, Peter x23353 peter.far...@broadridge.com To: IBM-MAIN IBM-MAIN@LISTSERV.UA.EDU Sent: Mon, Nov 25, 2013 7:43 am Subject: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? It has been suggested to management here that there could be potentially ignificant CPU savings from re-engineering application programs such that any ORT's are done in a separate step, so that a program with a single internal ORT would be broken up into a pre-SORT process followed by an external SORT of he massaged data followed by a post-process of the SORTed data. The first obvious factor is that SORT (at least Syncsort and DFSORT) are *far* ore efficient at I/O than any COBOL program can be. It is also obvious that he data volume would affect the relative CPU cost of the two methods, with mall volume possibly favoring an internal SORT and large(r) volume possibly avoring the external SORT process, FSVO large(r). Compressed (z/OS ompression, not disk subsystem compression) vs non-compressed data files could lso be another factor in CPU differences. Has anyone else been asked to measure whether this claim is true or not, and if rue where the break point in volume might be? TIA for any insight you can provide. Peter - This message and any attachments are intended only for the use of the addressee nd may contain information that is privileged and confidential. If the reader f the message is not the intended recipient or an authorized representative of he intended recipient, you are hereby notified that any dissemination of this ommunication is strictly prohibited. If you have received this communication in rror, please notify us immediately by e-mail and delete the message and any ttachments from your system. -- or IBM-MAIN subscribe / signoff / archive access instructions, end email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
The scheme you are considering is---if I understand it---that of read prepin==|preprocess|==write sortin read sortin==|external sort|==write sortout read sortout==|postprocess|==write postout It involves six i/o operations per record, and it thus has little to recommend it. If instead you use Exits 15 and 35 of the external sort, either DFSORT or SYNCSORT, having the preprocessor hand unsorted records to the external sort using Exit 15 and the postprocessor take sorted records back from the external sort at Exit 35, all in one job step, you can save four of these I/O operations. This second scheme is in my experience a very muich better one than the one you are considering. It gives you all of the benefits of using the external sort and avoids gratuitous I/O. Note that the highly efficient i/o operations of SYNCSORT and DFSORT are their internal ones. They must and do use access methods to read sortin and write sortout. They do of course use these access methods more efficiently than many/most COBOL programs, but the big i/o savings are elsewhere. John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
Alan: Actually, recoding the JCL is very simple. I have a business partner that can provide a solution that can recode thousands of JCL steps in a matter of minutes to any desired coding, including site standards, if applicable. Regards, Mitch McCluhan, Legacy Modernization Consultant -Original Message- From: Alan Field alan_c_fi...@bluecrossmn.com To: IBM-MAIN IBM-MAIN@LISTSERV.UA.EDU Sent: Mon, Nov 25, 2013 8:49 am Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? Peter, Review/research the COBOL compiler FASTSRT option. If you are using it hat you suggest will possibly make things worse. If you aren't using it, it may be a cleaner solution that recoding JCL. to chieve the desired savings. Alan Field echnical Engineer Principal CBS Minnesota Phone: 651.662.3546 Mobile: 651.428.8826 From: Farley, Peter x23353 peter.far...@broadridge.com o: IBM-MAIN@LISTSERV.UA.EDU, ate: 11/25/2013 09:43 ubject:Has anyone measured CPU savings using external SORT's vs nternal (COBOL) SORT's? ent by:IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU It has been suggested to management here that there could be potentially ignificant CPU savings from re-engineering application programs such that ny SORT's are done in a separate step, so that a program with a single nternal SORT would be broken up into a pre-SORT process followed by an xternal SORT of the massaged data followed by a post-process of the ORTed data. The first obvious factor is that SORT (at least Syncsort and DFSORT) are far* more efficient at I/O than any COBOL program can be. It is also bvious that the data volume would affect the relative CPU cost of the two ethods, with small volume possibly favoring an internal SORT and large(r) olume possibly favoring the external SORT process, FSVO large(r). ompressed (z/OS compression, not disk subsystem compression) vs on-compressed data files could also be another factor in CPU differences. Has anyone else been asked to measure whether this claim is true or not, nd if true where the break point in volume might be? TIA for any insight you can provide. Peter - This message and any attachments are intended only for the use of the ddressee and may contain information that is privileged and confidential. f the reader of the message is not the intended recipient or an uthorized representative of the intended recipient, you are hereby otified that any dissemination of this communication is strictly rohibited. If you have received this communication in error, please otify us immediately by e-mail and delete the message and any attachments rom your system. -- or IBM-MAIN subscribe / signoff / archive access instructions, end email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN - or IBM-MAIN subscribe / signoff / archive access instructions, end email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
facetiously Perhaps they should also look at removing all embedded SQL call into a pre-step to unload the required data to a temporary file, read it, write a new temporary output file, then have a post-step which loads the changed DB2 data back into the appropriate tables. That would also reduce the CPU usage (at least of the step in question), right? /facetiously You've had a cattle stampede. I know because there is a lot of B.S. left behind. Or perhaps the idea came from a C programmer who thinks that COBOL.is using something like C's built in qsort() function. On Mon, Nov 25, 2013 at 10:41 AM, Alan Field alan_c_fi...@bluecrossmn.comwrote: Peter, Review/research the COBOL compiler FASTSRT option. If you are using it what you suggest will possibly make things worse. If you aren't using it, it may be a cleaner solution that recoding JCL. to achieve the desired savings. Alan Field Technical Engineer Principal BCBS Minnesota Phone: 651.662.3546 Mobile: 651.428.8826 From: Farley, Peter x23353 peter.far...@broadridge.com To: IBM-MAIN@LISTSERV.UA.EDU, Date: 11/25/2013 09:43 Subject:Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? Sent by:IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU It has been suggested to management here that there could be potentially significant CPU savings from re-engineering application programs such that any SORT's are done in a separate step, so that a program with a single internal SORT would be broken up into a pre-SORT process followed by an external SORT of the massaged data followed by a post-process of the SORTed data. The first obvious factor is that SORT (at least Syncsort and DFSORT) are *far* more efficient at I/O than any COBOL program can be. It is also obvious that the data volume would affect the relative CPU cost of the two methods, with small volume possibly favoring an internal SORT and large(r) volume possibly favoring the external SORT process, FSVO large(r). Compressed (z/OS compression, not disk subsystem compression) vs non-compressed data files could also be another factor in CPU differences. Has anyone else been asked to measure whether this claim is true or not, and if true where the break point in volume might be? TIA for any insight you can provide. Peter -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- This is clearly another case of too many mad scientists, and not enough hunchbacks. Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
I mean the second, using the COBOL SORT verb to invoke the SORT from within the program. And I tend to agree with you. Just looking for reasons other than the ones I have thought of to refute the claim that was made. Peter -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Shmuel Metz (Seymour J.) Sent: Monday, November 25, 2013 11:30 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? In 985915eee6984740ae93f8495c624c6c231f711...@jscpcwexmaa1.bsg.ad.adp.com, on 11/25/2013 at 10:43 AM, Farley, Peter x23353 peter.far...@broadridge.com said: It has been suggested to management here that there could be potentially significant CPU savings from re-engineering application programs such that any SORT's are done in a separate step, so that a program with a single internal SORT would be broken up into a pre-SORT process followed by an external SORT of the massaged data followed by a post-process of the SORTed data. By internal sort do you mean a sort programmed in COBOL, or do you mean invoking the sort utility from within the COBOL program? If the latter, why would separate job steps be more efficient? For that matter, why wouldn't it be less efficient? -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
FASTSRT is restricted though. According to the FM (Programming Guide): You cannot use an input procedure on an input file nor an output procedure on an output file. Most processing here requires both of those, frequently with far more processing than can be supported by the SORT control features suggested by the FM as possible replacement(s) for INPUT/OUTPUT PROCEDURES (viz., INREC, OUTFILE, OUTREC, INCLUDE, OMIT, STOPAFT, SKIPREC, SUM). Peter -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Alan Field Sent: Monday, November 25, 2013 11:42 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? Peter, Review/research the COBOL compiler FASTSRT option. If you are using it what you suggest will possibly make things worse. If you aren't using it, it may be a cleaner solution that recoding JCL. to achieve the desired savings. -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
*Chuckle* Probably true about the bovine excrement, but it still has to be shown to be something that management would not want to step in. Thanks for the grin that generated. Peter -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of John McKown Sent: Monday, November 25, 2013 11:57 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? facetiously Perhaps they should also look at removing all embedded SQL call into a pre-step to unload the required data to a temporary file, read it, write a new temporary output file, then have a post-step which loads the changed DB2 data back into the appropriate tables. That would also reduce the CPU usage (at least of the step in question), right? /facetiously You've had a cattle stampede. I know because there is a lot of B.S. left behind. Or perhaps the idea came from a C programmer who thinks that COBOL is using something like C's built in qsort() function. -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
Note that my COBOL sort days are long ago so what I am saying may no longer be true. What used to happen with E35 is that SORT did NOT create the final sortout file: the program was given the sorted records from the last merge set. So you saved, at a minimum, that set of I/O. Each COBOL return from the E35 exit was really the equivalent of a write to the sortout file from the sort standpoint. If I remember correctly, there was also I/O savings using E15, but I do not remember what that savings was. Lloyd From: John Gilmore jwgli...@gmail.com To: IBM-MAIN@LISTSERV.UA.EDU Sent: Monday, November 25, 2013 11:49 AM Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? The scheme you are considering is---if I understand it---that of read prepin==|preprocess|==write sortin read sortin==|external sort|==write sortout read sortout==|postprocess|==write postout It involves six i/o operations per record, and it thus has little to recommend it. If instead you use Exits 15 and 35 of the external sort, either DFSORT or SYNCSORT, having the preprocessor hand unsorted records to the external sort using Exit 15 and the postprocessor take sorted records back from the external sort at Exit 35, all in one job step, you can save four of these I/O operations. This second scheme is in my experience a very muich better one than the one you are considering. It gives you all of the benefits of using the external sort and avoids gratuitous I/O. Note that the highly efficient i/o operations of SYNCSORT and DFSORT are their internal ones. They must and do use access methods to read sortin and write sortout. They do of course use these access methods more efficiently than many/most COBOL programs, but the big i/o savings are elsewhere. John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
Thanks for the careful analysis, John. Very helpful. I had not thought of the option of converting to JCL SORT's using E15/E35 programs. That might actually yield potential savings, at least for programs that only use a single internal SORT request. The ones that use multiple SORT's in one program are a bit more work to break apart, and might or might not yield sufficient savings to justify the investment in re-engineering. Peter -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of John Gilmore Sent: Monday, November 25, 2013 11:49 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? The scheme you are considering is---if I understand it---that of read prepin==|preprocess|==write sortin read sortin==|external sort|==write sortout read sortout==|postprocess|==write postout It involves six i/o operations per record, and it thus has little to recommend it. If instead you use Exits 15 and 35 of the external sort, either DFSORT or SYNCSORT, having the preprocessor hand unsorted records to the external sort using Exit 15 and the postprocessor take sorted records back from the external sort at Exit 35, all in one job step, you can save four of these I/O operations. This second scheme is in my experience a very muich better one than the one you are considering. It gives you all of the benefits of using the external sort and avoids gratuitous I/O. Note that the highly efficient i/o operations of SYNCSORT and DFSORT are their internal ones. They must and do use access methods to read sortin and write sortout. They do of course use these access methods more efficiently than many/most COBOL programs, but the big i/o savings are elsewhere. -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
Peter: This story spans 2 decades of stories and hist and OS/s (and sorts). In the 70's we were gasping for CPU and looked at practically everything. We tried doing things to economize on CPU resources. We tore apart programs and tried to quantify items. We even had the assembler people write sort exits E15, E35 (and god yes E61) amongst others. I think looking back it came down with COBOL (I have long lost the papers we wrote). That the worst performance was what you did when you did during the input procedure and output procedure. By did I mean how much other I/O and CPU the program did. BUT in the end splitting up programs did little in saving time (CPU ELAPSED). Yes sure there is step init/ending time etc etc but we are usually talking small amounts of time . Don't get me wrong adding up times amounts to at best 10-15 minutes a day, but usually re-examining tape unmount/remount time was most of it. With a few judicious disp=(new,pass) it saved most of it. With disk there is little to manage so if you manage the resources well its very little to save. USING DISK input output takes it down even further, so doing a good job on Disk should help. The second thing is that difficulty in sorting with a program is that when it breaks (and it does) your people have to be pretty good debuggers. We had a few programmers who tried to dump their S0C4's and S0C7's on us. My rule of thumb is that only when they can present a dump that us SYSPROGS can't explain while we take any responsibility. I have kick out several programmers who were to lazy to debug. Ed On Nov 25, 2013, at 12:28 PM, Farley, Peter x23353 wrote: I mean the second, using the COBOL SORT verb to invoke the SORT from within the program. And I tend to agree with you. Just looking for reasons other than the ones I have thought of to refute the claim that was made. Peter -Original Message- From: IBM Mainframe Discussion List [mailto:IBM- m...@listserv.ua.edu] On Behalf Of Shmuel Metz (Seymour J.) Sent: Monday, November 25, 2013 11:30 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? In 985915eee6984740ae93f8495c624c6c231f711...@jscpcwexmaa1.bsg.ad.adp.co m, on 11/25/2013 at 10:43 AM, Farley, Peter x23353 peter.far...@broadridge.com said: It has been suggested to management here that there could be potentially significant CPU savings from re-engineering application programs such that any SORT's are done in a separate step, so that a program with a single internal SORT would be broken up into a pre-SORT process followed by an external SORT of the massaged data followed by a post-process of the SORTed data. By internal sort do you mean a sort programmed in COBOL, or do you mean invoking the sort utility from within the COBOL program? If the latter, why would separate job steps be more efficient? For that matter, why wouldn't it be less efficient? -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
Thanks for the thoughts, Ed. Appreciated. -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Ed Gould Sent: Monday, November 25, 2013 3:46 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? Peter: This story spans 2 decades of stories and hist and OS/s (and sorts). In the 70's we were gasping for CPU and looked at practically everything. We tried doing things to economize on CPU resources. We tore apart programs and tried to quantify items. We even had the assembler people write sort exits E15, E35 (and god yes E61) amongst others. I think looking back it came down with COBOL (I have long lost the papers we wrote). That the worst performance was what you did when you did during the input procedure and output procedure. By did I mean how much other I/O and CPU the program did. BUT in the end splitting up programs did little in saving time (CPU ELAPSED). Yes sure there is step init/ending time etc etc but we are usually talking small amounts of time . Don't get me wrong adding up times amounts to at best 10-15 minutes a day, but usually re-examining tape unmount/remount time was most of it. With a few judicious disp=(new,pass) it saved most of it. With disk there is little to manage so if you manage the resources well its very little to save. USING DISK input output takes it down even further, so doing a good job on Disk should help. The second thing is that difficulty in sorting with a program is that when it breaks (and it does) your people have to be pretty good debuggers. We had a few programmers who tried to dump their S0C4's and S0C7's on us. My rule of thumb is that only when they can present a dump that us SYSPROGS can't explain while we take any responsibility. I have kick out several programmers who were to lazy to debug. Ed On Nov 25, 2013, at 12:28 PM, Farley, Peter x23353 wrote: I mean the second, using the COBOL SORT verb to invoke the SORT from within the program. And I tend to agree with you. Just looking for reasons other than the ones I have thought of to refute the claim that was made. Peter -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's?
John, SyncSort has for many years not used any regular access methods in the normal course of processing SORTIN or SORTOUT. There are exceptions to this such as compressed files where we have to use BSAM, but for the most part, we do not use traditional access methods. As to the original subject matter, it is impossible to make a single general statement about what way is the best way to design a process. If you are using a COBOL program or exit to transform data or select a subset of records, in general it is faster both in elapsed time and CPU time to use the many features of a sort (INCLUDE/OMIT/INREC/OUTREC/OUTFILE) rather than an exit. As with everything in computing, your mileage may vary. Chris Blaicher Principal Software Engineer, Software Development Syncsort Incorporated 50 Tice Boulevard, Woodcliff Lake, NJ 07677 P: 201-930-8260 | M: 512-627-3803 E: cblaic...@syncsort.com -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of John Gilmore Sent: Monday, November 25, 2013 11:49 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Has anyone measured CPU savings using external SORT's vs internal (COBOL) SORT's? Note that the highly efficient i/o operations of SYNCSORT and DFSORT are their internal ones. They must and do use access methods to read sortin and write sortout. They do of course use these access methods more efficiently than many/most COBOL programs, but the big i/o savings are elsewhere. John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN ATTENTION: - The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
