Thanks Barry for the walk down memory lane. I had a few years working on SMF in my early IBM days. First I was the receiving programmer to enhance SMF for VS2 rel 1 in Poughkeepsie when it moved from Boulder to Pok. I did some minor SMF enhancements for VS2 rel 1.
Then later in the late 70's I also was team leader for this project to enhance SMF and will add to this text below a little. We listened to Barry M and Share as they helped us understand the customer needs. Actually we knew the sizing effort had to be carefully handled at IBM and knew how much people months we could recommend to spend on the project and if sized correctly the project may not be approved. We low balled the sizing. Once we got it approved we felt we could get the needed people added to finish the project and we did. I later took the hits for our poor sizing but we met the customer needs as the team felt the passion to make these enhancements for Share customers. I also presented for IBM at Share in San Francisco to a packed ballroom the announcement for MVS SE2 and described how the release met many of the Share requirements for SMF. "Estimated project by doubling the then-known size of SMF (Source Lines of Code) and used that (and IBM internal estimates) for the actual estimated man-hours. The final implementation took four times as much as the estimate, requiring 12 people 1.5 to 2 years for SE2 SMF Writer redesign. Because no one else in VS 2 cared much about SMF, the development was independent, which allowed many things to be considered and many went in and out of the final SMF enhancements." Joe Winterton Release Mgr - OMEGAMON IBM Systems Group - RTP BLD 501 - 2G04 919-543-2919 Cell -914-954-0483 - [email protected] From: Barry Merrill <[email protected]> To: [email protected] Date: 03/03/2017 07:01 PM Subject: Re: When did SMF come along? Sent by: IBM Mainframe Discussion List <[email protected]> Remainder of article from MXG Newsletter FIFTEEN, November 1989. The History of SMF A. It uses volume and data set information from the SMF data set to create and update records in the inventory data sets. There is one inventory for direct access resources and one for tape. B. From volume information in the inventories it produces a volume inventory report and SCRATCH volume reports. C. From data set information in the inventory it produces a variety of data set reports. The data set inventory records are selected and arranged according to such parameters as data set name, volume indentification, account number, expiration date and last usage date. D. It creates control cards for IEHPROGM to SCRATCH data sets which have expired. E. It creates control cards for the Resource Management Utility to REMOVE data set inventory records from the inventories for data sets which have been SCRATCHed or DELETEd." "F. It accepts control cards to remove volume and data set information from the inventories. G. It can compress the inventories. Each inventory is a partitioned data set (PDS). Each directory entry in the PDS reflects a particular volume. The member itself contains information about data sets on that volume. This was followed by twelve pages identifying each field in each inventory record and its source SMF record (5,14,15,17,18,19, or 20)." and then the JCL: "2.1.5.6 Job Control Language and Control Statements Used by the Utility The Resource Management Utility can be invoked by the following job control language //jobname JOB positional parameters and keywords //stepname EXEC PGM=utility name //SYSPRINT DD parameters describing SYSOUT device //SYSDATIN DD parameters describing direct access inventory PDS //SYSTAINV DD parameters describing tape inventory PDS //SYSMAN DD parameters describing the SMF data set //SYSREPRT DD parameters describing report device //SYSPUNCH DD parameters describing punch data set //SYSIN DD parameters describing control statement data set " "The control statement data set (described on the SYSIN DD statement) contains one or more of the following operations: A. UPDATE B. REPORT C. REMOVE D. COMPRESS" Then followed twelve pages detail the syntax and use of the many operands for each operation. The sixteen error messages produced by the Resource Managment Utility, IEH901E through IEH916I are then documented, and give the first clue as to the planned name for the Resource Management Utility program name of IEHMAN! "2.3.2.2 Resource Management Utility Requirements The utility program is designed to be in a planned overlay structure with a minimum of 15K required for executable code at one time. In addition, core storage is required for buffers. The number of bytes needed for buffers is device dependent. Maximum buffer sizes (in bytes) are shown in the table below: Device Type Maximum buffer size 2301 Drum 21,000 2302 Drum 5,400 2303 Drum 5,300 2311 Disk 4,000 2314 Disk 7,900 2321 Data Cell 2,400" And even a performance evaluation of the utility: "The direct access inventory has entries for ten volumes, each with nine extensions, giving a total of 100 members in the directory. Each member is 3500 bytes long and contains 15 data set inventory records, giving a total of 1500 data set entries. An UPDATE is estimated to take 9 seconds. A REPORT VOLID operation, which produces a list of direct access volume inventory information is estimated to take 1 second. A REPORT DS operation, which produces an unordered list of all data sets on the printer, is estimated to take 82 seconds. Further time would be required to produce a sorted list. A prototype of the utility was coded and tested in several different versions. MVTTRACE was used to obtain timings of the final version described below. SYS1.MAN data set resided on a 9-track 2400 tape drive, model 3, and was recorded at 800 bpi. The direct access inventory resided on a 2311 device with a member blocksize of 3200 bytes. The printer was a 1403 device with 120 print positions. The CPU was a model 50, the system MVT, Release 14." "The first UPDATE run required 12.2 minutes. This initialized the inventory data sets. A. The input stream from SYS1.MAN contained the following records arranged as if coming from an MVT environment: 60 job commencement (type 20) 60 job termination (type 5) 1500 old non-temporary (type 14) 1500 new non-temporary (type 15) 60 temporary (type 16) 60 SCRATCH (type 17) 20 RENAME (type 18) 80 direct access volume dismount (type 19) The resulting direct access inventory contained information about 1862 data sets, of which 39 had been SCRATCHed. There were 197 directory names in the directory, of which 148 were base members and 49 were extension members. The average number of data sets per volume was 12.6. The average length of data set inventory records was 208 bytes. B. The second UPDATE required 19.7 minutes." In a separate section of the Final Specification the error expectations of SMF were predicted: "6.4.2.1 APAR records for IEBPRTPCH, a utility program estimated to be of a similar size (13,000 bytes) were used to project expected errors and severity. 6.4.2.3 All Severity 1 errors should be resolved before integration. Only two OS utility programs have ever received Severity 1 APAR's after release. and the accompanying table showed After Customer Ship Expected: 6 months 18 months Severity 2, 3, 4 errors 7 15 Man hours to correct 40 40 Machine hours to correct 5 5" And then along came the Final "Final": --------------------------------------------- S/360-OS-1010-01-Pok Final Programming Functional Specifications IBM Operating System/360 Data Set Management and D.A. Space Accounting (Subset 2) Internally Dated June 25, 1969. --------------------------------------------- This document was much thinner than its predecessor, and states on its cover letter: "This revision of the referenced specification contains modifications in the following major areas. 1. The Resource Management Utility program description has been deleted. THUS DIED SMF Reporting. This final specification also eliminated the type 16 (temporary data set) by redesigning the type 14 and 15 records to what we now have. In addition, design specifications that have departed from the initial specifications were identified: "1.3.2 From Initial Specifications: These specifications depart from the Objectives and Initial Specifications in the following respects: a. No I/O device timing will be performed. c. The TCT of Subset 1 will be expanded to include information previously intended for a new control block (the IOCT). This will eliminate the need to modify the DEB." The final error table was now modified " After Customer Ship Expected: 6 months 18 months Severity 2, 3, 4 errors 5 (was 7) 20 (was 15) Man hours required 4 (was 40) 10 (was 40) Machine hours required 4 (was 5) 10 (was 5)" "Systems Management Utility - IEHMAN" The IEHMAN Planning Guide describes the same features that were specified in the SMF design for Subset 2, called the "Resource Management Utility"! Not only did the IEHMAN Planning Guide exist within IBM, but through an error, a small number of copies were actually shipped from Mechanicsburg, Pa., to some IBM customer sites! Once the error was discovered, IBM immediately recalled all copies! Ken Plambeck was the author of the IEHMAN package. The project had 10-12 people 12 hrs/day, but the product was killed when the manager could not get sponsors. Even though IEHMAN was never announced nor ever released, it showed that IBM designers, even in 1966-1967, knew that analysis tools would be needed to exploit the SMF data. ------------------------------------------------ In May, 1969, C28-6712-0, Planning for System Mangement Facilities (62 pages) described the version of SMF planned for Release 18. ------------------------------------------------ but the real announcement of availability was: ------------------------------------------------ IBM System 360/Operating System Release 18 Guide GC28-6718-1 July, 1969 ------------------------------------------------ "System Management Facilities SMF is a new feature of the operating system, selectable at system generation in conjunction with the MVT configuration. SMF may not be specified in conjunction with Model 65 Multiprocessing. SMF provides two basic functions: Collecting and recording system-, job-, and step-related data for each job processed by the system. Monitoring job processing via exits from the control program to installation-provided routines at specific points during the processing of each job." Data Collection: SMF writes 13 types of records to a data set resident on either tape or direct access. The records contain such information as: system configuration, job and job step identification, CPU wait time, CPU usage by each job and job step, and I/O device usage and main storage usage by each job step. The writing of SMF records may be supressed at IPL. An installation-provided routine can periodically analyze the SMF dataset to evaluate system efficiency, performance, and usage. Job Monitoring: SMF provides five exits within the control program to installation routines: From the reader/interpreter of the job just before each job control statement is interpreted. From the initiator/terminator when a job is selected for initiation. From the initiator/terminator when a step is selected for initiation. From the intiator/terminator when a step and/or job is terminated. From the timer second level interruption handler (SLIH) if a specified CPU or wait time limit for a job or step is exceeded." If no wait time limit is specified, the default value provided by IBM is 15 minutes; in previous releases the default value was 30 minutes. A new macro instruction (SMFWTM) may be used in exit routines to write records to the SMF data set. A test procedure (TESTEXIT) is provided in SYS1.SAMPLIB to facilitate routine testing. You must also provide analysis and report routines to process the SMF data set." ------------------------------------------------ IBM System/360 Operating System Consolidated Document Release 18 GY28-6681-2 Third Edition (October, 1969) ------------------------------------------------ "Updated Version of Release 18, designated as Release 18.6, may now be ordered from PID. Approximately 40 PTFs have been applied to the distribution libraries, correcting more than 80 APARs. (Release 18.0 text:) A second release of SMF will support MFT, M65 Multiprocessing and Remote Job Entry. Graphics Job Processing (originally planned for the second release of SMF) is supported in this initial release of SMF. Primary Control Program (PCP) and Conversational Remote Job Entry (CRJE) are not supported. Performance If SMF is chosen at System Generation, performance will be reduced. The performance reduction will be dependent upon the installation's use of the following: . SMF buffer size . Device used for the SMF data set . SMF data set allocation size . Number of jobs run per day . Execution time of the installation exits The performance reduction to the system when all System Management Facilities are functioning can be less than 5%. Storage Requirements A fixed amount of main storage is required when SMF is chosen at System Generation. In MVT a maximum of 1500 bytes is added to the main resident storage. Supervisor Queue Space is used for data collection tables, new job queue entries, and the user defined SMF buffer. The variable storage depends on the number of active jobs in the system and the SMF options chosen." ------------------------------------------------ IBM System/360 Operating System: Release 19 Guide GC28-6733-1 June 1970 ------------------------------------------------ Announced Subset 2 of SMF and support for MFT and M65 Multiprogramming and RJE with all twenty-one SMF records (0-15,17-20) with one new record, Type 13 - MFT Partitions and one new Exit (IEFUSO) for SYSOUT data set limit exceeded ------------------------------------------------ System Programming Guide Release 19 ------------------------------------------------ "Example of SYS1.MANX Space Requirements ID bytes 1 IPL per day 0 20 Devices online 8,19 2 Vary Onlines per Hour 9 2 Vary Offlines per Hour 11 1 Device Allocation per Hour 10 1 Scratch Dataset per 4 Hours 17 1 Rename Dataset per 4 Hours 18 24 Accumulated Wait Time 1 Total for these records 2,025 Job Processing 1 Start of Job 20 1 End of Job 5 1 Dismount 2 Volumes per job 19 3 Steps per job 4 Step Processing 3 DD statements per step 4 2 Input datasets per step 14 2 Output datasets per step 15 2 Output writers per step 6 Total for one job 6,303 Total for 48 jobs in 4 hours 302,688 SMF headers Record Descriptor Words 5,044 Block Descriptor Words 1,684 TOTAL SMF DATA FOR 4 HOURS: 311,411" ------------------------------------------------ IBM System/360 Operating System: Release 20 Guide GC28-6730-0 January 1971 ------------------------------------------------ Announced 20.0 with support for S/360 and new S/370 155 processor (S/370 165 in April) and indicates to expect 20.6 in 6-8 months. ------------------------------------------------ IBM System/360 Operating System System Management Facilities GC28-6712-4 (Fourth Edition, February 1971) ------------------------------------------------ Applied to Release 20.1 and identifies the eleven new SMF records created with Release 20 (now totaling 31 record types): Type 21 - ESV Record Type 30 - Start TSO Record Type 31 - TIOC Initialization Record Type 32 - Driver Record Type 33 - Driver Modify Record Type 34 - TSO Step Termination Type 35 - TSO Logoff Record Type 38 - Initial TSO Configuration Record Type 40 - Dynamic Allocation Record (not documented!) Type 41 - Modify TSO Record Type 42 - Stop TSO Record ------------------------------------------------ IBM System/360 Operating System: Release 21 Guide GC28-6730-2 March, 1972 (Release 21.0) ------------------------------------------------ Announced the REC parameter and minor change in format of SMF records, and indicated that 21.6 would be along in 4-6 months. ------------------------------------------------ IBM System/360 Operating System: Release 21 Guide GC28-6730-04 August, 1972 (Release 21.6) ------------------------------------------------ was announced, with no SMF enhancements. In October, 1972, I first used the Statistical Analysis System, SAS, to read SMF records from OS/360 Release 20.6 at State Farm Insurance. At that time, SAS cost $100! By early 1973, I reported our successful results to user groups (SAM, TESDATA) and ACM (SIGSIM, Chicago). John Chapman demanded that I present at SHARE. In March, 1974 (SHARE XLII, Houston), in the CME project, I presented SMF/SAS; CME scheduled an open session for the Chicago SHARE. That summer, IBM announced SGP, their Statistics Gathering Package, written by Bill Tetzloff. The open session at the August, 1974, SHARE meeting in Chicago began with Bill's SGP product description followed by State Farm Auto's presentation of their use of SAS and SMF data. Over 750 people (half of SHARE attendance) were present! While SGP was truly a valiant IBM effort to provide reporting from an extract of a fixed set of SMF fields, it was not easily tailored, was cast in concrete, and appeared inflexible. This audience then saw the SAS language for the first time, and saw SAS actually used for productive SMF analysis. At the end, one attendee pointedly asked IBM, "Now that you have seen SAS, is there any reason why we should still consider SGP?" This discovery that the SAS language was highly suited for analysis of SMF and RMF data led to many SHARE sessions that demonstrated that SMF data analysis really did save money and answered data center management's questions (response, capacity, service objectives, etc.). SAS was the tool that made SMF analysis practical, in 1974. The early releases of MVS became available: VS/2 Announced August 1972 ?/73? MVS 2.2 MF-1 Type 70-77,79 ?/75? MVS 3.0 5/76 MVS 3.7 8/76 SU7 Supervisor 2 8/76 SU20 RMF 9/76 SU11 TSO CMD. Package 9/76 SU13 TSO/VTAM 3/78 SU58 TSO/VTAM Level 2 3/78 SU50 MVS SE1 (for 3.7) 7/78 SU78 TSO Session Manager Rel. 2 3/79 SU65 MVS SE1.1 (for 3.7) 3/79 MVS 3.8 (SCP2) 8/79 SU74 MVS SE2 (for 3.8) Type 23, 30, 32, 90 1/85 NPDA V3 R2 (Type 37) 2/85 DFSORT R7 (Type 16) 3/85 NLDM R3 370 (Type 39) 7/86 DB2 (Type 100,101,102) By the mid 1970s, large TSO shops (several hundred concurrent logged on users) began noting degredation due to the BSAM SMF writer: - ENQ/DEQ was used by all SMF records, a very slow, serialized server for every logical record written. - TCLOSE to update VTOC after every block was written moved the disk drive arm - the drive shook like a washing machine! - WRITE VERIFY doubled the I/O time In addition, the 1975 SHARE-IBM meeting in New York was called because users were not migrating from MVT to MVS, and were blaming SMF accounting issues, technical issues about actual numbers in the records (eg., more absolute CPU time, the loss of "storage measurement" a la "K-Core Hours" from MVT, the inclusion of PCI counts into SMF EXCP buckets) caused, Ron Hankison, then IBM Representative to SHARE MVS Group, to become the local SMF expert within IBM. >From Ron Hankison 1989 interview: Accounting requirements from SHARE and GUIDE had built up, and nothing had changed since the original OS implementation (VS 2.2 and VS 1.6 had only added a few fields regarding paging and Virtual storage). TCLOSE was out of control, the constraint was apparent and there was a backlog of user requirements. Project goals were to: Fix the performance constraint Add functionality Restructure after 5-10 years experience Estimated project by doubling the then-known size of SMF (Source Lines of Code) and used that (and IBM internal estimates) for the actual estimated man-hours. The final implementation took four times as much as the estimate, requiring 12 people 1.5 to 2 years for SE2 SMF Writer redesign. Because no one else in VS 2 cared much about SMF, the development was independent, which allowed many things to be considered and many went in and out of the final SMF enhancements. Primary developers were: Barb Butler Bill McTiernan Steve Rosengarn Joe Winterton ------------------------------------------------ Programming Objectives and Initial Programming Functional Specifications MVS Accounting Facility August 1, 1977 ------------------------------------------------ "1.4 Summary of Specifications The rewrite of SMF will resolve the numerous, known problems with SMF. The performance of SMF will be much improved by the elimination of bottlenecks during the collection and writing of the data. Additional data will be available for recording that fills in several known exposures for resource utilization and system activities. Flexibility will be built in that allows the installation improved control over the recording and make tradeoffs between the integrity of the data collected and the performance impact. Complexity will be reduced by the capability of establishing a common file to contain all the pertinent information needed to manage the installation. Additional useability items will make this package very appealing to DP installations." 1.5 Summary of RAS Characteristics Due to the critical nature of the data being handled, minimizing the opssibility of loss of data will be stressed in the design of this package. The improvements described in this document will be covered by either an FRR or ESTAE routine to ensure that loss of data is held to a minimum in the event of a failure in the SMF component. Optional facilities will be provided to minimize loss of data due to system failures. Programming techniques known to have demonstrated improved quality will be used during implementation and test." 2. User Requirements Addressed Started Task Reporting Interval Reporting Performance/Overhead of Data Collection Recording Selectivity Proliferation of Data and Records Installation Tracking Completeness Accounting Direction Proliferation of Tools 7. Performance The overall reduction of SMF overhead and its impact on system thruput will be significant in those environments that have a high volume of data that is recorded to SMF. The performance improvement of the collection routine and its branch entry capability will provide a much improved interface for those components that should be recording to SMF but were concerned about the SMF bottlenecks. The path length for a Write to the SMF data set is approximately 60,000 instructions. The frequency of that event is installation dependent but probably averages about once every 13 logical records. The size and frequency of record receipts will be increasing rapidly as processor speeds increase. The revised path length by using the VSAM ICIP in SRB mode is approximately 2500 instructions, resulting in a reduction of 57,500 instructions per Write." Summary of the author's opinions: 1. SHARE was instrumental in the creation of SMF. 2. Users had a fair idea of what data was needed. 3. IBM designers in 1966 knew more than users of the range of data that we would ultimately need. 4. The iterative process between IBM designers and IBM users provided realistic validation of the design before implementation. 5. Designer's specifications and wants exceeded the funding and time for implementation. 6. The 1966-1969 IBM design and implementation process was better structured, managed, and documented than your company's most recent managed software project in 1989. 7. Based on this example of IBM design practices of twenty years ago, imagine the detail we would find in today's IBM design documents! 8. SMF made IBM pre-eminent in the business of real data processing by giving DP management actual measures of the resource usage and the service (response) for each user. DP could then "manage by objectives" and prove to the president the value and costs of the services DP provided the company. No other vendor of hardware and software has provided the accurate measurements that IBM has given its customers. 9. As good as it is, it still isn't perfect: MVS/ESA added three important CPU measures to the type 30 (task level) record, RCT - Region Control Task CPU SLIH - Second Level Interrupt Handler CPU HPT - Hiperspace CPU but we do not have these same CPU measures in the type 72 (performance group) record. SHARE REQUIREMENT, ANY ONE? Contributors: With sincere thanks for the dedicated developers designers and users of SMF data, I especially salute these contributors to this research: Bob Brockish, (retired) IBM Tom Bell, Rivendel Consultants Lynn Weissenreider, IBM Boulder Brian Currah, BDC Computer Services Saul Schiffman, IBM Japan Aron Eisenpress, CUNY Ron Hankison, IBM Kingston Mario Morino, Morino Associates Dick Armstrong, IBM Gaithersburg Jim Doyle, IBM Poughkeepsie Jack Higgins, IBM Publications especially for their treasure-trove of original IBM project documentation as well as their time in reminiscing on personal remembrances. Barry Merrill Merrilly yours, Herbert W. Barry Merrill, PhD President-Programmer Merrill Consultants MXG Software 10717 Cromwell Drive technical questions: [email protected] Dallas, TX 75229 http://www.mxg.com admin questions: [email protected] tel: 214 351 1966 fax: 214 350 3694 ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
