Re: TIOT filling up: too many dynamic concatenations
David: I am just now digging through 2000+ IBM-MAIN messages, so this is a bit old by now. It has been a very long time since I was digging through dynamic allocation and concatenations as part of a JES2 user mod. What I remember was that my code had a loop where I read a dataset name, allocated it to a system assigned DDNAME, and then added it to the current concatenation. The problem was that the TIOT filled up very quickly because it was full of holes of ever increasing size. What was happening was the new concatenation was larger than any of the available holes in the TIOT, so it got placed at the end of the used space, and the TIOT ended up full of free space blocks of size 2x, 3x, 4x, 5x, etc, followed by the real proclib allocation space. It seemed that IBM was not going back and consolidating the free space blocks caused by the earlier concatenations. Assuming that this is still the case, one thing that you could try doing would be to allocate all of the datasets and then do a single massive concatenation, rather than doing all of the intermediate ones. I hope this helps. -jack - Original Message - From: David Eisenberg [EMAIL PROTECTED] Newsgroups: bit.listserv.ibm-main To: IBM-MAIN@BAMA.UA.EDU Sent: Wednesday, April 09, 2008 2:35 PM Subject: Re: TIOT filling up: too many dynamic concatenations Ed, Field S99ERROR (after the dynamic concatenation request to DYNALLOC) is coming back with a value of X'0238': Space unavailable in task input output table (TIOT). The manual says that the application should Reduce the total number of allocated DDs and devices. Deallocate data sets that are not needed simultaneously. The only way I've found to reduce the total number of allocated DDs is to reduce the number of files in the folder. I can't deallocate any of the component allocations once they've been concatenated. I did use IDF to step through and watch the TIOT grow. With each new concatenation call, there is an entry which keeps growing in size until the TIOT fills up. I think that what I need to know is if I'm missing some approach/technique that can get around what is probably a legitimate limitation. David -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
On Tue, 15 Apr 2008 21:45:34 -0300, Shmuel Metz (Seymour J.) wrote: But this provokes an interesting question: If (as I believe) HFS data sets are dynamically allocated in the OMVS address space (that's what's cited when I try to delete one), isn't OMVS at extreme hazard of impacting DVC etc. limits? No. Don't confuse an *FS with the Unix files that it contains. OMVS and the colony address spaces allocate entire file systems, not individual Unix files. Ah! colony address spaces; plural. Will OMVS create additional colony address spaces as hundreds of filesystems are mounted, as needed to avoid system limits on TIOT size, etc.? -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
Paul Gilmartin wrote: Ah! colony address spaces; plural. Will OMVS create additional colony address spaces as hundreds of filesystems are mounted, as needed to avoid system limits on TIOT size, etc.? This should not be an issue. Authorized programs are not constrained by traditional limits on TIOT size and 24-bit virtual storage. They can use non-contiguous 31-bit XTIOT entries (S99TIOEX), 31-bit DSAB entries (S99DSABA), and specify that UCB addresses not be captured (S99ACUCB). Such programs can support literally hundreds of thousands of simultaneous allocations. -- Edward E Jaffe Phoenix Software International, Inc 5200 W Century Blvd, Suite 800 Los Angeles, CA 90045 310-338-0400 x318 [EMAIL PROTECTED] http://www.phoenixsoftware.com/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
In a message dated 4/16/2008 12:28:54 P.M. Central Daylight Time, [EMAIL PROTECTED] writes: Authorized programs are not constrained by traditional limits on TIOT size and 24-bit virtual storage. ... Such programs can support literally hundreds of thousands of simultaneous allocations. They can also do I/O without any allocation, TIOT entry, DD statement, enqueue, open, etc. All they need is a UCB address, some CCWs somewhere, and about 150 bytes of ECSA. Bill Fairchild Rocket Software **It's Tax Time! Get tips, forms and advice on AOL Money Finance. (http://money.aol.com/tax?NCID=aolcmp0030002850) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
(IBM Mainframe Discussion List) wrote: They can also do I/O without any allocation, TIOT entry, DD statement, enqueue, open, etc. All they need is a UCB address, some CCWs somewhere, and about 150 bytes of ECSA. Of course ... using the low-level STARTIO interface. But, dynamic data set allocation serves a useful purpose in MVS. And, OMVS uses it. -- Edward E Jaffe Phoenix Software International, Inc 5200 W Century Blvd, Suite 800 Los Angeles, CA 90045 310-338-0400 x318 [EMAIL PROTECTED] http://www.phoenixsoftware.com/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
On Wed, 16 Apr 2008 10:28:00 -0700, Edward Jaffe [EMAIL PROTECTED] wrote: Paul Gilmartin wrote: Ah! colony address spaces; plural. Will OMVS create additional colony address spaces as hundreds of filesystems are mounted, as needed to avoid system limits on TIOT size, etc.? This should not be an issue. Authorized programs are not constrained by traditional limits on TIOT size and 24-bit virtual storage. They can use non-contiguous 31-bit XTIOT entries (S99TIOEX), 31-bit DSAB entries (S99DSABA), and specify that UCB addresses not be captured (S99ACUCB). Such programs can support literally hundreds of thousands of simultaneous allocations. I hinted at this a week ago: http://bama.ua.edu/cgi-bin/wa?A2=ind0804L=ibm-mainD=1amp;O=DT=0P=91054 but was waiting for someone with more expertise in this area to provide more information. Thanks, Mark -- Mark Zelden Sr. Software and Systems Architect - z/OS Team Lead Zurich North America / Farmers Insurance Group - ZFUS G-ITO mailto:[EMAIL PROTECTED] z/OS Systems Programming expert at http://expertanswercenter.techtarget.com/ Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
In [EMAIL PROTECTED], on 04/09/2008 at 10:03 PM, McKown, John [EMAIL PROTECTED] said: And doing QSAM I/O does avoid some problems with the LF or CRLF at the end of the logical records. Or NL if it's EBCDIC. -- Shmuel (Seymour J.) Metz, SysProg and JOAT ISO position; see http://patriot.net/~shmuel/resume/brief.html We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
In [EMAIL PROTECTED], on 04/09/2008 at 05:52 PM, Paul Gilmartin [EMAIL PROTECTED] said: But this provokes an interesting question: If (as I believe) HFS data sets are dynamically allocated in the OMVS address space (that's what's cited when I try to delete one), isn't OMVS at extreme hazard of impacting DVC etc. limits? No. Don't confuse an *FS with the Unix files that it contains. OMVS and the colony address spaces allocate entire file systems, not individual Unix files. And if the NFS is served from a RAID, does each volume in that server's RAID count separately? If you're running NFS on z/OS, doesn't it use Unix services to do its I/O for Unix files? Barring some really strange changes, it wouldn't have any allocations except for the MVS classic data sets that it is serving. For those, each extra volume would require an additional UCB pointer in the TIOT, assuming parallel mount. As for data sets served by an external NFS server, I wouldn't expect allocation to be involved, so there shouldn't be any need for extra TIOT space. -- Shmuel (Seymour J.) Metz, SysProg and JOAT ISO position; see http://patriot.net/~shmuel/resume/brief.html We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
Why do you concatenate them that way? Why not just find/open/process each independently? Or if you prefer, find all of them, then in a loop (allocate, open, process, deallocate) each individually? Yes, it is a bit more difficult to program, but it is infinitely scalable. If this hasn't changed lately, you'd also need to FREEBUF the QSAM buffers after each CLOSE or that might be the cause of the next out of storage condition. QSAM buffers are not freed by the system upon CLOSE. -- Peter Hunkeler Credit Suisse -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
Remember, z/OS emulates UNIX -- a CAT/READ/de-allocate is still just as expensive. It may quack like a duck; walk like a duck; it is an emulated duck. There is no such thing as allocation and deallocation in UNIX, incluiding z/OS UNIX. The z/OS UNIX kernel, more specifically, the physical file system code allocates and opens file system data sets at mount time. Thereafter it fullfills all the I/O requests from UNIX processes. I don't see an emulation here. I/O is expensive, no matter what kind of file system lies behind it. I haven't done any research on this but I bet in terms of resource usage doing a UNIX read is not to much different to doing an MVS read. -- Peter Hunkeler Credit Suisse -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
-Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of Hunkeler Peter (KIUK 3) Sent: Thursday, April 10, 2008 1:14 AM To: IBM-MAIN@BAMA.UA.EDU Subject: Re: TIOT filling up: too many dynamic concatenations Why do you concatenate them that way? Why not just find/open/process each independently? Or if you prefer, find all of them, then in a loop (allocate, open, process, deallocate) each individually? Yes, it is a bit more difficult to program, but it is infinitely scalable. If this hasn't changed lately, you'd also need to FREEBUF the QSAM buffers after each CLOSE or that might be the cause of the next out of storage condition. QSAM buffers are not freed by the system upon CLOSE. -- Peter Hunkeler Good point! I found that out the hard way when I did multiple OPEN/CLOSE on a single DCB. Ran out of storage. This was back in the MVS/XA days. It is also another reason to bite the bullet and rewrite to use native UNIX I/O. No such problems. -- John McKown Senior Systems Programmer HealthMarkets Keeping the Promise of Affordable Coverage Administrative Services Group Information Technology The information contained in this e-mail message may be privileged and/or confidential. It is for intended addressee(s) only. If you are not the intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication is strictly prohibited and could, in certain circumstances, be a criminal offense. If you have received this e-mail in error, please notify the sender by reply and delete this message without copying or disclosing it. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
I did use IDF to step through and watch the TIOT grow. With each new concatenation call, there is an entry which keeps growing in size until the TIOT fills up. My apologies if this has already been answered but I haven't seen it. This behavior is counter intuitive; the TIOT is filling up 3:1 compared to what one would expect. When you examine the TIOT, I presume that the entry that keeps growing is the one for MYFILE. Does it grow by a single entry each time, or is there more than a single entry added? If so, what is being added over and above expectation? If the MYFILE entry seems to be growing appropriately, what else is being added to the TIOT? Are allocations used in the concatention remaining as entries? This should be easy to test. Get a snapshot before and after the concatenation and analyze exactly what has changed. It could be that entries are not being reused by the concatenation but just used for the information that they contain. Date: Wed, 9 Apr 2008 13:35:28 -0500 From: [EMAIL PROTECTED] Subject: Re: TIOT filling up: too many dynamic concatenations To: IBM-MAIN@BAMA.UA.EDU Ed, Field S99ERROR (after the dynamic concatenation request to DYNALLOC) is coming back with a value of X'0238': Space unavailable in task input output table (TIOT). The manual says that the application should Reduce the total number of allocated DDs and devices. Deallocate data sets that are not needed simultaneously. The only way I've found to reduce the total number of allocated DDs is to reduce the number of files in the folder. I can't deallocate any of the component allocations once they've been concatenated. I did use IDF to step through and watch the TIOT grow. With each new concatenation call, there is an entry which keeps growing in size until the TIOT fills up. I think that what I need to know is if I'm missing some approach/technique that can get around what is probably a legitimate limitation. David _ Going green? See the top 12 foods to eat organic. http://green.msn.com/galleries/photos/photos.aspx?gid=164ocid=T003MSN51N1653A -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
Hunkeler Peter (KIUK 3) wrote: Why do you concatenate them that way? Why not just find/open/process each independently? Or if you prefer, find all of them, then in a loop (allocate, open, process, deallocate) each individually? Yes, it is a bit more difficult to program, but it is infinitely scalable. If this hasn't changed lately, you'd also need to FREEBUF the QSAM buffers after each CLOSE or that might be the cause of the next out of storage condition. QSAM buffers are not freed by the system upon CLOSE. This behavior changed with DFSMS/MVS 1.1 circa MVS/ESA 4.3. If you tell QSAM to use 31-bit buffers (who would choose otherwise?), it will free them automatically at CLOSE time. -- Edward E Jaffe Phoenix Software International, Inc 5200 W Century Blvd, Suite 800 Los Angeles, CA 90045 310-338-0400 x318 [EMAIL PROTECTED] http://www.phoenixsoftware.com/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
My apologies if this has already been answered but I haven't seen it. No apologies necessary! The entry that was filling up was actually not MYFILE; that entry remained at 20 bytes. The ever-growing entry seemed to contain all of the history of the concatenated DDs. With each new allocation and concatenation, it grew larger. After reading all of these very informative replies, I think that the best course of action for me will be to modify the application to process one file at a time. It may not be perfect, but it will work, and it will be scalable. Thanks so much! David -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
David, Could you clarify whether you perform the concatenation just once per run, or am I understanding that you perform multiple concatenations during each run? -- Tom Schmidt On Thu, 10 Apr 2008 15:13:58 -0500, David Eisenberg wrote: The entry that was filling up was actually not MYFILE; that entry remained at 20 bytes. The ever-growing entry seemed to contain all of the history of the concatenated DDs. With each new allocation and concatenation, it grew larger. After reading all of these very informative replies, I think that the best course of action for me will be to modify the application to process one file at a time. It may not be perfect, but it will work, and it will be scalable. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
David, I would have to agree with the posters that felt that the best way to handle this would be to use the z/OS UNIX services to read the files. While this does mean that you will need 2 code paths, one for UNIX files and a second one for classic datasets, the advantage of this approach is that you will not need to issue the SVC 99 to build the TIOT entry. Dynamic allocation isn't needed for the UNIX services, since z/OS I/O structures like JFCB's, DEB's, etc aren't required. Wayne Driscoll Product Developer NOTE: All opinions are strictly my own. -Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of David Eisenberg Sent: Thursday, April 10, 2008 3:14 PM To: IBM-MAIN@BAMA.UA.EDU Subject: Re: TIOT filling up: too many dynamic concatenations My apologies if this has already been answered but I haven't seen it. No apologies necessary! The entry that was filling up was actually not MYFILE; that entry remained at 20 bytes. The ever-growing entry seemed to contain all of the history of the concatenated DDs. With each new allocation and concatenation, it grew larger. After reading all of these very informative replies, I think that the best course of action for me will be to modify the application to process one file at a time. It may not be perfect, but it will work, and it will be scalable. Thanks so much! David -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
On Thu, 10 Apr 2008 15:34:29 -0500, Wayne Driscoll wrote: David, I would have to agree with the posters that felt that the best way to handle this would be to use the z/OS UNIX services to read the files. While this does mean that you will need 2 code paths, one for UNIX files and a second one for classic datasets, the advantage of this approach is that you will not need to issue the SVC 99 to build the TIOT entry. Dynamic allocation isn't needed for the UNIX services, since z/OS I/O structures like JFCB's, DEB's, etc aren't required. The options are certainly not mutually exclusive. He can process the files one at a time _and_ use UNIX services to perform the processing. And if the processing is to read and write buffers into a pipe ALLOCATEd to his prior SYSIN/SYSUT1, the disruption to the existing code is minimized. -Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of David Eisenberg After reading all of these very informative replies, I think that the best course of action for me will be to modify the application to process one file at a time. It may not be perfect, but it will work, and it will be scalable. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
TIOT filling up: too many dynamic concatenations
Anyone, I have a mainframe assembler application which is invoking Unix system services to get the names of all of the files in an NFS-mounted folder. The application dynamically allocates and logically concatenates these files into one giant dataset, then uses QSAM macros to read it. The DYNALLOC calls work this way: first, I dynamically allocate the first file in the folder with DDNAME MYFILE. Then the program enters a loop, performing these steps for each remaining file in the folder: 1) Dynamically allocate the file, asking the system to provide the DDNAME (I observe that these are getting the ddnames SYS1, SYS2, etc). 2) Dynamically concatenate MYFILE with the SYSx dataset just allocated (with the permanently allocated attribute on). This works beautifully; when I exit the loop, I can OPEN and GET all the records successfully from MYFILE. The problem is that I have reached a practical limit of approximately 540 files in the folder, because when I reach that point, I get a dynamic concatenation ABEND due to the TIOT filling up. I am told that our TIOT size is the default of 32K, which would allow for a maximum of 1,635 DDs in a job step. It would seem, however, that something in my allocation/concatenation loop is preventing me from reaching that number of files. There are only a handful of other DDs allocated to the step (e.g., STEPLIB, etc). If I were able to handle up to 750 (or perhaps 1,000) files at a time, it would be of immense help. At the moment, our only option seems to be to split up the files into multiple folders of 500 files each. Do I have any other options? Thanks so much. David -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
David Eisenberg wrote: The problem is that I have reached a practical limit of approximately 540 files in the folder, because when I reach that point, I get a dynamic concatenation ABEND due to the TIOT filling up. I am told that our TIOT size is the default of 32K, which would allow for a maximum of 1,635 DDs in a job step. It would seem, however, that something in my allocation/concatenation loop is preventing me from reaching that number of files. There are only a handful of other DDs allocated to the step (e.g., STEPLIB, etc). Which abend code? Is there no dump? A dump will show you what's in the TIOT. -- Edward E Jaffe Phoenix Software International, Inc 5200 W Century Blvd, Suite 800 Los Angeles, CA 90045 310-338-0400 x318 [EMAIL PROTECTED] http://www.phoenixsoftware.com/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
-Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of David Eisenberg Sent: Wednesday, April 09, 2008 1:17 PM To: IBM-MAIN@BAMA.UA.EDU Subject: TIOT filling up: too many dynamic concatenations Anyone, I have a mainframe assembler application which is invoking Unix system services to get the names of all of the files in an NFS-mounted folder. The application dynamically allocates and logically concatenates these files into one giant dataset, then uses QSAM macros to read it. [snip] If I were able to handle up to 750 (or perhaps 1,000) files at a time, it would be of immense help. At the moment, our only option seems to be to split up the files into multiple folders of 500 files each. Do I have any other options? Thanks so much. David Why do you concatenate them that way? Why not just find/open/process each independently? Or if you prefer, find all of them, then in a loop (allocate, open, process, deallocate) each individually? Yes, it is a bit more difficult to program, but it is infinitely scalable. -- John McKown Senior Systems Programmer HealthMarkets Keeping the Promise of Affordable Coverage Administrative Services Group Information Technology The information contained in this e-mail message may be privileged and/or confidential. It is for intended addressee(s) only. If you are not the intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication is strictly prohibited and could, in certain circumstances, be a criminal offense. If you have received this e-mail in error, please notify the sender by reply and delete this message without copying or disclosing it. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
Ed, Field S99ERROR (after the dynamic concatenation request to DYNALLOC) is coming back with a value of X'0238': Space unavailable in task input output table (TIOT). The manual says that the application should Reduce the total number of allocated DDs and devices. Deallocate data sets that are not needed simultaneously. The only way I've found to reduce the total number of allocated DDs is to reduce the number of files in the folder. I can't deallocate any of the component allocations once they've been concatenated. I did use IDF to step through and watch the TIOT grow. With each new concatenation call, there is an entry which keeps growing in size until the TIOT fills up. I think that what I need to know is if I'm missing some approach/technique that can get around what is probably a legitimate limitation. David -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
Yes, it is a bit more difficult to program, but it is infinitely scalable. Of course, you are correct. I took this approach because I have inherited a pre-existing application that used to read a single mainframe dataset. It's only recently that the capability to read multiple files via NFS was made necessary. My approach worked very nicely until very recently. If I have to modify the application to work the way you've suggested, then of course, I will. David -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
-Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of David Eisenberg Sent: Wednesday, April 09, 2008 1:17 PM To: IBM-MAIN@BAMA.UA.EDU Subject: TIOT filling up: too many dynamic concatenations Anyone, I have a mainframe assembler application which is invoking Unix system services to get the names of all of the files in an NFS-mounted folder. The application dynamically allocates and logically concatenates these files into one giant dataset, then uses QSAM macros to read it. The DYNALLOC calls work this way: first, I dynamically allocate the first file in the folder with DDNAME MYFILE. Then the program enters a loop, performing these steps for each remaining file in the folder: 1) Dynamically allocate the file, asking the system to provide the DDNAME (I observe that these are getting the ddnames SYS1, SYS2, etc). 2) Dynamically concatenate MYFILE with the SYSx dataset just allocated (with the permanently allocated attribute on). This works beautifully; when I exit the loop, I can OPEN and GET all the records successfully from MYFILE. The problem is that I have reached a practical limit of approximately 540 files in the folder, because when I reach that point, I get a dynamic concatenation ABEND due to the TIOT filling up. I am told that our TIOT size is the default of 32K, which would allow for a maximum of 1,635 DDs in a job step. It would seem, however, that something in my allocation/concatenation loop is preventing me from reaching that number of files. There are only a handful of other DDs allocated to the step (e.g., STEPLIB, etc). If I were able to handle up to 750 (or perhaps 1,000) files at a time, it would be of immense help. At the moment, our only option seems to be to split up the files into multiple folders of 500 files each. Do I have any other options? Thanks so much. snip Try putting DYNAMNBR=1024 on your EXEC card. Regards, Steve Thompson -- All opinions expressed by me are my own and may not necessarily reflect those of my employer. -- -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
Hi Ed: Maybe not the best solution but you can increase the size of the TIOT in the ALLOCxx parmlib member in the z/OS MVS Initialization and Tuning Reference _http://publibz.boulder.ibm.com/epubs/pdf/iea2e261.pdf_ (http://publibz.boulder.ibm.com/epubs/pdf/iea2e261.pdf) page 73. The maximum size is 64K. Regards, Gene **Planning your summer road trip? Check out AOL Travel Guides. (http://travel.aol.com/travel-guide/united-states?ncid=aoltrv000316) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
Try putting DYNAMNBR=1024 on your EXEC card. Just tried it; no good. Same result as before. Thanks, though. David -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
-Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of David Eisenberg Sent: Wednesday, April 09, 2008 1:42 PM To: IBM-MAIN@BAMA.UA.EDU Subject: Re: TIOT filling up: too many dynamic concatenations Yes, it is a bit more difficult to program, but it is infinitely scalable. Of course, you are correct. I took this approach because I have inherited a pre-existing application that used to read a single mainframe dataset. It's only recently that the capability to read multiple files via NFS was made necessary. My approach worked very nicely until very recently. If I have to modify the application to work the way you've suggested, then of course, I will. David Ah. That makes sense. I hate changing code. Hum, you could use the UNIX cat program to catenate all the files together into a single file somewhere, then read that. In a shell script, I'd do something like: #!/bin/sh cd /nfs/directory/to/scan cat * ~/all.files process ~/all.files You didn't say what language you are writing in. But you can dynamically invoke the /bin/cat program from almost anything via the BPXWUNIX subroutine. -- John McKown Senior Systems Programmer HealthMarkets Keeping the Promise of Affordable Coverage Administrative Services Group Information Technology The information contained in this e-mail message may be privileged and/or confidential. It is for intended addressee(s) only. If you are not the intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication is strictly prohibited and could, in certain circumstances, be a criminal offense. If you have received this e-mail in error, please notify the sender by reply and delete this message without copying or disclosing it. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
You didn't say what language you are writing in. Mainframe assembler. Your suggestion would work, but then I would have to get into an argument with the network guys when I tell them that I need twice as much disk just so that I can do a physical concatenation of all of the files. I don't think I have a leg to stand on, given that there is a programmatic, scalable solution (i.e., processing each file on its own). I think that I might have to modify the appllication... David -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
Try putting DYNAMNBR=1024 on your EXEC card. Just tried it; no good. Same result as before. That's because your TIOT size is still 32K. As somebody stated, the short term solution is to bump it up to 64K. The long term would be (imo) re-write the programme to open one file at a time. - Too busy driving to stop for gas! -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
-Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of David Eisenberg Sent: Wednesday, April 09, 2008 2:19 PM To: IBM-MAIN@BAMA.UA.EDU Subject: Re: TIOT filling up: too many dynamic concatenations You didn't say what language you are writing in. Mainframe assembler. Your suggestion would work, but then I would have to get into an argument with the network guys when I tell them that I need twice as much disk just so that I can do a physical concatenation of all of the files. I don't think I have a leg to stand on, given that there is a programmatic, scalable solution (i.e., processing each file on its own). I think that I might have to modify the appllication... David I think that is the best idea. Hopefully it won't be too difficult. I can think of some other methods, but they are very UNIXy and weird. Such as creating a named pipe, fork()'ing then exec()'ing /bin/cat in the child to write to the pipe which is then read with QSAM in the parent. This would eliminate the second copy of the data. -- John McKown Senior Systems Programmer HealthMarkets Keeping the Promise of Affordable Coverage Administrative Services Group Information Technology The information contained in this e-mail message may be privileged and/or confidential. It is for intended addressee(s) only. If you are not the intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication is strictly prohibited and could, in certain circumstances, be a criminal offense. If you have received this e-mail in error, please notify the sender by reply and delete this message without copying or disclosing it. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
On Wed, 9 Apr 2008 13:16:58 -0500, David Eisenberg wrote: I have a mainframe assembler application which is invoking Unix system services to get the names of all of the files in an NFS-mounted folder. The application dynamically allocates and logically concatenates these files into one giant dataset, then uses QSAM macros to read it. ,,, Do I have any other options? Thanks so much. Create a named pipe (FIFO). Allocate that FIFO as input to your existing application. Run your directory scanner concurrently (fork(), spawn(), or ATTACH) with your existing program. As it encounters each file in the folder, it should copy it into the FIFO (Hmmm. Need to keep a descriptor of the FIFO open, not open it each time.) Only one allocation. Actually, you might call cat(1) with as many file arguments as you've mentioned, and pipe its output into your assembler program. Plan C: Take your question to MVS-OE. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
On Wed, 9 Apr 2008 13:16:58 -0500, David Eisenberg wrote: I have a mainframe assembler application which is invoking Unix system services to get the names of all of the files in an NFS-mounted folder. The application dynamically allocates and logically concatenates these files into one giant dataset, then uses QSAM macros to read it. ... Do I have any other options? Thanks so much. Yes. Create a named pipe (FIFO) and allocate it as input to your assembler program. Run your folder scanner concurrently (fork(), spawn(), or ATTACH) with the assembler program. have it copy each file in the folder into the FIFO (Hmmm. Must open a descriptor once and write to that, not open multiple times). Only one allocation, not hundreds. Actually, for the number of files you mention, you should be able to supply the list as an argument to cat (cat *) and pipe the output to your assembler program. Plan C: Take the question to MVS-OE. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
On Wed, 9 Apr 2008 14:54:53 -0500, McKown, John wrote: I think that is the best idea. Hopefully it won't be too difficult. I can think of some other methods, but they are very UNIXy and weird. Such as creating a named pipe, fork()'ing then exec()'ing /bin/cat in the child to write to the pipe which is then read with QSAM in the parent. This would eliminate the second copy of the data. Ahem. What's weird about that? Sorry for the double post. I knew I fumbled on the Send button, and when I didn't see the message promptly, I retyped it. Still not seeing it, I sent again. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
The dataclass Dynamic Volume Count (DVC) is (DSN DD + DVC amount) for each dataset allocated. If the DVC is 20, the that would equate to 21 DDs in TIOT times however many datasets allocated. Terry Traylor -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
-Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of Traylor, Terry Sent: Wednesday, April 09, 2008 4:11 PM To: IBM-MAIN@BAMA.UA.EDU Subject: Re: TIOT filling up: too many dynamic concatenations The dataclass Dynamic Volume Count (DVC) is (DSN DD + DVC amount) for each dataset allocated. If the DVC is 20, the that would equate to 21 DDs in TIOT times however many datasets allocated. Terry Traylor Strange that DVC would apply to a UNIX file, but it doesn't surprise me. -- John McKown Senior Systems Programmer HealthMarkets Keeping the Promise of Affordable Coverage Administrative Services Group Information Technology The information contained in this e-mail message may be privileged and/or confidential. It is for intended addressee(s) only. If you are not the intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication is strictly prohibited and could, in certain circumstances, be a criminal offense. If you have received this e-mail in error, please notify the sender by reply and delete this message without copying or disclosing it. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
On Wed, 9 Apr 2008 16:15:26 -0500, McKown, John wrote: -Original Message- [mailto:[EMAIL PROTECTED] On Behalf Of Traylor, Terry Sent: Wednesday, April 09, 2008 4:11 PM The dataclass Dynamic Volume Count (DVC) is (DSN DD + DVC amount) for each dataset allocated. If the DVC is 20, the that would equate to 21 DDs in TIOT times however many datasets allocated. Strange that DVC would apply to a UNIX file, but it doesn't surprise me. But wouldn't that DVC apply in the OMVS address space (or, in the OP's case, the MVSNFSC address space) and not impact the problem program? And if the NFS is served from a RAID, does each volume in that server's RAID count separately? But I'll defer to your undoubtedly superior familiarity with z/OS internals and not be surprised myself. Nor infer that Terry T. was merely flaunting his knowledge of z/OS arcana regardless of irrelevance to the OP's concern. But this provokes an interesting question: If (as I believe) HFS data sets are dynamically allocated in the OMVS address space (that's what's cited when I try to delete one), isn't OMVS at extreme hazard of impacting DVC etc. limits? -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
IBM Mainframe Discussion List IBM-MAIN@BAMA.UA.EDU wrote on 04/09/2008 02:16:58 PM: I have a mainframe assembler application which is invoking Unix system services to get the names of all of the files in an NFS-mounted folder. The application dynamically allocates and logically concatenates these files into one giant dataset, then uses QSAM macros to read it. The DYNALLOC calls work this way: first, I dynamically allocate the first file in the folder with DDNAME MYFILE. Then the program enters a loop, performing these steps for each remaining file in the folder: 1) Dynamically allocate the file, asking the system to provide the DDNAME (I observe that these are getting the ddnames SYS1, SYS2, etc). 2) Dynamically concatenate MYFILE with the SYSx dataset just allocated (with the permanently allocated attribute on). This works beautifully; when I exit the loop, I can OPEN and GET all the records successfully from MYFILE. The problem is that I have reached a practical limit of approximately 540 files in the folder, because when I reach that point, I get a dynamic concatenation ABEND due to the TIOT filling up. I am told that our TIOT size is the default of 32K, which would allow for a maximum of 1,635 DDs in a job step. It would seem, however, that something in my allocation/concatenation loop is preventing me from reaching that number of files. There are only a handful of other DDs allocated to the step (e.g., STEPLIB, etc). If I were able to handle up to 750 (or perhaps 1,000) files at a time, it would be of immense help. At the moment, our only option seems to be to split up the files into multiple folders of 500 files each. Do I have any other options? Thanks so much. Is it necessary to concatenate all of the files? Could you restructure the program so that it allocates, opens, reads, closes, and unallocates each file sequentially? If the purpose is simply to read and process all of the records in all of the files, it would seem that this would accomplish the same objective, and the records would be processed in the same order. Jim Mulder z/OS System Test IBM Corp. Poughkeepsie, NY -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
On Wed, 9 Apr 2008 17:52:47 -0500, Paul Gilmartin [EMAIL PROTECTED] wrote: But this provokes an interesting question: If (as I believe) HFS data sets are dynamically allocated in the OMVS address space (that's what's cited when I try to delete one), isn't OMVS at extreme hazard of impacting DVC etc. limits? I doubt it. This is out of my area of expertise but I would think it uses the XTIOT since it is authorized and uses DYNALLOC (the S99TIOEX flag must be set in the the S99RB). This is what DB2 does. Mark -- Mark Zelden Sr. Software and Systems Architect - z/OS Team Lead Zurich North America / Farmers Insurance Group - ZFUS G-ITO mailto:[EMAIL PROTECTED] z/OS Systems Programming expert at http://expertanswercenter.techtarget.com/ Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
On Wed, 9 Apr 2008 19:13:19 -0400, Jim Mulder wrote: Is it necessary to concatenate all of the files? Could you restructure the program so that it allocates, opens, reads, closes, and unallocates each file sequentially? If the purpose is simply to read and process all of the records in all of the files, it would seem that this would accomplish the same objective, and the records would be processed in the same order. How would the performance of a scheme that allocates, opens, reads, closes, and unallocates each file sequentially (or concatenated, which still does most of the same work) compare with allocating a single POSIX pipe and using BPX1* system calls to copy the various NFS files into it? My gut feeling is that Unix System services is the big winner here. The NFS files must be opened, read, and closed either way; why add the overhead of allocation, unallocation, and z/OS OPEN and CLOSE? -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
How would the performance of a scheme that allocates, opens, reads, closes, and unallocates each file sequentially (or concatenated, which still does most of the same work) compare with allocating a single POSIX pipe and using BPX1* system calls to copy the various NFS files into it? My gut feeling is that Unix System services is the big winner here. The NFS files must be opened, read, and closed either way; why add the overhead of allocation, unallocation, and z/OS OPEN and CLOSE? Remember, z/OS emulates UNIX -- a CAT/READ/de-allocate is still just as expensive. It may quack like a duck; walk like a duck; it is an emulated duck. - Too busy driving to stop for gas! -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TIOT filling up: too many dynamic concatenations
-Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of Paul Gilmartin Sent: Wednesday, April 09, 2008 7:47 PM To: IBM-MAIN@BAMA.UA.EDU Subject: Re: TIOT filling up: too many dynamic concatenations On Wed, 9 Apr 2008 19:13:19 -0400, Jim Mulder wrote: Is it necessary to concatenate all of the files? Could you restructure the program so that it allocates, opens, reads, closes, and unallocates each file sequentially? If the purpose is simply to read and process all of the records in all of the files, it would seem that this would accomplish the same objective, and the records would be processed in the same order. How would the performance of a scheme that allocates, opens, reads, closes, and unallocates each file sequentially (or concatenated, which still does most of the same work) compare with allocating a single POSIX pipe and using BPX1* system calls to copy the various NFS files into it? My gut feeling is that Unix System services is the big winner here. The NFS files must be opened, read, and closed either way; why add the overhead of allocation, unallocation, and z/OS OPEN and CLOSE? -- gil Well, iffin it twere me, I'd replace the QSAM logic with directly reading the UNIX file using UNIX facilities (BPX1OPN / BPX1RED / BPX1CLO) . But I don't know how much work the OP wants / has time to do. Sometimes doing the minimal change is the best. Other times, it is better to simply bite the bullet and rewrite. And doing QSAM I/O does avoid some problems with the LF or CRLF at the end of the logical records. They would be presented with the BPX1RED, but not with QSAM. And QSAM, I think, will pad short records to the LRECL for RECFM=FB or emulate the RDW for RECFM=VB files. From my understanding, and I can easily be wrong, doing the I/O via the UNIX subroutines does not do any sort of DYNALLOC. I am fairly sure that all the I/O is somehow done by the OMVS task. Much like the I/O is done by the DB2 system and not the job doing the SELECT statements. Of course, this does lengthen the path length due to the need to talk to the OMVS system. I don't know how that is actually done. My uneducated guess would be with some sort of ss-PC routine. But OCO forbids me from actually knowing how anything works. Wouldn't want the unwashed masses to know too much, now would we? -- John McKown Senior Systems Programmer HealthMarkets Keeping the Promise of Affordable Coverage Administrative Services Group Information Technology The information contained in this e-mail message may be privileged and/or confidential. It is for intended addressee(s) only. If you are not the intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication is strictly prohibited and could, in certain circumstances, be a criminal offense. If you have received this e-mail in error, please notify the sender by reply and delete this message without copying or disclosing it. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
