Re: DFSORT maximum input records
>> UTF-8 is a variable-length encoding. Is the length of a key field specified >> in characters or in octets? Check this link which explains how DFSORT process Unicode data. Search for (UTF-8/16/32) in the link below https://www.ibm.com/docs/en/zos/2.5.0?topic=descriptions-dfsort-data-formats >> I suppose there's little to gain by acceleration when file size=0. 32 MB is the bare minimum requirement for Sort accelerator. >> Does it suffice to have DCB=LRECL=25 even though no record exceeds 24 bytes >> in length? Yes, it would, but status of the job would be depending on the option VLSHRT which pads short records with binary zeroes. Thanks, Kolusu DFSORT Development IBM Corporation -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
On Fri, 13 Jan 2023 16:03:37 +, Sri h Kolusu wrote: > >DFSORT calls LE to process locale and convert the keys and Unicode services >for processing UTF-8/16/32 data. > UTF-8 is a variable-length encoding. Is the length of a key field specified in characters or in octets? >Sort accelerator is NOT compatible with LOCALE. It is listed under >restrictions which is documented here > >https://www.ibm.com/docs/en/zos/2.5.0?topic=works-integrated-accelerator-z-sort > Thanks. I see: Please note that not all sorts would be eligible to use ZSORT. Below is a list of restrictions that will disable ZSORT and will revert to using traditional sorting technique: ... - Unsupported sort fields specified (examples Unicode, Locale, ALTSEQ …) - Unknown file size or file size=0. I suppose there's little to gain by acceleration when file size=0. ... - For variable records, the record length (LRECL) must be greater than 24 Does that mean that for such files the 2**32 limit still applies? Does it suffice to have DCB=LRECL=25 even though no record exceeds 24 bytes in length? Industry-wide jargon uses "stable" where DFSORT uses "EQUAL". Thanks -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
LOL On 1/13/2023 4:17 AM, Radoslaw Skorupka wrote: W dniu 12.01.2023 o 21:07, Tom Brennan pisze: On 1/12/2023 9:37 AM, Radoslaw Skorupka wrote: Maximum input records for DFSORT is infinity, Good to know! I'm testing that to be sure, but it's still running... Wrong. You should generate test data first. :-) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
>> Is that affected by LOCALE? The best-performing way to deal with some >> natural language locales is probably to translate any string to a single >> case after suffixing a bitmap indicating the original case of characters. >> The length of such a mask would need to be included in the total key length. Does DFSORT use strcoll() for such locales? Paul, DFSORT calls LE to process locale and convert the keys and Unicode services for processing UTF-8/16/32 data. >>. The Ref. has a page of usage notes on ZSORT where I don't see LOCALE >>mentioned. Sort accelerator is NOT compatible with LOCALE. It is listed under restrictions which is documented here https://www.ibm.com/docs/en/zos/2.5.0?topic=works-integrated-accelerator-z-sort You would get a reason code of 211 which is explained as a part of the message ICE267I https://www.ibm.com/docs/en/zos/2.5.0?topic=messages-ice267i Thanks, Kolusu DFSORT Development IBM Corporation -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
W dniu 12.01.2023 o 21:07, Tom Brennan pisze: On 1/12/2023 9:37 AM, Radoslaw Skorupka wrote: Maximum input records for DFSORT is infinity, Good to know! I'm testing that to be sure, but it's still running... Wrong. You should generate test data first. :-) -- Radoslaw Skorupka Lodz, Poland -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
On Fri, 13 Jan 2023 00:05:44 +, Sri h Kolusu wrote: > >There is distinction between maximum supported LRECL (32K) and maximum >supported SORT CONTROL FIELDS. The total number of bytes occupied by all sort >control fields must not exceed 4092 (or, when EQUALS option is in operation, >4088 bytes). > Is that affected by LOCALE? The best-performing way to deal with some natural language locales is probably to translate any string to a single case after suffixing a bitmap indicating the original case of characters. The length of such a mask would need to be included in the total key length. Does DFSORT use strcoll() for such locales? DFP keys? >Did you overlook this statement at the end my last post “Btw if you are >running on Z15 and higher you can use sort accelerator(with OPTION ZSORT) >which does NOT have that limit.” > I confess I shot from the hip without reading it. The Ref. has a page of usage notes on ZSORT where I don't see LOCALE mentioned. Some years ago, I went to SR reporting that DFSORT gave correct results for EN_CA but incorrect for EN_US, compared with a desktop sort. SR and I agreed that the problem was in LE, not DFSORT. I went to LE which rebutted with "of course the desktop results are different: desktop is ASCII; DFSORT is EBCDIC." I didn't waste more time arguing with a fool. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
EQUALS was not specified in my SORT parms, but it was listed in the OPTIONS display of the output as EQUALS=Y. I am in the process of getting the program rerun with NOEQUALS specified. C- On Thu, Jan 12, 2023 at 6:06 PM Sri h Kolusu wrote: > >> Is 32 bits an optimum choice? A larger pseudo-key would have some > impact on performance and reduce the maximum supported LRECL. > > Gil, > > There is distinction between maximum supported LRECL (32K) and maximum > supported SORT CONTROL FIELDS. The total number of bytes occupied by all > sort control fields must not exceed 4092 (or, when EQUALS option is in > operation, 4088 bytes). > > >> And Tom Brennan's fiendish test case would still break it. Perhaps > "max records" should be a PARM option -- 2**32 is a good compatible default. > > Did you overlook this statement at the end my last post “Btw if you are > running on Z15 and higher you can use sort accelerator(with OPTION ZSORT) > which does NOT have that limit.” > > >> A few contributors have suggested SPLIT; SORT ...; MERGE. If EQUALS is > (truly!?) needed, would that technique meet the need? > > Yes, it can. Before they go down that elaborate exercise, OP needs to > answer the question “Is EQUALS really needed?” Unless OP cares about the > order of duplicate records , there is NO point of having EQUALS and without > equals there is no limit to what DFSORT can sort. > > Thanks, > Kolusu > DFSORT Development > IBM Corporation > > > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
>> Is 32 bits an optimum choice? A larger pseudo-key would have some impact on >> performance and reduce the maximum supported LRECL. Gil, There is distinction between maximum supported LRECL (32K) and maximum supported SORT CONTROL FIELDS. The total number of bytes occupied by all sort control fields must not exceed 4092 (or, when EQUALS option is in operation, 4088 bytes). >> And Tom Brennan's fiendish test case would still break it. Perhaps "max >> records" should be a PARM option -- 2**32 is a good compatible default. Did you overlook this statement at the end my last post “Btw if you are running on Z15 and higher you can use sort accelerator(with OPTION ZSORT) which does NOT have that limit.” >> A few contributors have suggested SPLIT; SORT ...; MERGE. If EQUALS is >> (truly!?) needed, would that technique meet the need? Yes, it can. Before they go down that elaborate exercise, OP needs to answer the question “Is EQUALS really needed?” Unless OP cares about the order of duplicate records , there is NO point of having EQUALS and without equals there is no limit to what DFSORT can sort. Thanks, Kolusu DFSORT Development IBM Corporation -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
On Thu, 12 Jan 2023 20:45:29 +, Sri h Kolusu wrote: > >As Steve smith pointed out it Is your EQUALS parm that is limiting the number >of records to be sorted. Do you really need EQUALS ( which is retain the >order of duplicates). ? If you don't need it then you can pass NOEQUALS as an >option > My guess about the (OCO?) implementation is that this restriction arises from a technique of appending a 32-bit fictitious key to each record on input and removing it on output. Is 32 bits an optimum choice? A larger pseudo-key would have some impact on performance and reduce the maximum supported LRECL. And Tom Brennan's fiendish test case would still break it. Perhaps "max records" should be a PARM option -- 2**32 is a good compatible default. A few contributors have suggested SPLIT; SORT ...; MERGE. If EQUALS is (truly!?) needed, would that technique meet the need? -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: [EXTERNAL] Re: DFSORT maximum input records
Is EQUALS implied if you're running a program sort instead of a JCL sort? I thought Chuck said he was running the sort inside a program which is why he couldn't just split the data apart and merge it back together. Rex -Original Message- From: IBM Mainframe Discussion List On Behalf Of Sri h Kolusu Sent: Thursday, January 12, 2023 2:45 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: [EXTERNAL] Re: DFSORT maximum input records >> DFSORT issues a message that I have exceeded the sort input record maximum. Chuck, As Steve smith pointed out it Is your EQUALS parm that is limiting the number of records to be sorted. Do you really need EQUALS ( which is retain the order of duplicates). ? If you don't need it then you can pass NOEQUALS as an option OPTION NOEQUALS SORT FIELDS=(P,M,F,A) Or Simply SORT FIELDS=(P,M,F,A),NOEQUALS Btw if you are running on Z15 and higher you can use sort accelerator(with OPTION ZSORT) which does NOT have that limit. Further if you have any questions please let me know Thanks, Sri Hari Kolusu DFSORT Development IBM Corporation -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- The information contained in this message is confidential, protected from disclosure and may be legally privileged. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any disclosure, distribution, copying, or any action taken or action omitted in reliance on it, is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by replying to this message and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
I imagine this wouldn't get you around the problem, but if you were sort the records in a separate DFSORT step, then feed the sorted data to your program, would it still complain about the number of records since it would encounter no need for any sorting at all? Just a thought. --- Bob Bridges, robhbrid...@gmail.com, cell 336 382-7313 /* The most important accomplishment of our lives is not what we accomplish, but what we become. -Rick Joyner, 2015-11 */ -Original Message- From: IBM Mainframe Discussion List On Behalf Of Charles Hardee Sent: Thursday, January 12, 2023 11:43 I am trying to sort roughly 4295217295, records using DFSORT. DFSORT issues a message that I have exceeded the sort input record maximum. It appears that this maximum is 4294967295. Since this is a program calling sort internally, and does so several times, I cannot do the "normal" thing of splitting the file down into smaller files, sort them, then merge them back together. Anyone have any thoughts on this? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
>> DFSORT issues a message that I have exceeded the sort input record maximum. Chuck, As Steve smith pointed out it Is your EQUALS parm that is limiting the number of records to be sorted. Do you really need EQUALS ( which is retain the order of duplicates). ? If you don't need it then you can pass NOEQUALS as an option OPTION NOEQUALS SORT FIELDS=(P,M,F,A) Or Simply SORT FIELDS=(P,M,F,A),NOEQUALS Btw if you are running on Z15 and higher you can use sort accelerator(with OPTION ZSORT) which does NOT have that limit. Further if you have any questions please let me know Thanks, Sri Hari Kolusu DFSORT Development IBM Corporation -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
On 1/12/2023 9:37 AM, Radoslaw Skorupka wrote: Maximum input records for DFSORT is infinity, Good to know! I'm testing that to be sure, but it's still running... -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
Once you have busted a limit like this, it's probably time to rethink the design. What is the data source? If you are processing this many records, are they from a single source? DB2 has a tablespace limit of 128TB. Does your file fit into that size? Loading a table that size with an index(es) that reflect your desired sort sequence would be a hog but could satisfy retrieval with an appropriate ORDER BY. ack On Fri, Jan 13, 2023 at 5:34 AM Charles Hardee wrote: > Yes, we've been informed that SYNCSORt doesn't have this limitation but > getting it in-house takes major efforts since it's a new product to us. > We were hoping for some other DFSORT option to at least let us move > forward. > > Thanks everyone for listening and your replies. > Chuck > > On Thu, Jan 12, 2023 at 12:21 PM rpinion865 < > 042a019916dd-dmarc-requ...@listserv.ua.edu> wrote: > > > This isn't an answer to your problem, and things may have changed > > considerably since the early 1980's. But, SYNCSORT used to have > > the TAPE sort option. Way be when, in the early 1980's, I bumped > > up against DF/SORT's limit of that time, we happened to have a > > trial copy of SYNCSORT. SYNCSORT saved the day, and that helped > > win the decision to replace DF/SORT with SYNCSORT. > > > > But, like I said, that's ancient history :) > > > > > > > > > > Sent with Proton Mail secure email. > > > > --- Original Message --- > > On Thursday, January 12th, 2023 at 1:01 PM, Charles Hardee < > > charleshhar...@gmail.com> wrote: > > > > > > > I have plenty of SORTWKxx DDs. > > > I don't think that is the problem. > > > > > > I am getting an IEX121A message. > > > According to the DFSORT Messages, Codes and Diagnosis Guide for z/OS > 2.5: > > > > > > ICE121A FILE SIZE IS TOO LARGE > > > Explanation > > > Critical. The amount of data to be sorted exceeded > > > a DFSORT implementation limit for the maximum > > > number of records that can be sorted. DFSORT cannot > > > handle the amount of data regardless of environmental > > > factors, such as the amount of work data set space > > > provided. Specific implementation limits include: > > > > > > 1. EQUALS in effect - 4294967295 records > > > > > > 2. VLSHRT in effect - 2147483647 records > > > > > > 3. Blockset technique not selected - 2147483647 records > > > > > > System action > > > The program terminates. > > > > > > Programmer response > > > For cases 1 and 2, perform the sort on subsets of the > > > data set containing less than the indicated maximum > > > number of records. Then merge the sorted subsets > > > into a single data set. > > > > > > For case 3, rerun the job with a SORTDIAG DD > > > statement to get message ICE800I, which indicates > > > the reason Blockset could not be used. If possible, > > > remove the condition preventing the use of Blockset. > > > Alternatively, perform the sort on subsets of the > > > data set containing less than the indicated maximum > > > number of records. Then merge the sorted subsets > > > into a single data set. > > > > > > My record count is roughly 250,000 more than the "implementation > maximum" > > > of 4294967295. > > > > > > As I indicated, this is a program doing several internal sorts against > > the > > > data so splitting, sorting and merging externally is not an option. > > > > > > Chuck > > > > > > On Thu, Jan 12, 2023 at 11:33 AM Farley, Peter < > > > 031df298a9da-dmarc-requ...@listserv.ua.edu> wrote: > > > > > > > Apologies for the typo, the Syncsort option is DYNALLOC, not DYNALOC. > > > > > > > > -Original Message- > > > > From: IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU On > Behalf > > > > Of Farley, Peter > > > > Sent: Thursday, January 12, 2023 12:12 PM > > > > To: IBM-MAIN@LISTSERV.UA.EDU > > > > Subject: Re: DFSORT maximum input records > > > > > > > > Not sure if DFSORT has an equivalent option, but Syncsort has a > > parameter > > > > DYNALOC that can change the maximum number of SORTWKnn DD's it will > > > > allocate. Our local Syncsort default is 64, but for really big sorts > > (when > > > > we get the Syncsort equivalent "too big to sort" message, which says > > "SORT > &g
Re: DFSORT maximum input records
EQUALS adds an internal sort field of 4 bytes to hold an input sequence number, which is what you appear to be having trouble with. You haven't mentioned it, but removing EQUALS will solve your problem. If that's not feasible you can, if necessary, use some fancy stuff to add your own, say, a 6 or 8-byte sequence field on the fly, and use it as the least-significant key (i.e. so you can remove the EQUALS keyword and avoid the limit). btw, EQUALS adds significant overhead to the sort, besides the limitation. Don't use it just for fun. If SYNCSORT doesn't have this limitation, it must be using a bigger sequence field. Maybe you could ask IBM to allow for that. sas On Thu, Jan 12, 2023 at 1:02 PM Charles Hardee wrote: > I have plenty of SORTWKxx DDs. > I don't think that is the problem. > > I am getting an IEX121A message. > According to the DFSORT Messages, Codes and Diagnosis Guide for z/OS 2.5: > > ICE121A FILE SIZE IS TOO LARGE > Explanation > Critical. The amount of data to be sorted exceeded > a DFSORT implementation limit for the maximum > number of records that can be sorted. DFSORT cannot > handle the amount of data regardless of environmental > factors, such as the amount of work data set space > provided. Specific implementation limits include: > > 1. EQUALS in effect - 4294967295 records > > 2. VLSHRT in effect - 2147483647 records > > 3. Blockset technique not selected - 2147483647 records > > System action > The program terminates. > > Programmer response > For cases 1 and 2, perform the sort on subsets of the > data set containing less than the indicated maximum > number of records. Then merge the sorted subsets > into a single data set. > > For case 3, rerun the job with a SORTDIAG DD > statement to get message ICE800I, which indicates > the reason Blockset could not be used. If possible, > remove the condition preventing the use of Blockset. > Alternatively, perform the sort on subsets of the > data set containing less than the indicated maximum > number of records. Then merge the sorted subsets > into a single data set. > > My record count is roughly 250,000 more than the "implementation maximum" > of 4294967295. > > As I indicated, this is a program doing several internal sorts against the > data so splitting, sorting and merging externally is not an option. > > Chuck > > On Thu, Jan 12, 2023 at 11:33 AM Farley, Peter < > 031df298a9da-dmarc-requ...@listserv.ua.edu> wrote: > > > Apologies for the typo, the Syncsort option is DYNALLOC, not DYNALOC. > > > > -Original Message- > > From: IBM Mainframe Discussion List On Behalf > > Of Farley, Peter > > Sent: Thursday, January 12, 2023 12:12 PM > > To: IBM-MAIN@LISTSERV.UA.EDU > > Subject: Re: DFSORT maximum input records > > > > Not sure if DFSORT has an equivalent option, but Syncsort has a parameter > > DYNALOC that can change the maximum number of SORTWKnn DD's it will > > allocate. Our local Syncsort default is 64, but for really big sorts > (when > > we get the Syncsort equivalent "too big to sort" message, which says > "SORT > > CAPACITY EXCEEDED") we use DYNALOC(SYSDA,255). > > > > HTH > > > > Peter > > > > -Original Message- > > From: IBM Mainframe Discussion List On Behalf > > Of Charles Hardee > > Sent: Thursday, January 12, 2023 11:43 AM > > To: IBM-MAIN@LISTSERV.UA.EDU > > Subject: DFSORT maximum input records > > > > Hello Listers, > > > > I've searched the IBM-MAIN archives and found nothing to help me so I > > thought I'd better ask the question. > > > > I am trying to sort roughly 4295217295, records using DFSORT. > > DFSORT issues a message that I have exceeded the sort input record > maximum. > > It appears that this maximum is 4294967295. > > > > Since this is a program calling sort internally, and does so several > > times, I cannot do the "normal" thing of splitting the file down into > > smaller files, sort them, then merge them back together. > > > > Anyone have any thoughts on this? > > > > Thanks, > > Chuck > > -- > > > > > > This message and any attachments are intended only for the use of the > > addressee and may contain information that is privileged and > confidential. > > If the reader of the message is not the intended recipient or an > authorized > > representative of the intended recipient, you are hereby notified that > any > > dissemination of this communication is strictly prohibited. If you have > > received this communication in error, please notify us imme
Re: DFSORT maximum input records
I have run into large files with DFSORT, and when I asked them they told me that there are hard limits: * EQUALS in effect - 4 294 967 295 records * VLSHRT in effect - 2 147 483 647 records * Blockset technique not selected - 2 147 483 647 records I wonder if you could do something in your process to either segment the sorts into two pieces based on the high order part of the sort key, or if you could split, sort, and merge files outside of your program. It sounds to me that it is a special program that has to touch 4.3billion records in its run. Thank you and best regards, Billy Ashton -- Original Message -- From "Charles Hardee" To IBM-MAIN@listserv.ua.edu Date 1/12/2023 11:42:51 AM Subject DFSORT maximum input records Hello Listers, I've searched the IBM-MAIN archives and found nothing to help me so I thought I'd better ask the question. I am trying to sort roughly 4295217295, records using DFSORT. DFSORT issues a message that I have exceeded the sort input record maximum. It appears that this maximum is 4294967295. Since this is a program calling sort internally, and does so several times, I cannot do the "normal" thing of splitting the file down into smaller files, sort them, then merge them back together. Anyone have any thoughts on this? Thanks, Chuck -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
Yes, we've been informed that SYNCSORt doesn't have this limitation but getting it in-house takes major efforts since it's a new product to us. We were hoping for some other DFSORT option to at least let us move forward. Thanks everyone for listening and your replies. Chuck On Thu, Jan 12, 2023 at 12:21 PM rpinion865 < 042a019916dd-dmarc-requ...@listserv.ua.edu> wrote: > This isn't an answer to your problem, and things may have changed > considerably since the early 1980's. But, SYNCSORT used to have > the TAPE sort option. Way be when, in the early 1980's, I bumped > up against DF/SORT's limit of that time, we happened to have a > trial copy of SYNCSORT. SYNCSORT saved the day, and that helped > win the decision to replace DF/SORT with SYNCSORT. > > But, like I said, that's ancient history :) > > > > > Sent with Proton Mail secure email. > > --- Original Message --- > On Thursday, January 12th, 2023 at 1:01 PM, Charles Hardee < > charleshhar...@gmail.com> wrote: > > > > I have plenty of SORTWKxx DDs. > > I don't think that is the problem. > > > > I am getting an IEX121A message. > > According to the DFSORT Messages, Codes and Diagnosis Guide for z/OS 2.5: > > > > ICE121A FILE SIZE IS TOO LARGE > > Explanation > > Critical. The amount of data to be sorted exceeded > > a DFSORT implementation limit for the maximum > > number of records that can be sorted. DFSORT cannot > > handle the amount of data regardless of environmental > > factors, such as the amount of work data set space > > provided. Specific implementation limits include: > > > > 1. EQUALS in effect - 4294967295 records > > > > 2. VLSHRT in effect - 2147483647 records > > > > 3. Blockset technique not selected - 2147483647 records > > > > System action > > The program terminates. > > > > Programmer response > > For cases 1 and 2, perform the sort on subsets of the > > data set containing less than the indicated maximum > > number of records. Then merge the sorted subsets > > into a single data set. > > > > For case 3, rerun the job with a SORTDIAG DD > > statement to get message ICE800I, which indicates > > the reason Blockset could not be used. If possible, > > remove the condition preventing the use of Blockset. > > Alternatively, perform the sort on subsets of the > > data set containing less than the indicated maximum > > number of records. Then merge the sorted subsets > > into a single data set. > > > > My record count is roughly 250,000 more than the "implementation maximum" > > of 4294967295. > > > > As I indicated, this is a program doing several internal sorts against > the > > data so splitting, sorting and merging externally is not an option. > > > > Chuck > > > > On Thu, Jan 12, 2023 at 11:33 AM Farley, Peter < > > 0000031df298a9da-dmarc-requ...@listserv.ua.edu> wrote: > > > > > Apologies for the typo, the Syncsort option is DYNALLOC, not DYNALOC. > > > > > > -Original Message- > > > From: IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU On Behalf > > > Of Farley, Peter > > > Sent: Thursday, January 12, 2023 12:12 PM > > > To: IBM-MAIN@LISTSERV.UA.EDU > > > Subject: Re: DFSORT maximum input records > > > > > > Not sure if DFSORT has an equivalent option, but Syncsort has a > parameter > > > DYNALOC that can change the maximum number of SORTWKnn DD's it will > > > allocate. Our local Syncsort default is 64, but for really big sorts > (when > > > we get the Syncsort equivalent "too big to sort" message, which says > "SORT > > > CAPACITY EXCEEDED") we use DYNALOC(SYSDA,255). > > > > > > HTH > > > > > > Peter > > > > > > -Original Message- > > > From: IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU On Behalf > > > Of Charles Hardee > > > Sent: Thursday, January 12, 2023 11:43 AM > > > To: IBM-MAIN@LISTSERV.UA.EDU > > > Subject: DFSORT maximum input records > > > > > > Hello Listers, > > > > > > I've searched the IBM-MAIN archives and found nothing to help me so I > > > thought I'd better ask the question. > > > > > > I am trying to sort roughly 4295217295, records using DFSORT. > > > DFSORT issues a message that I have exceeded the sort input record > maximum. > > > It appears that this maximum is 4294967295. > > > > > > Since this is a program calling s
Re: DFSORT maximum input records
This isn't an answer to your problem, and things may have changed considerably since the early 1980's. But, SYNCSORT used to have the TAPE sort option. Way be when, in the early 1980's, I bumped up against DF/SORT's limit of that time, we happened to have a trial copy of SYNCSORT. SYNCSORT saved the day, and that helped win the decision to replace DF/SORT with SYNCSORT. But, like I said, that's ancient history :) Sent with Proton Mail secure email. --- Original Message --- On Thursday, January 12th, 2023 at 1:01 PM, Charles Hardee wrote: > I have plenty of SORTWKxx DDs. > I don't think that is the problem. > > I am getting an IEX121A message. > According to the DFSORT Messages, Codes and Diagnosis Guide for z/OS 2.5: > > ICE121A FILE SIZE IS TOO LARGE > Explanation > Critical. The amount of data to be sorted exceeded > a DFSORT implementation limit for the maximum > number of records that can be sorted. DFSORT cannot > handle the amount of data regardless of environmental > factors, such as the amount of work data set space > provided. Specific implementation limits include: > > 1. EQUALS in effect - 4294967295 records > > 2. VLSHRT in effect - 2147483647 records > > 3. Blockset technique not selected - 2147483647 records > > System action > The program terminates. > > Programmer response > For cases 1 and 2, perform the sort on subsets of the > data set containing less than the indicated maximum > number of records. Then merge the sorted subsets > into a single data set. > > For case 3, rerun the job with a SORTDIAG DD > statement to get message ICE800I, which indicates > the reason Blockset could not be used. If possible, > remove the condition preventing the use of Blockset. > Alternatively, perform the sort on subsets of the > data set containing less than the indicated maximum > number of records. Then merge the sorted subsets > into a single data set. > > My record count is roughly 250,000 more than the "implementation maximum" > of 4294967295. > > As I indicated, this is a program doing several internal sorts against the > data so splitting, sorting and merging externally is not an option. > > Chuck > > On Thu, Jan 12, 2023 at 11:33 AM Farley, Peter < > 031df298a9da-dmarc-requ...@listserv.ua.edu> wrote: > > > Apologies for the typo, the Syncsort option is DYNALLOC, not DYNALOC. > > > > -Original Message- > > From: IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU On Behalf > > Of Farley, Peter > > Sent: Thursday, January 12, 2023 12:12 PM > > To: IBM-MAIN@LISTSERV.UA.EDU > > Subject: Re: DFSORT maximum input records > > > > Not sure if DFSORT has an equivalent option, but Syncsort has a parameter > > DYNALOC that can change the maximum number of SORTWKnn DD's it will > > allocate. Our local Syncsort default is 64, but for really big sorts (when > > we get the Syncsort equivalent "too big to sort" message, which says "SORT > > CAPACITY EXCEEDED") we use DYNALOC(SYSDA,255). > > > > HTH > > > > Peter > > > > -Original Message- > > From: IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU On Behalf > > Of Charles Hardee > > Sent: Thursday, January 12, 2023 11:43 AM > > To: IBM-MAIN@LISTSERV.UA.EDU > > Subject: DFSORT maximum input records > > > > Hello Listers, > > > > I've searched the IBM-MAIN archives and found nothing to help me so I > > thought I'd better ask the question. > > > > I am trying to sort roughly 4295217295, records using DFSORT. > > DFSORT issues a message that I have exceeded the sort input record maximum. > > It appears that this maximum is 4294967295. > > > > Since this is a program calling sort internally, and does so several > > times, I cannot do the "normal" thing of splitting the file down into > > smaller files, sort them, then merge them back together. > > > > Anyone have any thoughts on this? > > > > Thanks, > > Chuck > > -- > > > > This message and any attachments are intended only for the use of the > > addressee and may contain information that is privileged and confidential. > > If the reader of the message is not the intended recipient or an authorized > > representative of the intended recipient, you are hereby notified that any > > dissemination of this communication is strictly prohibited. If you have > > received this communication in error, please notify us immediately by > > e-mail and delete the message and any attachments from your system. > > > > --
Re: DFSORT maximum input records
I have plenty of SORTWKxx DDs. I don't think that is the problem. I am getting an IEX121A message. According to the DFSORT Messages, Codes and Diagnosis Guide for z/OS 2.5: ICE121A FILE SIZE IS TOO LARGE Explanation Critical. The amount of data to be sorted exceeded a DFSORT implementation limit for the maximum number of records that can be sorted. DFSORT cannot handle the amount of data regardless of environmental factors, such as the amount of work data set space provided. Specific implementation limits include: 1. EQUALS in effect - 4294967295 records 2. VLSHRT in effect - 2147483647 records 3. Blockset technique not selected - 2147483647 records System action The program terminates. Programmer response For cases 1 and 2, perform the sort on subsets of the data set containing less than the indicated maximum number of records. Then merge the sorted subsets into a single data set. For case 3, rerun the job with a SORTDIAG DD statement to get message ICE800I, which indicates the reason Blockset could not be used. If possible, remove the condition preventing the use of Blockset. Alternatively, perform the sort on subsets of the data set containing less than the indicated maximum number of records. Then merge the sorted subsets into a single data set. My record count is roughly 250,000 more than the "implementation maximum" of 4294967295. As I indicated, this is a program doing several internal sorts against the data so splitting, sorting and merging externally is not an option. Chuck On Thu, Jan 12, 2023 at 11:33 AM Farley, Peter < 031df298a9da-dmarc-requ...@listserv.ua.edu> wrote: > Apologies for the typo, the Syncsort option is DYNALLOC, not DYNALOC. > > -Original Message- > From: IBM Mainframe Discussion List On Behalf > Of Farley, Peter > Sent: Thursday, January 12, 2023 12:12 PM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: Re: DFSORT maximum input records > > Not sure if DFSORT has an equivalent option, but Syncsort has a parameter > DYNALOC that can change the maximum number of SORTWKnn DD's it will > allocate. Our local Syncsort default is 64, but for really big sorts (when > we get the Syncsort equivalent "too big to sort" message, which says "SORT > CAPACITY EXCEEDED") we use DYNALOC(SYSDA,255). > > HTH > > Peter > > -Original Message- > From: IBM Mainframe Discussion List On Behalf > Of Charles Hardee > Sent: Thursday, January 12, 2023 11:43 AM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: DFSORT maximum input records > > Hello Listers, > > I've searched the IBM-MAIN archives and found nothing to help me so I > thought I'd better ask the question. > > I am trying to sort roughly 4295217295, records using DFSORT. > DFSORT issues a message that I have exceeded the sort input record maximum. > It appears that this maximum is 4294967295. > > Since this is a program calling sort internally, and does so several > times, I cannot do the "normal" thing of splitting the file down into > smaller files, sort them, then merge them back together. > > Anyone have any thoughts on this? > > Thanks, > Chuck > -- > > > This message and any attachments are intended only for the use of the > addressee and may contain information that is privileged and confidential. > If the reader of the message is not the intended recipient or an authorized > representative of the intended recipient, you are hereby notified that any > dissemination of this communication is strictly prohibited. If you have > received this communication in error, please notify us immediately by > e-mail and delete the message and any attachments from your system. > > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
On Thu, 12 Jan 2023 10:42:51 -0600 Charles Hardee wrote: :>I've searched the IBM-MAIN archives and found nothing to help me so I :>thought I'd better ask the question. :>I am trying to sort roughly 4295217295, records using DFSORT. :>DFSORT issues a message that I have exceeded the sort input record maximum. :>It appears that this maximum is 4294967295. :>Since this is a program calling sort internally, and does so several times, :>I cannot do the "normal" thing of splitting the file down into smaller :>files, sort them, then merge them back together. :>Anyone have any thoughts on this? 2**32-1 = 4294967295 Just saying. -- Binyamin Dissen http://www.dissensoftware.com Director, Dissen Software, Bar & Grill - Israel -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
Maximum input records for DFSORT is infinity, *except* some options like EQUALS. For EQUALS the maximum is 4Gi records (means 2**32). Sometimes it is good idea to change SORT to MERGE. -- Radoslaw Skorupka Lodz, Poland -Original Message- From: IBM Mainframe Discussion List On Behalf Of Charles Hardee Sent: Thursday, January 12, 2023 11:43 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: DFSORT maximum input records Hello Listers, I've searched the IBM-MAIN archives and found nothing to help me so I thought I'd better ask the question. I am trying to sort roughly 4295217295, records using DFSORT. DFSORT issues a message that I have exceeded the sort input record maximum. It appears that this maximum is 4294967295. Since this is a program calling sort internally, and does so several times, I cannot do the "normal" thing of splitting the file down into smaller files, sort them, then merge them back together. Anyone have any thoughts on this? Thanks, Chuck -- -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
Apologies for the typo, the Syncsort option is DYNALLOC, not DYNALOC. -Original Message- From: IBM Mainframe Discussion List On Behalf Of Farley, Peter Sent: Thursday, January 12, 2023 12:12 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: DFSORT maximum input records Not sure if DFSORT has an equivalent option, but Syncsort has a parameter DYNALOC that can change the maximum number of SORTWKnn DD's it will allocate. Our local Syncsort default is 64, but for really big sorts (when we get the Syncsort equivalent "too big to sort" message, which says "SORT CAPACITY EXCEEDED") we use DYNALOC(SYSDA,255). HTH Peter -Original Message- From: IBM Mainframe Discussion List On Behalf Of Charles Hardee Sent: Thursday, January 12, 2023 11:43 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: DFSORT maximum input records Hello Listers, I've searched the IBM-MAIN archives and found nothing to help me so I thought I'd better ask the question. I am trying to sort roughly 4295217295, records using DFSORT. DFSORT issues a message that I have exceeded the sort input record maximum. It appears that this maximum is 4294967295. Since this is a program calling sort internally, and does so several times, I cannot do the "normal" thing of splitting the file down into smaller files, sort them, then merge them back together. Anyone have any thoughts on this? Thanks, Chuck -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT maximum input records
Not sure if DFSORT has an equivalent option, but Syncsort has a parameter DYNALOC that can change the maximum number of SORTWKnn DD's it will allocate. Our local Syncsort default is 64, but for really big sorts (when we get the Syncsort equivalent "too big to sort" message, which says "SORT CAPACITY EXCEEDED") we use DYNALOC(SYSDA,255). HTH Peter -Original Message- From: IBM Mainframe Discussion List On Behalf Of Charles Hardee Sent: Thursday, January 12, 2023 11:43 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: DFSORT maximum input records Hello Listers, I've searched the IBM-MAIN archives and found nothing to help me so I thought I'd better ask the question. I am trying to sort roughly 4295217295, records using DFSORT. DFSORT issues a message that I have exceeded the sort input record maximum. It appears that this maximum is 4294967295. Since this is a program calling sort internally, and does so several times, I cannot do the "normal" thing of splitting the file down into smaller files, sort them, then merge them back together. Anyone have any thoughts on this? Thanks, Chuck -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
DFSORT maximum input records
Hello Listers, I've searched the IBM-MAIN archives and found nothing to help me so I thought I'd better ask the question. I am trying to sort roughly 4295217295, records using DFSORT. DFSORT issues a message that I have exceeded the sort input record maximum. It appears that this maximum is 4294967295. Since this is a program calling sort internally, and does so several times, I cannot do the "normal" thing of splitting the file down into smaller files, sort them, then merge them back together. Anyone have any thoughts on this? Thanks, Chuck -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
