Re: Syncsort to DFSORT - my time has come.
Apparently, the specification of format (or at least CH) on Syncsort JOINKEYS is optional. I don't have it specified in another job that has been working for some time. > -Original Message- > From: IBM Mainframe Discussion List On > Behalf Of Sri h Kolusu > Sent: Thursday, October 29, 2020 8:58 AM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: Re: Syncsort to DFSORT - my time has come. > > >>For compatibility and migration purposes, is there a way for DFSORT to > accept this specification? > > Unfortunately NO. The other product had to introduce the formats for > compensating the rich functionality of JNF1CNTL and JNF2CNTL in DFSORT. > DFSORT gives you the entire INREC formatting with JNF1 and JNF2. > > >>I could check and see if Syncsort actually required this specification or > if I did it by habit. > > If the key formats are of the same format (like in this case both key > fields are character data) , you absolutely don't require the format. > However if your key has ZD and PD combination then you can use JNF1/JNF2 > to > normalize one of the key to match the other. > > for example as 4 byte Packed decimal field is compared to 9 byte zoned > decimal number > > JOINKEYS FILE=F1,FIELDS=(1,4,PD,A) > JOINKEYS FILE=F2,FIELDS=(6,9,ZD,A) > > You can change this to > > JOINKEYS FILE=F1,FIELDS=(81,9,A) > JOINKEYS FILE=F2,FIELDS=(6,9,A) > > //JNF1CNTL DD * > INREC OVERLAY=(81:1,4,PD,ZD,LENGTH=9) $ convert 4 byte PD to 9 byte ZD > format > /* > > > Thanks, > Kolusu > DFSORT Development > IBM Corporation > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: How best to copy all UNIX files one z/OS to another
True, don't know what I was thinking :) Shouldn't have started w/ pax. Wanted to say, unmount, DFDSS, etc. Anyway, if there are easier ways.. - KB ‐‐‐ Original Message ‐‐‐ On Thursday, October 29, 2020 7:21 PM, Paul Gilmartin <000433f07816-dmarc-requ...@listserv.ua.edu> wrote: > On Wed, 28 Oct 2020 18:49:46 +, Michael Brennan wrote: > > > After the pax, unmount your new ZFS/HFS file. DFDSS dump it, terse the dump > > then FTP the tersed file. At the receiving site, unterse and restore. > > That seems to be an exercise in seeing how many needless utilities you can > exploit. Just transfer the pax archive. > > And the OP says bandwidth is not a constraint; no need to compress. > Even so, pax has a "-z" option. > > -- gil > > > > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: UNIX fork() performance (was: SMF to capture ... )
Good work! On 30/10/2020 10:40 am, Andrew Rowley wrote: On 30/10/2020 2:32 am, Kirk Wolf wrote: Your performance work in this area is very interesting to me. I would love to hear more if you ever write up your findings, even informally. I have been looking at SMF data and trying to build a picture of the work that was running from the SMF records. I have written up some of what I have found here: https://www.blackhillsoftware.com/news/2019/08/27/comparing-bash-and-bin-sh-on-z-os/ and here: https://www.blackhillsoftware.com/news/2019/06/20/understanding-z-os-unix-work-with-easysmf/ When a job runs unix work much of the work can appear in SMF records for the unix tasks, not the original job. E.g. for the script I was testing, running under bash only 0.02s of CPU time appears in the SMF record for the job. But there are SMF records for about 9000 unix tasks which add up to about 66s of CPU. Under /bin/sh, the job CPU time appears the same at 0.02s, but with the records from the unix work it is about 4.7s. Either way, you miss most of the work if you just look at the job records. Interestingly, I tried running the script under COZBATCH and it broke the logic to associate the unix tasks with the main job. I will have to look into that. COZBATCH did run steps in a separate address space and generated about the same number of SMF records, but it looked different to BPXBATCH with _BPX_SHAREAS. I don't really understand what I see in the SDSF PS display. The same script, run under COZBATCH and BPXBATCH with _BPX_SHAREAS (sorry about the wrap): JOBNAME JobID Status Owner State PID PPID ASID ASIDX Command ANDREWRB JOB03101 FILE SYS KERNEL WAIT ANDREWR 1F 66115 16843349 51 0033 cut -c1 ANDREWRB STC03080 RUNNING ANDREWR 1R 16843329 16843349 79 004F /bin/sh /home/andrewr/breakit.sh ANDREWRB JOB03101 WAITING FOR CHILD ANDREWR 1W 16843349 50397780 51 0033 /bin/sh /home/andrewr/breakit.sh ANDREWRB JOB03101 WAITING FOR CHILD ANDREWR 1W 50397780 67175122 51 0033 /bin/sh ANDREWRB JOB03101 FILE SYS KERNEL WAIT ANDREWR 1F 67175122 1 51 0033 COZBATCH JOBNAME JobID Status Owner State PID PPID ASID ASIDX Command ANDREWRE STC03080 RUNNING ANDREWR 1R 33620729 50397388 79 004F setfacl -m u:IBMUSER:rwx broken/24793 ANDREWRE STC03080 WAITING FOR CHILD ANDREWR 1W 50397388 83952212 79 004F /bin/sh /home/andrewr/breakit.sh ANDREWRE JOB03106 FILE SYS KERNEL WAIT ANDREWR 1F 83952212 1 51 0033 BPXBATCH What is the extra "/bin/sh /home/andrewr/breakit.sh" command in the separate address space under COZBATCH? Under COZBATCH, PS consistently showed the cut command which is part of the pipe. Under BPXBATCH, the only command which seemed to show up was setfacl. Presumably that is because that was the command where most of the time was spent. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: UNIX fork() performance (was: SMF to capture ... )
On 30/10/2020 2:32 am, Kirk Wolf wrote: Your performance work in this area is very interesting to me. I would love to hear more if you ever write up your findings, even informally. I have been looking at SMF data and trying to build a picture of the work that was running from the SMF records. I have written up some of what I have found here: https://www.blackhillsoftware.com/news/2019/08/27/comparing-bash-and-bin-sh-on-z-os/ and here: https://www.blackhillsoftware.com/news/2019/06/20/understanding-z-os-unix-work-with-easysmf/ When a job runs unix work much of the work can appear in SMF records for the unix tasks, not the original job. E.g. for the script I was testing, running under bash only 0.02s of CPU time appears in the SMF record for the job. But there are SMF records for about 9000 unix tasks which add up to about 66s of CPU. Under /bin/sh, the job CPU time appears the same at 0.02s, but with the records from the unix work it is about 4.7s. Either way, you miss most of the work if you just look at the job records. Interestingly, I tried running the script under COZBATCH and it broke the logic to associate the unix tasks with the main job. I will have to look into that. COZBATCH did run steps in a separate address space and generated about the same number of SMF records, but it looked different to BPXBATCH with _BPX_SHAREAS. I don't really understand what I see in the SDSF PS display. The same script, run under COZBATCH and BPXBATCH with _BPX_SHAREAS (sorry about the wrap): JOBNAME JobID Status Owner State PID PPID ASID ASIDX Command ANDREWRB JOB03101 FILE SYS KERNEL WAIT ANDREWR 1F 66115 16843349 51 0033 cut -c1 ANDREWRB STC03080 RUNNING ANDREWR 1R 16843329 16843349 79 004F /bin/sh /home/andrewr/breakit.sh ANDREWRB JOB03101 WAITING FOR CHILD ANDREWR 1W 16843349 50397780 51 0033 /bin/sh /home/andrewr/breakit.sh ANDREWRB JOB03101 WAITING FOR CHILD ANDREWR 1W 50397780 67175122 51 0033 /bin/sh ANDREWRB JOB03101 FILE SYS KERNEL WAIT ANDREWR 1F 67175122 1 51 0033 COZBATCH JOBNAME JobID Status Owner State PID PPID ASID ASIDX Command ANDREWRE STC03080 RUNNING ANDREWR 1R 33620729 50397388 79 004F setfacl -m u:IBMUSER:rwx broken/24793 ANDREWRE STC03080 WAITING FOR CHILD ANDREWR 1W 50397388 83952212 79 004F /bin/sh /home/andrewr/breakit.sh ANDREWRE JOB03106 FILE SYS KERNEL WAIT ANDREWR 1F 83952212 1 51 0033 BPXBATCH What is the extra "/bin/sh /home/andrewr/breakit.sh" command in the separate address space under COZBATCH? Under COZBATCH, PS consistently showed the cut command which is part of the pipe. Under BPXBATCH, the only command which seemed to show up was setfacl. Presumably that is because that was the command where most of the time was spent. -- Andrew Rowley Black Hill Software -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Another DFSORT Question
> The control statements below work fine. They do what I wanted. > Today my question is if these SORT Control statements can be optimized? > This is not urgent. Cameron, Your job definitely can be optimized. But I want to make a observation that you might be skipping some results. You are assuming that the ddnames DMNETMAP and DMSGFIL have the dataset name on the same line. 1. ) For example if your input has the following Input, you will NOT pick the DSN names //DMNETMAP DD DISP=SHR, //DSN=$HLQ.NETMAP //DMPUBLIB DD DISP=SHR, // DD DSN=$HLQ.PROCESL //DMMSGFIL DD DISP=OLD, //DSN=$HLQ.MSG4 2.) Similarly the stepnames can any where from 1 byte to 8 bytes, however your picking bytes from 4 thru 8 . Assume the following input //S1 EXEC PGM=DMBATCH,REGION=1024K,PARM=(YYSLYNN) . 3.) What happens if SYSIN cards are generated from previous step as a temp dataset and passed over to DMBATCH step? ex: https://www.ibm.com/support/pages/how-submit-instream-process-jcl-using-dgadbatc-dmbatch Check how & is generated. How do you plan to handle the & contents? 4.) Also the program DMBATCH can have SYSIN2 also, do you plan to extract the information? 5.) Technically SYSIN can be a PDS with member name. So the maximum length of PDS+Member = 54 bytes ( 44 Dsn name+ 8 byte member name + 2 byte for opening and closing parenthesis). However your control cards don't seem to be handling them. Let me know the answers to be above questions and I will show you the optimized control cards. Thanks, Kolusu DFSORT Development IBM Corporation -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: UNIX fork() performance (was: SMF to capture ... )
On 30/10/2020 12:58 am, Paul Gilmartin wrote: I suspect he's confusing "address space" with "execution environment": https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12 If you're referring to me: it's possible, but I don't think so. Referring to the comparison of bash and /bin/sh I linked previously, I can see the address space from the job number field in SMF. The main bash process runs in STC06733 (a BPX initiator address space). If I look at e.g the setfacl command I can see STC06734 (i.e. a different address space) initially runs bash, but then sub step 0 ends and sub step 1 starts, running the setfacl command. That appears to be a fork of bash followed by exec of setfacl in a new sub step as described here: https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.ieag200/ext.htm If I compare /bin/sh: The top level SH is running in STC06733. setfacl runs in STC06736, sometimes STC06737. It is sub step 0 - there is no sub step for the shell. There is one SH step for each iteration of the loop - I think this is the pipe. The mkdir steps seen under bash disappear completely - I guess it is built into the shell. When I tried with _BPX_SHAREAS all the individual steps for the shell commands disappeared, except for the SH step -- Andrew Rowley Black Hill Software -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: UNIX fork() performance (was: SMF to capture ... )
On Thu, 29 Oct 2020 16:21:00 -0500, Kirk Wolf wrote: >/bin/sh always uses spawn except in cases where it uses fork. Most of the >latter cases are pretty obvious: command &, `command` / $(command), etc. > I would expect one case requiring fork() is "( list )" because "list" inherits non-exported shell variables while running in a separate execution environment. E.g.: 510 $ x=foobar; ( echo $x; x=wombat; echo $x ); echo $x foobar wombat foobar >To have the spawns be local (same address space), you simply need to set >_BPX_SHAREAS=YES before the command. In my example, /etc/profile is the >last place to set it via "export _BPX_SHAREAS=YES". Thanks, gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Another DFSORT Question
Hello , I have put together this set of SORT control statements to scan through an IEBPTPCH dump of a JCL or PROC PDS. The DFSORT statements will look for any JOB Steps that execute program DMBATCH (this is an NDM Program). The statements will gather up some DD Statement information related to the program. I wanted to collect the DMNETMAP, DMMSGFIL and the SYSIN Control Statements data set names. The SYSIN could be a concatenated sequence of datasets. The control statements below work fine. They do what I wanted. Today my question is if these SORT Control statements can be optimized? This is not urgent. Any ideas are appreciated. Thanks, ** ** * SCAN THROUGH THE IEBPTPCH STREAM AND GENERATE AN OUTPUT FILE. * ** * WE IGNORE ALL COMMENT RECORDS.* * WE ONLY SELECT RECORDS WITH THE SCANNED PDS MEMBER NAMEAND* * RECORDS THAT EXECUTE PROCNAME * * THE OUTPUT FROM THE PROCESS WILL HAVE * * JCL/PROC LIBRARY NAME IN 001 THROUGH 003 * * JCL/PROC MEMBER NAME IN 005 THROUGH 012 * * DMNETMAP DATASET NAME IN 014 THROUGH 058 * * DMMSGFIL DATASET NAME IN 060 THROUGH 104 * * SYSIN DATASET NAMEIN 106 THROUGH 150 * ** ** OMIT COND=(02,03,CH,EQ,C'//*',OR, * SKIP COMMENTS (02,02,CH,NE,C'//',AND, 2,11,CH,NE,C'MEMBER NAME')) * SKIP DATA RECORDS * * THE MEMBER GROUP STARTS WITH THE "MEMBER NAME" RECORD. * INREC IFTHEN=(WHEN=GROUP, BEGIN=(2,11,CH,EQ,C'MEMBER NAME'), PUSH=(082:SEQ=8,* SEQUENCE # 091:15,08)), * JCL MEMBER NAME * * THE DMBATCH GROUP APPLIES TO THE ENTIRE DMBATCH STEP. * IT BEGINS WITH "PGM=DMBATCH" AND WILL END AFTER THE * DMBATCH STEP. * IFTHEN=(WHEN=GROUP, BEGIN=(4,40,SS,EQ,C'PGM=DMBATCH'), END=(004,05,CH,NE,C'SYSIN',AND, 004,05,CH,NE,C' ',AND, 1PUSH=(099:04,08)), * STEP NAME * * THE DMNETMAP GROUP APPLIES TO THE ENTIRE DMBATCH STEP. * IT BEGINS WITH THE DDNAME, AND ENDS WITH THE NEXT * EXEC STATEMENT. * IFTHEN=(WHEN=GROUP, BEGIN=(4,08,CH,EQ,C'DMNETMAP'), END=(4,20,SS,EQ,C' EXEC '), PUSH=(107:21,40)), * DMNETMAP DSN * * THE DMMSGFIL GROUP APPLIES TO THE ENTIRE DMBATCH STEP. * IT BEGINS WITH THE DDNAME, AND ENDS WITH THE NEXT * EXEC STATEMENT. * IFTHEN=(WHEN=GROUP, BEGIN=(4,08,CH,EQ,C'DMMSGFIL'), END=(4,20,SS,EQ,C' EXEC '), PUSH=(147:21,40)), * DMMSGFIL DSN * * GROUP STARTS WITH SYSIN DD STATEMENT AND ENDS * WHEN ANY CONCATENATION SEQUENCE ENDS. * IFTHEN=(WHEN=GROUP, BEGIN=(4,05,CH,EQ,C'SYSIN'), END=(4,05,CH,NE,C'SYSIN',AND, 4,05,CH,NE,C' '), PUSH=(187:04,05)), * SYSIN IFTHEN=(WHEN=(187,05,CH,EQ,C'SYSIN',AND,* INSTREAM RECORDS 012,40,SS,EQ,C' * '), BUILD=(001:JP1,* LIBRARY DATASET C',', 005:091,08, * MEMBER NAME C',', 014:107,40, * DMNETMAP DATASET C',', 055:147,40, * DMMSGFIL DATASET C',', 096:C'INSTREAM SYSIN DD * DATA', C',', 137:099,08, * DMBATCH STEP NAME 145:C'*')), IFTHEN=(WHEN=(187,05,CH,EQ,C'SYSIN',AND,* DATASET NAME 017,40,SS,EQ,C'DSN='), BUILD=(001:JP1,* LIBRARY DATASET C',', 005:091,08, * MEMBER NAME C',', 014:107,40, * DMNETMAP DATASET C',', 055:147,40, * DMMSGFIL DATASET C',', 096:021,40, * SYSIN DATASET C',', 137:099,08, * DMBATCH STEP NAME 145:C'D'))
Re: UNIX fork() performance (was: SMF to capture ... )
/bin/sh always uses spawn except in cases where it uses fork. Most of the latter cases are pretty obvious: command &, `command` / $(command), etc. To have the spawns be local (same address space), you simply need to set _BPX_SHAREAS=YES before the command. In my example, /etc/profile is the last place to set it via "export _BPX_SHAREAS=YES". Kirk http://dovetail.com PS> Gil, I'm astonished that you're astonished - we've been talking about BPXBATCH, BPXBATSL, AOPBATCH, COZBATCH (originally called DTLSPAWN) and_BPX_SHAREAS etc, etc on ibm-main and mvs-oe since 2006-2007 :-) On Thu, Oct 29, 2020 at 11:06 AM Paul Gilmartin < 000433f07816-dmarc-requ...@listserv.ua.edu> wrote: > On Thu, 29 Oct 2020 10:32:35 -0500, Kirk Wolf wrote: > > > >//SHELL EXEC PGM=COZBATCH > >//IN DD DSN=SYS1.MACLIB(ACB),DISP=SHR > >//OUTDD SYSOUT=* > >//STDIN DD * > ># run the user's login shell in the same address space > >fromdsn //DD:IN | grep IBM | todsn //DD:OUT > >// > I'm astonished that neither fromdsn nor todsn was forked. Is > that because of _BPX_SHAREAS=YES? Where was that set? > > -- gil > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Modify UNIX File Format
On Thu, 29 Oct 2020 19:17:50 +, Frank Swarbrick wrote: >That worked. I never in a million years would have found it. How did you >know? > Several years ago, I asked the same question on MVS-OE and WJS answered. >Your URL pointed to the binder instruction, though, not the UNIX command, >which is >https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.bpxa500/xattr.htm > I spotted that and fixed it, but not in time. >DVFJS:/u/dvfjs:>touch attrtest >DVFJS:/u/dvfjs:>chtag -t -c 819 attrtest >DVFJS:/u/dvfjs:>extattr -F CRLF attrtest >DVFJS:/u/dvfjs:>ls -FalTHp attrtest >t ISO8859-1 T=on -rw--- crlf 1 DVFJSDEPT9971 0 Oct 29 >14:13 attrtest > and REXX has ADDRESS SYSCALL chattr. It's annoying that many of these are unavailable via the JCL DD FILEDATA() option. But if they were I suppose access methods would be obliged to honor them as record separators. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Modify UNIX File Format
That worked. I never in a million years would have found it. How did you know? Your URL pointed to the binder instruction, though, not the UNIX command, which is https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.bpxa500/xattr.htm DVFJS:/u/dvfjs:>touch attrtest DVFJS:/u/dvfjs:>chtag -t -c 819 attrtest DVFJS:/u/dvfjs:>extattr -F CRLF attrtest DVFJS:/u/dvfjs:>ls -FalTHp attrtest t ISO8859-1 T=on -rw--- crlf 1 DVFJSDEPT9971 0 Oct 29 14:13 attrtest Thanks! From: IBM Mainframe Discussion List on behalf of Paul Gilmartin <000433f07816-dmarc-requ...@listserv.ua.edu> Sent: Thursday, October 29, 2020 1:08 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Modify UNIX File Format On Thu, 29 Oct 2020 18:31:03 +, Frank Swarbrick wrote: >Using the z/OS UNIX Directory List Utility (ISPF 3.17) command MF (Modify >Format) you are able to change the file format. It prompts you with a panel >looking something like this: >... >Is there a corresponding UNIX command to perform this action? For example, >chtag can be used to change the CCSID. But I can't find a command to change >the file format value. > I believe "extattr": https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.ieab100/exta.htm -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Modify UNIX File Format
On Thu, 29 Oct 2020 14:08:12 -0500, Paul Gilmartin wrote: >On Thu, 29 Oct 2020 18:31:03 +, Frank Swarbrick wrote: > >>Using the z/OS UNIX Directory List Utility (ISPF 3.17) command MF (Modify >>Format) you are able to change the file format. It prompts you with a panel >>looking something like this: >>... >>Is there a corresponding UNIX command to perform this action? For example, >>chtag can be used to change the CCSID. But I can't find a command to change >>the file format value. >> >I believe "extattr": > > https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.ieab100/exta.htm > Oops! Better URL: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.bpxa500/xattr.htm It's annoying that many of these aren't available via the JCL DD FILEDATA() option. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Modify UNIX File Format
On Thu, 29 Oct 2020 18:31:03 +, Frank Swarbrick wrote: >Using the z/OS UNIX Directory List Utility (ISPF 3.17) command MF (Modify >Format) you are able to change the file format. It prompts you with a panel >looking something like this: >... >Is there a corresponding UNIX command to perform this action? For example, >chtag can be used to change the CCSID. But I can't find a command to change >the file format value. > I believe "extattr": https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.ieab100/exta.htm -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Modify UNIX File Format
Using the z/OS UNIX Directory List Utility (ISPF 3.17) command MF (Modify Format) you are able to change the file format. It prompts you with a panel looking something like this: Modify z/OS UNIX File Format Command ===> Pathname . : /u/dvfjs/cics_headers.txt Type . . . : File Format . . . 6 1. NA 3. NL 5. LF 7. LFCR 9. Record 2. Binary 4. CR 6. CRLF 8. CRNL CCSID . . . 819 Enter "/" to select option / Automatic Conversion Is there a corresponding UNIX command to perform this action? For example, chtag can be used to change the CCSID. But I can't find a command to change the file format value. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
GSE 2020 Virtual event - November 2nd - 12th
Hi everyone, You may already be aware that the GSE UK Virtual Conference for 2020, will take place between 2nd and 12th November and will in place of the annual physical conference that will return in 2021. One of the big differences between this virtual and the physical conference is that there will be no charge to attend. We have over 150 hours of content during the two week period, including 17 sessions on the security stream featuring some well known presenters! More details about the conference, including the agenda and registration are available here > https://conferences.gse.org.uk/2020. As the conference is being hosted out of the UK, all times quoted on the agenda are in the GMT time zone. Please note that sessions are also being recorded. If you decide to register and you are not from a GSE member company, no problem – when completing the registration form, click non-member and enter your company name. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: How best to copy all UNIX files one z/OS to another
On Thu, 29 Oct 2020 11:52:35 -0400, Ken Smith wrote: > >But - are there FTP command parms that would copy the Unix files directly >and retain all attributes? (no intermediate pax/dss file). > Not FTP, but ssh: ( cd $HOME & pax -w . ) | ssh user@other-lpar 'cd & pax -vr' I do it routinely. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: UNIX fork() performance (was: SMF to capture ... )
On Thu, 29 Oct 2020 10:32:35 -0500, Kirk Wolf wrote: > >//SHELL EXEC PGM=COZBATCH >//IN DD DSN=SYS1.MACLIB(ACB),DISP=SHR >//OUTDD SYSOUT=* >//STDIN DD * ># run the user's login shell in the same address space >fromdsn //DD:IN | grep IBM | todsn //DD:OUT >// I'm astonished that neither fromdsn nor todsn was forked. Is that because of _BPX_SHAREAS=YES? Where was that set? -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Syncsort to DFSORT - my time has come.
>>For compatibility and migration purposes, is there a way for DFSORT to accept this specification? Unfortunately NO. The other product had to introduce the formats for compensating the rich functionality of JNF1CNTL and JNF2CNTL in DFSORT. DFSORT gives you the entire INREC formatting with JNF1 and JNF2. >>I could check and see if Syncsort actually required this specification or if I did it by habit. If the key formats are of the same format (like in this case both key fields are character data) , you absolutely don't require the format. However if your key has ZD and PD combination then you can use JNF1/JNF2 to normalize one of the key to match the other. for example as 4 byte Packed decimal field is compared to 9 byte zoned decimal number JOINKEYS FILE=F1,FIELDS=(1,4,PD,A) JOINKEYS FILE=F2,FIELDS=(6,9,ZD,A) You can change this to JOINKEYS FILE=F1,FIELDS=(81,9,A) JOINKEYS FILE=F2,FIELDS=(6,9,A) //JNF1CNTL DD * INREC OVERLAY=(81:1,4,PD,ZD,LENGTH=9) $ convert 4 byte PD to 9 byte ZD format /* Thanks, Kolusu DFSORT Development IBM Corporation -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: How best to copy all UNIX files one z/OS to another
I've used pax to archive (aka zip/tar) => transmit => unpax and for a whole File System dfDSS copy of FS to archive => transmit => restore FS Transmit can mean FTP of the intermediate file or using a shared volume For routine copies, one could include a job routed (via jes) to the other system to unpack. But - are there FTP command parms that would copy the Unix files directly and retain all attributes? (no intermediate pax/dss file). On Thu, Oct 29, 2020 at 10:11 AM Lloyd Fuller wrote: > And nobody has mentioned that pax will write or read an MVS file so you > are not restricted to Open MVS for the intermediate files. > > Regards. > > Lloyd > > Sent from my iPad > > > On Oct 29, 2020, at 10:01 AM, Charles Mills wrote: > > > > *Size* (disk space) might be a constraint so compression is good. > > > > Charles > > > > > > -Original Message- > > From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] > On Behalf Of Paul Gilmartin > > Sent: Thursday, October 29, 2020 6:51 AM > > To: IBM-MAIN@LISTSERV.UA.EDU > > Subject: Re: How best to copy all UNIX files one z/OS to another > > > >> On Wed, 28 Oct 2020 18:49:46 +, Michael Brennan wrote: > >> > >> After the pax, unmount your new ZFS/HFS file. DFDSS dump it, terse the > dump then FTP the tersed file. At the receiving site, unterse and restore. > >> > > That seems to be an exercise in seeing how many needless utilities you > can > > exploit. Just transfer the pax archive. > > > > And the OP says bandwidth is not a constraint; no need to compress. > > Even so, pax has a "-z" option. > > > > -- gil > > > > -- > > For IBM-MAIN subscribe / signoff / archive access instructions, > > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > > > > -- > > For IBM-MAIN subscribe / signoff / archive access instructions, > > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Syncsort to DFSORT - my time has come.
Thank you. Mos of our "fancy" sort stuff is mine, so I can make these changes. But, and this is just asking, not really expecting. For compatibility and migration purposes, is there a way for DFSORT to accept this specification? I have 4 LPARs. This job runs in 3 of them. 2 still run Syncsort. My migration will take a little more time. I could check and see if Syncsort actually required this specification or if I did it by habit. > -Original Message- > From: IBM Mainframe Discussion List On > Behalf Of Sri h Kolusu > Sent: Thursday, October 29, 2020 3:05 AM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: Re: Syncsort to DFSORT - my time has come. > > Dave, > > DFSORT Joinkeys does not require the FORMAT of the key field as a binary > match is done implicitly. > > So change your control cards to the following(Just removed the CH on the > Fields statement) > > JOINKEYS FILE=F1,FIELDS=(1,9,A),TYPE=F > JOINKEYS FILE=F2,FIELDS=(6,9,A), > INCLUDE=(6,9,CH,EQ,C'MCR02143I'),TYPE=V > > > Further if you have any questions please let me know > > > Thanks, > Kolusu > DFSORT Development > IBM Corporation > > > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: UNIX fork() performance (was: SMF to capture ... )
Andrew, Your performance work in this area is very interesting to me. I would love to hear more if you ever write up your findings, even informally. We had a customer a few years ago who seem to have performance problems with fork/execs related to BPXAS address space startups. IBM worked on it for a long time and then somehow it magically went away and no one could say why. I'm not sure exactly what you are doing in your benchmarks, but in COZBATCH we do an explicit local spawn of your login shell so as to get it to run in the same address space. Here's proof that pipeline shell commands can run in the job step address space: //SHELL EXEC PGM=COZBATCH //IN DD DSN=SYS1.MACLIB(ACB),DISP=SHR //OUTDD SYSOUT=* //STDIN DD * # run the user's login shell in the same address space fromdsn //DD:IN | grep IBM | todsn //DD:OUT // fromdsn(DD:IN)[N]: 79 records/6320 bytes read; 6399 bytes written in 0 milliseconds. todsn(DD:OUT)[N]: 243 bytes read; 3 records/240 bytes written in 0 milliseconds. CoZBatch[I]: returning rc=exitcode=0 there are all sorts of undocumented pitfalls surrounding where /bin/sh does a spawn vs fork. For example, a "builtin" is always forked when used in a pipeline, except for when it is last and you have -o pipecurrent set. You can hack this by using /bin/cp or /bin/cat (say, with a //DD:xxx). Gil will point out that this usage isn't explicitly documented by IBM (as with many things). Of course, this isn't that interesting to people who believe that things run "in z/OS Unix" rather than the reality that pretty much any job/task can *use* z/OS UNIX services, filesystems, programs, etc. :-)We find that using Unix shell scripts in z/OS batch works really nicely. On Wed, Oct 28, 2020 at 6:41 PM Andrew Rowley wrote: > On 29/10/2020 8:40 am, Kirk Wolf wrote: > > /bin/sh (when it uses spawn() and _BPX_SHAREAS=YES/MUST) will try to > start > > the child process in the **same** address space as the parent. > > > > I just reran the same script with _BPX_SHAREAS=YES to see what it looked > like. > > It cut the number of SMF records from 20,000 to 4,000 with only 1 > process showing up per iteration - maybe required for the pipe? > > Elapsed time didn't change. CPU time was reduced by about 20%. > > With _BPX_SHAREAS=YES you could no longer see information on the > commands making up the shell script. From the runs without > _BPX_SHAREAS=YES the SMF data showed most of the elapsed time was due to > the setfacl command. > > -- > > Andrew Rowley > Black Hill Software > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: How best to copy all UNIX files one z/OS to another
And nobody has mentioned that pax will write or read an MVS file so you are not restricted to Open MVS for the intermediate files. Regards. Lloyd Sent from my iPad > On Oct 29, 2020, at 10:01 AM, Charles Mills wrote: > > *Size* (disk space) might be a constraint so compression is good. > > Charles > > > -Original Message- > From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On > Behalf Of Paul Gilmartin > Sent: Thursday, October 29, 2020 6:51 AM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: Re: How best to copy all UNIX files one z/OS to another > >> On Wed, 28 Oct 2020 18:49:46 +, Michael Brennan wrote: >> >> After the pax, unmount your new ZFS/HFS file. DFDSS dump it, terse the dump >> then FTP the tersed file. At the receiving site, unterse and restore. >> > That seems to be an exercise in seeing how many needless utilities you can > exploit. Just transfer the pax archive. > > And the OP says bandwidth is not a constraint; no need to compress. > Even so, pax has a "-z" option. > > -- gil > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: How best to copy all UNIX files one z/OS to another
On Thu, 29 Oct 2020 07:01:31 -0700, Charles Mills wrote: >*Size* (disk space) might be a constraint so compression is good. > A pipe is your friend. Why should you need disk space other than for the source and the target, which I suspect you don't intend to compress? >From: Paul Gilmartin >Sent: Thursday, October 29, 2020 6:51 AM\ > >And the OP says bandwidth is not a constraint; no need to compress. >Even so, pax has a "-z" option. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: How best to copy all UNIX files one z/OS to another
*Size* (disk space) might be a constraint so compression is good. Charles -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Paul Gilmartin Sent: Thursday, October 29, 2020 6:51 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: How best to copy all UNIX files one z/OS to another On Wed, 28 Oct 2020 18:49:46 +, Michael Brennan wrote: >After the pax, unmount your new ZFS/HFS file. DFDSS dump it, terse the dump >then FTP the tersed file. At the receiving site, unterse and restore. > That seems to be an exercise in seeing how many needless utilities you can exploit. Just transfer the pax archive. And the OP says bandwidth is not a constraint; no need to compress. Even so, pax has a "-z" option. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: UNIX fork() performance (was: SMF to capture ... )
On Wed, 28 Oct 2020 16:40:23 -0500, Kirk Wolf wrote: >On Tue, Oct 27, 2020 at 5:37 PM Andrew Rowley wrote: > >> /bin/sh is executing the commands in a separate address space but >> without the sub-step created by fork-exec (I guess this is the spawn >> method that Kirk Wolf described). >> >/bin/sh (when it uses spawn() and _BPX_SHAREAS=YES/MUST) will try to start >the child process in the **same** address space as the parent. > I suspect he's confusing "address space" with "execution environment": https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12 >Get some processes running at once and use "ps" with right options or SDSF >PS and you can see which processes share the same ASID. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: How best to copy all UNIX files one z/OS to another
On Wed, 28 Oct 2020 18:49:46 +, Michael Brennan wrote: >After the pax, unmount your new ZFS/HFS file. DFDSS dump it, terse the dump >then FTP the tersed file. At the receiving site, unterse and restore. > That seems to be an exercise in seeing how many needless utilities you can exploit. Just transfer the pax archive. And the OP says bandwidth is not a constraint; no need to compress. Even so, pax has a "-z" option. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: How best to copy all UNIX files one z/OS to another
On Thu, 29 Oct 2020 03:34:19 +, kekronbekron wrote:
>Wondering why no one has suggested the all new USS file/directory dumping
>capability in DFDSS.
>...
>the output will probably be huge.
>
I believe you've answered your own question.
>See if Co:Z SFTP can help in any way w.r.t managing the transfer process.
>
Do you need anything more than "awk -w | ssh useer@host "awk -r"?
awk has various options to control UID and GID of created files.
How would you deal with non-portable filenames created by e.g.:
#! /bin/sh
# Doc: Create files with non-portable names.
set -x
mkdir Weirdos & cd Weirdos & : >test || exit $?
set +x
awk 'BEGIN {
for ( I = 1; I <128; ++I ) {
FName = sprintf( "%03d foo%cbar", I, I )
if ( match( FName, "/" ) ) continue
printf( "%03d\n", I ) >FName
close( FName ) }
}'
-- giil
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT wizard help
Could this be done quite easily using sed?
Create a sed script file
# split.sed
s/},{/}\
{/g
And then run
sed -f split.sed input.file > output.records
Your file looks like it is json. You will also need to remove the square
brackets that surround the array of records.
I'll leave this as an exercise for the reader.
Ray
-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On
Behalf Of Ituriel do Neto
Sent: 29 October 2020 11:06
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: DFSORT wizard help
Gentlemen,
The logical record i want starts with a "{" and ends with "}"
Thanks
Ituriel
Em quarta-feira, 28 de outubro de 2020 19:13:58 BRT, Paul Gilmartin
<000433f07816-dmarc-requ...@listserv.ua.edu> escreveu:
On Wed, 28 Oct 2020 20:40:54 +, Ituriel do Neto wrote:
>Hi,
>I have an omvs file and would like to separate the fields into multiple
>lines.The input file has the following format:
>
What's your criterion for separating fields into lines? Your example is
insufficiently informative.
> Ý{"date":"20200616-0400","MMSU":11206,"IH":35,"X":302},{"date":"20200616-0500","MMSU":11235,"IH":29,"X":303},{"date":"20200616-0600","MMSU":11269,"IH":34,"X":304},{"date":"20200616-0700","MMSU":11309,"IH":40,"X":305},{"date":"20200616-0800","MMSU":11352,"IH":43,"X":306},{"date":"20200616-0900","MMSU":11403,"IH":51,"X":307},{"date":"20200616-1000","MMSU":11459,"IH":56,"X":308},{"date":"20200616-1100","MMSU":11516,"IH":57,"X":309},{"date":"20200616-1200","MMSU":11585,"IH":69,"X":310},{"date":"20200616-1300","MMSU":11639,"IH":54,"X":311},{"date":"20200616-1400","MMSU":11689,"IH":50,"X":312},{"date":"20200616-1500","MMSU":11742,"IH":53,"X":313},{"date":"20200616-1600","MMSU":11805,"IH":63,"X":314},{"date":"20200616.
>And I would like to create a dataset in this
>format:{"date":"20200616-0400","MMSU":11206,"IH":35,"X":302}
>{"date":"20200616-0500","MMSU":11235,"IH":29,"X":303}{"date":"20200616-0600","MMSU":11269,"IH":34,"X":304}...
>I know SORT can to magic, but I'm not getting success.Any help will be
>appreciated.
-- gil
--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
--
This e-mail message has been scanned and cleared by Google Message Security
and the UNICOM Global security systems. This message is for the named
person's use only. If you receive this message in error, please delete it
and notify the sender.
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: DFSORT wizard help
Gentlemen,
The logical record i want starts with a "{" and ends with "}"
Thanks
Ituriel
Em quarta-feira, 28 de outubro de 2020 19:13:58 BRT, Paul Gilmartin
<000433f07816-dmarc-requ...@listserv.ua.edu> escreveu:
On Wed, 28 Oct 2020 20:40:54 +, Ituriel do Neto wrote:
>Hi,
>I have an omvs file and would like to separate the fields into multiple
>lines.The input file has the following format:
>
What's your criterion for separating fields into lines? Your example
is insufficiently informative.
> Ý{"date":"20200616-0400","MMSU":11206,"IH":35,"X":302},{"date":"20200616-0500","MMSU":11235,"IH":29,"X":303},{"date":"20200616-0600","MMSU":11269,"IH":34,"X":304},{"date":"20200616-0700","MMSU":11309,"IH":40,"X":305},{"date":"20200616-0800","MMSU":11352,"IH":43,"X":306},{"date":"20200616-0900","MMSU":11403,"IH":51,"X":307},{"date":"20200616-1000","MMSU":11459,"IH":56,"X":308},{"date":"20200616-1100","MMSU":11516,"IH":57,"X":309},{"date":"20200616-1200","MMSU":11585,"IH":69,"X":310},{"date":"20200616-1300","MMSU":11639,"IH":54,"X":311},{"date":"20200616-1400","MMSU":11689,"IH":50,"X":312},{"date":"20200616-1500","MMSU":11742,"IH":53,"X":313},{"date":"20200616-1600","MMSU":11805,"IH":63,"X":314},{"date":"20200616.
>And I would like to create a dataset in this
>format:{"date":"20200616-0400","MMSU":11206,"IH":35,"X":302}
>{"date":"20200616-0500","MMSU":11235,"IH":29,"X":303}{"date":"20200616-0600","MMSU":11269,"IH":34,"X":304}...
>I know SORT can to magic, but I'm not getting success.Any help will be
>appreciated.
-- gil
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: How best to copy all UNIX files one z/OS to another
Always in support of the OLD flat file system..seen the need to help roll into the MVS--z/OS fold. But over the years it has become reversed .Problem dealing with the OLD style MS/Unix style systems. And the problem dealing with the OMVS systems locking up and causing the z/OS to fail. -Original Message- From: IBM Mainframe Discussion List On Behalf Of kekronbekron Sent: Wednesday, October 28, 2020 10:34 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: How best to copy all UNIX files one z/OS to another ** EXTERNAL EMAIL - USE CAUTION ** Wondering why no one has suggested the all new USS file/directory dumping capability in DFDSS. I would also run ls -alf in a batch job against all the 'old' mount points to get a listing of owners & permissions. Batch because the output will probably be huge. See if Co:Z SFTP can help in any way w.r.t managing the transfer process. - KB ‐‐‐ Original Message ‐‐‐ On Thursday, October 29, 2020 3:10 AM, Michael Brennan wrote: > After you get your unix files to the target system, If you need to > change ownership of all the directories the following will come in handy: > > //JS10 EXEC PGM=IKJEFT01 > //SYSTSPRT DD SYSOUT=* > //SYSTSIN DD * > BPXBATCH SH chown -R USERID:GROUPID /To_Directory/ > > Where USERID is the RACF/ACF2/TSS id that you want to be the owner and > Where GROUPID is the Group you want to be the owner. > > -- > -- > -- > -- > -- > --- > > For IBM-MAIN subscribe / signoff / archive access instructions, send > email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN Email Disclaimer This E-mail contains confidential information belonging to the sender, which may be legally privileged information. This information is intended only for the use of the individual or entity addressed above. If you are not the intended recipient, or an employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution, or the taking of any action in reliance on the contents of the E-mail or attached files is strictly prohibited. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Syncsort to DFSORT - my time has come.
Dave, DFSORT Joinkeys does not require the FORMAT of the key field as a binary match is done implicitly. So change your control cards to the following(Just removed the CH on the Fields statement) JOINKEYS FILE=F1,FIELDS=(1,9,A),TYPE=F JOINKEYS FILE=F2,FIELDS=(6,9,A), INCLUDE=(6,9,CH,EQ,C'MCR02143I'),TYPE=V Further if you have any questions please let me know Thanks, Kolusu DFSORT Development IBM Corporation -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
