Re: [IBM External] Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[Paul Gilmartin]

On Wed, 29 Sep 2021 06:54:09 +, Martin Packer wrote:
>
>Between steps can't be pipes, can be VIO. Between jobs can be pipes, can't
>be VIO.
>
Pipes could work between steps if the PIPE SUBSYS has an implied ELASTIC
stage.  But that's just reinventing VIO.  PIPE SUBSYS must do some buffering
to handle BSAM RECFM=VB.  Or does it require that RECFM and BLKSIZE
or the producer and consumer be identical?

>That second sentence depends on the ability to schedule two jobs (possibly
>originally steps of the same job) alongside each other.
>
Alas, the Initiator won't do the latter for you.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: [IBM External] The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[kekronbekron]

BTW, are most of these recommendations still valid?

- KB

‐‐‐ Original Message ‐‐‐

On Wednesday, September 29th, 2021 at 7:34 PM, Martin Packer 
 wrote:

> Thank you KB!
>
> You can imagine my disappointment at finding all the links to it at the
>
> IBM Redbooks site to be broken.
>
> Cheers, Martin
>
> Martin Packer
>
> WW z/OS Performance, Capacity and Architecture, IBM Technology Sales
>
> +44-7802-245-584
>
> email: martin_pac...@uk.ibm.com
>
> Twitter / Facebook IDs: MartinPacker
>
> Blog: https://mainframeperformancetopics.com
>
> Mainframe, Performance, Topics Podcast Series (With Marna Walle):
>
> https://anchor.fm/marna-walle
>
> Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA
>
> From: "kekronbekron" 02dee3fcae33-dmarc-requ...@listserv.ua.edu
>
> To: IBM-MAIN@LISTSERV.UA.EDU
>
> Date: 29/09/2021 14:46
>
> Subject: [EXTERNAL] Re: [IBM External] The Business Case for
>
> BatchPipes in the z/OS Base (was: ... Pipes ...)
>
> Sent by: "IBM Mainframe Discussion List" IBM-MAIN@LISTSERV.UA.EDU
>
> Well my friends, I found it for ye.
>
> https://web.archive.org/web/20060224031157/http://www.redbooks.ibm.com/redbooks/pdfs/sg242557.pdf
>
> Saying this to nobody on this thread, but to some kind of people
>
> Now tell me what's important... knowing how to get to stuff or knowing by
>
> memory all the new parmlib members / keywords in zOS 2.5
>
> -   KB
>
> Original Message
>
> On Wednesday, September 29th, 2021 at 6:45 PM, Martin Packer
>
> martin_pac...@uk.ibm.com wrote:
>
> > I think it's gone. Does anyone still have a PDF of it?
> >
> > Thanks, Martin
> >
> > Martin Packer
> >
> > WW z/OS Performance, Capacity and Architecture, IBM Technology Sales
> >
> > +44-7802-245-584
> >
> > email: martin_pac...@uk.ibm.com
> >
> > Twitter / Facebook IDs: MartinPacker
> >
> > Blog:
>
> https://mainframeperformancetopics.com
>
> > Mainframe, Performance, Topics Podcast Series (With Marna Walle):
>
> https://anchor.fm/marna-walle
>
> > Youtube channel:
>
> https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA
>
> > From: "René Jansen" rene.vincent.jan...@gmail.com
> >
> > To: IBM-MAIN@LISTSERV.UA.EDU
> >
> > Date: 29/09/2021 09:12
> >
> > Subject: [EXTERNAL] Re: [IBM External] The Business Case for
> >
> > BatchPipes in the z/OS Base (was: ... Pipes ...)
> >
> > Sent by: "IBM Mainframe Discussion List" IBM-MAIN@LISTSERV.UA.EDU
> >
> > Martin,
> >
> > Do you have that book somewhere still? I spent 15 minutes on google but
> >
> > lots of links lead nowhere.
> >
> > NetRexx does have a rather complete CMS Pipelines implementation (
> >
> > www.netrexx.org <
>
> http://www.netrexx.org/
>
> > > ) and I was wondering what I need to do to also have these ’halfpipes’
> >
> > and other batch interfaces, using JZOS or other.
> >
> > Best regards,
> >
> > René.
> >
> > > On 29 Sep 2021, at 08:54, Martin Packer martin_pac...@uk.ibm.com
> >
> > wrote:
> >
> > > Between steps can't be pipes, can be VIO. Between jobs can be pipes,
> >
> > can't
> >
> > > be VIO.
> > >
> > > That second sentence depends on the ability to schedule two jobs
> >
> > (possibly
> >
> > > originally steps of the same job) alongside each other.
> > >
> > > Fun stuff but / and somewhat complex - which is what inspired me to
> >
> > start
> >
> > > writing what would become SG24-2557 Parallel Sysplex Batch Performance
> >
> > in
> >
> > > late 1990. :-)
> > >
> > > I did a lot of presenting on Pipes to individual customers and
> >
> > conferences
> >
> > > in the 1990s. It would be fun to do it again... :-)
> > >
> > > Cheers, Martin
> > >
> > > Sent from my iPad
> > >
> > > > On 29 Sep 2021, at 05:23, Paul Gilmartin
> > > >
> > > > 000433f07816-dmarc-requ...@listserv.ua.edu wrote:
> > > >
> > > > On Mon, 27 Sep 2021 11:39:17 -0500, Hobart Spitz wrote:
> > > >
> > > > > Intra-JOB pipe - This would be similar to VIO; it doesn't exist
> > > > >
> > > > > otherwise.
> > > >
> > > > > I think it would be a great feature.
> > > >
> > > > Between steps?

Re: [IBM External] The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[Martin Packer]

Thank you KB!

You can imagine my disappointment at finding all the links to it at the 
IBM Redbooks site to be broken.

Cheers, Martin

Martin Packer

WW z/OS Performance, Capacity and Architecture, IBM Technology Sales

+44-7802-245-584

email: martin_pac...@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog: https://mainframeperformancetopics.com

Mainframe, Performance, Topics Podcast Series (With Marna Walle): 
https://anchor.fm/marna-walle

Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From:   "kekronbekron" <02dee3fcae33-dmarc-requ...@listserv.ua.edu>
To: IBM-MAIN@LISTSERV.UA.EDU
Date:   29/09/2021 14:46
Subject:[EXTERNAL] Re: [IBM External] The Business Case for 
BatchPipes in the z/OS Base (was: ... Pipes ...)
Sent by:"IBM Mainframe Discussion List" 



Well my friends, I found it for ye.

https://web.archive.org/web/20060224031157/http://www.redbooks.ibm.com/redbooks/pdfs/sg242557.pdf
 



*Saying this to nobody on this thread, but to some kind of people*
Now tell me what's important... knowing how to get to stuff or knowing by 
memory all the new parmlib members / keywords in zOS 2.5

- KB

�\�\�\�\�\�\�\ Original Message �\�\�\�\�\�\�\

On Wednesday, September 29th, 2021 at 6:45 PM, Martin Packer 
 wrote:

> I think it's gone. Does anyone still have a PDF of it?
>
> Thanks, Martin
>
> Martin Packer
>
> WW z/OS Performance, Capacity and Architecture, IBM Technology Sales
>
> +44-7802-245-584
>
> email: martin_pac...@uk.ibm.com
>
> Twitter / Facebook IDs: MartinPacker
>
> Blog: 
https://mainframeperformancetopics.com 

>
> Mainframe, Performance, Topics Podcast Series (With Marna Walle):
>
> 
https://anchor.fm/marna-walle 

>
> Youtube channel: 
https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA 

>
> From: "René Jansen" rene.vincent.jan...@gmail.com
>
> To: IBM-MAIN@LISTSERV.UA.EDU
>
> Date: 29/09/2021 09:12
>
> Subject: [EXTERNAL] Re: [IBM External] The Business Case for
>
> BatchPipes in the z/OS Base (was: ... Pipes ...)
>
> Sent by: "IBM Mainframe Discussion List" IBM-MAIN@LISTSERV.UA.EDU
>
> Martin,
>
> Do you have that book somewhere still? I spent 15 minutes on google but
>
> lots of links lead nowhere.
>
> NetRexx does have a rather complete CMS Pipelines implementation (
>
> www.netrexx.org <
>
> 
http://www.netrexx.org/ 

>
> > ) and I was wondering what I need to do to also have these ’halfpipes’
>
> and other batch interfaces, using JZOS or other.
>
> Best regards,
>
> René.
>
> > On 29 Sep 2021, at 08:54, Martin Packer martin_pac...@uk.ibm.com
>
> wrote:
>
> > Between steps can't be pipes, can be VIO. Between jobs can be pipes,
>
> can't
>
> > be VIO.
> >
> > That second sentence depends on the ability to schedule two jobs
>
> (possibly
>
> > originally steps of the same job) alongside each other.
> >
> > Fun stuff but / and somewhat complex - which is what inspired me to
>
> start
>
> > writing what would become SG24-2557 Parallel Sysplex Batch Performance
>
> in
>
> > late 1990. :-)
> >
> > I did a lot of presenting on Pipes to individual customers and
>
> conferences
>
> > in the 1990s. It would be fun to do it again... :-)
> >
> > Cheers, Martin
> >
> > Sent from my iPad
> >
> > > On 29 Sep 2021, at 05:23, Paul Gilmartin
> > >
> > > 000433f07816-dmarc-requ...@listserv.ua.edu wrote:
> > >
> > > On Mon, 27 Sep 2021 11:39:17 -0500, Hobart Spitz wrote:
> > >
> > > > Intra-JOB pipe - This would be similar to VIO; it doesn't exist
> > > >
> > > > otherwise.
> > >
> > > > I think it would be a great feature.
> > >
> > > Between steps? One might just as well use VIO.
> > >
> > > -- gil
> > >
> > > For IBM-MAIN subscribe / signoff / archive access instructions,
> > >
> > > send email to lists...@listserv.ua.edu with the message: INFO
>
> IBM-MAINUnless stated otherwise above:
>
> > IBM United Kingdom Limited - Registered in England and Wales with 
number
>
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>
> 3AU
>
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> >
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
>
> For IBM-MAIN subscribe / signoff / archive access instructions,
>
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> Unless stated otherwise above:
>
> IBM

Re: [IBM External] The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[Ron Wells]

Ahh basicsI remember

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
kekronbekron
Sent: Wednesday, September 29, 2021 8:46 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: [IBM External] The Business Case for BatchPipes in the z/OS Base 
(was: ... Pipes ...)

** EXTERNAL EMAIL - USE CAUTION **


Well my friends, I found it for ye.

https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fweb.archive.org%2Fweb%2F20060224031157%2Fhttp%3A%2F%2Fwww.redbooks.ibm.com%2Fredbooks%2Fpdfs%2Fsg242557.pdfdata=04%7C01%7CRon.Wells%40OMF.COM%7C5d18d65d03ac481b280408d9834f7854%7C57c0053cb5f84a1e8bb6e8afa09f3b82%7C0%7C0%7C63768520084735%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=QrpAr3pkkI4EqsiRBLS8xMBCqEn59a3xvdfeO6woEGk%3Dreserved=0


*Saying this to nobody on this thread, but to some kind of people* Now tell me 
what's important... knowing how to get to stuff or knowing by memory all the 
new parmlib members / keywords in zOS 2.5

- KB

‐‐‐ Original Message ‐‐‐

On Wednesday, September 29th, 2021 at 6:45 PM, Martin Packer 
 wrote:

> I think it's gone. Does anyone still have a PDF of it?
>
> Thanks, Martin
>
> Martin Packer
>
> WW z/OS Performance, Capacity and Architecture, IBM Technology Sales
>
> +44-7802-245-584
>
> email: martin_pac...@uk.ibm.com
>
> Twitter / Facebook IDs: MartinPacker
>
> Blog:
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmain
> frameperformancetopics.com%2Fdata=04%7C01%7CRon.Wells%40OMF.COM%7
> C5d18d65d03ac481b280408d9834f7854%7C57c0053cb5f84a1e8bb6e8afa09f3b82%7
> C0%7C0%7C637685200847310001%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=nYYf
> Xc%2BO%2F4BhKOG9G5yNW5OgSVXUpPSDg1eDT1tBvx0%3Dreserved=0
>
> Mainframe, Performance, Topics Podcast Series (With Marna Walle):
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fanch
> or.fm%2Fmarna-walledata=04%7C01%7CRon.Wells%40OMF.COM%7C5d18d65d0
> 3ac481b280408d9834f7854%7C57c0053cb5f84a1e8bb6e8afa09f3b82%7C0%7C0%7C6
> 37685200847310001%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi
> V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=6laUrA7rfLQ9FG
> Ion0vaCk8qODQGTGLBs8kcXaNsOfY%3Dreserved=0
>
> Youtube channel:
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> youtube.com%2Fchannel%2FUCu_65HaYgksbF6Q8SQ4oOvAdata=04%7C01%7CRo
> n.Wells%40OMF.COM%7C5d18d65d03ac481b280408d9834f7854%7C57c0053cb5f84a1
> e8bb6e8afa09f3b82%7C0%7C0%7C637685200847310001%7CUnknown%7CTWFpbGZsb3d
> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C
> 3000sdata=prb%2FQMce629xpliec12VBsH378IG7cbKVc66rA6WW7g%3Dre
> served=0
>
> From: "René Jansen" rene.vincent.jan...@gmail.com
>
> To: IBM-MAIN@LISTSERV.UA.EDU
>
> Date: 29/09/2021 09:12
>
> Subject: [EXTERNAL] Re: [IBM External] The Business Case for
>
> BatchPipes in the z/OS Base (was: ... Pipes ...)
>
> Sent by: "IBM Mainframe Discussion List" IBM-MAIN@LISTSERV.UA.EDU
>
> Martin,
>
> Do you have that book somewhere still? I spent 15 minutes on google
> but
>
> lots of links lead nowhere.
>
> NetRexx does have a rather complete CMS Pipelines implementation (
>
> https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.n
> etrexx.org%2Fdata=04%7C01%7CRon.Wells%40OMF.COM%7C5d18d65d03ac481
> b280408d9834f7854%7C57c0053cb5f84a1e8bb6e8afa09f3b82%7C0%7C0%7C6376852
> 00847310001%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz
> IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=7G0d%2BLopX5dw7x2O6f
> Imcn83jLCAoRLLJVM0aObRkrg%3Dreserved=0 <
>
> https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.n
> etrexx.org%2Fdata=04%7C01%7CRon.Wells%40OMF.COM%7C5d18d65d03ac481
> b280408d9834f7854%7C57c0053cb5f84a1e8bb6e8afa09f3b82%7C0%7C0%7C6376852
> 00847310001%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz
> IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=7G0d%2BLopX5dw7x2O6f
> Imcn83jLCAoRLLJVM0aObRkrg%3Dreserved=0
>
> > ) and I was wondering what I need to do to also have these ’halfpipes’
>
> and other batch interfaces, using JZOS or other.
>
> Best regards,
>
> René.
>
> > On 29 Sep 2021, at 08:54, Martin Packer martin_pac...@uk.ibm.com
>
> wrote:
>
> > Between steps can't be pipes, can be VIO. Between jobs can be pipes,
>
> can't
>
> > be VIO.
> >
> > That second sentence depends on the ability to schedule two jobs
>
> (possibly
>
> > originally steps of the same job) alongside each other.
> >
> > Fun stuff but / and somewhat complex - which is what inspired me to
>
> sta

Re: [IBM External] The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[kekronbekron]

Well my friends, I found it for ye.

https://web.archive.org/web/20060224031157/http://www.redbooks.ibm.com/redbooks/pdfs/sg242557.pdf


*Saying this to nobody on this thread, but to some kind of people*
Now tell me what's important... knowing how to get to stuff or knowing by 
memory all the new parmlib members / keywords in zOS 2.5

- KB

‐‐‐ Original Message ‐‐‐

On Wednesday, September 29th, 2021 at 6:45 PM, Martin Packer 
 wrote:

> I think it's gone. Does anyone still have a PDF of it?
>
> Thanks, Martin
>
> Martin Packer
>
> WW z/OS Performance, Capacity and Architecture, IBM Technology Sales
>
> +44-7802-245-584
>
> email: martin_pac...@uk.ibm.com
>
> Twitter / Facebook IDs: MartinPacker
>
> Blog: https://mainframeperformancetopics.com
>
> Mainframe, Performance, Topics Podcast Series (With Marna Walle):
>
> https://anchor.fm/marna-walle
>
> Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA
>
> From: "René Jansen" rene.vincent.jan...@gmail.com
>
> To: IBM-MAIN@LISTSERV.UA.EDU
>
> Date: 29/09/2021 09:12
>
> Subject: [EXTERNAL] Re: [IBM External] The Business Case for
>
> BatchPipes in the z/OS Base (was: ... Pipes ...)
>
> Sent by: "IBM Mainframe Discussion List" IBM-MAIN@LISTSERV.UA.EDU
>
> Martin,
>
> Do you have that book somewhere still? I spent 15 minutes on google but
>
> lots of links lead nowhere.
>
> NetRexx does have a rather complete CMS Pipelines implementation (
>
> www.netrexx.org <
>
> http://www.netrexx.org/
>
> > ) and I was wondering what I need to do to also have these ’halfpipes’
>
> and other batch interfaces, using JZOS or other.
>
> Best regards,
>
> René.
>
> > On 29 Sep 2021, at 08:54, Martin Packer martin_pac...@uk.ibm.com
>
> wrote:
>
> > Between steps can't be pipes, can be VIO. Between jobs can be pipes,
>
> can't
>
> > be VIO.
> >
> > That second sentence depends on the ability to schedule two jobs
>
> (possibly
>
> > originally steps of the same job) alongside each other.
> >
> > Fun stuff but / and somewhat complex - which is what inspired me to
>
> start
>
> > writing what would become SG24-2557 Parallel Sysplex Batch Performance
>
> in
>
> > late 1990. :-)
> >
> > I did a lot of presenting on Pipes to individual customers and
>
> conferences
>
> > in the 1990s. It would be fun to do it again... :-)
> >
> > Cheers, Martin
> >
> > Sent from my iPad
> >
> > > On 29 Sep 2021, at 05:23, Paul Gilmartin
> > >
> > > 000433f07816-dmarc-requ...@listserv.ua.edu wrote:
> > >
> > > On Mon, 27 Sep 2021 11:39:17 -0500, Hobart Spitz wrote:
> > >
> > > > Intra-JOB pipe - This would be similar to VIO; it doesn't exist
> > > >
> > > > otherwise.
> > >
> > > > I think it would be a great feature.
> > >
> > > Between steps? One might just as well use VIO.
> > >
> > > -- gil
> > >
> > > For IBM-MAIN subscribe / signoff / archive access instructions,
> > >
> > > send email to lists...@listserv.ua.edu with the message: INFO
>
> IBM-MAINUnless stated otherwise above:
>
> > IBM United Kingdom Limited - Registered in England and Wales with number
>
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>
> 3AU
>
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> >
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
>
> For IBM-MAIN subscribe / signoff / archive access instructions,
>
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> Unless stated otherwise above:
>
> IBM United Kingdom Limited - Registered in England and Wales with number
>
> 741598.
>
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> 
>
> For IBM-MAIN subscribe / signoff / archive access instructions,
>
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: [IBM External] The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[Martin Packer]

I think it's gone. Does anyone still have a PDF of it?

Thanks, Martin

Martin Packer

WW z/OS Performance, Capacity and Architecture, IBM Technology Sales

+44-7802-245-584

email: martin_pac...@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog: https://mainframeperformancetopics.com

Mainframe, Performance, Topics Podcast Series (With Marna Walle): 
https://anchor.fm/marna-walle

Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From:   "René Jansen" 
To: IBM-MAIN@LISTSERV.UA.EDU
Date:   29/09/2021 09:12
Subject:[EXTERNAL] Re: [IBM External] The Business Case for 
BatchPipes in the z/OS Base (was: ... Pipes ...)
Sent by:"IBM Mainframe Discussion List" 



Martin,

Do you have that book somewhere still? I spent 15 minutes on google but 
lots of links lead nowhere.

NetRexx does have a rather complete CMS Pipelines implementation (
www.netrexx.org <
http://www.netrexx.org/ 
>) and I was wondering what I need to do to also have these ’halfpipes’ 
and other batch interfaces, using JZOS or other.

Best regards,

René. 


> On 29 Sep 2021, at 08:54, Martin Packer  
wrote:
> 
> 
> 
> Between steps can't be pipes, can be VIO. Between jobs can be pipes, 
can't
> be VIO.
> 
> That second sentence depends on the ability to schedule two jobs 
(possibly
> originally steps of the same job) alongside each other.
> 
> Fun stuff but / and somewhat complex - which is what inspired me to 
start
> writing what would become SG24-2557 Parallel Sysplex Batch Performance 
in
> late 1990. :-)
> 
> I did a lot of presenting on Pipes to individual customers and 
conferences
> in the 1990s. It would be fun to do it again... :-)
> 
> Cheers, Martin
> 
> Sent from my iPad
> 
>> On 29 Sep 2021, at 05:23, Paul Gilmartin
> <000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
>> 
>> On Mon, 27 Sep 2021 11:39:17 -0500, Hobart Spitz wrote:
>>> 
>>> Intra-JOB pipe - This would be similar to VIO; it doesn't exist
> otherwise.
>>> I think it would be a great feature.
>>> 
>> Between steps?  One might just as well use VIO.
>> 
>> -- gil
>> 
>> --
>> For IBM-MAIN subscribe / signoff / archive access instructions,
>> send email to lists...@listserv.ua.edu with the message: INFO 
IBM-MAINUnless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 
3AU
> 
> 
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: [IBM External] The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[René Jansen]

Martin,

Do you have that book somewhere still? I spent 15 minutes on google but lots of 
links lead nowhere.

NetRexx does have a rather complete CMS Pipelines implementation 
(www.netrexx.org <http://www.netrexx.org/>) and I was wondering what I need to 
do to also have these ’halfpipes’ and other batch interfaces, using JZOS or 
other.

Best regards,

René. 


> On 29 Sep 2021, at 08:54, Martin Packer  wrote:
> 
> 
> 
> Between steps can't be pipes, can be VIO. Between jobs can be pipes, can't
> be VIO.
> 
> That second sentence depends on the ability to schedule two jobs (possibly
> originally steps of the same job) alongside each other.
> 
> Fun stuff but / and somewhat complex - which is what inspired me to start
> writing what would become SG24-2557 Parallel Sysplex Batch Performance in
> late 1990. :-)
> 
> I did a lot of presenting on Pipes to individual customers and conferences
> in the 1990s. It would be fun to do it again... :-)
> 
> Cheers, Martin
> 
> Sent from my iPad
> 
>> On 29 Sep 2021, at 05:23, Paul Gilmartin
> <000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
>> 
>> On Mon, 27 Sep 2021 11:39:17 -0500, Hobart Spitz wrote:
>>> 
>>> Intra-JOB pipe - This would be similar to VIO; it doesn't exist
> otherwise.
>>> I think it would be a great feature.
>>> 
>> Between steps?  One might just as well use VIO.
>> 
>> -- gil
>> 
>> --
>> For IBM-MAIN subscribe / signoff / archive access instructions,
>> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAINUnless 
>> stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number 
> 741598. 
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
> 
> 
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: [IBM External] Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[Martin Packer]

Between steps can't be pipes, can be VIO. Between jobs can be pipes, can't
be VIO.

That second sentence depends on the ability to schedule two jobs (possibly
originally steps of the same job) alongside each other.

Fun stuff but / and somewhat complex - which is what inspired me to start
writing what would become SG24-2557 Parallel Sysplex Batch Performance in
late 1990. :-)

I did a lot of presenting on Pipes to individual customers and conferences
in the 1990s. It would be fun to do it again... :-)

Cheers, Martin

Sent from my iPad

> On 29 Sep 2021, at 05:23, Paul Gilmartin
<000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
>
> On Mon, 27 Sep 2021 11:39:17 -0500, Hobart Spitz wrote:
>>
>> Intra-JOB pipe - This would be similar to VIO; it doesn't exist
otherwise.
>> I think it would be a great feature.
>>
> Between steps?  One might just as well use VIO.
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAINUnless 
> stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[Paul Gilmartin]

On Mon, 27 Sep 2021 11:39:17 -0500, Hobart Spitz wrote:
>
>Intra-JOB pipe - This would be similar to VIO; it doesn't exist otherwise.
>I think it would be a great feature. 
>
Between steps?  One might just as well use VIO.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[David Crayford]

Thanks for the clarity. So BatchPipes are the plumbing to implement 
streaming almost like the JCL equivalent of the Apache Kafka Streams API?


On 28/09/2021 4:00 pm, Martin Packer wrote:

I think you have to remember that BatchPipes/MVS' origin story is
connecting existing batch job steps. I wouldn't want customers to have to
re-write e.g. in java.

One advantage is that it's easier to rework a pipeline - whether as a
fitting or between batch jobs - than rework some java code.

As someone who first proselytised Pipes in 1992 and first wrote about it
in a Redbook in 1997 (and wrote about it again in 2011 and 2013) you can
consider me a fan. Note: I don't have an enormous amount of influence on
those that make decisions about either IBM flavour of pipes. I do
recognise it's not as simple as "set the code free". There is the cost of
bringing it to market and supporting it. The latter in particular would be
much more costly than it's ever been before - if it were built in.

So, I hope we can make both Pipelines and BatchPipes/MVS available as part
of z/OS. I'd love to be talking about them again and supporting customers
using them. But I'm just a field guy.

Cheers, Martin

Martin Packer

WW z/OS Performance, Capacity and Architecture, IBM Technology Sales

+44-7802-245-584

email: martin_pac...@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog: https://mainframeperformancetopics.com

Mainframe, Performance, Topics Podcast Series (With Marna Walle):
https://anchor.fm/marna-walle

Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From:   "David Crayford" 
To: IBM-MAIN@LISTSERV.UA.EDU
Date:   28/09/2021 08:17
Subject:[EXTERNAL] Re: The Business Case for BatchPipes in the
z/OS Base (was: ... Pipes ...)
Sent by:"IBM Mainframe Discussion List" 



This first question I would ask is does IBM actually "own" BatchPipes or
did the flog it off to an ISV like most of their other software? If it's
the latter then they will be in no position to make it freely available.

Secondly, rather than pine for something that isn't available why not
just switch technologies and use a language that supports functional
programming features that are similar to pipes. Even Java has supported
functional programming since Java 8 came with streams in 2014.

https://stackify.com/streams-guide-java-8/



On 28/09/2021 12:39 am, Hobart Spitz wrote:

Gil wrote:

On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote:

I'm going to pivot here.  I'm putting my support behind putting

BatchPipes

in the z/OS base (rather than just Pipes).  If you agree, please
write/support such a requirement and/or educate your management to get
interested.  BatchPipes includes BatchPipesWorks, a not so current,

but

still highly useful, version of TSO Pipelines.


Is an RFE for an update required?  Conway's Law.

Great question.  I guess it would depend on whether the were a lot of
current BatchPipes customers needed the missing builtin stages and/o
fix(es) or whether it was more important to get BatchPipes in the z/OS
base.  It might also depend on how much additional work would be

required

to get a more current version of Pipes into BatchPipes, and whether the
staff, skills, and funding were available.  IMHO, the update would delay
the benefits of BatchPipes in the base (especially global warming
mitigation), and not be important to current BatchPipes customers.  In

my

estimation, BatchPipesWorks has 90% of the needed stages and 99% of the
fix(es) as would be in an up to date version of TSO Pipelines.  I find

the

IBM decision making process obscure, so these may not be the

considerations

that affect the final decision.

AFAIK, there is only one person now working on Pipes, so I must be

missing

something in applying Conway's law.


Does BatchPipes support connecting two Classic modules with an

intervening

small Pipeline filter?  How?  Is a coordinated third job needed?

Let me clarify some terms, answer you questions in the process, and

clarify

my previous post:


Inter-JOB pipe - This is a ppe that let's two JOB pass records from one

to

the next thru memory.  The BatchPipes subsystem(s) is/are required.
AFAIK:  There are no obvious limitations to the topology.  You can have
multiple JOB connected in a single "stream", split and/or join streams,

or

even have loop(s).  In this respect, it is similar to the PIPE command,
except that the record flow with split and rejoined streams may not be
predictable between JOBs.

Half Pipe Fitting or Intra-STEP pipe - This is a series of Pipes stages
that operate between a program in a step and the storage medium.

Intra-JOB pipe - This would be similar to VIO; it doesn't exist

otherwise.

I think it would be a great feature.  Hence I incorrectly used the term

to

refer to Half Pipe Fittings in a generic sense.  Apologies for any
confusion.


AFAIK, and if I understood your question, connecting two

Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[Martin Packer]

I think you have to remember that BatchPipes/MVS' origin story is 
connecting existing batch job steps. I wouldn't want customers to have to 
re-write e.g. in java.

One advantage is that it's easier to rework a pipeline - whether as a 
fitting or between batch jobs - than rework some java code.

As someone who first proselytised Pipes in 1992 and first wrote about it 
in a Redbook in 1997 (and wrote about it again in 2011 and 2013) you can 
consider me a fan. Note: I don't have an enormous amount of influence on 
those that make decisions about either IBM flavour of pipes. I do 
recognise it's not as simple as "set the code free". There is the cost of 
bringing it to market and supporting it. The latter in particular would be 
much more costly than it's ever been before - if it were built in.

So, I hope we can make both Pipelines and BatchPipes/MVS available as part 
of z/OS. I'd love to be talking about them again and supporting customers 
using them. But I'm just a field guy.

Cheers, Martin

Martin Packer

WW z/OS Performance, Capacity and Architecture, IBM Technology Sales

+44-7802-245-584

email: martin_pac...@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog: https://mainframeperformancetopics.com

Mainframe, Performance, Topics Podcast Series (With Marna Walle): 
https://anchor.fm/marna-walle

Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From:   "David Crayford" 
To: IBM-MAIN@LISTSERV.UA.EDU
Date:   28/09/2021 08:17
Subject:[EXTERNAL] Re: The Business Case for BatchPipes in the 
z/OS Base (was: ... Pipes ...)
Sent by:"IBM Mainframe Discussion List" 



This first question I would ask is does IBM actually "own" BatchPipes or 
did the flog it off to an ISV like most of their other software? If it's 
the latter then they will be in no position to make it freely available.

Secondly, rather than pine for something that isn't available why not 
just switch technologies and use a language that supports functional 
programming features that are similar to pipes. Even Java has supported 
functional programming since Java 8 came with streams in 2014.

https://stackify.com/streams-guide-java-8/ 



On 28/09/2021 12:39 am, Hobart Spitz wrote:
> Gil wrote:
>> On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote:
>>> I'm going to pivot here.  I'm putting my support behind putting 
BatchPipes
>>> in the z/OS base (rather than just Pipes).  If you agree, please
>>> write/support such a requirement and/or educate your management to get
>>> interested.  BatchPipes includes BatchPipesWorks, a not so current, 
but
>>> still highly useful, version of TSO Pipelines.
>>>
>> Is an RFE for an update required?  Conway's Law.
> Great question.  I guess it would depend on whether the were a lot of
> current BatchPipes customers needed the missing builtin stages and/o
> fix(es) or whether it was more important to get BatchPipes in the z/OS
> base.  It might also depend on how much additional work would be 
required
> to get a more current version of Pipes into BatchPipes, and whether the
> staff, skills, and funding were available.  IMHO, the update would delay
> the benefits of BatchPipes in the base (especially global warming
> mitigation), and not be important to current BatchPipes customers.  In 
my
> estimation, BatchPipesWorks has 90% of the needed stages and 99% of the
> fix(es) as would be in an up to date version of TSO Pipelines.  I find 
the
> IBM decision making process obscure, so these may not be the 
considerations
> that affect the final decision.
>
> AFAIK, there is only one person now working on Pipes, so I must be 
missing
> something in applying Conway's law.
>
>> Does BatchPipes support connecting two Classic modules with an 
intervening
>> small Pipeline filter?  How?  Is a coordinated third job needed?
> Let me clarify some terms, answer you questions in the process, and 
clarify
> my previous post:
>
>
> Inter-JOB pipe - This is a ppe that let's two JOB pass records from one 
to
> the next thru memory.  The BatchPipes subsystem(s) is/are required.
> AFAIK:  There are no obvious limitations to the topology.  You can have
> multiple JOB connected in a single "stream", split and/or join streams, 
or
> even have loop(s).  In this respect, it is similar to the PIPE command,
> except that the record flow with split and rejoined streams may not be
> predictable between JOBs.
>
> Half Pipe Fitting or Intra-STEP pipe - This is a series of Pipes stages
> that operate between a program in a step and the storage medium.
>
> Intra-JOB pipe - This would be similar to VIO; it doesn't exist 
otherwise.
> I think it would be a great feature.  Hence I incorrectly used the term 
to
> refer to Half Pipe Fittings in a generic sens

Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[David Crayford]

This first question I would ask is does IBM actually "own" BatchPipes or 
did the flog it off to an ISV like most of their other software? If it's 
the latter then they will be in no position to make it freely available.


Secondly, rather than pine for something that isn't available why not 
just switch technologies and use a language that supports functional 
programming features that are similar to pipes. Even Java has supported 
functional programming since Java 8 came with streams in 2014.


https://stackify.com/streams-guide-java-8/


On 28/09/2021 12:39 am, Hobart Spitz wrote:

Gil wrote:

On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote:

I'm going to pivot here.  I'm putting my support behind putting BatchPipes
in the z/OS base (rather than just Pipes).  If you agree, please
write/support such a requirement and/or educate your management to get
interested.  BatchPipes includes BatchPipesWorks, a not so current, but
still highly useful, version of TSO Pipelines.


Is an RFE for an update required?  Conway's Law.

Great question.  I guess it would depend on whether the were a lot of
current BatchPipes customers needed the missing builtin stages and/o
fix(es) or whether it was more important to get BatchPipes in the z/OS
base.  It might also depend on how much additional work would be required
to get a more current version of Pipes into BatchPipes, and whether the
staff, skills, and funding were available.  IMHO, the update would delay
the benefits of BatchPipes in the base (especially global warming
mitigation), and not be important to current BatchPipes customers.  In my
estimation, BatchPipesWorks has 90% of the needed stages and 99% of the
fix(es) as would be in an up to date version of TSO Pipelines.  I find the
IBM decision making process obscure, so these may not be the considerations
that affect the final decision.

AFAIK, there is only one person now working on Pipes, so I must be missing
something in applying Conway's law.


Does BatchPipes support connecting two Classic modules with an intervening
small Pipeline filter?  How?  Is a coordinated third job needed?

Let me clarify some terms, answer you questions in the process, and clarify
my previous post:


Inter-JOB pipe - This is a ppe that let's two JOB pass records from one to
the next thru memory.  The BatchPipes subsystem(s) is/are required.
AFAIK:  There are no obvious limitations to the topology.  You can have
multiple JOB connected in a single "stream", split and/or join streams, or
even have loop(s).  In this respect, it is similar to the PIPE command,
except that the record flow with split and rejoined streams may not be
predictable between JOBs.

Half Pipe Fitting or Intra-STEP pipe - This is a series of Pipes stages
that operate between a program in a step and the storage medium.

Intra-JOB pipe - This would be similar to VIO; it doesn't exist otherwise.
I think it would be a great feature.  Hence I incorrectly used the term to
refer to Half Pipe Fittings in a generic sense.  Apologies for any
confusion.


AFAIK, and if I understood your question, connecting two classic programs
in the same step with a series of Pipes stages may not be  possible now, at
least without resorting to Assembler.  I think the main reason is the
potential for DD name conflict, both in the TIOT (DD name table) and in
JCL.  Most COBOL programs require SYSOUT, so the output could be
intermixed.  Much less tolerable, I suspect, would be a common DD (e.g.
OUTPUT) conflict.  Putting a record that doesn't belong between two records
that must be consecutive would probably be a disaster.

That said, it may be possible for a REXX filter to start two COBOL (e.g.)
programs as subtasks (address attach ... , etc.) and feed and receive their
records via an Assembler(?) Pipe Fitting.  You would have to resolve any DD
name conflicts.  I don't think anyone has contemplated the capability for
creating multiple TIOTs in a single step, let alone how to reference them
in JCL.  I think such things have been done under CMS using Assembler.

Also:

pipe - a generic connection, in memory, between two processes or tasks.

Pipes - the TSO/CMS Pipelines software, singular.

Pipe - a specific instance of Pipes.

PIPE - the command that runs Pipes.


OREXXMan
Would you rather pass data in move mode (*nix piping) or locate mode
(Pipes) or via disk (JCL)?  Why do you think you rarely see *nix commands
with more than a dozen filters, while Pipelines specifications are commonly
over 100s of stages, and 1000s of stages are not uncommon.
IBM has been looking for an HLL for program products; REXX is that language.


On Mon, Sep 27, 2021 at 9:10 AM Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:


On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote:


I'm going to pivot here.  I'm putting my support behind putting BatchPipes
in the z/OS base (rather than just Pipes).  If you agree, please
write/support such a requirement and/

Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[Hobart Spitz]

Gil wrote:
>On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote:
>>I'm going to pivot here.  I'm putting my support behind putting BatchPipes
>>in the z/OS base (rather than just Pipes).  If you agree, please
>>write/support such a requirement and/or educate your management to get
>>interested.  BatchPipes includes BatchPipesWorks, a not so current, but
>>still highly useful, version of TSO Pipelines.
>>
>Is an RFE for an update required?  Conway's Law.

Great question.  I guess it would depend on whether the were a lot of
current BatchPipes customers needed the missing builtin stages and/o
fix(es) or whether it was more important to get BatchPipes in the z/OS
base.  It might also depend on how much additional work would be required
to get a more current version of Pipes into BatchPipes, and whether the
staff, skills, and funding were available.  IMHO, the update would delay
the benefits of BatchPipes in the base (especially global warming
mitigation), and not be important to current BatchPipes customers.  In my
estimation, BatchPipesWorks has 90% of the needed stages and 99% of the
fix(es) as would be in an up to date version of TSO Pipelines.  I find the
IBM decision making process obscure, so these may not be the considerations
that affect the final decision.

AFAIK, there is only one person now working on Pipes, so I must be missing
something in applying Conway's law.

> Does BatchPipes support connecting two Classic modules with an intervening
> small Pipeline filter?  How?  Is a coordinated third job needed?

Let me clarify some terms, answer you questions in the process, and clarify
my previous post:


Inter-JOB pipe - This is a ppe that let's two JOB pass records from one to
the next thru memory.  The BatchPipes subsystem(s) is/are required.
AFAIK:  There are no obvious limitations to the topology.  You can have
multiple JOB connected in a single "stream", split and/or join streams, or
even have loop(s).  In this respect, it is similar to the PIPE command,
except that the record flow with split and rejoined streams may not be
predictable between JOBs.

Half Pipe Fitting or Intra-STEP pipe - This is a series of Pipes stages
that operate between a program in a step and the storage medium.

Intra-JOB pipe - This would be similar to VIO; it doesn't exist otherwise.
I think it would be a great feature.  Hence I incorrectly used the term to
refer to Half Pipe Fittings in a generic sense.  Apologies for any
confusion.


AFAIK, and if I understood your question, connecting two classic programs
in the same step with a series of Pipes stages may not be  possible now, at
least without resorting to Assembler.  I think the main reason is the
potential for DD name conflict, both in the TIOT (DD name table) and in
JCL.  Most COBOL programs require SYSOUT, so the output could be
intermixed.  Much less tolerable, I suspect, would be a common DD (e.g.
OUTPUT) conflict.  Putting a record that doesn't belong between two records
that must be consecutive would probably be a disaster.

That said, it may be possible for a REXX filter to start two COBOL (e.g.)
programs as subtasks (address attach ... , etc.) and feed and receive their
records via an Assembler(?) Pipe Fitting.  You would have to resolve any DD
name conflicts.  I don't think anyone has contemplated the capability for
creating multiple TIOTs in a single step, let alone how to reference them
in JCL.  I think such things have been done under CMS using Assembler.

Also:

pipe - a generic connection, in memory, between two processes or tasks.

Pipes - the TSO/CMS Pipelines software, singular.

Pipe - a specific instance of Pipes.

PIPE - the command that runs Pipes.


OREXXMan
Would you rather pass data in move mode (*nix piping) or locate mode
(Pipes) or via disk (JCL)?  Why do you think you rarely see *nix commands
with more than a dozen filters, while Pipelines specifications are commonly
over 100s of stages, and 1000s of stages are not uncommon.
IBM has been looking for an HLL for program products; REXX is that language.


On Mon, Sep 27, 2021 at 9:10 AM Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:

> On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote:
>
> >I'm going to pivot here.  I'm putting my support behind putting BatchPipes
> >in the z/OS base (rather than just Pipes).  If you agree, please
> >write/support such a requirement and/or educate your management to get
> >interested.  BatchPipes includes BatchPipesWorks, a not so current, but
> >still highly useful, version of TSO Pipelines.
> >
> Is an RFE for an update rrequired?  Conway's Law.
>
> Does BatchPipes support connecting two Classic modules with an intervening
> small Pipeline filter?  How?  Is a coordinated third job needed?
>
> >The reasons are:  [Snip!  See the archive.]
>
> -- gil
>
> ---

Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[Paul Gilmartin]

On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote:

>I'm going to pivot here.  I'm putting my support behind putting BatchPipes
>in the z/OS base (rather than just Pipes).  If you agree, please
>write/support such a requirement and/or educate your management to get
>interested.  BatchPipes includes BatchPipesWorks, a not so current, but
>still highly useful, version of TSO Pipelines.
>
Is an RFE for an update rrequired?  Conway's Law.

Does BatchPipes support connecting two Classic modules with an intervening
small Pipeline filter?  How?  Is a coordinated third job needed?

>The reasons are:  [Snip!  See the archive.]

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)

[Hobart Spitz]

I'm going to pivot here.  I'm putting my support behind putting BatchPipes
in the z/OS base (rather than just Pipes).  If you agree, please
write/support such a requirement and/or educate your management to get
interested.  BatchPipes includes BatchPipesWorks, a not so current, but
still highly useful, version of TSO Pipelines.

The reasons are:

   1. The audience that could benefit from BatchPipes in the base is much
   larger than that of TSO Pipes in the base.  The former includes
   organizations that:
  - Have a limited batch window or want to reduce JOB run times.
  - Are constrained on developer resources.
  - Want to reduce hardware requirements.
  - Require quick resolution to production problems.
  - Have a policy to reduce their carbon footprint and/or their
  contribution to the climate crisis.
   2. It is unrealistic to expect wide buy-in for TSO Pipes since most z/OS
   sites are highly dependent on JCL.
   3. BatchPipes in the z/OS base would be the biggest enhancement to JCL
   since it (JCL) came into existence.
   4. BatchPipes in the base would improve the competitiveness of z/OS.
   5. IBM, vendors, and customers would benefit from the capability by
   being able to write quicker running, more general, less complicated JCL,
   knowing that BatchPipes would be available on all target systems.
   6. All parties could potentially benefit from having the PIPE command
   available on all target systems.
   7. BatchPipes fittings could enhance and extend existing programs and
   utilities with a consistent, intuitive, and uncomplicated control
   language.  Too many utilities have unique, inconsistent, and/or complex
   control languages.
   8. Contribute to global warming mitigation by reducing electricity usage
   due to processors, storage hardware and cooling.  This is more significant
   with BatchPipes as there is likely to be a larger impact in a shorter
   amount of time than with just Pipes alone.  The latter would be adopted
   more slowly and less broadly.  Climate crisis mitigation efforts may be
   exempt from the rumored requirement that IBM is legally barred from
   offering software at a loss.  (IMHO,  the huge number of competitors to IBM
   in today's market, suggests that any such requirement is obsolete and
   should be removed.)
   9. There would be some portability between z/VM and z/OS.
   10. Despite wide support, Pipes requirements have not budged for
   decades  .

Anyone so inclined, is welcome to submit such a requirement, adapt my Pipes
requirement, and/or work with me on a new requirement.  Of course, if you
agree, vote for the requirement.  Can the current Pipes requirement be
construed to support BatchPipes?  Is it too much to expect BatchPipes to be
added to the z/OS base without the delay of waiting for requirement(s),
voting, and acceptance.

Alternatively, BatchPipes and the REXX Compiler, possibly along with other
software, could be packaged together as a combined product, perhaps with a
performance or global warming mitigation theme .  Another possibility would
be to offer BatchPipes Light, where the intra-JOB piping was offered free
or in the base.  Half pipe fitting might be a loss-leader to attract
customers to the full product.

As a developer, I love REXX and I love Pipes.  Together, they are
programmer heaven, more so than OO.  Whenever I write something new,
whether in z/OS or z/VM, I write a REXX EXEC.  Under z/OS, I SPAWN the EXEC
when I want to run it in batch.

I have used BatchPipes only briefly, but I know Pipes fairly well.  Pipes
are the language of BatchPipes fittings, which essentially is an
enhancement that allows Pipes stages to run on most DDs between the program
and the storage media.

I don't like BatchPipes.  They require opsys or sysprog assistance and are
too much trouble to use.  They support inter-JOB piping which requires a
subsystem and makes the real valuable part (Half Pipe Fittings) complicated
and hard to use.  Since I almost never use JCL, I have no use for them.  I
can, however, see their use and, as listed above, their value to the z/OS
community as a whole.

That said, for the rest of this note, I will PRETEND to be someone with
production support responsibilities.  If some of these things have no
connection to anything that any site would actually do, I apologize.  I'm
just a color-blind painter trying to paint a picture that will give you the
idea.


In an imaginary future, some production control person may write:

I love BatchPipes!  Since we started using it, our batch production run
time has steadily dropped and is now just 20% of what it was.  We were able
to shorten the run-times of the longest running JOBs to a small fraction.
The DBAs love that they can start their maintenance and backups earlier and
with plenty of time to spare.

Much of the time, instead of writing a new

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Hobart Spitz]

Martin wote:
>I'm not familiar with FANOUT but if it writes a record to, say, two
destinations, it's got to copy one of them.

Actually not.  I know it sounds like magic, but there is still only one
copy.  I'll give you my best rendition of what I think is happening.  Don't
worry if you don't get the whole picture.  I had to work with Pipes for
many years before I understood this.

Pipes manages the storage for records.  When a new record is created (or
modified) Pipes allocates the space for it.  That pointer is what a stage
gets when it asks for an input record.  When an input record is no longer
needed by a stage, Pipes is informed.  This is referred to as "consuming"
the record.  In fact, the record is not "consumed", as in grinding with
teeth and digesting, rather it is how a stage tells Pipes that it's done
with the record.

Changes are made to new storage at a new record address.  Changing a record
means getting new storage and building a new record, with the changes.  The
input record is not changed, and it can be inspected by other streams via
FANOUT or any multiwrite stage.  When the storage area of a record is no
longer used by any stage, i.e., no stage has a pointer to it, that storage
can be reclaimed by Pipes.

I don't know the exact mechanism, but that's it at a high level.  No matter
how many streams and stages use a record, there is only one copy. DUP, for
example, could just write the record over and over using the same pointer.
(I don't know that for a fact, but John Hartmann is a lot smarter than I.)

The majority of Pipes building stages do not modify their input records.
(With REXX user written stages, there always is a new copy on output, even
if no changes were made to the data.)


OREXXMan
Would you rather pass data in move mode (*nix piping) or locate mode
(Pipes) or via disk (JCL)?  Why do you think you rarely see *nix commands
with more than a dozen filters, while Pipelines specifications are commonly
over 100s of stages, and 1000s of stages are not uncommon.
IBM has been looking for an HLL for program products; REXX is that language.


On Thu, Sep 23, 2021 at 9:44 AM Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:

> On Thu, 23 Sep 2021 09:30:43 +0100, Martin Packer wrote:
>
> >I'm not familiar with FANOUT but if it writes a record to, say, two
> >destinations, it's got to copy one of them.
> >
> It could be deferred; Copy-on-Write, optimizing for what Hobart earlier
> calledd the "typical case" of stages that don't modify the data.
> But incurring the complexity of a responsibility count.
>
>
> >From:   "Hobart Spitz"
> >Date:   23/09/2021 04:18
> >
> >>  I'm guessing the atypical case is a stage such as FANOUT which
> >> necessarily copies the data.
> >
> >Not sure what you mean by atypical.
> >
> I apologize; I trimmed your earlier mention of "typical case".
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Fri, 24 Sep 2021 08:01:27 -0500, Hobart Spitz wrote:

>Mike wrote:
>>Could something be developed similar to a SORTOUT exit that implements
>this switch?
>BatchPipes fittings are like sort exits on steroids:  They can be applied
>to almost any DD, ...
>
SORTWKnn?

A problem posed earlier was to estimate the size of the sort.
I''m imagining a FANOUT to two pipelines:
o One | COUNT RECORDS to measure the size
o One | ELASTIC | DD:SORTIN

A critique of this idea is left as an exercise for the student.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Hobart Spitz]

Phil, thanks again for the helpful feedback and the great food for thought.

Your point is well taken.

I'll try to break it down as best I can:  MIPS or CPU usage, main memory or
working set, and input/output operations.  (I'd be happy if someone else
has better information.)

MIPS is dependent on at least three things:  Arithmetic Logical Unit (ALU)
usage, instruction pipeline activity, and cache misses.  With Pipes, ALU
and instruction pipeline usage would be more intense, while cache misses
would be less so.  Many characters, ones that the Pipes stages never even
look at, won't even enter the cache and ALU.  Some data can be processed
through thousands of stages and only be cache missed a few times, because
stages are typically dispatched one after the other to process the same
record.  Compared to UNIX piping, where every character of a text file must
go through every stage, even if it is going to be ignored, Pipes is much
faster.  As Melinda Varian says (approximately):  This is an advantage of
file structure aware operating systems.  I could include Pipes as
systems software.

Main memory (i.e., non-cache) and working set affect how fast instructions
can be executed.  As with characters, records can enter and leave the
working set just a few times, and still be processed by thousands of
stages.  So the working sets of Pipes will mostly tend to be much lower.
Stages like TAKE and DROP can do their work without ever referencing a
single byte of the records being processed.  (With UNIX head and tail,
every character must be inspected by the CPU.)

Finally, there is I/O, which is where Pipes and even UNIX piping really
shine.   The fastest input and output operations are the ones that never
take place.  Records moved from program to program in memory skip I/O.  I/O
generally involves some kind of mechanical movement, and which is far
slower than the CPU, caches or real memory.

Taken together, a set of Pipes stages, whether in a TSO PIPE command or in
a BatchPipes Fitting on a DD, \will have a smaller average working set and
spend less time in the system than alternatives.  If we estimate costs as
WS*ER (working set size times elapsed residency), even a conservative 50%
reduction in each would mean about a 75% reduction in hardware usage.
(WS*0.5)*(ER*0.5) = WS*ER*0.25.

If you have been paying attention, I have purposely glossed over some
details.  Unreferenced characters may be cache loaded because they reside
on the same cache line as referenced characters.  Also, record length and
blocking factor, among other things, affect how fast data can reach main
memory, the caches, and then the ALU.  On the average, I don't think these
two details change the relative effects of Pipes processing versus
conventional processing..

I suspect that UNIX piping was the inspiration for Pipes (A.K.A. CMS/TSO
Pipelines and BatchPipesWorks) both in suggesting in-memory "I/O" as well
as improvements to the concept.

I hope this helps.

OREXXMan
Would you rather pass data in move mode (*nix piping) or locate mode
(Pipes) or via disk (JCL)?  Why do you think you rarely see *nix commands
with more than a dozen filters, while Pipelines specifications are commonly
over 100s of stages, and 1000s of stages are not uncommon.
IBM has been looking for an HLL for program products; REXX is that language.


On Mon, Sep 20, 2021 at 8:10 PM Phil Smith III  wrote:

> Hobart Spitz wrote, in part:
>
> >This is a great comment.  I hadn't given that much thought to the
> question.
>
> >Not to split hairs, but I didn't say MIPS, I said hardware.
>
> >If I had to guess, MIPS usage might actually increase slightly, because
> the
> >Pipes dispatcher has to switch between stages twice for every record that
> >is passed.
>
>
>
> Sure, just sayin' you'd want to be very clear about what you do mean.
>
>
>
> I'm not quite sure what you mean by "more MIPS but less hardware", though?
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Hobart Spitz]

Mike wrote:
>Could something be developed similar to a SORTOUT exit that implements
this switch?
BatchPipes fittings are like sort exits on steroids:  They can be applied
to almost any DD, can use most Pipes filters, do not require compiled code,
and they are not restricted to a single program.  I plan to say more
later.  In a nutshell:  I think BatchPipes, including the Pipe command,
should be in the z/OS base.  It would be the biggest enhancement to JCL
ever, and would be of much more interest to production oriented management
who care about stability.


OREXXMan
Would you rather pass data in move mode (*nix piping) or locate mode
(Pipes) or via disk (JCL)?  Why do you think you rarely see *nix commands
with more than a dozen filters, while Pipelines specifications are commonly
over 100s of stages, and 1000s of stages are not uncommon.
IBM has been looking for an HLL for program products; REXX is that language.


On Mon, Sep 20, 2021 at 7:11 PM Mike Schwab  wrote:

> So, in a sense, instead of pipes, the programs could be modified so
> that instead of outputting a record, call the next program passing the
> record as input.
>
> Could something be developed similar to a SORTOUT exit that implements
> this switch?
>
> On Mon, Sep 20, 2021 at 4:27 PM Hobart Spitz  wrote:
> >
> > Phil;
> >
> > This is a great comment.  I hadn't given that much thought to the
> question.
> >
> > Not to split hairs, but I didn't say MIPS, I said hardware.
> >
> > If I had to guess, MIPS usage might actually increase slightly, because
> the
> > Pipes dispatcher has to switch between stages twice for every record that
> > is passed.  Access method overheard would drop.  Buffered access methods,
> > in most cases, only have to increment the pointer into the block buffer,
> > check for end-of-buffer and return to the caller.  I don't know for sure
> > which is larger.  Maybe someone more knowledgeable than I can shed some
> > light.
> >
> > I would say the real savings would be in elapsed run time and working set
> > size.  Run time, due to eliminating something like 80-95% of I/O
> operations
> > for intra-JOB datasets.  Working set reduction would save on real memory.
> > (See below.)  Run time is probably more of a concern to customers,
> > especially those with tight batch windows.  That said, working set size
> > reduction would mean that processors would likely spend more, if not all,
> > time pegged at 100% busy, because so many more address spaces (TSO and
> JOB)
> > would be in a swapped-in and ready state than before.  Depending on what
> > metrics the capacity planners are looking at, CPU sales might actually
> > increase.  As I think about it more, if thru-put increases, new data
> could
> > be generated more quickly and other times of hardware could be more in
> > demand during peak load times.  I just don't know enough to say for sure.
> >
> > Phil and others know what follows.
> >
> > For those who don't know, in the typical case, a record passes through
> all
> > possible stages before the next record begins the same trip.  Each record
> > stays in the working page set, at least partially, during the entire
> time.
> > Parts that are referenced have a good chance of staying cache resident
> > between stages.
> >
> > Think of it this way:  You can visualize UNIX piping as a series of
> > hourglasses open at both ends and connected in a tower.  Each grain of
> sand
> > must stop at every "stage" and wait its turn to go through the narrow
> > opening at the waist of each hourglass.  In Pipes, most stages have no
> > delay and it's like a single tall hourglass tube with only one narrow
> > point.  My best guess is that Pipes, in this analogy, would have only
> > 5%-15% of the narrow openings as an equivalent UNIX piping command,
> meaning
> > that the data (sand) would flow 85-95% faster in the Pipes "hourglass".
> >
> >
> > OREXXMan
> > Would you rather pass data in move mode (*nix piping) or locate mode
> > (Pipes) or via disk (JCL)?  Why do you think you rarely see *nix commands
> > with more than a dozen filters, while Pipelines specifications are
> commonly
> > over 100s of stages, and 1000s of stages are not uncommon.
> > IBM has been looking for an HLL for program products; REXX is that
> language.
> >
> >
> > On Mon, Sep 20, 2021 at 12:48 PM Phil Smith III  wrote:
> >
> > > Hobart Spitz wrote, in part:
> > >
> > > >The case *for *Pipes in the z/OS base.:
> > >
> > > > 2. Hardware usage would drop for customers.
> > >
> > >
&

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Thu, 23 Sep 2021 09:30:43 +0100, Martin Packer wrote:

>I'm not familiar with FANOUT but if it writes a record to, say, two
>destinations, it's got to copy one of them.
> 
It could be deferred; Copy-on-Write, optimizing for what Hobart earlier
calledd the "typical case" of stages that don't modify the data.
But incurring the complexity of a responsibility count.


>From:   "Hobart Spitz"
>Date:   23/09/2021 04:18
>
>>  I'm guessing the atypical case is a stage such as FANOUT which
>> necessarily copies the data.
>
>Not sure what you mean by atypical.  
> 
I apologize; I trimmed your earlier mention of "typical case".

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Seymour J Metz]

Yes, checkpointing is increasingly important in a high volume workd, but it is 
also increasingly more difficult. There is an OS facility for restarting from a 
checkpoint, but it has significant restrictions and I wonder whether it has 
been used in the last half century.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Timothy Sipples [sipp...@sg.ibm.com]
Sent: Thursday, September 23, 2021 12:26 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: The Business Case for Pipes in the z/OS Base (was: Re: REXX - 
Interpret or Value - Which is better?)

I misplaced the original post, but somewhere in this thread someone
commented that checkpointing is less important. I think I disagree, so
just a quick comment from me.

Yes, absolutely, there's much more computing power and much better I/O.
There are also lots of efficiency gains -- much better compilers, for
example. However, if anything the data volumes and related requirements
are growing even faster. We've also seen recent, real world incidents
involving major organizations failing to meet batch processing deadlines
with serious consequences, in some cases to whole national economies. My
anecdotal observation is that checkpointing is becoming more important at
least on z/OS, not less. By sheer coincidence I'm having a technical
conversation this afternoon that (when you boil it down to its essence) is
"please implement a certain type of checkpointing."

I interpreted this particular remark as a side comment, not really
anything that genuinely affects whether pipes are useful in some cases.
Yes, pipes are useful. It's not necessary to bash checkpointing in defense
of pipes, or vice versa.

- - - - - - - - - -
Timothy Sipples
I.T. Architect Executive
Digital Asset & Other Industry Solutions
IBM Z & LinuxONE
- - - - - - - - - -
E-Mail: sipp...@sg.ibm.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: [IBM External] The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Martin Packer]

There are lots of things that breaks pipelines - and so recovery is a 
necessary thing to think about and build in.

I guess the difference between Pipes and pipes :-) is one of granularity. 
A (CMS-style) pipeline is probably one you're more willing to restart in 
its entirety.

Cheers, Martin

Martin Packer

WW z/OS Performance, Capacity and Architecture, IBM Technology Sales

+44-7802-245-584

email: martin_pac...@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog: https://mainframeperformancetopics.com

Mainframe, Performance, Topics Podcast Series (With Marna Walle): 
https://anchor.fm/marna-walle

Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From:   "Timothy Sipples" 
To: IBM-MAIN@LISTSERV.UA.EDU
Date:   23/09/2021 05:27
Subject:[IBM External] The Business Case for Pipes in the z/OS 
Base (was: Re: REXX - Interpret or Value - Which is better?)
Sent by:"IBM Mainframe Discussion List" 



I misplaced the original post, but somewhere in this thread someone 
commented that checkpointing is less important. I think I disagree, so 
just a quick comment from me.

Yes, absolutely, there's much more computing power and much better I/O. 
There are also lots of efficiency gains -- much better compilers, for 
example. However, if anything the data volumes and related requirements 
are growing even faster. We've also seen recent, real world incidents 
involving major organizations failing to meet batch processing deadlines 
with serious consequences, in some cases to whole national economies. My 
anecdotal observation is that checkpointing is becoming more important at 
least on z/OS, not less. By sheer coincidence I'm having a technical 
conversation this afternoon that (when you boil it down to its essence) is 

"please implement a certain type of checkpointing."

I interpreted this particular remark as a side comment, not really 
anything that genuinely affects whether pipes are useful in some cases. 
Yes, pipes are useful. It's not necessary to bash checkpointing in defense 

of pipes, or vice versa.

- - - - - - - - - -
Timothy Sipples
I.T. Architect Executive
Digital Asset & Other Industry Solutions
IBM Z & LinuxONE
- - - - - - - - - -
E-Mail: sipp...@sg.ibm.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Martin Packer]

I'm not familiar with FANOUT but if it writes a record to, say, two 
destinations, it's got to copy one of them.

Cheers, Martin

Martin Packer

WW z/OS Performance, Capacity and Architecture, IBM Technology Sales

+44-7802-245-584

email: martin_pac...@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog: https://mainframeperformancetopics.com

Mainframe, Performance, Topics Podcast Series (With Marna Walle): 
https://anchor.fm/marna-walle

Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From:   "Hobart Spitz" 
To: IBM-MAIN@LISTSERV.UA.EDU
Date:   23/09/2021 04:18
Subject:[EXTERNAL] Re: The Business Case for Pipes in the z/OS 
Base (was: Re: REXX - Interpret or Value - Which is better?)
Sent by:"IBM Mainframe Discussion List" 



Paul said:
>  I'm guessing the atypical case is a stage such as FANOUT which
necessarily
copies the data.

Not sure what you mean by atypical.  FANOUT is typical in the respect that
it doesn't create an actual copy of the input record.  It just looks like
it.  FANOUT, and non-record-changing stages, pass the same input record
pointer to their downstream stage(s).  This is what makes Pipes so
efficient.  No working set expansion and less reloading of just purged
cache data.



OREXXMan
Would you rather pass data in move mode (*nix piping) or locate mode
(Pipes) or via disk (JCL)?  Why do you think you rarely see *nix commands
with more than a dozen filters, while Pipelines specifications are 
commonly
over 100s of stages, and 1000s of stages are not uncommon.
IBM has been looking for an HLL for program products; REXX is that 
language.


On Wed, Sep 22, 2021 at 3:13 AM Martin Packer 
wrote:

> Conversely a pipe as input is not necessarily a good input medium for a
> sort. 10 years ago I contributed to a Batch Modernization Redbook on 
this,
> emphasising the need for BatchPipes input to DFSORT to be accompanied by 
a
> FILSZ / AVGRLEN estimate pair.
>
> Bringing it back to pipes, I wonder if it's feasible to tell a sorting
> stage (whether DFSORT (yes please Sri Hari) or otherwise) the input 
size.
> Otherwise we could have blow ups or bad performance at scale.
>
> BTW I'm all in favour of pipes as a first class citizen but note I have
> little influence in this regard.
>
> Cheers, Martin
>
> Martin Packer
>
> WW z/OS Performance, Capacity and Architecture, IBM Technology Sales
>
> +44-7802-245-584
>
> email: martin_pac...@uk.ibm.com
>
> Twitter / Facebook IDs: MartinPacker
>
> Blog: 
https://mainframeperformancetopics.com 

>
> Mainframe, Performance, Topics Podcast Series (With Marna Walle):
> 
https://anchor.fm/marna-walle 

>
> Youtube channel: 
https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA 

>
>
>
> From:   "Paul Gilmartin" 
<000433f07816-dmarc-requ...@listserv.ua.edu>
> To: IBM-MAIN@LISTSERV.UA.EDU
> Date:   22/09/2021 03:50
> Subject:[EXTERNAL] Re: The Business Case for Pipes in the z/OS
> Base (was: Re: REXX - Interpret or Value - Which is better?)
> Sent by:"IBM Mainframe Discussion List" 

>
>
>
> On Tue, 21 Sep 2021 21:03:14 -0500, Mike Schwab  wrote:
>
> >If a SORT (or other similar temporary data store) program is one of
> >the pipe programs, when the EXEC PGM= program closes the output file
> >then the program holding the data needs to output the stored data to
> >output ddnames (pipe or output files).
> >
> Are you thinking of MS-DOS pseudo-"pipes" where the upstream program
> wrote a temporary file under-the-covers and the downstream program
> processed it?  A pipe in syntax only.  Even Windows is better nowadays.
>
> SORT is a bad conceptual example for Pipethink because SORT can't
> write its first output record until it has read its last input record.
> Better
> to envision a filter which re-formats log records from a long-running 
(or
> never-terminating) program, writing a file to be browsed with SDSF or
> tail -f in real time.
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 
3AU
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subsc

The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Timothy Sipples]

I misplaced the original post, but somewhere in this thread someone 
commented that checkpointing is less important. I think I disagree, so 
just a quick comment from me.

Yes, absolutely, there's much more computing power and much better I/O. 
There are also lots of efficiency gains -- much better compilers, for 
example. However, if anything the data volumes and related requirements 
are growing even faster. We've also seen recent, real world incidents 
involving major organizations failing to meet batch processing deadlines 
with serious consequences, in some cases to whole national economies. My 
anecdotal observation is that checkpointing is becoming more important at 
least on z/OS, not less. By sheer coincidence I'm having a technical 
conversation this afternoon that (when you boil it down to its essence) is 
"please implement a certain type of checkpointing."

I interpreted this particular remark as a side comment, not really 
anything that genuinely affects whether pipes are useful in some cases. 
Yes, pipes are useful. It's not necessary to bash checkpointing in defense 
of pipes, or vice versa.

- - - - - - - - - -
Timothy Sipples
I.T. Architect Executive
Digital Asset & Other Industry Solutions
IBM Z & LinuxONE
- - - - - - - - - -
E-Mail: sipp...@sg.ibm.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Hobart Spitz]

Paul said:
>  I'm guessing the atypical case is a stage such as FANOUT which
necessarily
copies the data.

Not sure what you mean by atypical.  FANOUT is typical in the respect that
it doesn't create an actual copy of the input record.  It just looks like
it.  FANOUT, and non-record-changing stages, pass the same input record
pointer to their downstream stage(s).  This is what makes Pipes so
efficient.  No working set expansion and less reloading of just purged
cache data.



OREXXMan
Would you rather pass data in move mode (*nix piping) or locate mode
(Pipes) or via disk (JCL)?  Why do you think you rarely see *nix commands
with more than a dozen filters, while Pipelines specifications are commonly
over 100s of stages, and 1000s of stages are not uncommon.
IBM has been looking for an HLL for program products; REXX is that language.


On Wed, Sep 22, 2021 at 3:13 AM Martin Packer 
wrote:

> Conversely a pipe as input is not necessarily a good input medium for a
> sort. 10 years ago I contributed to a Batch Modernization Redbook on this,
> emphasising the need for BatchPipes input to DFSORT to be accompanied by a
> FILSZ / AVGRLEN estimate pair.
>
> Bringing it back to pipes, I wonder if it's feasible to tell a sorting
> stage (whether DFSORT (yes please Sri Hari) or otherwise) the input size.
> Otherwise we could have blow ups or bad performance at scale.
>
> BTW I'm all in favour of pipes as a first class citizen but note I have
> little influence in this regard.
>
> Cheers, Martin
>
> Martin Packer
>
> WW z/OS Performance, Capacity and Architecture, IBM Technology Sales
>
> +44-7802-245-584
>
> email: martin_pac...@uk.ibm.com
>
> Twitter / Facebook IDs: MartinPacker
>
> Blog: https://mainframeperformancetopics.com
>
> Mainframe, Performance, Topics Podcast Series (With Marna Walle):
> https://anchor.fm/marna-walle
>
> Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA
>
>
>
> From:   "Paul Gilmartin" <000433f07816-dmarc-requ...@listserv.ua.edu>
> To: IBM-MAIN@LISTSERV.UA.EDU
> Date:   22/09/2021 03:50
> Subject:[EXTERNAL] Re: The Business Case for Pipes in the z/OS
> Base (was: Re: REXX - Interpret or Value - Which is better?)
> Sent by:"IBM Mainframe Discussion List" 
>
>
>
> On Tue, 21 Sep 2021 21:03:14 -0500, Mike Schwab  wrote:
>
> >If a SORT (or other similar temporary data store) program is one of
> >the pipe programs, when the EXEC PGM= program closes the output file
> >then the program holding the data needs to output the stored data to
> >output ddnames (pipe or output files).
> >
> Are you thinking of MS-DOS pseudo-"pipes" where the upstream program
> wrote a temporary file under-the-covers and the downstream program
> processed it?  A pipe in syntax only.  Even Windows is better nowadays.
>
> SORT is a bad conceptual example for Pipethink because SORT can't
> write its first output record until it has read its last input record.
> Better
> to envision a filter which re-formats log records from a long-running (or
> never-terminating) program, writing a file to be browsed with SDSF or
> tail -f in real time.
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Martin Packer]

Conversely a pipe as input is not necessarily a good input medium for a 
sort. 10 years ago I contributed to a Batch Modernization Redbook on this, 
emphasising the need for BatchPipes input to DFSORT to be accompanied by a 
FILSZ / AVGRLEN estimate pair.

Bringing it back to pipes, I wonder if it's feasible to tell a sorting 
stage (whether DFSORT (yes please Sri Hari) or otherwise) the input size. 
Otherwise we could have blow ups or bad performance at scale.

BTW I'm all in favour of pipes as a first class citizen but note I have 
little influence in this regard.

Cheers, Martin

Martin Packer

WW z/OS Performance, Capacity and Architecture, IBM Technology Sales

+44-7802-245-584

email: martin_pac...@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog: https://mainframeperformancetopics.com

Mainframe, Performance, Topics Podcast Series (With Marna Walle): 
https://anchor.fm/marna-walle

Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From:   "Paul Gilmartin" <000433f07816-dmarc-requ...@listserv.ua.edu>
To: IBM-MAIN@LISTSERV.UA.EDU
Date:   22/09/2021 03:50
Subject:[EXTERNAL] Re: The Business Case for Pipes in the z/OS 
Base (was: Re: REXX - Interpret or Value - Which is better?)
Sent by:"IBM Mainframe Discussion List" 



On Tue, 21 Sep 2021 21:03:14 -0500, Mike Schwab  wrote:

>If a SORT (or other similar temporary data store) program is one of
>the pipe programs, when the EXEC PGM= program closes the output file
>then the program holding the data needs to output the stored data to
>output ddnames (pipe or output files).
>
Are you thinking of MS-DOS pseudo-"pipes" where the upstream program
wrote a temporary file under-the-covers and the downstream program
processed it?  A pipe in syntax only.  Even Windows is better nowadays.

SORT is a bad conceptual example for Pipethink because SORT can't
write its first output record until it has read its last input record. 
Better
to envision a filter which re-formats log records from a long-running (or
never-terminating) program, writing a file to be browsed with SDSF or
tail -f in real time.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Mon, 20 Sep 2021 16:26:59 -0500, Hobart Spitz wrote:
>...
>For those who don't know, in the typical case, a record passes through all
>possible stages before the next record begins the same trip.  Each record
>stays in the working page set, at least partially, during the entire time.
>Parts that are referenced have a good chance of staying cache resident
>between stages.
>
I'm guessing the atypical case is a stage such as FANOUT which necessarily
copies the data.

>...  My best guess is that Pipes, in this analogy, would have only
>5%-15% of the narrow openings as an equivalent UNIX piping command, meaning
>that the data (sand) would flow 85-95% faster in the Pipes "hourglass".
> ...
I suspect the cost of moving data is overwhelmed by the cost of process
switching.  And z/OS UNIX is probably the worst of all UNIXem because
its design hasn't been optimized for process switching.

(I wonder whether nowadays z/OS creates more address spaces for job
step initiation or for fork().  I'm confident that the design  remains
optimized for the former, regardless.)

But remember that nowadays silicon is ofteen cheaper than carbon.
With POSIX pipes I can:
407 $ ls | wc
  2   2  17
(got it right the first time.  A useless example, admittedly.)

What would that look like using JCL an Batchpipes, replacing "ls"
with LISTDS and "wc" with (utility of your chiice)?  (I wouln't get it
right the first time.)

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Mike Schwab]

Yes, I was not aware of the subsystem idea and was just throwing out an idea.

On Tue, Sep 21, 2021 at 10:14 PM Paul Gilmartin
<000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
>
> On Mon, 20 Sep 2021 22:30:32 -0500, Mike Schwab wrote:
>
> >How about
> >//ddname DD DISP=(NEW,DELETE,DELETE),
> >// DCB=(DSORG=PS[only],RECFM=F/FA/FM/V/VA/VM/U?/UA?/UM?/[no 
> >B]),BLKSIZE=n),
> >// PIPES=(PROGNAME,'100/32k byte parm to select or modify records',
> >// ddname1,ddname2,...,ddnameN),
> >and the OPEN (ddname,OUTPUT) loads the PIPES program in (sub)task memory)
> >and sets up the PUT/WRITE to DDNAME to call the program instead?
> >The close unloads the PIPES program in (subtask memory).
> >VSAM writes accepted like a DSORG=PS?
> >Any other parameters possible?
> >
> What manual did you find that in?  I don't see it in the JCL Ref.
>
> Are you makiing it up?  How is it preferable to SUBSYS=PIPE?  (BP01?
> I see each among various IBM pages.)
>
> >A pipes DDNAME can use a PIPES parameter for a subsequent program.
> >
> "subsequent"?  Not "concurrent"?  (But perhaps you mean lexically subsequent
> but temporally concurrent.)
>
> >Of course, ddnames can only be used by the EXEC PGM= or one of the
> >pipes programs.
> >And all pipes programs remain in memory until step end.
> >In case of abend / restart the checkpoint is taken in EXEC PGM= and
> >abandon updates / writes in any pipes program after the checkpoint.
> > ...
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



-- 
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Mon, 20 Sep 2021 22:30:32 -0500, Mike Schwab wrote:

>How about
>//ddname DD DISP=(NEW,DELETE,DELETE),
>// DCB=(DSORG=PS[only],RECFM=F/FA/FM/V/VA/VM/U?/UA?/UM?/[no B]),BLKSIZE=nnnnn),
>// PIPES=(PROGNAME,'100/32k byte parm to select or modify records',
>// ddname1,ddname2,...,ddnameN),
>and the OPEN (ddname,OUTPUT) loads the PIPES program in (sub)task memory)
>and sets up the PUT/WRITE to DDNAME to call the program instead?
>The close unloads the PIPES program in (subtask memory).
>VSAM writes accepted like a DSORG=PS?
>Any other parameters possible?
>
What manual did you find that in?  I don't see it in the JCL Ref.

Are you makiing it up?  How is it preferable to SUBSYS=PIPE?  (BP01?
I see each among various IBM pages.)

>A pipes DDNAME can use a PIPES parameter for a subsequent program.
>
"subsequent"?  Not "concurrent"?  (But perhaps you mean lexically subsequent
but temporally concurrent.)

>Of course, ddnames can only be used by the EXEC PGM= or one of the
>pipes programs.
>And all pipes programs remain in memory until step end.
>In case of abend / restart the checkpoint is taken in EXEC PGM= and
>abandon updates / writes in any pipes program after the checkpoint.
> ...

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Tue, 21 Sep 2021 21:03:14 -0500, Mike Schwab  wrote:

>If a SORT (or other similar temporary data store) program is one of
>the pipe programs, when the EXEC PGM= program closes the output file
>then the program holding the data needs to output the stored data to
>output ddnames (pipe or output files).
>
Are you thinking of MS-DOS pseudo-"pipes" where the upstream program
wrote a temporary file under-the-covers and the downstream program
processed it?  A pipe in syntax only.  Even Windows is better nowadays.

SORT is a bad conceptual example for Pipethink because SORT can't
write its first output record until it has read its last input record.  Better
to envision a filter which re-formats log records from a long-running (or
never-terminating) program, writing a file to be browsed with SDSF or
tail -f in real time.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Mike Schwab]

If a SORT (or other similar temporary data store) program is one of
the pipe programs, when the EXEC PGM= program closes the output file
then the program holding the data needs to output the stored data to
output ddnames (pipe or output files).

On Tue, Sep 21, 2021 at 5:25 AM Mike Schwab  wrote:
>
> Oh, and PIPES=(program,'parm',ddname1,ddname2,ddnaen) the ddname1 gets
> as input the one record written to the //ddname DD PIPES ddname.
>
> On Mon, Sep 20, 2021 at 10:30 PM Mike Schwab  wrote:
> >
> > How about
> > //ddname DD DISP=(NEW,DELETE,DELETE),
> > // DCB=(DSORG=PS[only],RECFM=F/FA/FM/V/VA/VM/U?/UA?/UM?/[no 
> > B]),BLKSIZE=n),
> > // PIPES=(PROGNAME,'100/32k byte parm to select or modify records',
> > // ddname1,ddname2,...,ddnameN),
> > and the OPEN (ddname,OUTPUT) loads the PIPES program in (sub)task memory)
> > and sets up the PUT/WRITE to DDNAME to call the program instead?
> > The close unloads the PIPES program in (subtask memory).
> > VSAM writes accepted like a DSORG=PS?
> > Any other parameters possible?
> >
> > A pipes DDNAME can use a PIPES parameter for a subsequent program.
> > Of course, ddnames can only be used by the EXEC PGM= or one of the
> > pipes programs.
> > And all pipes programs remain in memory until step end.
> > In case of abend / restart the checkpoint is taken in EXEC PGM= and
> > abandon updates / writes in any pipes program after the checkpoint.
> >
> > On Mon, Sep 20, 2021 at 9:12 PM Paul Gilmartin
> > <000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
> > >
> > > On Mon, 20 Sep 2021 19:10:16 -0500, Mike Schwab wrote:
> > >
> > > >So, in a sense, instead of pipes, the programs could be modified so
> > > >that instead of outputting a record, call the next program passing the
> > > >record as input.
> > > >
> > > No.
> > > Rather than requiring every utility to be modified in order to use the
> > > capability, it should be provided at the access method level or lower,
> > > transparent, so the top programs are oblivious to it.
> > >
> > > And there should be provision for inserting filters between a producer
> > > and a consumer to convert incompatible data formats.
> > >
> > > And ... what am I thinking?
> > >
> > > -- gil
> > >
> > > --
> > > For IBM-MAIN subscribe / signoff / archive access instructions,
> > > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> >
> >
> >
> > --
> > Mike A Schwab, Springfield IL USA
> > Where do Forest Rangers go to get away from it all?
>
>
>
> --
> Mike A Schwab, Springfield IL USA
> Where do Forest Rangers go to get away from it all?



-- 
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Mike Schwab]

Oh, and PIPES=(program,'parm',ddname1,ddname2,ddnaen) the ddname1 gets
as input the one record written to the //ddname DD PIPES ddname.

On Mon, Sep 20, 2021 at 10:30 PM Mike Schwab  wrote:
>
> How about
> //ddname DD DISP=(NEW,DELETE,DELETE),
> // DCB=(DSORG=PS[only],RECFM=F/FA/FM/V/VA/VM/U?/UA?/UM?/[no 
> B]),BLKSIZE=nnnnn),
> // PIPES=(PROGNAME,'100/32k byte parm to select or modify records',
> // ddname1,ddname2,...,ddnameN),
> and the OPEN (ddname,OUTPUT) loads the PIPES program in (sub)task memory)
> and sets up the PUT/WRITE to DDNAME to call the program instead?
> The close unloads the PIPES program in (subtask memory).
> VSAM writes accepted like a DSORG=PS?
> Any other parameters possible?
>
> A pipes DDNAME can use a PIPES parameter for a subsequent program.
> Of course, ddnames can only be used by the EXEC PGM= or one of the
> pipes programs.
> And all pipes programs remain in memory until step end.
> In case of abend / restart the checkpoint is taken in EXEC PGM= and
> abandon updates / writes in any pipes program after the checkpoint.
>
> On Mon, Sep 20, 2021 at 9:12 PM Paul Gilmartin
> <000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
> >
> > On Mon, 20 Sep 2021 19:10:16 -0500, Mike Schwab wrote:
> >
> > >So, in a sense, instead of pipes, the programs could be modified so
> > >that instead of outputting a record, call the next program passing the
> > >record as input.
> > >
> > No.
> > Rather than requiring every utility to be modified in order to use the
> > capability, it should be provided at the access method level or lower,
> > transparent, so the top programs are oblivious to it.
> >
> > And there should be provision for inserting filters between a producer
> > and a consumer to convert incompatible data formats.
> >
> > And ... what am I thinking?
> >
> > -- gil
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
>
>
> --
> Mike A Schwab, Springfield IL USA
> Where do Forest Rangers go to get away from it all?



-- 
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Mike Schwab]

How about
//ddname DD DISP=(NEW,DELETE,DELETE),
// DCB=(DSORG=PS[only],RECFM=F/FA/FM/V/VA/VM/U?/UA?/UM?/[no B]),BLKSIZE=n),
// PIPES=(PROGNAME,'100/32k byte parm to select or modify records',
// ddname1,ddname2,...,ddnameN),
and the OPEN (ddname,OUTPUT) loads the PIPES program in (sub)task memory)
and sets up the PUT/WRITE to DDNAME to call the program instead?
The close unloads the PIPES program in (subtask memory).
VSAM writes accepted like a DSORG=PS?
Any other parameters possible?

A pipes DDNAME can use a PIPES parameter for a subsequent program.
Of course, ddnames can only be used by the EXEC PGM= or one of the
pipes programs.
And all pipes programs remain in memory until step end.
In case of abend / restart the checkpoint is taken in EXEC PGM= and
abandon updates / writes in any pipes program after the checkpoint.

On Mon, Sep 20, 2021 at 9:12 PM Paul Gilmartin
<000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
>
> On Mon, 20 Sep 2021 19:10:16 -0500, Mike Schwab wrote:
>
> >So, in a sense, instead of pipes, the programs could be modified so
> >that instead of outputting a record, call the next program passing the
> >record as input.
> >
> No.
> Rather than requiring every utility to be modified in order to use the
> capability, it should be provided at the access method level or lower,
> transparent, so the top programs are oblivious to it.
>
> And there should be provision for inserting filters between a producer
> and a consumer to convert incompatible data formats.
>
> And ... what am I thinking?
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



-- 
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Mon, 20 Sep 2021 19:10:16 -0500, Mike Schwab wrote:

>So, in a sense, instead of pipes, the programs could be modified so
>that instead of outputting a record, call the next program passing the
>record as input.
>
No.
Rather than requiring every utility to be modified in order to use the
capability, it should be provided at the access method level or lower,
transparent, so the top programs are oblivious to it.

And there should be provision for inserting filters between a producer
and a consumer to convert incompatible data formats.

And ... what am I thinking?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Phil Smith III]

Hobart Spitz wrote, in part:

>This is a great comment.  I hadn't given that much thought to the question.

>Not to split hairs, but I didn't say MIPS, I said hardware.

>If I had to guess, MIPS usage might actually increase slightly, because the
>Pipes dispatcher has to switch between stages twice for every record that
>is passed.

 

Sure, just sayin' you'd want to be very clear about what you do mean.

 

I'm not quite sure what you mean by "more MIPS but less hardware", though?


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Mike Schwab]

So, in a sense, instead of pipes, the programs could be modified so
that instead of outputting a record, call the next program passing the
record as input.

Could something be developed similar to a SORTOUT exit that implements
this switch?

On Mon, Sep 20, 2021 at 4:27 PM Hobart Spitz  wrote:
>
> Phil;
>
> This is a great comment.  I hadn't given that much thought to the question.
>
> Not to split hairs, but I didn't say MIPS, I said hardware.
>
> If I had to guess, MIPS usage might actually increase slightly, because the
> Pipes dispatcher has to switch between stages twice for every record that
> is passed.  Access method overheard would drop.  Buffered access methods,
> in most cases, only have to increment the pointer into the block buffer,
> check for end-of-buffer and return to the caller.  I don't know for sure
> which is larger.  Maybe someone more knowledgeable than I can shed some
> light.
>
> I would say the real savings would be in elapsed run time and working set
> size.  Run time, due to eliminating something like 80-95% of I/O operations
> for intra-JOB datasets.  Working set reduction would save on real memory.
> (See below.)  Run time is probably more of a concern to customers,
> especially those with tight batch windows.  That said, working set size
> reduction would mean that processors would likely spend more, if not all,
> time pegged at 100% busy, because so many more address spaces (TSO and JOB)
> would be in a swapped-in and ready state than before.  Depending on what
> metrics the capacity planners are looking at, CPU sales might actually
> increase.  As I think about it more, if thru-put increases, new data could
> be generated more quickly and other times of hardware could be more in
> demand during peak load times.  I just don't know enough to say for sure.
>
> Phil and others know what follows.
>
> For those who don't know, in the typical case, a record passes through all
> possible stages before the next record begins the same trip.  Each record
> stays in the working page set, at least partially, during the entire time.
> Parts that are referenced have a good chance of staying cache resident
> between stages.
>
> Think of it this way:  You can visualize UNIX piping as a series of
> hourglasses open at both ends and connected in a tower.  Each grain of sand
> must stop at every "stage" and wait its turn to go through the narrow
> opening at the waist of each hourglass.  In Pipes, most stages have no
> delay and it's like a single tall hourglass tube with only one narrow
> point.  My best guess is that Pipes, in this analogy, would have only
> 5%-15% of the narrow openings as an equivalent UNIX piping command, meaning
> that the data (sand) would flow 85-95% faster in the Pipes "hourglass".
>
>
> OREXXMan
> Would you rather pass data in move mode (*nix piping) or locate mode
> (Pipes) or via disk (JCL)?  Why do you think you rarely see *nix commands
> with more than a dozen filters, while Pipelines specifications are commonly
> over 100s of stages, and 1000s of stages are not uncommon.
> IBM has been looking for an HLL for program products; REXX is that language.
>
>
> On Mon, Sep 20, 2021 at 12:48 PM Phil Smith III  wrote:
>
> > Hobart Spitz wrote, in part:
> >
> > >The case *for *Pipes in the z/OS base.:
> >
> > > 2. Hardware usage would drop for customers.
> >
> >
> >
> > From IBM's perspective, that might not be a positive argument. It should
> > be-they're hopefully not fooling themselves that they have a lock on
> > enterprise computing any more, so anything that makes life more palatable
> > for the remaining faithful at low cost to IBM should be A Good Thing-but I
> > can easily imagine someone saying, "We estimate this will reduce MIPS
> > purchases by x%, that's bad, don't do it".
> >
> >
> >
> > Just sayin'.
> >
> >
> >
> > ...phsiii
> >
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> >
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



-- 
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Hobart Spitz]

Phil;

This is a great comment.  I hadn't given that much thought to the question.

Not to split hairs, but I didn't say MIPS, I said hardware.

If I had to guess, MIPS usage might actually increase slightly, because the
Pipes dispatcher has to switch between stages twice for every record that
is passed.  Access method overheard would drop.  Buffered access methods,
in most cases, only have to increment the pointer into the block buffer,
check for end-of-buffer and return to the caller.  I don't know for sure
which is larger.  Maybe someone more knowledgeable than I can shed some
light.

I would say the real savings would be in elapsed run time and working set
size.  Run time, due to eliminating something like 80-95% of I/O operations
for intra-JOB datasets.  Working set reduction would save on real memory.
(See below.)  Run time is probably more of a concern to customers,
especially those with tight batch windows.  That said, working set size
reduction would mean that processors would likely spend more, if not all,
time pegged at 100% busy, because so many more address spaces (TSO and JOB)
would be in a swapped-in and ready state than before.  Depending on what
metrics the capacity planners are looking at, CPU sales might actually
increase.  As I think about it more, if thru-put increases, new data could
be generated more quickly and other times of hardware could be more in
demand during peak load times.  I just don't know enough to say for sure.

Phil and others know what follows.

For those who don't know, in the typical case, a record passes through all
possible stages before the next record begins the same trip.  Each record
stays in the working page set, at least partially, during the entire time.
Parts that are referenced have a good chance of staying cache resident
between stages.

Think of it this way:  You can visualize UNIX piping as a series of
hourglasses open at both ends and connected in a tower.  Each grain of sand
must stop at every "stage" and wait its turn to go through the narrow
opening at the waist of each hourglass.  In Pipes, most stages have no
delay and it's like a single tall hourglass tube with only one narrow
point.  My best guess is that Pipes, in this analogy, would have only
5%-15% of the narrow openings as an equivalent UNIX piping command, meaning
that the data (sand) would flow 85-95% faster in the Pipes "hourglass".


OREXXMan
Would you rather pass data in move mode (*nix piping) or locate mode
(Pipes) or via disk (JCL)?  Why do you think you rarely see *nix commands
with more than a dozen filters, while Pipelines specifications are commonly
over 100s of stages, and 1000s of stages are not uncommon.
IBM has been looking for an HLL for program products; REXX is that language.


On Mon, Sep 20, 2021 at 12:48 PM Phil Smith III  wrote:

> Hobart Spitz wrote, in part:
>
> >The case *for *Pipes in the z/OS base.:
>
> > 2. Hardware usage would drop for customers.
>
>
>
> From IBM's perspective, that might not be a positive argument. It should
> be-they're hopefully not fooling themselves that they have a lock on
> enterprise computing any more, so anything that makes life more palatable
> for the remaining faithful at low cost to IBM should be A Good Thing-but I
> can easily imagine someone saying, "We estimate this will reduce MIPS
> purchases by x%, that's bad, don't do it".
>
>
>
> Just sayin'.
>
>
>
> ...phsiii
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Seymour J Metz]

It would not be difficult to use the API to bring in stream I/O under TSO, but 
supporting PAM and SAM is another matter.

Concurrency is another reason for porting oorexx. As for z/VM, it's not your 
father's CMS.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Paul Gilmartin [000433f07816-dmarc-requ...@listserv.ua.edu]
Sent: Monday, September 20, 2021 10:48 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - 
Interpret or Value - Which is better?)

On Mon, 20 Sep 2021 14:20:04 +, Seymour J Metz wrote:

>If portability is an issue then we need ANSI stream I/O in all REXX 
>environments. If we had that we wouldn't need EXECIO for new code.
>
It's ironic.  The z/OS Rexx interpreter supports ANSI stream, but only under 
OMVS.
I wonder whether the API could start Rexx with the needed function package in 
other
environments.  And only for UNIX files, not Classic.

The z/OS Rexx compiler appears to support Stream I/O, but not for UNIX files.

Conway's Law.  An  OMVS developer pleaded for directing message and TRACE
output to stderr (the Regina default) rather than stdout.  The Rexx designers
rebuffed that.  Even though the Rexx programming interface provides distinct
calls for those functions.

I've been RTFM.  It appears that the "other side" of DDNAME is SUBSYS=PIPE.
Would that work even in a single job step, with use of alternate DDNAMES as
necessary?

And concurrency.  ADDRESS ATTCHMVS is a deception.  It always WAITs for
subtask completion rather than allowing it to run in the background. Concurrency
would be enormously difficult in CMS; relatively easy in z/OS.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Phil Smith III]

Hobart Spitz wrote, in part:

>The case *for *Pipes in the z/OS base.:

> 2. Hardware usage would drop for customers.

 

>From IBM's perspective, that might not be a positive argument. It should
be-they're hopefully not fooling themselves that they have a lock on
enterprise computing any more, so anything that makes life more palatable
for the remaining faithful at low cost to IBM should be A Good Thing-but I
can easily imagine someone saying, "We estimate this will reduce MIPS
purchases by x%, that's bad, don't do it".

 

Just sayin'.

 

...phsiii


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Dave Jones]

+1 for this idea.
DJ

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Mon, 20 Sep 2021 14:20:04 +, Seymour J Metz wrote:

>If portability is an issue then we need ANSI stream I/O in all REXX 
>environments. If we had that we wouldn't need EXECIO for new code.
>
It's ironic.  The z/OS Rexx interpreter supports ANSI stream, but only under 
OMVS.
I wonder whether the API could start Rexx with the needed function package in 
other
environments.  And only for UNIX files, not Classic.

The z/OS Rexx compiler appears to support Stream I/O, but not for UNIX files.

Conway's Law.  An  OMVS developer pleaded for directing message and TRACE
output to stderr (the Regina default) rather than stdout.  The Rexx designers
rebuffed that.  Even though the Rexx programming interface provides distinct
calls for those functions.

I've been RTFM.  It appears that the "other side" of DDNAME is SUBSYS=PIPE.
Would that work even in a single job step, with use of alternate DDNAMES as
necessary?

And concurrency.  ADDRESS ATTCHMVS is a deception.  It always WAITs for
subtask completion rather than allowing it to run in the background. Concurrency
would be enormously difficult in CMS; relatively easy in z/OS.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Seymour J Metz]

If portability is an issue then we need ANSI stream I/O in all REXX 
environments. If we had that we wouldn't need EXECIO for new code.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3



From: IBM Mainframe Discussion List  on behalf of 
Hobart Spitz 
Sent: Sunday, September 19, 2021 12:53 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - 
Interpret or Value - Which is better?)

Thank you for your relevant and helpful comments.  They are very much
appreciated, as I omitted some topics.  I'll do my best to address them
here.  Pardon the lack of brevity.

Concerning EXECIO:
   Yes, the z/OS and z/VM EXECIO versions are mostly incompatible.
Once you have Pipes you don't need or want EXECIO.  Here's why:

   - Pipes can read from or write to either a DD or a dataset.  z/OS EXECIO
   can only use a DD.
   - TSO Pipes has a format that is syntactically and semantically similar
   to that in CMS Pipes.  It is used with a PDS/E allocated to a DD.
   - When a dataset is specified, Pipes defaults to OLD when writing.  This
   is something that z/OS access methods don't even check.  You could still
   accidentally shoot yourself in the foot, but it's not obvious in JCL  In
   Pipes to have to explicitly override the default by coding SHR next to the
   dataset name.  I don't know why you would want to.  Consider the
   organization that doesn't protect itself from writing with a DISP=SHR in
   JCL, either with education, standards or exits.  OT:  This is why you
   should *always* put DISP= on the first line of a JCL DD and the DSN on a
   continuation.  Otherwise, if someone performs maintenance and accidentally
   ends up with DISP=SHR on an output DD, for a long time there may be no
   errors and it may run fine even in production.  That is, until the day that
   another process writes with SHR or reads while the dataset is in an
   inconsistent state.  There could be a lot of confusion.  I would be glad to
   hear from someone knowledgeable that I'm wrong on this and that I missed
   some access method change that has made the concern unnecessary.
   - Pipes lets you read member(s) through a DD and names(s) specified in
   the pipe stage or by feeding member names to certain stages.  With EXECIO,
   you must put the member name on the allocation.  If you've ever read a
   large number of members from a PDS/E with EXECIO, you know it takes
   forever.  You must go through TSO ALLOC command attach, enqueue, catalog
   search, VTOC search, directory search, and OPEN (to name those I know
   about), and then free and dequeue the entire allocation before starting the
   next member.  (OT:  ISPF services allows you to allocate a PDS/E and then
   process multiple members.  What takes minutes with EXECIO takes seconds
   with ISPF and Pipes.)
   - With Pipes, you don't have to write a loop to process or build records
   in a stem or in the stack.  Since the records are ready available to the
   steam, you process them from there in any way that your application
   dictates.
   - Pipes does parallel enqueues, allocations and opens via its COMMIT
   mechanism.  In a nutshell, during the initialization stage (commit level <
   0), any stage can issue a negative return code.  It stops further regular
   processing and gives stages the opportunity to release any resources
   obtained.  This is similar to what happens with batch JOB JCL.  Recovering
   from an enqueue, allocation. or open failure can be complicated with
   multiple instances of EXECIO.

Concerning GLOBALV:

   I have looked at GLOBALV files and they do not appear to be too
difficult to read and write compatibly with REXX and/or Pipes.  IBM
documents the format.  SESSION values could be saved in a VIO dataset for
performance and tranciency.  Thus writing a GLOBALV for TSO seems highly
practical.  If there was no better option, one could write logic for just
those functions that are used by the CMS code being ported.


Concerning checkpointing:

   - Checkpointing was originally implemented because large organizations
   had regular jobs that ran for hours. If there was an abend and a long
   running job had to be rerun, whole sets of dependent jobs would be delayed,
   throwing weekly payroll, monthly payables, or bill cycle receivables, etc.
   behind schedule.  The company could be put at risk of penalties for late
   payroll, interest charges for overdue payments, or delayed receivables
   income, etc.  Because today's platforms are orders of magnitude faster,
   there are many fewer such jobs today, even given increased volumes.  Many
   checkpointing concerns are moot today.  (This includes for example baring
   temporary datasets in production because a job that now runs in 10 minutes
   or less might have to be restarted when it would have run fine from
   scratch.  It will take that time to review the problem, fix it and set

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Sun, 19 Sep 2021 11:53:18 -0500, Hobart Spitz wrote:
>...
>Concerning EXECIO:
>   - Pipes can read from or write to either a DD or a dataset.  z/OS EXECIO
>   can only use a DD.
>  
o Can that DD be allocated to a UNIX file, with FILEDATA any of BINARY, TEXT, 
or RECORD?
o Is it savvy to files tagged with FILEDATA and/or CCSID?
o Does it restrict concurrent use of UNIX files as ISPF does?:
  
<https://www.ibm.com/docs/en/zos/2.1.0?topic=integrity-zos-unix-file-enqueue>

>... -   I would be glad to
>   hear from someone knowledgeable that I'm wrong on this and that I missed
>   some access method change that has made the concern unnecessary.
>
As I mentioned earlier, the SPFEDIT member ENQs with DISP=SHR
avoid the need to lock an entire PDS in order to update a single member:
<https://www.ibm.com/docs/en/zos/2.1.0?topic=integrity-member-name-enqueue>

>   - Pipes lets you read member(s) through a DD and names(s) specified in
>   the pipe stage or by feeding member names to certain stages. 
>   you must put the member name on the allocation.  If you've ever read a
>   large number of members from a PDS/E 
>
Does this support mixed concatenations of PDS, PDSE and UNIX directories?
(It should.  This is intrinsic to  BPAM.)  What happens if the UNIX files have
unlike tags?

Does Pipelines support the "other side" of DDNAMEs?  I.e. can Pipelines
output connectors be associated with input DDs of Classic utilities such as
OLD and NEW of ISRSUPC, or SYSUT2 of an arbitrary Classic utility be
associated with a Pipeline input cnonector.  I could do this with a Rube
Goldberg of SYSCALL PIPE, BPXWDYN, SYSCALL SPAWN,  ...
It would be nice to have it in an opaque package.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Sun, 19 Sep 2021 11:53:18 -0500, Hobart Spitz wrote:
>...
>Once you have Pipes you don't need or want EXECIO.  ...
>  
At times the effort of converting existing art outweighs the benefit.  And  
POSIX
pipes are portable to desktop systems; CMS/TSO Pipelines is not.

>   - Pipes can read from or write to either a DD or a dataset.  z/OS EXECIO
>   can only use a DD.

>   - When a dataset is specified, Pipes defaults to OLD when writing.  This
>   is something that z/OS access methods don't even check.  You could still
>   accidentally shoot yourself in the foot, but it's not obvious in JCL  In
>   Pipes to have to explicitly override the default by coding SHR next to the
>   dataset name.  I don't know why you would want to.  
>
Pipes ought also support the SPFEDIT ENQ:
<https://www.ibm.com/docs/en/zos/2.1.0?topic=integrity-member-name-enqueue>
as FTP and NFS do in order to allow safely updating one member of a PDS by one
job while another job processes a different member, and to allow creating 
different
members concurenttly, supported by PDSE but not PDS.  With "1000s of stages"
a programmer might not easily check for this.

Is there any protection against FANOUT's having downstream stages that 
destructively
update the same data sett?  Does Pipes do "ENQ RET=HAVE"?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Hobart Spitz]

Thank you for your relevant and helpful comments.  They are very much
appreciated, as I omitted some topics.  I'll do my best to address them
here.  Pardon the lack of brevity.

Concerning EXECIO:
   Yes, the z/OS and z/VM EXECIO versions are mostly incompatible.
Once you have Pipes you don't need or want EXECIO.  Here's why:

   - Pipes can read from or write to either a DD or a dataset.  z/OS EXECIO
   can only use a DD.
   - TSO Pipes has a format that is syntactically and semantically similar
   to that in CMS Pipes.  It is used with a PDS/E allocated to a DD.
   - When a dataset is specified, Pipes defaults to OLD when writing.  This
   is something that z/OS access methods don't even check.  You could still
   accidentally shoot yourself in the foot, but it's not obvious in JCL  In
   Pipes to have to explicitly override the default by coding SHR next to the
   dataset name.  I don't know why you would want to.  Consider the
   organization that doesn't protect itself from writing with a DISP=SHR in
   JCL, either with education, standards or exits.  OT:  This is why you
   should *always* put DISP= on the first line of a JCL DD and the DSN on a
   continuation.  Otherwise, if someone performs maintenance and accidentally
   ends up with DISP=SHR on an output DD, for a long time there may be no
   errors and it may run fine even in production.  That is, until the day that
   another process writes with SHR or reads while the dataset is in an
   inconsistent state.  There could be a lot of confusion.  I would be glad to
   hear from someone knowledgeable that I'm wrong on this and that I missed
   some access method change that has made the concern unnecessary.
   - Pipes lets you read member(s) through a DD and names(s) specified in
   the pipe stage or by feeding member names to certain stages.  With EXECIO,
   you must put the member name on the allocation.  If you've ever read a
   large number of members from a PDS/E with EXECIO, you know it takes
   forever.  You must go through TSO ALLOC command attach, enqueue, catalog
   search, VTOC search, directory search, and OPEN (to name those I know
   about), and then free and dequeue the entire allocation before starting the
   next member.  (OT:  ISPF services allows you to allocate a PDS/E and then
   process multiple members.  What takes minutes with EXECIO takes seconds
   with ISPF and Pipes.)
   - With Pipes, you don't have to write a loop to process or build records
   in a stem or in the stack.  Since the records are ready available to the
   steam, you process them from there in any way that your application
   dictates.
   - Pipes does parallel enqueues, allocations and opens via its COMMIT
   mechanism.  In a nutshell, during the initialization stage (commit level <
   0), any stage can issue a negative return code.  It stops further regular
   processing and gives stages the opportunity to release any resources
   obtained.  This is similar to what happens with batch JOB JCL.  Recovering
   from an enqueue, allocation. or open failure can be complicated with
   multiple instances of EXECIO.

Concerning GLOBALV:

   I have looked at GLOBALV files and they do not appear to be too
difficult to read and write compatibly with REXX and/or Pipes.  IBM
documents the format.  SESSION values could be saved in a VIO dataset for
performance and tranciency.  Thus writing a GLOBALV for TSO seems highly
practical.  If there was no better option, one could write logic for just
those functions that are used by the CMS code being ported.


Concerning checkpointing:

   - Checkpointing was originally implemented because large organizations
   had regular jobs that ran for hours. If there was an abend and a long
   running job had to be rerun, whole sets of dependent jobs would be delayed,
   throwing weekly payroll, monthly payables, or bill cycle receivables, etc.
   behind schedule.  The company could be put at risk of penalties for late
   payroll, interest charges for overdue payments, or delayed receivables
   income, etc.  Because today's platforms are orders of magnitude faster,
   there are many fewer such jobs today, even given increased volumes.  Many
   checkpointing concerns are moot today.  (This includes for example baring
   temporary datasets in production because a job that now runs in 10 minutes
   or less might have to be restarted when it would have run fine from
   scratch.  It will take that time to review the problem, fix it and set up
   the job for restart.  Too many people are copying JCL or following less
   than well documented standards that they and/or their management don't
   understand.)   The remaining long running jobs are prime candidates for
   being rewritten in Pipes.
   - Rewriting a long running JOB or UNIX piping command in Pipes can cut
   the run time dramatically.  A rule of thumb for the time saved is the total
   I/O time for all intermediate datasets used to pass data from step to
   step.  A 

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Fri, 17 Sep 2021 16:21:25 -0400, Steve Thompson wrote:

>EXECIO is not supported the same between z/os and CMS. This is a pain. 
> 
Yes.  What are your specifics.  Mine include:

o Lack of the VAR and, especially STRING, options, requiring extra code.
  Is that an accommodation to syntactic limitations of the CLIST/TSO parser?

o Requirement for the OPEN option in some cases.  I suspect this is a
  feckless accommodation to a customer who complained that
  EXECIO 0 DISKW fileid (FINIS created an empty file on TSO but not
  on CMS.  That's merely the behavior of the underlying file system.

o Lack of the PRINT, PUNCH, etc. forms.  That's a difference in the
  Data Management conventions of the supporting OS.

o Difference in the fileid syntax.  As in the previous bullet.  I've easily
  dealt with that by dual-path.

The page at  lauds Rexx portability.
I see it as an argument for the other side.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Paul Gilmartin]

On Fri, 17 Sep 2021 14:56:15 -0500, Hobart Spitz wrote:
>...
>   15. Is consistent with policies for combating global warming of
>   customers, vendors and the public.   ...
>  
OK.  But 40% of the U.S. electorate would consider that an argument "against".

Think of the environmental impact of cryptocurrency mining (GIYF).  Would
TSO Pipelines and/or ICSF mitigate that?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Farley, Peter x23353]

One additional "against" (until the problem is solved):

The knowledge of how to structure recovery from abends in one (or more) pipe 
stages is not well known in application circles.  Most batch recovery today 
involves stopping any successor jobs (usually via scheduler dependency on 
successful completion of the abending job(s)), then resolving the program 
abend(s) and rerunning the failing job(s).

How does one structure a multi-hundred stage pipeline to enable easy and rapid 
recovery from an abend at any stage?  I've been in the mainframe programming 
business since 1972 and I would not know how to structure such a beast.  
Restarting from the beginning is not an option with the increasingly gargantuan 
volumes of data we must process and the tighter and tighter SLA's our customers 
demand.

Peter

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Hobart Spitz
Sent: Friday, September 17, 2021 3:56 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: The Business Case for Pipes in the z/OS Base (was: Re: REXX - 
Interpret or Value - Which is better?)

IMHO, the Business Cases on Pipes in the z/OS Base are as follows.  (Pipes is 
already available as part of BatchPipes.)

The case *for *Pipes in the z/OS base.:

   1. Development costs would drop for customers, vendors, and IBM, since
   everyone could use Pipes in their software.
   2. Hardware usage would drop for customers.  In addition to avoiding
   I/O, Pipes uses a record address-and-length descriptor.  A record can
   flow from stage to stage with the only data movement being for changed
   records.  Potential data needed by a stage could have already been in the
   working set and/or cache-loaded by the previous stage.  (A methodology for
   identifying the cost/benefits by JOB and application would allow the best
   to be reworked first.  Thus Pipes would pay for itself in the shortest
   amount of time.)
   3. Product efficiency for vendors (IBM and others) would improve.
   (Arguably it's the other side of the coin in #2.)
   4. Tight integration with REXX, CLIST and JCL.
   5. Portability to and from z/VM.  This breaks down differently for
   different groups:
  - Customers: Cheaper porting to/from z/OS.  (Porting to other IBM
  Series is expensive and time-consuming, AFAIK.)
  - Vendors:  Write once for both platforms.
  - IBM:  Rather than customers moving to non-IBM platforms, when z/OS
  or z/VM don't meet their needs, those customers would have another option
  to stay with IBM.
   6. You can process both RecFM F records and RecFM V records with the
   same stages.
   7. Pipes can be used on both EXEC and DD JCL statements.  This is
   primarily for REXX-a-phobes.  Pipe commands in REXX are amazing; I've used
   the combination on both z/OS and z/VM (and predecessors).  Pipes with
   CLISTs is almost as good, AFAIK.
   8. Increased competitiveness for IBM hardware and software.  This would
   especially apply to UNIX customers who have exceeded the capabilities of
   their platforms.
   9. CMS/TSO Pipes is better than UNIX piping, and REXX is better than C.
   With today's processors, C's performance advantage over REXX is not
   significant, and dwarfed by low developer productivity (your bullet, your
   gun, your foot) of C.  Strategically using Pipes with REXX, can give better
   performance that UNIX piping and C.
   10. Since both C++ and Pipes are internally pointer based, you could get
   similar benefits by using OO techniques exclusively.  How are you going to
   convert a COBOL and JCL application to C++?  Not as easily as going to REXX
   and Pipes.
   11. z/OS is a file-structure-aware operating system.  It does not
   use embedded control characters in text files or impose record boundary
   knowledge in binary files.  Pipes reads and writes byte stream data by
   converting to and from record address-and-length descriptor format.  This
   means that, in most cases, any sequence of stages perform equally well on
   RecFM F, RecFM V, byte stream text data and deblocked UNIX binary files.
   12. Addresses staff shortages due to baby-boomer retirements and CoViD
   impacts.
   13. Reduces batch window requirements.  With less I/O, JOBs finish
   faster.
   14. It's my understanding that there are people inside IBM who are
   behind Pipes going into the z/OS base.  Pipes is part of the z/VM base.
   Can z/OS be that far behind?
   15. Is consistent with policies for combating global warming of
   customers, vendors and the public.  Fewer CPU cycles wasted means less heat
   to be dissipated and less electricity to be produced.  The UNIX stream and
   C string models are obsolete in this light; every character must go through
   the CPU to get to the end of a record or string.   Not so in Pipes or SAM
   access methods.  Rarely do you see a UNIX command with more than a dozen
   filters; Pipes commands go into the 100s or 1000s of stages.  In general,
   the more stages 

Re: The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Steve Thompson]

EXECIO is not supported the same between z/os and CMS. This is a pain. 

GLOBALV probably needs porting to solve other problems. I was tempted to do 
that on the last project I was on where we had REXX working between the two 
platforms. 

Sent from my iPhone — small keyboarf, fat fungrs, stupd spell manglr. Expct 
mistaks 


> On Sep 17, 2021, at 3:57 PM, Hobart Spitz  wrote:
> 
> IMHO, the Business Cases on Pipes in the z/OS Base are as follows.  (Pipes
> is already available as part of BatchPipes.)
> 
> The case *for *Pipes in the z/OS base.:
> 
>   1. Development costs would drop for customers, vendors, and IBM, since
>   everyone could use Pipes in their software.
>   2. Hardware usage would drop for customers.  In addition to avoiding
>   I/O, Pipes uses a record address-and-length descriptor.  A record can
>   flow from stage to stage with the only data movement being for changed
>   records.  Potential data needed by a stage could have already been in the
>   working set and/or cache-loaded by the previous stage.  (A methodology for
>   identifying the cost/benefits by JOB and application would allow the best
>   to be reworked first.  Thus Pipes would pay for itself in the shortest
>   amount of time.)
>   3. Product efficiency for vendors (IBM and others) would improve.
>   (Arguably it's the other side of the coin in #2.)
>   4. Tight integration with REXX, CLIST and JCL.
>   5. Portability to and from z/VM.  This breaks down differently for
>   different groups:
>  - Customers: Cheaper porting to/from z/OS.  (Porting to other IBM
>  Series is expensive and time-consuming, AFAIK.)
>  - Vendors:  Write once for both platforms.
>  - IBM:  Rather than customers moving to non-IBM platforms, when z/OS
>  or z/VM don't meet their needs, those customers would have another option
>  to stay with IBM.
>   6. You can process both RecFM F records and RecFM V records with the
>   same stages.
>   7. Pipes can be used on both EXEC and DD JCL statements.  This is
>   primarily for REXX-a-phobes.  Pipe commands in REXX are amazing; I've used
>   the combination on both z/OS and z/VM (and predecessors).  Pipes with
>   CLISTs is almost as good, AFAIK.
>   8. Increased competitiveness for IBM hardware and software.  This would
>   especially apply to UNIX customers who have exceeded the capabilities of
>   their platforms.
>   9. CMS/TSO Pipes is better than UNIX piping, and REXX is better than C.
>   With today's processors, C's performance advantage over REXX is not
>   significant, and dwarfed by low developer productivity (your bullet, your
>   gun, your foot) of C.  Strategically using Pipes with REXX, can give better
>   performance that UNIX piping and C.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

The Business Case for Pipes in the z/OS Base (was: Re: REXX - Interpret or Value - Which is better?)

[Hobart Spitz]

IMHO, the Business Cases on Pipes in the z/OS Base are as follows.  (Pipes
is already available as part of BatchPipes.)

The case *for *Pipes in the z/OS base.:

   1. Development costs would drop for customers, vendors, and IBM, since
   everyone could use Pipes in their software.
   2. Hardware usage would drop for customers.  In addition to avoiding
   I/O, Pipes uses a record address-and-length descriptor.  A record can
   flow from stage to stage with the only data movement being for changed
   records.  Potential data needed by a stage could have already been in the
   working set and/or cache-loaded by the previous stage.  (A methodology for
   identifying the cost/benefits by JOB and application would allow the best
   to be reworked first.  Thus Pipes would pay for itself in the shortest
   amount of time.)
   3. Product efficiency for vendors (IBM and others) would improve.
   (Arguably it's the other side of the coin in #2.)
   4. Tight integration with REXX, CLIST and JCL.
   5. Portability to and from z/VM.  This breaks down differently for
   different groups:
  - Customers: Cheaper porting to/from z/OS.  (Porting to other IBM
  Series is expensive and time-consuming, AFAIK.)
  - Vendors:  Write once for both platforms.
  - IBM:  Rather than customers moving to non-IBM platforms, when z/OS
  or z/VM don't meet their needs, those customers would have another option
  to stay with IBM.
   6. You can process both RecFM F records and RecFM V records with the
   same stages.
   7. Pipes can be used on both EXEC and DD JCL statements.  This is
   primarily for REXX-a-phobes.  Pipe commands in REXX are amazing; I've used
   the combination on both z/OS and z/VM (and predecessors).  Pipes with
   CLISTs is almost as good, AFAIK.
   8. Increased competitiveness for IBM hardware and software.  This would
   especially apply to UNIX customers who have exceeded the capabilities of
   their platforms.
   9. CMS/TSO Pipes is better than UNIX piping, and REXX is better than C.
   With today's processors, C's performance advantage over REXX is not
   significant, and dwarfed by low developer productivity (your bullet, your
   gun, your foot) of C.  Strategically using Pipes with REXX, can give better
   performance that UNIX piping and C.
   10. Since both C++ and Pipes are internally pointer based, you could get
   similar benefits by using OO techniques exclusively.  How are you going to
   convert a COBOL and JCL application to C++?  Not as easily as going to REXX
   and Pipes.
   11. z/OS is a file-structure-aware operating system.  It does not
   use embedded control characters in text files or impose record boundary
   knowledge in binary files.  Pipes reads and writes byte stream data by
   converting to and from record address-and-length descriptor format.  This
   means that, in most cases, any sequence of stages perform equally well on
   RecFM F, RecFM V, byte stream text data and deblocked UNIX binary files.
   12. Addresses staff shortages due to baby-boomer retirements and CoViD
   impacts.
   13. Reduces batch window requirements.  With less I/O, JOBs finish
   faster.
   14. It's my understanding that there are people inside IBM who are
   behind Pipes going into the z/OS base.  Pipes is part of the z/VM base.
   Can z/OS be that far behind?
   15. Is consistent with policies for combating global warming of
   customers, vendors and the public.  Fewer CPU cycles wasted means less heat
   to be dissipated and less electricity to be produced.  The UNIX stream and
   C string models are obsolete in this light; every character must go through
   the CPU to get to the end of a record or string.   Not so in Pipes or SAM
   access methods.  Rarely do you see a UNIX command with more than a dozen
   filters; Pipes commands go into the 100s or 1000s of stages.  In general,
   the more stages the better the efficiency, since you are doing more work
   with the same amount of I/O.  If one reads the tea leaves, it seems
   inevitable that there will be some kind of restriction on activities that
   cause global warming; whether it's a tax, regulation, cap-and-trade, etc.,
   I can't guess.  The point is that it is likely that many heavy resource
   using applications and processes will have to be converted to Pipes-like
   mechanisms, on all platforms to fight global warming.  The question is, do
   you want to start now when it can be done as part of a well thought-out
   cost-effective plan or do it in a rush later, when the costs and risks have
   ballooned?

The case *a**gainst *Pipes, IMHO:

   1. Short term loss of revenue to due better hardware utilization for the
   same work.  Because of improved competitiveness the dip should be short
   lived if it happens at all.  Not all customers will embrace Pipes at the
   same rate or at the same time.  On the other hand, there is always the
   chance that a few successes turn into a snowball.
   2. From the customer perspective

Re: RFE TSO Pipes - the TOP voted RFE among all brands

[Roberto Halais]

Output of PIPEQ COMMAND:

FPLINX086I CMS/TSO Pipelines, 5654-030/5655-A17 1.0110
(Version.Release/Mod) - Generated 7 Apr 1998 at 14:00:22




On Mon, Jul 19, 2021 at 10:22 AM Lionel B. Dyck  wrote:

> The RFE for TSO PIPES is now the top voted RFE - let's keep voting - if you
> haven't then please vote, if you have then encourage others to vote.
>
>
>
> The URL is http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe
> <
> http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=47699
> >
> _ID=47699
>
>
>
> The current vote is 221 votes.
>
>
>
> IBM - it is time to release PIPES for TSO - let's keep z/OS productive and
> moving forward.
>
>
>
>
>
> Lionel B. Dyck <><
>
> Website: https://www.lbdsoftware.com
>
> Github: https://github.com/lbdyck
>
>
>
> "Worry more about your character than your reputation. Character is what
> you
> are, reputation merely what others think you are."   - - - John Wooden
>
>
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
-- 
Politics: Poli (many) - tics (blood sucking parasites)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

RFE TSO Pipes - the TOP voted RFE among all brands

[Lionel B. Dyck]

The RFE for TSO PIPES is now the top voted RFE - let's keep voting - if you
haven't then please vote, if you have then encourage others to vote.

 

The URL is http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe
<http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=47699>
_ID=47699

 

The current vote is 221 votes.

 

IBM - it is time to release PIPES for TSO - let's keep z/OS productive and
moving forward.

 

 

Lionel B. Dyck <><

Website: https://www.lbdsoftware.com

Github: https://github.com/lbdyck

 

"Worry more about your character than your reputation. Character is what you
are, reputation merely what others think you are."   - - - John Wooden

 


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: TSO Pipes (and BatchPipes)

[Lionel B Dyck]

Ray - thank you - so the CMS Pipes is 20 years newer than the TSO Pipes (at 
least the BatchPipes version).


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of Ray 
Pearce
Sent: Wednesday, April 22, 2020 11:29 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: TSO Pipes (and BatchPipes)

On CMS I get
FPLINX086I CMS Pipelines, 5741-A07 1.0112 (Version.Release/Mod) - Generated 8 
Sep 2016 at 15:22:22

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf 
Of Hobart Spitz
Sent: 22 April 2020 17:04
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: TSO Pipes (and BatchPipes)

If there are any z/VMers subscribed, it would be interesting to also compare 
the PIPE Q output under CMS.

Thanks.

OREXXMan
JCL is the buggy whip of 21st century computing.  Stabilize it.
Put Pipelines in the z/OS base.  Would you rather process data in move mode or 
locate mode?
IBM has been looking for an HLL for program products; REXX is that language.


On Wed, Apr 22, 2020 at 10:57 AM Lionel B Dyck  wrote:

> I don't have access to TSO Pipes other than what is included with 
> BatchPipes but it seems that it is a tad dated - it is actually old 
> enough to drink:
>
>
>
> BPW00086I BatchPipes/MVS BatchPipeWorks, 5655-065/SCPSP 1.0109
> (Version.Release/Mod) - Generated 12 Oct 1996 at 13:59:00
>
>
>
> If someone has the non-BP version of TSO Pipes it would be interesting 
> to see the results of the PIPES Q command showing the date.
>
>
>
> For those of you who would like to encourage IBM to release TSO Pipes 
> as part of the z/OS base please go to this URL and vote 
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe
> <
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=
> 47699
> > _ID=47699  (note the current vote is 188 which places it at the  
> > top
> in
> the z/OS area).
>
>
>
> Having TSO Pipes as a native part of the z/OS distribution would help 
> everyone as software developers would then be free to use it when 
> developing applications, products, and tools knowing that every site 
> has it available.
>
>
>
>
>
> Lionel B. Dyck <
> Website:  <https://www.lbdsoftware.com> https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is 
> what you are, reputation merely what others think you are." - John 
> Wooden
>
>
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send 
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
This e-mail message has been scanned and cleared by Google Message Security and 
the UNICOM Global security systems. This message is for the named person's use 
only. If you receive this message in error, please delete it and notify the 
sender. 

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: TSO Pipes (and BatchPipes)

[Ray Pearce]

On CMS I get
FPLINX086I CMS Pipelines, 5741-A07 1.0112 (Version.Release/Mod) - Generated 8 
Sep 2016 at 15:22:22

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf 
Of Hobart Spitz
Sent: 22 April 2020 17:04
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: TSO Pipes (and BatchPipes)

If there are any z/VMers subscribed, it would be interesting to also
compare the PIPE Q output under CMS.

Thanks.

OREXXMan
JCL is the buggy whip of 21st century computing.  Stabilize it.
Put Pipelines in the z/OS base.  Would you rather process data in move mode
or locate mode?
IBM has been looking for an HLL for program products; REXX is that language.


On Wed, Apr 22, 2020 at 10:57 AM Lionel B Dyck  wrote:

> I don't have access to TSO Pipes other than what is included with
> BatchPipes
> but it seems that it is a tad dated - it is actually old enough to drink:
>
>
>
> BPW00086I BatchPipes/MVS BatchPipeWorks, 5655-065/SCPSP 1.0109
> (Version.Release/Mod) - Generated 12 Oct 1996 at 13:59:00
>
>
>
> If someone has the non-BP version of TSO Pipes it would be interesting to
> see the results of the PIPES Q command showing the date.
>
>
>
> For those of you who would like to encourage IBM to release TSO Pipes as
> part of the z/OS base please go to this URL and vote
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe
> <
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=47699
> > _ID=47699  (note the current vote is 188 which places it at the  top
> in
> the z/OS area).
>
>
>
> Having TSO Pipes as a native part of the z/OS distribution would help
> everyone as software developers would then be free to use it when
> developing
> applications, products, and tools knowing that every site has it available.
>
>
>
>
>
> Lionel B. Dyck <
> Website:  <https://www.lbdsoftware.com> https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is what
> you are, reputation merely what others think you are." - John Wooden
>
>
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

-- 
This e-mail message has been scanned and cleared by Google Message Security 
and the UNICOM Global security systems. This message is for the named 
person's use only. If you receive this message in error, please delete it 
and notify the sender. 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: TSO Pipes (and BatchPipes)

[Hobart Spitz]

If there are any z/VMers subscribed, it would be interesting to also
compare the PIPE Q output under CMS.

Thanks.

OREXXMan
JCL is the buggy whip of 21st century computing.  Stabilize it.
Put Pipelines in the z/OS base.  Would you rather process data in move mode
or locate mode?
IBM has been looking for an HLL for program products; REXX is that language.


On Wed, Apr 22, 2020 at 10:57 AM Lionel B Dyck  wrote:

> I don't have access to TSO Pipes other than what is included with
> BatchPipes
> but it seems that it is a tad dated - it is actually old enough to drink:
>
>
>
> BPW00086I BatchPipes/MVS BatchPipeWorks, 5655-065/SCPSP 1.0109
> (Version.Release/Mod) - Generated 12 Oct 1996 at 13:59:00
>
>
>
> If someone has the non-BP version of TSO Pipes it would be interesting to
> see the results of the PIPES Q command showing the date.
>
>
>
> For those of you who would like to encourage IBM to release TSO Pipes as
> part of the z/OS base please go to this URL and vote
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe
> <
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=47699
> > _ID=47699  (note the current vote is 188 which places it at the  top
> in
> the z/OS area).
>
>
>
> Having TSO Pipes as a native part of the z/OS distribution would help
> everyone as software developers would then be free to use it when
> developing
> applications, products, and tools knowing that every site has it available.
>
>
>
>
>
> Lionel B. Dyck <
> Website:  <https://www.lbdsoftware.com> https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is what
> you are, reputation merely what others think you are." - John Wooden
>
>
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: TSO Pipes (and BatchPipes)

[Jack J. Woehr]

On 4/22/20 9:56 AM, Lionel B Dyck wrote:

it is actually old enough to drink:


It's not that long until MVS itself is old enough to collect social 
security ...


--
Jack J. Woehr # Science is more than a body of knowledge. It's a way of
www.well.com/~jax # thinking, a way of skeptically interrogating the universe
www.softwoehr.com # with a fine understanding of human fallibility. - Carl Sagan

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

TSO Pipes (and BatchPipes)

[Lionel B Dyck]

I don't have access to TSO Pipes other than what is included with BatchPipes
but it seems that it is a tad dated - it is actually old enough to drink:

 

BPW00086I BatchPipes/MVS BatchPipeWorks, 5655-065/SCPSP 1.0109
(Version.Release/Mod) - Generated 12 Oct 1996 at 13:59:00

 

If someone has the non-BP version of TSO Pipes it would be interesting to
see the results of the PIPES Q command showing the date.

 

For those of you who would like to encourage IBM to release TSO Pipes as
part of the z/OS base please go to this URL and vote
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe
<https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=47699
> _ID=47699  (note the current vote is 188 which places it at the  top in
the z/OS area).

 

Having TSO Pipes as a native part of the z/OS distribution would help
everyone as software developers would then be free to use it when developing
applications, products, and tools knowing that every site has it available.

 

 

Lionel B. Dyck <
Website:  <https://www.lbdsoftware.com> https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what
you are, reputation merely what others think you are." - John Wooden

 


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: BatchPipes and TSO Pipes

[Paul Gilmartin]

On Sun, 22 Mar 2020 16:20:41 -0500, Lionel B Dyck wrote:

>Haven't looked into pipes in omvs (unix) but that is an interesting idea.
> 
You'll find a collection of utilities quite different from what you're used to.

It's tediious but possible to use omvs pipes to connect Classic OS utilities.
No one has shown me that TSO Pipelines can connect omvs utilities.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: BatchPipes and TSO Pipes

[Lionel B Dyck]

Haven't looked into pipes in omvs (unix) but that is an interesting idea.

thx


Lionel B. Dyck <
Website: http://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
scott Ford
Sent: Sunday, March 22, 2020 3:15 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: BatchPipes and TSO Pipes

Can’t you also use Unix Pipes? Just a thought Lionel

On Sun, Mar 22, 2020 at 1:52 PM Lionel B Dyck  wrote:

> Martin - thank you
>
>
> Lionel B. Dyck <
> Website: http://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is 
> what you are, reputation merely what others think you are." - John 
> Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On 
> Behalf Of Martin Packer
> Sent: Sunday, March 22, 2020 12:43 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: BatchPipes and TSO Pipes
>
> You're best of searching with the string "batchpipeworks" as that's 
> what it was called when SmartBatch was created. I believe the term 
> survived into BatchPipes/MVS V2.
>
> Not all CMS Pipelines functions are supported.
>
> Cheers, Martin
>
> Martin Packer
>
> zChampion, Systems Investigator & Performance Troubleshooter, IBM
>
> +44-7802-245-584
>
> email: martin_pac...@uk.ibm.com
>
> Twitter / Facebook IDs: MartinPacker
>
> Blog:
> https://www.ibm.com/developerworks/mydeveloperworks/blogs/MartinPacker
>
> Podcast Series (With Marna Walle): https://developer.ibm.com/tv/mpt/ 
> or
>
>
> https://itunes.apple.com/gb/podcast/mainframe-performance-topics/id112
> 794357
> 3?mt=2
> <https://itunes.apple.com/gb/podcast/mainframe-performance-topics/id11
> 27943573?mt=2>
>
>
> Youtube channel: 
> https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA
>
>
>
> From:   Lionel B Dyck 
> To: IBM-MAIN@LISTSERV.UA.EDU
> Date:   22/03/2020 17:24
> Subject:[EXTERNAL] Re: BatchPipes and TSO Pipes
> Sent by:IBM Mainframe Discussion List 
>
>
>
> That telecourse looks outstanding - thank you
>
>
> Lionel B. Dyck <
> Website:
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lbdsoftware.co
> m=Dw
>
> IFaQ=jf_iaSHvJObTbx-siA1ZOg=BsPGKdq7-Vl8MW2-WOWZjlZ0NwmcFSpQCLphNz
> nBSDQ&
>
> m=pvIHxGgF3aJff3x0OLzJEPZPeaTiZBaOn7cLSXVwS_4=JToobsms7DK2vd66akWU37
> MZPctq
> GvCvZcJf1BvVpUU=
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lbdsoftware.c
> om=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=BsPGKdq7-Vl8MW2-WOWZjlZ0NwmcFSp
> QCLphNznBSDQ=pvIHxGgF3aJff3x0OLzJEPZPeaTiZBaOn7cLSXVwS_4=JToobsms7
> DK2vd66akWU37MZPctqGvCvZcJf1BvVpUU=>
>
>
> "Worry more about your character than your reputation.  Character is 
> what you are, reputation merely what others think you are." - John 
> Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On 
> Behalf Of Dave Jones
> Sent: Sunday, March 22, 2020 11:51 AM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: BatchPipes and TSO Pipes
>
> You might find this of some use, Lionel :
>  CMS Pipelines Telecourse (an HTML selfstudy course) V1.3
>   
> https://www.vm.ibm.com/download/packages/descript.cgi?TCVM2
>
> DJ
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send 
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send 
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with 
> number 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 
> 3AU
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send 
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send 
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
--
Scott Ford
IDMWORKS
z/OS Development

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: BatchPipes and TSO Pipes

[scott Ford]

Can’t you also use Unix Pipes? Just a thought Lionel

On Sun, Mar 22, 2020 at 1:52 PM Lionel B Dyck  wrote:

> Martin - thank you
>
>
> Lionel B. Dyck <
> Website: http://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is what
> you are, reputation merely what others think you are." - John Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On Behalf
> Of
> Martin Packer
> Sent: Sunday, March 22, 2020 12:43 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: BatchPipes and TSO Pipes
>
> You're best of searching with the string "batchpipeworks" as that's what it
> was called when SmartBatch was created. I believe the term survived into
> BatchPipes/MVS V2.
>
> Not all CMS Pipelines functions are supported.
>
> Cheers, Martin
>
> Martin Packer
>
> zChampion, Systems Investigator & Performance Troubleshooter, IBM
>
> +44-7802-245-584
>
> email: martin_pac...@uk.ibm.com
>
> Twitter / Facebook IDs: MartinPacker
>
> Blog:
> https://www.ibm.com/developerworks/mydeveloperworks/blogs/MartinPacker
>
> Podcast Series (With Marna Walle): https://developer.ibm.com/tv/mpt/
> or
>
>
> https://itunes.apple.com/gb/podcast/mainframe-performance-topics/id112794357
> 3?mt=2
> <https://itunes.apple.com/gb/podcast/mainframe-performance-topics/id1127943573?mt=2>
>
>
> Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA
>
>
>
> From:   Lionel B Dyck 
> To: IBM-MAIN@LISTSERV.UA.EDU
> Date:   22/03/2020 17:24
> Subject:[EXTERNAL] Re: BatchPipes and TSO Pipes
> Sent by:IBM Mainframe Discussion List 
>
>
>
> That telecourse looks outstanding - thank you
>
>
> Lionel B. Dyck <
> Website:
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lbdsoftware.com=Dw
>
> IFaQ=jf_iaSHvJObTbx-siA1ZOg=BsPGKdq7-Vl8MW2-WOWZjlZ0NwmcFSpQCLphNznBSDQ&
>
> m=pvIHxGgF3aJff3x0OLzJEPZPeaTiZBaOn7cLSXVwS_4=JToobsms7DK2vd66akWU37MZPctq
> GvCvZcJf1BvVpUU=
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lbdsoftware.com=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=BsPGKdq7-Vl8MW2-WOWZjlZ0NwmcFSpQCLphNznBSDQ=pvIHxGgF3aJff3x0OLzJEPZPeaTiZBaOn7cLSXVwS_4=JToobsms7DK2vd66akWU37MZPctqGvCvZcJf1BvVpUU=>
>
>
> "Worry more about your character than your reputation.  Character is what
> you are, reputation merely what others think you are." - John Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On Behalf
> Of
> Dave Jones
> Sent: Sunday, March 22, 2020 11:51 AM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: BatchPipes and TSO Pipes
>
> You might find this of some use, Lionel :
>  CMS Pipelines Telecourse (an HTML selfstudy course) V1.3
>   https://www.vm.ibm.com/download/packages/descript.cgi?TCVM2
>
> DJ
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send email
> to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send email
> to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
-- 
Scott Ford
IDMWORKS
z/OS Development

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: BatchPipes and TSO Pipes

[Lionel B Dyck]

Martin - thank you


Lionel B. Dyck <
Website: http://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what
you are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of
Martin Packer
Sent: Sunday, March 22, 2020 12:43 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: BatchPipes and TSO Pipes

You're best of searching with the string "batchpipeworks" as that's what it
was called when SmartBatch was created. I believe the term survived into
BatchPipes/MVS V2.

Not all CMS Pipelines functions are supported.

Cheers, Martin

Martin Packer

zChampion, Systems Investigator & Performance Troubleshooter, IBM

+44-7802-245-584

email: martin_pac...@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog: 
https://www.ibm.com/developerworks/mydeveloperworks/blogs/MartinPacker

Podcast Series (With Marna Walle): https://developer.ibm.com/tv/mpt/or 
  
https://itunes.apple.com/gb/podcast/mainframe-performance-topics/id112794357
3?mt=2


Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From:   Lionel B Dyck 
To: IBM-MAIN@LISTSERV.UA.EDU
Date:   22/03/2020 17:24
Subject:[EXTERNAL] Re: BatchPipes and TSO Pipes
Sent by:IBM Mainframe Discussion List 



That telecourse looks outstanding - thank you


Lionel B. Dyck <
Website: 
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lbdsoftware.com=Dw
IFaQ=jf_iaSHvJObTbx-siA1ZOg=BsPGKdq7-Vl8MW2-WOWZjlZ0NwmcFSpQCLphNznBSDQ&
m=pvIHxGgF3aJff3x0OLzJEPZPeaTiZBaOn7cLSXVwS_4=JToobsms7DK2vd66akWU37MZPctq
GvCvZcJf1BvVpUU= 


"Worry more about your character than your reputation.  Character is what
you are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of
Dave Jones
Sent: Sunday, March 22, 2020 11:51 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: BatchPipes and TSO Pipes

You might find this of some use, Lionel :
 CMS Pipelines Telecourse (an HTML selfstudy course) V1.3
  https://www.vm.ibm.com/download/packages/descript.cgi?TCVM2

DJ

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@listserv.ua.edu with the message: INFO IBM-MAIN




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: BatchPipes and TSO Pipes

[Martin Packer]

You're best of searching with the string "batchpipeworks" as that's what 
it was called when SmartBatch was created. I believe the term survived 
into BatchPipes/MVS V2.

Not all CMS Pipelines functions are supported.

Cheers, Martin

Martin Packer

zChampion, Systems Investigator & Performance Troubleshooter, IBM

+44-7802-245-584

email: martin_pac...@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog: 
https://www.ibm.com/developerworks/mydeveloperworks/blogs/MartinPacker

Podcast Series (With Marna Walle): https://developer.ibm.com/tv/mpt/or 
  
https://itunes.apple.com/gb/podcast/mainframe-performance-topics/id1127943573?mt=2


Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From:   Lionel B Dyck 
To: IBM-MAIN@LISTSERV.UA.EDU
Date:   22/03/2020 17:24
Subject:[EXTERNAL] Re: BatchPipes and TSO Pipes
Sent by:IBM Mainframe Discussion List 



That telecourse looks outstanding - thank you


Lionel B. Dyck <
Website: 
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lbdsoftware.com=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=BsPGKdq7-Vl8MW2-WOWZjlZ0NwmcFSpQCLphNznBSDQ=pvIHxGgF3aJff3x0OLzJEPZPeaTiZBaOn7cLSXVwS_4=JToobsms7DK2vd66akWU37MZPctqGvCvZcJf1BvVpUU=
 


"Worry more about your character than your reputation.  Character is what 
you are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf 
Of Dave Jones
Sent: Sunday, March 22, 2020 11:51 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: BatchPipes and TSO Pipes

You might find this of some use, Lionel :
 CMS Pipelines Telecourse (an HTML selfstudy course) V1.3
  https://www.vm.ibm.com/download/packages/descript.cgi?TCVM2

DJ

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email 
to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: BatchPipes and TSO Pipes

[Lionel B Dyck]

That telecourse looks outstanding - thank you


Lionel B. Dyck <
Website: http://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Dave Jones
Sent: Sunday, March 22, 2020 11:51 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: BatchPipes and TSO Pipes

You might find this of some use, Lionel :
CMS Pipelines Telecourse (an HTML selfstudy course) V1.3
  https://www.vm.ibm.com/download/packages/descript.cgi?TCVM2

DJ

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: BatchPipes and TSO Pipes

[Dave Jones]

You might find this of some use, Lionel :
CMS Pipelines Telecourse (an HTML selfstudy course) V1.3
  https://www.vm.ibm.com/download/packages/descript.cgi?TCVM2

DJ

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: BatchPipes and TSO Pipes

[Paul Gilmartin]

On Sun, 22 Mar 2020 08:20:36 -0500, Lionel B Dyck wrote:

>I have access to a system where BP is installed and discovered that TSO
>Pipes works - to a limited degree.  CALLPIPE and ADDPIPE are missing even
>though both are clearly documented in the reference manual. Does anyone have
>any pointers to any doc that would document what parts of Pipes are missing
>with BP?
> 
First, ask on  http://vm.marist.edu/htbin/wlvindex?CMSPIP-L
Subscription info at bottom.  Also, try (2016):
http://www.vm.ibm.com/library/710pdfs/71625200.pdf

Dammit!  Google is getting nasty wrapping URLs so they can track!

Does anyone know how to associate a Pipelines connector
with an OMVS/Openextensions descriptor?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

BatchPipes and TSO Pipes

[Lionel B Dyck]

I have access to a system where BP is installed and discovered that TSO
Pipes works - to a limited degree.  CALLPIPE and ADDPIPE are missing even
though both are clearly documented in the reference manual. Does anyone have
any pointers to any doc that would document what parts of Pipes are missing
with BP?

 

Now that I can use Pipes under TSO I'm being reminded of its power that I
found during my few years using it under z/VM where I only had a very basic
introduction to it. Now I'm trying to learn more but finding a lack of
learning resources other than trial and error. There are some SHARE
presentations and some info on the Marist site that is helpful. Does anyone
have a pointer to a Pipes for novices?

 

 

Lionel B. Dyck <
Website:  <http://www.lbdsoftware.com/> http://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what
you are, reputation merely what others think you are." - John Wooden

 


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

PIPES (was Sdsf rexx)

[Dyck, Lionel B. (TRA)]

Check out the RFE for TSO Pipes that currently has 168 votes and is an 
'uncommitted candidate'. It really needs a solid business case for IBM to move 
forward is what I suspect.

If you search the RFE system for Servers and Systems Software and then product 
z/OS this is the TOP RFE by votes with Full ISPF Support for PDSE Member 
Generations a close 2nd.

https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=47699

--
Lionel B. Dyck (Contractor)  <
Mainframe Systems Programmer – RavenTek Solution Partners


-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf 
Of Hobart Spitz
Sent: Friday, May 11, 2018 10:09 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: [EXTERNAL] Re: [TSO-REXX] Sdsf rexx

Lionel:

You are so right.  That's why it needs to be in the base.

(There are sites that still have it.)

Ok, people!!!  Go out there and rattle some cages!!!  Your boss, your data
center manager, you sales rep., your SHARE rep., your neighbor's dog, and
anyone else you can think of.  (Skip your manager and/or the dog if they
bite.  :-)  )

LET'S GET PIPELINES IN THE Z/OS BASE!!!

OREXXMan
JCL is the buggy whip of 21st century computing.
We want Pipelines in the z/OS base.

On Fri, May 11, 2018 at 11:01 AM, Dyck, Lionel B. (TRA) <lionel.d...@va.gov>
wrote:

> Ask your friendly IBM rep, or search the IBM site, and you won't find it.
> It may be there but it isn't obvious and unless it is integrated it won't
> be exploited by ISVs (or IBM).
>
> --
> Lionel B. Dyck (Contractor)  <
> Mainframe Systems Programmer – RavenTek Solution Partners
>
>
> -Original Message-
> From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On
> Behalf Of Hobart Spitz
> Sent: Friday, May 11, 2018 9:57 AM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: [EXTERNAL] Re: [TSO-REXX] Sdsf rexx
>
> It's already integrated.  It's just not in the z/OS base.  And you have to
> pay extra.
>
> OREXXMan
> JCL is the buggy whip of 21st century computing.
> We want Pipelines in the z/OS base.
>
> On Fri, May 11, 2018 at 10:54 AM, Dyck, Lionel B. (TRA) <
> lionel.d...@va.gov>
> wrote:
>
> > PIPES - it would be very nice if IBM were to just integrate PIPES into
> > TSO/REXX and then we can move on and make z/OS an even better platform
> for
> > the future.
> >
> > 
> --
> > Lionel B. Dyck (Contractor)  <
> > Mainframe Systems Programmer – RavenTek Solution Partners
> >
> > -Original Message-
> > From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On
> > Behalf Of Hobart Spitz
> > Sent: Friday, May 11, 2018 9:48 AM
> > To: IBM-MAIN@LISTSERV.UA.EDU
> > Subject: [EXTERNAL] Re: [TSO-REXX] Sdsf rexx
> >
> > "pipe listpds" dsn "| stem members."
> >
> > OREXXMan
> > JCL is the buggy whip of 21st century computing.
> > We want Pipelines in the z/OS base.
> >
> > On Fri, May 11, 2018 at 9:00 AM, Carmen Vitullo <cvitu...@hughes.net>
> > wrote:
> >
> > > outrap with LISTD works well
> > > soemthing like this works well
> > >
> > >
> > >
> > > X = Listdsi(DSN 'DIRECTORY NORECALL')
> > > If X=16 Then Do
> > > If SYSREASON = 9 Then Do
> > > Say '** Data set has been migrated .. Abort'
> > > Exit
> > > End
> > > If SYSREASON = 5 Then Do
> > > Say '** Data set not cataloged .. Abort'
> > > Exit
> > > End
> > > Else Do
> > > Say '** Servere error in LISTDSI...Rc=16 - SYSREASON=' SYSREASON
> > > Exit
> > > End
> > > End
> > >
> > >
> > > or even LM services ?
> > > a clist example
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > CONTROL PROMPT NOFLUSH NOMSG END(ENDO)
> > > ISPEXEC CONTROL ERRORS RETURN
> > > CONTROL END(ENDO)
> > > ISPEXEC LMINIT DATASET('') +
> > > DATAID(PDS) ENQ(SHR) ORG(ORG)
> > > ISPEXEC LMOPEN DATAID() OPTION(INPUT)
> > > SET  =
> > > GETMBR: +
> > > ISPEXEC LMMLIST DATAID() OPTION(LIST) MEMBER(MEMPGM)
> > > SET  = 
> > > IF  = 0 THEN DO
> > > SET =()
> > > SET =(1:,)
> > > SET =
> > > ISPEXEC VPUT (MEMPGM)...
> > > and so on
> > >
> > >
> > >
> > > Carmen Vitullo
> > >
> > > -

Re: Pipes

[Cheryl Watson]

Hi Lionel,

Google brings up that Hobart has a current LinkedIn entry and this
interesting link which contains a paper from him on pipelines -
https://www.mail-archive.com/cms-pipelines@vm.marist.edu/msg02783.html.
Marty's also on LinkedIn.

Good luck!
Cheryl

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On
Behalf Of Dyck, Lionel B. (TRA)
Sent: Friday, April 7, 2017 9:07 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Pipes

Does anyone have a SHARE proceedings CD with these two presentations that
you would be willing to share:

"Plumbers Who Dump CVTs and the Control Blocks that Love Them:MVS/TSO
Pipelines for Systems Programmers", SHARE February 1999, Session 2875,
Hobart Spitz, Paine Webber

"Two Dimensional Pipes:How CMS Pipelines Differs from UNIX Pipes", SHARE
97, Summer 2001, Session 9109, Marty Zimelis, Computer Associates


--
Lionel B. Dyck
Mainframe Systems Programmer - TRA
Enterprise Operations (Station 200) (005OP6.3.10) Information and
Technology, IT Operations and Services


--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Pipes

[Nims,Alva John (Al)]

I have dug up "Plumbers Who Dump CVTs and the Control Blocks that Love Them:
MVS/TSO Pipelines for Systems Programmers", SHARE February 1999, Session 2875, 
Hobart Spitz, Paine Webber

For Lionel and will send it to him off-list and now starting the search for the 
CD/DVD that has it, but it is not looking good, I may not have any 2001 discs.

Al Nims
Systems Admin/Programmer 3
UFIT
University of Florida
(352) 273-1298

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf 
Of Dyck, Lionel B. (TRA)
Sent: Friday, April 07, 2017 9:07 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Pipes

Does anyone have a SHARE proceedings CD with these two presentations that you 
would be willing to share:

"Plumbers Who Dump CVTs and the Control Blocks that Love Them:MVS/TSO 
Pipelines for Systems Programmers", SHARE February 1999, Session 2875, Hobart 
Spitz, Paine Webber

"Two Dimensional Pipes:How CMS Pipelines Differs from UNIX Pipes", SHARE 
97, Summer 2001, Session 9109, Marty Zimelis, Computer Associates


--
Lionel B. Dyck
Mainframe Systems Programmer - TRA
Enterprise Operations (Station 200) (005OP6.3.10) Information and Technology, 
IT Operations and Services


--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Pipes

[Dyck, Lionel B. (TRA)]

Does anyone have a SHARE proceedings CD with these two presentations that you 
would be willing to share:

"Plumbers Who Dump CVTs and the Control Blocks that Love Them:MVS/TSO 
Pipelines for Systems Programmers", SHARE February 1999, Session 2875, Hobart 
Spitz, Paine Webber

"Two Dimensional Pipes:How CMS Pipelines Differs from UNIX Pipes", SHARE 
97, Summer 2001, Session 9109, Marty Zimelis, Computer Associates


--
Lionel B. Dyck
Mainframe Systems Programmer - TRA
Enterprise Operations (Station 200) (005OP6.3.10)
Information and Technology, IT Operations and Services


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COBOL and POSIX pipes

[Paul Gilmartin]

On Thu, 16 Feb 2017 19:32:37 -0800, Anne & Lynn Wheeler wrote:
>
>Unix traditional records are variable length deliminated by trailing
>null/zero byte.
> 
For text files on disk it was far more traditonal to use LF or  NL
rather than NUL as the delimiter.

>lots of traditional unix API programming would read w/o
>length restriction and common attack is to provide extremely long record
>that would overwrite end of buffer being used ... resulting failure
>and/or compromise. Lots of pressure to get UNIX (c language) programmers
>to use API that specify maximum length read.
>
read(), fread(), fgets(), and snprintf() all provide a maximum length
argument.  Among the antiquated outliers are gets() and sprintf().
Those should be neither used nor taught nowadays.

Traditional automobiles lacked seat belts, air bags, and collision
warning systems.  Modern automotive designers aren't allowed to
omit most of those features, although deregulation may change
that.  Few such restrictions apply to software designers.

And it's pointless to boast, repeatedly, about how much better
we are nowadays.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COBOL and POSIX pipes

[Anne & Lynn Wheeler]

john.archie.mck...@gmail.com (John McKown) writes:
> ​It adds some really nice features to legacy z/OS. But UNIX files can be
> confusing to z/OS programmers because they are more like "memory" than
> "disk" in that they are simply an ordered sequence of _bytes_, not
> _records_. The file system itself does not have _any_ interpretation o​f
> how those bytes are grouped into logical records. z/OS programmers, in
> general, are used to reading individual records when reading a data set.
> When you read a UNIX file, the program must tell the UNIX kernel ("access
> method") how many bytes you want to read. UNIX will return __NO MORE__ than
> that number of bytes. It could return fewer, if there are not that many
> left before "end of file" (an on some other rare occasions). Your code must
> then somehow know where the data you want (aka "this record") ends. Which
> means either the file is composed of fixed length records, hard coded in
> the program, or there is meta information encoded in the file data itself
> which indicates a length (similar to the LLBB field in a z/OS variable
> length data set) for each record. You can't rely on the system "handing"
> you a "logical record: when you do a UNIX read simply because there is no
> such thing, in a general sense.

Unix traditional records are variable length deliminated by trailing
null/zero byte. lots of traditional unix API programming would read w/o
length restriction and common attack is to provide extremely long record
that would overwrite end of buffer being used ... resulting failure
and/or compromise. Lots of pressure to get UNIX (c language) programmers
to use API that specify maximum length read.

I had this discussion with Dennis Ritchie that traditional IBM variable
length used two-byte explicit length prefixing ... while the unix method
saved a byte per record, with just a single (null/zero) byte postfixing
the record ... back in the days of "small memory" and pdp-7 ... birth of
unix
http://www.linfo.org/pdp-7.html

trivia: some of the CTSS people went to the 5th flr and did Multics and
others went to the ibm science center on the 4th floor and did virtual
machines, bunch of online applications, the internal network (also used
for university bitnet/earn) and invented GML. folklore is that some of
the AT worked on multics and then returned home and did unix
https://en.wikipedia.org/wiki/Multics#UNIX
dennis
https://en.wikipedia.org/wiki/Dennis_Ritchie

past posts mentioning ibm science center, 4th flr, 545 tech sq
http://www.garlic.com/~lynn/subtopic.html#545tech
CTSS
https://en.wikipedia.org/wiki/Compatible_Time-Sharing_System

I joke that when release 1 cp67 was delivered to the university in
Jan1968 it had some code that I completely rewrote ... but then saw very
similar code in unix 20yrs later (possibly all having traced back to
CTSS?). recent posts mentioning early cp67
http://www.garlic.com/~lynn/2017b.html#26 Virtualization's Past Helps Explain 
Its Current Importance
http://www.garlic.com/~lynn/2017b.html#27 Virtualization's Past Helps Explain 
Its Current Importance
http://www.garlic.com/~lynn/2017b.html#28 Virtualization's Past Helps Explain 
Its Current Importance
http://www.garlic.com/~lynn/2017b.html#29 Virtualization's Past Helps Explain 
Its Current Importance
http://www.garlic.com/~lynn/2017b.html#30 Virtualization's Past Helps Explain 
Its Current Importance
http://www.garlic.com/~lynn/2017b.html#32 Virtualization's Past Helps Explain 
Its Current Importance

I've also frequently commented that the original IBM mainframe TCP/IP
product was implemented in vs/pascal and *never* had any of the length
related vulnerabiilties and exploits that have been epidemic in C
language implementations. some past posts
http://www.garlic.com/~lynn/subintegrity.html#buffer

-- 
virtualization experience starting Jan1968, online at home since Mar1970

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COBOL and POSIX pipes (was: A design question)

[John McKown]

On Thu, Feb 16, 2017 at 7:49 PM, scott Ford  wrote:

> You guys are amazing. I havent used Unix on Z/OS much, a little. It has
> advantages for sure.
>
> Scott
>
>
​It adds some really nice features to legacy z/OS. But UNIX files can be
confusing to z/OS programmers because they are more like "memory" than
"disk" in that they are simply an ordered sequence of _bytes_, not
_records_. The file system itself does not have _any_ interpretation o​f
how those bytes are grouped into logical records. z/OS programmers, in
general, are used to reading individual records when reading a data set.
When you read a UNIX file, the program must tell the UNIX kernel ("access
method") how many bytes you want to read. UNIX will return __NO MORE__ than
that number of bytes. It could return fewer, if there are not that many
left before "end of file" (an on some other rare occasions). Your code must
then somehow know where the data you want (aka "this record") ends. Which
means either the file is composed of fixed length records, hard coded in
the program, or there is meta information encoded in the file data itself
which indicates a length (similar to the LLBB field in a z/OS variable
length data set) for each record. You can't rely on the system "handing"
you a "logical record: when you do a UNIX read simply because there is no
such thing, in a general sense.

The main drawback to z/OS UNIX is the relatively extreme overhead of doing
a fork() or spawn(). UNIX shells do those by the _ton_. Which explains the
BPXAS instances, which are the UNIX equivalent of a WLM batch initiator
(even runs the same program, IEFIIC, just with a special PARM= value).
Another drawback, to me, of z/OS UNIX is that the supplied UNIX tools (like
awk, find and /bin/sh) are _not_ as good as the GNU versions. Thankfully
Rocket Software has stepped into the gap with a nice port of many of them
for _no cost_, with optional paid support.

-- 
"Irrigation of the land with seawater desalinated by fusion power is
ancient. It's called 'rain'." -- Michael McClary, in alt.fusion

Maranatha! <><
John McKown

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COBOL and POSIX pipes (was: A design question)

[scott Ford]

You guys are amazing. I havent used Unix on Z/OS much, a little. It has
advantages for sure.

Scott

On Mon, Feb 13, 2017 at 1:35 PM, John McKown 
wrote:

> On Mon, Feb 13, 2017 at 12:28 PM, Paul Gilmartin <
> 000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
>
> > On Mon, 13 Feb 2017 11:58:57 -0600, John McKown wrote:
> > >
> > >...having already used
> > >a UNIX subroutine, I guess I figured it was easier to use another UNIX
> > >subroutine rather that setting up an FD along with an OPEN / READ loop /
> > >CLOSE.​
> > >
> > >... I think that the HLASM is basically
> > >being used as a "QSAM for RACF" type interface.​
> > >
> > If the replies returned by RACF are of uniform length or the assembler
> > program can pad them, BPX1RED is easiest.  I'm accustomed to
> > dealing with utility output where records are variable length,
> > contain no control characters, and are NL-terminated.  For that,
> > I might as well ALLOC and use the real QSAM.
> >
>
> ​Another point in your favor would be that the resultant code would be more
> understandable to a regular COBOL programmer. The only "oddity" would be
> the setup, in its calls to BPX1PIP and BPXWDYN.​
>
>
>
> >
> > -- gil
> >
> >
>
> --
> Our calculus classes are an integral part of your education.
>
> Maranatha! <><
> John McKown
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COBOL and POSIX pipes (was: A design question)

[John McKown]

On Mon, Feb 13, 2017 at 12:28 PM, Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:

> On Mon, 13 Feb 2017 11:58:57 -0600, John McKown wrote:
> >
> >...having already used
> >a UNIX subroutine, I guess I figured it was easier to use another UNIX
> >subroutine rather that setting up an FD along with an OPEN / READ loop /
> >CLOSE.​
> >
> >... I think that the HLASM is basically
> >being used as a "QSAM for RACF" type interface.​
> >
> If the replies returned by RACF are of uniform length or the assembler
> program can pad them, BPX1RED is easiest.  I'm accustomed to
> dealing with utility output where records are variable length,
> contain no control characters, and are NL-terminated.  For that,
> I might as well ALLOC and use the real QSAM.
>

​Another point in your favor would be that the resultant code would be more
understandable to a regular COBOL programmer. The only "oddity" would be
the setup, in its calls to BPX1PIP and BPXWDYN.​



>
> -- gil
>
>

-- 
Our calculus classes are an integral part of your education.

Maranatha! <><
John McKown

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COBOL and POSIX pipes (was: A design question)

[Paul Gilmartin]

On Mon, 13 Feb 2017 11:58:57 -0600, John McKown wrote:
>
>...having already used
>a UNIX subroutine, I guess I figured it was easier to use another UNIX
>subroutine rather that setting up an FD along with an OPEN / READ loop /
>CLOSE.​
>
>... I think that the HLASM is basically
>being used as a "QSAM for RACF" type interface.​
> 
If the replies returned by RACF are of uniform length or the assembler
program can pad them, BPX1RED is easiest.  I'm accustomed to
dealing with utility output where records are variable length,
contain no control characters, and are NL-terminated.  For that,
I might as well ALLOC and use the real QSAM.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COBOL and POSIX pipes (was: A design question)

[John McKown]

On Mon, Feb 13, 2017 at 11:51 AM, Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:

> On Mon, 13 Feb 2017 11:32:36 -0600, John McKown wrote:
> >
> >
> ​
>
> >
> I was thinking of BPX1PIP and BPXWDYN( "alloc path('/dev/fd/[pipe
> output]') ..." ).
> No generated pathname.
>

​Ah. I hadn't thought of that. An interesting idea, but having already used
a UNIX subroutine, I guess I figured it was easier to use another UNIX
subroutine rather that setting up an FD along with an OPEN / READ loop /
CLOSE.​



>
> >> Does BPX1ATM dup() descriptors as spawn() does?  If so the parent must
> >> close() its copy of the input descriptor or it never sees EOF.  Happens
> to
> >> me all the time.
> >
> >​BPX1ATM is "attach_and_exec_mvs", which basically seems, to me, to be a
> >wrapper around the ATTACHX function. So the HLASM program is a subtask of
> >the COBOL program. ​
> >
> I take that as "doesn't dup()."
>

​Right, I should have said that explicitly. Why would it? What use would a
dup() serve?​ Both tasks (aka "threads") are running in the same UNIX
process and so (should?) have access to all of that process' file
descriptors.



>
> >​There is a BPX1SEL (
> >https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.
> 0/com.ibm.zos.v2r1.bpxb100/sel.htm?view=kc
> >) which I guess could be used. But, at least to me, using the BPX1ATM is
> >conceptually easier. ...
> >
> You need something like select() if you want to monitor multiple
> input streams, such as stdout and stderr from a child process.
>

​Yes, but that is not the case here. I think that the HLASM is basically
being used as a "QSAM for RACF" type interface.​


>
> -- gil
>
>
-- 
Our calculus classes are an integral part of your education.

Maranatha! <><
John McKown

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COBOL and POSIX pipes (was: A design question)

[Paul Gilmartin]

On Mon, 13 Feb 2017 11:32:36 -0600, John McKown wrote:
>
>... The COBOL
>> >program would use BPX1RED to read from the read_file_descriptor. The HLASM
>> >
>> Is this better than using BPXWDYN('ALLOC ...') the descriptor and using
>> conventional COBOL I/O)?
>
>​Better? Not really. Just different. And doesn't require a MKFIFO with
>"random" name.​
> 
I was thinking of BPX1PIP and BPXWDYN( "alloc path('/dev/fd/[pipe output]') 
..." ).
No generated pathname.

>> Does BPX1ATM dup() descriptors as spawn() does?  If so the parent must
>> close() its copy of the input descriptor or it never sees EOF.  Happens to
>> me all the time.
>
>​BPX1ATM is "attach_and_exec_mvs", which basically seems, to me, to be a
>wrapper around the ATTACHX function. So the HLASM program is a subtask of
>the COBOL program. ​
> 
I take that as "doesn't dup()."

>​There is a BPX1SEL (
>https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.bpxb100/sel.htm?view=kc
>) which I guess could be used. But, at least to me, using the BPX1ATM is
>conceptually easier. ...
> 
You need something like select() if you want to monitor multiple
input streams, such as stdout and stderr from a child process.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COBOL and POSIX pipes (was: A design question)

[John McKown]

On Mon, Feb 13, 2017 at 10:58 AM, Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:

> On Mon, 13 Feb 2017 09:44:51 -0600, John McKown wrote:
> >
> >> 3.  Is it possible to have the Assembler routine place the extracted
> data into a unix
> >>  pipe or a C type=memory file and then i can retrieve it ?
> >
> >​You can call the UNIX BPX* routines from COBOL, so you can use something
> >like BPX1PIP to create an unnamed pipe. The COBOL program could then ​pass
> >the "write_file_descriptor" to the HLASM program as a parm. The COBOL
> >program would use BPX1RED to read from the read_file_descriptor. The HLASM
> >
> Is this better than using BPXWDYN('ALLOC ...') the descriptor and using
> conventional COBOL I/O)?
>

​Better? Not really. Just different. And doesn't require a MKFIFO with
"random" name.​



>
> >would use BPX1WRT to write to the passed in descriptor. But critically
> >important is that the HLASM program must be ATTACHX'd (BPX1ATM service can
> >do this) and not LINK'd to (as a normal COBOL CALL would do). Otherwise
> >
> Does BPX1ATM dup() descriptors as spawn() does?  If so the parent must
> close() its copy of the input descriptor or it never sees EOF.  Happens to
> me all the time.
>

​BPX1ATM is "attach_and_exec_mvs", which basically seems, to me, to be a
wrapper around the ATTACHX function. So the HLASM program is a subtask of
the COBOL program. ​



>
> >you'll most likely get a lock out with either the TCB either waiting in
> the
> >COBOL code for the HLASM to write some thing, or in the HLASM code waiting
> >(buffer full) for the COBOL to read something. I just wanted to throw this
> >last in for completeness. You might also need some way to communicate to
> >the COBOL routine that the HLASM routine has written an "EOF" and has
> ended.
> >
> close() the input end of the pipe should suffice.  COBOL should see EOF.
>

​Ah. I hadn't thought of that. Thanks for the info.


>
>
> On Mon, 13 Feb 2017 16:45:33 +0100, Peter Hunkeler wrote:
> >
> >Any you'd need to run the Cobol and the assembler routine in parallel,
> not as main routine calling a subroutine. UNIX pipes need both ends to be
> acitve for them to work.
> >
> I've sometimes cheated on this, mostly with stderr, assuming that
> any error messages will fit in a 131KB buffer.  Hard to handle both
> stdout and stderr otherwise in a Rexx parent.  Is there a select()
> SYSCALL?
>

​There is a BPX1SEL (
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.bpxb100/sel.htm?view=kc
) which I guess could be used. But, at least to me, using the BPX1ATM is
conceptually easier. Mainly because the HLASM routine would not need to
"return" to the COBOL routine except at the very end. Otherwise, Scott
could just do a normal CALL to "get the next record" and have the HLASM
routine be able to "read next" the RACF information, which would mean
saving "positioning" information over the CALL. I also got the impression
that the HLASM and COBOL are pretty much already written at using a
"socket" type interface is mainly to eliminate the use of an intermediate
"disk file buffer". It would also enhance security some because the data
would never make it to a disk device, just memory.

I was also thinking of the COBOL program passing a record buffer address to
the HLASM program, then coordinating access to the buffer using a semaphone
(may require two?) (on == ready to read; off == ready to write), but that
looked a bit more difficult to me.


>
> -- gil
>
>


-- 
Our calculus classes are an integral part of your education.

Maranatha! <><
John McKown

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

COBOL and POSIX pipes (was: A design question)

[Paul Gilmartin]

On Mon, 13 Feb 2017 09:44:51 -0600, John McKown wrote:
>
>> 3.  Is it possible to have the Assembler routine place the extracted data 
>> into a unix
>>  pipe or a C type=memory file and then i can retrieve it ?
>
>​You can call the UNIX BPX* routines from COBOL, so you can use something
>like BPX1PIP to create an unnamed pipe. The COBOL program could then ​pass
>the "write_file_descriptor" to the HLASM program as a parm. The COBOL
>program would use BPX1RED to read from the read_file_descriptor. The HLASM
>
Is this better than using BPXWDYN('ALLOC ...') the descriptor and using
conventional COBOL I/O)?

>would use BPX1WRT to write to the passed in descriptor. But critically
>important is that the HLASM program must be ATTACHX'd (BPX1ATM service can
>do this) and not LINK'd to (as a normal COBOL CALL would do). Otherwise
>
Does BPX1ATM dup() descriptors as spawn() does?  If so the parent must
close() its copy of the input descriptor or it never sees EOF.  Happens to
me all the time.

>you'll most likely get a lock out with either the TCB either waiting in the
>COBOL code for the HLASM to write some thing, or in the HLASM code waiting
>(buffer full) for the COBOL to read something. I just wanted to throw this
>last in for completeness. You might also need some way to communicate to
>the COBOL routine that the HLASM routine has written an "EOF" and has ended.
> 
close() the input end of the pipe should suffice.  COBOL should see EOF.


On Mon, 13 Feb 2017 16:45:33 +0100, Peter Hunkeler wrote:
>
>Any you'd need to run the Cobol and the assembler routine in parallel, not as 
>main routine calling a subroutine. UNIX pipes need both ends to be acitve for 
>them to work.
> 
I've sometimes cheated on this, mostly with stderr, assuming that
any error messages will fit in a 131KB buffer.  Hard to handle both
stdout and stderr otherwise in a Rexx parent.  Is there a select()
SYSCALL?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: What are pipes? [was:RE: Two RFE's to please vote for]

[Farley, Peter x23353]

Check the VM/CMS Command Reference on the IBM website for the PIPE command.  
More info including the PIPE author's version of the PIPE reference manual here:

http://vm.marist.edu/~pipeline/

HTH

Peter

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf 
Of Rick Troth
Sent: Friday, May 20, 2016 9:31 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Two RFE's to please vote for

What are these pipes of which you speak? They sound interesting. Do they 
stream information like lead and copper and PVC plumbing streams water? 
Do they come with couplers and tees and elbows and valves and filters?

-- 


This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please notify us immediately by e-mail and delete the message and any 
attachments from your system.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN