IMHO, the Business Cases on Pipes in the z/OS Base are as follows. (Pipes
is already available as part of BatchPipes.)
The case *for *Pipes in the z/OS base.:
1. Development costs would drop for customers, vendors, and IBM, since
everyone could use Pipes in their software.
2. Hardware usage would drop for customers. In addition to avoiding
I/O, Pipes uses a record address-and-length descriptor. A record can
flow from stage to stage with the only data movement being for changed
records. Potential data needed by a stage could have already been in the
working set and/or cache-loaded by the previous stage. (A methodology for
identifying the cost/benefits by JOB and application would allow the best
to be reworked first. Thus Pipes would pay for itself in the shortest
amount of time.)
3. Product efficiency for vendors (IBM and others) would improve.
(Arguably it's the other side of the coin in #2.)
4. Tight integration with REXX, CLIST and JCL.
5. Portability to and from z/VM. This breaks down differently for
different groups:
- Customers: Cheaper porting to/from z/OS. (Porting to other IBM
Series is expensive and time-consuming, AFAIK.)
- Vendors: Write once for both platforms.
- IBM: Rather than customers moving to non-IBM platforms, when z/OS
or z/VM don't meet their needs, those customers would have another option
to stay with IBM.
6. You can process both RecFM F records and RecFM V records with the
same stages.
7. Pipes can be used on both EXEC and DD JCL statements. This is
primarily for REXX-a-phobes. Pipe commands in REXX are amazing; I've used
the combination on both z/OS and z/VM (and predecessors). Pipes with
CLISTs is almost as good, AFAIK.
8. Increased competitiveness for IBM hardware and software. This would
especially apply to UNIX customers who have exceeded the capabilities of
their platforms.
9. CMS/TSO Pipes is better than UNIX piping, and REXX is better than C.
With today's processors, C's performance advantage over REXX is not
significant, and dwarfed by low developer productivity (your bullet, your
gun, your foot) of C. Strategically using Pipes with REXX, can give better
performance that UNIX piping and C.
10. Since both C++ and Pipes are internally pointer based, you could get
similar benefits by using OO techniques exclusively. How are you going to
convert a COBOL and JCL application to C++? Not as easily as going to REXX
and Pipes.
11. z/OS is a file-structure-aware operating system. It does not
use embedded control characters in text files or impose record boundary
knowledge in binary files. Pipes reads and writes byte stream data by
converting to and from record address-and-length descriptor format. This
means that, in most cases, any sequence of stages perform equally well on
RecFM F, RecFM V, byte stream text data and deblocked UNIX binary files.
12. Addresses staff shortages due to baby-boomer retirements and CoViD
impacts.
13. Reduces batch window requirements. With less I/O, JOBs finish
faster.
14. It's my understanding that there are people inside IBM who are
behind Pipes going into the z/OS base. Pipes is part of the z/VM base.
Can z/OS be that far behind?
15. Is consistent with policies for combating global warming of
customers, vendors and the public. Fewer CPU cycles wasted means less heat
to be dissipated and less electricity to be produced. The UNIX stream and
C string models are obsolete in this light; every character must go through
the CPU to get to the end of a record or string. Not so in Pipes or SAM
access methods. Rarely do you see a UNIX command with more than a dozen
filters; Pipes commands go into the 100s or 1000s of stages. In general,
the more stages the better the efficiency, since you are doing more work
with the same amount of I/O. If one reads the tea leaves, it seems
inevitable that there will be some kind of restriction on activities that
cause global warming; whether it's a tax, regulation, cap-and-trade, etc.,
I can't guess. The point is that it is likely that many heavy resource
using applications and processes will have to be converted to Pipes-like
mechanisms, on all platforms to fight global warming. The question is, do
you want to start now when it can be done as part of a well thought-out
cost-effective plan or do it in a rush later, when the costs and risks have
ballooned?
The case *a**gainst *Pipes, IMHO:
1. Short term loss of revenue to due better hardware utilization for the
same work. Because of improved competitiveness the dip should be short
lived if it happens at all. Not all customers will embrace Pipes at the
same rate or at the same time. On the other hand, there is always the
chance that a few successes turn into a snowball.
2. From the customer perspective, converting JCL and COBOL to REXX and
Pipes may not appear to justify the expense and/or the skills to do so may
be lacking. If a mechanism could be created to convert JCL to Pipes on the
fly that may change the picture. It's not obvious, but most sequential
datasets used to pass data from step to step could be replaced with a pipe,
provided you can come up with a way to start all steps in parallel, giving
each their own TIOT and other control blocks. Giving each step it's own
address space would require cross address space addressing. Blocks of
addresses would have to be reserved for each pipe stage step, and be
addressable, read only, by other pipe stage steps. I'm not up on all the
latest addressing mechanisms, but perhaps data-spaces would be applicable
for this; only data needs to be addressable between step stages.
3. AFAIK, the newer versions of Pipes are not SMP/E ready.
4. The funding and staffing are not available. This is probably due to
misguided politics, including pro-UNIX and hard-code LE factions.
5. The Pipe command is not well known or well marketed as part of
BatchPipes. This should change.
6. There is not enough customer management push or technical
understanding of the enormous advantages
The last points in each section may be the most important. Push from
customers concerned about global warming may be the straw that gets IBM to
do what is right.
Action items for anyone who is interested:
- Make your data center, budgeting and other management aware of how
they can save resources, time and money by installing BatchPipes.
- Bug your IBM representative information on the products BatchPipes and
TSO Pipelines. Too many of them don't even know what they are.
- Get a free trial installation of BatchPipes.
- Code up some Pipes and non-pipes timing and resource comparisons
Publish the results. If nothing else, try sorting a stem (many long
records), once the usual way and once by concatenation a null string each
time you assign a record to a new position. The latter should force the
original copy to be garbage collected when the time comes.
- If you are not familiar with Pipes, go to
http://vm.marist.edu/~pipeline/ . There is lots of documentation and
code. It's all free.
- Put in a requirement to package the REXX compiler together with Pipes
and/or put the compiler in the z/OS base. You don't get UNIX without a C
compiler. You don't get z/VM without the REXX compiler. Why does z/OS not
include the REXX compiler. (Do I smell an LE monkey wrench?)
- Request that TSO Pipes (a.k.a. BatchPipesWorks) be broken off from
BatchPipes and made available standalone. IMHO, the inter-JOB piping
capability of BatchPipes was a kludgy mistake, confirmed by the low
customer interest. Intra-JOB capability can be done with the TSO Pipe
command.
- Request that BatchPipes works run without a sub-system where the
inter-JOB functionality is not needed.
- Get OOREXX and play with pipe.rex and usepipe.rex. Compare the
pipe.rex performance to a non-pipe equivalent.
While I'm on the subject, kudos to Mike Cowlishaw (who created REXX) and
John Hartmann (who created Pipes).
OREXXMan
Would you rather pass data in move mode (*nix piping) or locate mode
(Pipes) or via disk (JCL)? Why do you think you rarely see *nix commands
with more than a dozen filters, while Pipelines specifications are commonly
over 100s of stages, and 1000s of stages are not uncommon.
IBM has been looking for an HLL for program products; REXX is that language.
On Wed, Sep 15, 2021 at 7:40 AM Paul Gilmartin <
[email protected]> wrote:
> On Wed, 15 Sep 2021 05:50:13 -0500, Lionel B. Dyck wrote:
>
> >z/OS REXX does have a PIPES but it is only available currently as part of
> SmartBatch (or is it BatchPipes).
> >
> >There is an RFE asking for PIPES for TSO that is the highest voted RFE at
> the moment with 237 votes.
> >
> >
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=47699
> >
> >I'm not holding my breath for IBM to do it - if they really wanted to
> enhance the z/OS REXX experience they would but that doesn't seem to be
> their goal.
> >
> TANSTAAFL. What's the business case for IBM?
>
> -- gil
>
> ----------------------------------------------------------------------
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to [email protected] with the message: INFO IBM-MAIN
>
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO IBM-MAIN
