Hello Hobart,
Do you have any RFE already opened for that? I would surely vote for it
(plus this might be a good place to broadcast its ref. number)!
Thanks, Lucas
On Nov 16, 2017 14:36, "Hobart Spitz" <orexx...@gmail.com> wrote:
Someone on TSO-REXX wrote:
> Yes and if IBM would give us PIPES (BatchPipesWorks Product) then you can
use PIPE to read from a file into a STEM or STACK and vice-versa.
There have been other such posts.
For those who don't know TSO PIPEs is like an assembly line with reusable,
standardized stations. (You can create your own, on the fly even.)
IMHO, making TSO PIPEs part of the z/OS base is long, long overdue. Over
the last 20 years, I have been associated with two requirements for Pipes
in the OS base that have been submitted at SHARE, but which produced no
results. I have it on good authority that that if you take all the
requests for TSO Pipelines, sort in REXX, and other requirements addressed
by TSO PIPEs, that there would be about 75% positive RFE votes.
Background on the term "piping":
The early versions of UNIX were written on small machines (32K?!), with
slow/small disk drives. Writing temporary results out to disk was too slow
and inefficient, so they came up a pipe operator ( | ), which connected
the input of one command to the output of the previous one, thus avoiding
physical I/O.. E.g. *ls | wc*. The data passed thru an in-memory buffer
with both commands running, and UNIX dispatching each as needed. It also
allowed intermediate results to be very much larger than the available disk
space, since those intermediate results didn't have to be written to disk.
CMS/TSO Pipelines does piping somewhat like this and much more. Once
difference is that data doesn't have to move, record descriptors do that
job for unchanged records.
I was first exposed to this concept, when I was supporting a team of
previously UNIX-based developers working on TSO for the first time. One of
the team came to me and asked how to connect the output of one program to
the input of the next in JCL. On the whiteboard, I proceeded to write the
JCL for creating a temporary dataset in one job step (DD with DSN, UNIT,
DCB, DISP, etc.) and the JCL to read that dataset in the next job step.
When I turned around, the questioner was gone. A short time later, his
officemate came in with the same question. The original questioner thought
I was pulling his leg (since one could do that with a single character in
UNIX) and he had left before I was done.
For those that may not know:
- TSO PIPEs implements dynamic, record-level, deterministic piping that
is significantly more advanced than that on UNIX, Linux, and USS. (More
on
that below.)
- If installed, TSO PIPEs can be used by invoking the "pipe": command
anywhere that a TSO command can be used, including: READY Prompt, ISPF
option 6, ISPF command line with the TSO prefix, REXX EXECs, CLISTs, TSO
batch, and the ISPF service "select cmd(pipe ... )".
- PIPEs is part of the z/VM base but is optional under z/OS. Hence this
post.
- TSO PIPEs, TSO Pipelines, and BatchPipesWorks, are all names for
essentially the same product, which is roughly a subset of CMS Pipes.
(My
understanding is that TSO PIPEs is not the same as the USS command
"pipe";
you need to direct the command to TSO.)
- Under z/VM, and under z/OS where available, it is well integrated into
both environments.
- On a very high-level conceptual level, PIPEs are somewhat similar to
SQL and 4GLs in that many common, low level processing details (such as
initialization, looping, and end of data) are taken care of internally.
Thus, many types of selections, matchings, transformations, etc. can be
done with compact commands and frequently without convention programming
in
languages like COBOL, PL/I, and C/C++. (It is different from SQL and
4GLs
in that Pipes can process many kinds of data, not just one type of data
or
source.)
- PIPEs can read both USS text and most binary files and can convert
them to record orientation (address plus length descriptors).
- PIPEs can also be invoked from JCL thusly: // EXEC
PGM=PIPE,PARM='< IN.FILE|SORT|> OUT.FILE'
PIPEs in the z/OS base would greatly improve the competitiveness of
mainframe systems, while cutting development, hardware and support costs
for IBM, third-parties and customers. By having Pipes available to all
internal and external developers, efficiency and productivity would
improve. This would help IBM competitiveness and bottom line, as well as
that of mainframe sites, and software vendors. If you've been following
IBM after quarterly reports are covered in the news, you'll know that IBM
needs a shot in the arm like this. Results have been poor for many
quarters. Over the long term it would mean improved profitability for the
entire mainframe division. There might even be IBM customers who are saved
from moving off mainframes (because of costs) by such a move. In short,
making PIPEs part of the z/OS base would more than pay for itself in
reduced costs, retained customers, and additional sales. If I were an IBM
stockholder, a manager at a mainframe customer, or part of a third-party
development team, I would be asking for TSO PIPEs from anyone who would
listen, and maybe some who wouldn't. :-)
Costs to IBM would not be an issue. TSO PIPEs already exists as a subset
of CMS PIPEs. Even in it's current subset state, it would satisfy the vast
majority of requirements (if not all) to deliver the product with the base,
thus minimizing the additional cost.
UNIX and variants cannot achieve what PIPEs can. (I have some experience
with UNIX and Linux, but only a little exposure to USS, so feel free to
correct any misstatements or incorrect details, especially in this
section.)
- The *NIX byte stream model requires that every character of a text
file be inspected for end-of-record and forcing each and every character
to
the faster caches. There is no way to skip an entire record in a text
file. The PIPEs model of record orientation means that records can be
skipped, in their entirety, or by inspecting a part, without paging-in
and
cache-loading the whole record, should it span page or cache line
boundary. Certainly, the faster caches don't need to be flooded with
unneeded, unused bytes.
- In *NIX each and every byte must be copied from the sending process,
to the buffer, to the receiving process. In PIPEs, only the record
descriptor (address and length) gets copied, while the data remains in
place. (Actually, it's probable that the descriptor is passed by
address.)
- Where binary (non-text) files are used (e.g. for performance or random
access), this precludes the use of most *NIX/USS text-oriented built-in
filters, without extra effort. E.g., extra
encoding/translating/encapsulation
to prevent binary data from being interpreted as end-of-record sequences,
to ensure the logical records appear as such.
- *NIX/USS pipeing byte stream can be split only with difficulty (think
about one job step creating two temporary datasets), and, because the
data
flow is not deterministic, they can't be rejoined without special coding.
With PIPEs, splitting and rejoining streams is routine and useful,
because
the record flow characteristics are known and documented for built-in
stages.
PIPEs make REXX an even more compelling tool in batch. I would nominate
PIPEs as a candidate for the JCL of the 21st century., but that's a long
story for another time. In both conventional multi-step jobs and more
contemporary applications (e.g. ETL, data warehousing, etc.) multiple steps
can be achieved in a single command more efficiently and with less original
code than is generally done in z/OS today.
PIPEs has interfaces for DB2, PDS/Es, Byte Files, ISPF tables and
variables, as well as sequential files, to name a few.
If you have any doubt about the usefulness of Pipes, take a poll of your
z/VM friends. They would tell you that they couldn't live without Pipes,
not because z/VM otherwise lacks anything, but because Pipes is so powerful
and efficient.
I urge anyone and everyone who uses REXX or JCL to do what they can to make
bring PIpes to the z/OS base. Fee free to forward this post to anyone who
you think cares and/or who can help.
I hope that someone more vocal and persuasive than I can take up this issue
and get some needed action.
In the meantime, check to see if you have the TSO PIPE command already
installed. Type "[tso] pipe q"; if you get a version message, you have
it. Next step: "[tso] pipe help menu" and/or "[tso] pipe ahelp menu".
Failing that, you can convince your management to order and install TSO
PIPEs. Your site will benefit, and it might just be enough to get the
powers that be in IBM to wake up. The TSO PIPE command was available
separately from BatchPipes. I assume it still is.
IBM needs to get its act together. The stock is right back to where it was
before the z encryption announcement, Buffet is selling his IBM holding
(possibly related), and the previous quarters have been a long string of
disappointments. Putting PIPEs into the z/OS base may be just the step
needed.
My apologies for any errors, inaccuracies, or too harsh words.
Whew! Kudos if you read the whole thing.
- Hobart Spitz
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
