Ref: <[email protected]>
On 18 June 2019, Paul Gilmartin wrote:
> I have a similar impression of Pipelines. An example I may have used
> before in POSIX shell:
> { stage1; stage2; stage3; } | stage4
I understand this concatenates the output of the first three programs (in
order?)
and passes it as input to the fourth one. Cute. So what about input (it seems
the
first program in that set takes the input). So my pipeline would be
.. stage1 | append stage2 | append stage3 | stage4 ..
> I found an analogous example in one of the Plunging, IIRC. It has
> STAGESEPs, and connectors, and a FANIN. Ouch! It might just be
> dialect: ; is a STAGESEP; { } is a FANIN, and the connections
> are implied by the position of the stages.
Your mapping of the syntax is flawed. In your example the '|' connects the
stages,
just like the CMS Pipelines stage separator. The ';' (don't forget the last one)
in combination with the { } forms a kind of input manifold.
You could use STAGESEP ; and use the ; rather than the pipe character, but
that's
not what you have on UNIX. If you're Python fan, you might think the '.' is the
obvious choice to pipe processes, sometimes.
There are lots of ways to funnel streams; FANIN funnels them one stream after
the
other; GATHER takes records alternating from each stream; MERGE takes them based
on the key value in each stream; FANINANY takes whichever is available. Some
stages
like LOOKUP take records from different streams for different purposes.
In my CMS Pipelines introduction, it takes 30 minutes before we get to things
that
you can't do with UNIX piping. For those very simple things, CMS Pipelines
syntax
is a bit different, but not really harder. I don't see how the simple UNIX
syntax
could be used to compose real pipelines. Or something like NodeJS which may be
elegant for simple things, but unable to describe complicated things.
> I know; Pipelines is better for concurrent stages. Ive used it successfully,
> always with the manual open before me (even though it cant employ multiple
> cores but could it on TSO?)
It does not. CMS Pipelines has abstractions for TCP/IP and IUCV connections, so
it is easy to compose a pipeline refinery that has multiple CMS users involved
to do the work. The obvious reason would be to control access to data, but
you could also use it to dispatch heavy tasks over workers (even on different
systems). One z/OS you could also use BatchPipes to connect different parts of
your application running in parallel.
Sir Rob the Plumber
