Phil, thanks again for the helpful feedback and the great food for thought.
Your point is well taken. I'll try to break it down as best I can: MIPS or CPU usage, main memory or working set, and input/output operations. (I'd be happy if someone else has better information.) MIPS is dependent on at least three things: Arithmetic Logical Unit (ALU) usage, instruction pipeline activity, and cache misses. With Pipes, ALU and instruction pipeline usage would be more intense, while cache misses would be less so. Many characters, ones that the Pipes stages never even look at, won't even enter the cache and ALU. Some data can be processed through thousands of stages and only be cache missed a few times, because stages are typically dispatched one after the other to process the same record. Compared to UNIX piping, where every character of a text file must go through every stage, even if it is going to be ignored, Pipes is much faster. As Melinda Varian says (approximately): This is an advantage of file structure aware operating systems. I could include Pipes as systems software. Main memory (i.e., non-cache) and working set affect how fast instructions can be executed. As with characters, records can enter and leave the working set just a few times, and still be processed by thousands of stages. So the working sets of Pipes will mostly tend to be much lower. Stages like TAKE and DROP can do their work without ever referencing a single byte of the records being processed. (With UNIX head and tail, every character must be inspected by the CPU.) Finally, there is I/O, which is where Pipes and even UNIX piping really shine. The fastest input and output operations are the ones that never take place. Records moved from program to program in memory skip I/O. I/O generally involves some kind of mechanical movement, and which is far slower than the CPU, caches or real memory. Taken together, a set of Pipes stages, whether in a TSO PIPE command or in a BatchPipes Fitting on a DD, \will have a smaller average working set and spend less time in the system than alternatives. If we estimate costs as WS*ER (working set size times elapsed residency), even a conservative 50% reduction in each would mean about a 75% reduction in hardware usage. (WS*0.5)*(ER*0.5) = WS*ER*0.25. If you have been paying attention, I have purposely glossed over some details. Unreferenced characters may be cache loaded because they reside on the same cache line as referenced characters. Also, record length and blocking factor, among other things, affect how fast data can reach main memory, the caches, and then the ALU. On the average, I don't think these two details change the relative effects of Pipes processing versus conventional processing.. I suspect that UNIX piping was the inspiration for Pipes (A.K.A. CMS/TSO Pipelines and BatchPipesWorks) both in suggesting in-memory "I/O" as well as improvements to the concept. I hope this helps. OREXXMan Would you rather pass data in move mode (*nix piping) or locate mode (Pipes) or via disk (JCL)? Why do you think you rarely see *nix commands with more than a dozen filters, while Pipelines specifications are commonly over 100s of stages, and 1000s of stages are not uncommon. IBM has been looking for an HLL for program products; REXX is that language. On Mon, Sep 20, 2021 at 8:10 PM Phil Smith III <[email protected]> wrote: > Hobart Spitz wrote, in part: > > >This is a great comment. I hadn't given that much thought to the > question. > > >Not to split hairs, but I didn't say MIPS, I said hardware. > > >If I had to guess, MIPS usage might actually increase slightly, because > the > >Pipes dispatcher has to switch between stages twice for every record that > >is passed. > > > > Sure, just sayin' you'd want to be very clear about what you do mean. > > > > I'm not quite sure what you mean by "more MIPS but less hardware", though? > > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO IBM-MAIN > ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
