On Tue, Jul 14, 2009 at 1:54 AM, Ian S. Worthington<[email protected]> wrote:
> Redirecting the fanout via ?dup0:|add0: was the bit I was missing. I often > seem to stumble on that occasional requirement: is there any guidance > anywhere about when its required and when its not, and how to do it right? I > had tried something similar here, but it clearly wasn't quite the flavour I > needed. You just can't connect the stream to two different places. In a few cases it is only syntactical sugar, but normally you need to have this explicit control to determine in what order records are made available on each stream. This is why we can do pipes like this | x: fanout | copy | s: spec w1 1 select 1 w1 nw | .. \ x: | spec w2 | reverse | s: Since fanout writes the record first to the primary and then the secondary, the "copy" stage holds that record while the other copy of it goes to the secondary output, gets the 2nd word reversed, and made available to the "spec" stage. That one can, with both streams available then start to product its output record. If we could not reason about record delay here, this would not work. Some plumbers will, when they are up to their knees in the water and can't get the leak fixed, use "elastic" to fix the stalled pipeline. > Elastic: You're quite right in your guess as to how this got added. But, as I > said in a follow up note to Rob, this is a huge file being read off of tape > and buffer isn't going to work here. You use elastic when there's a variable amount of records to hold during the run of the pipeline. If you need to buffer all (like you have to with your lookup secondary input) you should use "buffer" instead. But from what I read it's doable without buffering all. You will need to buffer some records (for example the details when waiting for a corresponding master record). Oh, and you did not say whether the master matches more than one detail in your data - lookup expects so, and will hold the entire master table until the end of your input. Melinda's "plunging on" may help on multi-stream plumbing. And John's "CMS Pipelines Explained" has more about record delay. Rob
