Many thanks Glen. Redirecting the fanout via ?dup0:|add0: was the bit I was missing. I often seem to stumble on that occasional requirement: is there any guidance anywhere about when its required and when its not, and how to do it right? I had tried something similar here, but it clearly wasn't quite the flavour I needed.
Elastic: You're quite right in your guess as to how this got added. But, as I said in a follow up note to Rob, this is a huge file being read off of tape and buffer isn't going to work here. And yes, I really need to cure my aversion to uppercase command names. But it bites me so rarely (ever?) and I hate uppercase so much, that it's difficult. I'll sign up for rehab one day. "My name is Ian and its been 7 days since my last lowercase command..." i ------ Original Message ------ Received: 05:55 PM COT, 07/13/2009 From: Glenn Knickerbocker <[email protected]> To: [email protected] Subject: Re: Pipeline fanin problem > "Ian S. Worthington" wrote: > > What I don't appear to be able to get right is the trivial bit: Files STARTLU > > and START000 need to be fanned together into a single file, but all my > > attempts to do so seem to end in label redefinition hell. > > I'm guessing what you're probably missing is that you need a FANOUT to > make two copies of file START000, one to feed to LOOKUP and the other to > fan into your output: > > 'PIPE (end ?)', > ' <' fn ' other ' ofm, > '|start: nlocate 41.5 ,START, ', > '|nstart: faninany ', > '| buffer', > '|l2: lookup 18.8 detail', /* details in; matches on luname, > write detail records */ > '|add0: fanin 1 0', > '| >' fn ' startlu ' ofm, > '?start:', > '|0000: nlocate 33.4 ,0000, ', > '|nstart:', > '?0000:', > '|dup0: fanout' , > '|l2:', /* masters in; unref details out */ > '| >' fn 'nstartlu' ofm, > '?dup0:', > '|add0:' > > I used BUFFER instead of ELASTIC because there's no need to try to read > any of the file before EOF on the input.(*) Note that the entire master > file will come at the top of your output. If you want the records in > the original order, since your LOOKUP compares the same columns in the > detail and master records, you can feed the master records back into the > file before the LOOKUP instead of after, and they'll always be selected: > > .. > '?dup0:', > '|nstart:' > > If your input file is on disk, though, you might as well just read it > twice instead, since reading a file from disk a second time with < is > generally cheaper than buffering it in storage: > > 'PIPE (end ?)', > ' <' fn ' other ' ofm, > '|l2: lookup 18.8 detail', /* details in; matches on luname, > write detail records */ > '| >' fn ' startlu ' ofm, > '? <' fn ' other ' ofm, > '| locate 41.5 ,START,', > '| locate 33.4 ,0000, ', > '|l2:', /* masters in; unref details out */ > '| >' fn 'nstartlu' ofm > > (*) ELASTIC generally says to me, "This is stalling and I don't know > why, so I'll just keep throwing in slop until it runs." There are > specific situations where it's really needed, but usually either > BUFFER or COPY is more appropriate. > > Also, the lowercase "pipe" indicates that you must not be using > ADDRESS COMMAND, so you'll stumble on any execs with the same name > as commands you're issuing. Every time I think I should rename my > PIPE EXEC, I decide it's worth keeping around to detect this and > alert exec authors to it--especially since I tend to inherit a lot > of the code I use. > > ¬R >
