On Fri, Feb 19, 2010 at 1:07 AM, Alan Winson <[email protected]> wrote:
> 3. I would expect the pipe to use the same amount of storage no matter how > many files are read. Those records should move through the pipe one at a > time, right? They are detail input to lookup, not master input. The pipe > works fine when only one or two files are read, but runs out of memory with > 11 inputs. It works with 11 inputs when I increase the amount of memory. Your "lookup autoadd" stores all records with unique key in the table, so its requirement depends on the total number of records read. Does not matter how many different files. If you only need the keys, you could use the KEYONLY option to reduce size of the table elements. And if the records arrive in sorted order or so, you might be able to use the "unique" stage (which needs to keep only one record in memory). With some more info about your input files and the required output, the plumbers gathered here may come up with half a dozen alternatives ;-) Sir Rob the Plumber
