On Fri, Feb 19, 2010 at 4:15 PM, Alan Winson <[email protected]> wrote:
> about 20% more unique keys than reading one file. The lookup stage deletes > each detail record that is added and adds a new detail record with the same > key (and the updated sums). I guess the reason that storage grows is either I'll sit down with my pipeline wrench in a spare minute, but caution: when you delete the master from lookup, the memory is not freed. That has been on my wish list for a long time, but I doubt it will happen this year for my birthday. My paper on using lookup has an example of a pipeline that recycles the lookup stage now and then to free memory and reload the valid entries in the table. Rob
