On 2 February 2018 at 03:11, Paul Raulerson <[email protected]> wrote:
> > Timing is usually done with signal and/or semaphores - or better yet with > message > queues. :) > With 'relative timing' I mean the flow of records in two parallel paths, for example selecting a subset of the records to modify, and then combine the two streams again. When stages would buffer records, you would need to number records so you merge in the right order. I don't suppose you would seriously suggest implementing a messaging system to pass records between stages in the pipeline that each take a few dozen instructions to do their work. For one of the things I work on, the application runs some 15,000 stages in parallel (just a 64K buffer between them would add up to 1 GB). It's true that systems are 3-4 orders of magnitude larger than when the dispatcher in CMS Pipelines was written, but it has been demonstrated that a 'straightforward' implementation in Java even today is too resource intensive to be practical. Rob
