On 25 May 2012, at 12:16, Sanne Grinovero wrote: > On 25 May 2012 12:10, Manik Surtani <[email protected]> wrote: >> Well, the start() bit maybe less useful, but how do you know when the >> processor has been fed everything it needs, to be able to clean up? > > Too much usage of separate threads is blinding you from usage of known > patterns :-)
ROFL! Good point. > The Processor is a Visitor, so processEntriesWith( processor ) has the > CacheLoader invoke on itself, so it's all being coordinated by the > same thread which did initialize() and will to #shutdownWorkers() > after all entries are known to have been loaded and processed. > > If the Processor is using an internal async queue, that's not known to > the API contract, the shutdownWorkers() implementation will have to > deal with it but that's not hard as it's invoked after all entries > having been pushed to such queue. > >> >> On 25 May 2012, at 12:02, Sanne Grinovero wrote: >> >>> On 25 May 2012 11:33, Manik Surtani <[email protected]> wrote: >>>> Yes, as a one-off, but there should be a mechanism to set up internal >>>> structures and clean up/send finalisation messages to Hibernate Search or >>>> completion RPCs, etc. >>> >>> Ah got it. Maybe we need it, but I was initally - maybe naively - >>> expecting to deal with initialization myself : >>> >>> MassIndexingWorkCollector pwc = new MassIndexingWorkCollector(); >>> //implements Processor >>> pwc.initialize(.custom stuff..) //not defined on Processor >>> [cacheLoader?].processEntriesWith(pwc); // Blocking! so we know when >>> we finished loading all entries. >>> pwc.shutdownWorkers(); //not defined on Processor >>> >>> minimal API ;-) >>> But I guess when implementing for real I might need something like that. >>> >>> Sanne >>> >>>> >>>> On 25 May 2012, at 11:31, Sanne Grinovero wrote: >>>> >>>>> On 25 May 2012 10:57, Manik Surtani <[email protected]> wrote: >>>>>> #processEntriesWith(Processor p) >>>>>> >>>>>> Processor extends Lifecycle { // Lifecycle for start() and stop() >>>>>> methods… >>>>>> void process(CacheEntry e); >>>>>> void process(Collection<CacheEntry> e); >>>>>> boolean processMoreEntries(); >>>>>> } >>>>> >>>>> why the LifeCycle start/stop ? >>>>> I expect to use it as a one-off, not as something which is >>>>> "permanently hooked": looks like you' re thinking about a different >>>>> problem? >>>>> >>>>> The use case I'm thinkin of is when we need to iterate on all entries >>>>> in the cachestore, such as : >>>>> - Map/Reduce >>>>> - evaluating the average value of some attribute >>>>> - word counting >>>>> - MassIndexer >>>>> >>>>> In all the use cases I'm having in mind, you want to process all >>>>> entries, and only once. >>>>> So #processMoreEntries would be redundant, and I think we should >>>>> choose just one between CacheEntry or Collection<CacheEntry>.. let's >>>>> go with the simple CacheEntry ? >>>>> Should be able to avoid creation of short lived collections, and when >>>>> passing collections one would likely need to iterate on each element >>>>> anyway to route the invocation so some internal process(CacheEntry e); >>>>> >>>>> -- Sanne >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> [email protected] >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Manik Surtani >>>> [email protected] >>>> twitter.com/maniksurtani >>>> >>>> Lead, Infinispan >>>> http://www.infinispan.org >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> [email protected] >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> [email protected] >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> [email protected] >> twitter.com/maniksurtani >> >> Lead, Infinispan >> http://www.infinispan.org >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> [email protected] >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > [email protected] > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani [email protected] twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org _______________________________________________ infinispan-dev mailing list [email protected] https://lists.jboss.org/mailman/listinfo/infinispan-dev
