dsimcha Wrote:
> == Quote from Jerry Quinn ([email protected])'s article
> > Timon Gehr Wrote:
> > > On 09/12/2011 07:23 PM, Jerry Quinn wrote:
> > > > I'm looking at porting an app that maintains a work queue to be
> > > > processed by
> one of N engines and written out in order. At first glance, std.parallelism
> already provides the queue, but the Task concept appears to assume that
> there's no
> startup cost per thread.
> > > >
> > > > Am I missing something or do I need to roll a shared queue object?
> > >
> > > I don't know if I get you right, but std.parallelism uses a task pool.
> > > Usually no threads are started or stopped during processing.
> > OK I guess what I'm looking for is WorkerLocalStorage. I can create an
> > engine
> per thread. However, I probably need to have each thread do the
> initialization
> work. If I create the engine on the main thread, it won't be properly
> accessed by
> the worker thread, right? I.e. I need each thread to run engine.init() which
> will
> do a whole pile of loading and setup first before I can start feeding data to
> the
> pool.
> > The impression I get is that
> > for (int i=0; i < nthreads; i++)
> > taskPool.workerLocalStorage(new engine(datafile))
> > will not get me what I want.
>
> The parameter for taskPool.workerLocalStorage is lazy, so it's even easier:
>
> taskPool.workerLocalStorage(new engine(datafile));
>
> will create one new engine **per thread** because the lazy parameter is
> passed to
> it and evaluated **once per thread**. I'm not sure what an engine is in this
> context, though.
An engine is simply a complex class that takes a lot of memory and setup work
to prepare. I'm working on language processing, so we use large multiGB models.
Great, I'm getting closer. So the evaluation happens in the worker thread, not
the main thread, right?
The other thing I think I have to do is sequence the initializations.
Otherwise the disk/network will thrash trying to load N copies simultaneously.
I imagine this can be done with a shared variable inside the engine.
> As far as initialization, you could do something like:
>
> auto isInitialized = taskPool.workerLocalStorage(false);
>
> void doStuff() {
> if(!isInitialized.get) {
> engines.get.initialize();
> isInitialized.get = true;
> }
>
> // Do some real processing.
> }
I prefer to do all the initializations before work starts so that timing can be
more accurate. These things take a lot to get fully loaded and ready to run.
Thanks for all the help! It might be useful to indicate in the workerLocalData
call that the lazy processing happens in the worker's thread.
Jerry
