Adrian Crum wrote: > --- On Thu, 4/1/10, Adam Heath <[email protected]> wrote: >> Adrian Crum wrote: >>> --- On Thu, 4/1/10, Adam Heath <[email protected]> >> wrote: >>>> Adrian Crum wrote: >>>>> I multi-threaded the data load by having one >> thread >>>> parse the XML files >>>>> and put the results in a queue. Another >> thread >>>> services the queue and >>>>> loads the data. I also multi-threaded the >> EECAs - but >>>> that has an issue >>>>> I need to solve. >>>> We need to be careful with that. >> EntitySaxReader >>>> supports reading >>>> extremely large data files; it doesn't read the >> entire >>>> thing into >>>> memory. So, any such event dispatch system >> needs to >>>> keep the parsing >>>> from getting to far ahead. >>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/BlockingQueue.html >> Not really. That will block the calling thread when >> no data is available. > > Yeah, really. > > 1. Construct a FIFO queue, fire up n consumers to service the queue. > 2. Consumers block, waiting for queue elements. > 3. Producer adds elements to queue. Consumers unblock. > 4. Queue reaches capacity, producer blocks, waiting for room. > 5. Consumers empty the queue. > 6. Goto step 2.
And that's a blocking algo, which is bad. If you only have a limited number of threads, then anytime one of them blocks, the thread becomes unavailable to do real work. What needs to happen in these cases is that the thread removes it self from the thread pool, and the consumer thread then had to resubmit the producer. The whole point of SEDA is to not have unbounded resource usage. If a thread gets blocked, then that implies that another new thread will be needed to keep the work queue proceeding. > > > > >
