Adrian Crum wrote: > --- On Thu, 4/1/10, Adam Heath <[email protected]> wrote: >> Adrian Crum wrote: >>> --- On Thu, 4/1/10, Adam Heath <[email protected]> >> wrote: >>>> Adrian Crum wrote: >>>>> --- On Thu, 4/1/10, Adam Heath <[email protected]> >>>> wrote: >>>>>> Adrian Crum wrote: >>>>>>> I multi-threaded the data load by >> having one >>>> thread >>>>>> parse the XML files >>>>>>> and put the results in a queue. >> Another >>>> thread >>>>>> services the queue and >>>>>>> loads the data. I also multi-threaded >> the >>>> EECAs - but >>>>>> that has an issue >>>>>>> I need to solve. >>>>>> We need to be careful with that. >>>> EntitySaxReader >>>>>> supports reading >>>>>> extremely large data files; it doesn't >> read the >>>> entire >>>>>> thing into >>>>>> memory. So, any such event dispatch >> system >>>> needs to >>>>>> keep the parsing >>>>>> from getting to far ahead. >>>>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/BlockingQueue.html >>>> Not really. That will block the calling >> thread when >>>> no data is available. >>> Yeah, really. >>> >>> 1. Construct a FIFO queue, fire up n consumers to >> service the queue. >>> 2. Consumers block, waiting for queue elements. >>> 3. Producer adds elements to queue. Consumers >> unblock. >>> 4. Queue reaches capacity, producer blocks, waiting >> for room. >>> 5. Consumers empty the queue. >>> 6. Goto step 2. >> And that's a blocking algo, which is bad. > > Huh? You just asked for a blocking algorithm: "So, any such event dispatch > system needs to keep the parsing from getting to far ahead."
No, I didn't ask for a blocking algorithm. When the outgoing queue is full, the producer needs to pause itself, so that it's thread can be used for other things. Consider a single, shared thread pool, used system wide. There are only 8 threads available, as there are only 6 real cpus available. This thread pool is used to keep the system from getting overloaded, running too many things at once, and thrashing. If any of the work items being processed by one of these threads blocks, then the system will loose a thread for doing other work. And if A blocks on B, which blocks on C, then D, you've lost 4 threads.
