Adrian Crum wrote: > --- On Thu, 4/1/10, Adam Heath <[email protected]> wrote: >> Adrian Crum wrote: >>> --- On Thu, 4/1/10, Adam Heath <[email protected]> >> wrote: >>>> Adrian Crum wrote: >>>>> --- On Thu, 4/1/10, Adam Heath <[email protected]> >>>> wrote: >>>>>> Adrian Crum wrote: >>>>>>> --- On Thu, 4/1/10, Adam Heath <[email protected]> >>>>>> wrote: >>>>>>>> Adrian Crum wrote: >>>>>>>>> --- On Thu, 4/1/10, Adam Heath >> <[email protected]> >>>>>>>> wrote: >>>>>>>>>> Adrian Crum wrote: >>>>>>>>>>> I multi-threaded the >> data load >>>> by >>>>>> having one >>>>>>>> thread >>>>>>>>>> parse the XML files >>>>>>>>>>> and put the results in >> a >>>> queue. >>>>>> Another >>>>>>>> thread >>>>>>>>>> services the queue and >>>>>>>>>>> loads the data. I >> also >>>> multi-threaded >>>>>> the >>>>>>>> EECAs - but >>>>>>>>>> that has an issue >>>>>>>>>>> I need to solve. >>>>>>>>>> We need to be careful with >> that. >>>>>>>> EntitySaxReader >>>>>>>>>> supports reading >>>>>>>>>> extremely large data >> files; it >>>> doesn't >>>>>> read the >>>>>>>> entire >>>>>>>>>> thing into >>>>>>>>>> memory. So, any such >> event >>>> dispatch >>>>>> system >>>>>>>> needs to >>>>>>>>>> keep the parsing >>>>>>>>>> from getting to far >> ahead. >>>>>>>>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/BlockingQueue.html >>>>>>>> Not really. That will block >> the >>>> calling >>>>>> thread when >>>>>>>> no data is available. >>>>>>> Yeah, really. >>>>>>> >>>>>>> 1. Construct a FIFO queue, fire up n >> consumers >>>> to >>>>>> service the queue. >>>>>>> 2. Consumers block, waiting for queue >>>> elements. >>>>>>> 3. Producer adds elements to queue. >> Consumers >>>>>> unblock. >>>>>>> 4. Queue reaches capacity, producer >> blocks, >>>> waiting >>>>>> for room. >>>>>>> 5. Consumers empty the queue. >>>>>>> 6. Goto step 2. >>>>>> And that's a blocking algo, which is bad. >>>>> Huh? You just asked for a blocking algorithm: >> "So, any >>>> such event dispatch system needs to keep the >> parsing from >>>> getting to far ahead." >>>> >>>> No, I didn't ask for a blocking algorithm. >> When the >>>> outgoing queue is >>>> full, the producer needs to pause itself, so that >> it's >>>> thread can be >>>> used for other things. >>> I guess you could make the producer consume a queue >> element, then try adding the new one again. So: >> >> Nope, not good enough. It would be possible for the >> producer thread >> to stuck for a long time, producing/consuming. If >> there are several >> such workflows like this in the thread pool, then the >> threads become >> unavailable for doing other work. > > Are we talking about theoretical software or OFBiz? What thread pool? The > application server's? I have been referring to the existing OFBiz entity > import/export code. If an entity import takes n mS in the current > single-threaded code, and the same import takes n/x mS using multi-threaded > code, then hasn't the performance improved?
Data loading can take place from webtools. And several requests could be submitted at once. There's no reason to try and process them all at the same time, if the cpu is loaded. Just queue up the requests. Plus(this part is theorhetical), when ofbiz is more segmented, other things would go thru same pool. And thrashing would be reduced. I'm not suggesting we go thru and change ofbiz to some kind of segmented event dispatcher. But the basic infrastructure is simple enough to write, it doesn't hurt to do it right in the first place. > >> CPU is a limited resource. > > CPUs are cheap. Just buy more. ;-) Go survive a slashdotting.
