Hi Jacques, I'm working on implementing the priority queue approach at the moment for a client. All things going well it will be in production in a couple of weeks and I'll report back then with a patch.
Regards Scott On Tue, 26 Feb 2019 at 03:11, Jacques Le Roux <jacques.le.r...@les7arts.com> wrote: > Hi, > > I put this comment there with OFBIZ-10002 trying to document why we have 5 > as hardcoded value of /max-threads/ attribute in /thread-pool/ element > (serviceengine.xml). At this moment Scott already mentioned[1]: > > /Honestly I think the topic is generic enough that OFBiz doesn't need > to provide any information at all. Thread pool sizing is not exclusive to > OFBiz and it would be strange for anyone to modify the numbers without > first researching sources that provide far more detail than a few sentences > in our config files will ever cover./ > > I agree with Scott and Jacopo that jobs are more likely IO rather than CPU > bounded. So I agree that we should take that into account, change the > current algorithm and remove this somehow misleading comment. Scott's > suggestion in his 2nd email sounds good to me. So If I understood well we > could > use an unbounded but finally limited queue, like it was before. > > Although with all of that said, after a quick second look it appears > that > the current implementation doesn't try poll for more jobs than the > configured limit (minus already queued jobs) so we might be fine with > an > unbounded queue implementation. We'd just need to alter the call to > JobManager.poll(int limit) to not pass in > executor.getQueue().remainingCapacity() and instead pass in something > like > (threadPool.getJobs() - executor.getQueue().size()) > > I'm fine with that as it would continue to prevent hitting physical > limitations and can be tweaked by users as it's now. Note that it seems > though > uneasy to tweak as we received already several "complaints" about it. > > Now one of the advantage of a PriorityBlockingQueue is priority. And to > take advantage of that we can't rely on "/natural ordering"/ and need to > implement Comparable (which does no seem easy). Nicolas provided some > leads below and this should be discussed. The must would be to have that > parametrised, of course. > > My 2 cts > // > > [1] https://markmail.org/message/ixzluzd44rgloa2j > > Jacques > > Le 06/02/2019 à 14:24, Nicolas Malin a écrit : > > Hello Scott, > > > > On a customer project we use massively the job manager with an average > of one hundred thousand job per days. > > > > We have different cases like, huge long jobs, async persistent job, fast > regular job. The mainly problem that we detect has been (as you notified) > > the long jobs that stuck poller's thread and when we restart OFBiz (we > are on continuous delivery) we hadn't windows this without crash some jobs. > > > > To solve try with Gil to analyze if we can load some weighting on job > definition to help the job manager on what jobs on the pending queue it can > > push on queued queue. We changed own vision to create two pools, one for > system maintenance and huge long jobs managed by two ofbiz instances and an > > other to manage user activity jobs also managed by two instances. We > also added on service definition an information to indicate the > predilection pool > > > > This isn't a big deal and not resolve the stuck pool but all blocked > jobs aren't vital for daily activity. > > > > For crashed job, we introduced in trunk service lock that we set before > an update and wait a windows for the restart. > > > > At this time for all OOM detected we reanalyse the origin job and tried > to decompose it by persistent async service to help loading repartition. > > > > If I had more time, I would be oriented job improvement to : > > > > * Define an execution plan rule to link services and poller without > touch any service definition > > > > * Define configuration by instance for the job vacuum to refine by > service volumetric > > > > This feedback is a little confused Scott, maybe you found interesting > things > > > > Nicolas > > > > On 30/01/2019 20:47, Scott Gray wrote: > >> Hi folks, > >> > >> Just jotting down some issues with the JobManager over noticed over the > >> last few days: > >> 1. min-threads in serviceengine.xml is never exceeded unless the job > count > >> in the queue exceeds 5000 (or whatever is configured). Is this not > obvious > >> to anyone else? I don't think this was the behavior prior to a > refactoring > >> a few years ago. > >> 2. The advice on the number of threads to use doesn't seem good to me, > it > >> assumes your jobs are CPU bound when in my experience they are more > likely > >> to be I/O bound while making db or external API calls, sending emails > etc. > >> With the default setup, it only takes two long running jobs to > effectively > >> block the processing of any others until the queue hits 5000 and the > other > >> threads are finally opened up. If you're not quickly maxing out the > queue > >> then any other jobs are stuck until the slow jobs finally complete. > >> 3. Purging old jobs doesn't seem to be well implemented to me, from what > >> I've seen the system is only capable of clearing a few hundred per > minute > >> and if you've filled the queue with them then regular jobs have to queue > >> behind them and can take many minutes to finally be executed. > >> > >> I'm wondering if anyone has experimented with reducing the queue the > size? > >> I'm considering reducing it to say 100 jobs per thread (along with > >> increasing the thread count). In theory it would reduce the time real > jobs > >> have to sit behind PurgeJobs and would also open up additional threads > for > >> use earlier. > >> > >> Alternatively I've pondered trying a PriorityBlockingQueue for the job > >> queue (unfortunately the implementation is unbounded though so it isn't > a > >> drop-in replacement) so that PurgeJobs always sit at the back of the > >> queue. It might also allow prioritizing certain "user facing" jobs > (such > >> as asynchronous data imports) over lower priority less time critical > jobs. > >> Maybe another option (or in conjunction) is some sort of "swim-lane" > >> queue/executor that allocates jobs to threads based on prior execution > >> speed so that slow running jobs can never use up all threads and block > >> faster jobs. > >> > >> Any thoughts/experiences you have to share would be appreciated. > >> > >> Thanks > >> Scott > >> > > >
