Hi Scott,

It seems we have some issues currently with our job scheduler [1]
which seems to be some sort of memory leak. We are also experiencing
some performance issues and other anomalies. It seems like a good time
to perhaps revisit the whole thing.

Are you suggesting to replace LinkedBlockingQueue with
PriorityBlockingQueue? If so I think it might actually be a better
option. I think being unbounded _might_ actually resolve some of the
pain points we're facing. I didn't get why it's not a drop-in
replacement though. It matches the signature of the call in the
executor service unless i'm missing something somewhere?

[1] https://issues.apache.org/jira/browse/OFBIZ-10592

On Wed, Jan 30, 2019 at 10:59 PM Scott Gray
<scott.g...@hotwaxsystems.com> wrote:
> Hi folks,
> Just jotting down some issues with the JobManager over noticed over the
> last few days:
> 1. min-threads in serviceengine.xml is never exceeded unless the job count
> in the queue exceeds 5000 (or whatever is configured).  Is this not obvious
> to anyone else?  I don't think this was the behavior prior to a refactoring
> a few years ago.
> 2. The advice on the number of threads to use doesn't seem good to me, it
> assumes your jobs are CPU bound when in my experience they are more likely
> to be I/O bound while making db or external API calls, sending emails etc.
> With the default setup, it only takes two long running jobs to effectively
> block the processing of any others until the queue hits 5000 and the other
> threads are finally opened up.  If you're not quickly maxing out the queue
> then any other jobs are stuck until the slow jobs finally complete.
> 3. Purging old jobs doesn't seem to be well implemented to me, from what
> I've seen the system is only capable of clearing a few hundred per minute
> and if you've filled the queue with them then regular jobs have to queue
> behind them and can take many minutes to finally be executed.
> I'm wondering if anyone has experimented with reducing the queue the size?
> I'm considering reducing it to say 100 jobs per thread (along with
> increasing the thread count).  In theory it would reduce the time real jobs
> have to sit behind PurgeJobs and would also open up additional threads for
> use earlier.
> Alternatively I've pondered trying a PriorityBlockingQueue for the job
> queue (unfortunately the implementation is unbounded though so it isn't a
> drop-in replacement) so that PurgeJobs always sit at the back of the
> queue.  It might also allow prioritizing certain "user facing" jobs (such
> as asynchronous data imports) over lower priority less time critical jobs.
> Maybe another option (or in conjunction) is some sort of "swim-lane"
> queue/executor that allocates jobs to threads based on prior execution
> speed so that slow running jobs can never use up all threads and block
> faster jobs.
> Any thoughts/experiences you have to share would be appreciated.
> Thanks
> Scott

Reply via email to