[
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489526#comment-16489526
]
Peter Bacsko commented on OOZIE-3160:
-------------------------------------
Looks like we have to do the following if we have to avoid busy waiting:
1. Have a {{ScheduledThreadPoolExecutor}} per priority level
2. We submit tasks in a wrapped {{Callable}} to these executors depending on
priority
3. There's a separate thread for each priority where we call {{queue.take()}}.
In our case, the queue will be a {{DelayWorkQueue}} which blocks {{take()}}
until an item expires.
#3 looks like a limitation, however we only have 3 priority levels in Oozie, so
it isn't really a problem.
Since we have to deal with priorities, we need a common {{ExecutorService}}
where the queue is not an ordinary blocking queue, but a
{{PriorityBlockingQueue}} (this comes with the JDK). This ensures that
submitted tasks are ordered based on their priority. So higher priority tasks
will run before lower ones.
Problem is, there is no anti-starvation logic in {{PriorityBlockingQueue}}. To
avoid this, we have to track the added elements in a separate data structure.
This could be a set of priority queues, which we peek at certain points. If a
certain item reached the anti-starvation threshold, we do a promotion inside
the main {{PriorityBlockingQueue}} (remove then re-add it with a higher
priority).
Whoah, this is a big pain. There must be an easier solution to this... I know I
do overcomplicate simple things from time to time.
[~gezapeti], [~andras.piros], [~rkanter] any thoughts?
> PriorityDelayQueue put()/take() can cause significant CPU load due to busy
> waiting
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
> Issue Type: Bug
> Components: core
> Environment: all platforms
> Reporter: jj
> Assignee: Peter Bacsko
> Priority: Major
> Attachments: 11111111111111.png, 222222222222222222.png,
> OOZIE-3160-POC01.patch
>
>
> oozie process always consume high cpu. in my mechine,around 10%.
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement<E> take() throws InterruptedException {
> QueueElement<E> e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
