[
https://issues.apache.org/jira/browse/TEZ-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated TEZ-1187:
-----------------------------------
Attachment: TEZ-1187.WIP.01.patch
[~sseth], Please take a look at this Shared Executor service:
* The service creates and uses a shared executor service underneath.
* It can create ScheduledExecutorService instances which behave almost like an
ExecutorService. The differences are:
- No support for a Single threaded executor service, which guarantees, serial
execution of tasks.
- The shutdownNow does not return the non-scheduled Runnables.
- No fair scheduling b/w various ScheduledExecutorService instances created
using the Shared Executor.
Currently the shared executor is a fixed thread pool executor. We can change it
to CachedThreadPoolExecutor.
[~rajesh.balamohan] Had concerns with both the thread pool:
* With FixedThreadPool, all the threads can go into blocked state and a task
which can unblock this will not be scheduled and can result in a deadlock.
* With cachedThreadPool, number of threads is not limited and can cause
unbounded thread growth resulting in too many threads and system could end up
in a crawl.
Another solution would be to create a thread pool which can monitor blocked
threads and if all are in blocked state it will allocate newer threads and
schedule tasks in them. And it will stop scheduling new tasks if too many
runnable threads are active at a given time. This custom thread pool can also
be used to ensure some fairness in scheduling of tasks.
> Share Thread pools between different tasks
> ------------------------------------------
>
> Key: TEZ-1187
> URL: https://issues.apache.org/jira/browse/TEZ-1187
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Harish Jaiprakash
> Attachments: TEZ-1187.WIP.01.patch
>
>
> Thread pools are used all over for fetchers, sort etc. When running a single
> task - this is already a problem, and gets worse when running multiple tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
