[
https://issues.apache.org/jira/browse/IMPALA-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang reassigned IMPALA-9457:
--------------------------------------
Assignee: (was: Vihang Karajgaonkar)
> Lazy start of disk threads in I/O manager
> -----------------------------------------
>
> Key: IMPALA-9457
> URL: https://issues.apache.org/jira/browse/IMPALA-9457
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Tim Armstrong
> Priority: Major
> Labels: ramp-up, supportability
>
> Currently DiskIoMgr starts all the I/O threads upfront for all supported
> filesystems. This means there are 100s of idle threads in most impalads that
> never do anything. It would be sensible to start the threads for a disk only
> when the first range is submitted. It's not immediately obvious where the
> best place to do this is. A couple of ideas:
> * Try to do it in ScheduleContext in a lightweight way, e.g. check an atomic
> to see if it's been initialised, then acquire a lock and create the threads
> if needed. Propagating the status if thread creation fails may be the tricky
> part
> * Start up one thread per disk, so I/O can always make progress, and start an
> extra thread per disk each time a range is pulled off the queue in
> DiskQueue::GetNextRequestRange() so that the number of threads ramps up as
> scan ranges are submitted. It could potentially be clever and try to track
> how many threads are parked and only create new threads if 0 threads are
> parked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
