[
https://issues.apache.org/jira/browse/IMPALA-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-9255:
----------------------------------
Description:
In Planner.computeResourceReqs(), we assume that all fragment's peak resource
consumption occurs at the same time, resulting in higher than necessary
estimates. There are a few things that need to change here to address this
inefficiency:
* We should consider changing the backend logic so that fragments don't start
up and claim resources until their dependencies are ready.
* We need to change the backend logic so that resources can be released when
needed, e.g. the join build fragment's thread should exit once the build is
done, instead of waiting for the probe threads to all exit
* We need to update the planner to reflect the new behaviour.
was:
In Planner.computeResourceReqs(), we assume that all fragment's peak resource
consumption occurs at the same time. In actually, with the separate join build,
there are dependencies between fragments and we could take those into account
to reduce the estimated resource requirements.
> Avoid unnecessary concurrent resource consumption by fragments.
> ---------------------------------------------------------------
>
> Key: IMPALA-9255
> URL: https://issues.apache.org/jira/browse/IMPALA-9255
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Tim Armstrong
> Priority: Major
> Labels: multithreading
>
> In Planner.computeResourceReqs(), we assume that all fragment's peak resource
> consumption occurs at the same time, resulting in higher than necessary
> estimates. There are a few things that need to change here to address this
> inefficiency:
> * We should consider changing the backend logic so that fragments don't
> start up and claim resources until their dependencies are ready.
> * We need to change the backend logic so that resources can be released when
> needed, e.g. the join build fragment's thread should exit once the build is
> done, instead of waiting for the probe threads to all exit
> * We need to update the planner to reflect the new behaviour.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
