[
https://issues.apache.org/jira/browse/IMPALA-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell reassigned IMPALA-14784:
--------------------------------------
Assignee: Joe McDonnell
> Switch end to end parallel tests to python-xdist's --dist=worksteal
> -------------------------------------------------------------------
>
> Key: IMPALA-14784
> URL: https://issues.apache.org/jira/browse/IMPALA-14784
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> Python-xdist added a "worksteal" distribution mode that can rebalance work
> between the threads when a thread goes idle. This is mainly useful towards
> the end of a parallel test run to reduce skew. On exhaustive end-to-end
> parallel tests, the last 2% of tests take hours:
> {noformat}
> 02:26:10
> ..............................xxx....................................... [
> 98%]
> 03:38:48
> ........................................................................ [
> 98%]
> 04:42:39
> ........................................................................ [
> 99%]
> 05:08:04
> ........................................................................ [
> 99%]
> 05:08:21 ....ss
> [100%]{noformat}
> Looking at the report log, there is massive skew. At the end, a single worker
> is working alone on its remaining tests while other workers are idle. Work
> stealing would make a big difference here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
