Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24035
Change subject: IMPALA-14784: Upgrade to python-xdist==3.5.0 and use --dist=worksteal ...................................................................... IMPALA-14784: Upgrade to python-xdist==3.5.0 and use --dist=worksteal On exhaustive jobs, the end-to-end parallel tests show enormous skew. The last 1-2% of tests takes hours, and logs indicate that the last 1257 tests execute on a single worker. pytest-xdist introduced a 'worksteal' algorithm in 3.2.0 that can rebalance the work. Exhaustive end-to-end parallel tests take about 5:45 right now with Python 3.8. A Python 3.11 run with the worksteal policy ran them in 2:39. The improvement on core exhaustive tests would be much smaller, because it doesn't suffer the same level of skew. Testing: - Ran end-to-end exhaustive tests - Ran a core job Change-Id: I6916bbef94b380a516356763dfabb3777c682637 --- M infra/python/deps/requirements.txt M tests/run-tests.py 2 files changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/24035/1 -- To view, visit http://gerrit.cloudera.org:8080/24035 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6916bbef94b380a516356763dfabb3777c682637 Gerrit-Change-Number: 24035 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell <[email protected]>
