Hello Noemi Pap-Takacs, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24035
to look at the new patch set (#2).
Change subject: IMPALA-14784: Upgrade to python-xdist==3.5.0 and use
--dist=worksteal
......................................................................
IMPALA-14784: Upgrade to python-xdist==3.5.0 and use --dist=worksteal
On exhaustive jobs, the end-to-end parallel tests show enormous
skew. The last 1-2% of tests takes hours, and logs indicate that
the last 1257 tests execute on a single worker.
pytest-xdist introduced a 'worksteal' algorithm in 3.2.0 that
can rebalance the work. Exhaustive end-to-end parallel tests
take about 5:20, while the same tests run in about 2:40 with
the worksteal policy. The improvement on core exhaustive
tests is much smaller, because it doesn't suffer the same
level of skew.
pytest-xdist changed the way they assign tests to workers,
and it exposed an issue with TestAcid::test_lock_timings().
The test sets the query option lock_max_wait_time_s on the
session, but it never unsets it. When multiple copies of
the test run on a single worker, the test case for a timeout
of 300 seconds with lock_max_wait_time_s unset is actually
using a value of lock_max_wait_time_s=5. This reworks the
test to set lock_max_wait_time_s via execute_query()'s
query_options argument rather than on the session itself.
Testing:
- Ran end-to-end exhaustive tests
- Ran a core job
- Verified that TestAcid::test_lock_timings() can run multiple
times with a single worker without failing
Change-Id: I6916bbef94b380a516356763dfabb3777c682637
---
M infra/python/deps/requirements.txt
M tests/query_test/test_acid.py
M tests/run-tests.py
3 files changed, 18 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/24035/2
--
To view, visit http://gerrit.cloudera.org:8080/24035
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6916bbef94b380a516356763dfabb3777c682637
Gerrit-Change-Number: 24035
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
