[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

Roundup Robot added the comment: New changeset 7d69b214e668 by Zachary Ware in branch '2.7': Issue #24751: Fix running regrtest with '-w' flag in case of test failures. https://hg.python.org/cpython/rev/7d69b214e668 -- ___ Python tracker

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

Changes by Zachary Ware : -- assignee: -> zach.ware resolution: -> fixed stage: needs patch -> resolved status: open -> closed ___ Python tracker ___ __

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

Zachary Ware added the comment: Ah. The problem is on 2.7 only; 3.x calls sorted() on the set operation. The set operation should just go away, though; we don't count ENV_CHANGED as 'bad' anymore. Will fix shortly. -- ___ Python tracker

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

Changes by R. David Murray : -- resolution: fixed -> stage: resolved -> needs patch status: closed -> open ___ Python tracker ___ ___

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

David Bolen added the comment: While running a manual test (make buildbottest) on my 2.7 Ubuntu buildbot, I ran into an exception in this patch: The tail end of the test run: [401/401/1] test_signal 379 tests OK. 1 test failed: test_curses 21 tests skipped: test_aepack test_al test_app

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

Roundup Robot added the comment: New changeset 6987a9c7dde9 by Zachary Ware in branch '2.7': Issue #24751: When running regrtest with '-w', don't fail if re-run succeeds. https://hg.python.org/cpython/rev/6987a9c7dde9 New changeset 9964edf2fd1e by Zachary Ware in branch '3.4': Issue #24751: When

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

Zachary Ware added the comment: Here's a patch. -- components: +Tests keywords: +patch stage: -> patch review type: -> behavior versions: +Python 2.7, Python 3.4, Python 3.5, Python 3.6 Added file: http://bugs.python.org/file40076/issue24751.diff __

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

R. David Murray added the comment: I think option 1 is to be preferred. One of the things we've been talking about for the workflow is gating on the buildbots passing, and the way that works with flaky tests is if the check fails, you just run the test again so you get a green and the patch c

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

New submission from Zachary Ware: The buildbots all run the test suite with the '-w', which re-runs any tests that failed in the main test sequence at a higher verbosity level. More often than not it seems the re-run tests succeed, but the exit code is still 1 so the build is marked as a fail