[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

[Roundup Robot]

Roundup Robot added the comment:

New changeset 7d69b214e668 by Zachary Ware in branch '2.7':
Issue #24751: Fix running regrtest with '-w' flag in case of test failures.
https://hg.python.org/cpython/rev/7d69b214e668

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

[Zachary Ware]

Changes by Zachary Ware :


--
assignee:  -> zach.ware
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

[Zachary Ware]

Zachary Ware added the comment:

Ah.  The problem is on 2.7 only; 3.x calls sorted() on the set operation.  The 
set operation should just go away, though; we don't count ENV_CHANGED as 'bad' 
anymore.

Will fix shortly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

[R. David Murray]

Changes by R. David Murray :


--
resolution: fixed -> 
stage: resolved -> needs patch
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

[David Bolen]

David Bolen added the comment:

While running a manual test (make buildbottest) on my 2.7 Ubuntu buildbot, I 
ran into an exception in this patch:

The tail end of the test run:

[401/401/1] test_signal
379 tests OK.
1 test failed:
test_curses
21 tests skipped:
test_aepack test_al test_applesingle test_bsddb185 test_cd test_cl
test_dl test_gl test_imgfile test_kqueue test_linuxaudiodev
test_macos test_macostools test_msilib test_ossaudiodev
test_scriptpackages test_startfile test_sunaudiodev test_winreg
test_winsound test_zipfile64
Those skips are all expected on linux2.
Re-running failed tests in verbose mode
Traceback (most recent call last):
  File "./Lib/test/regrtest.py", line 1598, in 
main()
  File "./Lib/test/regrtest.py", line 655, in main
for test in bad[:]:
TypeError: 'set' object has no attribute '__getitem__'


The code is attempting to iterate over a sliced copy of bad (bad[:]) due to 
later possible mutation, but by that point, if you had failures, bad is a set, 
from the block shortly above where it subtracts out the environment changed 
list.  I was testing 2.7, but I think the issue affects all branches.

Perhaps list(bad) instead of bad[:]?

--
nosy: +db3l

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

[Roundup Robot]

Roundup Robot added the comment:

New changeset 6987a9c7dde9 by Zachary Ware in branch '2.7':
Issue #24751: When running regrtest with '-w', don't fail if re-run succeeds.
https://hg.python.org/cpython/rev/6987a9c7dde9

New changeset 9964edf2fd1e by Zachary Ware in branch '3.4':
Issue #24751: When running regrtest with '-w', don't fail if re-run succeeds.
https://hg.python.org/cpython/rev/9964edf2fd1e

New changeset 9d1f6022261d by Zachary Ware in branch '3.5':
Issue #24751: Merge with 3.4
https://hg.python.org/cpython/rev/9d1f6022261d

New changeset 6f67c74608b6 by Zachary Ware in branch 'default':
Closes #24751: Merge with 3.5
https://hg.python.org/cpython/rev/6f67c74608b6

--
nosy: +python-dev
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

[Zachary Ware]

Zachary Ware added the comment:

Here's a patch.

--
components: +Tests
keywords: +patch
stage:  -> patch review
type:  -> behavior
versions: +Python 2.7, Python 3.4, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file40076/issue24751.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

[R. David Murray]

R. David Murray added the comment:

I think option 1 is to be preferred.  One of the things we've been talking 
about for the workflow is gating on the buildbots passing, and the way that 
works with flaky tests is if the check fails, you just run the test again so 
you get a green and the patch can be gated in.  So from that perspective if the 
tests pass on rerun the result is most useful if it is green.

Unless we want to say amber is OK for gating...but in terms of cognative load I 
think green is better.  After all, our current green state is morally 
equivalent to running the tests again and having them pass..

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24751] regrtest/buildbot: test run marked as failure even when re-run succeeds

[Zachary Ware]

New submission from Zachary Ware:

The buildbots all run the test suite with the '-w', which re-runs any tests 
that failed in the main test sequence at a higher verbosity level.  More often 
than not it seems the re-run tests succeed, but the exit code is still 1 so the 
build is marked as a failure.

The simplest action I'd like would be to exit(0) iff all re-run tests pass on 
the re-run.  Alternatively, we could try to get a bit fancier and exit with 
some other return code, and adjust the build master to interpret that return 
code as "passed, with warnings" and mark the build as amber rather than red.

--
keywords: buildbot, easy
messages: 247632
nosy: zach.ware
priority: low
severity: normal
status: open
title: regrtest/buildbot: test run marked as failure even when re-run succeeds

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com