Re: [Python-Dev] please help triage VSTS failures

[Steve Dower]

Hmm... I would guess that it would create a new "build” and likely lose the 
link to the original (though it would still be kept, just harder to find). Then 
again, the existing support for rerunning a release through VSTS keeps all the 
previous attempts... I’ll mention it, but the quickest fix here may not be 
immediately sufficient.

Sent from my Windows 10 phone

From: Nathaniel Smith
Sent: Friday, May 18, 2018 17:34
To: Steve Dower
Cc: Zachary Ware; Python-Dev
Subject: Re: [Python-Dev] please help triage VSTS failures

On Fri, May 18, 2018 at 1:13 PM, Steve Dower <steve.do...@python.org> wrote:
> According to the VSTS dev team, an easy “rerun this build” button and
> filtering by changed paths are coming soon, which should clean things up.

If you're talking to them, please ask them to make sure that the
"rerun this build" button doesn't erase the old log. (That's what it
does on Travis. Appveyor is better.) The problem is that when you have
a flaky/intermittent failure, your todo list is always (a) rerun the
build so at least it's not blocking whatever this unrelated change is,
(b) file some sort of bug, or comment on some existing bug, and link
to the log to help track down the intermittent failure. If you click
the "rebuild" button on Travis, then it solves (a), while deleting the
information you need for (b) – and for rare intermittent bugs you
might not have much information to go on besides that build log.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please help triage VSTS failures

[Gregory P. Smith]

Hah, yes, that was un-intuitive for me.  I was looking for something
labelled "logs" to click on.  thanks.  (I hope MS is taking notes on UX
issues here)

So these failures were in fact the known flakes and not infrastructure.
good!

On the high level view of VSTS output on a failure:

"Issues:
phase1
Cmd.exe exited with code '2'."

is not useful.  Can we make that display the tail of the failing phase's
log?  that'd avoid needing to find the inner most red X and click on it
which is what I'll always need to do otherwise as there doesn't appear to
be a direct hyperlink to that.

It looks like VSTS also had an API for surfacing individual test results
into their Tests / Test Results summary pane?  Doing something to integrate
with that would likely be a nicer UI.  We have something similar at work,
here's how we make unittest.TestCase emit the JUnit XML files of test
results for CI systems to display:

https://github.com/abseil/abseil-py/blob/master/absl/testing/xml_reporter.py


If there are industry standard format(s) that make Python test suites play
nicer for reporting on CI systems such as that, they'd be worthy of stdlib
inclusion.

-gps


On Fri, May 18, 2018 at 5:35 PM Nathaniel Smith  wrote:

> On Fri, May 18, 2018 at 1:13 PM, Steve Dower 
> wrote:
> > According to the VSTS dev team, an easy “rerun this build” button and
> > filtering by changed paths are coming soon, which should clean things up.
>
> If you're talking to them, please ask them to make sure that the
> "rerun this build" button doesn't erase the old log. (That's what it
> does on Travis. Appveyor is better.) The problem is that when you have
> a flaky/intermittent failure, your todo list is always (a) rerun the
> build so at least it's not blocking whatever this unrelated change is,
> (b) file some sort of bug, or comment on some existing bug, and link
> to the log to help track down the intermittent failure. If you click
> the "rebuild" button on Travis, then it solves (a), while deleting the
> information you need for (b) – and for rare intermittent bugs you
> might not have much information to go on besides that build log.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please help triage VSTS failures

[Nathaniel Smith]

On Fri, May 18, 2018 at 1:13 PM, Steve Dower  wrote:
> According to the VSTS dev team, an easy “rerun this build” button and
> filtering by changed paths are coming soon, which should clean things up.

If you're talking to them, please ask them to make sure that the
"rerun this build" button doesn't erase the old log. (That's what it
does on Travis. Appveyor is better.) The problem is that when you have
a flaky/intermittent failure, your todo list is always (a) rerun the
build so at least it's not blocking whatever this unrelated change is,
(b) file some sort of bug, or comment on some existing bug, and link
to the log to help track down the intermittent failure. If you click
the "rebuild" button on Travis, then it solves (a), while deleting the
information you need for (b) – and for rare intermittent bugs you
might not have much information to go on besides that build log.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please help triage VSTS failures

[Terry Reedy]

On 5/18/2018 4:13 PM, Steve Dower wrote:


Close/reopen PR is the best way to trigger a rebuild right now.


It may be the way to retrigger VSTS, but if one want to merge, and 
either of Travis or AppVeyor pass, tossing the success is a foolish 
thing to do.  Either may fail on a rebuild.



--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please help triage VSTS failures

[Yury Selivanov]

On Fri, May 18, 2018 at 4:15 PM Steve Dower  wrote:
[..]
> The asyncio instability is apparently really hard to fix. There were 2-3
people looking into it yesterday on one of the other systems, but
apparently we haven’t solved it yet (my guess is lingering state from a
previous test). The multissl script was my fault for not realising that we
don’t use it on 3.6 builds, but that should be fixed already. Close/reopen
PR is the best way to trigger a rebuild right now.

I asked Andrew Svetlov to help with asyncio CI triage.  Hopefully we'll
resolve most of them early next week.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please help triage VSTS failures

[Steve Dower]

Unfamiliar maybe, though I’m a big fan of separating build and test logs. If 
anyone is motivated enough to make unittest/regrtest generate Junit format XML 
then we can get a nice breakdown by individual test, too, which would save 
scrolling through the log entirely.

The asyncio instability is apparently really hard to fix. There were 2-3 people 
looking into it yesterday on one of the other systems, but apparently we 
haven’t solved it yet (my guess is lingering state from a previous test). The 
multissl script was my fault for not realising that we don’t use it on 3.6 
builds, but that should be fixed already. Close/reopen PR is the best way to 
trigger a rebuild right now.

According to the VSTS dev team, an easy “rerun this build” button and filtering 
by changed paths are coming soon, which should clean things up. We *could* add 
our own detection for doc-only changes and skip most steps – I’m happy to add 
that in if someone can help with the right git incantation.

Top-posted from my Windows phone

From: Zachary Ware
Sent: Friday, May 18, 2018 9:15
To: Python-Dev
Subject: Re: [Python-Dev] please help triage VSTS failures

On Fri, May 18, 2018 at 10:55 AM, Gregory P. Smith <g...@krypto.org> wrote:
> These both look like VSTS infrastructure falling over on PRs:
>
>  https://python.visualstudio.com/cpython/_build?buildId=522
>
>  https://python.visualstudio.com/cpython/_build?buildId=523
>
> I don't see anywhere that gives information about the failures. (*)

Somewhat non-intuitively, logs are available by clicking something in
the tree in the left column (in particular, the inner-most thing with
a red X).  Do be sure to choose "Logs" in the right pane rather than
"Tests"; "Tests" appears to be a separate feature that we don't use.

> These CI failures on different platforms are both for the same change, on
> different branches, for a documentation only change.  That passed on the
> other branches and platforms for the same change.

The Windows failure appears to be the test_asyncio instability that
has been popping up everywhere the past couple of days.  The Linux
failure looks like a mis-use of Tools/ssl/multissltests.py in the
setup, but I thought I'd seen a checkin go by that removed that.

-- 
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please help triage VSTS failures

[Zachary Ware]

On Fri, May 18, 2018 at 10:55 AM, Gregory P. Smith  wrote:
> These both look like VSTS infrastructure falling over on PRs:
>
>  https://python.visualstudio.com/cpython/_build?buildId=522
>
>  https://python.visualstudio.com/cpython/_build?buildId=523
>
> I don't see anywhere that gives information about the failures. (*)

Somewhat non-intuitively, logs are available by clicking something in
the tree in the left column (in particular, the inner-most thing with
a red X).  Do be sure to choose "Logs" in the right pane rather than
"Tests"; "Tests" appears to be a separate feature that we don't use.

> These CI failures on different platforms are both for the same change, on
> different branches, for a documentation only change.  That passed on the
> other branches and platforms for the same change.

The Windows failure appears to be the test_asyncio instability that
has been popping up everywhere the past couple of days.  The Linux
failure looks like a mis-use of Tools/ssl/multissltests.py in the
setup, but I thought I'd seen a checkin go by that removed that.

-- 
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] please help triage VSTS failures

[Gregory P. Smith]

These both look like VSTS infrastructure falling over on PRs:

 https://python.visualstudio.com/cpython/_build?buildId=522

 https://python.visualstudio.com/cpython/_build?buildId=523

I don't see anywhere that gives information about the failures. (*)

These CI failures on different platforms are both for the same change, on
different branches, for a documentation only change.  That passed on the
other branches and platforms for the same change.

-gps

(*) I refuse to "Download logs as a zip file".  I'm in a web browser, if
the information I might need is potentially buried somewhere in a zip file
of logs, that is a waste of my time. I'm not going to do it. The web UI
*needs* to find and display the relevant failure info from any logs
directly.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com