TL;DR
- UI got more verbose, elements unnecessary hidden
- Too much vertical space used
- Passes and failures shouldn't be separated
- failure log becomes longer because of failure snippets
User persona
Developers:
- check the results of their own pushes to Try and own pushes to
official repositories and their backouts (rarely star own pushes on non-Try)
- only a few developers push patches of other people
- might download the B builds from Try to test them locally
- check the tree status by the favicon of the Treeherder tab
- hopefully check the failures
- aren't aware of current most common issue like intermittents or infra
- wait eagerly that that one Try push finishes
Sheriffs:
- interact with all kind of pushes on all repositories
- also push patches for other people
- keep an eye on the number of unstarred failures by looking at the
number in the tab title even when it's in the background
- check the tree status by the favicon of the Treeherder tab
- are looking for new issues and changed behavior over time (scrolling,
Similar Jobs link, Sig link, filtering)
- try to identify slow jobs/infra issues
- wait eagerly for that retriggers to finish
- have to find mergeable revisions
- have multiple trees open
- try to get the unstarred count to 0
Concerns shared between developers and sheriffs:
- Passes are separated from failures. This makes it harder to check if
there was a retrigger (automatic or not) and if it passed. This
shouldn't get implemented like that. It will also make it more difficult
to just scroll down and check if a job run, failed or got coalesced.
- Retriggering a job is more hidden in a menu. This should be fixed.
- Use of vertical space: The wireframes only show the merge commits, the
others are hidden behind a "+14" - will this be done by a "smart"
algorithm? How will this look for e.g. a push with 10 commits total for
2 bugs? On a 15'', a push with pgo etc. uses 62 lines for the results
here (of 41 platforms). This won't get reduced much because it doesn't
get much wider, test failures and builds get the left space (pun
intended). It should always fit completely on a screen. And as a
developer, being able to have as many results for a build/test from as
many pushes as possible on the screen is my desired outcome (and much
faster than to open a new tab with the "Sig" link or the slow/broken
"Similar jobs" tab. The height might even get taller if Tier 2 (and 3)
get their own lines.
Concerns from a sheriff point of view:
- Adding snippets to the test failures will further increase the length
of the failure list which already is a scroll "marathon" with
autoclassifier at the moment. Can a valuable log snippet be
automatically extracted? I doubt it (think about strack traces),
especially with the length concern.
- There is only one link to one log viewer, for the link to both one has
to open the menu. At the moment, a sheriff can use the link to the
structured log to shared it with a developer and the developer will be
taken to the first failure when they open the link and also have a link
to the push on Treeherder on the left where they can self-serve their
needs. As sheriff, I only use the normal, plaintext full length log
because it's possible to do a full text search and I don't have to wait
when to scroll up or down.
- The link to "Similar jobs" has been hidden in the "Job details" so
that opening it takes more time (if it worked reliably)
- The search on top can hide its filters. P.5 shows a platform
restriction which - from the design - doesn't seem to get exposed by
default. It's easy to forget about a filter when you switch to a
different tab and return.
Changes which could be dropped:
- The "known" and "unknown" failure styles sound handy, a substantial
part of the known failures can be caused by a change be know much more
frequent, unknown ones can be unrelated (like infra). This won't provide
much value.
- There are 3 columns (failures, builds and tests). In the current UI,
the builds always are put in front of the list, here we spend more
space, but people shouldn't have an issue to find the builds already at
the moment (if they try the first job for a platform, they get it).
Issues not addressed in the wireframes:
- Name of pusher still much more accessible than the name of the patch
author. Sheriff has always to check if they are different when a patch
has issues and the developer has to be pinged on IRC. Suggestion:
Highlight when patch author is different from pusher.
- It's still possible to select multiple trees. Do people use this or
makes it sense to drop the feature and use multiple tabs?
Sebastian
Original-Nachricht
Betreff: Feedback requested: UI changes for Treeherder
Von: Jonathan Griffin
An:
Datum: 2016-10-10 20:
