CIA/Treeherder updates

[Sarah Clements]

Hi all,

The CIA/Treeherder team would like to share a few of the projects and
features that we worked on in H1 that you may not be aware of. They include:

Cost reduction

Cross-team effort reduced across-the-board monthly spend by 40% in H1 with
plans for another 25% or more in H2.

Developer productivity [gbrown]

Geoff reduced the backlog of testing bugs affecting developer productivity
with a focus on busted mach commands, improvements to crash reporting and
test debugging. There were 66 bugs resolved in H1 with an additional 44
bugs identified for continued efforts in H2.

Outreachy Accessibility project [camd and sclements]

Cameron and I mentored Outreachy intern Mellina Yonashiro during her 3
month internship from Dec - March. In addition to improving accessibility
for all users in the Treeherder family of apps, Mellina worked with Marco
Zehe and James Teh to create screen-reader friendly alternative views for
the graphs in Perfherder and Intermittent Failures View.

UX research and design project [camd and sclements]

Cameron and I observed and interviewed six developers as they used the
Treeherder family of apps. Based on our findings, we decided to focus
primarily on improving Push Health in order to make it a dedicated
developers view. With the assistance of UX designer Victoria Wang, we spent
5 weeks on an iterative design process (mockups -> feedback from users ->
mockup changes). More details (and other improvements) to come.

Treeherder filter tasks by test or manifest path [armenzg]

Armen worked on a feature that enables job filtering by test or manifest
paths. As an example, the screenshot below shows the task in which
“devtools/client/framework/browser-toolbox/test/browser.ini” is executed
in. You can see it live here
.
More details outlined in this blog post

.



Treeherder Github projects now have proper commit sorting [armenzg]

Github does not have the concepts of pushes, however, Treeherder does.
There are few Github projects running on Treeherder and they each have
different automation systems to merge code into their respective master
branches. Unfortunately, this created various edge cases where a recent
Github code landing on master would show up later in Treeherder’s timeline.
This was fixed earlier this year and makes for easier tree sheriffing.


*Treeherder's Add new Jobs (Search) feature has been enhanced*

The Add New Jobs (Search) feature in Treeherder, it has recently been
updated to take advantage of new advanced search filtering operators
. These should be
practically equivalent to mach try fuzzy's search operators, although
specific queries from mach try fuzzy might not trigger an identical set of
jobs when used in Treeherder because of differences in how lists of
available jobs are supplied to both tools. Examples | Fuse.js



Perfherder improvements

The performance test team has made many improvements to the Perfherder UI
such as improved search for alerts and allowing retriggers of up to 10
performance jobs directly from the Compare View. They’ve reached a
milestone (80% complete) with the backfill bot, which will backfill missing
performance jobs.

Best,
Sarah
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Deprecation notice - change to Treeherder's jobdetail API

[Sarah Clements]

The reason my answer has been overly broad is because the type of data that
is being parsed by TinderboxPrint lines is inconsistent. Some of its
CPU/resource stats that should be reported to Perfherder, some of it is
urls embedded in html, and some is other bits of data (for example):
```

"title": "(unnecessary roots)",
"value": "1001",
"url": null

```

So the answer to "what do we do instead' is "it depends" but we'll help
people figure it out on a case-by-case basis.

Sarah


On Mon, Apr 13, 2020 at 10:38 AM Nicholas Alexander 
wrote:

> Hi all,
>
> On Mon, Apr 13, 2020 at 10:29 AM Sarah Clements 
> wrote:
>
>> Nick,
>>
>> I want to clarify something I said in my previous email that might be
>> confusing:
>> > We currently process data from the jobInfo and extra.artifacts
>> properties of jobs we ingest (this goes into the JobDetails table).
>>
>> Disregard this, it's mostly specific to Treeherder (I think we're
>> primarily using extra.artifacts for Perfherder and I am cautious about
>> using it as a catch-all for random bits of information). An alternative for
>> surfacing whatever information is currently being parsed by TinderboxPrint
>> lines is to create a structured artifact (if, as I mentioned previously,
>> it's not stats that should be reported to Perfherder). If this doesn't work
>> for your scenario, we can pull in people more familiar with
>> tests/harnesses/etc to help figure out what you need and how to surface it
>> (:jmaher might be a good person to ask).
>>
>> When you get an idea of what you need to do, let me know an estimation of
>> your timeline. I can extend the deadline by another month or two (*just
>> for TinderboxPrint deprecation*) so everyone who needs to can get
>> alternatives implemented can.
>>
>
> To be clear: I don't need or expect an extension of this deadline.  I'm
> not thrilled that we don't have a crisp answer to "what do I do instead",
> but I trust that the people closest to this problem -- namely, you -- are
> balancing things reasonably and I fully support the decision you are
> presenting.
>
> Best,
> Nick
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Deprecation notice - change to Treeherder's jobdetail API

[Sarah Clements]

Nick,

I want to clarify something I said in my previous email that might be
confusing:
> We currently process data from the jobInfo and extra.artifacts properties
of jobs we ingest (this goes into the JobDetails table).

Disregard this, it's mostly specific to Treeherder (I think we're primarily
using extra.artifacts for Perfherder and I am cautious about using it as a
catch-all for random bits of information). An alternative for surfacing
whatever information is currently being parsed by TinderboxPrint lines is
to create a structured artifact (if, as I mentioned previously, it's not
stats that should be reported to Perfherder). If this doesn't work for your
scenario, we can pull in people more familiar with tests/harnesses/etc to
help figure out what you need and how to surface it (:jmaher might be a
good person to ask).

When you get an idea of what you need to do, let me know an estimation of
your timeline. I can extend the deadline by another month or two (*just for
TinderboxPrint deprecation*) so everyone who needs to can get alternatives
implemented can.

Sarah
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Deprecation notice - change to Treeherder's jobdetail API

[Sarah Clements]

Hi Steve,

You will still be able to see all uploaded artifacts in the Treeherder job
detail panel. The difference is that we will be retrieving them from
taskcluster directly rather than storing them in our database and
retrieving them that way.

Regarding your use of log lines, if you can create a structured artifact
(JSON, as you mentioned) that would probably be the way to go. If it's
stats, that should be reported to Perfherder. If that doesn't work for your
particular scenario, then we can loop in other people (maybe :jmaher) who
have more knowledge of tests/harnesses/etc to figure out how to surface the
info you need. In general, we don't really have an intended replacement per
say for TinderboxPrint. It's more a matter of figuring out who is using it,
whether we already have that data stored elsewhere in Treeherder and if
not, how we can surface it using existing avenues.

Let me know if you need a longer timeline for implementing this. I can
extend the deadline by another month or two (*just for TinderboxPrint
deprecation*) so everyone who needs to can get alternatives implemented can.

Sarah


On Fri, Apr 10, 2020 at 11:03 AM Steve Fink  wrote:

> On 4/10/20 10:06 AM, Andrew McCreight wrote:
> > On Thu, Apr 9, 2020 at 6:51 PM Sarah Clements 
> wrote:
> >
> >> Hello,
> >>
> >> As part of maintenance work and performance improvements to Treeherder,
> >> we're making changes to some of the data that's stored in our database.
> On *April
> >> 30th*, we will no longer be retrieving uploaded artifacts for jobs and
> >> storing them in the JobDetail table.
> >>
> >> Uploaded artifacts can be retrieved via a taskcluster API, with the
> >> appropriate rootUrl, taskId and runId. TaskId and runId (synonymous with
> >> retryId in Treeherder) can be surfaced from any of our /jobs/ API's. See
> >> this
> >> <
> https://github.com/mnoorenberghe/mozscreenshots/issues/43#issue-592994567>
> >> for implementation details.
> >>
> > As somebody who has dumped files to MOZ_UPLOAD_DIR and used the job
> detail
> > tab to download those files (just this week, in fact!), but has only a
> > vague idea of what the "taskcluster API" even is, let alone a "rootURL"
> or
> > the rest of it, is there any introductory documentation about how I might
> > retrieve files written to MOZ_UPLOAD_DIR? Am I going to have to write
> some
> > kind of Python script to retrieve my files in the future? That GitHub
> issue
> > seems to require a lot of background information that I do not have.
>
> I'm in a similar boat. I very frequently set additional options on my
> try pushes in order to generate nondefault files in MOZ_UPLOAD_DIR. It
> is unclear to me from this description whether I will still be able to
> see those files in the treeherder UI after this change.
>
> >
> >> We're also planning to deprecate the parsing and storage of
> TinderboxPrint
> >> log lines in the JobDetail table. They'll no longer be shown in
> >> Treeherder's job detail panel but you can still read them in the raw
> logs
> >> via the log-viewer.
> >>
> >> If you have any questions or concerns (or need a longer timeline to make
> >> changes that rely on this data), don't hesitate to reach out.
>
> Similarly, I also regularly depend on being able to emit log lines from
> my jobs that will be easily visible in treeherder (*without* reading the
> entire raw log), both for my own use and others'. Is there a replacement
> for this functionality? I am not at all sad to see the specific
> implementation go ("TinderboxPrint" plus an opaque set of rules for what
> can go after it), and would be happy to convert my jobs over to a
> different system. If that involves something like writing out structured
> json to a separate file, that's still fine with me. (I'm only speaking
> for myself; some people may depend on being able to simply drop in
> `fprintf(stderr, "TinderboxPrint: ...")` into their code and have it
> show up.)
>
> Needing to load the raw log files isn't great because (1) it takes a
> noticeable amount of time for large files (I'm sure this is much worse
> for some locations), and (2) when you're doing things like this, you may
> very well need to be loading several of them.
>
> If there *is* an intended replacement, I could suggest other things that
> I'd find useful (or would like to preserve with the current setup). For
> example, I have a job that checks for multiple types of errors, and
> wants to output a link to documentation specific to the class of error
> that was detected. Not critical functionality, but it definitely makes
>

Re: Deprecation notice - change to Treeherder's jobdetail API

[Sarah Clements]

Hi Nick and Andrew,

Thanks for the questions.

The job details panel in the UI will continue to display uploaded
artifacts, the difference is we'll be retrieving them from taskcluster
directly rather than storing them in JobDetails API and then retrieving
them that way. I mentioned this in case anyone is relying on the JobDetails
API specifically to get that information.

> I don't see any slated replacement for print lines.  A great deal of
useful information is summarized in these lines (e.g., we dump
libxul/classes.dex/and APK sizes using this mechanism -- or at least we
did).  What's happening here?

We currently process data from the jobInfo and extra.artifacts properties
of jobs we ingest (this goes into the JobDetails table). Can some of the
data in TinderboxPrint lines be surfaced in the properties of jobs instead
(or those aforementioned fields)? Or reported to Perfherder, if it consists
of stats (some of which might already be via the PERFHERDER_DATA log
lines)? What specific information are you needing to see?

> As somebody who has dumped files to MOZ_UPLOAD_DIR and used the job
detail tab to download those files (just this week, in fact!), but has only
a vague idea of what the "taskcluster API" even is, let alone a "rootURL"
or the rest of it, is there any introductory documentation about how I
might retrieve files written to MOZ_UPLOAD_DIR? Am I going to have to write
some kind of Python script to retrieve my files in the future? That GitHub
issue seems to require a lot of background information that I do not have.

You only need to make a change if you were using the JobDetails API
specifically to retrieve the uploaded artifacts, either via a script or
library. If that's not your use case, you can disregard :)

Otherwise, this is the API:
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task//runs/0/artifacts
<-- for all artifacts or with `/` to select just one specific
artifact. We can set up a time to chat on slack/riot or video if you need
guidance on implementation for use with a script or library.

Sarah


On Thu, Apr 9, 2020 at 9:05 PM Nicholas Alexander 
wrote:

> Hello Sarah, others,
>
> On Thu, Apr 9, 2020 at 6:51 PM Sarah Clements 
> wrote:
>
>> Hello,
>>
>> As part of maintenance work and performance improvements to Treeherder,
>> we're making changes to some of the data that's stored in our database. On 
>> *April
>> 30th*, we will no longer be retrieving uploaded artifacts for jobs and
>> storing them in the JobDetail table.
>>
>> Uploaded artifacts can be retrieved via a taskcluster API, with the
>> appropriate rootUrl, taskId and runId. TaskId and runId (synonymous with
>> retryId in Treeherder) can be surfaced from any of our /jobs/ API's. See
>> this
>> <https://github.com/mnoorenberghe/mozscreenshots/issues/43#issue-592994567>
>> for implementation details.
>>
>> We're also planning to deprecate the parsing and storage of
>> TinderboxPrint log lines in the JobDetail table. They'll no longer be shown
>> in Treeherder's job detail panel but you can still read them in the raw
>> logs via the log-viewer.
>>
>> If you have any questions or concerns (or need a longer timeline to make
>> changes that rely on this data), don't hesitate to reach out.
>>
>
> I have worked with both of these APIs and been frustrated with them (and
> filed tickets that are now being closed), so I'm pleased to see changes in
> this area.  However, I have some questions.
>
> Does the TH UI that exposes artifacts change as part of this?  I.e., does
> it get harder to find the package produced by a build job?
>
> I don't see any slated replacement for print lines.  A great deal of
> useful information is summarized in these lines (e.g., we dump
> libxul/classes.dex/and APK sizes using this mechanism -- or at least we
> did).  What's happening here?
>
> Best,
> Nick
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Deprecation notice - change to Treeherder's jobdetail API

[Sarah Clements]

Hello,

As part of maintenance work and performance improvements to Treeherder,
we're making changes to some of the data that's stored in our
database. On *April
30th*, we will no longer be retrieving uploaded artifacts for jobs and
storing them in the JobDetail table.

Uploaded artifacts can be retrieved via a taskcluster API, with the
appropriate rootUrl, taskId and runId. TaskId and runId (synonymous with
retryId in Treeherder) can be surfaced from any of our /jobs/ API's. See
this

for implementation details.

We're also planning to deprecate the parsing and storage of TinderboxPrint
log lines in the JobDetail table. They'll no longer be shown in
Treeherder's job detail panel but you can still read them in the raw logs
via the log-viewer.

If you have any questions or concerns (or need a longer timeline to make
changes that rely on this data), don't hesitate to reach out.

Sarah
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

OrangeFactor/Intermittent Failures View Update

[Sarah Clements]

Hi!

As of today, the OrangeFactor url is redirecting to Intermittent Failures
View  as part of
the decommission process. The API's that replace OrangeFactors API's are
Failures, FailuresByBug and FailureCount (you can find more info in the
Treeherder API docs ).

The Treeherder team will be dedicating more resources to improving
Intermittent Failures View, so please file any bugs or feature
requests for Intermittent
Failures View

rather than for OrangeFactor.

- Sarah
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform