On Monday, March 4, 2013 5:42:39 AM UTC-8, Ed Morley wrote:
> (CCing auto-to...@mozilla.com)
>
> jmaher and jhammel will be able to comment more on the talos specifics,
> but few thoughts off the top of my head:
>
> It seems like we're conflating multiple issues here:
> 1) "[talos] is a separate repo from mc"
And also
1a) Talos itself is a big pain for developers to use and debug regressions
in, not to mention add tests to, which they basically don't.
It seems that some of this may have changed recently, especially around using
the new framework--I haven't used it in a while. I think Talos still does fail
on creating tests, though, because lots of things just don't fit its
assumptions.
> 2) "[it's a hassle to] test the test to be sure it’s working"
> 3) "[it's a hassle to get results] populated in graph server"
> 4) "[we need to] come up with a good way to manage the life cycle of
> active perf tests so graph server doesn’t become polluted"
> Switching from the talos harness to mochitest doesn't fix #2 (we still
> have to test, and I don't see how it magically becomes any easier
> without extra work - that could have been applied to talos instead) or
> #3/#4 (orthogonal problem). It also seems like a brute force way of
> fixing #1 (we could just check talos into mozilla-central).
I think that part was mostly supposed to address (1a).
> Instead, I think we should be asking:
>
> 1) Is the best test framework for performance testing: [a] talos (with
> improvements), [b] mochitest (with a significant amount of work to make
> it compatible), or [c] a brand new framework?
I think that question doesn't have one answer. For JS, it's clearly "something
else", but it's not even really a framework--it's just running standard
benchmarks.
For other areas, there are likely different answers. That's why I was so
excited about the self-serve idea. (Interestingly, I got schooled on this
subject in a similar vein recently on bug tracking. :-) )
> 2) Regardless of framework used, would checking it into mozilla-central
> improve dev workflow enough to outweigh the downsides (see bug 787200
> for history on that discussion)?
Thanks for the bug link. It seems like putting Talos itself into m-c has
significant disadvantages. I'm not sure what to do with other/new perf tests.
> 3) Regardless of framework used, how can we make the
> development/testing/staging cycle less painful?
I liked the original proposal a lot for this.
> 4) Regardless of framework used, who should be responsible for ensuring
> we actively prune performance tests that are no longer relevant?
I gave an idea for how to do this in my reply to the original proposal. I
didn't say who would do it, but I was assuming the maintainers/operators of
graph-server, with the notion that they would be highly empowered to remove
anything that no one asked them to keep or that didn't otherwise have a
well-documented, easily understood rationale.
Dave
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
