Comments inline. On Monday, November 19, 2012 4:44:08 PM UTC-8, Henrik Skupin wrote: > Hi all, > > > > There is a lot of work happening on the WebRTC code and I have been > > asked a while back to get automated tests written. So our team has > > started with some plain Mochitests about a month ago but failed to run > > them due to a lot of crash related bugs. As result I have started to get > > some crashtests into the tree, but we were never able to enable them due > > to leaks in the WebRTC code. Given all those problems and a high demand > > for us on the WebAPI project we stepped away for a while. > > > > Now that I'm back on the WebRTC project the situation is probably better > > now. So I'm going to analyze various problems we had before to get those > > solved and new tests implemented. That said we have to cover two types > > of tests which are crashtests and mochitests. As best we want to see > > those running on tbpl for mozilla-central builds. But there are some > > roadblockers in the way we should try to get fixed over the next couple > > of days (or weeks). I think as earlier as better. So I propose the > > following: > > > > 1. Lets get the crashtests enabled on m-c by fixing all the leaks and > > failures: This probably doesn't take that long given that only a single > > crashtest is causing leaks at the moment (bug 813339). But I can't tell > > how much work is necessary to get this fixed. Also there is one > > remaining test failure for Cipc tests (bug 811873). For myself it's most > > likely only the latter which is remaining to get investigated.
That seems like the right first step. I agree. > > > > 2. Create more crashtests: Seeing all those crashes in the WebRTC code > > we definitely have to create crashtests in time when the patches land. > > Otherwise it's getting harder to get them reproduced and to verify their > > stability later, because other check-ins could have caused other crashes > > meanwhile. I have seen it a couple of times in the last weeks. > Agreed here as well. In fact, this is probably more critical at this point in the WebRTC code then getting the mochitests up and running for some basic tests due to the point you've made - the crash bug count is quite high and still climbing. > > > 3. Create a basic set of mochitests: With more stable code we probably > > can create some basic mochitests now. Therefore I would ask Jason to > > give me a good list of those tests he most likely has in his mind. Those > > should be added to the WebRTC QA page for an easier reference. I will > > then pick items, get those filed as bugs, and work on the > > implementation. Those I'm currently not working on I will mark as > > mentored so our wonderful community can pick up tasks too. > A bunch of this work was already taken care of during the brainstorm Rob and I had a while back. There's already a few bugs already on file on the WebRTC side for tests to get started with. For the tests that weren't filed in bugs, the etherpad here - https://etherpad.mozilla.org/automation-webrtc already calls a bunch of others to go after. There's also the various amount of bugs already on file that could be used probably more functional-based mochitest automation. For more formal test runs, getUserMedia (not the full stack) has a bunch of ideas that could be turned into automation based off of my defined test cases here - https://wiki.mozilla.org/Platform/Features/Camera_API_-_Phase_2_%28getUserMedia%29/Test_Plan#Test_Cases. For the full stack (e.g. peer connection), the tests I have defined are more smoke tests off of the etherpad + reusing the existing test case pages. I'll plan on expanding that as I go along, but with those docs, that will probably keep you busy for a good while. > > > Just some more notes: > > > > - When I find leaks for any type of tests I will investigate those and > > get them filed. I wish that we can get those fixed in a short interval. > > Would that be possible? I agree it's important to get these leaks fixed. However, there are many "bad" bug types on file right now all competing for development resources such as: - sec-critical bugs - crash bugs - memory leak bugs - basic functional bustage of the webrtc stack We need to balance the priority across all of those areas carefully. Probably the right thing to do if you hit a memory leak bug that has you blocked is to clearly raise it in a comment/whiteboard of some sorts to indicate that you are blocked. That'll help indicate that the bug needs a higher priority. > > > > - From my judgement I would create crashtests and mochitests on a 3:1 > > ratio, which indeed depends on the amount of remaining crashtests to > > write or to transform from an already given testcase. Hmm...I'd actually put that ratio on the higher end even more for crashtests right now given the amount of crash bugs coming in. That's my opinion though. > > > > - Keep in mind that those created Mochitests can leak! We will not be > > able to check those into mozilla-central until the leak has been fixed. > > Means we could land those temporarily on the alder branch so we get at > > least coverage across platforms. > > > > - Any mochitest and crashtest we are running gets implemented with faked > > media streams. We will never be able to use real devices. That means we > > still need manual testcases. Right. The getUserMedia pieces are fully covered with a bunch of manual test cases. WebRTC full stack pretty much has some high level basics defined from the brainstorm Rob & I did, although I'll be working out more details as I go along. > > > > - It would be a great help for me to get a good list of possible > > mochitests (as mentioned above). That way I will be able to get started > > as soon as possible on their implementation. See the comment above - take advantage of the existing bugs on file + the brainstorming etherpad. > > > > > > Could be that I missed something. So please ask if something is unclear > > or needs a discussion. I really would like to see green results on tbpl > > for webrtc tests in the near future and a good process is crucial here. > > > > Thanks! > > > > -- > > Henrik _______________________________________________ dev-media mailing list [email protected] https://lists.mozilla.org/listinfo/dev-media
