Re: Switching Jetpack to use the runtests.py automation
On Mon, Aug 4, 2014 at 6:21 PM, Gregory Szorc g...@mozilla.com wrote: On 8/4/14, 10:39 AM, Dave Townsend wrote: I've done a little investigation into marionette and I've found a few issues with it: Firstly it doesn't look like running marionette directly or through mach allows developers to select individual directories or test files to run, rather it is a one-shot affair. This is very inconvenient for development. Secondly marionette doesn't seem to be built to scale to many test types. It uses regular expressions on filenames to determine the test type, as it happens the Jetpack tests do use a different form to the existing marionette tests so it's not out of the question but still makes me wary of adding a new test type. Thirdly I can't run marionette tests locally, they consistently fail quite badly. These problems make marionette a less than desirable option for use as a base for our test harness right now so I plan to get my work to make mochitests run jetpack tests completed this week and submit it for review. If Marionette becomes a better choice in the future a lot of the work I'm doing right now carries over, it will be simpler to switch from mochitest to marionette than it is switching to mochitest. The issues listed seem fixable. I would rather we spend energy improving Marionette than piling yet more things on top of mochitest's haphazard base. The various automation failures in the past few weeks should be reason enough to avoid mochitest and go with a better-engineered and tested solution (marionette). Who is going to do that work? I have patches that vastly improve the testing situation for jetpack tests by allowing other developers run them more easily, making them easier for releng to manage and most importantly making them meet tbpl visibility requirements. They involve tiny changes to the mochitest harness code. We're already hidden by default on tinderbox and are hitting problems because of it, I'd rather go ahead and finish up this work than wait for some future time when marionette can be upgraded to meet our requirements. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Switching Jetpack to use the runtests.py automation
If this only involves tiny changes to mochitest and it's ready, I'd go ahead and do that. I am interested in seeing what your requirements are, though, and figuring out if we could meet them later with a better architected solution, whether it's Marionette or something else. Mochitest is kind of a monster, and the more we hack on corner cases the more fragile and unmaintainable it becomes. Jonathan On 8/5/2014 9:20 AM, Dave Townsend wrote: On Mon, Aug 4, 2014 at 6:21 PM, Gregory Szorc g...@mozilla.com wrote: On 8/4/14, 10:39 AM, Dave Townsend wrote: I've done a little investigation into marionette and I've found a few issues with it: Firstly it doesn't look like running marionette directly or through mach allows developers to select individual directories or test files to run, rather it is a one-shot affair. This is very inconvenient for development. Secondly marionette doesn't seem to be built to scale to many test types. It uses regular expressions on filenames to determine the test type, as it happens the Jetpack tests do use a different form to the existing marionette tests so it's not out of the question but still makes me wary of adding a new test type. Thirdly I can't run marionette tests locally, they consistently fail quite badly. These problems make marionette a less than desirable option for use as a base for our test harness right now so I plan to get my work to make mochitests run jetpack tests completed this week and submit it for review. If Marionette becomes a better choice in the future a lot of the work I'm doing right now carries over, it will be simpler to switch from mochitest to marionette than it is switching to mochitest. The issues listed seem fixable. I would rather we spend energy improving Marionette than piling yet more things on top of mochitest's haphazard base. The various automation failures in the past few weeks should be reason enough to avoid mochitest and go with a better-engineered and tested solution (marionette). Who is going to do that work? I have patches that vastly improve the testing situation for jetpack tests by allowing other developers run them more easily, making them easier for releng to manage and most importantly making them meet tbpl visibility requirements. They involve tiny changes to the mochitest harness code. We're already hidden by default on tinderbox and are hitting problems because of it, I'd rather go ahead and finish up this work than wait for some future time when marionette can be upgraded to meet our requirements. ___ tools mailing list to...@lists.mozilla.org https://lists.mozilla.org/listinfo/tools ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Switching Jetpack to use the runtests.py automation
I've done a little investigation into marionette and I've found a few issues with it: Firstly it doesn't look like running marionette directly or through mach allows developers to select individual directories or test files to run, rather it is a one-shot affair. This is very inconvenient for development. Secondly marionette doesn't seem to be built to scale to many test types. It uses regular expressions on filenames to determine the test type, as it happens the Jetpack tests do use a different form to the existing marionette tests so it's not out of the question but still makes me wary of adding a new test type. Thirdly I can't run marionette tests locally, they consistently fail quite badly. These problems make marionette a less than desirable option for use as a base for our test harness right now so I plan to get my work to make mochitests run jetpack tests completed this week and submit it for review. If Marionette becomes a better choice in the future a lot of the work I'm doing right now carries over, it will be simpler to switch from mochitest to marionette than it is switching to mochitest. On Tue, Jul 15, 2014 at 10:56 PM, Henrik Skupin hsku...@mozilla.com wrote: Gregory Szorc wrote on 07/15/2014 09:04 PM: On 7/15/14, 11:49 AM, Dave Townsend wrote: Since forever Jetpack tests in the Firefox trees have been run using our custom python CFX tool which is based on a fork of an ancient version of mozrunner. This causes us a number of problems. Keeping up with tree visibility rules is hard. Some features from newer versions of mozrunner like crash stack handling aren't available and our attempts to update to the newer mozbase have been blocked on trying to get some of our forked code accepted. It also makes it hard for Mozilla other developers to run our tests as CFX has a very different syntax to the other test suites. We've started investigating switching away from CFX and instead using the python automation that the mochitests use. This would work somewhat similarly to browser-chrome tests, runtests.py will startup Firefox and overlay some XUL and JS on the main window from where we can run the existing JS parts of the Jetpack test suites. There are many benefits here. The runtests.py code is well used and known to be resilient. It supports things like screenshots on failures and crash stacks that Jetpack tests don't currently handle. We'll use manifest files like the other test suites so disabling tests per platform will be easy. Excellent mach integration will make running individual tests simple. It also makes it possible to use commonjs style tests elsewhere in the tree. Release engineering should find managing the Jetpack tests a lot easier as they behave just like other mochitests. My initial experiment last week shows that this will work. The first part of our tests (package tests) is running and passing on my local machine and I expect to have the add-on tests working this week. I wanted to give everyone a heads up about this work to give you all a chance to ask questions or raise objections. The changes to runtests and the build system are minimal, just adding support for new manifest types really but I will be needing reviews for those. We'll also have to make the buildbot changes to switch over to use these new tests but I expect that to be pretty straightforward. Was Marionette considered? From what little I know (jgriffin and others can correct me), Marionette seems like the logical base for this test suite. Adding the tools mailing list, so that members of the A-team are aware of this thread, and can answer appropriately. -- Henrik Skupin Senior Test Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Switching Jetpack to use the runtests.py automation
On 8/4/14, 10:39 AM, Dave Townsend wrote: I've done a little investigation into marionette and I've found a few issues with it: Firstly it doesn't look like running marionette directly or through mach allows developers to select individual directories or test files to run, rather it is a one-shot affair. This is very inconvenient for development. Secondly marionette doesn't seem to be built to scale to many test types. It uses regular expressions on filenames to determine the test type, as it happens the Jetpack tests do use a different form to the existing marionette tests so it's not out of the question but still makes me wary of adding a new test type. Thirdly I can't run marionette tests locally, they consistently fail quite badly. These problems make marionette a less than desirable option for use as a base for our test harness right now so I plan to get my work to make mochitests run jetpack tests completed this week and submit it for review. If Marionette becomes a better choice in the future a lot of the work I'm doing right now carries over, it will be simpler to switch from mochitest to marionette than it is switching to mochitest. The issues listed seem fixable. I would rather we spend energy improving Marionette than piling yet more things on top of mochitest's haphazard base. The various automation failures in the past few weeks should be reason enough to avoid mochitest and go with a better-engineered and tested solution (marionette). ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Switching Jetpack to use the runtests.py automation
I think I speak for everyone who's debugged JP failures when I say: Huzzah! Thanks for doing this Mossop :-) On Tue, Jul 15, 2014 at 11:49 AM, Dave Townsend dtowns...@oxymoronical.com wrote: Since forever Jetpack tests in the Firefox trees have been run using our custom python CFX tool which is based on a fork of an ancient version of mozrunner. This causes us a number of problems. Keeping up with tree visibility rules is hard. Some features from newer versions of mozrunner like crash stack handling aren't available and our attempts to update to the newer mozbase have been blocked on trying to get some of our forked code accepted. It also makes it hard for Mozilla other developers to run our tests as CFX has a very different syntax to the other test suites. We've started investigating switching away from CFX and instead using the python automation that the mochitests use. This would work somewhat similarly to browser-chrome tests, runtests.py will startup Firefox and overlay some XUL and JS on the main window from where we can run the existing JS parts of the Jetpack test suites. There are many benefits here. The runtests.py code is well used and known to be resilient. It supports things like screenshots on failures and crash stacks that Jetpack tests don't currently handle. We'll use manifest files like the other test suites so disabling tests per platform will be easy. Excellent mach integration will make running individual tests simple. It also makes it possible to use commonjs style tests elsewhere in the tree. Release engineering should find managing the Jetpack tests a lot easier as they behave just like other mochitests. My initial experiment last week shows that this will work. The first part of our tests (package tests) is running and passing on my local machine and I expect to have the add-on tests working this week. I wanted to give everyone a heads up about this work to give you all a chance to ask questions or raise objections. The changes to runtests and the build system are minimal, just adding support for new manifest types really but I will be needing reviews for those. We'll also have to make the buildbot changes to switch over to use these new tests but I expect that to be pretty straightforward. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Switching Jetpack to use the runtests.py automation
On 7/15/14, 11:49 AM, Dave Townsend wrote: Since forever Jetpack tests in the Firefox trees have been run using our custom python CFX tool which is based on a fork of an ancient version of mozrunner. This causes us a number of problems. Keeping up with tree visibility rules is hard. Some features from newer versions of mozrunner like crash stack handling aren't available and our attempts to update to the newer mozbase have been blocked on trying to get some of our forked code accepted. It also makes it hard for Mozilla other developers to run our tests as CFX has a very different syntax to the other test suites. We've started investigating switching away from CFX and instead using the python automation that the mochitests use. This would work somewhat similarly to browser-chrome tests, runtests.py will startup Firefox and overlay some XUL and JS on the main window from where we can run the existing JS parts of the Jetpack test suites. There are many benefits here. The runtests.py code is well used and known to be resilient. It supports things like screenshots on failures and crash stacks that Jetpack tests don't currently handle. We'll use manifest files like the other test suites so disabling tests per platform will be easy. Excellent mach integration will make running individual tests simple. It also makes it possible to use commonjs style tests elsewhere in the tree. Release engineering should find managing the Jetpack tests a lot easier as they behave just like other mochitests. My initial experiment last week shows that this will work. The first part of our tests (package tests) is running and passing on my local machine and I expect to have the add-on tests working this week. I wanted to give everyone a heads up about this work to give you all a chance to ask questions or raise objections. The changes to runtests and the build system are minimal, just adding support for new manifest types really but I will be needing reviews for those. We'll also have to make the buildbot changes to switch over to use these new tests but I expect that to be pretty straightforward. Was Marionette considered? From what little I know (jgriffin and others can correct me), Marionette seems like the logical base for this test suite. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Switching Jetpack to use the runtests.py automation
On 7/15/2014 2:49 PM, Dave Townsend wrote: Since forever Jetpack tests in the Firefox trees have been run using our custom python CFX tool which is based on a fork of an ancient version of mozrunner. This causes us a number of problems. Keeping up with tree visibility rules is hard. Some features from newer versions of mozrunner like crash stack handling aren't available and our attempts to update to the newer mozbase have been blocked on trying to get some of our forked code accepted. It also makes it hard for Mozilla other developers to run our tests as CFX has a very different syntax to the other test suites. We've started investigating switching away from CFX and instead using the python automation that the mochitests use. This would work somewhat similarly to browser-chrome tests, runtests.py will startup Firefox and overlay some XUL and JS on the main window from where we can run the existing JS parts of the Jetpack test suites. There are many benefits here. The runtests.py code is well used and known to be resilient. It supports things like screenshots on failures and crash stacks that Jetpack tests don't currently handle. We'll use manifest files like the other test suites so disabling tests per platform will be easy. Excellent mach integration will make running individual tests simple. It also makes it possible to use commonjs style tests elsewhere in the tree. Release engineering should find managing the Jetpack tests a lot easier as they behave just like other mochitests. My initial experiment last week shows that this will work. The first part of our tests (package tests) is running and passing on my local machine and I expect to have the add-on tests working this week. I wanted to give everyone a heads up about this work to give you all a chance to ask questions or raise objections. The changes to runtests and the build system are minimal, just adding support for new manifest types really but I will be needing reviews for those. We'll also have to make the buildbot changes to switch over to use these new tests but I expect that to be pretty straightforward. ___ I am totally into the sentiment here, but I'm not sure that I'm into your exact plan of action. The Jetpack tests definitely have a lot of issues due to not using the same infrastructure as other test harnesses, so I'd be glad to see that get fixed. On the other hand, the Mochitest harness is already unduly complicated by trying to do too many things--plain Mochitests, chrome Mochitests, browser-chrome tests and all the other variants are all shoehorned in there and it's a mess. I'd rather not shoehorn yet another test framework into the Mochitest umbrella. However! In the modern era Mochitest is mostly comprised of some glue code that uses mozbase modules to do all the hard work. I think you can cobble together a reasonable facsimile that runs the Jetpack tests in a cleaner fashion by using the mozbase modules directly. Alternately, as gps pointed out in his reply, you might want to look at Marionette as a starting point. The Marionette test harness uses mozbase as well, but gives you the power of the Marionette protocol to control the browser. If we were writing the Mochitest harness from scratch today I would base it on top of Marionette. -Ted ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Switching Jetpack to use the runtests.py automation
Gregory Szorc wrote on 07/15/2014 09:04 PM: On 7/15/14, 11:49 AM, Dave Townsend wrote: Since forever Jetpack tests in the Firefox trees have been run using our custom python CFX tool which is based on a fork of an ancient version of mozrunner. This causes us a number of problems. Keeping up with tree visibility rules is hard. Some features from newer versions of mozrunner like crash stack handling aren't available and our attempts to update to the newer mozbase have been blocked on trying to get some of our forked code accepted. It also makes it hard for Mozilla other developers to run our tests as CFX has a very different syntax to the other test suites. We've started investigating switching away from CFX and instead using the python automation that the mochitests use. This would work somewhat similarly to browser-chrome tests, runtests.py will startup Firefox and overlay some XUL and JS on the main window from where we can run the existing JS parts of the Jetpack test suites. There are many benefits here. The runtests.py code is well used and known to be resilient. It supports things like screenshots on failures and crash stacks that Jetpack tests don't currently handle. We'll use manifest files like the other test suites so disabling tests per platform will be easy. Excellent mach integration will make running individual tests simple. It also makes it possible to use commonjs style tests elsewhere in the tree. Release engineering should find managing the Jetpack tests a lot easier as they behave just like other mochitests. My initial experiment last week shows that this will work. The first part of our tests (package tests) is running and passing on my local machine and I expect to have the add-on tests working this week. I wanted to give everyone a heads up about this work to give you all a chance to ask questions or raise objections. The changes to runtests and the build system are minimal, just adding support for new manifest types really but I will be needing reviews for those. We'll also have to make the buildbot changes to switch over to use these new tests but I expect that to be pretty straightforward. Was Marionette considered? From what little I know (jgriffin and others can correct me), Marionette seems like the logical base for this test suite. Adding the tools mailing list, so that members of the A-team are aware of this thread, and can answer appropriately. -- Henrik Skupin Senior Test Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
