Are crashtests allowed to access chrome (e.g. set preferences)?
Hi all, I want to start with crashtests for webrtc crashes which have been fixed recently. Sadly this feature is disabled on mozilla-central and mozilla-aurora. So I would have to set two preferences to get access to all the features. Now I see that crashtests are getting run through the reftest framework and wonder if there is a way to modify preferences before the test, or is that forbidden? In either way, what would be the best way to get those tests into the tree? I assume a place like /dom/media/tests/crashtests/ is a good location? Thanks! Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Are crashtests allowed to access chrome (e.g. set preferences)?
Bobby Holley wrote on 10/15/12 2:12 PM: Now that bug 792029 has landed, you should be able to use SpecialPowers in crashtests. Wow, great response time and great answer! Thank you a lot Bobby! -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Are crashtests allowed to access chrome (e.g. set preferences)?
Ted Mielczarek wrote on 10/15/12 2:16 PM: The reftest manifest format (which crashtest uses, since crashtests run in the reftest harness) has explicit support for setting preferences per-test: http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/README.txt#160 Sounds good. I will make use of it. Sadly it cannot set the preference globally for all the tests referenced by that crashtests.list file. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Are crashtests allowed to access chrome (e.g. set preferences)?
L. David Baron wrote on 10/15/12 4:31 PM: There's a reviewed patch in https://bugzilla.mozilla.org/show_bug.cgi?id=788967 that adds support for this, but apparently it had failures on try. Exactly what I need. Will watch when it lands. Thanks! -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Proposal for reorganizing test directories
Hi all, When I started to work on tests for WebRTC and WebAPI lately I have noticed that there are no clear specifications where tests have to be added. Some are located in a tests subdirectory (like /dom/feature/tests/) while others are under a testing folder (like /dom/testing/mochitest). This is totally confusing and as an internal discussion in the automation and tools team has been shown, we would support a location of tests which are under the feature which means directly beneath the actual source. The proposal would be the following hierarchy: |-- feature/ |-- tests/ |-- browser-chrome/ |-- chrome/ |-- crashtests/ |-- marionette/ |-- mochitests/ |-- reftests/ |-- xpcshell |-- [..]/ So tests for each type of test framework will get their own subfolder under tests. That will make it easier for everyone to get started and we can retain a clear structure of our tests. With this post we want to get feedback from module owners and how they feel with such a reorganization of the tests. If we agree all on it I would like to go ahead and submit patches in smaller chunks most likely on a per module basis. I know that will be a lot of work, so I will do it beside my actual work and try to get as many contributors as possible to helping out via mentored bugs. Once agreed on our team will also add proper documentation on MDN as part of our current documentation revamp project for test frameworks. Thanks a lot, -- Henrik Skupin Software Engineer in Test Mozilla Corporation -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal for reorganizing test directories
Justin Lebar wrote on 10/25/12 10:06 AM: I'd probably be a lot more sympathetic to this proposal if I understood in a concrete way how making my life a little harder here would make your life a little easier. What problem exactly are you trying to solve? Good points. So let me give one example. I was looking at the following URL: http://mxr.mozilla.org/mozilla-central/source/dom/tests/mochitest/ Why have those tests be placed into this location and not beside the actual implementation code? It's hard when you work on tests (e.g. fixing brokenness) and you have to check the implementation. It takes a bit of time you have to spend on figuring out where the source actually lives. I personally like the fact that we don't have the tests in dom/browser-element/tests/mochitest. The extra directory would have just one child, so from my perspective, it would add very little. (I have to type this path all the time.) I don't know how many other folders you'd need to change in a similar way. I don't think I have a problem with it. If there is only one single type of test harness used, we could have the tests directly under a folder with it's name. Once more are getting added we could consider to move e.g. the mochitests to tests/mochitest/. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
How to correctly handle window.location.reload() in a crashtest
Hi, A crashtest for WebRTC requires to reload the current page. So that can be done by window.location.reload(). Sadly this will cause an infinite loop when run in the automation. So which practice do we have to set a flag once the window has been reloaded once? Are there any guides how to do it? One option which comes into my mind is to set a cookie. But I wonder if there are better solutions. Thanks -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: How to correctly handle window.location.reload() in a crashtest
Kyle Huey wrote on 11/15/12 12:26 AM: Can you use an iframe? I tried and it would work if we wouldn't crash in another area. So using an iframe is certainly another crashtest. But thanks for the information! -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Switching Jetpack to use the runtests.py automation
Gregory Szorc wrote on 07/15/2014 09:04 PM: On 7/15/14, 11:49 AM, Dave Townsend wrote: Since forever Jetpack tests in the Firefox trees have been run using our custom python CFX tool which is based on a fork of an ancient version of mozrunner. This causes us a number of problems. Keeping up with tree visibility rules is hard. Some features from newer versions of mozrunner like crash stack handling aren't available and our attempts to update to the newer mozbase have been blocked on trying to get some of our forked code accepted. It also makes it hard for Mozilla other developers to run our tests as CFX has a very different syntax to the other test suites. We've started investigating switching away from CFX and instead using the python automation that the mochitests use. This would work somewhat similarly to browser-chrome tests, runtests.py will startup Firefox and overlay some XUL and JS on the main window from where we can run the existing JS parts of the Jetpack test suites. There are many benefits here. The runtests.py code is well used and known to be resilient. It supports things like screenshots on failures and crash stacks that Jetpack tests don't currently handle. We'll use manifest files like the other test suites so disabling tests per platform will be easy. Excellent mach integration will make running individual tests simple. It also makes it possible to use commonjs style tests elsewhere in the tree. Release engineering should find managing the Jetpack tests a lot easier as they behave just like other mochitests. My initial experiment last week shows that this will work. The first part of our tests (package tests) is running and passing on my local machine and I expect to have the add-on tests working this week. I wanted to give everyone a heads up about this work to give you all a chance to ask questions or raise objections. The changes to runtests and the build system are minimal, just adding support for new manifest types really but I will be needing reviews for those. We'll also have to make the buildbot changes to switch over to use these new tests but I expect that to be pretty straightforward. Was Marionette considered? From what little I know (jgriffin and others can correct me), Marionette seems like the logical base for this test suite. Adding the tools mailing list, so that members of the A-team are aware of this thread, and can answer appropriately. -- Henrik Skupin Senior Test Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Support for 32bit on OS X and testing infrastructure
As I have read [1] that Google will drop support for 32bit mode of Chrome with the upcoming 39 release in November. I wonder what our support plans for 32bit mode on OS X look like. Is it something we want to keep, to allow e.g. users of Chrome, who cannot run a 64bit version of Chrome, to switch over to Firefox? Or is it something we would like to follow? I just wonder because of bugs like 960509, which went unnoticed for a while. Do we actually have test coverage of the 32bit version via TBPL, or not at all? How stable are we? Would we be able to catch regressions in the case of no tests are running for this mode? Via bug 967055 there is at least the question to enable Mozmill tests for the 32bit binary. Before I want to continue with it, I would like to have some feedback here first. Thanks, Henrik [1] http://googleappsupdates.blogspot.de/2014/09/google-chrome-64-bit-for-mac-and-windows.html ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Support for 32bit on OS X and testing infrastructure
Robert Strong wrote on 09/19/2014 06:59 PM: Regarding dropping support, Silverlight on Mac does not support 64 bit and we run it using 32 bit. So at the very least we will need html5 for sites like Netflix before we can drop 32 bit support on OS X. I see. So I will investigate what's necessary to get those builds running in parallel via Mozmill CI. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Support for 32bit on OS X and testing infrastructure
Robert Strong wrote on 09/22/2014 09:50 AM: I see. So I will investigate what's necessary to get those builds running in parallel via Mozmill CI. Specifically, the 32bit plugin-container and libraries that it loads. The one and only plugin we make use of for tests is Adobe Flash. There is no other plugin we currently test with Mozmill. So by enabling tests for 32bit builds, those tests will also run. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Upcoming changes to Mac package layout, signing
Robert Strong wrote on 09/22/2014 11:07 AM: Hi Robert, All of the major changes for Mac v2 signing have landed on the Oak branch. This will allow us to test installing and updating before landing on mozilla-central. [..] If no serious issues are found we are hoping to be able to land on mozilla-central later this week. Thanks a lot for this update. In case of our Mozmill update tests, those have been updated yesterday for Nightly to be able to handle the new signed builds. If all goes well with updates today, I will backport my patch on bug 1063584 to Aurora today. There are only two issues left for the mozmill-automation package, which I hopefully can finish up by today or tomorrow. https://github.com/mozilla/mozmill-automation/milestones/2.0.8.1 That means we should then be able to run our update tests against builds of the oak tree, which should give an extra bit of safety. Cheers, -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Testing framework does not wait for correct dialog sometimes.
ISHIKAWA,chiaki wrote on 10/11/2014 03:22 PM: Hi, But then I realize that the current dialog mechanism may not have such an identifier. (Maybe we can use the string or part of the string that is shown in the dialog as a key to distinguish different dialogs? Of course, change to existing dialog mechanism may be in order. test script can be enhanced later. We can add a wait-for-dialog API with [..] I noticed such erratic behavior of unexpected dialog fooling the test program into thinking it is seeing an expected dialog during |make mozmill| since Can I ask which kind of Mozmill tests are you executing and for which application? For me it sounds like Thunderbird. So what I know is that the devs created another module layer between the tests and Mozmill to handle the specific behaviors. The mentioned wait for API sounds like that. Would be good to see any URL of that implementation. Do you have that? -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Replacing Mozmill driven tests for Firefox with Marionette
Andrew Halberstadt wrote on 05/13/2015 04:24 PM: I thought thunderbird was using a very out-of-date version of mozmill which was already unmaintained anyway? Or did you already upgrade to the latest version? Yes, its a very outdated version of Mozmill from the hotfix-1.5 branch, which we do not support since more than a year. Given that I have to add that we cannot maintain something which is not in use at all. It's sad to say but if something is broken the TB devs will have to get it fixed in their customized Mozmill version. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Replacing Mozmill driven tests for Firefox with Marionette
Hi all, I would like to finally announce that we are going to discontinue any kind of development for the Mozmill framework and tests for Firefox. The reason is that there is simply no man power to further develop this tool. Also it's plainly only myself who has the knowledge of how it works and how it grew over the past 6 years. It's simply not scalable anymore. Since end of last year we (members of the A-Team) were actively working on replacing our tests. As you can imagine there is lots of work to do. To being able to better spread work around we basically have chosen the wider known Marionette framework. It's already in use by some products, has the standardized Webdriver API, and has a better platform for today's requirements compared to Mozmill. So lots of new features and bug fixes went into Marionette during the last quarter. All of those were necessary to support at least the most important features of Mozmill like running different locales, exercising restart tests, and others. As result we now have Marionette in a state to get started in replacing Mozmill tests. We even have some tests converted last quarter, so the important security and update tests could be flipped over to Marionette now. The new test suite itself is named Firefox UI Tests and can be accessed via: https://github.com/mozilla/firefox-ui-tests. Currently I'm working on replacing the Mozmill jobs in Mozmill CI with the Marionette tests. So the first test results should be available soon. As reporting platform we want to support treeherder, which will be integrated over the next couple of weeks. So test results will be broader visible. One other big step forward is that we are now working on getting the update tests for releases to run on RelEng infrastructure. Armen Zambrano is helping us here. With that move the sign-offs from QA for update tests won't be necessary anymore. RelEng will be informed directly about update failures. All the above should give you a quick overview and understanding what's going on with Mozmill and Marionette. For the individual topics I might follow-up with separate blog posts and a more detailed description of work happening in those areas. Please let us know if you have questions or want to participate in that work. I'm always up for mentoring people from the community. Best, -- Henrik Skupin Senior Test Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Data migration from ftp.mozilla.org to S3, 1pm Pacific Tuesday 20th October
Nick Thomas wrote on 10/21/2015 11:46 AM: > * to reach older builds you can use > http://ftp-origin-scl3.mozilla.org/pub/mozilla.org/firefox/ > * please don't get comfortable with that, we'll just be using > archive.mozilla.org soon For how long will the ftp-origin-scl3.mozilla.org host be online? We had to temporarily switch mozdownload to make use of that host, to circumvent the gaps of builds and the changed HTML structure for parsing. It will take us some days to get it all fixed and I don't want to use archive.mozilla.org again as long as not all builds have been transfered over. Thanks -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Firefox-ui-tests – Platform Operations Project of the Month
Hello from Platforms Operations! Once a month we highlight one of our projects to help the Mozilla community discover a useful tool or an interesting contribution opportunity. This month’s project is firefox-ui-tests! What are firefox-ui-tests? === Firefox UI tests are a test suite for integration tests which are based on the Marionette automation framework and are majorly used for user interface centric testing of Firefox. The difference to pure Marionette tests is, that Firefox UI tests are interacting with the chrome scope (browser interface) and not content scope (websites) by default. Also the tests have access to a page object model called Firefox Puppeteer. It eases the interaction with all ui elements under test, and especially makes interacting with the browser possible even across different localizations of Firefox. That is a totally unique feature compared to all the other existing automated test suites. You can read more about it on my blog post: http://www.hskupin.info/2016/06/02/firefox-ui-tests-platform-operations-project-of-the-month/ Update == Just to announce it separately, as of yesterday we bumped our test results on Treeherder to Tier-2, which will give us more visibility to developers on integration branches. Tests per checkin are run on Taskcluster which means it's Linux64 only for the moment. We will expand testing once new platforms are available. Cheers, -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Generic Task Cluster Tasks / File Based Task Scheduling
Gregory Szorc wrote on 02/17/2016 07:59 PM: > * You can now define tasks that are neither "build" nor "test" tasks. This > mechanism is probably where you should place one-off tasks such as linting, > docs generation, code analysis, etc. See > https://hg.mozilla.org/integration/mozilla-inbound/rev/623765c2381e for > more on this feature. This is great. Something else which comes into my mind while reading your email is if we could reduce the turnaround time e.g. for try builds in case such a build only contains test changes and wouldn't need a new binary to be produced. Not sure if that would work but if we could grab an already existent binary from the inbound changeset the try push is based off, a lot of machine power could be saved (compile time vs. download time). -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
firefox-ui-tests are now in mozilla-central with TaskCluster support
Over the last years the formerly known Mozmill tests and now Firefox ui tests have been located in their own repository. That meant that we always got regressions due to changes developers have been made in Firefox. Hunting down those regressions and fixing them was always a time consuming job. Beside all that we were also responsible for merging our branches accordingly to our 6 week cycle. To finally get rid of that I was working all the last weeks to make the following changes: * Added mozharness scripts for executing firefox-ui-tests and enhanced base mozharness modules to cope with various other feature requirements we had. * Moved our tests into mozilla-central under testing/firefox-ui/ and updated test packaging. Backports happened down to Firefox 45.0 so that we will benefit from the upcoming ESR release too. * Updated mozmill-ci (which now runs our firefox-ui-tests) to use test packages across all supported branches down to current Firefox 45.0ESR. With that we can finally get rid of our Github repository. It turned out to not be an easy process given that we do not only run our tests against en-US but also a couple of other locales for nightly and release builds. So a couple of blockers had to be fixed side-by-side in packaging, build notifications via Mozilla Pulse etc. * Added support for TaskCluster to run our tests once the build has been finished. Similar to other tests this only works for linux64 debug right now. Test results can be found on Treeherder in the Fxfn group. Keep in mind that those are still Tier-3, so you will have to make those tests visible. If you have to make changes to our tests and want to push to try, simply use '-u firefox-ui-functional,firefox-ui-functional-e10s' in the try comment (trychooser will be updated soon). With all the above close to be done now, I'm starting to write documentation. This will enable us to push results as Tier-2, and also have a base set for contributors to get started hacking on our tests. If you have questions please ask or join our #automation channel on IRC. Cheers, Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: firefox-ui-tests are now in mozilla-central with TaskCluster support
Matthew N. wrote on 03/01/2016 06:18 PM: > Could you give an overview of what is tested by this suite and how it > differs from mochitest-browser-chrome? It seems one difference is that > some(?) tests run on non-en-US. So Andrew already told a lot, and just to emphasize here we really do not want to work with any JS globals etc when navigating through the ui but always use ui elements directly. The reason is that we also run our tests with a couple of other locales and for each of those the command, access keys and others can differ. The before mentioned firefox-puppeteer library is a helper to make test creation even more easier. So you don't have to remember the real DOM id, label, anon attribute or what-ever when accessing such elements. We have a nice hierarchy like: self.browser.navbar.back_button.click() The number of tests are still low given that we haven't had the time to port more tests from mozmill. That means as of now we have converted only the most important tests including remote security checks for various protocols and some locationbar and privatebrowsing tests. We would like to get more tests added especially those which would require a restart of Firefox, or those for testing shortcuts etc across locales. So we hope that you all might want to create one in case you have a need for. > Could you also add a description to > https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing? Yes I will and this is actually one of my remaining goals this quarter. The work for that will be tracked on bug 1237552. In case you want to know more about the code right now, you can find everything here: http://mxr.mozilla.org/mozilla-central/source/testing/puppeteer/firefox/ http://mxr.mozilla.org/mozilla-central/source/testing/firefox-ui/ Let me know if there are still open questions. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent to deprecate: MacOS 10.6-10.8 support
Mike Hommey wrote on 05/11/2016 05:06 AM: >> The post states "Mozilla will end support for Firefox on OS X 10.6, 10.7, >> and 10.8 in August, 2016." This means that we will end support with the >> Firefox 48 release. i.e. Firefox 48 will not support OS X 10.6-10.8. > > That's why the post should have given a version number instead of an > ambiguous date. Bug 1269790 only landed on mozilla-central. Shouldn't it then also be backported to mozilla-aurora so it will hit the 48.0 release? -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Increased number of e10s related crashes for firefox-ui-tests on mozilla-central and integration branches
Hi, Over the last couple of days I noticed a spike of crashes specifically for our firefox-ui-tests as run on Taskcluster (Linux64, Ubuntu 12.04 docker image). The crashes are completely related to our e10s jobs. There are a lot of different crash signatures, but mostly these are: [@ libpthread-2.15.so + 0xbd84] [@ libc-2.15.so + 0xe7993] https://treeherder.mozilla.org/#/jobs?repo=autoland=firefox%20ui%20e10s Does anyone know if there were related changes for e10s which might have caused those crashes to appear? I'm afraid that we might have some unknown top crashers which would go to aurora by the merge early next week. Thanks -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: The integration/autoland repo
Gregory Szorc wrote on 06/28/2016 08:44 PM: > As of a few minutes ago, when you land commits from MozReview they will be > pushed to https://hg.mozilla.org/integration/autoland instead of > https://hg.mozilla.org/integration/mozilla-inbound. > > For now, think of integration/autoland as just another mozilla-inbound or > fx-team. In fact, the sheriffs will be treating the autoland repo just like > they treat inbound and fx-team. But that will change in the weeks ahead. I'm just curious when autoland will actually be merged to mozilla-central the first time. Given that we have this repository for a while now, and sheriffs merged mozilla-central into it a couple of times, I do not see any single merge of autoland to mozilla-central yet. It's blocking some of my patches to reach Nightly builds. So for me the current situation is not satisfying. Can you please clarify the merge strategy? > *Please do not pull from the autoland repo.* If you do, future changes that > will introduce divergent commits/DAG heads will make your life painful. This gives problems if you have patches for different bugs but which have overlapping code blocks. This requires you to keep already landed commits around until code found its way to mozilla-central and mozilla-inbound. Given that this can take up to a day or more I don't feel satisfied right now in using autoland for my mozreview patches. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: The integration/autoland repo
Gijs Kruitbosch wrote on 07/01/2016 11:47 PM: > On 01/07/2016 20:52, Henrik Skupin wrote: >> I do not see any single merge of autoland to mozilla-central > > https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=61ed5c0d64195c58de57489147046aeaf14252d3 > > is a merge from autoland to m-c earlier today, and you are CC'd on the > bug... I was able to sort out the issue with Carsten on IRC. He just posted a message to this list about the current issues with autoland->mc updates. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Marionette tests for Android are Tier-2 now
Over the last couple of weeks I focused my work a bit on getting cleaner jobs for Marionette on Android. Here 9 out of 10 chunks were basically failing all the time. So lots of work was expected. [1] actually tracked everything. To get more passing tests I did not only skip/disable those affected tests, but also make them work with Fennec as run on the Android emulator. That's not always easy given the missing Gecko log output. So in some cases it was more a try and error scenario. Beside that also other work had to happen, like implementing correct tab handling for Fennec in Marionette [2], coping with the overall slowness of the test execution [3], and others. Given my priorities I was not able to get everything fixed and implemented. But at least we now have mostly greenish test jobs with a low intermittent failure rate. As such a bump to Tier-2 on Treeherder was not a problem anymore, and was done on [4]. And that even for mozilla-aurora and mozilla-beta. The outcome for us who are working on Marionette and the Webdriver implementation is, that we are able to test our bug fixes and patches for new features more easily, and also finally get visual feedback for broken jobs of try builds on mozreview. As such it helps a lot in keeping up with support for both applications. If we stay with the low number of intermittents I will go ahead and request Tier-1 level in the next couple of days. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=3D1297394 [2] https://bugzilla.mozilla.org/show_bug.cgi?id=3D1323185 [3] https://bugzilla.mozilla.org/show_bug.cgi?id=3D1322277 [4] https://bugzilla.mozilla.org/show_bug.cgi?id=3D1332205 Cheers, -- Henrik Skupin Senior Software Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reorganization of Firefox-UI tests in mozilla-central
Gijs Kruitbosch wrote on 09/02/2016 11:37 AM: > I am not familiar with this bit of our build architecture, but as far as > I can tell from a quick look it builds bits of the zipfile off the > objdir. So it collects mochitests from $OBJDIR/_tests/testing/mochitest, > where (again, AFAICT) things get installed via manifests. This is only happening for old test suites but not for more recent ones. But also for reftests we do no longer have to copy all the files to the obj dir first. We package them directly from the source tree into the various *.tests.zip files. But this is all related to directories and not individual files. > We should be doing the same for firefox-ui tests. Not just to avoid > duplication of files in archives, but because otherwise, if we want to > add new tests somewhere else, we both have to add the manifests to the > build system and then modify this build system python file to make sure > they get included in the test archive. That would be wrong. No, this is not what we want to do. Anything related to the obj dir causes a lot of operations and cpu cycles to build those archives (see bug 1283919 comment 1). With having the manifests in place we can also run the tests directly from the source tree. It's true that the situation with a master manifest is not ideal right now, and especially for marionette we have it located in testing/marionette/harness/marionette/test/. Anyway for firefox-ui tests I want to have testing/firefox-ui/tests/(functional|update)-tests.ini. >From here we can reference everything. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reorganization of Firefox-UI tests in mozilla-central
Gijs Kruitbosch wrote on 09/02/2016 01:22 PM: > AIUI, if we can run tests directly from a checkout, we no longer need > the common.tests.zip archive to exist (or at least not to have the tests > in them), so the problem is moot, no? Those files are used by test machines for all of our test suites. That is so they do not have to clone the full tree. I'm not aware of any planned changes here. But build people might know more. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reorganization of Firefox-UI tests in mozilla-central
Gijs Kruitbosch wrote on 09/02/2016 01:22 PM: > AIUI, if we can run tests directly from a checkout, we no longer need > the common.tests.zip archive to exist (or at least not to have the tests > in them), so the problem is moot, no? I think so. There should not be a need to have to run "mach build package-tests" before running any test from the source tree. mach itself takes care about copying necessary bits to the obj dir if necessary. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reorganization of Firefox-UI tests in mozilla-central
L. David Baron wrote on 09/04/2016 10:02 AM: > Recently, sheriffing practices have changed so that intermittent > failure bugs that show up in different tests now have a separate > bugzilla bug for each test they occur in. This causes: > > 1. a large number of rarely-occurring bugs -- enough that > intermittent-orange bugs are a significant fraction of the bugs > reported, but most of the bugs document only a single failure (in > many cases, a problem that has also occurred on other tests, > tracked in different bugs). This encourages ignoring the bugs and > just looking at orangefactor for what's interesting/relevant. > > 2. spreading out a single problem across >100 bugs [1] leads to the > severity of that problem being ignored for extended periods of > time. We are all aware of this and lately the following bug got filed: https://bugzilla.mozilla.org/show_bug.cgi?id=1299274 So maybe a solution is not that far away. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reorganization of Firefox-UI tests in mozilla-central
Gijs Kruitbosch wrote on 09/01/2016 06:24 PM: > As I did over IRC, I would like to strongly object to the continued use > of per-test-type subfolders in our test directories. You can already use > a specific mach command per test type, and the tests are listed in > different manifests, *and* there's all the different filename > conventions (browser_, test_html, test_xul, .js) that > further point out what type of test you're looking at. The subfolders > add nothing useful. The problematic piece here will be the package-tests step which currently picks complete subfolders. It would mean if we mix-up tests for firefox-ui-tests and eg. mochitests all would end-up twice in the common.tests.zip archive. If we want to get away from using subfolders we would have to improve the test archiver (https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/test_archive.py) first to not only collect the directories of referenced manifests, but also only pick those tests which are referenced and leave all others behind. This would apply to all test suites currently covered by this mozbuild action. > Finally, "firefox_ui" (as well as "ui") as a name for a directory is > going to cause all kinds of confusion for people exploring the repo > without detailed knowledge of what's going on. Additionally, it's not > like most of the mochitest-browser tests aren't tests of the Firefox > UI... If we absolutely must have some kind of subdirectory because of > reasons I have yet to hear, I think "integration" would be a better > choice of name as far as subdirs of "test" go (as juxtaposed to "unit" > for our xpcshell tests). I would totally be fine with integration as name, unless we don't hit the above mentioned problem that tests for other harnesses end-up in the same folder. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reorganization of Firefox-UI tests in mozilla-central
Matthew N. wrote on 09/02/2016 01:15 AM: >>> /browser/components/sessionstore/test/firefox_ui/ >>> * update tests:/toolkit/mozapps/update/tests/firefox_ui/ > > Is there a plan to merge those with > /toolkit/mozapps/update/tests/marionette? It seems unusual to use both > when this new test suite is basically a layer on top of Marionette as > you said. Those marionette tests have been created specifically for B2G: https://dxr.mozilla.org/mozilla-central/source/testing/marionette/harness/marionette/tests/update-tests.ini But the firefox-ui harness can only be used for Firefox. Reason is that we have our own testcase classes and several enhancements to the harness. An integration of those into Marionette base was denied to not make Marionette that bloated. As result we have to keep our layering for now. > * Chrome mochitests, plain mochitests and xpcshell share the same prefix > (test_) so they are interleaved together in directory listings. And exactly this makes it hard to only collect the relevant files (tests, support files) via test_archive.py as mentioned in my other email. > "firefox_ui", "ui", and "integration" all overlap with what > mochitest-browser-chrome is about IMO and I think naming the suite > "Firefox-UI" was confusing from the beginning. If I was a new > contributor working on a UI feature and decided I wanted to write tests, > I wouldn't want to be misled into thinking I should write a "Firefox-UI" > test when mochitest-browser-chrome is actually the desired suite. I > would suggest "puppeteer" or "marionette" for directory names to avoid > confusion. As of now we only have puppeteer available for Firefox. And it will not be a hard requirement to use hopefully soon once we decoupled the harness from it. Also we just got Marionette support for Fennec. It means beside testing/puppeteer/firefox it might be good to get shared helper code for Fennec specific tests into testing/puppeteer/fennec. Using marionette here would also be confusing because those tests won't work with the marionette command. Maybe at the end we should try again to get our testcase classes and harness extensions into Marionette again. But depending on the future that we may want to use Selenium for our tests directly, I would wait and not spend my time with such a refactoring. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Reorganization of Firefox-UI tests in mozilla-central
Hello, Via bug 1272145 we want to move our existing Firefox-UI tests from /testing/firefox-ui/ closer to the code which these are testing, so that the former location only contains the test harness and appropriate unit tests. Link to the current set of tests: https://dxr.mozilla.org/mozilla-central/source/testing/firefox-ui/tests For those of you who haven't heard about these tests yet, they are basically Marionette tests with an additional layer (firefox-puppeteer) on top to ease the test creation for ui specific checks. Benefits we have are restarting and quitting the browser, and running the tests in any localized build of Firefox. As listed below you can find the current type of tests and a proposed location: * locationbar tests:/browser/base/content/test/urlbar/firefox_ui/ * private browsing tests: /browser/components/privatebrowsing/test/firefox_ui/ * safe browsing tests: /browser/components/safebrowsing/content/test/firefox_ui/ * session store tests: /browser/components/sessionstore/test/firefox_ui/ * update tests: /toolkit/mozapps/update/tests/firefox_ui/ Do those locations sound good? I have heard at least once that "firefox_ui" might not be the best choice as folder name, but that's how the harness is called, and corresponds to what we have for other harnesses too. Locations for the following tests are not clear yet: * l10n tests (shortcuts): not clear yet (maybe under /browser/base/content/test/) * security tests: not clear yet L10n related tests mostly cover shortcuts and that the correct command is invoked to find failures like bug 1173735. Nearly all of the security tests are running checks against a real server with various SSL certificates (DV, OV, EV) and protocol versions. We will have a meeting soon to determine which of those tests are needed and which ones can be removed. So we might also be able to find the correct location for the tests. For now I would like to know if the proposal is fine or if we should go a completely different route. Thanks -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reorganization of Firefox-UI tests in mozilla-central
Henrik Skupin wrote on 01/09/16 17:37: After a longer time without a response I would like to give a follow-up on where we stand at the moment and what the future will be for those tests. But first, I want to say thanks to everyone who participated in this thread. The received feedback was very helpful and showed that a couple of topics are questionable and needed further discussion. With all that in mind we decided to get away with the original idea of moving the firefox-ui tests to the appropriate components in the state they are in right now. This will ensure that: * no other subdirectory for a new kind of test has to be added * don't make it harder for developers to decide which harness to use Instead our team decided to work towards the goal in transforming the firefox-ui-functional tests into plain Marionette tests. The work as such is not that trivial given that a couple of things have to be obeyed, and not everything is clear yet - especially not how we want to continue in testing nightly and release builds via mozmill-ci. But that shouldn't stop us to make writing Marionette tests for Firefox easier. Just to give an overview, here are the sub-projects I have to work on: * Decouple the Firefox Puppeteer package from FirefoxTestCase and make it an optional (mixin) class, which then could even be used with MarionetteTestCase if wanted. * Get rid of large portions of the firefox-ui harness except for update tests which would still require a separate harness due to their own command line arguments. * Refactor Marionette tests in domain specific jobs. As you know we run all the existing tests (unit, browser, layout, toolkit) in a single job and report as `Mn` to Treeherder. This should be split up into chunks. There will be a follow-up email on this topic soon. * Ensure that those jobs which report as Tier-1 will not use remote testcase data, and consider a new Tier-2 group for others (necessary for lot of the fx-ui security tests) * Refactor existing firefox-ui-functional tests into plain Marionette tests and get browser peer review before moving them to the appropriate tests folders. * Keep automation (mozharness, mozmill-ci) intact in parallel so we do not loose test coverage as needed by QE. I'm sure that the above outlined goals will be in a better alignment with you all. If not, or if there are questions, please let me know. Thanks, -- Henrik Skupin Senior Software Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Reduction of intermittent failures for Marionette tests
Hi all, Over the past months I tried to always have a look on newly filed intermittent failures for Marionette tests on Bugzilla. I tried to triage them early and to bring those up to interested parties. This was working great for a couple of scenarios like the massive amount of hangs and crashes for e10s tests. But I also did a lot of work myself to stabilize the tests by making sure we correctly handle: * The external process controlled by Marionette (tracking bug 1283906) * Chrome windows and tabs (bug 1259055) With the latest landing of the patch on bug 1299216 to improve the crash handling of Marionette, which also adds support for content crashes, I will mark this work as done. The results of all this work you can already see on integration branches. Marionette tests have a very low intermittent failure rate and fail only a handful of times over a day. I hope that we can keep that rate for a while now to make the sheriffs happy. -- Henrik Skupin Senior Software Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Let Firefox shutdown for content crashes via MOZ_CRASHREPORTER_SHUTDOWN
Hi, Recently I have added a feature to Firefox which let it shutdown on its own in case of a content crash happening (bug 1311016). It is available starting with Firefox 52, and the behavior can be controlled via the MOZ_CRASHREPORTER_SHUTDOWN environment variable. This feature is kinda helpful for test harnesses to ensure a proper continuation of the remaining tests. As of now only Marionette makes use of it (bug 1299216) but I feel that other harnesses might also benefit from it. So if there are needs please file an appropriate bug to get this support added to mochitest and others. -- Henrik Skupin Senior Software Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Upcoming rename of the marionette-client package to marionette-harness
Hi, Please note that we will soon rename the marionette-client Python package to marionette-harness. This work is currently happening on bug 1320073. The reason for it is that there is lots of confusion between the package names and the folder names in-tree: > testing/marionette/client -> marionette-driver > testing/marionette/harness -> marionette-client As you can see we will fix one part only because we cannot rename marionette-driver to marionette-client now. We may consider to do so for Firefox 59 which is the overnext ESR release. But the changes for the harness package will definitely also land for the 52 ESR release and as such will be uplifted to mozilla-aurora soon. What does it mean for you? - If your code is located in in the tree you don't have to do anything. My patch will update every single instance where the old package name was used. For external code you will have to update your package dependencies once the code has been landed and the new package released to PyPI. If you don't do so, no further updates will be available via PyPI. In case you have questions please let us know. Thanks, -- Henrik Skupin Senior Software Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Upcoming rename of the marionette-client package to marionette-harness
Xidorn Quan wrote on 02/12/16 00:49:
> Note that this kind of changes may lead to weird Python error. We have
> seen one in bug 1321468.
>
> If you recently see any unexpected Python error when running mach
> commands, try execute "./mach clobber python" first. ("./mach clobber"
> would *not* work because it only purges things inside the objdir, no
> *.pyc files elsewhere.)
Thanks for following up on this topic! One thing I could suggest is to
simply get rid of the generation of .pyc files when running Python
commands. This can be done by exporting the following variable:
export PYTHONDONTWRITEBYTECODE=1
--
Henrik Skupin
Senior Software Engineer
Mozilla Corporation
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Heads up: archive.m.o is no longer updating tinderbox builds, AWSY is effectively dead
Eric Rahm wrote on 06/03/2017 22:17: > I'm unaware of a bug for this decision, but https://archive.mozilla.org > stopped adding tinderbox builds back on January 18th. I haven't heard about that myself. So I'm curious if that is a regression by a change in Releng automation? I would suggest you simply file a bug in RelEng:General Automation in Bugzilla. Please put me on CC. > AWSY also depends on mozdownload [3] which will need to be fixed to handle > not being able to download from archive.m.o as well. I don't believe > mozdownload has an active owner at this point. I'm still the owner of this tool / project, but as said above I haven't been made aware of a change here. So if we do not publish tinderbox builds to archive.mozilla.org anymore - and if that is intended - we indeed would need another way to download this builds. And that most likely by adding support for Taskcluster. Please also note that nothing of my projects including Mozmill-CI is using Tinderbox builds. So if something is broken in the future please file issues for those tools / packages which are not working as expected. Cheers, -- Henrik Skupin Senior Software Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Update: Phabricator and commit series
Mark Côté wrote on 03.07.18 21:34: > We’re aiming for late July for these implementations, which will allow us > to close down MozReview soon after, in time for the data center closure. What does that mean for not yet landed patches? Will there be a way to still retrieve the commit series after this shutdown via Bugzilla? Or are those lost? -- Henrik Skupin Senior Software Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal to adjust testing to run on PGO builds only and not test on OPT builds
Nicholas Alexander wrote on 03.01.19 18:41: > 1) automation builds need a special configuration piece in place to > properly support artifact builds. Almost certainly that's not in place for > PGO builds, since it's such an unusual thing to do: "you want to pack PGO > binaries into a development build... why?" But there's really no reason we > can't do that in automation so I've filed > https://bugzilla.mozilla.org/show_bug.cgi?id=15175323 for these things. This is actually: https://bugzilla.mozilla.org/show_bug.cgi?id=1517533 Thanks for filing those bugs. -- Henrik Skupin Senior Software Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Fission Newsletter #2
Nika Layzell wrote on 09.08.19 19:33: >Wait for document loads to complete before trying to run code inside the >target window, as a process switch may occur after the frame or browser is >created. For frames in content, this usually means waiting for the load >event. While working to get the new screenshot API into Marionette I noticed a couple of problems, which were all related to page load events inside OOP iframes. The underlying reason is that the top-browsing context doesn't wait yet, until all the documents of OOP frames have been finished loading. See bug 1559841. Please be aware of that if you run into intermittent failures. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Using MozPhab without Arcanist
glob wrote on 23.10.19 17:56: > It's available now - make sure you're running the latest version by > running `moz-phab self-update`. That's what I did yesterday, but as it looks like the self-update actually didn't update my version to the latest MozPhab-0.1.55. I will check soon with this version. Thanks! Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Using MozPhab without Arcanist
Piotr Zalewa wrote on 22.10.19 17:25: > Since today MozPhab does not require Arcanist to submit patches in all > supported VCS's. It's an optional and experimental feature. Add the > `--no-arc` option to switch it on. Great to hear, but is there also an ETA when this mode will be available for users of Mercurial? When I use this option I get: > NotImplementedError: Mercurial revisions can't be submitted without Arcanist Thanks -- Henrik Skupin Senior Software Engineer Mozilla Corporation ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
