On Wed, Nov 16, 2016 at 12:25 PM, Gratian Lup <lgrat...@gmail.com> wrote:
> On Wednesday, November 16, 2016 at 5:23:58 AM UTC-8, Ted Mielczarek wrote: > > Gratian, > > > > One of my coworkers reminded me of something that might be an option for > > you--we have scripts that would allow you to provide a Firefox build > > that you generated (at a publicly accessible URL) and trigger test jobs > > on that build in our CI infrastructure. If that's something that sounds > > useful to you we can definitely make that happen. > > > > You'd have to produce a Firefox build, run the `mach package` and `mach > > package-tests` targets, upload a few zip files from the $objdir/dist > > directory to somewhere accessible via public HTTP and then run a Python > > script to schedule test jobs against those files. > > > > -Ted > > Hi Ted, > > Thanks a lot for your help! > Using Windows 7 or 8 to do the tests should be fine - I actually don't see > any reason it shouldn't be OK to also do other builds and tests on them > when not used for Firefox. > > The idea of testing on your infrastructure is tempting, but probably would > consume too many resources, since this new testing system is intended to be > used both overnight against the latest good compiler build, but also by > every developer on its own while working on new features - this would be > quite a lot of people. Doing a test build now would still be a good idea, > though, at least to see if everything passes in the right environment. > Something Mozilla has been wanting to do for a while is stand up "nightly" builds using the latest, unreleased toolchain from upstream. This would help us identify compiler compatibility issues early. And it may even help compiler toolchain developers find unwanted regressions. In theory, Microsoft developers could gain access to our "Try" server, which allows you to push changes made to Firefox in version control (including its automation config) and run the results on our infrastructure. We have a file in the repository referencing a zip file of the Visual Studio toolchain to use. So, it would be possible for someone at Microsoft to create a zip file of an unreleased VC++ toolchain and have our automation run against that. Of course, there are scaling and privacy issues to work out. (I'm guessing Microsoft LCA will have an opinion on mostly-in-the-public-domain Mozilla infrastructure accessing pre-release Microsoft software.) > > I have a few more questions about running the tests: > > 1. How exactly should the build artifacts be copied to the test machine? > Something like ./mach package? After copying over, running the tests with > ./mach is going to pick the binaries after copying, or is some ./mach > "unpack" needed? I assume the entire mozilla-central enlistment is also > required. > `mach package` will get you a standalone archive or installer suitable for deploying to test machines. For historical reasons, not all `mach *test` commands integrate really well with pre-built archives/installers. There is a whole other layer that deals with running tests in automation. That being said, some `mach` commands do accept an argument with an alternate path to a binary to use. e.g. `mach reftest --appname <path>` and `mach mochitest --appname <path>`. Running tests from arbitrary binaries is definitely something we'd like to make a bit easier. So if there's a feature you'd like, don't hesitate to ask. > 2. Can I see on Treeherder the exact command line that was used to launch > the test suite? I looked over the log files and didn't find anything like > that. > In some tests. e.g. in https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1479739510/mozilla-central_win7_vm_test-web-platform-tests-1-bm129-tests1-windows-build17.txt.gz the main command is: c:/mozilla-build/python27/python' '-u' 'scripts/scripts/web_platform_tests.py' '--cfg' 'web_platform_tests/prod_config_windows.py' '--test-type=testharness' '--total-chunks' '5' '--this-chunk' '1' '--blob-upload-branch' 'mozilla-central' '--download-symbols' 'ondemand' This is invoking a tool called "mozharness" which is our "execute in automation" layer. This tool performs tons of setup work, such as obtaining a version control checkout, creating Python virtualenvs, etc. Once that's all in place, it executes a test runner. e.g. 08:22:59 INFO - Calling ['c:\\slave\\test\\build\\venv\\Scripts\\python', '-u', 'c:\\slave\\test\\build\\tests\\web-platform\\runtests.py', '--log-raw=-', '--log-raw=c:\\slave\\test\\build\\blobber_upload_dir\\wpt_raw.log', '--log-errorsummary=c:\\slave\\test\\build\\blobber_upload_dir\\wpt_errorsummary.log', '--binary=c:\\slave\\test\\build\\application\\firefox\\firefox.exe', '--symbols-path=https://queue.taskcluster.net/v1/task/G49XlpNvSWK9pkFLL2pT_A/artifacts/public/build/firefox-53.0a1.en-US.win32.crashreporter-symbols.zip', '--stackwalk-binary=c:\\slave\\test\\build\\win32-minidump_stackwalk.exe', '--stackfix-dir=c:\\slave\\test\\build\\tests\\bin', '--test-type=testharness', '--disable-e10s', '--total-chunks=5', '--this-chunk=1', '--prefs-root=c:\\slave\\test\\build\\tests\\web-platform/prefs', '--processes=1', '--config=c:\\slave\\test\\build\\tests\\web-platform/wptrunner.ini', '--ca-cert-path=c:\\slave\\test\\build\\tests\\web-platform/certs/cacert.pem', '--host-key-path=c:\\slave\\test\\build\\tests\\web-platform/certs/web-platform.test.key', '--host-cert-path=c:\\slave\\test\\build\\tests\\web-platform/certs/web-platform.test.pem', '--certutil-binary=c:\\slave\\test\\build\\tests/bin/certutil'] with output_timeout 1000 There is a lot of complexity here. Most of it unique to our automation infrastructure. I recommend you avoid mozharness completely and try to execute things with "mach." (This is the direction we're trying to take things.) Again, if you need features to make that easier, please ask and we'll see what we can do. > I'm going to try again with Update 3, I might have used instead the most > recent build, which can indeed show some new errors, the frontend team do a > lot of changes. > > Thanks, > Gratian > _______________________________________________ > dev-builds mailing list > dev-builds@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-builds >
_______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
