On Wed, Nov 16, 2016, at 03:27 AM, lgrat...@gmail.com wrote: > Hello, > > *I posted this on multiple mailing lists, I'm not sure which is the right > place for these questions.
Hi Gratian, Finding the right communication channel always feels like a problem at Mozilla. :) dev-builds is a fine venue for this discussion. > I'm a developer on the Microsoft Visual C++ compiler (code optimizer) and > I'm looking into extending our test suite with popular open-source > projects. This helps us to find bugs earlier, ensuring that these > projects are not broken by frontend or backend changes. It also helps the > projects themselves, by making upgrades to the never compiler easier - > ideally without any problems. This sounds great! We always struggle with updating to a new toolchain, whether with compiler bugs or issues in our codebase surfaced by correctness fixes in the compiler. For example, here's the dependency tree of bugs we fixed that are marked as blocking our "Support Visual C++ 2015" bug: https://bugzilla.mozilla.org/showdependencytree.cgi?id=1119082&hide_resolved=0 > I have several questions about building Firefox on Windows and especially > about running the correctness tests: > > 1. What is the latest version of Visual Studio that should be used? I > tried first VS2015 Update 3 and there seems to be a C++ error about > "constexpr". Next I tried Update 2 and that one finishes the build. We are using VS2015u3 for our builds in automation currently, so it should work: https://dxr.mozilla.org/mozilla-central/rev/79feeed4293336089590320a9f30a813fade8e3c/browser/config/tooltool-manifests/win32/releng.manifest#32 Are you building from the mozilla-central repository (our development repository) or something else? > 2. What are the tests that should be run to test correctness? The idea > here is to ensure that new optimizations don't introduce new bugs, which > should be exposed by new test failures in Firefox. I looked over the QA > Automated testing page and there seem to be a lot of different test > suites - running all is probably not feasible. Is there a subset that is > run when a checkin is done, for example? We run a lot of tests on every Firefox checkin. Here are all the Windows build and test jobs from an arbitrary recent mozilla-central revision: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=f8ba9c9b401f57b0047ddd6932cb830190865b38&filter-searchStr=Windows The secret decoder key for all those letters is here (I'm not sure if it's fully kept up-to-date): https://treeherder.mozilla.org/userguide.html We do have an elaborate system that tries not to run every single test on every single checkin, but toolchain changes have a way of showing up as test failures in very strange places, especially with things like our PGO builds. (Ask me sometime about debugging an issue that showed up when I turned on PGO in the Firefox 3 release cycle...) That being said, if you were going to run any tests I would probably recommend some of the Mochitest test suites, since they launch a browser and run a lot of HTML+JS, which exercises a good amount of our code, and the JIT tests, which exercise the JavaScript engine pretty heavily. The JS team tends to write more complicated C++ than other groups within Mozilla, so they also hit a lot of compiler issues. > 3. What Windows OS do you use for testing? Window 10, Windows 8, etc? Any > other configuration work needed, such as disabling the > antivirus/firewall? I will have to confirm specifics, but we currently run tests on Windows 7 (x86) and Windows 8 (x64), with just a small handful of tests running on Windows 10 (x64). I'm fairly certain we do a lot of configuration on our test machines, but I don't have the details handy. We generally attempt to restrict network access from our test suites so that network issues do not cause test failures, but we do run multiple servers on localhost during testing, so at the very least that needs to be enabled. Note also that we do our builds and tests on separate machines in automation. Running the tests from a build on the same machine should work fine, that's what developers do locally, it just doesn't scale well in automation and we also want to do our builds on Server and run our tests on a desktop OS. > After the build completed, I tried to run a few of the tests and there > seem to be many failures even with a debug build. I assume that the > checked in tests are supposed to pass, so it's likely a Windows config > problem or failure to access extra files from the Web. > > For example, ./mach reftests had about 700 failures. ./mach gtest, > jsapi-tests, jsbrowsertest had none. The tests were run under Windows 10 > Anniversary update with the AV disabled. Many tests are extremely sensitive to the OS environment, unfortunately. I don't think we have reftests running in Windows 10, so there's likely to be issues there due to test expectations. You might have better luck with them on a Windows 7 machine, which we do run in automation. Some test suites are not as sensitive. In your case maybe you'd get the best mileage out of running the JIT tests for now? I don't think there's a standalone mach command to run them, sadly, but I think `./mach build js/src/check-jit-test` will run them. You will probably need to add `ac_add_options --enable-js-shell` to your mozconfig to make that work. I'd love to get you sorted out, if only for the selfish reasons that you can fix issues before new versions of VC get released. :) Regards, -Ted _______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
