This is all fantastic, especially the codegen tests. Does buildbot record the codegen tests in a way such that they're potentially trackable on isrustfastyet?
On Tue, Jul 16, 2013 at 8:33 PM, Graydon Hoare <[email protected]> wrote: > Hi, > > I've added some new machinery to the testing systems in rust and wanted > to draw a little attention to them. At least this will all be true > shortly when https://github.com/mozilla/rust/pull/7829 lands. > > They're (lightly) documented at the bottom of: > > https://github.com/mozilla/rust/wiki/Doc-unit-testing > > and > > https://github.com/mozilla/rust/wiki/Note-testsuite > > The point-form version is: > > - #[bench] tests can now save their results as json. > run the testrunner with --save-metrics=foo.json > > - If you want to avoid regressing on something, use > --ratchet-metrics=foo.json instead; it will save and > reload with each run, and consider any regression beyond > a given noise threshold to be a test-fail. > > (Removing, renaming, omitting or adding metrics does not cause > failure. it emits a warning and assumes you meant to do it.) > > - If you configure with --ratchet-bench then all the crate #[bench] > benchmarks will be ratcheted in your workspace. We will turn this > on on the buildbot soon-ish. At that point, adding a #[bench] > function will make it impervious to perf regressions, so add them > regularly but also with some care and thought. > > - If you're on a particularly noisy machine, you can override the > inferred noise thresholds with --ratchet-noise-percent=N > > - There are new compiletests in src/test/codegen. These consist > of pairs of files, foo.rs and foo.cc, each containing a non-mangled > extern C function 'test'. The runner will compile the rust one with > rustc and the C++ one with clang, extract their LLVM bitcode, > disassemble it and compare the sizes. The result is recorded in > a metrics file and _ratcheted by default_. > > - The codegen tests also leave behind foo-extract.ll and > foo-clang-extract.ll files, which are helpful for eyeballing > problems in our codegen. Please populate the codegen directory > liberally. All you need to do is write a function in rust and > a function that "does the same thing" in C++. > > If you have any questions, or find none of it works for you and want > help making it work, please let me know. I'll hopefully turn on > --ratchet-bench and export the metrics files from the buildbot soon. > > -Graydon > _______________________________________________ > Rust-dev mailing list > [email protected] > https://mail.mozilla.org/listinfo/rust-dev >
_______________________________________________ Rust-dev mailing list [email protected] https://mail.mozilla.org/listinfo/rust-dev
