Re: [lldb-dev] [cfe-dev] [Openmp-dev] [llvm-dev] RFC: End-to-end testing
On Wed, Oct 16, 2019 at 1:09 PM Roman Lebedev via cfe-dev < cfe-...@lists.llvm.org> wrote: > FWIW I'm personally cautiously non-optimistic about this, > but maybe i'm just not seeing the whole picture of the proposal. > > Both checking final asm, and checking more than one layer of abstraction > feels overreaching and very prone to breakage/too restrictful. > Even minimal changes to the scheduling for particular CPU can cause many > instructions to reorder. > I'm not sure what effect that will have on middle-end pass development, > too. > > A change affects these end-to-end tests, what then? > Just blindly regenerate every affected test? > This will be further complicated once clang isn't the only upstream > front-end. > Agreed that the broader a test is, the more careful one has to be about making it reliable in spite of other changes - sometimes that's really difficult (if you're trying to get a particular instruction selection or register allocation) but in others it can be fairly reliable if done carefully to sufficiently restrict optimizations, etc. (having function calls to external functions to act as sinks/sources for values, etc, for instance - picking places where the output is already "optimal" and trivially/obviously so (for whatever set of constraints you've provided - not heroic optimizations, etc) to ensure that it's fairly stable) - Dave > > Roman. > > On Wed, Oct 16, 2019 at 11:00 PM David Greene via cfe-dev > wrote: > > > > Renato Golin via Openmp-dev writes: > > > > > We already have tests in clang that check for diagnostics, IR and > > > other things. Expanding those can handle 99.9% of what Clang could > > > possibly do without descending into assembly. > > > > I agree that for a great many things this is sufficient. > > > > > Assembly errors are more complicated than just "not generating VADD", > > > and that's easier done in the TS than LIT. > > > > Can you elaborate? I'm talking about very small tests targeted to > > generate a specific instruction or small number of instructions. > > Vectorization isn't the best example. Something like verifying FMA > > generation is a better example. > > > > -David > > ___ > > cfe-dev mailing list > > cfe-...@lists.llvm.org > > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev > ___ > cfe-dev mailing list > cfe-...@lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev > ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [cfe-dev] [Openmp-dev] [llvm-dev] RFC: End-to-end testing
On Wed, Oct 16, 2019 at 12:54 PM David Greene via cfe-dev < cfe-...@lists.llvm.org> wrote: > Renato Golin via Openmp-dev writes: > > > But if we have some consensus on doing a clean job, then I would > > actually like to have that kind of intermediary check (diagnostics, > > warnings, etc) on most test-suite tests, which would cover at least > > the main vectorisation issues. Later, we could add more analysis > > tools, if we want. > > I think this makes a lot of sense. > > > It would be as simple as adding CHECK lines on the execution of the > > compilation process (in CMake? Make? wrapper?) and keep the check > > files with the tests / per file. > > Yep. > > > I think we're on the same page regarding almost everything, but > > perhaps I haven't been clear enough on the main point, which I think > > it's pretty simple. :) > > Personally, I still find source-to-asm tests to be highly valuable and I > don't think we need test-suite for that. Such tests don't (usually) > depend on system libraries (headers may occasionally be an issue but I > would argue that the test is too fragile in that case). > > So maybe we separate concerns. Use test-suite to do the kind of > system-level testing you've discussed but still allow some tests in a > monorepo top-level directory that test across components but don't > depend on system configurations. > I'm inclined to the direction suggested by others that the monorepo is orthogonal to this issue and top level tests might not be the right thing. lldb already does end-to-end testing in its tests, for instance. Clang does in some tests (the place I always hit is anything that's configured API-wise on the MCContext - there's no way to test that configuration on the clang boundary, so the only test that we can write is one that tests the effect of that API/programmatic configuration done by clang to the MCContext (function sections, for instance) - in some cases I've just skipped the testing, in others I've written the end-to-end test in clang (& an LLVM test for the functionality that uses llvm-mc or similar)). > If people really object to a top-level monorepo test directory I guess > they could go into test-suite but that makes it much more cumbersome to > run what really should be very simple tests. > >-David > ___ > cfe-dev mailing list > cfe-...@lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev > ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
