I have a bit of concern about this sort of thing - worrying it'll lead to people being less cautious about writing the more isolated tests. That said, clearly there's value in end-to-end testing for all the reasons you've mentioned (& we do see these problems in practice - recently DWARF indexing broke when support for more nuanced language codes were added to Clang).
Dunno if they need a new place or should just be more stuff in test-suite, though. On Tue, Oct 8, 2019 at 9:50 AM David Greene via cfe-dev < cfe-...@lists.llvm.org> wrote: > [ I am initially copying only a few lists since they seem like > the most impacted projects and I didn't want to spam all the mailing > lists. Please let me know if other lists should be included. ] > > I submitted D68230 for review but this is not about that patch per se. > The patch allows update_cc_test_checks.py to process tests that should > check target asm rather than LLVM IR. We use this facility downstream > for our end-to-end tests. It strikes me that it might be useful for > upstream to do similar end-to-end testing. > > Now that the monorepo is about to become the canonical source of truth, > we have an opportunity for convenient end-to-end testing that we didn't > easily have before with svn (yes, it could be done but in an ugly way). > AFAIK the only upstream end-to-end testing we have is in test-suite and > many of those codes are very large and/or unfocused tests. > > With the monorepo we have a place to put lit-style tests that exercise > multiple subprojects, for example tests that ensure the entire clang > compilation pipeline executes correctly. We could, for example, create > a top-level "test" directory and put end-to-end tests there. Some of > the things that could be tested include: > > - Pipeline execution (debug-pass=Executions) > - Optimization warnings/messages > - Specific asm code sequences out of clang (e.g. ensure certain loops > are vectorized) > - Pragma effects (e.g. ensure loop optimizations are honored) > - Complete end-to-end PGO (generate a profile and re-compile) > - GPU/accelerator offloading > - Debuggability of clang-generated code > > Each of these things is tested to some degree within their own > subprojects, but AFAIK there are currently no dedicated tests ensuring > such things work through the entire clang pipeline flow and with other > tools that make use of the results (debuggers, etc.). It is relatively > easy to break the pipeline while the individual subproject tests > continue to pass. > > I realize that some folks prefer to work on only a portion of the > monorepo (for example, they just hack on LLVM). I am not sure how to > address those developers WRT end-to-end testing. On the one hand, > requiring them to run end-to-end testing means they will have to at > least check out and build the monorepo. On the other hand, it seems > less than ideal to have people developing core infrastructure and not > running tests. > > I don't yet have a formal proposal but wanted to put this out to spur > discussion and gather feedback and ideas. Thank you for your interest > and participation! > > -David > _______________________________________________ > cfe-dev mailing list > cfe-...@lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev >
_______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev