Re: [lldb-dev] [Openmp-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
On Fri, 18 Oct 2019 at 16:30, David Greene wrote: > I have been viewing test-suite as a kind of second-level/backup testing > that catches things not flagged by "make check-all." Is that a > reasonable interpretation? I was hoping to get some end-to-end tests > under "make check-all" because that's easier for developers to run in > their workflows. It is a common understanding, which makes the test-suite less useful, but that's not really relevant. No one needs to the test-suite as part of their development processes, because we have bots for that. If you have decent piece-wise tests in Clang/LLVM, you really don't need end-to-end tests in Clang/LLVM, because the test-suite will run on bots and you will be told if you break them. Most errors will be picked up by piece-wise tests, and the minority where e2e make a difference can be reactive, rather than pro-active fixing. > Is there a proposal somewhere of what companies would be expected to do? > It's difficult for us engineers to talk to management without a concrete > set of expectations, resource requirements, etc. There were talks about upgrading Buildbot (the service), moving to Jenkins or something else (Travis?). None of them have the majority of the community behind, and that's the main problem. IIRC, the arguments (definitely outdated, probably wrong) were: Buildbot: - Pros: we already have a big infra based on it, it's passable, an upgrade could ameliorate a lot of problems without creating many new ones. - Cons: it's old tech and requires extensive coding to make it work Jenkins: - Pros: Most companies use that already, it's more modetn than Apple's GreenBot is based on that, lots of plugins and expertise in the community - Cons: It requires Java running on the client, which not all targets like. Alternatives require a separate server to run as a slave and connect to targets. Travis: - Pros: It's natively compatible with Github (is is still the case?) and it could be the easiest to connect with our new repo for CI - Cons: less expertise, I guess, and other things I don't really know. > Nice! That's much better. Yes, it won't scale but it's much clearer > about what is being run. Perhaps adding a new column as to what components we test in each one would be nice. --renato ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Openmp-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
On Thu, 17 Oct 2019 at 18:10, David Greene wrote: > From other discussion, it sounds like at least some people are open to > asm tests under clang. I think that should be fine. But there are > probably other kinds of end-to-end tests that should not live under > clang. That is my position as well. Some tests, especially similar to existing ones, are fine. But if we really want to do do complete tests and stress more than just grepping a couple of instructions, should be in a better suited place. > How often would such tests be run as part of test-suite? Every time the TS is executed. Some good work has been put on it to run with CMake etc, so it should be trivial to to run that before commits, but it *does* require more than just "make check-all". On CI, a number of bots run those as often as they can, non-stop. > Honestly, it's not really clear to me exactly which bots cover what, how > often they run and so on. Is there a document somewhere describing the > setup? Not really. The main Buildbot page is a mess and the system is very old. There is a round table at the dev meeting to discuss the path forward. This is not the first, though. We have been discussing this for a number of years, but getting people / companies to commit to testing is not trivial. I created a page for the Arm bots (after many incarnations, it ended up here: http://ex40-01.tcwglab.linaro.org/) to make that simpler. But that wouldn't scale, nor it fixes the real problems. ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Openmp-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
On Thu, 17 Oct 2019 at 16:28, Robinson, Paul wrote: > This is no different than today. Many tests in Clang require a specific > target to exist. Grep clang/test for "registered-target" for example; > I get 577 hits. Integration tests (here called "end-to-end" tests) > clearly need to specify their REQUIRES conditions correctly. Right, why I wrote in the beginning that Clang already has tests like that. So, if all David wants is to extend those tests, than I think this thread was a heck of a time wasting exercise. :) It's nothing new, nothing deeply controversial and it's in the list of things we know are not great, but accept anyway. I personally don't think it's a good idea (for reasons already expressed in this thread), and that has brought me trouble when I was setting up the Arm bots. I had to build the x86 target, even though I never used it, just because of some tests. Today, Arm bots are faster, so it doesn't matter much, but new hardware will still have that problem. I would like, long term, to have the right tests on the right places. > Monorepo isn't the relevant thing. It's all about the build config. I didn't mean it would, per se. Yes, it's about build config, but setting up CI with SVN means you have to actively checkout repos, while in monorepo, they all come together, so it's easier to forget they are tangled, or to hack around build issues (like I did when I marked the x86 to build) and never look back (that was 7 years ago). > I have to say, it's highly unusual for me to make a commit that > does *not* produce blame mail from some bot running lit tests. > Thankfully it's rare to get one that is actually my fault. I was hoping to reduce that. :) > I can't remember *ever* getting blame mail related to test-suite. > Do they actually run? Do they ever catch anything? Do they ever > send blame mail? I have to wonder about that. They do run, on both x86 and Arm at least, in different configurations, including correctness and benchmark mode, on anything between 5 and 100 commits, continuously. They rarely catch much nowadays because the toolchain is stable and no new tests are being added. They work very well, though, for external system tests and benchmarks, and people use it downstream a lot. They do send blame mail occasionally, but only after all the others, and people generally ignore them. Bot owners usually have to pressure people, create bugs, revert patches or just fix the issues themselves. ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Openmp-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
Renato wrote: > If you want to do the test in Clang all the way to asm, you need to > make sure the back-end is built. Clang is not always build with all > back-ends, possibly even none. This is no different than today. Many tests in Clang require a specific target to exist. Grep clang/test for "registered-target" for example; I get 577 hits. Integration tests (here called "end-to-end" tests) clearly need to specify their REQUIRES conditions correctly. > To do that in the back-end, you'd have to rely on Clang being built, > which is not always true. A frontend-based test in the backend would be a layering violation. Nobody is suggesting that. > Hacking our test infrastructure to test different things when a > combination of components is built, especially after they start to > merge after being in a monorepo, will complicate tests and increase > the likelihood that some tests will never be run by CI and bit rot. Monorepo isn't the relevant thing. It's all about the build config. Any test introduced by any patch today is expected to be run by CI. This expectation would not be any different for these integration tests. > On the test-suite, you can guarantee that the whole toolchain is > available: Front and back end of the compilers, assemblers (if > necessary), linkers, libraries, etc. > > Writing a small source file per test, as you would in Clang/LLVM, > running LIT and FileCheck, and *always* running it in the TS would be > trivial. I have to say, it's highly unusual for me to make a commit that does *not* produce blame mail from some bot running lit tests. Thankfully it's rare to get one that is actually my fault. I can't remember *ever* getting blame mail related to test-suite. Do they actually run? Do they ever catch anything? Do they ever send blame mail? I have to wonder about that. Mehdi wrote: > David Greene wrote: >> Personally, I still find source-to-asm tests to be highly valuable and I >> don't think we need test-suite for that. Such tests don't (usually) >> depend on system libraries (headers may occasionally be an issue but I >> would argue that the test is too fragile in that case). >> >> So maybe we separate concerns. Use test-suite to do the kind of >> system-level testing you've discussed but still allow some tests in a >> monorepo top-level directory that test across components but don't >> depend on system configurations. >> >> If people really object to a top-level monorepo test directory I guess >> they could go into test-suite but that makes it much more cumbersome to >> run what really should be very simple tests. > > The main thing I see that will justify push-back on such test is the > maintenance: you need to convince everyone that every component in LLVM > must also maintain (update, fix, etc.) the tests that are in other > components (clang, flang, other future subproject, etc.). Changing the > vectorizer in the middle-end may require now to understand the kind of > update a test written in Fortran (or Haskell?) is checking with some > Hexagon assembly. This is a non-trivial burden when you compute the > full matrix of possible frontend and backends. So how is this different from today? If I put in a patch that breaks Hexagon, or compiler-rt, or LLDB, none of which I really understand... or omg Chrome, which isn't even an LLVM project... it's still my job to fix whatever is broken. If it's some component where I am genuinely clueless, I'm expected to ask for help. Integration tests would not be any different. Flaky or fragile tests that constantly break for no good reason would need to be replaced or made more robust. Again this is no different from any other flaky or fragile test. I can understand people being worried that because an integration test depends on more components, it has a wider "surface area" of potential breakage points. This, I claim, is exactly the *value* of such tests. And I see them breaking primarily under two conditions. 1) Something is broken that causes other component-level failures. Fixing that component-level problem will likely fix the integration test as well; or, the integration test must be fixed the same way as the component-level tests. 2) Something is broken that does *not* cause other component-level failures. That's exactly what integration tests are for! They verify *interactions* that are hard or maybe impossible to test in a component-level way. The worry I'm hearing is about a third category: 3) Integration tests fail due to fragility or overly-specific checks. ...which should be addressed in exactly the same way as our overly fragile or overly specific component-level tests. Is there some reason they wouldn't be? --paulr ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Openmp-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
On Wed, 16 Oct 2019 at 21:00, David Greene wrote: > Can you elaborate? I'm talking about very small tests targeted to > generate a specific instruction or small number of instructions. > Vectorization isn't the best example. Something like verifying FMA > generation is a better example. To check that instructions are generated from source, a two-step test is the best approach: - Verify that Clang emits different IR for different options, or the right IR for a new functionality - Verify that the affected targets (or at least two of the main ones) can take that IR and generate the right asm Clang can emit LLVM IR for any target, but you don't necessarily need to build the back-ends. If you want to do the test in Clang all the way to asm, you need to make sure the back-end is built. Clang is not always build with all back-ends, possibly even none. To do that in the back-end, you'd have to rely on Clang being built, which is not always true. Hacking our test infrastructure to test different things when a combination of components is built, especially after they start to merge after being in a monorepo, will complicate tests and increase the likelihood that some tests will never be run by CI and bit rot. On the test-suite, you can guarantee that the whole toolchain is available: Front and back end of the compilers, assemblers (if necessary), linkers, libraries, etc. Writing a small source file per test, as you would in Clang/LLVM, running LIT and FileCheck, and *always* running it in the TS would be trivial. --renato ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Openmp-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
On Wed, Oct 16, 2019 at 6:05 PM David Greene wrote:
> > I'm inclined to the direction suggested by others that the monorepo is
> > orthogonal to this issue and top level tests might not be the right
> thing.
> >
> > lldb already does end-to-end testing in its tests, for instance.
> >
> > Clang does in some tests (the place I always hit is anything that's
> > configured API-wise on the MCContext - there's no way to test that
> > configuration on the clang boundary, so the only test that we can write
> is
> > one that tests the effect of that API/programmatic configuration done by
> > clang to the MCContext (function sections, for instance) - in some cases
> > I've just skipped the testing, in others I've written the end-to-end test
> > in clang (& an LLVM test for the functionality that uses llvm-mc or
> > similar)).
>
> I'd be totally happy putting such tests under clang. This whole
> discussion was spurred by D68230 where some noted that previous
> discussion had determined we didn't want source-to-asm tests in clang
> and the test update script explicitly forbade it.
>
> If we're saying we want to reverse that decision, I'm very glad!
>
Unfortunately LLVM's community is by no means a monolith, so my opinion
here doesn't mean whoever expressed their opinion there has changed their
mind.
& I generally agree that end-to-end testing should be very limited - but
there are already some end-to-end-ish tests in clang and I don't think
they're entirely wrong there. I don't know much about the vectorization
tests - but any test that requires a tool to maintain/generate makes me a
bit skeptical and doubly-so if we were testing all of those end-to-end too.
(I'd expect maybe one or two sample/example end-to-end tests, to test
certain integration points, but exhaustive testing would usually be left to
narrower tests (so if you have one subsystem with three codepaths {1, 2, 3}
and another subsystem with 3 codepaths {A, B, C}, you don't test the full
combination of {1, 2, 3} X {A, B, C} (9 tests), you test each set
separately, and maybe one representative sample end-to-end (so you end up
with maybe 7-8 tests))
Possible I know so little about the vectorization issues in particular that
my thoughts on testing don't line up with the realities of that particular
domain.
- Dave
>
> -David
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Openmp-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
> I'm inclined to the direction suggested by others that the monorepo is > orthogonal to this issue and top level tests might not be the right thing. > > lldb already does end-to-end testing in its tests, for instance. > > Clang does in some tests (the place I always hit is anything that's > configured API-wise on the MCContext - there's no way to test that > configuration on the clang boundary, so the only test that we can write is > one that tests the effect of that API/programmatic configuration done by > clang to the MCContext (function sections, for instance) - in some cases > I've just skipped the testing, in others I've written the end-to-end test > in clang (& an LLVM test for the functionality that uses llvm-mc or > similar)). I'd be totally happy putting such tests under clang. This whole discussion was spurred by D68230 where some noted that previous discussion had determined we didn't want source-to-asm tests in clang and the test update script explicitly forbade it. If we're saying we want to reverse that decision, I'm very glad! -David ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Openmp-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
"Robinson, Paul via Openmp-dev" writes: >> I always ran check-all before every patch, FWIW. > > Yep. Although I run check-all before *starting* on a patch, to make sure > the starting point is clean. It usually is, but I've been caught enough > times to be slightly wary. This is interesting. I literally have never seen a clean check-all. I suspect that is because we have more components built than (most?) others along with multiple targets. -David ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Openmp-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
"Robinson, Paul via Openmp-dev" writes:
> David Greene, will you be at the LLVM Dev Meeting? If so, could you sign
> up for a Round Table session on this topic? Obviously lots to discuss
> and concerns to be addressed.
That's a great idea. I will be there. I'm also hoping to help run a
routable on complex types so we'll need times that don't conflict. What
times work well for folks?
> (1) Executable tests. These obviously require an execution platform; for
> feasibility reasons this means host==target and the guarantee of having
> a linker (possibly but not necessarily LLD) and a runtime (possibly but
> not necessarily including libcxx). Note that the LLDB tests and the
> debuginfo-tests project already have this kind of dependency, and in the
> case of debuginfo-tests, this is exactly why it's a separate project.
Ok. I'd like to learn more about debuginfo-tests and how they're set
up.
> (2) Non-executable tests. These are near-identical in character to the
> existing clang/llvm test suites and I'd expect lit to drive them. The
> only material difference from the majority(*) of existing clang tests is
> that they are free to depend on LLVM features/passes. The only difference
> from the majority of existing LLVM tests is that they have [Obj]{C,C++} as
> their input source language.
Right. These are the kinds of tests I've been thinking about.
-David
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
