On Wed, Apr 27, 2016 at 9:14 PM, Nate Finch <[email protected]> wrote: > So, I don't really think the method of testing should determine where a test > lives or how it is run. I could test the exact same things with a more > common unit test - check the tls config we use when dialing the API is using > tls 1.2, that it only uses these specific ciphersuites, etc. In fact, we > have some unit tests that do just that, to verify that SSL is disabled. > However, then we'd need to remember to write those same tests for every > place we make a tls.Config.
The distinction here is between testing the code (execution) and verifying that the code base follows proscribed constraints, whether technical or organizational. Perhaps you could call the latter "meta-testing". :) For testing the code, it is really useful to have the tests next to the code. For meta-tests, e.g. static-analysis against the code base as a whole, the relationship with any particular package is tenuous at best. So I agree with the others that "tests" which focus on the code-base as a whole, via static analysis or otherwise, should be grouped together in the directory tree. There's prior art with the featuretests package, though it's not quite the same thing. I also agree that Lingo would be an effective tool in cases like this where we are enforcing a code-base-wide policy, e.g. "don't import testing packages in non-test files" or "use the helper function when creating a tls.Config". However, that does not mean we can't wrap calls to Lingo in test methods (more on this below). > > The thing I like about having this as part of the unit tests is that it's > zero friction. They already gate landings. We can write them and run them > them just like we write and run go tests 1000 times a day. They're not > special. There's no other commands I need to remember to run, scripts I > need to remember to set up. It's go test, end of story. I agree that is useful, particularly if we can isolate them, e.g. with "go test -short" or the testing tags that I proposed. I don't think the objection is necessarily about running these "meta" tests via "go test". Personally I kind of like that approach. It wouldn't be hard to write test suites with methods which call out to the tools. That would thereby allow us to meet all our testing needs through "go test" while allowing us to continue using the other tools independently to meet additional specific needs. For me, the objection is more about *where* such tests live. > > The comment about Lingo is valid, though I think we have room for both in > our processes. Lingo, in my mind, is more appropriate at review-time, which > allows us to write lingo rules that may not have 100% confidence. They can > be strong suggestions rather than gating rules. The type of test I wrote > should be a gating rule - there are no false positives. Again, having granular testing tags would allow us to make that distinction in a way that the test tooling could easily distinguish. > > To give a little more context, I wrote the test as a suite, where you can > add tests to hook into the code parsing, so we can trivially add more tests > that use the full parsed code, while only incurring the 16.5 second parsing > hit once for the entire suite. That doesn't really affect this discussion > at all, but I figured people might appreciate that this could be extended > for more than my one specific test. Fair enough. > I certainly wouldn't advocate people > writing new 17 seconds tests all over the place. Oh good! :) -eric -- Juju-dev mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
