On 04/27/2016 09:51 PM, Nate Finch wrote: > So, this is exactly why I didn't want to mention the nature of the > test, because we'd get sidetracked. I'll make another thread to talk > about that specific test. > > I do still want to talk about what we can do for unit tests that take > a long time. I think giving developers the option to skip long tests > is handy - getting a reasonable amount of coverage when you're in the > middle of the develop/test/fix cycle. It would be really useful for > when you're making changes that affect a lot of packages and so you > end up having to run full tests over and over. Of course, running > just the short tests would not give you 100% confidence, but once > you've fixed everything so the short tests pass, *then* you could do a > long run for thorough coverage.
I believe Cheryl has something like this in the works and will be sending a note out on it soon. > This is a very low friction way to increase developer productivity, > and something we can implement incrementally. It can also lead to > better test coverage over all. If you write 10 unit tests that > complete in milliseconds, but were thinking about writing a couple > longer-running unit tests that make sure things are working > end-to-end, you don't have the disincentive of "well, this will make > everyone's full test runs 30 seconds longer", since you can always > skip them with -short. > > The only real negative I see is that it makes it less painful to write > long tests for no reason, which would still affect landing times.... > but hopefully everyone is still aware of the impact of long-running > tests, and will avoid them whenever possible. I will gently point out that we were prepared to land a test that takes ~17s to run without discussion. The motivations are honest and good, but how many others think the same? This is how our test suite grows to be unmanageable. I also agree with Andrew that the nature of the test should be the delineating factor. Right now we tend to view everything through the lens of the Go testing suite; it's a hammer, and everything is a nail. Moving forward, I think we should try much harder to delineate between the different types of tests in the so-called test pyramid, <http://martinfowler.com/bliki/TestPyramid.html> place like tests with like tests, and then run classes of tests when and where they're most appropriate. This is definitely static analysis and should be run in the same stage as other static analysis tools. - Katherine
-- Juju-dev mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
