On 12/19/2015 07:57 PM, Remi Bergsma wrote:
> I disagree with testing based on complexity. You simply cannot know the > implications upfront, as that is why you run the tests. What seems small, can > break it all. > > Example: > This commit seems an easy_fix, right? Just a findbugs issue resolved. > https://github.com/apache/cloudstack/commit/6a4927f660f776bcbd12ae45f4e63ae2c2e96774 Ok, definition of easy is for _me_: fixing a typo in a debug log output, adding license headers, fixing a comment. adding documentation. So we just have minor and major, fine for me. But would you run integration tests for adding missing license header? > It was just merged indeed, exactly as you propose. But it did cause a major > outage. And I don’t even use HyperV. > Details: https://github.com/apache/cloudstack/pull/761 > > This is why all PRs to 4.6 and 4.7 (200+) that we merged over the last couple > of months were tested against a real cloud. No easy fixes and minor changes. > We always need to run the full integration tests. This would be the best case (and good job btw!) but for how long can you made it without automation? We need automated tests, and if possible full blown automated integration testing. I agree. > When new functionality is proposed, there aren’t many people willing to write > unit and integration tests to cover it. Until that changes, testing can only > guard whatever the tests cover. And when we merge new stuff without tests, > the total coverage goes down making the tests less relevant. In fact, when we > resolve a bug we should write a tests along with it. I know of one guy that > does that on a regular basis. It is ~2016. No excuse. I know we can not test everything, we are dealing with hardware. But I would rather say, "merge it, it covers tests and they passed" then "merge it, it has 2 LGTM". > It’s not so simple as it seems unfortunately. It has never been simple, and I didn't say so, but it should be our goal to get there. Right? Regards René
