Hi Rene, We’re at the stage that running the integration tests is a one-liner and 7 hour wait [1]. It’s running a subset [2] of available Marvin tests.
The problem is that Marvin is slow and sometimes flaky (error found, rerun is fine). So the test results need manual review before you can publish them. This alone is a lot of work. The biggest issue is that there is no community hardware that can run this and there is no way to hook tests into Github either (unless we get full access). About LGTM, I usually want one based on the integration tests and one based on code review. Regards, Remi [1] https://github.com/schubergphilis/MCT-shared/blob/master/helper_scripts/cloudstack/check-pr.sh [2] https://github.com/schubergphilis/MCT-shared/blob/master/helper_scripts/cloudstack/run_marvin_router_tests.sh On 19/12/15 23:11, "Rene Moser" <m...@renemoser.net> wrote: > > >On 12/19/2015 07:57 PM, Remi Bergsma wrote: > >> I disagree with testing based on complexity. You simply cannot know the >> implications upfront, as that is why you run the tests. What seems small, >> can break it all. >> >> Example: >> This commit seems an easy_fix, right? Just a findbugs issue resolved. >> https://github.com/apache/cloudstack/commit/6a4927f660f776bcbd12ae45f4e63ae2c2e96774 > >Ok, definition of easy is for _me_: fixing a typo in a debug log output, >adding license headers, fixing a comment. adding documentation. > >So we just have minor and major, fine for me. But would you run >integration tests for adding missing license header? > >> It was just merged indeed, exactly as you propose. But it did cause a major >> outage. And I don’t even use HyperV. >> Details: https://github.com/apache/cloudstack/pull/761 >> >> This is why all PRs to 4.6 and 4.7 (200+) that we merged over the last >> couple of months were tested against a real cloud. No easy fixes and minor >> changes. We always need to run the full integration tests. > >This would be the best case (and good job btw!) but for how long can you >made it without automation? We need automated tests, and if possible >full blown automated integration testing. I agree. > >> When new functionality is proposed, there aren’t many people willing to >> write unit and integration tests to cover it. Until that changes, testing >> can only guard whatever the tests cover. And when we merge new stuff without >> tests, the total coverage goes down making the tests less relevant. In fact, >> when we resolve a bug we should write a tests along with it. I know of one >> guy that does that on a regular basis. > >It is ~2016. No excuse. I know we can not test everything, we are >dealing with hardware. But I would rather say, "merge it, it covers >tests and they passed" then "merge it, it has 2 LGTM". > >> It’s not so simple as it seems unfortunately. > >It has never been simple, and I didn't say so, but it should be our goal >to get there. Right? > >Regards >René