I think getting reasonable coverage from the pure unit tests is good goal. I know Mark has done some work in the past to get code coverage reports for these different categories working with jenkins, maybe we can get that integrated with the geode CI. I think a combination of these unit tests and module specific categories could help some get the fast feedback as Ashvin suggested.
I also agree with Ashvin that the full test suite should be required for commits. One of the problems with commit-then-test is that it becomes hard to track down who broke the tests if there are a lot of changes going in. -Dan On Mon, Jun 15, 2015 at 5:05 PM, Ashvin A <[email protected]> wrote: > Hi, > > We need a fast verification method for testing any changes, I agree. I > think test categories/groups is a step in the correct direction. Then as a > developer, I can quickly verify my changes at the module level. As Kirk > said, we will invest in writing new tests and also identifying and fixing > tests taking very long. > > However I think it is important to execute all tests before committing any > changes. New developers will prefer shortcuts, but it is very difficult to > weed out bugs later. I would not encourage reduced test set as a commit > criteria. I fear that tests will be ignored once a reduced set is created. > > $.01 > > Thanks, > Ashvin > > On Mon, Jun 15, 2015 at 4:39 PM, Kirk Lund <[email protected]> wrote: > > > Hi Jun, > > > > Here's some info on Geode tests. > > > > 1) junit > > > > These are unit tests involving mocks as well as end-to-end functional > > tests. These are currently marked with @Category(UnitTest.class) or > > @Category(IntegrationTest.class). > > > > We need to encourage Geode developers to write a lot of more of the > > UnitTest category tests. There's really not enough of these at this point > > in time. There needs to be a strong commitment to writing these unit > tests > > and refactoring older code to make it easier or even possible for certain > > classes. > > > > 2) dunit > > > > The majority of developer tests are "dunit" tests which extend > > DistributedTestCase. These create and manipulate 6 JVMs. Think of these > as > > end-to-end functional tests for a cluster. Unfortunately it take hours to > > execute them all and there are some reliability issues in these tests. > I'm > > working on replacing DistributedTestCase with a custom junit runner so > that > > the syntax of these tests can be updated to using JUnit 4 annotations and > > rules. > > > > 3) hydra > > > > These are bigger QA-developed tests that run longer and may use many more > > JVMs than a dunit. Moving hydra and these tests into Geode is further > down > > the road after moving the rest of the dunit tests. > > > > > > On Mon, Jun 15, 2015 at 4:00 PM, jun aoki <[email protected]> wrote: > > > > > Hi Dan, > > > > > > Unit testing is super important for open source projects, so I'm glad > > some > > > high quality testing is coming in to the project! > > > > > > Along with Roman's comment, it will be nice those tests ; > > > 1. are unit level test where mocks are appropriately used and no > running > > > instances are required. (IMO, unit test should be lightweight, and more > > > heavy integration tests should be done separately) > > > 2. completes up to 30mins so that contributors can get a quicker > > feedback. > > > (we are hoping to get hundreds of contributors so be aware of this! :) > ) > > > > > > As Roman suggested, should the unit test of GEODE-6 have basic > acceptance > > > tests (can be subset of the several-hour test set) and the open source > CI > > > should only tests the subset coverage? > > > > > > > > > On Mon, Jun 15, 2015 at 10:07 AM, Roman Shaposhnik < > [email protected] > > > > > > wrote: > > > > > > > On Mon, Jun 15, 2015 at 9:42 AM, Dan Smith <[email protected]> > wrote: > > > > > Welcome Jun! > > > > > > > > > > I like all of your suggestions! One thing to be aware of regarding > > the > > > CI > > > > > jobs is that the test time is going to go up a lot once GEODE-6 is > > > > > resolved. We'll be adding several hours of tests. That will make > > > having a > > > > > test patch CI job all that more valuable, but it also affects how > > much > > > > > resources need to be devoted to that. > > > > > > > > Does it just mean that we'd have to have a 'smoke test' testsuite? > > > > > > > > Thanks, > > > > Roman. > > > > > > > > > > > > > > > > -- > > > -jun > > > > > >
