On Thu, Dec 1, 2011 at 6:34 PM, Jesse Yates <[email protected]> wrote: > We need to characterize the unit tests because some are longer than > others, and we want to split them up in order of complexity. >
Nkeywal has done this, no? He's made them into small/medium/large. Some tests may need to be moved between categories but the hard graft has been done here? You introduced the IntegrationTest category that is run with failsafe. Some of the current large unit tests might get moved over to this new IntegrationTest category. > Keywal, this next bit is divergent from what I was saying the other day... > > The next level up should be testing between integration of components. I'm not sure I grok this. Next level up from unit tests? But unit tests, for both medium and large, are allowed spin up mini clusters so they are already testing integration of components? If you are suggesting that these kind of minicluster tests get moved out to IntegrationTest category, then I'd disagree; or, I disagree that should happen any time soon (too much work, too disruptive). > Again this should be run by the CI; If you are saying the IntegrationTests should be run the CI, I'm thinking not. I'm thinking they'd be run on occasion when trying to proof a build or release candidate or trying to verify an hotfix before production deploy > I've revised my thinking such that this could a different category of > unit tests (maybe @Integration that run in their own jvm, in a > different step). > This helps separate concerns between the individual class testing and > the testing of multiple pieces working together. > I don't think this viable. What I do think viable is committing the last nkeywal suggestion of 'mvn test' running current small and medium tests but then we have the CI run small/medium/large. Then, IntegrationTests are a new category altogether of tests that go against the API only and that presume a cluster (built by failsafe or otherwise); the tests do not do cluster setup/teardown. > A good place for these tests would be the hbase-it module discussed in > HBASE-4336, since that module can just depend on the rest of the > modules (giving full access to the components). Within that package, > we could conceivably have the multi-item integration tests > (@Integration form above) and then the 'real cluster' tests as well. > I'd be down w/ this. Currently IntegrationTest* are mixed in w/ unit tests but it would be best if they were off in a module of their own. > +1 Having tests that just use the top level API (e.g. what the FB guys > are using on their dev cluster after running the full test suite) is > really important to make sure we 'real' test cases to ensure > functionality. > > Also, agreeing with Roman, making the plugin to enable that is going > to rough, but we should be able to at least find a way to (at least > manually at first) have those api level tests run on a real cluster, > even if it is a company X and they just post the results. Even then a > weekly build/run of that suite could be sufficient at first (though > having that for every commit would be must better - anyone know > someone at Amazon we can scrounge credits from?). > +1 on above too. > However, I do think that if tests start failing in _any_ of the above > phases then the patch that caused it needs to not be committed OR the > tests need to be revised (which would also be part of the patch, so my > original point stands). That is indicative of broken functionality and > trunk should be kept as stable as possible. > Depends. Patch may have been committed before an IntegationTest is run. I think we should allow this. For sure, if IntegrationTests are broke, we need to figure which patch broke things. > Yeah, devs shouldn't have to run the full set of tests before > submitting a patch, but they should easily be able to run everything > up to (but not including) the api-level testing. Yes. Nkeywal has this going on in his patch. I'd say too that the failsafe plugin should out of the box let devs run all IntegrationTest. failsafe default might be to spin up a mini hbase cluster. > What would be cool though would be that they could even run the > api-level testing, but that is just runs on a local MiniCluster, but > that's a lot of work out :). I don't think so. In fact I think it a prereq for easy IntegrationTest dev'ing. > > ps. I hate to be 'that guy' but we need a consistent way of describing > the tests we talking about, but lets push that off until figure out > what the heck we actually want to do. > The way I see it, I think we have a proposal on what to do. Nkeywal has done a refactoring of our unit tests into categories but we need a new category of tests, tests that are optionally long-running against clusters of variable size doing scales beyond what would be tenable on Apache's or anyone's CI. These are the tests we'd give our bigtop brothers and that we'd run when we wanted to do a verification beyond what our unit tests can do. Good on you Jesse, St.Ack
