The following discussion is closely related to HBASE-4712. We should reach general consensus so that the execution of future test strategy is smooth.
On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <[email protected]>wrote: > I was considering 'integration tests' as a separate concern from the > large/medium/small _unit_ tests. > > That is, in fact, why the failsafe plugin was added (and is designed for). > > Currently, we have a lot of tests that fall in the realm of integration > tests (testing integration between various pieces, rather than single > functionality alone). The most obvious indicator that something is an > integration test is when it is using the MiniHBaseCluster since that is > really testing the integration between the new feature and the rest of the > system. > > The surefire plugin should really be testing unit tests, and then those > can be classified as small/medium/large. However, the large would be > _extremely_ rare cases. > > If the community is completely moving to just using the annotations for > denoting size of tests, rather than using the integrationTest/unit test > stuff, thats fine. However, I still think that if we do that, we should > have a separate designation for @IntegrationTest (but that is really > already covered in the fact that they would be named differently and > therefore run by the different plugin). The reason we need to split them > out is to have that separate tier of testing - once they pass unit level, > they can be tested at the integration level and then on the acceptance test > level (though we don't have any of those yet). > > This probably means more reclassification of tests. > > Making this explicit is important so people know at what point in the > process their patch breaks (as well as staging the testing being good > practice in general). Therefore, the small/medium/large classifications are > not complete. Further, we should should still be able to use the general > maven testing (mvn test, mvn verify) to run all the tests as expected, > rather than having to know which things extra commands to run. To that end, > we need to integrate > HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in > the book and people know how to work with the current > testing layout. > > I've been a bit behind on the testing discussions on dev@, so didn't get > to put these thoughts in, but I think they can still be added on. This was > also part of what Doug, Stack and I were kind of envisioning in the > original email to the list. > > Thanks. > > -Jesse > > > On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <[email protected]> wrote: > >> Today's tests are: >> - 416 small tests, executed in ~3 minutes >> - 489 medium tests, executed in ~35 minutes (without parallelization) >> - 280 large tests, executed in ~90 minutes (without parallelization) >> >> So may be you could put this new test as large? Will you want to run this >> test for all central builds? Or do you see it as a specific tests that will >> be executed only sometimes? >> >> >> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <[email protected]> wrote: >> >>> So I guess it's likely to be filtered as it's "not large". >>> >>> >>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <[email protected]> wrote: >>> >>>> N: >>>> Currently IntegrationTestConstraint doesn't have @Category annotation. >>>> >>>> FYI >>>> >>>> >>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <[email protected]> wrote: >>>> >>>>> Yes, I put the large test as integration tests (may be 3 weeks ago?). >>>>> So if you mix naming pattern with categories, you can end up with no tests >>>>> to do. >>>>> >>>>> It seems logical to me to put large tests in the failsafe module. If >>>>> you want to use the failsafe tests for something else, I can remove then. >>>>> >>>>> see the doc in >>>>> HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>(last >>>>> comment) and the patch >>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on how >>>>> it designed to work today. >>>>> >>>>> Cheers, >>>>> >>>>> N. >>>>> >>>>> >>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <[email protected]> wrote: >>>>> >>>>>> Copying N. >>>>>> >>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hmmm, that's odd. It was working last week, maybe it was the recent >>>>>>> changes N put in. >>>>>>> >>>>>>> I'll look into it. >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> - Jesse Yates >>>>>>> >>>>>>> Sent from my iPhone. >>>>>>> >>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <[email protected]> wrote: >>>>>>> >>>>>>> > Hi, >>>>>>> > Please categorize new tests for 4605. >>>>>>> > >>>>>>> > When I issued this command: >>>>>>> > mvn failsafe:integration-test >>>>>>> > >>>>>>> > I got: >>>>>>> > >>>>>>> > Results : >>>>>>> > >>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 >>>>>>> > >>>>>>> > Do you have some idea ? >>>>>>> > >>>>>>> > Thanks >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > > > -- > ------------------- > Jesse Yates > 240-888-2200 > @jesse_yates > >
