+1 Overall idea. I don't know if we should consider the integration tests as the primary thing that bigtop runs. Since they are deploying to a small cluster (right?), the tests that FB is supposed to be open sourcing soon should probably be what they run (and lets call those acceptance tests).
While integration tests could be run by bigtop, they should definitely be part of our testing of patches as well - no patches should go into trunk that don't pass both the unit tests and integration tests. Then we push in bigger patches or cut releases we make sure that the acceptance tests pass. At least, that's what I was thinking about how testing works. -Jesse On Tue, Nov 29, 2011 at 10:08 PM, Ted Yu <[email protected]> wrote: > From Stack: > ------- > I think we can have it so everyone gets what they need; > small/medium/large are done by surefire and then we have a new > intergration test green field that is run by failsafe that can help us > get tests for bigtop et al. to run. > > ------ > > I think the above proposal makes sense. > > Please share your comments. > > > On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <[email protected]> wrote: > >> The following discussion is closely related to HBASE-4712. >> >> We should reach general consensus so that the execution of future test >> strategy is smooth. >> >> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <[email protected]>wrote: >> >>> I was considering 'integration tests' as a separate concern from the >>> large/medium/small _unit_ tests. >>> >>> That is, in fact, why the failsafe plugin was added (and is designed >>> for). >>> >>> Currently, we have a lot of tests that fall in the realm of integration >>> tests (testing integration between various pieces, rather than single >>> functionality alone). The most obvious indicator that something is an >>> integration test is when it is using the MiniHBaseCluster since that is >>> really testing the integration between the new feature and the rest of the >>> system. >>> >>> The surefire plugin should really be testing unit tests, and then those >>> can be classified as small/medium/large. However, the large would be >>> _extremely_ rare cases. >>> >>> If the community is completely moving to just using the annotations for >>> denoting size of tests, rather than using the integrationTest/unit test >>> stuff, thats fine. However, I still think that if we do that, we should >>> have a separate designation for @IntegrationTest (but that is really >>> already covered in the fact that they would be named differently and >>> therefore run by the different plugin). The reason we need to split them >>> out is to have that separate tier of testing - once they pass unit level, >>> they can be tested at the integration level and then on the acceptance test >>> level (though we don't have any of those yet). >>> >>> This probably means more reclassification of tests. >>> >>> Making this explicit is important so people know at what point in the >>> process their patch breaks (as well as staging the testing being good >>> practice in general). Therefore, the small/medium/large classifications are >>> not complete. Further, we should should still be able to use the general >>> maven testing (mvn test, mvn verify) to run all the tests as expected, >>> rather than having to know which things extra commands to run. To that end, >>> we need to integrate >>> HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in >>> the book and people know how to work with the current >>> testing layout. >>> >>> I've been a bit behind on the testing discussions on dev@, so didn't >>> get to put these thoughts in, but I think they can still be added on. This >>> was also part of what Doug, Stack and I were kind of envisioning in the >>> original email to the list. >>> >>> Thanks. >>> >>> -Jesse >>> >>> >>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <[email protected]> wrote: >>> >>>> Today's tests are: >>>> - 416 small tests, executed in ~3 minutes >>>> - 489 medium tests, executed in ~35 minutes (without parallelization) >>>> - 280 large tests, executed in ~90 minutes (without parallelization) >>>> >>>> So may be you could put this new test as large? Will you want to run >>>> this test for all central builds? Or do you see it as a specific tests that >>>> will be executed only sometimes? >>>> >>>> >>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <[email protected]> wrote: >>>> >>>>> So I guess it's likely to be filtered as it's "not large". >>>>> >>>>> >>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <[email protected]> wrote: >>>>> >>>>>> N: >>>>>> Currently IntegrationTestConstraint doesn't have @Category annotation. >>>>>> >>>>>> FYI >>>>>> >>>>>> >>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <[email protected]> wrote: >>>>>> >>>>>>> Yes, I put the large test as integration tests (may be 3 weeks >>>>>>> ago?). So if you mix naming pattern with categories, you can end up >>>>>>> with no >>>>>>> tests to do. >>>>>>> >>>>>>> It seems logical to me to put large tests in the failsafe module. If >>>>>>> you want to use the failsafe tests for something else, I can remove >>>>>>> then. >>>>>>> >>>>>>> see the doc in >>>>>>> HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>(last >>>>>>> comment) and the patch >>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on >>>>>>> how it designed to work today. >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> N. >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <[email protected]> wrote: >>>>>>> >>>>>>>> Copying N. >>>>>>>> >>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the >>>>>>>>> recent changes N put in. >>>>>>>>> >>>>>>>>> I'll look into it. >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> - Jesse Yates >>>>>>>>> >>>>>>>>> Sent from my iPhone. >>>>>>>>> >>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <[email protected]> wrote: >>>>>>>>> >>>>>>>>> > Hi, >>>>>>>>> > Please categorize new tests for 4605. >>>>>>>>> > >>>>>>>>> > When I issued this command: >>>>>>>>> > mvn failsafe:integration-test >>>>>>>>> > >>>>>>>>> > I got: >>>>>>>>> > >>>>>>>>> > Results : >>>>>>>>> > >>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 >>>>>>>>> > >>>>>>>>> > Do you have some idea ? >>>>>>>>> > >>>>>>>>> > Thanks >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> ------------------- >>> Jesse Yates >>> 240-888-2200 >>> @jesse_yates >>> >>> >> > -- ------------------- Jesse Yates 240-888-2200 @jesse_yates
