Sorry, about the message with title: "dev Digest of: thread.25579", my email has been acting up and I forgot to change the subject line when pulling from the archives.
I'm reposting this to the right thread for the archives and in case anyone missed it. -Jesse On Thu, Dec 1, 2011 at 6:34 PM, Jesse Yates <[email protected]> wrote: > I'm generally on the same page here, but this is what I'm (currently) > thinking about how it could work: > > > > We need to characterize the unit tests because some are longer than others, > and we want to split them up in order of complexity. > > Keywal, this next bit is divergent from what I was saying the other day... > > The next level up should be testing between integration of components. Again > this should be run by the CI; really this is more of a finer grain separation > of what was previously just unit tests. > I've revised my thinking such that this could a different category of unit > tests (maybe @Integration that run in their own jvm, in a different step). > This helps separate concerns between the individual class testing and the > testing of multiple pieces working together. > > > Onto using the failsafe plugin. > > I like the idea of really leveraging the failsafe plugin to spin up the tests > on some of 'real' instance. The (canonical) simple example of using failsafe > is to spin up a Jetty server and test whatever stuff you are doing with a > 'real' > > > Also, integrating with BigTop to do some of that stuff would also be pretty > sweet (and good for both communities). > > A good place for these tests would be the hbase-it module discussed in > HBASE-4336, since that module can just depend on the rest of the modules > (giving full access to the components). Within that package, we could > conceivably have the multi-item integration tests (@Integration form above) > and then the 'real cluster' tests as well. > > > +1 Having tests that just use the top level API (e.g. what the FB guys are > using on their dev cluster after running the full test suite) is really > important to make sure we 'real' test cases to ensure functionality. > > > Also, agreeing with Roman, making the plugin to enable that is going to > rough, but we should be able to at least find a way to (at least manually at > first) have those api level tests run on a real cluster, even if it is a > company X and they just post the results. Even then a weekly build/run of > that suite could be sufficient at first (though having that for every commit > would be must better - anyone know someone at Amazon we can scrounge credits > from?). > > > However, I do think that if tests start failing in _any_ of the above phases > then the patch that caused it needs to not be committed OR the tests need to > be revised (which would also be part of the patch, so my original point > stands). That is indicative of broken functionality and trunk should be kept > as stable as possible. > > > Yeah, devs shouldn't have to run the full set of tests before submitting a > patch, but they should easily be able to run everything up to (but not > including) the api-level testing. > What would be cool though would be that they could even run the api-level > testing, but that is just runs on a local MiniCluster, but that's a lot of > work out :). > A lot of that stability comes form having a staged testing cycle as well as > helping devs cut down their testing cycles, so anything we do to help cut > that time down is good stuff in my book.. > > -- Jesse > ------------------- > > > ps. I hate to be 'that guy' but we need a consistent way of describing the > tests we talking about, but lets push that off until figure out what the heck > we actually want to do. > > > Jesse Yates > 240-888-2200 > @jesse_yates > > > >> ---------- Forwarded message ---------- >> From: Ted Yu <[email protected]> >> To: [email protected] >> Date: Mon, 28 Nov 2011 12:23:43 -0800 >> Subject: scoping integration tests >> The following discussion is closely related to HBASE-4712. >> >> We should reach general consensus so that the execution of future test >> strategy is smooth. >> >> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <[email protected] >> >wrote: >> >> > I was considering 'integration tests' as a separate concern from the >> > large/medium/small _unit_ tests. >> > >> > That is, in fact, why the failsafe plugin was added (and is designed >> for). >> > >> > Currently, we have a lot of tests that fall in the realm of integration >> > tests (testing integration between various pieces, rather than single >> > functionality alone). The most obvious indicator that something is an >> > integration test is when it is using the MiniHBaseCluster since that is >> > really testing the integration between the new feature and the rest of >> the >> > system. >> > >> > The surefire plugin should really be testing unit tests, and then those >> > can be classified as small/medium/large. However, the large would be >> > _extremely_ rare cases. >> > >> > If the community is completely moving to just using the annotations for >> > denoting size of tests, rather than using the integrationTest/unit test >> > stuff, thats fine. However, I still think that if we do that, we should >> > have a separate designation for @IntegrationTest (but that is really >> > already covered in the fact that they would be named differently and >> > therefore run by the different plugin). The reason we need to split them >> > out is to have that separate tier of testing - once they pass unit >> level, >> > they can be tested at the integration level and then on the acceptance >> test >> > level (though we don't have any of those yet). >> > >> > This probably means more reclassification of tests. >> > >> > Making this explicit is important so people know at what point in the >> > process their patch breaks (as well as staging the testing being good >> > practice in general). Therefore, the small/medium/large classifications >> are >> > not complete. Further, we should should still be able to use the general >> > maven testing (mvn test, mvn verify) to run all the tests as expected, >> > rather than having to know which things extra commands to run. To that >> end, >> > we need to integrate HBASE-4712< >> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book >> and people know how to work with the current >> > testing layout. >> > >> > I've been a bit behind on the testing discussions on dev@, so didn't >> get >> > to put these thoughts in, but I think they can still be added on. This >> was >> > also part of what Doug, Stack and I were kind of envisioning in the >> > original email to the list. >> > >> > Thanks. >> > >> > -Jesse >> > >> > >> > On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <[email protected]> wrote: >> > >> >> Today's tests are: >> >> - 416 small tests, executed in ~3 minutes >> >> - 489 medium tests, executed in ~35 minutes (without parallelization) >> >> - 280 large tests, executed in ~90 minutes (without parallelization) >> >> >> >> So may be you could put this new test as large? Will you want to run >> this >> >> test for all central builds? Or do you see it as a specific tests that >> will >> >> be executed only sometimes? >> >> >> >> >> >> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <[email protected]> wrote: >> >> >> >>> So I guess it's likely to be filtered as it's "not large". >> >>> >> >>> >> >>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <[email protected]> wrote: >> >>> >> >>>> N: >> >>>> Currently IntegrationTestConstraint doesn't have @Category >> annotation. >> >>>> >> >>>> FYI >> >>>> >> >>>> >> >>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <[email protected]> >> wrote: >> >>>> >> >>>>> Yes, I put the large test as integration tests (may be 3 weeks >> ago?). >> >>>>> So if you mix naming pattern with categories, you can end up with >> no tests >> >>>>> to do. >> >>>>> >> >>>>> It seems logical to me to put large tests in the failsafe module. If >> >>>>> you want to use the failsafe tests for something else, I can remove >> then. >> >>>>> >> >>>>> see the doc in HBASE-4712< >> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the >> patch >> >>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on >> how >> >>>>> it designed to work today. >> >>>>> >> >>>>> Cheers, >> >>>>> >> >>>>> N. >> >>>>> >> >>>>> >> >>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <[email protected]> >> wrote: >> >>>>> >> >>>>>> Copying N. >> >>>>>> >> >>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates < >> >>>>>> [email protected]> wrote: >> >>>>>> >> >>>>>>> Hmmm, that's odd. It was working last week, maybe it was the >> recent >> >>>>>>> changes N put in. >> >>>>>>> >> >>>>>>> I'll look into it. >> >>>>>>> >> >>>>>>> Thanks! >> >>>>>>> >> >>>>>>> - Jesse Yates >> >>>>>>> >> >>>>>>> Sent from my iPhone. >> >>>>>>> >> >>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <[email protected]> wrote: >> >>>>>>> >> >>>>>>> > Hi, >> >>>>>>> > Please categorize new tests for 4605. >> >>>>>>> > >> >>>>>>> > When I issued this command: >> >>>>>>> > mvn failsafe:integration-test >> >>>>>>> > >> >>>>>>> > I got: >> >>>>>>> > >> >>>>>>> > Results : >> >>>>>>> > >> >>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 >> >>>>>>> > >> >>>>>>> > Do you have some idea ? >> >>>>>>> > >> >>>>>>> > Thanks >> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> > >> > >> > -- >> > ------------------- >> > Jesse Yates >> > 240-888-2200 >> > @jesse_yates >> > >> > >> >> >> ---------- Forwarded message ---------- >> From: Ted Yu <[email protected]> >> To: [email protected] >> Date: Tue, 29 Nov 2011 22:08:30 -0800 >> Subject: Re: scoping integration tests >> From Stack: >> ------- >> I think we can have it so everyone gets what they need; >> small/medium/large are done by surefire and then we have a new >> intergration test green field that is run by failsafe that can help us >> get tests for bigtop et al. to run. >> >> ------ >> >> I think the above proposal makes sense. >> >> Please share your comments. >> >> On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <[email protected]> wrote: >> >> > The following discussion is closely related to HBASE-4712. >> > >> > We should reach general consensus so that the execution of future test >> > strategy is smooth. >> > >> > On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <[email protected] >> >wrote: >> > >> >> I was considering 'integration tests' as a separate concern from the >> >> large/medium/small _unit_ tests. >> >> >> >> That is, in fact, why the failsafe plugin was added (and is designed >> >> for). >> >> >> >> Currently, we have a lot of tests that fall in the realm of integration >> >> tests (testing integration between various pieces, rather than single >> >> functionality alone). The most obvious indicator that something is an >> >> integration test is when it is using the MiniHBaseCluster since that is >> >> really testing the integration between the new feature and the rest of >> the >> >> system. >> >> >> >> The surefire plugin should really be testing unit tests, and then those >> >> can be classified as small/medium/large. However, the large would be >> >> _extremely_ rare cases. >> >> >> >> If the community is completely moving to just using the annotations for >> >> denoting size of tests, rather than using the integrationTest/unit test >> >> stuff, thats fine. However, I still think that if we do that, we should >> >> have a separate designation for @IntegrationTest (but that is really >> >> already covered in the fact that they would be named differently and >> >> therefore run by the different plugin). The reason we need to split >> them >> >> out is to have that separate tier of testing - once they pass unit >> level, >> >> they can be tested at the integration level and then on the acceptance >> test >> >> level (though we don't have any of those yet). >> >> >> >> This probably means more reclassification of tests. >> >> >> >> Making this explicit is important so people know at what point in the >> >> process their patch breaks (as well as staging the testing being good >> >> practice in general). Therefore, the small/medium/large >> classifications are >> >> not complete. Further, we should should still be able to use the >> general >> >> maven testing (mvn test, mvn verify) to run all the tests as expected, >> >> rather than having to know which things extra commands to run. To that >> end, >> >> we need to integrate HBASE-4712< >> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book >> and people know how to work with the current >> >> testing layout. >> >> >> >> I've been a bit behind on the testing discussions on dev@, so didn't >> get >> >> to put these thoughts in, but I think they can still be added on. This >> was >> >> also part of what Doug, Stack and I were kind of envisioning in the >> >> original email to the list. >> >> >> >> Thanks. >> >> >> >> -Jesse >> >> >> >> >> >> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <[email protected]> wrote: >> >> >> >>> Today's tests are: >> >>> - 416 small tests, executed in ~3 minutes >> >>> - 489 medium tests, executed in ~35 minutes (without parallelization) >> >>> - 280 large tests, executed in ~90 minutes (without parallelization) >> >>> >> >>> So may be you could put this new test as large? Will you want to run >> >>> this test for all central builds? Or do you see it as a specific >> tests that >> >>> will be executed only sometimes? >> >>> >> >>> >> >>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <[email protected]> wrote: >> >>> >> >>>> So I guess it's likely to be filtered as it's "not large". >> >>>> >> >>>> >> >>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <[email protected]> wrote: >> >>>> >> >>>>> N: >> >>>>> Currently IntegrationTestConstraint doesn't have @Category >> annotation. >> >>>>> >> >>>>> FYI >> >>>>> >> >>>>> >> >>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <[email protected]> >> wrote: >> >>>>> >> >>>>>> Yes, I put the large test as integration tests (may be 3 weeks >> ago?). >> >>>>>> So if you mix naming pattern with categories, you can end up with >> no tests >> >>>>>> to do. >> >>>>>> >> >>>>>> It seems logical to me to put large tests in the failsafe module. >> If >> >>>>>> you want to use the failsafe tests for something else, I can >> remove then. >> >>>>>> >> >>>>>> see the doc in HBASE-4712< >> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the >> patch >> >>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on >> how >> >>>>>> it designed to work today. >> >>>>>> >> >>>>>> Cheers, >> >>>>>> >> >>>>>> N. >> >>>>>> >> >>>>>> >> >>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <[email protected]> >> wrote: >> >>>>>> >> >>>>>>> Copying N. >> >>>>>>> >> >>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates < >> >>>>>>> [email protected]> wrote: >> >>>>>>> >> >>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the >> recent >> >>>>>>>> changes N put in. >> >>>>>>>> >> >>>>>>>> I'll look into it. >> >>>>>>>> >> >>>>>>>> Thanks! >> >>>>>>>> >> >>>>>>>> - Jesse Yates >> >>>>>>>> >> >>>>>>>> Sent from my iPhone. >> >>>>>>>> >> >>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <[email protected]> wrote: >> >>>>>>>> >> >>>>>>>> > Hi, >> >>>>>>>> > Please categorize new tests for 4605. >> >>>>>>>> > >> >>>>>>>> > When I issued this command: >> >>>>>>>> > mvn failsafe:integration-test >> >>>>>>>> > >> >>>>>>>> > I got: >> >>>>>>>> > >> >>>>>>>> > Results : >> >>>>>>>> > >> >>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 >> >>>>>>>> > >> >>>>>>>> > Do you have some idea ? >> >>>>>>>> > >> >>>>>>>> > Thanks >> >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> >> >> >> -- >> >> ------------------- >> >> Jesse Yates >> >> 240-888-2200 >> >> @jesse_yates >> >> >> >> >> > >> >> >> ---------- Forwarded message ---------- >> From: Jesse Yates <[email protected]> >> To: Ted Yu <[email protected]> >> Date: Tue, 29 Nov 2011 22:28:44 -0800 >> Subject: Re: scoping integration tests >> +1 Overall idea. >> >> I don't know if we should consider the integration tests as the primary >> thing that bigtop runs. Since they are deploying to a small cluster >> (right?), the tests that FB is supposed to be open sourcing soon should >> probably be what they run (and lets call those acceptance tests). >> >> While integration tests could be run by bigtop, they should definitely be >> part of our testing of patches as well - no patches should go into trunk >> that don't pass both the unit tests and integration tests. Then we push in >> bigger patches or cut releases we make sure that the acceptance tests >> pass. >> >> At least, that's what I was thinking about how testing works. >> >> -Jesse >> >> On Tue, Nov 29, 2011 at 10:08 PM, Ted Yu <[email protected]> wrote: >> >> > From Stack: >> > ------- >> > I think we can have it so everyone gets what they need; >> > small/medium/large are done by surefire and then we have a new >> > intergration test green field that is run by failsafe that can help us >> > get tests for bigtop et al. to run. >> > >> > ------ >> > >> > I think the above proposal makes sense. >> > >> > Please share your comments. >> > >> > >> > On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <[email protected]> wrote: >> > >> >> The following discussion is closely related to HBASE-4712. >> >> >> >> We should reach general consensus so that the execution of future test >> >> strategy is smooth. >> >> >> >> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <[email protected] >> >wrote: >> >> >> >>> I was considering 'integration tests' as a separate concern from the >> >>> large/medium/small _unit_ tests. >> >>> >> >>> That is, in fact, why the failsafe plugin was added (and is designed >> >>> for). >> >>> >> >>> Currently, we have a lot of tests that fall in the realm of >> integration >> >>> tests (testing integration between various pieces, rather than single >> >>> functionality alone). The most obvious indicator that something is an >> >>> integration test is when it is using the MiniHBaseCluster since that >> is >> >>> really testing the integration between the new feature and the rest >> of the >> >>> system. >> >>> >> >>> The surefire plugin should really be testing unit tests, and then >> those >> >>> can be classified as small/medium/large. However, the large would be >> >>> _extremely_ rare cases. >> >>> >> >>> If the community is completely moving to just using the annotations >> for >> >>> denoting size of tests, rather than using the integrationTest/unit >> test >> >>> stuff, thats fine. However, I still think that if we do that, we >> should >> >>> have a separate designation for @IntegrationTest (but that is really >> >>> already covered in the fact that they would be named differently and >> >>> therefore run by the different plugin). The reason we need to split >> them >> >>> out is to have that separate tier of testing - once they pass unit >> level, >> >>> they can be tested at the integration level and then on the >> acceptance test >> >>> level (though we don't have any of those yet). >> >>> >> >>> This probably means more reclassification of tests. >> >>> >> >>> Making this explicit is important so people know at what point in the >> >>> process their patch breaks (as well as staging the testing being good >> >>> practice in general). Therefore, the small/medium/large >> classifications are >> >>> not complete. Further, we should should still be able to use the >> general >> >>> maven testing (mvn test, mvn verify) to run all the tests as expected, >> >>> rather than having to know which things extra commands to run. To >> that end, >> >>> we need to integrate HBASE-4712< >> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book >> and people know how to work with the current >> >>> testing layout. >> >>> >> >>> I've been a bit behind on the testing discussions on dev@, so didn't >> >>> get to put these thoughts in, but I think they can still be added on. >> This >> >>> was also part of what Doug, Stack and I were kind of envisioning in >> the >> >>> original email to the list. >> >>> >> >>> Thanks. >> >>> >> >>> -Jesse >> >>> >> >>> >> >>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <[email protected]> wrote: >> >>> >> >>>> Today's tests are: >> >>>> - 416 small tests, executed in ~3 minutes >> >>>> - 489 medium tests, executed in ~35 minutes (without >> parallelization) >> >>>> - 280 large tests, executed in ~90 minutes (without parallelization) >> >>>> >> >>>> So may be you could put this new test as large? Will you want to run >> >>>> this test for all central builds? Or do you see it as a specific >> tests that >> >>>> will be executed only sometimes? >> >>>> >> >>>> >> >>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <[email protected]> wrote: >> >>>> >> >>>>> So I guess it's likely to be filtered as it's "not large". >> >>>>> >> >>>>> >> >>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <[email protected]> >> wrote: >> >>>>> >> >>>>>> N: >> >>>>>> Currently IntegrationTestConstraint doesn't have @Category >> annotation. >> >>>>>> >> >>>>>> FYI >> >>>>>> >> >>>>>> >> >>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <[email protected]> >> wrote: >> >>>>>> >> >>>>>>> Yes, I put the large test as integration tests (may be 3 weeks >> >>>>>>> ago?). So if you mix naming pattern with categories, you can end >> up with no >> >>>>>>> tests to do. >> >>>>>>> >> >>>>>>> It seems logical to me to put large tests in the failsafe module. >> If >> >>>>>>> you want to use the failsafe tests for something else, I can >> remove then. >> >>>>>>> >> >>>>>>> see the doc in HBASE-4712< >> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the >> patch >> >>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on >> >>>>>>> how it designed to work today. >> >>>>>>> >> >>>>>>> Cheers, >> >>>>>>> >> >>>>>>> N. >> >>>>>>> >> >>>>>>> >> >>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <[email protected]> >> wrote: >> >>>>>>> >> >>>>>>>> Copying N. >> >>>>>>>> >> >>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates < >> >>>>>>>> [email protected]> wrote: >> >>>>>>>> >> >>>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the >> >>>>>>>>> recent changes N put in. >> >>>>>>>>> >> >>>>>>>>> I'll look into it. >> >>>>>>>>> >> >>>>>>>>> Thanks! >> >>>>>>>>> >> >>>>>>>>> - Jesse Yates >> >>>>>>>>> >> >>>>>>>>> Sent from my iPhone. >> >>>>>>>>> >> >>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <[email protected]> >> wrote: >> >>>>>>>>> >> >>>>>>>>> > Hi, >> >>>>>>>>> > Please categorize new tests for 4605. >> >>>>>>>>> > >> >>>>>>>>> > When I issued this command: >> >>>>>>>>> > mvn failsafe:integration-test >> >>>>>>>>> > >> >>>>>>>>> > I got: >> >>>>>>>>> > >> >>>>>>>>> > Results : >> >>>>>>>>> > >> >>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 >> >>>>>>>>> > >> >>>>>>>>> > Do you have some idea ? >> >>>>>>>>> > >> >>>>>>>>> > Thanks >> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >>> -- >> >>> ------------------- >> >>> Jesse Yates >> >>> 240-888-2200 >> >>> @jesse_yates >> >>> >> >>> >> >> >> > >> >> >> -- >> ------------------- >> Jesse Yates >> 240-888-2200 >> @jesse_yates >> >> >> ---------- Forwarded message ---------- >> From: N Keywal <[email protected]> >> To: Ted Yu <[email protected]> >> Date: Wed, 30 Nov 2011 09:33:01 +0100 >> Subject: Re: scoping integration tests >> ok for me. >> >> I will update & the .pom the doc today >> >> On Wed, Nov 30, 2011 at 7:08 AM, Ted Yu <[email protected]> wrote: >> >> > From Stack: >> > ------- >> > I think we can have it so everyone gets what they need; >> > small/medium/large are done by surefire and then we have a new >> > intergration test green field that is run by failsafe that can help us >> > get tests for bigtop et al. to run. >> > >> > ------ >> > >> > I think the above proposal makes sense. >> > >> > Please share your comments. >> > >> > >> > On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <[email protected]> wrote: >> > >> >> The following discussion is closely related to HBASE-4712. >> >> >> >> We should reach general consensus so that the execution of future test >> >> strategy is smooth. >> >> >> >> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <[email protected] >> >wrote: >> >> >> >>> I was considering 'integration tests' as a separate concern from the >> >>> large/medium/small _unit_ tests. >> >>> >> >>> That is, in fact, why the failsafe plugin was added (and is designed >> >>> for). >> >>> >> >>> Currently, we have a lot of tests that fall in the realm of >> integration >> >>> tests (testing integration between various pieces, rather than single >> >>> functionality alone). The most obvious indicator that something is an >> >>> integration test is when it is using the MiniHBaseCluster since that >> is >> >>> really testing the integration between the new feature and the rest >> of the >> >>> system. >> >>> >> >>> The surefire plugin should really be testing unit tests, and then >> those >> >>> can be classified as small/medium/large. However, the large would be >> >>> _extremely_ rare cases. >> >>> >> >>> If the community is completely moving to just using the annotations >> for >> >>> denoting size of tests, rather than using the integrationTest/unit >> test >> >>> stuff, thats fine. However, I still think that if we do that, we >> should >> >>> have a separate designation for @IntegrationTest (but that is really >> >>> already covered in the fact that they would be named differently and >> >>> therefore run by the different plugin). The reason we need to split >> them >> >>> out is to have that separate tier of testing - once they pass unit >> level, >> >>> they can be tested at the integration level and then on the >> acceptance test >> >>> level (though we don't have any of those yet). >> >>> >> >>> This probably means more reclassification of tests. >> >>> >> >>> Making this explicit is important so people know at what point in the >> >>> process their patch breaks (as well as staging the testing being good >> >>> practice in general). Therefore, the small/medium/large >> classifications are >> >>> not complete. Further, we should should still be able to use the >> general >> >>> maven testing (mvn test, mvn verify) to run all the tests as expected, >> >>> rather than having to know which things extra commands to run. To >> that end, >> >>> we need to integrate HBASE-4712< >> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book >> and people know how to work with the current >> >>> testing layout. >> >>> >> >>> I've been a bit behind on the testing discussions on dev@, so didn't >> >>> get to put these thoughts in, but I think they can still be added on. >> This >> >>> was also part of what Doug, Stack and I were kind of envisioning in >> the >> >>> original email to the list. >> >>> >> >>> Thanks. >> >>> >> >>> -Jesse >> >>> >> >>> >> >>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <[email protected]> wrote: >> >>> >> >>>> Today's tests are: >> >>>> - 416 small tests, executed in ~3 minutes >> >>>> - 489 medium tests, executed in ~35 minutes (without >> parallelization) >> >>>> - 280 large tests, executed in ~90 minutes (without parallelization) >> >>>> >> >>>> So may be you could put this new test as large? Will you want to run >> >>>> this test for all central builds? Or do you see it as a specific >> tests that >> >>>> will be executed only sometimes? >> >>>> >> >>>> >> >>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <[email protected]> wrote: >> >>>> >> >>>>> So I guess it's likely to be filtered as it's "not large". >> >>>>> >> >>>>> >> >>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <[email protected]> >> wrote: >> >>>>> >> >>>>>> N: >> >>>>>> Currently IntegrationTestConstraint doesn't have @Category >> annotation. >> >>>>>> >> >>>>>> FYI >> >>>>>> >> >>>>>> >> >>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <[email protected]> >> wrote: >> >>>>>> >> >>>>>>> Yes, I put the large test as integration tests (may be 3 weeks >> >>>>>>> ago?). So if you mix naming pattern with categories, you can end >> up with no >> >>>>>>> tests to do. >> >>>>>>> >> >>>>>>> It seems logical to me to put large tests in the failsafe module. >> If >> >>>>>>> you want to use the failsafe tests for something else, I can >> remove then. >> >>>>>>> >> >>>>>>> see the doc in HBASE-4712< >> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the >> patch >> >>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on >> >>>>>>> how it designed to work today. >> >>>>>>> >> >>>>>>> Cheers, >> >>>>>>> >> >>>>>>> N. >> >>>>>>> >> >>>>>>> >> >>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <[email protected]> >> wrote: >> >>>>>>> >> >>>>>>>> Copying N. >> >>>>>>>> >> >>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates < >> >>>>>>>> [email protected]> wrote: >> >>>>>>>> >> >>>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the >> >>>>>>>>> recent changes N put in. >> >>>>>>>>> >> >>>>>>>>> I'll look into it. >> >>>>>>>>> >> >>>>>>>>> Thanks! >> >>>>>>>>> >> >>>>>>>>> - Jesse Yates >> >>>>>>>>> >> >>>>>>>>> Sent from my iPhone. >> >>>>>>>>> >> >>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <[email protected]> >> wrote: >> >>>>>>>>> >> >>>>>>>>> > Hi, >> >>>>>>>>> > Please categorize new tests for 4605. >> >>>>>>>>> > >> >>>>>>>>> > When I issued this command: >> >>>>>>>>> > mvn failsafe:integration-test >> >>>>>>>>> > >> >>>>>>>>> > I got: >> >>>>>>>>> > >> >>>>>>>>> > Results : >> >>>>>>>>> > >> >>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 >> >>>>>>>>> > >> >>>>>>>>> > Do you have some idea ? >> >>>>>>>>> > >> >>>>>>>>> > Thanks >> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >>> -- >> >>> ------------------- >> >>> Jesse Yates >> >>> 240-888-2200 >> >>> @jesse_yates >> >>> >> >>> >> >> >> > >> >> >> ---------- Forwarded message ---------- >> From: Stack <[email protected]> >> To: [email protected] >> Date: Wed, 30 Nov 2011 10:11:34 -0800 >> Subject: Re: scoping integration tests >> On Tue, Nov 29, 2011 at 10:28 PM, Jesse Yates <[email protected]> >> wrote: >> > I don't know if we should consider the integration tests as the primary >> > thing that bigtop runs. Since they are deploying to a small cluster >> > (right?), the tests that FB is supposed to be open sourcing soon should >> > probably be what they run (and lets call those acceptance tests). >> > >> > While integration tests could be run by bigtop, they should definitely >> be >> > part of our testing of patches as well - no patches should go into trunk >> > that don't pass both the unit tests and integration tests. Then we push >> in >> > bigger patches or cut releases we make sure that the acceptance tests >> pass. >> > >> >> Chatting w/ Roman and Mikhail yesterday at the Hackathon, and with a >> probably broader goals than perhaps what you've been thinking on >> Jesse, I think: >> >> + IntegrationTests* -- where IntegrationTest* is tests with the >> IntegrationTest prefix and are that run when 'mvn verify' as per your >> recent contrib Jesse -- differ from unit tests in that they always run >> against a cluster >> + IntegrationTests* do not expect to be able to put their fingers into >> regionservers and/or master to figure test/pass or fail; they treat >> the cluster as 'remote' >> + IntegrationTest* do not concern themselves with cluster setup and >> teardown -- this is done external to the test code by the failsafe >> plugin + configs (If we do this and the previous point, then our >> IntegrationTest* can be run by bigtop if I understood Roman properly >> yesterday). >> + Cluster setup and teardown needs to be pluggable (it seems like >> failsafe can do this for us). We need this because default cluster >> will be minicluster but then it needs to be easy for us to plug in a >> whirr cluster and/or for the likes of our fb homies to plugin their >> cluster setup, etc. >> + IntegerationTest* are sizeable. You can pass things like do 1M rows >> or 100B rows -- and configurable ("Use 'that' cluster") >> >> I don't think a patch needs to pass all integration tests before it >> gets committed. If a bit of functionality needs an integration-type >> test, then IMO it shoudl be cast as a 'large' unit test run by >> surefire. As per N, devs do 'mvn test' which will run small and >> medium unit tests and then our CI does small/medium/large. >> >> I don't want to get hung up on nomenclature. If there are long email >> threads on test taxonomy, then something has gone awry IMO. A clean >> surefire for unit tests and failsafe for integration tests where unit >> tests are whats runnable w/o need of outside dependencies -- e.g. an >> idling hbase or mr cluster -- and is what devs and our CI does seems >> like a pretty clean separation to me, one that even us hbase devs >> might grok. >> >> What do you reckon? >> >> St.Ack >> >> >> >> ---------- Forwarded message ---------- >> From: Roman Shaposhnik <[email protected]> >> To: [email protected] >> Date: Wed, 30 Nov 2011 11:16:21 -0800 >> Subject: Re: scoping integration tests >> +bigtop-dev >> >> On Wed, Nov 30, 2011 at 10:11 AM, Stack <[email protected]> wrote: >> > Chatting w/ Roman and Mikhail yesterday at the Hackathon, and with a >> > probably broader goals than perhaps what you've been thinking on >> > Jesse, I think: >> > >> > + IntegrationTests* -- where IntegrationTest* is tests with the >> > IntegrationTest prefix and are that run when 'mvn verify' as per your >> > recent contrib Jesse -- differ from unit tests in that they always run >> > against a cluster >> >> Small correction: I don't "always run against a cluster" is a useful >> assumption here. In fact, I really want those tests to be completly >> agnostic of whether they run against a Mini, Pseudo-distributed >> or a fully distributed cluster. >> >> > + IntegrationTest* do not concern themselves with cluster setup and >> > teardown -- this is done external to the test code by the failsafe >> > plugin + configs (If we do this and the previous point, then our >> > IntegrationTest* can be run by bigtop if I understood Roman properly >> > yesterday). >> >> Correct. An excellent example of how this can be done is pigsmoke >> artifact with its -Dpigunit.exectype.cluster=true knob (and a few >> others): >> https://issues.apache.org/jira/browse/PIG-1799 >> >> > + Cluster setup and teardown needs to be pluggable (it seems like >> > failsafe can do this for us). >> >> Only if somebody puts an effort of actually implementing a >> deployment plugin (perhaps based on Whirr?). This is a non-trivial >> exercise (although it is a nice project). >> >> > I don't want to get hung up on nomenclature. If there are long email >> > threads on test taxonomy, then something has gone awry IMO. A clean >> > surefire for unit tests and failsafe for integration tests where unit >> > tests are whats runnable w/o need of outside dependencies -- e.g. an >> > idling hbase or mr cluster -- and is what devs and our CI does seems >> > like a pretty clean separation to me, one that even us hbase devs >> > might grok. >> > >> > What do you reckon? >> >> +1 >> >> An extra point that I was trying to articulate very strongly to the GB >> guys >> yesterday is this: we need tests NOW even when we don't have the >> rest of the infrastructure in place (mvn verify, deployment, separation >> of tests, etc.) If they can start giving me access to the tests they've >> created I can start running them in Bigtop today. >> >> Sure they will be "one step removed" from a casual HBase developer >> but the bugs that I'm going to file will be quite real. >> >> Thanks, >> Roman. >> >> >> > > > -- > ------------------- > Jesse Yates > 240-888-2200 > @jesse_yates > >
