Re: Convention on dropping tests under the test framework
Eh... rebuild time for apps with no changes are negligible... it takes longer for selenium to spin up firefox. I would not even worry about the app rebuild time. --jason On Dec 12, 2006, at 7:23 AM, Prasad Kashyap wrote: On 12/11/06, David Jencks <[EMAIL PROTECTED]> wrote: I thought about this a little more and have some ideas about what I'd like. I dunno if this is remotely possible. - keep the test app and test code close together in svn. It's going to be a lot easier to work with a single test if these are close together, rather than like the testsupport stuff that is far from the tests that use it. - make it easy to run the tests on a single app... so there's an easy way to build the test app, start up a server, and run the tests for that app. The downside to this is that you are building the app everytime you run the test. This is doubly wasteful since the tests are run once for each of the container-assemblies. Granted, this app builds very fast but still there is a delta time that it adds to the test process. Not every app may build this fast. IMO, The most ideal scenario is what we do for the enterprise-testsuite/test-ejbcontainer. There, both the ejb beans and the tests are compiled elsewhere and published. They are simply invoked in the framework. Cheers Prasad
Re: Convention on dropping tests under the test framework
On 12/11/06, David Jencks <[EMAIL PROTECTED]> wrote: I thought about this a little more and have some ideas about what I'd like. I dunno if this is remotely possible. - keep the test app and test code close together in svn. It's going to be a lot easier to work with a single test if these are close together, rather than like the testsupport stuff that is far from the tests that use it. - make it easy to run the tests on a single app... so there's an easy way to build the test app, start up a server, and run the tests for that app. The downside to this is that you are building the app everytime you run the test. This is doubly wasteful since the tests are run once for each of the container-assemblies. Granted, this app builds very fast but still there is a delta time that it adds to the test process. Not every app may build this fast. IMO, The most ideal scenario is what we do for the enterprise-testsuite/test-ejbcontainer. There, both the ejb beans and the tests are compiled elsewhere and published. They are simply invoked in the framework. Cheers Prasad
Re: Convention on dropping tests under the test framework
On 12/10/06, David Jencks <[EMAIL PROTECTED]> wrote: On Dec 8, 2006, at 1:29 PM, Prasad Kashyap wrote: > David, > > Check out this patch http://people.apache.org/~prasad/manifestcp.patch > > Apply it from the geronimo/testsuite/depoyment-testsuite directory. > > It will create 2 directories under it. > -- The manifestcp will create the ear for. > -- The test-manifestcp will deploy and undeploy it. > > Throw your testcases under test-manifestcp. When I apply the patch I get files with many copies of the content. Assuming this builds for you can you commit it? I have committed this code. Please read the comments in deployment-testsuite/manifestcp/pom.xml Cheers Prasad
Re: Convention on dropping tests under the test framework
Starting and stopping the server for each app ? Wouldn't that be a tad too much. Currently the start/stop is designed to happen for a suite (eg. web). So a bunch of related tests (servlet tests, jsp tests, jsf tests etc) can be performed in a suite with required apps getting deployed and undeployed for the tests. Cheers Prasad On 12/11/06, David Jencks <[EMAIL PROTECTED]> wrote: I thought about this a little more and have some ideas about what I'd like. I dunno if this is remotely possible. - keep the test app and test code close together in svn. It's going to be a lot easier to work with a single test if these are close together, rather than like the testsupport stuff that is far from the tests that use it. - make it easy to run the tests on a single app... so there's an easy way to build the test app, start up a server, and run the tests for that app. - make it relatively quick to run all the tests which I think means starting a server, then for each app, deploying, running tests, and undeploying. My current understanding of what we have now is that the server is going to be started and stopped for each app if we've set things up for my second requirement :-) thanks david jencks On Dec 8, 2006, at 1:21 PM, Prasad Kashyap wrote: > I know what you mean. I just worked on the app that David was talking > about those do need a separate module by themselves. I didn't realise > his scenario until I saw the JIRA. > > I was talking about something like > testsuite/web-testsuite/test-servlet25. This example builds a war and > tests it in the same pom. > > Cheers > Prasad > > On 12/8/06, Jason Dillon <[EMAIL PROTECTED]> wrote: >> On Dec 8, 2006, at 6:30 AM, Prasad Kashyap wrote: >> > 1) just make a copy of test-deployment, (say call it cxf- >> deployment) >> > 2) use it's child profile to go thro the complete maven lifecycle - >> > compile, build and test your apps. >> > 3) it's parent (deployment-testsuite) will take care of the server >> > start/stop and reporting for you. >> >> This won't actually work as easily as you may have imagined Prasad. >> To effectively build some apps, you actually need to have them live >> in their own module, so that they can be built using the associated >> maven plugins. >> >> I do not believe that the current testsuite setup takes this into >> account... and this is why I think it is premature to simply roll out >> the same config/layout with out having some more real tests, which >> build these test applications and allow for effective organization. >> >> I don't think we are quite there yet... close, but still a bit off. >> >> --jason >> >>
Re: Convention on dropping tests under the test framework
On 12/10/06, David Jencks <[EMAIL PROTECTED]> wrote: On Dec 8, 2006, at 1:29 PM, Prasad Kashyap wrote: > David, > > Check out this patch http://people.apache.org/~prasad/manifestcp.patch > > Apply it from the geronimo/testsuite/depoyment-testsuite directory. > > It will create 2 directories under it. > -- The manifestcp will create the ear for. > -- The test-manifestcp will deploy and undeploy it. > > Throw your testcases under test-manifestcp. When I apply the patch I get files with many copies of the content. Assuming this builds for you can you commit it? IIRC there there aren't any tests needed for this, if it deploys it works. When it's committed I'll check. Sure. let me work on commiting it for you. I think I'd prefer everything including the tests to be under the manifestcp dir, but we can try to rearrange this later. I think we need to study how to best set this up. Usually, that shouldn't have been a problem but for the fact that it might be left out of the entire reporting structure of the test framework. Let me see how I can pull in the surefire-reports from the deploy/undeploy of the manifestcp.ear if it gets embedded inside this "pom" thanks david jencks Cheers Prasad > > Cheers > Prasad > > On 12/8/06, David Jencks <[EMAIL PROTECTED]> wrote: >> >> On Dec 8, 2006, at 6:30 AM, Prasad Kashyap wrote: >> >> > On 12/8/06, David Jencks <[EMAIL PROTECTED]> wrote: >> > >> >> For instance it might be useful to have a maven archetype to >> set up >> >> everything except the app to test and the actual test cases. >> >> >> >> thanks >> >> david jencks >> >> >> > >> > I have alrady written an archetype plugin called >> > testsuite-archetype-plugin that will let you create a testsuite >> with >> > an empty testset project under it. Please see >> > http://cwiki.apache.org/GMOxDEV/integration- >> > testing.html#IntegrationTesting-Gettingstarted >> > >> > However, in your case, since you don't want us to be createing >> any moe >> > testsuites till we have colected enough tests, this is what you >> should >> > do : >> > >> > 1) just make a copy of test-deployment, (say call it cxf- >> deployment) >> > 2) use it's child profile to go thro the complete maven lifecycle - >> > compile, build and test your apps. >> > 3) it's parent (deployment-testsuite) will take care of the server >> > start/stop and reporting for you. >> >> I wasn't able to make anything work where the app being tested was >> under the directory that's a copy of *-deployment. Now I'm working >> with a structure where the *-deployment is a child of the app build >> directory, and it seems to work. Once I straighten out the tests so >> they work I'll check it in and we can see if it can be reorganized to >> be simpler. >> >> I was wondering why the SeleniumTestSupport and ExtendedSelenium >> classes were in the tests themselves rather than in a support jar. >> I'd think these would be used by just about all tests. >> >> FIxing the tests is likely to take me all day. If you'd like to >> demonstrate what you think is correct structure you could set >> something up with the test ear from GERONIMO-2125 >> >> http://issues.apache.org/jira/secure/attachment/12336683/manifestcp- >> itest-v3.jar >> >> which has a maven project to build a test ear. With luck deploying >> it stlll works :-) >> >> thanks >> david jencks >> >> > >> > Cheers >> > Prasad >> > >> > Cheers >> > Prasad >> > >> > >> > >> > >> >> On Dec 7, 2006, at 8:31 PM, Prasad Kashyap wrote: >> >> >> >> > Since there were some questions today on where to drop new >> >> tests, I'll >> >> > take a stab at creating a convention. Feel free to offer your >> >> > suggestions so that we can modify it as we go along. >> >> > >> >> > We began by having 2 testsuites just as an example. >> >> > geronimo/testsuite/ >> >> > console-testsuite >> >> > deployment-testsuite >> >> > >> >> > But almost everything can fall under the category of the >> >> > deployment-testsuite since most tests do need some deployable >> >> > artifact. So I think we should use the deployment-testsuite >> >> purely for >> >> > testing the deploy tool. Especially, it should be used to test >> >> > features like hot-deploy, redeploy and undeploy which have had >> >> JIRAs >> >> > before. >> >> > >> >> > We should categorize the tests so that they reflect the broad >> >> > functional areas of the server. >> >> > >> >> > * web-testsuite (servlets, jsp, jstl) >> >> > * enterprise-testsuite (ejb, jpa, jms, jacc, jta, javamail, jaf >> >> etc) >> >> > * mgmt-testsuite (jee management, deployment) >> >> > * webservices-testsuite (wsee, jaxb, stax, saaj, ws-metadata >> etc) >> >> > * performance-testsuite (server startup time, server >> footprint etc) >> >> > * security-testsuite >> >> > * console-testsuite >> >> > * regression (compatibility) >> >> > >> >> > If nobody has any objection to this top categorization, I >> shall go >> >> > ahead and create these testsuites over the weekend. Meanwhile >> >> you may
Re: Convention on dropping tests under the test framework
I thought about this a little more and have some ideas about what I'd like. I dunno if this is remotely possible. - keep the test app and test code close together in svn. It's going to be a lot easier to work with a single test if these are close together, rather than like the testsupport stuff that is far from the tests that use it. - make it easy to run the tests on a single app... so there's an easy way to build the test app, start up a server, and run the tests for that app. - make it relatively quick to run all the tests which I think means starting a server, then for each app, deploying, running tests, and undeploying. My current understanding of what we have now is that the server is going to be started and stopped for each app if we've set things up for my second requirement :-) thanks david jencks On Dec 8, 2006, at 1:21 PM, Prasad Kashyap wrote: I know what you mean. I just worked on the app that David was talking about those do need a separate module by themselves. I didn't realise his scenario until I saw the JIRA. I was talking about something like testsuite/web-testsuite/test-servlet25. This example builds a war and tests it in the same pom. Cheers Prasad On 12/8/06, Jason Dillon <[EMAIL PROTECTED]> wrote: On Dec 8, 2006, at 6:30 AM, Prasad Kashyap wrote: > 1) just make a copy of test-deployment, (say call it cxf- deployment) > 2) use it's child profile to go thro the complete maven lifecycle - > compile, build and test your apps. > 3) it's parent (deployment-testsuite) will take care of the server > start/stop and reporting for you. This won't actually work as easily as you may have imagined Prasad. To effectively build some apps, you actually need to have them live in their own module, so that they can be built using the associated maven plugins. I do not believe that the current testsuite setup takes this into account... and this is why I think it is premature to simply roll out the same config/layout with out having some more real tests, which build these test applications and allow for effective organization. I don't think we are quite there yet... close, but still a bit off. --jason
Re: Convention on dropping tests under the test framework
David, Check out this patch http://people.apache.org/~prasad/manifestcp.patch Apply it from the geronimo/testsuite/depoyment-testsuite directory. It will create 2 directories under it. -- The manifestcp will create the ear for. -- The test-manifestcp will deploy and undeploy it. Throw your testcases under test-manifestcp. Cheers Prasad On 12/8/06, David Jencks <[EMAIL PROTECTED]> wrote: On Dec 8, 2006, at 6:30 AM, Prasad Kashyap wrote: > On 12/8/06, David Jencks <[EMAIL PROTECTED]> wrote: > >> For instance it might be useful to have a maven archetype to set up >> everything except the app to test and the actual test cases. >> >> thanks >> david jencks >> > > I have alrady written an archetype plugin called > testsuite-archetype-plugin that will let you create a testsuite with > an empty testset project under it. Please see > http://cwiki.apache.org/GMOxDEV/integration- > testing.html#IntegrationTesting-Gettingstarted > > However, in your case, since you don't want us to be createing any moe > testsuites till we have colected enough tests, this is what you should > do : > > 1) just make a copy of test-deployment, (say call it cxf-deployment) > 2) use it's child profile to go thro the complete maven lifecycle - > compile, build and test your apps. > 3) it's parent (deployment-testsuite) will take care of the server > start/stop and reporting for you. I wasn't able to make anything work where the app being tested was under the directory that's a copy of *-deployment. Now I'm working with a structure where the *-deployment is a child of the app build directory, and it seems to work. Once I straighten out the tests so they work I'll check it in and we can see if it can be reorganized to be simpler. I was wondering why the SeleniumTestSupport and ExtendedSelenium classes were in the tests themselves rather than in a support jar. I'd think these would be used by just about all tests. FIxing the tests is likely to take me all day. If you'd like to demonstrate what you think is correct structure you could set something up with the test ear from GERONIMO-2125 http://issues.apache.org/jira/secure/attachment/12336683/manifestcp- itest-v3.jar which has a maven project to build a test ear. With luck deploying it stlll works :-) thanks david jencks > > Cheers > Prasad > > Cheers > Prasad > > > > >> On Dec 7, 2006, at 8:31 PM, Prasad Kashyap wrote: >> >> > Since there were some questions today on where to drop new >> tests, I'll >> > take a stab at creating a convention. Feel free to offer your >> > suggestions so that we can modify it as we go along. >> > >> > We began by having 2 testsuites just as an example. >> > geronimo/testsuite/ >> > console-testsuite >> > deployment-testsuite >> > >> > But almost everything can fall under the category of the >> > deployment-testsuite since most tests do need some deployable >> > artifact. So I think we should use the deployment-testsuite >> purely for >> > testing the deploy tool. Especially, it should be used to test >> > features like hot-deploy, redeploy and undeploy which have had >> JIRAs >> > before. >> > >> > We should categorize the tests so that they reflect the broad >> > functional areas of the server. >> > >> > * web-testsuite (servlets, jsp, jstl) >> > * enterprise-testsuite (ejb, jpa, jms, jacc, jta, javamail, jaf >> etc) >> > * mgmt-testsuite (jee management, deployment) >> > * webservices-testsuite (wsee, jaxb, stax, saaj, ws-metadata etc) >> > * performance-testsuite (server startup time, server footprint etc) >> > * security-testsuite >> > * console-testsuite >> > * regression (compatibility) >> > >> > If nobody has any objection to this top categorization, I shall go >> > ahead and create these testsuites over the weekend. Meanwhile >> you may >> > drop your tests in the existing testsuites for now. I shall move >> them >> > appropriately. >> > >> > Lastly, how do we deal with super apps like daytrader that can span >> > across multiple testsuites ? >> > >> > Cheers >> > Prasad >> >>
Re: Convention on dropping tests under the test framework
I know what you mean. I just worked on the app that David was talking about those do need a separate module by themselves. I didn't realise his scenario until I saw the JIRA. I was talking about something like testsuite/web-testsuite/test-servlet25. This example builds a war and tests it in the same pom. Cheers Prasad On 12/8/06, Jason Dillon <[EMAIL PROTECTED]> wrote: On Dec 8, 2006, at 6:30 AM, Prasad Kashyap wrote: > 1) just make a copy of test-deployment, (say call it cxf-deployment) > 2) use it's child profile to go thro the complete maven lifecycle - > compile, build and test your apps. > 3) it's parent (deployment-testsuite) will take care of the server > start/stop and reporting for you. This won't actually work as easily as you may have imagined Prasad. To effectively build some apps, you actually need to have them live in their own module, so that they can be built using the associated maven plugins. I do not believe that the current testsuite setup takes this into account... and this is why I think it is premature to simply roll out the same config/layout with out having some more real tests, which build these test applications and allow for effective organization. I don't think we are quite there yet... close, but still a bit off. --jason
Re: Convention on dropping tests under the test framework
On 12/8/06, Jason Dillon <[EMAIL PROTECTED]> wrote: On Dec 8, 2006, at 6:33 AM, Prasad Kashyap wrote: >> What is a super app anyways? :-P > > Maybe I should have called it a uber app. Unlike other small apps that > we create and use to test a particular feature, DT can serve as the > test app to test features across many categories. Small apps to test a particular feature... a test-app? Don't need to go calling other normal applications something else :-P So sue me :-) I still don't understand why you even brought daytrader up in this context though... still would like to know that. Apart from the small itty bitty apps that we write as test-apps, I think it would be a good idea to test the features on a real world app, a super-duper-uber-app :-) like DayTrader. --jason Cheers Prasad
Re: Convention on dropping tests under the test framework
On Dec 8, 2006, at 6:30 AM, Prasad Kashyap wrote: 1) just make a copy of test-deployment, (say call it cxf-deployment) 2) use it's child profile to go thro the complete maven lifecycle - compile, build and test your apps. 3) it's parent (deployment-testsuite) will take care of the server start/stop and reporting for you. This won't actually work as easily as you may have imagined Prasad. To effectively build some apps, you actually need to have them live in their own module, so that they can be built using the associated maven plugins. I do not believe that the current testsuite setup takes this into account... and this is why I think it is premature to simply roll out the same config/layout with out having some more real tests, which build these test applications and allow for effective organization. I don't think we are quite there yet... close, but still a bit off. --jason
Re: Convention on dropping tests under the test framework
On Dec 8, 2006, at 9:34 AM, David Jencks wrote: I was wondering why the SeleniumTestSupport and ExtendedSelenium classes were in the tests themselves rather than in a support jar. I'd think these would be used by just about all tests. They were simply never moved to a common location. Looks like they have just been moved though... they should have probably made it into a "selenium" package in the testsupport module. FIxing the tests is likely to take me all day. If you'd like to demonstrate what you think is correct structure you could set something up with the test ear from GERONIMO-2125 http://issues.apache.org/jira/secure/attachment/12336683/manifestcp- itest-v3.jar which has a maven project to build a test ear. With luck deploying it stlll works :-) This is probably a good idea, get some more real tests put together to learn what works and what does not work with the current testsuite setup. --jason
Re: Convention on dropping tests under the test framework
On Dec 8, 2006, at 6:33 AM, Prasad Kashyap wrote: What is a super app anyways? :-P Maybe I should have called it a uber app. Unlike other small apps that we create and use to test a particular feature, DT can serve as the test app to test features across many categories. Small apps to test a particular feature... a test-app? Don't need to go calling other normal applications something else :-P I still don't understand why you even brought daytrader up in this context though... still would like to know that. --jason
Re: Convention on dropping tests under the test framework
On Dec 8, 2006, at 3:50 AM, Jason Dillon wrote: too lazy to insert a :-) every 10 words... so... just assume that I have. :-P dude, you're killing me :-P Matt Hogstrom [EMAIL PROTECTED]
Re: Convention on dropping tests under the test framework
On Dec 8, 2006, at 6:30 AM, Prasad Kashyap wrote: On 12/8/06, David Jencks <[EMAIL PROTECTED]> wrote: For instance it might be useful to have a maven archetype to set up everything except the app to test and the actual test cases. thanks david jencks I have alrady written an archetype plugin called testsuite-archetype-plugin that will let you create a testsuite with an empty testset project under it. Please see http://cwiki.apache.org/GMOxDEV/integration- testing.html#IntegrationTesting-Gettingstarted However, in your case, since you don't want us to be createing any moe testsuites till we have colected enough tests, this is what you should do : 1) just make a copy of test-deployment, (say call it cxf-deployment) 2) use it's child profile to go thro the complete maven lifecycle - compile, build and test your apps. 3) it's parent (deployment-testsuite) will take care of the server start/stop and reporting for you. I wasn't able to make anything work where the app being tested was under the directory that's a copy of *-deployment. Now I'm working with a structure where the *-deployment is a child of the app build directory, and it seems to work. Once I straighten out the tests so they work I'll check it in and we can see if it can be reorganized to be simpler. I was wondering why the SeleniumTestSupport and ExtendedSelenium classes were in the tests themselves rather than in a support jar. I'd think these would be used by just about all tests. FIxing the tests is likely to take me all day. If you'd like to demonstrate what you think is correct structure you could set something up with the test ear from GERONIMO-2125 http://issues.apache.org/jira/secure/attachment/12336683/manifestcp- itest-v3.jar which has a maven project to build a test ear. With luck deploying it stlll works :-) thanks david jencks Cheers Prasad Cheers Prasad On Dec 7, 2006, at 8:31 PM, Prasad Kashyap wrote: > Since there were some questions today on where to drop new tests, I'll > take a stab at creating a convention. Feel free to offer your > suggestions so that we can modify it as we go along. > > We began by having 2 testsuites just as an example. > geronimo/testsuite/ > console-testsuite > deployment-testsuite > > But almost everything can fall under the category of the > deployment-testsuite since most tests do need some deployable > artifact. So I think we should use the deployment-testsuite purely for > testing the deploy tool. Especially, it should be used to test > features like hot-deploy, redeploy and undeploy which have had JIRAs > before. > > We should categorize the tests so that they reflect the broad > functional areas of the server. > > * web-testsuite (servlets, jsp, jstl) > * enterprise-testsuite (ejb, jpa, jms, jacc, jta, javamail, jaf etc) > * mgmt-testsuite (jee management, deployment) > * webservices-testsuite (wsee, jaxb, stax, saaj, ws-metadata etc) > * performance-testsuite (server startup time, server footprint etc) > * security-testsuite > * console-testsuite > * regression (compatibility) > > If nobody has any objection to this top categorization, I shall go > ahead and create these testsuites over the weekend. Meanwhile you may > drop your tests in the existing testsuites for now. I shall move them > appropriately. > > Lastly, how do we deal with super apps like daytrader that can span > across multiple testsuites ? > > Cheers > Prasad
Re: Convention on dropping tests under the test framework
On 12/8/06, Jason Dillon <[EMAIL PROTECTED]> wrote: too lazy to insert a :-) every 10 words... so... just assume that I have. :-P On Dec 7, 2006, at 8:31 PM, Prasad Kashyap wrote: > Lastly, how do we deal with super apps like daytrader that can span > across multiple testsuites ? I don't understand what you mean by this at all. Why does it matter? What is a super app anyways? :-P Maybe I should have called it a uber app. Unlike other small apps that we create and use to test a particular feature, DT can serve as the test app to test features across many categories. --jason Cheers Prasad
Re: Convention on dropping tests under the test framework
On 12/8/06, David Jencks <[EMAIL PROTECTED]> wrote: For instance it might be useful to have a maven archetype to set up everything except the app to test and the actual test cases. thanks david jencks I have alrady written an archetype plugin called testsuite-archetype-plugin that will let you create a testsuite with an empty testset project under it. Please see http://cwiki.apache.org/GMOxDEV/integration-testing.html#IntegrationTesting-Gettingstarted However, in your case, since you don't want us to be createing any moe testsuites till we have colected enough tests, this is what you should do : 1) just make a copy of test-deployment, (say call it cxf-deployment) 2) use it's child profile to go thro the complete maven lifecycle - compile, build and test your apps. 3) it's parent (deployment-testsuite) will take care of the server start/stop and reporting for you. Cheers Prasad Cheers Prasad On Dec 7, 2006, at 8:31 PM, Prasad Kashyap wrote: > Since there were some questions today on where to drop new tests, I'll > take a stab at creating a convention. Feel free to offer your > suggestions so that we can modify it as we go along. > > We began by having 2 testsuites just as an example. > geronimo/testsuite/ > console-testsuite > deployment-testsuite > > But almost everything can fall under the category of the > deployment-testsuite since most tests do need some deployable > artifact. So I think we should use the deployment-testsuite purely for > testing the deploy tool. Especially, it should be used to test > features like hot-deploy, redeploy and undeploy which have had JIRAs > before. > > We should categorize the tests so that they reflect the broad > functional areas of the server. > > * web-testsuite (servlets, jsp, jstl) > * enterprise-testsuite (ejb, jpa, jms, jacc, jta, javamail, jaf etc) > * mgmt-testsuite (jee management, deployment) > * webservices-testsuite (wsee, jaxb, stax, saaj, ws-metadata etc) > * performance-testsuite (server startup time, server footprint etc) > * security-testsuite > * console-testsuite > * regression (compatibility) > > If nobody has any objection to this top categorization, I shall go > ahead and create these testsuites over the weekend. Meanwhile you may > drop your tests in the existing testsuites for now. I shall move them > appropriately. > > Lastly, how do we deal with super apps like daytrader that can span > across multiple testsuites ? > > Cheers > Prasad
Re: Convention on dropping tests under the test framework
too lazy to insert a :-) every 10 words... so... just assume that I have. :-P On Dec 7, 2006, at 8:31 PM, Prasad Kashyap wrote: Since there were some questions today on where to drop new tests, I'll take a stab at creating a convention. Feel free to offer your suggestions so that we can modify it as we go along. We began by having 2 testsuites just as an example. geronimo/testsuite/ console-testsuite deployment-testsuite These were only basic, basic examples for how it _might_ work. Probably needs more thought before its really usable... though console-testsuite, which does not need to build custom apps or other muck, might be fine as is. But other types of tests may need some custom app modules built, which is currently not part of the testsuite/* bits at all. But almost everything can fall under the category of the deployment-testsuite since most tests do need some deployable artifact. So I think we should use the deployment-testsuite purely for testing the deploy tool. Especially, it should be used to test features like hot-deploy, redeploy and undeploy which have had JIRAs before. The deployment-testsuite was intended to test the deployment functionality of Geronimo... not just any place to put tests which have deployables. And I just tossed that together to test module deployment, so maybe it should be refactored... or removed, or who knows. We should categorize the tests so that they reflect the broad functional areas of the server. * web-testsuite (servlets, jsp, jstl) * enterprise-testsuite (ejb, jpa, jms, jacc, jta, javamail, jaf etc) * mgmt-testsuite (jee management, deployment) * webservices-testsuite (wsee, jaxb, stax, saaj, ws-metadata etc) * performance-testsuite (server startup time, server footprint etc) * security-testsuite * console-testsuite * regression (compatibility) If nobody has any objection to this top categorization, I shall go ahead and create these testsuites over the weekend. Meanwhile you may drop your tests in the existing testsuites for now. I shall move them appropriately. Please wait on this until there is a real more complete set of tests which show that the setup/layout will scale for most/all types of tests... I am fairly sure that it will not scale asis. Lastly, how do we deal with super apps like daytrader that can span across multiple testsuites ? I don't understand what you mean by this at all. Why does it matter? What is a super app anyways? :-P --jason
Re: Convention on dropping tests under the test framework
I agree. We need some *real* tests first so that we can see if the setup we have no will work, or will not. Looks like it might need some more changes before we'd want to make a cookie cutter. --jason On Dec 8, 2006, at 12:35 AM, David Jencks wrote: I'm afraid I think it's way too early to try to categorize where tests should go. Based on my experience so far I think we need to work on simplifying how to set up a small app and have some simple tests for it first. Once we have enough of these to confuse ourselves about organization, and it's dead simple to write a new one, we can worry about categorizing them. For instance it might be useful to have a maven archetype to set up everything except the app to test and the actual test cases. thanks david jencks On Dec 7, 2006, at 8:31 PM, Prasad Kashyap wrote: Since there were some questions today on where to drop new tests, I'll take a stab at creating a convention. Feel free to offer your suggestions so that we can modify it as we go along. We began by having 2 testsuites just as an example. geronimo/testsuite/ console-testsuite deployment-testsuite But almost everything can fall under the category of the deployment-testsuite since most tests do need some deployable artifact. So I think we should use the deployment-testsuite purely for testing the deploy tool. Especially, it should be used to test features like hot-deploy, redeploy and undeploy which have had JIRAs before. We should categorize the tests so that they reflect the broad functional areas of the server. * web-testsuite (servlets, jsp, jstl) * enterprise-testsuite (ejb, jpa, jms, jacc, jta, javamail, jaf etc) * mgmt-testsuite (jee management, deployment) * webservices-testsuite (wsee, jaxb, stax, saaj, ws-metadata etc) * performance-testsuite (server startup time, server footprint etc) * security-testsuite * console-testsuite * regression (compatibility) If nobody has any objection to this top categorization, I shall go ahead and create these testsuites over the weekend. Meanwhile you may drop your tests in the existing testsuites for now. I shall move them appropriately. Lastly, how do we deal with super apps like daytrader that can span across multiple testsuites ? Cheers Prasad
Re: Convention on dropping tests under the test framework
I'm afraid I think it's way too early to try to categorize where tests should go. Based on my experience so far I think we need to work on simplifying how to set up a small app and have some simple tests for it first. Once we have enough of these to confuse ourselves about organization, and it's dead simple to write a new one, we can worry about categorizing them. For instance it might be useful to have a maven archetype to set up everything except the app to test and the actual test cases. thanks david jencks On Dec 7, 2006, at 8:31 PM, Prasad Kashyap wrote: Since there were some questions today on where to drop new tests, I'll take a stab at creating a convention. Feel free to offer your suggestions so that we can modify it as we go along. We began by having 2 testsuites just as an example. geronimo/testsuite/ console-testsuite deployment-testsuite But almost everything can fall under the category of the deployment-testsuite since most tests do need some deployable artifact. So I think we should use the deployment-testsuite purely for testing the deploy tool. Especially, it should be used to test features like hot-deploy, redeploy and undeploy which have had JIRAs before. We should categorize the tests so that they reflect the broad functional areas of the server. * web-testsuite (servlets, jsp, jstl) * enterprise-testsuite (ejb, jpa, jms, jacc, jta, javamail, jaf etc) * mgmt-testsuite (jee management, deployment) * webservices-testsuite (wsee, jaxb, stax, saaj, ws-metadata etc) * performance-testsuite (server startup time, server footprint etc) * security-testsuite * console-testsuite * regression (compatibility) If nobody has any objection to this top categorization, I shall go ahead and create these testsuites over the weekend. Meanwhile you may drop your tests in the existing testsuites for now. I shall move them appropriately. Lastly, how do we deal with super apps like daytrader that can span across multiple testsuites ? Cheers Prasad
Convention on dropping tests under the test framework
Since there were some questions today on where to drop new tests, I'll take a stab at creating a convention. Feel free to offer your suggestions so that we can modify it as we go along. We began by having 2 testsuites just as an example. geronimo/testsuite/ console-testsuite deployment-testsuite But almost everything can fall under the category of the deployment-testsuite since most tests do need some deployable artifact. So I think we should use the deployment-testsuite purely for testing the deploy tool. Especially, it should be used to test features like hot-deploy, redeploy and undeploy which have had JIRAs before. We should categorize the tests so that they reflect the broad functional areas of the server. * web-testsuite (servlets, jsp, jstl) * enterprise-testsuite (ejb, jpa, jms, jacc, jta, javamail, jaf etc) * mgmt-testsuite (jee management, deployment) * webservices-testsuite (wsee, jaxb, stax, saaj, ws-metadata etc) * performance-testsuite (server startup time, server footprint etc) * security-testsuite * console-testsuite * regression (compatibility) If nobody has any objection to this top categorization, I shall go ahead and create these testsuites over the weekend. Meanwhile you may drop your tests in the existing testsuites for now. I shall move them appropriately. Lastly, how do we deal with super apps like daytrader that can span across multiple testsuites ? Cheers Prasad