Re: Functional Test strategies with Maven
Hi Martijn! I'm joining this discussion a little late, but have some references to add which may help making up your mind: http://docs.codehaus.org/display/MAVENUSER/Maven+and+Integration+Testing http://www.jroller.com/carlossg/entry/functional_testing_with_maven_cargo hth, - martin On Monday 02 March 2009 martijnverb...@gmail.com wrote: Hi all, I'd like to get opinions from others on how they structure/deal with what we deem 'functional tests' To us these are tests that are broader in scope than a unit test but whose scope is still limited to that same module and require no 3rd party resources to run (we're able to Mock everything we need). Traditionally a Maven structure looks something like: foobar jar src main java test java war ... sar ... rar ... ear With the 'test' structure being used for Unit Tests. So we're wondering if it was wise to put the functional tests under jar--test--src--java (perhaps differentiating them by package structure) or introduce a new module (say functional-test) eg foobar jar src main java test java functional-test src test java war ... sar ... rar ... ear ... Is there a preferred technique/other options that people have used? signature.asc Description: This is a digitally signed message part.
Functional Test strategies with Maven
Hi all, I'd like to get opinions from others on how they structure/deal with what we deem 'functional tests' To us these are tests that are broader in scope than a unit test but whose scope is still limited to that same module and require no 3rd party resources to run (we're able to Mock everything we need). Traditionally a Maven structure looks something like: foobar jar src main java test java war ... sar ... rar ... ear With the 'test' structure being used for Unit Tests. So we're wondering if it was wise to put the functional tests under jar--test--src--java (perhaps differentiating them by package structure) or introduce a new module (say functional-test) eg foobar jar src main java test java functional-test src test java war ... sar ... rar ... ear ... Is there a preferred technique/other options that people have used?
Re: Functional Test strategies with Maven
Hi Martin, What is the reasoning beyond the separation of the functional tests? Is it because you would like to run them separately from your other unit tests? If so, have you considered using profiles to control what tests are run by Maven? If not, what goal are you trying to obtain by putting them in a separate area? Ian On Mon, Mar 2, 2009 at 3:33 PM, martijnverb...@gmail.com wrote: Hi all, I'd like to get opinions from others on how they structure/deal with what we deem 'functional tests' To us these are tests that are broader in scope than a unit test but whose scope is still limited to that same module and require no 3rd party resources to run (we're able to Mock everything we need). Traditionally a Maven structure looks something like: foobar jar src main java test java war ... sar ... rar ... ear With the 'test' structure being used for Unit Tests. So we're wondering if it was wise to put the functional tests under jar--test--src--java (perhaps differentiating them by package structure) or introduce a new module (say functional-test) eg foobar jar src main java test java functional-test src test java war ... sar ... rar ... ear ... Is there a preferred technique/other options that people have used?
Re: Re: Functional Test strategies with Maven
Are these functional tests invoked as jUnit / TestNG tests? I have written similar, integration type functional tests with mocks, but they were still based on jUnit / TestNG. I simply stored them in the same module name--test--src--java are being used for unit tests. I differentated between the two sets through the package names and used profiles to run these optionally. My reasoning was that its easiest to stick with the vanilla Maven directory structure as it is a standard. This approach certainly worked for me at the time. I guess that would be my suggestion. ps: I think you replied directly to me, rather reply to the group email address. On Mon, Mar 2, 2009 at 4:08 PM, martijnverb...@gmail.com wrote: Hi Ian, What is the reasoning beyond the separation of the functional tests? Is it because you would like to run them separately from your other unit tests? If so, have you considered using profiles to control what tests are run by Maven? The specific purpose in this case is to ensure that we've wired up messaging flow(s) within our module. This is done by sending a mocked event through the various routers/transformers etc that we have. They're not unit tests because they're not testing a discrete bit of functionality within a Java class. However, it is vital that these tests pass before we package that module, so I we want to run them as part of a the test life-cycle (as opposed to integration-test which is post package) or at least a life-cycle phase that is before package. I agree that we could use profiles to run this group of tests separately, but I'd like to understand where they should live in the source structure. All of the Maven literature that I have read talks of module name--test--src--java being used for unit tests, but tests such as mine fall into a grey area? Cheers, Martijn On Mon, Mar 2, 2009 at 3:33 PM, martijnverb...@gmail.com wrote: Hi all, I'd like to get opinions from others on how they structure/deal with what we deem 'functional tests' To us these are tests that are broader in scope than a unit test but whose scope is still limited to that same module and require no 3rd party resources to run (we're able to Mock everything we need). Traditionally a Maven structure looks something like: foobar jar src main java test java war ... sar ... rar ... ear With the 'test' structure being used for Unit Tests. So we're wondering if it was wise to put the functional tests under jar--test--src--java (perhaps differentiating them by package structure) or introduce a new module (say functional-test) eg foobar jar src main java test java functional-test src test java war ... sar ... rar ... ear ... Is there a preferred technique/other options that people have used?
Re: Re: Re: Functional Test strategies with Maven
Hi all, Are these functional tests invoked as jUnit / TestNG tests? I have written similar, integration type functional tests with mocks, but they were still based on jUnit / TestNG. I simply stored them in the same name--test--src--java are being used for unit tests. I differentiated between the two sets through the package names and used profiles to run these optionally. Yes they are JUnit tests and the idea of differentiating by package names is definitely one we've discussed. It's interesting to know that someone is using that technique! My reasoning was that its easiest to stick with the vanilla Maven directory structure as it is a standard. This approach certainly worked for me at the time. I think it would definitely work for us as well, but it's an interesting 'gap' in the standard maven project and so we hesitated on which way to go :) ps: I think you replied directly to me, rather reply to the group email address. Apologies for that, I'm using the iGoogle gmail interface, and the reply picks up the user and not the mailing list address first, add my stupidity on top of that and you get mailing list fail :) Original post back to Ian Hi Ian, What is the reasoning beyond the separation of the functional tests? Is it because you would like to run them separately from your other unit tests? If so, have you considered using profiles to control what tests are run by Maven? The specific purpose in this case is to ensure that we've wired up messaging flow(s) within our module. This is done by sending a mocked event through the various routers/transformers etc that we have. They're not unit tests because they're not testing a discrete bit of functionality within a Java class. However, it is vital that these tests pass before we package that module, so I we want to run them as part of a the test life-cycle (as opposed to integration-test which is post package) or at least a life-cycle phase that is before package. I agree that we could use profiles to run this group of tests separately, but I'd like to understand where they should live in the source structure. All of the Maven literature that I have read talks of module name--test--src--java being used for unit tests, but tests such as mine fall into a grey area? --- Thanks again for your insights Ian. Does anyone else follow a different/similar method? Cheers, Martijn On Mon, Mar 2, 2009 at 4:08 PM, martijnverb...@gmail.com wrote: Hi Ian, What is the reasoning beyond the separation of the functional tests? Is it because you would like to run them separately from your other unit tests? If so, have you considered using profiles to control what tests are run by Maven? The specific purpose in this case is to ensure that we've wired up messaging flow(s) within our module. This is done by sending a mocked event through the various routers/transformers etc that we have. They're not unit tests because they're not testing a discrete bit of functionality within a Java class. However, it is vital that these tests pass before we package that module, so I we want to run them as part of a the test life-cycle (as opposed to integration-test which is post package) or at least a life-cycle phase that is before package. I agree that we could use profiles to run this group of tests separately, but I'd like to understand where they should live in the source structure. All of the Maven literature that I have read talks of name--test--src--java being used for unit tests, but tests such as mine fall into a grey area? Cheers, Martijn On Mon, Mar 2, 2009 at 3:33 PM, martijnverb...@gmail.com wrote: Hi all, I'd like to get opinions from others on how they structure/deal with what we deem 'functional tests' To us these are tests that are broader in scope than a unit test but whose scope is still limited to that same module and require no 3rd party resources to run (we're able to Mock everything we need). Traditionally a Maven structure looks something like: foobar jar src main java test java war ... sar ... rar ... ear With the 'test' structure being used for Unit Tests. So we're wondering if it was wise to put the functional tests under jar--test--src--java (perhaps differentiating them by package structure) or introduce a new module (say functional-test) eg foobar
Re: Functional Test strategies with Maven
Martijn - I've seen this done a couple of different ways. I can't say that there is any particular set of advantages/disadvantages that makes one more attractive than another. I'm sure you've already thought about these, but here they are, just the same: #1 - The functional tests are named with a specific suffix (ex. _FT) so that they can be included/excluded by profile. They still live in the src/test/java tree alongside the unit tests. In this case, you must explicitly include and/or exclude tests based on a pattern matching their class names. #2 - Place all of your functional tests in a different source tree (ex. src/functionalTest/java). As with #1, you have to do some include/exclude type work, but in this case, you only have to do so on your profile that runs the functional tests. However, you also have to make sure to include this tree on your compilation. So, IMHO, there's no clear winner. I've been dealing with similar problems lately for Integration Tests that actually start up my web app and test it using Selenium. I have the same set of problems. I think I read that there are plans to enhance Maven with the knowledge of functional/integration tests, but I don't know what the timing for any such change is. Dave The specific purpose in this case is to ensure that we've wired up messaging flow(s) within our module. This is done by sending a mocked event through the various routers/transformers etc that we have. They're not unit tests because they're not testing a discrete bit of functionality within a Java class. However, it is vital that these tests pass before we package that module, so I we want to run them as part of a the test life-cycle (as opposed to integration-test which is post package) or at least a life-cycle phase that is before package. I agree that we could use profiles to run this group of tests separately, but I'd like to understand where they should live in the source structure. All of the Maven literature that I have read talks of module name--test--src--java being used for unit tests, but tests such as mine fall into a grey area? - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Re: Functional Test strategies with Maven
Hi David, Martijn - I've seen this done a couple of different ways. I can't say that there is any particular set of advantages/disadvantages that makes one more attractive than another. I'm sure you've already thought about these, but here they are, just the same: #1 - The functional tests are named with a specific suffix (ex. _FT) so that they can be included/excluded by profile. They still live in the src/test/java tree alongside the unit tests. In this case, you must explicitly include and/or exclude tests based on a pattern matching their class names. #2 - Place all of your functional tests in a different source tree (ex. src/functionalTest/java). As with #1, you have to do some include/exclude type work, but in this case, you only have to do so on your profile that runs the functional tests. However, you also have to make sure to include this tree on your compilation. So, IMHO, there's no clear winner. I've been dealing with similar problems lately for Integration Tests that actually start up my web app and test it using Selenium. I have the same set of problems. I think I read that there are plans to enhance Maven with the knowledge of functional/integration tests, but I don't know what the timing for any such change is. Thanks David, this helps (along with Ian's comments) confirm that there is currently no 'official' Maven life-cycle support for these types of tests (yet), hopefully as long as we're consistent in our approach any of the suggested solutions will be more than adequate :). Cheers, Martijn PS: I will also post separately about the 'true' integration tests scenario (where we have tests that can be JUnit run but involve many modules and/or projects and can also have 3rd party requirements such as a J2EE server up and running). The specific purpose in this case is to ensure that we've wired up messaging flow(s) within our module. This is done by sending a mocked event through the various routers/transformers etc that we have. They're not unit tests because they're not testing a discrete bit of functionality within a Java class. However, it is vital that these tests pass before we package that module, so I we want to run them as part of a the test life-cycle (as opposed to integration-test which is post package) or at least a life-cycle phase that is before package. I agree that we could use profiles to run this group of tests separately, but I'd like to understand where they should live in the source structure. All of the Maven literature that I have read talks of name--test--src--java being used for unit tests, but tests such as mine fall into a grey area? - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
RE: Re: Functional Test strategies with Maven
Trying to achieve consistency between organizations ahead of time could prove difficult. I would instead suggest trying for consistency within your organization and sharing notes about what worked and what didn't either on this mailing list or perhaps the Maven wiki. Perhaps that could help provide feedback for if/when integration testing is officially supported. /S -Original Message- From: martijnverb...@gmail.com [mailto:martijnverb...@gmail.com] Sent: Monday, March 02, 2009 11:26 AM To: users@maven.apache.org Subject: Re: Re: Functional Test strategies with Maven Hi David, Martijn - I've seen this done a couple of different ways. I can't say that there is any particular set of advantages/disadvantages that makes one more attractive than another. I'm sure you've already thought about these, but here they are, just the same: #1 - The functional tests are named with a specific suffix (ex. _FT) so that they can be included/excluded by profile. They still live in the src/test/java tree alongside the unit tests. In this case, you must explicitly include and/or exclude tests based on a pattern matching their class names. #2 - Place all of your functional tests in a different source tree (ex. src/functionalTest/java). As with #1, you have to do some include/exclude type work, but in this case, you only have to do so on your profile that runs the functional tests. However, you also have to make sure to include this tree on your compilation. So, IMHO, there's no clear winner. I've been dealing with similar problems lately for Integration Tests that actually start up my web app and test it using Selenium. I have the same set of problems. I think I read that there are plans to enhance Maven with the knowledge of functional/integration tests, but I don't know what the timing for any such change is. Thanks David, this helps (along with Ian's comments) confirm that there is currently no 'official' Maven life-cycle support for these types of tests (yet), hopefully as long as we're consistent in our approach any of the suggested solutions will be more than adequate :). Cheers, Martijn PS: I will also post separately about the 'true' integration tests scenario (where we have tests that can be JUnit run but involve many modules and/or projects and can also have 3rd party requirements such as a J2EE server up and running). The specific purpose in this case is to ensure that we've wired up messaging flow(s) within our module. This is done by sending a mocked event through the various routers/transformers etc that we have. They're not unit tests because they're not testing a discrete bit of functionality within a Java class. However, it is vital that these tests pass before we package that module, so I we want to run them as part of a the test life-cycle (as opposed to integration-test which is post package) or at least a life-cycle phase that is before package. I agree that we could use profiles to run this group of tests separately, but I'd like to understand where they should live in the source structure. All of the Maven literature that I have read talks of name--test--src--java being used for unit tests, but tests such as mine fall into a grey area? - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org -- CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: RE: Re: Functional Test strategies with Maven
Hi Steven, Oh absolutely, sorry when I said we I meant my organisation :) Cheers, Martijn On Mar 2, 2009 5:32pm, Cummings,Steven steven.cummi...@cerner.com wrote: Trying to achieve consistency between organizations ahead of time could prove difficult. I would instead suggest trying for consistency within your organization and sharing notes about what worked and what didn't either on this mailing list or perhaps the Maven wiki. Perhaps that could help provide feedback for if/when integration testing is officially supported. /S -Original Message- From: martijnverb...@gmail.com [mailto:martijnverb...@gmail.com] Sent: Monday, March 02, 2009 11:26 AM To: users@maven.apache.org Subject: Re: Re: Functional Test strategies with Maven Hi David, Martijn - I've seen this done a couple of different ways. I can't say that there is any particular set of advantages/disadvantages that makes one more attractive than another. I'm sure you've already thought about these, but here they are, just the same: #1 - The functional tests are named with a specific suffix (ex. _FT) so that they can be included/excluded by profile. They still live in the src/test/java tree alongside the unit tests. In this case, you must explicitly include and/or exclude tests based on a pattern matching their class names. #2 - Place all of your functional tests in a different source tree (ex. src/functionalTest/java). As with #1, you have to do some include/exclude type work, but in this case, you only have to do so on your profile that runs the functional tests. However, you also have to make sure to include this tree on your compilation. So, IMHO, there's no clear winner. I've been dealing with similar problems lately for Integration Tests that actually start up my web app and test it using Selenium. I have the same set of problems. I think I read that there are plans to enhance Maven with the knowledge of functional/integration tests, but I don't know what the timing for any such change is. Thanks David, this helps (along with Ian's comments) confirm that there is currently no 'official' Maven life-cycle support for these types of tests (yet), hopefully as long as we're consistent in our approach any of the suggested solutions will be more than adequate :). Cheers, Martijn PS: I will also post separately about the 'true' integration tests scenario (where we have tests that can be JUnit run but involve many modules and/or projects and can also have 3rd party requirements such as a J2EE server up and running). The specific purpose in this case is to ensure that we've wired up messaging flow(s) within our module. This is done by sending a mocked event through the various routers/transformers etc that we have. They're not unit tests because they're not testing a discrete bit of functionality within a Java class. However, it is vital that these tests pass before we package that module, so I we want to run them as part of a the test life-cycle (as opposed to integration-test which is post package) or at least a life-cycle phase that is before package. I agree that we could use profiles to run this group of tests separately, but I'd like to understand where they should live in the source structure. All of the Maven literature that I have read talks of name--test--src--java being used for unit tests, but tests such as mine fall into a grey area? - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org -- CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, USA at (+1) (816)221-1024. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
