On 19/12/2012, at 6:18 AM, Matt Khan wrote: > Classification: Public > > Hello > > We have a gradle based build/release/deployment system and we use > cucumber-jvm (https://github.com/cucumber/cucumber-jvm) for certain types of > testing. > > ***Some preamble about cucumber first which could be skipped if you’re > familiar with cucumber*** > > cucumber (or rather gherkin) is set of keywords organised into a tree like > structure like (taking the example from their page) > > Feature: Addition > Scenario: Add two numbers > Given I have entered 50 into the calculator > And I have entered 70 into the calculator > When I press add > Then the result should be 120 on the screen > > cucumber-jvm gives you 2 ways to run tests; via cli interface or via a junit > runner. The junit approach exposes each level in this hierarchy as a junit > Description where each description is simply it’s displayed name, in this > example we’d have > > a Description with name “Feature: Addition” with 1 child, which is > a Description with name “Scenario: Add two numbers” with 4 children, which are > a Description for each step with names “Given I have entered 50 into the > calculator” etc > > There are other variations (scenario outlines, data tables etc) but they all > fall into this sort of hierarchical structure. > > *** The meat of the problem *** > > As of today (gradle 1.3), the only option to run cucumber-jvm from gradle is > to create a custom javaexec task that runs the cucumber cli Main with build > specific kludges to make it agree with how gradle runs testng/junit (like > checking that tests have run). This is because cucumber’s use of junit does > appear to agree with gradle’s interpretation. There are, at least, 2 problems; > > 1) a blocking bug (an NPE in gradle code triggered by the fact that > cucumber generated Descriptions produce a null “methodName”) that prevents it > from working at all (described in this thread > -http://forums.gradle.org/gradle/topics/beforetest_descriptor_is_null_and_the_test_is_not_actually_run_trying_to_run_groovy_and_java_cucumber_step_definition) > 2) (if one avoids the NPE by just adding a null check) gradle renders > the results as a list of scenarios only > > The 2nd point means that key information is missing (on which step did the > test fail?). > > My Q is pretty simple... what’s the “right” way to get this working in gradle > such that the full hierarchy of information available is preserved? > Alternatively should we not expect to see the gradle test reporting encompass > this sort of content and instead focus on getting the execution side working > (without the need for custom javaexec tasks) but using cucumber to generate > test reports?
I'd like to see Gradle be able to deal with this. To fix this, we'd need to change the test reporting so that it does not assume that a test == a method in a class. It might also be worthwhile putting together a plugin that connects up Gradle's TestClassProcessor directly to cucumber, assuming it has some kind of native execution API. This way, we can get proper descriptions for these things in the reports (e.g. 'Feature', 'Scenario', 'Step' rather than generic 'Test'). Are you interested in helping with some of this stuff? -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
