Greetings,
I'm writing to report an issue I stumbled upon while trying to adjust to
the new (for me) world of implicit-subject expectation writing.
Consider this rather silly test case:
describe "something" do
context "in context" do
subject { "an object" }
it { should == "an object" }
end
end
When I run rspec with `--format documentation`, I see the nifty
documentation generated by the matcher itself:
something
in context
should == "an object"
However, this is what I see when I run with `--format json` (modulo pretty
printing):
{
"summary_line": "1 example, 0 failures",
"summary": {
"example_count": 1,
"duration": 0.000344,
"failure_count": 0,
"pending_count":0
},
"examples": [
{
"status": "passed",
"description": "should == \"an object\"",
"full_description": "something in context ",
"line_number": 4,
"file_path": "./spec/test_spec.rb"
}
]
}
Note that the full_description is missing the matcher documentation. And
with ci_reporter 1.8.4, this comes out as a sorry hash indeed, because
_only_ the full descriptions are reported to CI. Here's the trimmed XML
output.
SPEC-something.xml:
<testsuite errors="0" tests="0" failures="0" name="something" skipped="0"
time="0.000228">
...
</testsuite>
SPEC-something-in-context.xml:
<testsuite errors="0" tests="1" failures="0" name="something in context"
skipped="0" time="0.00102">
<testcase name="something in context " time="0.000651" />
...
</testsuite>
Now imagine you had 15-20 test cases whose names were all "something in
context". Ugh.
What I want to see, of course, is the generated matcher documentation being
added to the end of the full description of the example, and showing up in
my CI reports.
----
I couldn't find a report of this issue already, but please stop here and
wave me off if you've already got a handle on this. Otherwise, here's my
analysis and suggested fix.
----
At first I was digging through ci_reporter to see if there was something
wrong on that end, but you can see that the built-in JSON reporter is
running into the same problem: full_description is simply missing important
stuff.
The problem as I see it starts from the fact that the generated description
from the matcher gets stored into metadata[:description] after the example
is run, but metadata[:full_description] never gets updated. This results in
the following metadata hash for our example:
{
:caller => ...,
:description => "should == \"an object\"",
:description_args = [],
:example_group => ...,
:example_group_block => #<Proc: ...>,
:execution_result => ...,
}
No :full_description entry in sight.
This means that, when someone calls `example.description` after the test is
run, they get the generated matcher description, because the implementation
of Example#description sees that `metadata[:description]` is set. But when
someone calls `example.full_description` or
`example.metadata[:full_description]`, both fall through to this
implementation, from the ExampleMetadata module:
def full_description
build_description_from(self[:example_group][:full_description],
*self[:description_args])
end
Hmm... I see the parent's full description, the empty list of
description_args... but where's the example's description itself? We're not
taking into account that in this case the description didn't come from the
args at all, it came from the generated matcher. That, I believe, is the
root cause.
I think the best behavior is to respect the description args above all, but
_only if they're given_. Failing that, we should use the generated
description if it's available, and then we do I dunno what (stick with the
above, or go back to "example at ..."). Going with this behavior would fix
my issue.
So, my suggested fix is to have #full_description take
metadata[:description] into account:
def full_description
+ if self[:description]
+ build_description_from(self[:example_group][:full_description],
self[:description])
+ else
build_description_from(self[:example_group][:full_description],
*self[:description_args])
+ end
end
It's not the cleanest patch, because it appears on the surface to be
undermining the behavior I described above, prioritizing whatever it finds
in self[:description], generated or otherwise, above
self[:description_args]. However, I think it works, and I'm not sure how
better to express it. If metadata[:description] is set, I think it could
contain either the contents of metadata[:description_args] or the generated
matcher description (any other possibilities?). And it looks to me from
Example#assign_generated_description that if metadata[:description] is
already filled out (from the description args), the previous contents will
win. So really, I think what you're going to see is: if we fall through to
the else, self[:description_args] is going to be empty. But I have yet to
prove that.
Questions notwithstanding, I've tried a monkey patch with this and it does
indeed fix my issue. I can work it into a full test case and submission if
you want, but as I'm new to the group I'll need some guidance.
Thanks for reviewing, and for a fun little test framework!
-- Owen
_______________________________________________
rspec-users mailing list
rspec-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/rspec-users
