I have several customers using this feature. I will have to think
about whether the new proposal is going to work for them.
I guess we can document the process. They have to be saving the
responses and then they probably want to use a recognised name
for that step so they can easily find the response(s) saved for
that step. Then the only trick is whether to save the first or
the last response for that step - depending on whether they are
applying the filters to the referenceFile or not.
Perhaps its not so bad, but I would like to have the documentation
finished before removing the old step. I will have a go at this myself
if I get time.
Paul.
Marc Guillemot wrote:
Hi Paul,
I've intentionally removed this automatic creation as explained in my
original mail. I think that this functionality doesn't belong into a
test even if I understand that it may be useful in some cases.
Creating a reference file should not be automatic otherwise the risk to
miss the real reference file (due for instance to path/name problem)
without being aware of it is too high (we've had the case with
compareToExpected). More generally I think that a test should not
produce any file that has to go directly under source control without
human action.
Generating the reference file automatically without any hand action
would cause the test to ... not test anything what is not its aim. I
think that the right way to handle such a case is first to create a
dummy reference file, then let the test run and fail because the real
file differs from the reference one. Then, and only after a manual check
of the received real file, to place this file as reference file.
Marc.
Paul King wrote:
Hi Marc,
I think I misunderstood your intentions with verifyContent
vs compareToExpected. While compareToExpected had some
issues which needed fixing, it did perform some functionality
which seems to have now gone with verifyContent.
In particular, the ability to create the reference file
the first time you ran the test seems to have gone.
This is pretty key because in general it is very hard to
potentially create the reference file except by applying
the same filters later used when verifyingContent.
Two uses cases we have:
(1) A PDF file is generated from a third-party templating
system. The customer address, barcode etc are removed from
the PDF using a custom filter. This becomes the reference
file the first time the test is run. If the template is
changed, we simply delete the reference file and it is
recreated.
(2) A web service returns a zipped file as an attachment.
Using a custom filter, the attachment is extracted from
the web service response, unzipped and some of the
information contained within an XML file within the zip
becomes the reference file.
Is there a plan that would allow verifyContent to handle
these scenarios? I think we should not deprecate
compareToExpected until we have such functionality in place.
Paul.
Marc Guillemot wrote:
Hi,
I've several problems with current compareToExpected step:
- name doesn't fit with all our verifyXXX steps
- if reference file doesn't exist, it looks silently for it in a
subfolder named "expected"
- it doesn't fail if reference file doesn't exist at all but just
creates it
- it doesn't fail if an IO error occurs
Therefore I propose to deprecate compareToExpected and to introduce a
verifyContent which would offer the same functionalities as
compareToExpected but in a more deterministic way. This would look like:
<verifyContent referenceFile="someFile.xxx".../>
Any thoughts?
Marc.
_______________________________________________
WebTest mailing list
[email protected]
http://lists.canoo.com/mailman/listinfo/webtest
