On Tue, Apr 22, 2008 at 4:25 PM, Martijn van Oosterhout <[EMAIL PROTECTED]>
wrote:
> On Tue, Apr 22, 2008 at 10:31:53AM +0200, Zdenek Kotala wrote:
> > When you are able detect ordering difference you are able also check if
> it
> > is important for the test or not without any extra effort. Only what we
> > need is put some flag to test that order is not important.
>
> Not true. Sorting the file is going jumble all the results together.
> Since we perform many tests in one file, you're not going to be able to
> seperate them.
>
> > Regression test MUST BE bulletproof. If you get a error you must know
> that
> > it is really error (in postgresql or regtest) and must be fixed. When
> you
> > start to ignore some errors because it can happen sometimes you fall in
> the
> > trap soon.
>
> I think people are misunderstanding. You posted a bunch of diffs with
> that comment that they *appeared* to only be ordering differences. How
> good did you check? If an 8 become a 9 chances are you'd miss it.
> Having a second test checking the sorted results would at least
> preclude the chance that there really is something wrong.
>
> It was a guide, not a way of getting out of tests.
>
In the past, I had faced and tried to work on this exact problem... here's
what I had in mind:
in the .expected file, we would demarcate the section of lines we expect to
come in any order, by using two special markers. Then, when comparing the
actual output with expected output, we would take the demarcated group of
lines, and the corresponding lines from actual output, and compare them
after sorting.
For eg.
foo.expected:
select * from tenk where col1 <= 3 limit 3;
col1 | col2 | col3
-------------------------
?unsorted_result_start
1 | 10 | 100
2 | 20 | 200
3 | 30 | 300
?unsorted_result_end
foo.out:
select * from tenk where col1 <= 3 limit 3;
col1 | col2 | col3
-------------------------
3 | 30 | 300
2 | 20 | 200
1 | 10 | 100
So, the diff program should discard the lines beginning with '?' (meta
character), and then sort and match exactly the same number of lines.
There's another option of putting these '?' lines in a separate file
with corresponding begin/end line numbers of the unsorted group, and using
this as a parameter to the diffing program.
Of course, this needs a change in the (standard) diff that we use from
pg_regress!
Best regards,
--
[EMAIL PROTECTED]
[EMAIL PROTECTED] gmail | hotmail | indiatimes | yahoo }.com
EnterpriseDB http://www.enterprisedb.com
Mail sent from my BlackLaptop device
