(I hope this isn't dupe, the interface accepted the message, but it
didn't show up)
On Mar 22, 10:14 pm, Jonathan Pryor <[email protected]> wrote:
> On Mon, 2010-03-22 at 07:38 -0700, Anders wrote:
> > - In the repository there are both an "examples" and a "tests" folder,
> > both containing sql scripts for creating a northwind database for each
> > database vendor. It seems that it is the scripts from the "examples"
> > folder which should be used for running the unit tests (contrary to
> > what I would expect from the naming and contrary to what the "Unit
> > Tests" page in this groups says), since e.g. for the scripts in the
> > "tests" folder:
> [...]
> Short, short version: IGNORE the SQL scripts within 'examples', and use
> the ones in 'tests' (if possible, and you later mention a number of
> issues with this...).
> [...]
> Compare [the standard northwind schema in tests/instnwnd.*.sql] to e.g.
> examples/DbLinq.Pgsql.Example/sql/createDB_Northwind_pg.sql, where the
> schema is recreated but NOT the same dataset.
> [...]
Of course the test databases should contain the same data across
vendors such that tests can be reused. And I didn't assume that the
data in the "examples" scripts is the same as in the "tests" scripts.
But it seems that the data in the "examples" scripts is the same
across vendors. I have just eyeballed the "examples" scripts (lining
up the scripts pairwise in my editor and flipping between them) and
they seem identical. The only potential problem I spotted was that
current time is used for the "Orders"."OrderDate". This should
perhaps be fixed. I call it a potential problem since all vendors had
a single row for which the date was hard-coded to 16h of October 1996
such that the ReadTests_DateTimeFunctions.Get* tests and
LinqToSqlString16...19 tests are deterministic.
The reason that the "examples" scripts have other (far less) data than
the "tests" scripts, I assume, is exactly to make them better suited
for unit testing. For example the EntitySet.Refresh01 and .Refresh02
test takes about 2 minutes on my (admitted, not very fast) computer
using the full northwind database from the "tests" folder.
Also it seems significant that the data contexts used by the unit
tests are found in "examples/DbLinq.*.Example/nwind/Northwind.cs"
"next to" the sql scripts.
So for these reasons, in addition to the reasons I mentioned
previously (which means that the "tests/"-scripts couldn't have been
used for producing the previous test results page), I think, I will
still have to respectfully disagree that the "examples" scripts should
be ignored. (I know you have worked far more on this project than I,
so I am not in any way assuming to know more about this than you, I
just can't figure out why the "tests/" scripts would be correct)
But I guess it should now be verified that no test have been written
assuming a wrong schema such that it "silently succeeds" (i.e. without
testing anything) in the correct schema.
> > - I have created a small application "TestReport" [...]
> Please check in the tool.
Will do. Seeing the problems you point out below, let's call it work
in progress.
> 1. How are you running DbLinq.Sqlite_test_mono_strict.dll and what
> errors are you getting? It's intended to be run under Mono >= 2.6
> (NOT .NET). (If you look at the assembly references, you'll see that it
> does NOT reference DbLinq.dll, but ~only System.Data.Linq.dll.)
I was running it using the MS runtime, and assumed (based on the
naming of the assembly) that that was why it failed and therefor I
decided to let it rest until I knew if I was on the right track. As to
not pollute the table with misleading "FAIL"s, I just left that column
out (it isn't in the current version either).
It seems that the tests all fail with some form of
"System.InvalidOperationException : Incorrect AutoSync specification
for member 'OrderID'". Perhaps because something is in an unexpected
name-space.
> 2. You *MAY* want to consider an alternate table-generation approach.
>
> In an "ideal" world, all "known-bad" tests will have [Exclude]
> attributes on them in the "release" build. (This is true for SQL Server
> and SQLite, and may be true for PostgreSQL.) If this is the case for
> all databases, generating the pass/fail table becomes a simple exercise
> in Reflection -- any test with an [Explicit] attribute is a FAIL, and
> anything without the attribute is a PASS. Just reflect over the
> assemblies, and you're done.
>
> Generating a table in this fashion should be fast/trivial, but also
> requires that the [Explicit] attributes be kept in-sync so that Release
> builds are always "green" (which is certainly good...if we had actual
> maintainers for those databases, otherwise it's just overhead.)
Aren't there a difference between "known-bad" in the sense that "this
will never work for this vendor" (e.g. testing for store procedures in
a database that doesn't support them), and failing tests that are
failing just because the errors haven't been fixed yet. I don't assume
the later type is supposed to be disabled. It would seem quite odd to
me to disable tests just because they are failing.
BTW, I was wondering if it wouldn't be an idea to use categories for
disabling tests instead of the "preprocessor directives" and
"Explicit"-attribute (e.g. a test could be marked with
[Category("ExcludeMySql")] and when running with the mysql vendor,
that category would be excluded). That way only one test-assembly
would be needed and I think it would perhaps be easier to read those
annotations. But that is another discussion.
> 3. Please do NOT skip tests that work for everything. Instead, place
> them into a separate section ("Works for everything") so that we can
> easily specify what is actually supported. Otherwise we're left with
> saying "everything that doesn't work is supported", which isn't strictly
> true (we might not have written a test for it yet).
Okay, I just didn't think that table was so self-explanatory or
readable (there will be several tests for each "feature") that it
served as a feature list. I viewed it more as a todo-list and for that
purpose it seemed useful to suppress "completed" tasks, also it helped
keep the page under 200KB allowed by the wiki. Two pages perhaps.
--
Anders
--
You received this message because you are subscribed to the Google Groups
"DbLinq" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/dblinq?hl=en.
