On Mon, 2010-03-22 at 07:38 -0700, Anders wrote:
> I have been trying to get the unit tests to run for all vendors in
> order to update the "Tests" page on the wiki.
Yay!
While you're at it, please update the Unit Tests wiki page with
instructions on how to setup & configure the databases you're hitting:
http://groups.google.com/group/dblinq/web/unit-tests
> - In the repository there are both an "examples" and a "tests" folder,
> both containing sql scripts for creating a northwind database for each
> database vendor. It seems that it is the scripts from the "examples"
> folder which should be used for running the unit tests (contrary to
> what I would expect from the naming and contrary to what the "Unit
> Tests" page in this groups says), since e.g. for the scripts in the
> "tests" folder:
It's a matter of "parallel evolution" (and me being an idiot).
And Microsoft doing The Right Thing.
Short, short version: IGNORE the SQL scripts within 'examples', and use
the ones in 'tests' (if possible, and you later mention a number of
issues with this...).
I wasn't around when these were created, but (apparently) in the
meantime Microsoft released BOTH the Northwind schema AND the dataset
under the Ms-Pl, which is suitably free. This allows us to reasonably
"port" BOTH the schema AND the dataset to multiple databases (see
tests/instnwnd.*.sql).
Compare this to e.g.
examples/DbLinq.Pgsql.Example/sql/createDB_Northwind_pg.sql, where the
schema is recreated but NOT the same dataset.
The result is that, under the 'examples' approach, tests couldn't assert
~anything about the returned content (how many rows should be returned,
what the rows should contain, etc.) as every database could have
different contents...which is a nightmare, frankly. A recurring problem
was that we would have tests in name only: they'd perform a query, cause
DbLinq to generate some SQL and send it to the DB (all good), but the
tests wouldn't assert anything about the data returned.
For example, ReadTest.C6_NullParentEmplyee():
// PC patch: I get 4 results...
Assert.IsTrue(list.Count >= 3);
ReadTest.D14_ProjectedProductList()
Assert.IsTrue(list.Count > 0);
etc., etc. Sure, we get data back, but is it the *right* data?
Often times, NO, it isn't. Thus, in many cases the tests provided a
false sense of security.
If you don't have consistent data, you can't have consistent results,
and with the examples/*/*.sql regime this wasn't assured.
(Again, likely because I was an idiot, saw "Northwind", and ~immediately
installed the Microsoft Northwind dataset on my local server instead of
checking to see what examples did. The perils of ~no documentation.)
> - - the script for the Oracle vendor will create all uppercase table
> names, which will make the tests fail, since the tests uses Pascal
> case, also the username/password in the script doesn't match the
> credentials used in the ConnectionStrings.xml.
Indeed. I'm not sure what e.kotlyarov was thinking when he wrote the
tests/instnwnd.Oracle.sql file. Looks like someone needs to fix it.
:-/
> - - the script for the Ingres vendor will create a database where
> updates fails (perhaps because of data types that are unsupported by
> the data provider).
Likely another case of someone "porting" the SQL without actually
testing anything.
Joy.
:-(
Comparing tests/instnwnd.Ingres.sql to
examples/DbLinq.Ingres.Example/sql/create_Northwind.sql implies that the
major problem is due to lack of explicit sequencing (`CREATE
SEQUENCE ...`). Presumably a simple fix (saying the guy who has no
idea.)
> - - the script for the MySql vendor fails since the data is inserted
> in the wrong order (so foreign keys prevents the data for being
> inserted), tables are being dropped in the wrong order, views aren't
> being dropped, and dates are in the wrong format ('mm/dd/yyyy' instead
> of 'yyyy-mm-dd' see
> http://dev.mysql.com/doc/refman/5.1/en/date-and-time-types.html).
> So what is the scripts in the "tests" folder used for?
Yet another case of "Garbage In/Garbage Out" (aka "why do people bother
writing things if they're not going to test it?!").
> - I have created a small application "TestReport" (for lack of better
> name) in the "Tools" folder, that replaces the DbLinq-NUnit-test-
> results.xlsx spread sheet, such that the Office software is not
> required and no copy/paste is required (it searches for placeholders
> in the wiki-file instead). It also includes test for more vendors (all
> except the Sqlite_test_mono_strict since it fails for almost all tests
> on my setup) and for the non-vendor tests, and it suppresses tests
> that succeeds for all vendors. Would it be alright for me to check-in
> this change?
Please check in the tool.
That said, three points worth mentioning:
1. How are you running DbLinq.Sqlite_test_mono_strict.dll and what
errors are you getting? It's intended to be run under Mono >= 2.6
(NOT .NET). (If you look at the assembly references, you'll see that it
does NOT reference DbLinq.dll, but ~only System.Data.Linq.dll.)
I get 5 errors when running under Mono trunk, ~all due to Mono errors
(which I *think* I've filed, but can't remember).
(Under Mono 2.4, ~everything fails; don't bother. Under .NET,
~everything should fail, as .NET L2SQL can't connect to a SQLite DB.)
2. You *MAY* want to consider an alternate table-generation approach.
In an "ideal" world, all "known-bad" tests will have [Exclude]
attributes on them in the "release" build. (This is true for SQL Server
and SQLite, and may be true for PostgreSQL.) If this is the case for
all databases, generating the pass/fail table becomes a simple exercise
in Reflection -- any test with an [Explicit] attribute is a FAIL, and
anything without the attribute is a PASS. Just reflect over the
assemblies, and you're done.
Generating a table in this fashion should be fast/trivial, but also
requires that the [Explicit] attributes be kept in-sync so that Release
builds are always "green" (which is certainly good...if we had actual
maintainers for those databases, otherwise it's just overhead.)
3. Please do NOT skip tests that work for everything. Instead, place
them into a separate section ("Works for everything") so that we can
easily specify what is actually supported. Otherwise we're left with
saying "everything that doesn't work is supported", which isn't strictly
true (we might not have written a test for it yet).
Thanks,
- Jon
--
You received this message because you are subscribed to the Google Groups
"DbLinq" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/dblinq?hl=en.
