On 25/09/13 17:28, Tim Bunce wrote:
Hi all.
This email aims to start a discussion around clarifying the goals, scope
and requirements of a Common DBI Driver Test Suite, including learning
from the great work already done on DBI::Test.
Once we've kicked these around to the point were there's reasonable
agreement, then we'll start a new thread to discuss design/implementation.
For this thread though, goals, scope, and requirements are the focus.
First, let's spell out some basics...
Stakeholders: (who is affected by this work)
- driver developers
- cpantesters
- end users
Terminology:
* We'll use DBIT as the generic term for this thing, whatever it
turns out to look like or be called.
Goals: ("how will we know if this project is or is not a success")
G1. driver developers improve their compliance with the DBI API
and so improve consistency and portability of code.
This is what it's all about!
G2. driver developers adopt DBIT as a free test suite with good
coverage of the DBI API. This is the pay-back _for_ developers.
I think one thing many DBD test suites could benefit from is wrappers around many
of the DBI methods that wrap that method in tests e.g., when execute is called,
did it return undef, a true value (0E0 or a number > 0) or -1 depending on
whether it is a select statement or not. If test suites were converted to use
those I'm sure we'd find quite a few issues in DBDs but still using existing test
code.
G3. driver developers write and share reusable generic tests
(they'll still need to write their own driver-specific tests).
This is the pay-back _from_ developers.
G4. end users won't be affected by DBIT evolution causing install failures
i.e., DBIT runs as 'author testing', *at least for now*.
This is our promise to end users not to ruin their day.
G5. be usable for the widest range of DBI drivers, including non-SQL.
This is a promise to be inclusive.
G6. enable testing with proxies like DBD::Gofer.
This is a promise to be flexible.
G7. enable testing with DBI::PurePerl for pure-perl drivers.
This is a promise to support fatpackers and perl6 ;)
G8. Provide tools that drivers can use/extend for themselves.
I'm thinking specifically here of the creation of test files
with combinations of env vars and other settings.
E.g., test DBD::CSV with Text::CSV and Text::CSV_XS
G9. end users can find the level of DBI API compliance of a driver
i.e., by looking at the test configuration for the driver
to see what areas of functionality are configured to be skipped.
This in particular is something I'd like to see and expand on.
As someone who has attempted to use DBI to multiple DBD backends (and even
ignoring SQL differences) the biggest stumbling block is differences in DBDs
either outside of the DBI specification or because DBI allows a driver to not
implement something or allows a DBD multiple options. Just ask Peter
(ribasushi) how much code is in DBIx::Class to work around driver differences.
I've long thought (and I've a feeling proposed - but that might have just been
in discussions with Peter over a pint) a capability system beyond what get_info
provides. get_info is pretty much ODBC's SQLGetInfo and few drivers beyond
DBD::ODBC really support it that well. Just off the top of my head I'm thinking
about these sorts of differences:
a) does it support named parameters
b) does it need a ':' preceding named parameters or not
c) can named parameters be used multiple times in the same SQL
d) does it use :N for placeholders or ?
e) lobs - DBIs' lob read is not documented and so all DBDs do it a different way
f) returning cursors from functions or procedures
g) output parameters from procedures and functions and when they are available
(more_results is another one here)
h) DBIx::Class (used by many) and the work it has to do in the background to
compensate for differences in DBDs.
i) does it have its own execute_for_fetch - only implemented in 2 DBDs I know
of and severely restricted in DBD::ODBC due to ODBC driver differences.
j) can it support unicode
k) how do you enable unicode, sqlite_unicode etc
l) is last_insert_id usuable - not in some DBDs
m) can you safely put comments in SQL - even comments including ? or :param
n) does a do create a statement handle, i.e, is do the same as
$dbh->prepare->execute
o) can you rebind a placeholder with a different type
p) does a disconnect commit or rollback
q) does type_info/type_info_all return 19 or 20 columns
r) does it support DiscardString etc
s) does it support setting a type on bind_column
.
.
If I put my mind to it (and looked at my code from years ago when I was
involved in writing to multiple DBDs from the same application) I could proably
come up with a much longer list - Peter probably could too.
I know this is not DBIT as such and you might see it as a distraction (if you
do, ignore) but I think it would be worth while.
[I thought of adding an extra goal: "other code can determine the
limitations/quirks of a driver i.e., by looking at the test
configuration for the driver to see how it's configured." but to be
generic I think that would require a connected $dbh to determine server
version/config. It's worth keeping in mind though in terms of having an
object that represents the server that can be queried for info needed by
tests. Perhaps that'll end up as extensions to $dbh->get_info().]
yes please - see above.
Scope: (the boundaries and deliverables of the project)
S1. the DBIT will be a separate distribution.
S3. the DBI won't have a mandatory dependency on DBIT,
for now at least, driver developers are the priority.
S4. DBIT is not meant to test the underlying database.
If a driver implements a database itself then it'll
need it's own tests to provide good coverage of that.
[Regarding S4: We could enable individual distros to install modules
that add DBIT tests that'll be included in DBIT runs in certain cases,
e.g., for certain drivers. I want to avoid running lots of SQL tests
for drivers where that doesn't make sense. I'm sure we can find a way to
do that but still allow sharing of SQL tests for SqlEngine based DBDs.]
Requirements:
R1. be easy for driver developers to adopt,
so existing drivers migrate to using it.
R2. be easy for driver developers to extend/contribute to,
so driver developers contribute to it.
R3. work well with cpantesters,
so we get good coverage (perhaps extend Test::Database)
As a side note, I as going to add support to Test::Database for DBD::ODBC
because I thought it might get me more smokers actually running the tests. Then
I discovered it needed create database support, and that was not going to
happen with ODBC as each database has totally different syntax for that and
some databases need very high level permissions to create a database.
So, any thoughts on these goals, scope, and requirements?
(Please avoid design and implementation details for now. If you have any
concerns that relate to design and implementation that you feel need to
be considered now, try to express them in terms of the goals, scope, or
requirements that would lead to that design.)
Tim.
These seem like worthwhile goals. Whether I can be of any help, I don't know,
but I'll at least try and keep up with the discussion and provide any useful
feedback.
Martin
