Am 01.07.2013 um 23:55 schrieb Tim Bunce <tim.bu...@pobox.com>:
> On Mon, Jul 01, 2013 at 04:45:37PM +0200, Jens Rehsack wrote:
>> Hi all,
>>
>> I decided it's best to discuss it in public than in a github repository.
>> We can easier answer / quote / split discussions than in a push/pull battle.
>>
>>> # DESIGN
>>>
>>> Currently this is a list of open issues and discussion points...
>>>
>>> Topics can be removed once they're settled and the relevant docs have been
>>> updated.
>>>
>>>
>>> ## DBI::Test as a DBD author's tool
>>>
>>> This is the principle use-case for DBI::Test: to provide a common suite of
>>> tests for multiple drivers.
>>
>> Currently documented as
>>
>> --- BEGIN
>> This module aims at a transparent test suite for the DBI API
>> to be used from both sides of the API (DBI and DBD) to check
>> if the provided functionality is working and complete.
>> --- END
>>
>> Does it satisfy that point?
>
> The phrase "used from both sides of the API" seems unclear to me.
> Perhaps use this simpler wording:
>
> This module aims to be a test suite for the DBI API and
> an underlying DBD driver, to check
> if the provided functionality is working and complete.
Okay - taken and extended a bit (hope it's fine).
>>> We need to consider how evolution of DBI::Test will affect driver authors.
>>> Specifically, as DBI::Test add new tests it's quite likely that some drivers
>>> will fail that test, but that failure is not a regression for the driver.
>>
>> Well - from my point of view, it is. A lot of drivers (including those
>> coming with DBI) doesn't conformDBI API in a lot of cases. Step by
>> step (finally Tux) discovers them by enhancing DBD::CSV tests.
>
> Sure, but that's not a "regression" in the technical sense of being
> "previously working but now broken". It's just uncovered an old bug.
Well, agreed. "Don't know if it was every working, but it definitively
don't now" should probably be an AUTHOR case at first.
The men from Oslo.pm reported that there're some famous DBD's out there
(commercial DB DBD's) which have nearly no suitable test. Probably they
wouldn't install anymore for a long time if the authors can't choose the
scope of validation.
>>> So it seems reasonable for DBI::Test to be primarily a developer tool
>>> and not run as a standard part of the drivers' test suite, at least for now.
>>> In other words, DBI::Test would only be run if AUTHOR_TESTING is true.
>>
>> That's not designed for being an AUTHOR_TESTING tool. It's designed for
>> exactly the opposite - to allow authors detecting issues in a very wide
>> range of situations.
>
> I see this as a period of transition. AUTHOR_TESTING first, then
> AUTOMATED_TESTING (cpantesters), then a full dependency.
>
> The rate of progress through those steps should be up to the
> individual driver authors. In fact it simply has to be.
>
> So DBI::Test should aim to make progress through those steps
> as smooth and rewarding as possible. Low cost, high benefit.
> Otherwise driver authors are less likely to make the effort.
How about the developers can choose and DBI::Test will have
a default for waverers (the way you recommended).
As far as I understand the test protocol and toolchain, AUTHOR_TESTING
isn't set by default by anyone. How will any author test ever be
executed?
AUTOMATED_TESTING is set by the toolchain (?) and smoke bots.
All we need to do is generating an appropriate preamble.
$ENV{AUTHOR_TESTING} or notest => "We want that so, dude."
>> Choose DBD::DBM as first example: [...]
>> Second example: DBD::ODBC [...]
>> On design goal of DBI::Test is to manage those differences [...]
>
> Of course, that's all very good and I agree.
>
> Consider a Random User runing "cpanm DBD::Foo" to install DBD::Foo.
> My point is ONLY that that user shouldn't get a failure simply because a
> new DBI::Test release has a new test that causes DBD::Foo to fail
> because of an old bug.
Heh - and /I/ want this bug - because that's probably an edge case
/I/ didn't test. But I'm 100% sure we're talking about different
kind of bugs. You're probably talking about
+-----------+
| DBI::Test |
+-----------+
|
+------+------+
| |
\/ \/
+----------+ +----------------+
| DBD::DBM | | SQL::Statement |
+----------+ +----------------+
When you update/install DBD::DBM, and it fails because of a new
Test in DBI::Test, we made a mistake. We released new DBI::Test
without proving with dev-releases and DBI / S::S tests, how it
works.
I can see only 2 reasons for such a test fail:
1) We found a new old bug, which needs to be uncovered
==> the failure is probably wanted (we could make the
AUTHOR/ AUTOMATED/...-testing default per test case,
couldn't we?)
2) We made a mistake by introducing a test which is bad
==> the failure points to invalid self-test of DBI::Test
>>> That also allows us to duck the issue of whether DBD's should list
>>> DBI::Test as
>>> a dependency. At least for now.
>>
>> DBI lists DBI::Test as dependency. That's why we're having a lot of
>> fun when doing common stuff in a way not introducing unnecessary
>> dependencies (read: almost none except core).
>
> I don't see DBI::Test listed in Makefile.PL.
Did you checkout the dbi-test branch? To the best of my remembrance
there was a wish of a VIP DBI developer about feature branches ;)
(I only want to show that I'm willing to learn ^^).
> [...]
> But we're probably "splitting hairs" - there's not much value in
> debating primary vs secondary goals as they're both important :)
Taken :)
> [...]
>>> * It's not trying to test the database SQL syntax. As many tests as possible
>>> should be usable even for databases that don't use SQL at all.
>>
>> Ehm …
>> AFAIK we decided that otherwise. But it should be easy (or at least,
>> could be easily made possible) to improve the match-check to catch
>> only SQL based drivers for SQL tests.
>>
>> This is a job for the match => sub :)
>
> I suspect that's going to be limited, but I'm happy to be proved wrong.
For the moment, I wouldn't spend to much brain on those cases,
as they do not affect any existing and targeted driver.
Primary goal for the first shot are:
* DBI (and bundled drivers)
* SQL::Statement (with DBI::Mock extension)
* DBD::CSV
* DBD::SQLite
* DBD::Oracle
* DBD::DB2
* DBD::Unify
* DBIx::Class # ribasushi has some dirty ideas … ask him ^^
o DBD::Sys
o DBD::AnyData (if any volunteer takes on it or I find a tuit)
Yes, I see your point - but I'm old and my glasses … ;)
Is it okay when we move it to some kind of "thinking about that
later"?
>>> ## List some minimum and other edge cases we want to handle
>>>
>>> Example: Using the DBM with SQL::Nano parser.
>>>
>>> This means that, as far as possible, all tests should use very simple
>>> SQL and only one or two string columns.
>>
>> Nano can do as much as columns you want, but yes.
>
> Sure, but a basic key-value store won't, eg DBD::DBM without MLDBM.
Yes, that's true. Getting old ;)
But this is also a good thing the match sub for different test cases
should cover. And also the DSN generator ...
>> The goal is, that SQL::Statement provides own DBI::Test plugins.
>> And a configuration plugin for DBI::Test catching any DBI::DBD::SqlEngine
>> based DBD's and create additional test setups for them.
>>
>> Read: When DBI is installed after SQL::Statement, DBI's test suite
>> is enhanced by the SQL::Statement tests for DBI::DBD::SqlEngine, DBD::File
>> and DBD::DBM - if SQL::Statement becomes installed after DBI, it's
>> vice versa.
>
> Okay.
>
> My point is simply that a very large proportion of the DBI and DBD API
> and behaviour can be tested with statements that involve only two columns.
> So all tests that can be written with two columns should be.
100% agree.
> [...]
>>> ## How and where should database connections be made?
>>>
>>> I think the modules that implement tests should not perform connections.
>>> The $dbh to use should be provided as an argument.
>>
>> This is a must-have. Because we want to generate several test
>> variants for DBD::DBM with several backends (as well as for
>> DBD::CSV).
>
> I'm not sure if you're agreeing with me.
I do - but I don't know how to do it technically.
This is one of the remaining show stoppers I hoped to have an
enlightening meanwhile. I hoped to can steel some of the benefits
of BooK's Test::Database (which can do that stuff) without taking
the drawback.
Getting my mind free yesterday by killing some mobs in Frostgorge
Sound (http://wiki.guildwars2.com/wiki/Frostgorge_Sound) helps …
I would do some kind of DSN generator - for the first step it need's
only spit out suitable stuff for DBI::Test ("DBD::NullP:" - I fake that)
and DBI (NullP, ExampleP, DBD::DBM …). I will get an idea how to
generalize the generator when going forward - and wont release before
there is a general agreement (doesn't need to be perfect, but on the
right way).
As architectural base I'd take the idea from Log::Any - where is
an Adapter class for extending the logging with other backends, but
the Null adapter comes with the framework itself. I think, that could
be a way ...
>>> [...some specific implementation details...]
>>
>> Feature request :P
>> Nice one, but later - let's go out before YAPC::EU!
>> And it's voluntary work … :D
>
> Sure. And I'm happy to help. At the moment, though, I don't know if my
> implementing some of the items above would cause more pain than gain.
I have - regardless what next weeks discover - at least a few hours per
day for coding. This will be more than enough, more important are
ideas for removing show stoppers.
> I just want to be sure we're all helping in roughly the same direction :)
Yes, and even if it doesn't appeal to - I want the same and I'm very
thankful for your support.
> Tim.
--
Jens Rehsack
pkgsrc, Perl5
rehs...@cpan.org
