Re: DBI::Test's DESIGN.md
Am 01.07.2013 um 23:55 schrieb Tim Bunce tim.bu...@pobox.com:
On Mon, Jul 01, 2013 at 04:45:37PM +0200, Jens Rehsack wrote:
Hi all,
I decided it's best to discuss it in public than in a github repository.
We can easier answer / quote / split discussions than in a push/pull battle.
# DESIGN
Currently this is a list of open issues and discussion points...
Topics can be removed once they're settled and the relevant docs have been
updated.
## DBI::Test as a DBD author's tool
This is the principle use-case for DBI::Test: to provide a common suite of
tests for multiple drivers.
Currently documented as
--- BEGIN
This module aims at a transparent test suite for the DBI API
to be used from both sides of the API (DBI and DBD) to check
if the provided functionality is working and complete.
--- END
Does it satisfy that point?
The phrase used from both sides of the API seems unclear to me.
Perhaps use this simpler wording:
This module aims to be a test suite for the DBI API and
an underlying DBD driver, to check
if the provided functionality is working and complete.
Okay - taken and extended a bit (hope it's fine).
We need to consider how evolution of DBI::Test will affect driver authors.
Specifically, as DBI::Test add new tests it's quite likely that some drivers
will fail that test, but that failure is not a regression for the driver.
Well - from my point of view, it is. A lot of drivers (including those
coming with DBI) doesn't conformDBI API in a lot of cases. Step by
step (finally Tux) discovers them by enhancing DBD::CSV tests.
Sure, but that's not a regression in the technical sense of being
previously working but now broken. It's just uncovered an old bug.
Well, agreed. Don't know if it was every working, but it definitively
don't now should probably be an AUTHOR case at first.
The men from Oslo.pm reported that there're some famous DBD's out there
(commercial DB DBD's) which have nearly no suitable test. Probably they
wouldn't install anymore for a long time if the authors can't choose the
scope of validation.
So it seems reasonable for DBI::Test to be primarily a developer tool
and not run as a standard part of the drivers' test suite, at least for now.
In other words, DBI::Test would only be run if AUTHOR_TESTING is true.
That's not designed for being an AUTHOR_TESTING tool. It's designed for
exactly the opposite - to allow authors detecting issues in a very wide
range of situations.
I see this as a period of transition. AUTHOR_TESTING first, then
AUTOMATED_TESTING (cpantesters), then a full dependency.
The rate of progress through those steps should be up to the
individual driver authors. In fact it simply has to be.
So DBI::Test should aim to make progress through those steps
as smooth and rewarding as possible. Low cost, high benefit.
Otherwise driver authors are less likely to make the effort.
How about the developers can choose and DBI::Test will have
a default for waverers (the way you recommended).
As far as I understand the test protocol and toolchain, AUTHOR_TESTING
isn't set by default by anyone. How will any author test ever be
executed?
AUTOMATED_TESTING is set by the toolchain (?) and smoke bots.
All we need to do is generating an appropriate preamble.
$ENV{AUTHOR_TESTING} or notest = We want that so, dude.
Choose DBD::DBM as first example: [...]
Second example: DBD::ODBC [...]
On design goal of DBI::Test is to manage those differences [...]
Of course, that's all very good and I agree.
Consider a Random User runing cpanm DBD::Foo to install DBD::Foo.
My point is ONLY that that user shouldn't get a failure simply because a
new DBI::Test release has a new test that causes DBD::Foo to fail
because of an old bug.
Heh - and /I/ want this bug - because that's probably an edge case
/I/ didn't test. But I'm 100% sure we're talking about different
kind of bugs. You're probably talking about
+---+
| DBI::Test |
+---+
|
+--+--+
| |
\/\/
+--+ ++
| DBD::DBM | | SQL::Statement |
+--+ ++
When you update/install DBD::DBM, and it fails because of a new
Test in DBI::Test, we made a mistake. We released new DBI::Test
without proving with dev-releases and DBI / S::S tests, how it
works.
I can see only 2 reasons for such a test fail:
1) We found a new old bug, which needs to be uncovered
== the failure is probably wanted (we could make the
AUTHOR/ AUTOMATED/...-testing default per test case,
couldn't we?)
2) We made a mistake by introducing a test which is bad
== the failure points to invalid self-test of DBI::Test
That also allows us to duck the issue of whether DBD's should list
DBI::Test as
a dependency. At least for now.
DBI lists DBI::Test as dependency. That's why we're having a lot of
fun when doing common stuff in a way
Re: DBI::Test's DESIGN.md
On Tue, 2 Jul 2013 09:44:06 +0200, Jens Rehsack rehs...@gmail.com wrote: For the moment, I wouldn't spend to much brain on those cases, as they do not affect any existing and targeted driver. Primary goal for the first shot are: * DBI (and bundled drivers) * SQL::Statement (with DBI::Mock extension) * DBD::CSV * DBD::Pg PostgreSQL is now my primary target when I develop or test database related modules or scripts. It is a nice opponent to DBD::CSV. Pg is fast, complete and actively maintained, written from the database point-of-view using DBI. DBD::CSV is slow, incomplete (due to underlying restrictions) and written from the DBI point of view. The two have completely different routes from database to user, which is why these should make a neat couple. DBD::Pg *only* needs DBI, whereas DBD::CSV requires DBI, Text::CSV_XS, SQL::Statement, and DBD::File (in DBI) which implies more possible points of failure. * DBD::SQLite * DBD::Oracle * DBD::DB2 * DBD::Unify I have no plans to do serious work on that driver. We are moving away from that database which means that tuits will be hard to find to improved that one. DBD::Unify however took a basic approach in its initial test suite to just take the DBI docs and implement sequentially all the methods described by the time of writing, so the test suite was based on what DBI provides, not on what Unify offers. When tests failed, I wrote the corresponding DBD::Unify part or documented that that functionality would not work in Unify. So I have no objection if parts of those tests are copied or stolen from to complete the DBI::Test suite * DBIx::Class # ribasushi has some dirty ideas … ask him ^^ o DBD::Sys o DBD::AnyData (if any volunteer takes on it or I find a tuit) -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
DBI::Test's DESIGN.md
Hi all, I decided it's best to discuss it in public than in a github repository. We can easier answer / quote / split discussions than in a push/pull battle. # DESIGN Currently this is a list of open issues and discussion points... Topics can be removed once they're settled and the relevant docs have been updated. ## DBI::Test as a DBD author's tool This is the principle use-case for DBI::Test: to provide a common suite of tests for multiple drivers. Currently documented as --- BEGIN This module aims at a transparent test suite for the DBI API to be used from both sides of the API (DBI and DBD) to check if the provided functionality is working and complete. --- END Does it satisfy that point? We need to consider how evolution of DBI::Test will affect driver authors. Specifically, as DBI::Test add new tests it's quite likely that some drivers will fail that test, but that failure is not a regression for the driver. Well - from my point of view, it is. A lot of drivers (including those coming with DBI) doesn't conformDBI API in a lot of cases. Step by step (finally Tux) discovers them by enhancing DBD::CSV tests. Where possible, I merge those tests to SQL::Statement and/or DBD::DBM or DBD::File etc. To avoid such merge battles, we decided to start DBI::Test. And I really think, a newly introduced test showing a misbehavior should be a regression. Thinking about rt#84260, it definitively should. So it seems reasonable for DBI::Test to be primarily a developer tool and not run as a standard part of the drivers' test suite, at least for now. In other words, DBI::Test would only be run if AUTHOR_TESTING is true. That's not designed for being an AUTHOR_TESTING tool. It's designed for exactly the opposite - to allow authors detecting issues in a very wide range of situations. Choose DBD::DBM as first example: Our standard test case run's it using SDBM_File - regardless any other available backend. Typical Linux use case would be GDBM_File as backend, other Unices might provide NBDM_File. Progressive users might have BerkeleyDB installed and wants to know (during the update o DBI) if everything went ok. Second example: DBD::ODBC - I don't know if anyone always tested DBD::ODBC using a Gofer Proxy. Or DBD::Proxy or DBD::Multiplex. DBI::Test would allow doing exactly that by using plugins for test case configuration and test variants configuration (it doesn't makes sense to do a DBI::PurePerl test with DBD::ODBC or DBD::Oracle, but it does for DBD::CSV when using Text::CSV instead of Text::CSV_XS). On design goal of DBI::Test is to manage those differences and allow better testing and reporting of configuration edge cases. That also allows us to duck the issue of whether DBD's should list DBI::Test as a dependency. At least for now. DBI lists DBI::Test as dependency. That's why we're having a lot of fun when doing common stuff in a way not introducing unnecessary dependencies (read: almost none except core). ## DBI::Test as a DBI developer's tool The goal here would be to test the methods the DBI implements itself and the services the DBI provides to drivers, and also to test the various drivers shipped with the DBI. This is a secondary goal, but is important because DBI::Test will probably become the primary test suite for the drivrs that ship with DBI. If I understood those sentences correctly, this is the primary goal. ## Define what DBI::Test is NOT trying to do * It's not trying to test the database SQL behaviour (ORDER BY, JOINs etc). Databases (an drivers that implement their own databases) should have their own test suite for that. Yes and no. It doesn't provide tests for that by default. But it would allow to create a CPAN distribution DBI::Test::ANSI_SQL and deploy test cases for that. Anyone who has installed those module might have additional tests when running make test. This is wanted. * It's not trying to test the database SQL syntax. As many tests as possible should be usable even for databases that don't use SQL at all. Ehm … AFAIK we decided that otherwise. But it should be easy (or at least, could be easily made possible) to improve the match-check to catch only SQL based drivers for SQL tests. This is a job for the match = sub :) ## List some minimum and other edge cases we want to handle Example: Using the DBM with SQL::Nano parser. This means that, as far as possible, all tests should use very simple SQL and only one or two string columns. Nano can do as much as columns you want, but yes. The goal is, that SQL::Statement provides own DBI::Test plugins. And a configuration plugin for DBI::Test catching any DBI::DBD::SqlEngine based DBD's and create additional test setups for them. Read: When DBI is installed after SQL::Statement, DBI's test suite is enhanced by the SQL::Statement tests for DBI::DBD::SqlEngine, DBD::File and DBD::DBM - if SQL::Statement becomes installed after DBI, it's vice versa. Obviously
Re: DBI::Test's DESIGN.md
On Mon, Jul 01, 2013 at 04:45:37PM +0200, Jens Rehsack wrote: Hi all, I decided it's best to discuss it in public than in a github repository. We can easier answer / quote / split discussions than in a push/pull battle. # DESIGN Currently this is a list of open issues and discussion points... Topics can be removed once they're settled and the relevant docs have been updated. ## DBI::Test as a DBD author's tool This is the principle use-case for DBI::Test: to provide a common suite of tests for multiple drivers. Currently documented as --- BEGIN This module aims at a transparent test suite for the DBI API to be used from both sides of the API (DBI and DBD) to check if the provided functionality is working and complete. --- END Does it satisfy that point? The phrase used from both sides of the API seems unclear to me. Perhaps use this simpler wording: This module aims to be a test suite for the DBI API and an underlying DBD driver, to check if the provided functionality is working and complete. We need to consider how evolution of DBI::Test will affect driver authors. Specifically, as DBI::Test add new tests it's quite likely that some drivers will fail that test, but that failure is not a regression for the driver. Well - from my point of view, it is. A lot of drivers (including those coming with DBI) doesn't conformDBI API in a lot of cases. Step by step (finally Tux) discovers them by enhancing DBD::CSV tests. Sure, but that's not a regression in the technical sense of being previously working but now broken. It's just uncovered an old bug. So it seems reasonable for DBI::Test to be primarily a developer tool and not run as a standard part of the drivers' test suite, at least for now. In other words, DBI::Test would only be run if AUTHOR_TESTING is true. That's not designed for being an AUTHOR_TESTING tool. It's designed for exactly the opposite - to allow authors detecting issues in a very wide range of situations. I see this as a period of transition. AUTHOR_TESTING first, then AUTOMATED_TESTING (cpantesters), then a full dependency. The rate of progress through those steps should be up to the individual driver authors. In fact it simply has to be. So DBI::Test should aim to make progress through those steps as smooth and rewarding as possible. Low cost, high benefit. Otherwise driver authors are less likely to make the effort. Choose DBD::DBM as first example: [...] Second example: DBD::ODBC [...] On design goal of DBI::Test is to manage those differences [...] Of course, that's all very good and I agree. Consider a Random User runing cpanm DBD::Foo to install DBD::Foo. My point is ONLY that that user shouldn't get a failure simply because a new DBI::Test release has a new test that causes DBD::Foo to fail because of an old bug. That also allows us to duck the issue of whether DBD's should list DBI::Test as a dependency. At least for now. DBI lists DBI::Test as dependency. That's why we're having a lot of fun when doing common stuff in a way not introducing unnecessary dependencies (read: almost none except core). I don't see DBI::Test listed in Makefile.PL. ## DBI::Test as a DBI developer's tool The goal here would be to test the methods the DBI implements itself and the services the DBI provides to drivers, and also to test the various drivers shipped with the DBI. This is a secondary goal, but is important because DBI::Test will probably become the primary test suite for the drivrs that ship with DBI. If I understood those sentences correctly, this is the primary goal. I see sharing a common test suite among multiple drivers as the primary goal. Becoming a test suite for the DBI itself is a very useful side-effect. But we're probably splitting hairs - there's not much value in debating primary vs secondary goals as they're both important :) ## Define what DBI::Test is NOT trying to do * It's not trying to test the database SQL behaviour (ORDER BY, JOINs etc). Databases (an drivers that implement their own databases) should have their own test suite for that. Yes and no. It doesn't provide tests for that by default. But it would allow to create a CPAN distribution DBI::Test::ANSI_SQL and deploy test cases for that. Anyone who has installed those module might have additional tests when running make test. This is wanted. I can see value in a generic extension mechanism. * It's not trying to test the database SQL syntax. As many tests as possible should be usable even for databases that don't use SQL at all. Ehm … AFAIK we decided that otherwise. But it should be easy (or at least, could be easily made possible) to improve the match-check to catch only SQL based drivers for SQL tests. This is a job for the match = sub :) I suspect that's going to be limited, but I'm happy to be proved wrong. ## List some minimum and other
Re: DBI::Test's DESIGN.md
Apologies for top posting and I'll give this post the attention it deserves later (by all means track me down Jens if I forget) but one thing struck me immediately and that was sections in DBD pod like DBD::ODBC's deviations from the DBI specification at http://search.cpan.org/~mjevans/DBD-ODBC-1.43/ODBC.pm#Deviations_from_the_DBI_specification Some of these might be really difficult to deal with in test code. Over the years I've submitted many patches to tighten the DBI specification but at the end of the day having maintained a DBD for quite a while deviations from the specification are just reality. Trying to make all DBDs the same is a really difficult task which DBI does well for the most part but I keep coming up with new cases - just search in dbi-dev for threads containing clarification from me. In particular see last_insert_id which DBD::ODBC has almost no chance of supporting. As an aside the err, native and errstr are still a continual problem for some of the work I do in that there is only one per DBI and they are not per stmt - contrary to ODBC where the native error and error string is per stmt, dbc or env. Yes, I could have done something about this but it always seemed harder to address than it first seemed since it is engrained in DBI since day one. Or my memory could be worse than I think and Tim will correct me. Martin On 01/07/2013 22:55, Tim Bunce wrote: On Mon, Jul 01, 2013 at 04:45:37PM +0200, Jens Rehsack wrote: Hi all, I decided it's best to discuss it in public than in a github repository. We can easier answer / quote / split discussions than in a push/pull battle. # DESIGN Currently this is a list of open issues and discussion points... Topics can be removed once they're settled and the relevant docs have been updated. ## DBI::Test as a DBD author's tool This is the principle use-case for DBI::Test: to provide a common suite of tests for multiple drivers. Currently documented as --- BEGIN This module aims at a transparent test suite for the DBI API to be used from both sides of the API (DBI and DBD) to check if the provided functionality is working and complete. --- END Does it satisfy that point? The phrase used from both sides of the API seems unclear to me. Perhaps use this simpler wording: This module aims to be a test suite for the DBI API and an underlying DBD driver, to check if the provided functionality is working and complete. We need to consider how evolution of DBI::Test will affect driver authors. Specifically, as DBI::Test add new tests it's quite likely that some drivers will fail that test, but that failure is not a regression for the driver. Well - from my point of view, it is. A lot of drivers (including those coming with DBI) doesn't conformDBI API in a lot of cases. Step by step (finally Tux) discovers them by enhancing DBD::CSV tests. Sure, but that's not a regression in the technical sense of being previously working but now broken. It's just uncovered an old bug. So it seems reasonable for DBI::Test to be primarily a developer tool and not run as a standard part of the drivers' test suite, at least for now. In other words, DBI::Test would only be run if AUTHOR_TESTING is true. That's not designed for being an AUTHOR_TESTING tool. It's designed for exactly the opposite - to allow authors detecting issues in a very wide range of situations. I see this as a period of transition. AUTHOR_TESTING first, then AUTOMATED_TESTING (cpantesters), then a full dependency. The rate of progress through those steps should be up to the individual driver authors. In fact it simply has to be. So DBI::Test should aim to make progress through those steps as smooth and rewarding as possible. Low cost, high benefit. Otherwise driver authors are less likely to make the effort. Choose DBD::DBM as first example: [...] Second example: DBD::ODBC [...] On design goal of DBI::Test is to manage those differences [...] Of course, that's all very good and I agree. Consider a Random User runing cpanm DBD::Foo to install DBD::Foo. My point is ONLY that that user shouldn't get a failure simply because a new DBI::Test release has a new test that causes DBD::Foo to fail because of an old bug. That also allows us to duck the issue of whether DBD's should list DBI::Test as a dependency. At least for now. DBI lists DBI::Test as dependency. That's why we're having a lot of fun when doing common stuff in a way not introducing unnecessary dependencies (read: almost none except core). I don't see DBI::Test listed in Makefile.PL. ## DBI::Test as a DBI developer's tool The goal here would be to test the methods the DBI implements itself and the services the DBI provides to drivers, and also to test the various drivers shipped with the DBI. This is a secondary goal, but is important because DBI::Test will probably become the primary test suite for the drivrs that ship with DBI. If I understood those sentences
Re: DBI::Test's DESIGN.md
Am 02.07.2013 um 00:56 schrieb Martin J. Evans boh...@ntlworld.com: Apologies for top posting and I'll give this post the attention it deserves later (by all means track me down Jens if I forget) but one thing struck me immediately and that was sections in DBD pod like DBD::ODBC's deviations from the DBI specification at http://search.cpan.org/~mjevans/DBD-ODBC-1.43/ODBC.pm#Deviations_from_the_DBI_specification Some of these might be really difficult to deal with in test code. Over the years I've submitted many patches to tighten the DBI specification but at the end of the day having maintained a DBD for quite a while deviations from the specification are just reality. Trying to make all DBDs the same is a really difficult task which DBI does well for the most part but I keep coming up with new cases - just search in dbi-dev for threads containing clarification from me. In particular see last_insert_id which DBD::ODBC has almost no chance of supporting. As an aside the err, native and errstr are still a continual problem for some of the work I do in that there is only one per DBI and they are not per stmt - contrary to ODBC where the native error and error string is per stmt, dbc or env. Yes, I could have done something about this but it always seemed harder to address than it first seemed since it is engrained in DBI since day one. Or my memory could be worse than I think and Tim will correct me. That's all right, and even more worth finding a good solution for overall specification testing. But that (even if quick read Tim's reply before going to be convinced me) takes exactly the same line as Tim's suggestion for AUTHOR_TESTING … I will make a proposal in the reply of Tim, hope that doesn't hurt you :) Cheers -- Jens Rehsack rehs...@gmail.com
