On Mon, 1 Jul 2013 22:55:11 +0100, Tim Bunce <tim.bu...@pobox.com> wrote:
> On Mon, Jul 01, 2013 at 04:45:37PM +0200, Jens Rehsack wrote: > > Hi all, > > > > I decided it's best to discuss it in public than in a github repository. > > We can easier answer / quote / split discussions than in a push/pull battle. > > > > > # DESIGN > > > > > > Currently this is a list of open issues and discussion points... > > > > > > Topics can be removed once they're settled and the relevant docs have been > > > updated. > > > > > > > > > ## DBI::Test as a DBD author's tool > > > > > > This is the principle use-case for DBI::Test: to provide a common suite of > > > tests for multiple drivers. > > > > Currently documented as > > > > --- BEGIN > > This module aims at a transparent test suite for the DBI API > > to be used from both sides of the API (DBI and DBD) to check > > if the provided functionality is working and complete. > > --- END > > > > Does it satisfy that point? > > The phrase "used from both sides of the API" seems unclear to me. > Perhaps use this simpler wording: > > This module aims to be a test suite for the DBI API and > an underlying DBD driver, to check > if the provided functionality is working and complete. I personally do not see that as clarification or simpler, but if it makes the project clearer to the other people, I don't care about the worden, as long as it makes clear what we aim at. > > > We need to consider how evolution of DBI::Test will affect driver authors. > > > Specifically, as DBI::Test add new tests it's quite likely that some > > > drivers > > > will fail that test, but that failure is not a regression for the driver. > > > > Well - from my point of view, it is. A lot of drivers (including those > > coming with DBI) doesn't conformDBI API in a lot of cases. Step by > > step (finally Tux) discovers them by enhancing DBD::CSV tests. Which is both time-consuming and worthwhile. Each problem detected is likely to improve both sides of the coin eventually > Sure, but that's not a "regression" in the technical sense of being > "previously working but now broken". It's just uncovered an old bug. That is an important distinction we should keep in mind > > > So it seems reasonable for DBI::Test to be primarily a developer tool > > > and not run as a standard part of the drivers' test suite, at least for > > > now. > > > In other words, DBI::Test would only be run if AUTHOR_TESTING is true. > > > > That's not designed for being an AUTHOR_TESTING tool. It's designed for > > exactly the opposite - to allow authors detecting issues in a very wide > > range of situations. > > I see this as a period of transition. AUTHOR_TESTING first, then > AUTOMATED_TESTING (cpantesters), then a full dependency. I - being an author on DBI and sevral DBD's, would like to run the tests *always* on my development box (currently my laptop). Even if it takes more (or considerably more) time. I'd rather see an environmental setting that would temporarily disable the slow tests than have it off by default. Having it off by default on cpan installs is fine with me. Having an option to have it on/of in CPANTESTERS is up to us to decide: I can imaging having it ON by default for DBI::Test itself and OFF in all other environments. Which reminds me that now that the namespace has been freed, we should upload a simple dist that passes to claim it. > The rate of progress through those steps should be up to the > individual driver authors. In fact it simply has to be. Testing from DBI - or the completeness of the tests - should not have to be in sync with the number of tests actually run on the DBD side … > So DBI::Test should aim to make progress through those steps > as smooth and rewarding as possible. Low cost, high benefit. > Otherwise driver authors are less likely to make the effort. … with the above reason > > Choose DBD::DBM as first example: [...] > > Second example: DBD::ODBC [...] > > On design goal of DBI::Test is to manage those differences [...] > > Of course, that's all very good and I agree. > > Consider a Random User runing "cpanm DBD::Foo" to install DBD::Foo. > My point is ONLY that that user shouldn't get a failure simply because a > new DBI::Test release has a new test that causes DBD::Foo to fail > because of an old bug. Fully agree. I see DBI::Test's value more on the DBI developers/developing and DBD developer/developing side than on the end user side, who in the end only cares about a working module with hopefully no real bugs. > > > That also allows us to duck the issue of whether DBD's should list > > > DBI::Test as > > > a dependency. At least for now. random musing: could the DBI test suite be enhanced so that it runs the tests of DBI::Test *if* it is installed and only then? Then the whole dependency question is off. If an end-user has installed DBI::Test, he/she will run the tests, otherwise he/she won't notice the difference. The benefit is still with us, without making a nuisance to the rest of the world. > > DBI lists DBI::Test as dependency. That's why we're having a lot of > > fun when doing common stuff in a way not introducing unnecessary > > dependencies (read: almost none except core). > > I don't see DBI::Test listed in Makefile.PL. > > > > ## DBI::Test as a DBI developer's tool > > > > > > The goal here would be to test the methods the DBI implements itself and > > > the > > > services the DBI provides to drivers, and also to test the various drivers > > > shipped with the DBI. > > > > > > This is a secondary goal, but is important because DBI::Test will probably > > > become the primary test suite for the drivrs that ship with DBI. > > > > If I understood those sentences correctly, this is the primary goal. In hindsight, I don't think that we should aim at DBI::Test being a replacement of the driver test suite for either DBI or DBD::*. It is an additional test set. If however the DBD author has reached a full 100% compliance with DBI, and thus with DBI::Test, then having DBI::Test as a module dependency will enable all the new tests, which could then act as a replacement, but the choice is then completely at the DBD authors side. > I see sharing a common test suite among multiple drivers as the primary > goal. Becoming a test suite for the DBI itself is a very useful > side-effect. > > But we're probably "splitting hairs" - there's not much value in > debating primary vs secondary goals as they're both important :) Agree. I know my brain works not like Jens' brain and not like Tim's brain. I most often want code to play with before it is complete, whereas Jens like the whole picture somewhat complete before filling the gaps. Not much to be done 'bout that. Once I/we have some tests to play with, it would be easier (in my point of view) to just run the tests with e.g. DBD::CSV and add new tests/fixes along the way in baby steps when I have time to play. > > > ## Define what DBI::Test is NOT trying to do > > > > > > * It's not trying to test the database SQL behaviour (ORDER BY, JOINs > > > etc). > > > Databases (an drivers that implement their own databases) should have > > > their > > > own test suite for that. > > > > Yes and no. It doesn't provide tests for that by default. But it would > > allow to create a CPAN distribution DBI::Test::ANSI_SQL and deploy test > > cases for that. > > > > Anyone who has installed those module might have additional tests when > > running "make test". This is wanted. > > I can see value in a generic extension mechanism. That is beyond *my* focus area. > > > * It's not trying to test the database SQL syntax. As many tests as > > > possible > > > should be usable even for databases that don't use SQL at all. > > > > Ehm … > > AFAIK we decided that otherwise. But it should be easy (or at least, > > could be easily made possible) to improve the match-check to catch > > only SQL based drivers for SQL tests. > > > > This is a job for the match => sub :) > > I suspect that's going to be limited, but I'm happy to be proved wrong. I see value but before we have anything useful at all, I simply don't care. > > > ## List some minimum and other edge cases we want to handle > > > > > > Example: Using the DBM with SQL::Nano parser. > > > > > > This means that, as far as possible, all tests should use very simple > > > SQL and only one or two string columns. Where appropriate, yes: because simple tests are easy to debug and we are testing the API, not the complexity. Not that I deny any value in having tests with more than two columns, but still. > > Nano can do as much as columns you want, but yes. > > Sure, but a basic key-value store won't, eg DBD::DBM without MLDBM. > > > The goal is, that SQL::Statement provides own DBI::Test plugins. > > And a configuration plugin for DBI::Test catching any DBI::DBD::SqlEngine > > based DBD's and create additional test setups for them. > > > > Read: When DBI is installed after SQL::Statement, DBI's test suite > > is enhanced by the SQL::Statement tests for DBI::DBD::SqlEngine, DBD::File > > and DBD::DBM - if SQL::Statement becomes installed after DBI, it's > > vice versa. > > Okay. > > My point is simply that a very large proportion of the DBI and DBD API > and behaviour can be tested with statements that involve only two columns. > So all tests that can be written with two columns should be. I see no objection to that wish > > > ## Should we create .t files at all, and if so, how many? > > > > > > There's a need to have a separate process for some test cases, like > > > testing DBI vs DBI::PurePerl. But others, like Gofer (DBI_AUTOPROXY) > > > don't need a separate process. > > > > The question for so many ".t" files doesn't came from the requirement > > of own processes. It came from several perspectives: > > > > 1) make it easier to debug a single test fail > > (ever tried to figure out what really failed in > > SQL-Statement/t/06virtual.t when all possible test variant > > could be run? There're over 1500 tests in one file …) > > 2) make it possible to run as much as possible parallel > > (yes, we expect 18+ Core per chip on Haswell Xeon's and > > up to 8 processor sockets per machine …) > > *nods* > > > > Let's keep the generation of test files for now, but keep in mind the > > > possibility that some 'context combinations' might be handled > > > dynamically in future, i.e., inside the run_test() subroutine. > > > > Maybe or maybe not. You probably can't decide on generation > > if a setting can be clearly undone (like DBI_AUTOPROXY) or not > > (like DBI_SQL_NANO). > > > > Maybe we can improve DBI::DBD::SqlEngine at a point in future to > > have the same flexibility there (stealing the RootClass technology > > from DBI ^^). > > > > Anyway - this can be very tricky and have interesting side effects > > producing a lot of false positives which nobody wants. > > *nods* > > > > ## How and where should database connections be made? > > > > > > I think the modules that implement tests should not perform connections. > > > The $dbh to use should be provided as an argument. > > > > This is a must-have. Because we want to generate several test > > variants for DBD::DBM with several backends (as well as for > > DBD::CSV). > > I'm not sure if you're agreeing with me. neither do I. Some tests will for sure have no $dbh at all when testing from DBI. Many tests will just test if the API is available/implemented from both sides. $dbh often only makes sense when ran on the DBD side. I have no detailed ideas on how to define the $dbh on the DBD side other than maybe a lib/dbi-test-conf.json or somesuch > > > [...some specific implementation details...] > > > > Feature request :P > > Nice one, but later - let's go out before YAPC::EU! > > And it's voluntary work … :D > > Sure. And I'm happy to help. At the moment, though, I don't know if my > implementing some of the items above would cause more pain than gain. > > I just want to be sure we're all helping in roughly the same direction :) > > Tim. -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
