I am proposing a convention for extension, not an extension to core
compliance requirements.
I am also proposing to write at least one wrapper driver that will provide the
functionality against a synchronous DBD.
I imagine that asynchronous DBDs will, at least initially, exist
separately from their
synchronous counterparts, possibly with an extended DBD name conforming to a
standard suffix, like ora_a or ix_a -- a for asynchronous. In much
the same way that
reentrant libraries are suffixed with _r.
On Thu, 9 Sep 2004 00:07:12 -0700, Jonathan Leffler
<[EMAIL PROTECTED]> wrote:
> On Wed, 8 Sep 2004 14:39:45 -0500, David Nicol <[EMAIL PROTECTED]> wrote:
> > Right. And when your driver is providing a DBI interface to one of
> > them, to provide
> > a fileno() function that works, you may need to open a pipe and write
> > to one end of it when you are expecting some data. [...]
>
> We're talking past each other and not connecting. Probably too much
> asynchronous communication between incompatible database systems :-(
Recognizing the problem is the first step :)
> Or we have different plans for what DBI is and/or should be -- I've
> had my disagreements with Tim over this in the past, and can see that
> it may happen again in the future.
>
> I've gone back to what I believe is the first of your postings (27th
> August - though the email header has 28th August just before 01:00 in
> US/Eastern or maybe US/Central time zone; -05:00). It's a long
> posting. In it, you say:
>
> ==> In the application I am currently working on, I want prepare and
> ==> execute to return as soon as they can, even if error reports will get
> ==> deferred, and I would like to know if data is ready before attempting
> ==> to fetch a row, so I don't have to wait for the server to send it.
>
> These perceived requirements obviously colour your opinion on what is
> required - that's OK; Open Source software gets written when it
> scratches someone's itch.
>
> ==> Yes this class of issues can be trivially solved by demanding threading,
> ==> but that does not help when a(n unrealistic?) design constraint limits
> ==> you to one thread.
>
> This worries me. I really think that asynchronous support via
> threading is more sensible. It is also more likely to be widely
> achievable - IMNSHO. However, it is your itch, so you get to lay out
> your desires. However, be aware that not everyone suffers from the
> same sources of irritation as you do.
Supporting asynchronous operation via threading is unnecessary.
With threading, the application author can have one thread
work the database synchronously. End of story.
> ==> If pointed to the simplest TCP-based driver, I will cheerfully
> ==> add this stuff to demonstrate that its practical.
>
> Simple TCP drivers aren't the problem. It is the non-simple drivers
> with closed source where there is *NO* access to the file handles that
> are used internally, or where no file handles are used at all, and
> whether or not they are used is irrelevant since there is no access to
> the underlying transport layers regardless of what is used.
> Demonstrate how it will be done in DBD::Oracle -- I'd prefer it to be
> DBD::Informix, but that's probably a tad too partisan to be fair.
>
> What I don't see in your discussion is how you would scope
> asynchronicity. Is it at the $dbh level - so that the DBI->connect
> method takes a new attribute that says { Asynchronous => 1 }. Or is
> it something that is at the statement level, so $dbh->prepare method
> has to take the flag. Somewhere along the line, you have to indicate
> your desire for asynchronicity -- you cannot add it to everything
> without breaking a lot of happily working applications.
The attribute in my proposal is "Timeout" and its appearance indicates
that the applicaiton author wishes to use an asynchronous mode. I believe
that DBDs crash when presented with atteributes they don't understand, which
if it is true would mean the following would work in general:
eval {
$dbh = DBI->connect($dsn,$username,$password,
{RaiseError => 1, Timeout => 0});
};
$@ and
$dbh = DBI->connect("Async:$dsn",$username,$password,
{RaiseError => 1, Timeout => 0});
when Async is the name of an asynchronous wrapper DBD. I am open to
the idea that having an attribute which the existence of it is what matters
rather than having it set to a true value is too weird for DBI and the attribute
should be renamed to Asynchronous and Timeout would be an additional
attribute that defaults to zero. I find that cumbersome.
> ==> Working out the implementation is not our problem yet (well it is our
> ==> problem if we're maintaining a driver, but it's not our problem if we're
> ==> wearing "Lords Of The DBI Specification" hats)
>
> The Lords of the DBI Specification may NOT mandate features that
> cannot be readily be supported by the vast majority of DBD authors
> without impacting their entire lives for a year or two. It won't
> work. I have zero time to work on DBD::Informix as it is. Putting
> massive obstacles in the way of compliance with a revised
> specification is nonsense.
I am proposing a convention for an optional extension, NOT an extension
of the core compliance requirements.
> I proposed a scroll cursor mechanism a couple of years ago (at least),
> and outlined a DBMS-agnostic fallback mechanism written in pure Perl
> that, although not particularly pretty or fast, would have worked.
> But, I lacked the time to prosecute it through the DBI spec. However,
> it was made clear to me that the feature would not become a part of
> DBI without a fallback mechanism in place that DBI could provide for
> those drivers not capable of providing an override mechanism. I'd be
> happier if you showed how the mechanism would be provided
> automatically for those drivers that cannot provide it themselves --
> then you'll get my blessing. But while the imposition is on the DBD
> authors to jump through your hurdles, you will get resistance all the
> way from me.
With working asynchronous wrapper drivers, DBI could fall back to preferring
a particular wrapper style through configuration or per-platform installation
switches. DBI->connect could be extended to check for the Timeout attribute
and do the fallback-to-wrapper-on-failure trick shown above, internally.
With wokring async wrappers, the pressure is off, until someone decides
to write a DBD::Informix::Async (invoked with a $dsn beginning ix_a:) that
is somehow better than the wrapper.
> > The idea of a common interface is that it provides a standard way for
> > the code that uses the interface to request the feature.
>
> I'm not ready to sign up for that. I disagree with the starting
> point. DBI should enable access to databases, but it should not
> demand that DBD authors write gobs of code to meet requirements that
> their DBMS does not support. I prefer a 'more-or-less lowest common
> denominator' approach. Everyone else seems to prefer a "let's drag
> the whole world through contortions to make everything work everywhere
> approach". Granted, Microsoft does this -- I don't regard that as a
> good model for Open Source projects, though.
I am proposing a convention for an optional extension, to allow
portability between
DBDs that have asynchronous modes. Not an extension to the core DBI compliance
requirements. An optional flashy trick, not a raising of the bar.
> > When a query comes back empty, ready should return 0but_true because
> > there are zero rows to fetch. That's something new.
> >
> > When a driver is an interface around one or more streams, the file
> > descriptor number(s) of
> > that (those) streams can be passed back with the fileno method, which
> > will be a wrapper
> > around the core C<fileno> function. In other situations such as the
> > ones you list, creating
> > a pipe that will have a file descriptor number, just for communication
> > between the driver
> > and the application, is not difficult.
>
> Just for the record, even though Informix ESQL/C uses sockets or TLI
> widgets or pipes or other file descriptors internally, there is no
> access at the programming level to them. You are confusing Open
> Source applications - where the library details are available to you -
> with Closed Source where the library details are not available to you.
> DBD::Informix, and other drivers such as DBD::Oracle, are constrained
> by the API which the closed source company provides.
No I'm not. You are misconstruing me.
> I have no intention whatsoever of providing a bolt-on piece of
> gizmotronnery, opening pipes etc.
I'm not asking you to, unless you want to. An async wrapper will work
perfectly well against any DBD.
> If the DBI default mechanisms provide such support, great. If not,
> then support won't be forthcoming from the current maintainer of
> DBD::Informix -- alternative maintainers are sought but have been for
> about 5 years now with no-one showing any signs of interest.
>
> > Originally I thought, all polling would be done in the ready() method,
> > which would not
> > block. A request was made to provide a file descriptor for use in an
> > external event loop,
> > and this resulted in the extension of the fileno() method.
>
> In my totally biassed and not desparately cooperative opinion, it
> can't be done sensibly for Informix. Or, I don't want to do it. Or
> something along those lines.
Prioritization is good.
> > A fully asynchronous driver obviously doesn't need any CPU from the
> > application, which
> > could result in a situation where ready() would return true but there
> > has never been a
> > request for an event. So, we define the fileno() method in such a way
> > that a fully
> > asynchronous driver can mock-up a ready event to send to applications
> > that are doing
> > event loops, to maintain the interface. In the code I submitted to
> > the list, this was
> > accomplished by writing a lower-case 'a' to the pipe.
> >
> > The C language, AFAIK, does not distringuish between file handles and
> > socket handles.
>
> Nope; you're right - they're just small integers (for varying
> definitions of small - but the values normally fit in a 16-bit signed
> quantity).
>
> > Just to be clear, if my proposal to support nonblocking modes by
> > defining "ready"
> > and "fileno" methods into the DBI specification is accepted, a driver
> > that is providing
> > a deferred asynchronous interface to a database that does not use any
> > sort of file handle in the communication process will have to create
> > an anonymous pipe for the purpose of indicating the desire to be
> > polled to an application with an event loop. A
> > fully asynchronous driver will have to do that too.
>
> Then I'm not ready to accept your proposal. I may lose the battle -
> we have AutoCommit in the product and it still stinks in my opinion,
> so I've lost before, too. However, this could be the one that breaks
> the camel's back. Quite easily.
Did you ever write a wrapper driver that would provide your scrolling cursor
interface for a DBD that does not support it? Would working on that
cheer you up?
> > The semantics of the fileno FD set are as follows:
> >
> > * it MAY be readable at any time
> >
> > * it MAY become unreadable during readiness
> >
> > * it MUST become readable after C<execute>, and remain readable until
> > C<ready> has been called before ready returns a true value
> >
> > * it MUST become readable after C<ready> has returned false, and
> > remain readable until C<ready> has been called to return true.
> >
> > * readability DOES NOT IMPLY readiness
> >
> > * readiness DOES NOT IMPLY readability
>
> I'm not at all sure I understand this. It appears to be English, but
> the meanings of the words aren't (or don't seem to be) the meanings I
> normally associate with them, even in a programming context. You talk
> about an 'FD fileno set' in the singular, but at some point, you need
> to specify what it means in terms of individual members of the set --
> unless you are assuming that the set always has at most one member.
the proposed fileno method returns a reference to an array containing the
integer file descriptor numbers. Select-based event loops, such as the
one in Http::Server::Singlethreaded (my recent work), use these numbers to
tell Select what to examine for readability. It would be reasonable
for a DBD object
that was idle to return a reference to an empty array. It is easily
conceivable that
a deferred-asynchronous driver might be listening to multiple real
streams, and would
want some CPU whenever any of them became readable. That's why it's a set
rather than just one file descriptor number. I expect that fully
asynchronous drivers,
including wrapper drivers, will only use one pipe to indicate readiness
> > The reference model is a defered asynchronous driver to a database connected
> > via a stream. Drivers to databases that do not use streams for the
> > communication,
> > and fully asynchronous drivers, are required to pretend that they are
> > deferred-asynchronmous drivers that connect via a stream, by providing one or
> > more stream descriptors that can be used in an event loop. [...snip...]
>
> Midnight - the witching hour. Time for good little DBD authors to go
> to sleep. (Walk the dog, potty the kids, shut up shop, attempt to get
> some sleep).
>
> I'm sorry this sounds so curmudgeonly -- it probably is.
> Nevertheless, I do feel that you are imposing unnecessary workloads on
> driver writers because you have a particular constraint imposed on you
> that is not especially necessary to the larger world.
Thank you
> If you come up with a proposal that has a viable default mechanism
> that can be provided by DBI and overridden by DBD authors who have
> more capable DBMS APIs, then that's one thing -- you can do that as
> far as I am concerned.
Proposed migration path:
1: agree on interface details, including methods, semantics, driver_name suffix,
DBD subclass naming convention. For instance: is it better to have a
driver_name suffix, or a connect_async connection method, to load the
asynchronous
version of the driver or the wrapper if needed?
2: approve interface as experimental, optional, if-you-really-want-to.
3: write at least one wrapper driver providing async functionality to
synchronous drivers
4: consider bundling connect mode request logic and fallback async
wrappers with DBI
distribution
We can stay at step 3 indefinately, including a connect fallback code example
with the DBD::Async wrapper documentation. I could take us there
without anyone's
approval, but it would be yet another spontaneous self-directed
initative for me, and
I'm working on being a team player
> But if DBD authors have to be overloaded with
> fiendish work just to meet the requirements of an extreme minority, I
> will be obliged to use my (probably extremely limited) powers to veto
> the suggestion. That which is easy is acceptable; that which is hard
> is not.
DBD authors need do no work to support the Async wrapper driver. The
existing DBI compliance of your driver is sufficient. Bogging down synchronous
code, the vast majority, with checks to see if we're in an async mode,
is obviously
wrong. So async drivers will have their own names.
Those who want to -- who have drivers that currently provide nonstandard
asynchronous modes -- might publish conformant asynchronous versions
as DBD::Async::YourDatabase. This naming convention is part of phase 1.
> --
> Jonathan Leffler <[EMAIL PROTECTED]> #include <disclaimer.h>
> Guardian of DBD::Informix - v2003.04 - http://dbi.perl.org
> "I don't suffer from insanity - I enjoy every minute of it."
--
David L Nicol, genius at large
IT Consulting since 1986
