Hi Greg
First, I have to apologize for sending this and then not following up. I
actually sent it initially using google groups, since accessing NNTP
from behind the company firewall isn't easily possible. Strangely my
post didn't seem to turn up when the list was viewed via google groups.
I had assumed my post didn't make it onto the group, and in fact I sent
another one just yesterday. Then when I checked with a real NNTP client
I found my old post! So please ignore the post from yesterday, if it
shows up at all.
>
> What exactly is the slow part? DBD::Pg uses XS extensively as well,
> so it's good to know about real world limitations where the Perl
> bumps up against the C. Maybe we can fix that part instead of having
> to tie yet another language into the mix.
>
The application has grown over the years, to the point where there are
now a number of installations, each handling 100's of thousands of files
a day, parsing their content, processing and storing in the database.
The database size is in the 10's of TB, and can be processing many GB
per day. There are a number of transformations and operations performed
on the data before putting it in the database, and as it has grown we
have found that a pure perl solution has not been able to keep up.
I doubt you could do much to DBD::Pg to help with this.
> My first instinct is to make the version a requirement somehow.
> Not knowing your system, I'm guessing that's problematic.
Yes, we are doing that at the moment, using RPM dependancies. However it
makes it difficult to accomodate additional host OS variants, and there
is also the danger that an admin at one of the sites might install
another version of DBD::Pg from source.
> At first blush, I don't see any reason to deny that. It's just
> hooking to the underlying plumbing, there is no escalation of
> privs or anything. And we could certainly keep it as a
> super-advanced, here-be-dragons kind of attribute.
OK, I will try some experiments to see if I can come up with a workable
solution.
In the other post I attempted to make yesterday I suggested three
possible methods:
1. Allow access to the PGconn ptr from perl as an attribute or via a
method
2. Provide a C accessor function which can be called externally from
another XS module
3. Include a header file with DBD::Pg which defines access macros. Of
course I didn't think that through too well as it really does not solve
the problem, so perhaps forget that.
My gut feel is to go with option #2, since the raw PG connection is not
likely to be that useful from Perl and it allows us to have our module
accept a SV* dbi handle as an argument in a transparent way.
Regards
Mike
