Re: DBD::Oracle - is ora_lob_read mangling BLOB data?
Can you tell us which version of DBD::Oracle you are using. There is
quite a differance between versions when it comes to UTF8 handling.
Milo van der Leij wrote:
I am in the progress of changing some code that uses DBD::Oracle to use the documented
ora_lob_read function instead of the undocumented blob_read, and I suddenly hit the error
Malformed UTF-8 character (unexpected end of string) in length.
As it turns out, Perl's length function returns the number of characters in the
string. If Perl thinks the string contains text and the text is in utf-8, then
this can be different from the number of bytes in the string. I regret to say
that I am very unfamiliar with how Perl handles utf-8, but I am familiar enough
with Oracle to see that treating BLOB data as character data is incorrect.
I see the call to SvUTF8_on in ora_lob_read in Oracle.xs, I see how the value
of NLS_LANG plays in, and I see how it affects both CLOB and BLOB data, so I
understand why I am getting the result I am getting. But I have some questions:
(For someone who understands utf-8 in Perl:) Is there danger in binary data being treated as utf-8,
other than the length function crashing? Is there a workaround? (I see use bytes, but
Perl 5.10 doc for the length function says the internal encoding is variable, and the number
of bytes usually meaningless which scares me.)
(For someone who understands the stuff in Oracle.xs:) Is it even possible to
change ora_lob_read to only call SvUTF8_on for CLOB data and not for BLOB data?
My test case is included below. Connection info will need to be altered before
it runs. Commenting out the line with AL32UTF8 makes the problem go away.
use strict;
use DBI;
use DBD::Oracle qw(:ora_types);
$ENV{'NLS_LANG'} = '_.AL32UTF8';
my $hDB = DBI-connect('dbi:Oracle:TEST', 'SCOTT', 'TIGER');
my $hSQL = $hDB-prepare(qq{
DECLARE
b blob;
BEGIN
dbms_lob.createtemporary(b,true);
dbms_lob.writeappend(b,256,utl_raw.xrange('00','FF'));
:b := b;
END;
}, { ora_auto_lob = 0 });
my $b;
$hSQL-bind_param_inout( ':b', \$b, 0, { ora_type = ORA_BLOB } );
$hSQL-execute;
my $data = $hDB-ora_lob_read( $b, 1, 1024 );
print Read , length($data), bytes.\n;
Re: DBD::Oracle - is ora_lob_read mangling BLOB data?
My apologies. I am seeing this behavior with DBD::Oracle versions 1.17
and 1.22
- Original Message -
From: John Scoles
To: Milo van der Leij
Cc: dbi-dev@perl.org
Subject: Re: DBD::Oracle - is ora_lob_read mangling BLOB data?
Date: Thu, 26 Mar 2009 06:43:32 -0400
Can you tell us which version of DBD::Oracle you are using. There
is quite a differance between versions when it comes to UTF8
handling.
Milo van der Leij wrote:
I am in the progress of changing some code that uses DBD::Oracle
to use the documented ora_lob_read function instead of the
undocumented blob_read, and I suddenly hit the error Malformed
UTF-8 character (unexpected end of string) in length.
As it turns out, Perl's length function returns the number of
characters in the string. If Perl thinks the string contains
text and the text is in utf-8, then this can be different from
the number of bytes in the string. I regret to say that I am
very unfamiliar with how Perl handles utf-8, but I am familiar
enough with Oracle to see that treating BLOB data as character
data is incorrect.
I see the call to SvUTF8_on in ora_lob_read in Oracle.xs, I see
how the value of NLS_LANG plays in, and I see how it affects both
CLOB and BLOB data, so I understand why I am getting the result I
am getting. But I have some questions:
(For someone who understands utf-8 in Perl:) Is there danger in
binary data being treated as utf-8, other than the length
function crashing? Is there a workaround? (I see use bytes,
but Perl 5.10 doc for the length function says the internal
encoding is variable, and the number of bytes usually
meaningless which scares me.)
(For someone who understands the stuff in Oracle.xs:) Is it even
possible to change ora_lob_read to only call SvUTF8_on for CLOB
data and not for BLOB data?
My test case is included below. Connection info will need to be
altered before it runs. Commenting out the line with AL32UTF8
makes the problem go away.
use strict;
use DBI;
use DBD::Oracle qw(:ora_types);
$ENV{'NLS_LANG'} = '_.AL32UTF8';
my $hDB = DBI-connect('dbi:Oracle:TEST', 'SCOTT', 'TIGER');
my $hSQL = $hDB-prepare(qq{
DECLARE
b blob;
BEGIN
dbms_lob.createtemporary(b,true);
dbms_lob.writeappend(b,256,utl_raw.xrange('00','FF'));
:b := b;
END;
}, { ora_auto_lob = 0 });
my $b;
$hSQL-bind_param_inout( ':b', \$b, 0, { ora_type = ORA_BLOB } );
$hSQL-execute;
my $data = $hDB-ora_lob_read( $b, 1, 1024 );
print Read , length($data), bytes.\n;
--
Be Yourself @ mail.com!
Choose From 200+ Email Addresses
Get a Free Account at www.mail.com
Re: ANNOUNCE: IBM DB2 Database Driver for Perl DBI Version 1.7 released
IBM DB2 Database Driver for Perl DBI Version 1.71 has been uploaded on CPAN. Testers are please welcome to test the new features and report the bugs. DBI is an open standard application programming interface (API) that provides database access for client applications written in Perl. DBI defines a set of functions, variables, and conventions that provide a platform-independent database interface. The DBD::DB2 driver works with DBI and a DB2 client to access databases. **New In Release - Retrieving CLOB type gave truncated for DB2 version 9.1fp2 fixed. - 173663 - Segmentation fault unless db2profile is sourced - CPAN - 44505 Download Link http://www.cpan.org/authors/id/I/IB/IBMTORDB2/DBD-DB21.71.tar.gz Installation Instructions for both Linux and Unix and PPM repositories Information http://www-306.ibm.com/software/data/db2/perl/ Support Email Addresses * This release is supported by Open Source Application Development Team * You may also report your bugs via the CPAN resolution tracking system: http://rt.cpan.org/ by searching for module DBD-DB2 * Such bug reports can be sent by email to bug-dbd-...@rt.cpan.org; they also get sent to open...@us.ibm.com, etc. Download DB2 Express-C for free, go to: http://www-01.ibm.com/software/data/db2/express/download.html?S_CMP=ECDDWW01S_TACT=ACDB201 Getting started with DB2 Express-C: http://www.ibm.com/developerworks/wikis/display/DB2/FREE+Book-+Getting+Started+with+DB2+Express-C Thanks, Tarun Pasrija IBM OpenSource Application Development Team India Software Labs, Bangalore (India)
RFC: developing DBD::SQLite
Hello, So, out of my un-paid projects, my promise to take over release management of DBD::SQLite (from the still incommunicado previous owner) has now come to the front of my queue (now that Set::Relation 0.9.0 is out), so I'm now starting to think about it in detail and get to work over the next week or two. In that vein, the first and only major design change I intend to make right from the start is to stop bundling the SQLite library with the DBI driver and so the driver will have that library as an external dependency. While one of the selling points of DBD::SQLite versus other DBI drivers in the past was that it came with everything you need, with the advent of a single file amalgamation library being provided standard from sqlite.org, as well as the increasing availability of the SQLite library as its own shared system library install, I figure it isn't too difficult now for users to either obtain the library separately or use the one that came with their system, or the DBD::SQLite installer could automatically download it similar to how some other projects download their dependencies (Rakudo Perl 6 can download Parrot for example); so I don't think the ease of use of DBD::SQLite is diminished significantly by it no longer bundling the SQLite library. On the other side, there is a lot of benefit gained from not bundling. For one thing the size of the distribution as well as the source control is cut down significantly, since the DBI driver alone is orders of magnitude smaller. For another thing, occasional needs to update for compatibility aside, DBD::SQLite will always be up to date with the newest SQLite library; users just update the library and possibly recompile the DBD::SQLite they already have. And so DBD::SQLite won't need to be updated as often; it would just need updates to address incompatible changes to the SQLite library, or to fix bugs in or update features specific to DBD::SQLite itself. Another quasi-change is that DBD::SQLite will be designed to work specifically with the amalgamation version of the source files only, not the pre-amalgamated versions; I say quasi-change since Audrey Tang already did the work to convert DBD::SQLite to work this way, in the separate ::Amalgamation distro. Compatibility-wise, my DBD::SQLite will endeavour to work with all versions of SQLite 3.y.z, though note that only 3.4.0 for which the amalgamation file was a distinct download on sqlite.org (and 3.3.14 or so was the first that amalgamation was a make target). Or more specifically, I only plan to test with the latest SQLite source library available at the time (3.6.11 currently), as well as probably whatever version comes with Mac OS X Leopard. Supporting older versions will happen as I get advocates or testers for them. I also won't explicitly drop support for any Perl or DBI versions that the current DBD::SQLite supports, but I only intend to test it with the latest DBI and Perl 5.8.x+. A minor change is I will start out with using 3-part versions and have a dependency on version.pm 0.74, which is bundled with Perl 5.10.x and an easy install otherwise. Now a specific question for you: First assume the new DBD::SQLite can look in at least 3 places for a SQLite library to use, which are: 1. An amalgamation file that the user explicitly put in the distro directory (or that was similarly slipstreamed into a copy of the distro maybe by some OS package manager); 2. A SQLite system shared library that was installed either as part of the OS or later by a user; 3. Go and automatically fetch a copy of the latest amalgamation file from sqlite.org, similarly to how Rakudo Perl 6 can go fetch a copy of its Parrot dependency from the 'net. Now assuming that, changeable config options aside, there is an automatic default order that these alternate sources will be used by a hands-free CPAN/CPANPLUS/etc install, I ask you what options should be used and in what order? Now I think we probably can all agree that if choice #1 is satisfied (the user explicitly put a file to use in the distro dir or a packager did), then it is clear-cut to try to use that or die trying. What is less clear cut is whether the next automatic default action should be to use an already installed SQLite shared library or whether to download the latest from sqlite.org. I'm thinking that between #2 and #3, that #3 (get the latest) should always be the default choice, and that users have to explicitly override, such as by setting an environment variable, or whatever already happens with DBD::SQLite, to use the system one instead. So in fact the real question then is whether I should stay with the precedent existing with the current DBD::SQLite, which effectively is download from sqlite.org by default, or whether I should change this default. Any thoughts on that matter, or on anything else I raised? And while I suppose this is technically a fork, I'll still
Re: RFC: developing DBD::SQLite
On Mar 26, 2009, at 10:30 PM, Darren Duncan wrote: Hello, So, out of my un-paid projects, my promise to take over release management of DBD::SQLite (from the still incommunicado previous owner) has now come to the front of my queue (now that Set::Relation 0.9.0 is out), so I'm now starting to think about it in detail and get to work over the next week or two. In that vein, the first and only major design change I intend to make right from the start is to stop bundling the SQLite library with the DBI driver and so the driver will have that library as an external dependency. --1. Prefer a system-installed lib, but use the bundled lib if one cannot be found on the system. Don't make this harder for people to use. While one of the selling points of DBD::SQLite versus other DBI drivers in the past was that it came with everything you need, with the advent of a single file amalgamation library being provided standard from sqlite.org, as well as the increasing availability of the SQLite library as its own shared system library install, I figure it isn't too difficult now for users to either obtain the library separately or use the one that came with their system, or the DBD::SQLite installer could automatically download it similar to how some other projects download their dependencies (Rakudo Perl 6 can download Parrot for example); so I don't think the ease of use of DBD::SQLite is diminished significantly by it no longer bundling the SQLite library. Don't download it; a lot of times modules are installed where there is no access to the Net. And those libraries that download external dependencies never work very well (see Math::Pari). On the other side, there is a lot of benefit gained from not bundling. For one thing the size of the distribution as well as the source control is cut down significantly, since the DBI driver alone is orders of magnitude smaller. For another thing, occasional needs to update for compatibility aside, DBD::SQLite will always be up to date with the newest SQLite library; users just update the library and possibly recompile the DBD::SQLite they already have. And so DBD::SQLite won't need to be updated as often; it would just need updates to address incompatible changes to the SQLite library, or to fix bugs in or update features specific to DBD::SQLite itself. These are benefits to the developer of the module, not to the end user. I don't find them compelling. Another quasi-change is that DBD::SQLite will be designed to work specifically with the amalgamation version of the source files only, not the pre-amalgamated versions; I say quasi-change since Audrey Tang already did the work to convert DBD::SQLite to work this way, in the separate ::Amalgamation distro. Don't know anything about this. Compatibility-wise, my DBD::SQLite will endeavour to work with all versions of SQLite 3.y.z, though note that only 3.4.0 for which the amalgamation file was a distinct download on sqlite.org (and 3.3.14 or so was the first that amalgamation was a make target). Or more specifically, I only plan to test with the latest SQLite source library available at the time (3.6.11 currently), as well as probably whatever version comes with Mac OS X Leopard. Supporting older versions will happen as I get advocates or testers for them. I also won't explicitly drop support for any Perl or DBI versions that the current DBD::SQLite supports, but I only intend to test it with the latest DBI and Perl 5.8.x+. If you don't test it with other versions, how can you be sure that they're supported? I deal with this with pgTAP, BTW; I have to keep 5 separate PostgreSQL installations around to make sure it works with all of them. You could probably script it to test that it works with n versions of SQLite. That would obviously be an improvement over the current maintenance. A minor change is I will start out with using 3-part versions and have a dependency on version.pm 0.74, which is bundled with Perl 5.10.x and an easy install otherwise. Why? I see no benefit to this, and just imposes yet another inconvenience on users. Now a specific question for you: First assume the new DBD::SQLite can look in at least 3 places for a SQLite library to use, which are: 1. An amalgamation file that the user explicitly put in the distro directory (or that was similarly slipstreamed into a copy of the distro maybe by some OS package manager); 2. A SQLite system shared library that was installed either as part of the OS or later by a user; 3. Go and automatically fetch a copy of the latest amalgamation file from sqlite.org, similarly to how Rakudo Perl 6 can go fetch a copy of its Parrot dependency from the 'net. Now assuming that, changeable config options aside, there is an automatic default order that these alternate sources will be used by a hands-free CPAN/CPANPLUS/etc
