DBD::MariaDB 1.23

[pali]

Hello,

New version of DBD::MariaDB 1.23 is now on CPAN:

  https://metacpan.org/release/PALI/DBD-MariaDB-1.23

DBD::MariaDB is DBI driver for connecting to MariaDB and MySQL
databases. It is a fork of DBD::mysql driver which aims to work
correctly with Unicode and to have a better support for MariaDB,
while keeping compatibility with MySQL.

New version 1.23 is mostly bug fix release. Major changes are:

  * Fix compilation with MariaDB 10.2/10.3 client library
  * Fix compilation with MySQL 8.0+ client library
  * Fix tests for running against MySQL 8.0+ server
  * Fix mariadb_use_result attribute
  * Fix mariadb_sockfd attribute on Windows
  * Fix AutoCommit attribute with RaiseError=>0
  * Fix statements with multiple result sets in asynchronous mode
  * Add support to query and change mariadb_multi_statements attribute
  * Add connect option mariadb_auth_plugin for specifying auth plugin
  * Updated and extended documentation

Up-to-date documentation for DBD::MariaDB is available at:

  https://metacpan.org/pod/DBD::MariaDB

If you find a bug in DBD::MariaDB, please report it to issue tracker:

  https://github.com/perl5-dbi/DBD-MariaDB/issues

Note that GitHub project page was moved to perl5-dbi organization but
old URLs should still work thanks to automatic redirection.

Pali

Re: Strange issue with inserting varchar2()

[pali]

On Thursday 27 May 2021 23:35:32 Bruce  Johnson wrote:
> use bytes; $string=substr($orig,0,4000);

Hello! This is really suspicious. See **BOLD** description of 'bytes' module:

  **Use of this module for anything other than debugging purposes is strongly 
discouraged.**

  **If you feel that the functions here within might be useful for your 
application, this possibly indicates a mismatch between your mental model of 
Perl Unicode and the current reality.**

DBD::MariaDB 1.21

[pali]

Hello!

New version of DBD::MariaDB 1.21 is now on CPAN:

  https://metacpan.org/release/PALI/DBD-MariaDB-1.21

DBD::MariaDB is DBI driver for connecting to MariaDB and MySQL
databases. It is a fork of DBD::mysql driver which aims to work
correctly with Unicode and to have a better support for MariaDB.

In new version 1.21 are following changes:

  * Fix compilation on Windows
  * Fix tests for MariaDB 10.2.20+, 10.3.12+ and 10.4.1+
  * Fix processing of unfetched multi result sets
  * Cover a protocol design error for multi result statements
  * Fix $dbh->last_insert_id for async calls and multi result statements
  * Add more robust solution to prevent deinitializing OpenSSL functions
(https://github.com/gooddata/DBD-MariaDB/issues/119)
  * Set $sth->{Active} to off after fetching all data
(https://github.com/jhthorsen/mojo-mysql/pull/47#issuecomment-448095164)
  * Fix $sth->rows for async, multi results and when error occurred
  * Fix disconnect_all() method to work also during global destruction phase
  * Fix skipping test t/14destruct.t
  * Revert "MySQL bug #93276 is fixed in 8.0.15"
(https://github.com/gooddata/DBD-MariaDB/issues/137)

And up-to-date documentation for DBD::MariaDB is available at:

  https://metacpan.org/pod/DBD::MariaDB

If you find a bug in DBD::MariaDB, please report it to issue tracker:

  https://github.com/gooddata/DBD-MariaDB/issues


PS: Perl Foundation was selected as organization in Google Summer of
Code 2019 program and we have there DBD::MariaDB project for students:

  https://perl-gsoc-2019.github.io/ideas/perl5/DBD-MariaDB.html

Re: Translate between DBI and SQL

[pali]

On Friday 08 February 2019 22:37:17 Mike Martin wrote:
> Has anyone done any work on converting SQL queries between RDBMS and perl?
> 
> My particular interest is DBD::Pg but anything would be of use
> 
> It would be very useful when I am testing complex SQL, it's very easy to
> miss a \ or quote between the two
> 
> Thanks
> Mike

Hi! Maybe following module could be interesting:
https://metacpan.org/pod/DBIx::Perlish

Re: macOS Mojave "Can't link/include C library ' ', aborting"

[pali]

On Wednesday 16 January 2019 17:59:07 James Brown via dbi-users wrote:
> > On 16 Jan 2019, at 5:48 pm, Daniël van Eeden  > > wrote:
> > 
> > The --libs part of that looks wrong.
> > I would expect something like "... -lmysqlclient -lssl -lcrypto”, not a -l 
> > without library name.
> > Not sure why that might be.
> 
> Thanks Daniël. To compile Dovecot to link against MySQL I had to use:
> 
> export CPPFLAGS="-I/usr/local/opt/mysql@5.7/include/mysql 
> -I/usr/local/Cellar/openssl@1.1/1.1.1/include"
> export LDFLAGS="-L/usr/local/opt/mysql@5.7/lib 
> -L/usr/local/opt/openssl@1.1/lib"
> 
> ./configure --with-mysql --with-ssl=openssl
> 
> Would I have to do something like that?

Hi! After forking DBD::mysql to DBD::MariaDB, I fixed lot of problems
with parsing output of mysql_config and finding include paths and
libraries in Makefile.PL.

Makefile.PL in DBD::MariaDB accepts env variables DBD_MARIADB_CFLAGS
DBD_MARIADB_LIBS and override output from mysql_config when env variable
DBD_MARIADB_CONFIG is set to "skip".

Maybe something similar could work for DBD::mysql. But I do not know as
Makefile.PL in DBD::mysql had lot of bugs... Maybe it helps you.

DBD::MariaDB 1.10 released on CPAN

[pali]

Hello!

I would like to announce that DBD::MariaDB 1.10 was released on CPAN.

  https://metacpan.org/release/PALI/DBD-MariaDB-1.10

DBD::MariaDB is DBI driver for connecting to MariaDB and MySQL
databases. It is a fork of DBD::mysql driver which aims to work
correctly with Unicode and to have a better support for MariaDB.

Currently DBD::MariaDB is tested with more then 150 combination of
MariaDB/MySQL client libraries with different MariaDB/MySQL server
databases, including new versions of MariaDB 10.3/10.4 and MySQL 8.0.

  https://travis-ci.org/gooddata/DBD-MariaDB

New version 1.10 contains mostly bug fixes and workarounds for bugs in
MariaDB/MySQL client libraries. Also there are new features too:

  * Support for DBI driver-aware $sth->last_insert_id() method which is
replacement for MariaDB specific $sth->{mariadb_insertid} attribute

  * Default session collation is set to utf8mb4_unicode_ci (real Unicode)

List of all changes is available at DBD-MariaDB-1.10 release page:

  https://metacpan.org/release/PALI/DBD-MariaDB-1.10

And up-to-date documentation for DBD::MariaDB is available at:

  https://metacpan.org/pod/DBD::MariaDB

If you find a bug in DBD::MariaDB, please report it to issue tracker:

  https://github.com/gooddata/DBD-MariaDB/issues

Re: Generic way of fetching a list of databases

[pali]

On Friday 16 November 2018 17:04:50 Marius Feraru wrote:
> On Fri, Nov 16, 2018 at 10:38 AM  wrote:
> > Hi! If you want to list all databases then use DBI's data_sources method:
> > https://metacpan.org/pod/DBI#data_sources
> 
> And DBD::mysql points to this method for a while:
> https://github.com/perl5-dbi/DBD-mysql/blame/9f96fab985327773397a60020f5dcb76dfd8c396/lib/DBD/mysql.pm#L1432
> 
> Yet, it never worked for me, while their "legacy" version still does:
> $ perl -MData::Dump=dd -MDBI -e 'for $impl (qw[mysql MariaDB]) { $plm
> = lc $impl; $dbh =
> DBI->connect("dbi:$impl:;${plm}_read_default_group=mysql"); dd {
> listdbs => [ eval {$dbh->func("_ListDBs")} ], data_sources => [
> $dbh->data_sources ], dbd => "DBD::$impl", dbi_version =>
> $DBI::VERSION, dbd_version => ${"DBD::$impl\::VERSION"}, map { $_ =>
> $dbh->{"${plm}_$_"} } qw(clientinfo clientversion serverversion) } }'
> {
>   clientinfo => "8.0.13",
>   clientversion => 80013,
>   data_sources => [],
>   dbd => "DBD::mysql",
>   dbd_version => 4.048,
>   dbi_version => 1.642,
>   listdbs => ["information_schema", "mysql", "performance_schema", "sys"],
>   serverversion => 80013,
> }
> {
>   clientinfo => "8.0.13",
>   clientversion => 80013,
>   data_sources => [
> "DBI:MariaDB:information_schema",
> "DBI:MariaDB:mysql",
> "DBI:MariaDB:performance_schema",
> "DBI:MariaDB:sys",
>   ],
>   dbd => "DBD::MariaDB",
>   dbd_version => "1.00",
>   dbi_version => 1.642,
>   listdbs => [],
>   serverversion => 80013,
> }
> 
> Pali, seeing that your flavor's data_sources does work, I reckon it
> might be yet another update to upstream. I suggest keeping ListDBs
> retirement separate though. ;-)

In DBD::MariaDB was that legacy _ListDBs() method removed. Just because
DBI provides generic method data_sources().

Documentation for DBD::MariaDB also describes how to call data_sources()
method also without need to create $dbh connection. You need to specify
host, port, user, password:
https://metacpan.org/pod/DBD::MariaDB#data_sources

Btw, same additional arguments (host, port, user, password) work also
with DBD::mysql, just it is not properly documented.

IIRC, data_sources() method work also with PostgreSQL...
https://metacpan.org/pod/DBD::Pg#data_sources

> For the sake of discussion –and to disappoint Daniel, sorry– given
> this lack of coherence (NOT only at DBD level, e.g. privileges might
> be involved), I have not been able to rely on DBI::data_sources,
> having to employ DBMS specific queries instead.
> e.g. @mysql: while "show databases" usually still works, an
> "information_schema" based alternative/fallback could be useful.
> 
> Cheers!

Re: Generic way of fetching a list of databases

[pali]

On Friday 16 November 2018 13:09:21 Daniel Kasak wrote:
> Hi all.
> 
> I have a project that has to support pretty much every major database
> around, plus a number of more esoteric "big data" platforms as well. Until
> now, I've had a bunch of methods, implementing fetching databases, schemas,
> tables, etc, per database, with database-specific queries ( eg against
> information_schema ).
> 
> Some of the newer databases I'm trying to support have very little
> documentation, and in some cases no apparent way of fetching in the schema
> via SQL. I've had a conversation with one of the tech support people for a
> DB product who said that there were generic ODBC functions we can call for
> this kind of thing. Is this the case?
> 
> I've done quite a bit of search, but can't find any docs that mention
> fetching *databases* - either in ODBC docs or in Perl/DBI docs. The closest
> I've found that *might* have worked was DBI's tables() method:
>  https://metacpan.org/pod/DBI#tables
>   ... but:
> 
> - this doesn't work in cases where there is a separation between
> hierarchies at the database level ( eg postgres only lists schemas and
> tables in the current database )
> - this isn't returning *anything* for me with any of the ODBC drivers I've
> tried
> 
> So is it possible to retrieve a list of databases in a generic way? Failing
> that, assuming that there *is* some ODBC call ( as suggested by one DB
> vendor ) that I can use, is there some way of calling it from Perl with
> DBD::ODBC?
> 
> Thanks :)
> 
> Dan

Hi! If you want to list all databases then use DBI's data_sources method:
https://metacpan.org/pod/DBI#data_sources

That method is there exactly for this purpose to list all databases in
DBI's DSN format. If you are not interested in full DSN, just extract
database... See example:

$ perl -MDBI -E 'say $_ foreach map { $_ =~ /^DBI:[^:]*:(.*)/i } 
DBI->data_sources("mysql")'
information_schema
mysql
performance_schema
test
test2
test3
test4

Re: Wierd issue with printf and DBD::Pg

[pali]

On Monday 01 October 2018 11:13:48 Mike Martin wrote:
> HI
> If I use printf to round a numeric value before inserting into postgres
> table it is altered to 1 rather than the value when it is put into a table

Hi! Function printf takes format string with parameters and then it
prints to the standard output. And returns 1 (true) if is succeed.

> example
> CREATE TABLE public.chksize
> (
> size numeric(10,2), #does not matter what field type
> path1 character varying COLLATE pg_catalog."default"
> )
> 
> this creates a value of 1 for every value
> 
> my $ins=$dbh->prepare("INSERT into chksize (size,path1) VALUES (?,?) ");
> open my $list ,'chksizer.txt';
> no strict 'refs';
> my @list=(<$list>);
> foreach my $k (@list){
> chomp $k;
> my ($size,$path)=split /,/,$k;
> my $size1=printf('%.1f',$size/(1024*1024));

... so value '%.1f',$size/(1024*1024) is printed to STDOUT. Printing
succeed and therefore into $size1 is assigned 1 (true value).

You probably want to use sprintf function which returns formatted string
instead of printing it to STDOUT.

> $ins->bind_param(1,$size1);
> $ins->bind_param(2,$path);
> $ins->execute;
> }
> 
> without printf this inserts proper value
> 
> my $ins=$dbh->prepare("INSERT into chksize (size,path1) VALUES (?,?) ");
> open my $list ,'chksizer.txt';
> no strict 'refs';
> my @list=(<$list>);
> foreach my $k (@list){
> chomp $k;
> my ($size,$path)=split /,/,$k;
> my $size1=$size/(1024*1024);
> $ins->bind_param(1,$size1);
> $ins->bind_param(2,$path);
> $ins->execute;
> }
> 
> Any ideas what is happening here?
> 
> thanks
> 
> Mike

Re: Tim Bunce - RE: DBD-DBI Insert Binary Files and using Perl modules in cPanel environment

[pali]

Hi! I do not know if it helps you, but processing binary data with MySQL
or MariaDB server and Perl's DBI can be tricky due to limitation of
MySQL protocol. Some details I wrote to the DBD::MariaDB documentation:

https://metacpan.org/pod/DBD::MariaDB#Working-with-binary-data

DBD::MariaDB 1.00 released on CPAN

[pali]

Hello!

I would like to announce public release of the DBD::MariaDB 1.00. Now it
is available on CPAN and can be installed directly via cpan client.

 https://metacpan.org/release/PALI/DBD-MariaDB-1.00

There is a problem with perl mailing lists and perl email servers,
therefore it is possible that not all participants receive this message.

DBD::MariaDB is a DBI driver for connecting to the MariaDB or the MySQL
database. It is a fork of the DBD::mysql 4.041 driver. This fork's
primary focus is fixing Unicode bugs & security problems, adding support
for new versions of MariaDB and MySQL client libraries and other things
which are not possible to fix in DBD::mysql due to legacy problems.

Documentation for DBD::MariaDB and install instructions are on metacpan:

 https://metacpan.org/pod/DBD::MariaDB
 https://metacpan.org/pod/DBD::MariaDB::INSTALL

Public version 1.00 fixes only small problems found in previous beta
version 0.90_01. More information can be found in previous email:

 https://www.nntp.perl.org/group/perl.dbi.dev/2018/06/msg8174.html

Full changelog, including all fixed issues, is available in Changes file
and also formatted at metacpan release page:

 https://metacpan.org/release/PALI/DBD-MariaDB-1.00

If you found a bug in DBD::MariaDB, please report it to us via GitHub:

 https://github.com/gooddata/DBD-MariaDB/issues

DBD::MariaDB beta 0.90_01 released on CPAN

[pali]

Hello,

I would like to announce that after few months of DBD::MariaDB
development, beta version 0.90_01 of DBD::MariaDB is now on CPAN:

 https://metacpan.org/release/PALI/DBD-MariaDB-0.90_01

DBD::MariaDB is a DBI driver for connecting to the MariaDB or the MySQL
database. It is a fork of the DBD::mysql 4.041 driver. This fork's
primary focus is fixing Unicode bugs & security problems, adding support
for new versions of MariaDB and MySQL client libraries and other things
which are not possible to fix in DBD::mysql due to legacy problems. See
previous emails:

 https://www.nntp.perl.org/group/perl.dbi.dev/2017/08/msg8030.html
 https://www.nntp.perl.org/group/perl.dbi.dev/2018/01/msg8151.html

DBD::MariaDB is currently developed in GoodData. Our git repository is
public at GitHub and we are open for pull requests from the community,
as well.

 https://github.com/gooddata/DBD-MariaDB

Version 0.90_01 is marked as a beta. We would like to get feedback from
the community. If you find a bug or have a suggestion for improvement,
fell free to report it on GitHub:

 https://github.com/gooddata/DBD-MariaDB/issues

We are using Travis for testing and continuous integration with more
then hundred configurations of MySQL/MariaDB client/server versions and
different Perl versions. This setup ensures that the driver can be
compiled with any client library and can connect to remote MariaDB/MySQL
server in any version. Travis status is available at:

 https://travis-ci.org/gooddata/DBD-MariaDB

Major changes for DBD::MariaDB 0.90_01 since DBD::mysql 4.041 are:

- Fix for CVE-2017-10788 (use-after-free)
- Fix for CVE-2017-3302 (use-after-free)
- Fix for CVE-2018-2767 (TLS is not enforced)

- Compatibility fixes for MariaDB 10.2.4+, 10.3.0+
- Compatibility fixes for MariaDB Connector/C 2.3.0+, 3.0.2+
- Compatibility fixes for MySQL 5.7.6+, 8.0+

- Full Unicode support according to Perl Unicode model
- Automatically established utf8mb4 (or utf8) connection encoding
- Embedded server support integrated into main DBD::MariaDB driver
- Fix $dbh->ping(), $dbh->last_insert_id() and $dbh->get_info()
- Fix processing statements with nul bytes

The whole changelog for DBD::MariaDB 0.90_01 is available at:

 https://metacpan.org/release/PALI/DBD-MariaDB-0.90_01

Re: Extend API for last_insert_id

[pali]

On Tuesday 15 May 2018 21:12:19 Tim Bunce wrote:
> On Fri, May 04, 2018 at 02:10:18PM +0200, p...@cpan.org wrote:
> > Hello, do you have any opinion or comments?
> > 
> > Tim, you as a DBI maintainer, what do you think about those ideas?
> > 
> > On Friday 27 April 2018 16:03:59 p...@cpan.org wrote:
> > > 
> > > So I'm proposing change that caller would be allowed to call
> > > $dbh->last_insert_id() without any argument
> 
> Ok.
> 
> > > Second change: Add a new statement method $sth->last_insert_id().
> 
> That seems fine. I'd write the fallback code like this:
> 
> sub last_insert_id { return shift->{Database}->last_insert_id(@_) }

Changed.

And now I created a pull request with these changes:
https://github.com/perl5-dbi/dbi/pull/64

> Thanks Pali.
> 
> Tim.

Re: Extend API for last_insert_id

[pali]

Hello, do you have any opinion or comments?

Tim, you as a DBI maintainer, what do you think about those ideas?

On Friday 27 April 2018 16:03:59 p...@cpan.org wrote:
> Hello,
> 
> I would like to propose two new changes to DBI which extends
> last_insert_id API.
> 
> First one: Allow to call $dbh->last_insert_id() method without
> arguments. Currently this method needs to take four arguments (plus
> $dbh) and XS code validates their count:
> 
> XS_EUPXS(XS_DBD__Perl__db_last_insert_id)
> {
> dVAR; dXSARGS;
> if (items < 5 || items > 6)
>croak_xs_usage(cv,  "dbh, catalog, schema, table, field, 
> attr=Nullsv");
> ...
> }
> 
> More databases does not process these arguments and caller needs to
> supply four undefs.
> 
> So I'm proposing change that caller would be allowed to call
> $dbh->last_insert_id() without any argument and DBI would fill those
> missing arguments by undefs prior to calling DBI driver function. This
> does not change driver API, so all existing DBI drivers would work as
> before and allows application to not specify those useless four undef
> arguments.
> 
> Second change: Add a new statement method $sth->last_insert_id().
> Some databases (e.g. MariaDB) supports tracking last insert id when more
> statement or cursor handles are open. This would allow to call e.g.
> 
> $sth1->execute();
> $sth2->execute();
> ...
> $sth1->last_insert_id();
> $sth2->last_insert_id();
> 
> for drivers which would support it. $sth1->last_insert_id() would return
> insert it which belongs to last ->execute of $sth1 even there were more
> INSERT/execute calls (e.g. by $sth2).
> 
> Currently database handle method $dbh->last_insert_id() returns id of
> the most recent INSERT statement.
> 
> What do you think about those two enhancements?
> 
> Here is my prototype implementation for DBI:
> https://github.com/perl5-dbi/dbi/compare/master...pali:last-insert-id
> 
> DBI drivers can then implement own dbd_st_last_insert_id() function and
> provide correct id for the selected $sth statement.

Re: New DBI driver DBD::MariaDB

[pali]

On Tuesday 23 January 2018 11:14:13 p...@cpan.org wrote:
> * Better compatibility with the MariaDB client library

Basic unit tests passed on Travis with last MySQL versions 5.5.58,
5.6.38, 5.7.20, 8.0.3 and MariaDB versions 5.5.58, 10.0.33, 10.1.30,
10.2.12, 10.3.3. Also with MariaDB C/Connector 2.3.4 and 3.0.2.

Therefore DBD::MariaDB is working with MariaDB 10.2 and 10.3 series.

Whole test matrix (about 100 passed configurations) is available at:
https://travis-ci.org/gooddata/DBD-MariaDB

New DBI driver DBD::MariaDB

[pali]

Hello,

I would like to announce a new DBI driver for MariaDB and MySQL
databases. As stated in the email [1] which I sent to this mailing list
about a half year ago, we in the GoodData company decided to create a
fork of the old DBD::mysql driver under the name DBD::MariaDB.

The source code of this new driver is now public, avaiable in our git
repository on GitHub [2]. After we declare it stable we would like to
upload it to CPAN, too.

We hope that this new DBD::MariaDB driver would be useful also for
others and we are open for pull requests or other bug fixes from the
community.

Major changes in DBD::MariaDB since DBD::mysql 4.043 are:

* Correct Unicode support according to the standard Perl Unicode model
* Better compatibility with the MariaDB client library
* Fixes for security issues, SSL/TLS (BACKRONYM and Riddle vulnerabilities)
* Fixes for ZEROFILL columns and magic scalars (needed by Amavis)
* Extended Travis-CI testing

[1] - https://www.nntp.perl.org/group/perl.dbi.dev/2017/08/msg8030.html
[2] - https://github.com/gooddata/DBD-MariaDB

Re: DBD::mysql next steps

[pali]

On Friday 10 November 2017 10:13:55 Patrick M. Galbraith wrote:
> Greetings all!
> 
> Michiel and I have been talking, weighing options of what course to take in
> dealing with moving forward-- with the goal of both offering both stability
> and the choice to have the latest functionality and bug fixes as well as
> give contributors the opportunity to be part of overall improvements to the
> driver.
> 
> What we are going to do is:
> 
> Add to the connection the ability to turn on proper UTF handling with
> 'mysql_enable_proper_unicode'. This gives the user volition the option to 
> knowingly
> toggle whether they want the new functionality and understand that
> data structures returned or accepted by the driver might be different
> than without this setting.
> 
> The other options had their merits, but we think this will solve the issue
> while keeping the driver unified and prevent divergence.
> 
> Thank you for your input over the last couple months-- we look forward to
> moving ahead!
> 
> Patrick and Michiel
> 

Hi! And what are you going to do with all commits which you reverted in
4.043 version? As I remember, months ago you wrote that you reapply
fixes, but nothing happened. Instead you started merging conflicting
new pull requests which other people opened on github. This just prevent
any future reintroduction of fixes which were already done and was
reverted. Or you going to invent wheel again? I'm really surprised that
you now want to throw out everything which I did, including security
fixes for SSL/TLS or new versions of MariaDB/MySQL (with huge test
setup). Otherwise I do not understand why you have not looked at
reverted commit yet... Or are you expecting that new after one year I
should start rebasing all my commits which I did on top of master and
opening pull requests again and again? Sorry, but this really look that
you are not interested in new supporting new MariaDB version and now is
really a good time to create a fork of DBD::mysql which would contains
support for new MariaDB, huge test coverage and security fixes for
SSL/TLS.

Re: Bugtracker DBD::mysql

[pali]

On Tuesday 14 November 2017 10:56:58 Michiel Beijen wrote:
> Hi Pali,
> 
> On Mon, Nov 13, 2017 at 6:18 PM,  <p...@cpan.org> wrote:
> > And I would suggest to disable issue tracker on github as primary bug
> > tracker (according to DBD::mysql documentation) is on RT and also
> > probably all problems are reported there. The worst thing which can be
> > is to have two independent bug trackers, which is current situation.
> 
> You're right, it would be helpful and desirable to have only one bug tracker!
> 
> I've tried to migrate the existing RT tickets to Github before, but
> that did not work correctly because DBD::mysql uses a Github
> organization and the script I used at the time did not accept this.
> I'll put it higher on my to-do list again and look into getting that
> done.

Hi! As there are only few tickets on github, it would be easier to
disable creating new tickets on github and those few which are not
resolved yet either move on RT or (if github allows it; which I believe
yet) continue to exists.

And once any migration is ready, then it can be fully switched.

Still one bugtracker is better then two, even if it is not on github.

Re: DBD::mysql next steps

[pali]

And I would suggest to disable issue tracker on github as primary bug
tracker (according to DBD::mysql documentation) is on RT and also
probably all problems are reported there. The worst thing which can be
is to have two independent bug trackers, which is current situation.

Re: DBD::mysql next steps

[pali]

On Friday 10 November 2017 10:13:55 Patrick M. Galbraith wrote:
> Greetings all!
> 
> Michiel and I have been talking, weighing options of what course to take in
> dealing with moving forward-- with the goal of both offering both stability
> and the choice to have the latest functionality and bug fixes as well as
> give contributors the opportunity to be part of overall improvements to the
> driver.
> 
> What we are going to do is:
> 
> Add to the connection the ability to turn on proper UTF handling with
> 'mysql_enable_proper_unicode'. This gives the user volition the option to 
> knowingly
> toggle whether they want the new functionality and understand that
> data structures returned or accepted by the driver might be different
> than without this setting.
> 
> The other options had their merits, but we think this will solve the issue
> while keeping the driver unified and prevent divergence.
> 
> Thank you for your input over the last couple months-- we look forward to
> moving ahead!
> 
> Patrick and Michiel

Great! Thank you very much, I'm waiting for a new action which would
bring DBD::mysql back to the active development.

Re: DBD::mysql path forward

[pali]

Technical: Reason why it does not help nor solve the main problem: Due
to internals, DBD::mysql pseudo-randomly decide if input bind variable
is encoded to UTF-8 or not. And do it independently of how is option
enable_utf8 configured. As we know there are applications which misuse
this internal. Calling perlapi functions may change this internal state
under some circumstances and IIRC perl (itself) can change. Therefore it
is not safe to touching code around. Fixing Unicode support needs to
totally replace this code.

Night Light wrote that enable_utf8=0 does no encoding of input and this
is not truth. It is also root of all problems and applications which
misuse it.

Also we have no idea what else applications misuse. And because we were
not already able to detect it in testing period, I really doubt that we
would be able to do it in future...

On Friday 10 November 2017 09:34:19 p...@cpan.org wrote:
> I do not understand what (and how) your proposal with =2 solve. Probably
> nothing and people would again start after final release again
> complaining...
> 
> On Friday 10 November 2017 09:24:35 Night Light wrote:
> > Forking would take away my concerns (thank you all for suggesting that) but
> > one thing pops into my mind:
> > A decision to fork a mature module is not a light decision and should only
> > be considered as a last resort (when ran out of all options).
> > So far it is all based on the opinion of only one developer who finds it
> > hard or probably impossible to implement.
> > 
> > Can before forking at least a second person with C knowledge open dbdimp.c
> > and measure if it is indeed unfeasible to make the module backward
> > compatible by adding an option as suggested?
> > 
> > Michiel, do you know C? Can you do that 5-minute check?
> > 
> > The breaking is due to the fact that the prepared SQL statement and the
> > bound parameters will be apart from what is returned from MySQL is encoded
> > as well.
> > The mysql_enable_utf8/mysql_enable_utf8mb4 flags already arrange logic to
> > set if utf8 encoding/decoding should be done or not.
> > I'm not a C expert but this is what I find in dbdimp.c of 4.042 about it:
> > 
> > dbd_st_prepare_sv function:
> > bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> > get_statement(aTHX_ statement_sv, enable_utf8, , _len);
> > 
> > dbd_bind_ph function:
> > bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> > bind_param(_sth->params[idx], value, sql_type, idx+1, enable_utf8);
> > 
> > 
> > Would changing that into the following make the module backwards compatible?
> > bool enable_utf8 = (imp_dbh->enable_utf8 == 2 || imp_dbh->enable_utf8mb4 ==
> > 2);
> > 
> > enable_utf8 0 = no encoding of input and only decoding of output
> > enable_utf8 1 = decode only output (4.043 mode)
> > enable_utf8 2 = encoded input and decode output (4.042 mode)
> > 
> > On Fri, Nov 10, 2017 at 8:37 AM, Darren Duncan 
> > wrote:
> > 
> > > Michael, why can't you accept moving forward under a new module name?  Why
> > > does it have to be under the old name?  When people purposefully want to
> > > upgrade they purposefully choose the new module name in order to do so.
> > > What is the actual problem in that? -- Darren Duncan
> > >
> > > On 2017-11-09 10:59 PM, Michiel Beijen wrote:
> > >
> > >> On Fri, Nov 10, 2017 at 7:16 AM, Darren Duncan 
> > >> wrote:
> > >>
> > >>> I agree with everything Dan said here.  Its what I proposed, in fewer
> > >>> words.
> > >>> Do all new development under a new name, including all of Pali's work,
> > >>> and
> > >>> leave the current name for a product with no further effort applied to
> > >>> develop it. -- Darren Duncan
> > >>>
> > >>
> > >> This is NOT an option to me - it simply can't because the world moves
> > >> forward and because of bitrot. The 'old' version - the version that
> > >> works for most people, the current version of DBD::mysql, the one
> > >> which would then receive no more maintenance as it is no longer
> > >> compiles with the latest version of libmysqlclient and it does not
> > >> compile with libmariadb. This will only get worse in the future.
> > >>
> > >> I'll stick with my earlier proposal - I'll propose to go back to the
> > >> *current* latest DBD::mysql release which does not break backcompat
> > >> for our users; add the patches that we discarded when we rolled back
> > >> one by one, such as testing on many different lib/db options, memory
> > >> leaks and so on, and make a new release so we can be on the move
> > >> again.
> > >>
> > >> If possible I'd want to add back the *breaking* unicode changes that
> > >> were introduced but they should be either in a separate namespace OR
> > >> under a specific configuration option.
> > >> Currently this whole thing has cost us loosing MONTHS of progress and
> > >> then MONTHS of nothing and that is simply not good.
> > >>
> > >> Patrick: let me know if you're OK with this 

Re: DBD::mysql path forward

[pali]

On Friday 10 November 2017 10:09:59 demerphq wrote:
> Pali is there a concise summary of what we are arguing about here?

Short summary:

1. There are applications which misuse perl and DBD::mysql internals to
   pass blob or utf-8 text data from perl to MySQL database. We were not
   aware of them for a longer testing period of time as nobody
   complained until version 4.042 of DBD::mysql was released. There were
   trial releases, blog posts, asking testers for test, etc...

2. Unicode support in DBD::mysql is broken and cause data corruption.
   There are lot of reported bugs about it, e.g.:
   https://rt.cpan.org/Public/Bug/Display.html?id=25590
   https://rt.cpan.org/Public/Bug/Display.html?id=53130
   https://rt.cpan.org/Public/Bug/Display.html?id=60987
   https://rt.cpan.org/Public/Bug/Display.html?id=87428
   https://rt.cpan.org/Public/Bug/Display.html?id=119079
   https://rt.cpan.org/Public/Bug/Display.html?id=120141
   All those problems were fixed, but breaking applications which misuse
   internals. Basically Unicode support is opposite of the above.
   All fixes from 4.042 were reverted and version 4.043 was released.

3. DBD::mysql itself misuse libmysqlclient internals for a long time and
   now when MariaDB changed something, it cannot be used with DBD::mysql
   anymore.

4. Due to how Oracle (not) reacted and (not) fixed problems related to
   SSL, DBD::mysql has a security problems when SSL encryption is used.
   There are also other crashing problems in DBD::mysql.

5. Misusing internals of 3rd applications which use DBD::mysql is a
   problem because basically any change of low level parts in DBD::mysql
   can lead to breakage of those 3rd applications. And because nobody
   was able to detect it would break something there is a really big
   chance that bigger changes in DBD::mysql break something again.
   Fixing 3. is expected to be big change.

Re: DBD::mysql path forward

[pali]

On Friday 10 November 2017 13:40:40 demerphq wrote:
> On 10 November 2017 at 13:11,  <p...@cpan.org> wrote:
> > Should DBD::mysql maintainers now starting complaining to Oracle and
> > MariaDB, that they must revert their changes in MySQL and MariaDB, just
> > because DBD::mysql (as libmysqlclient application) stopped working,
> > because is misusing internals of C structures?
> 
> Pali, I wanted to give you some feedback on this thread. I have been
> monitoring this discussion for some time. My $workplace probably has
> one of the largest Mysql footprints in the world, and certainly in the
> Perl world. We have funded MySQL/DBI related work in the past, and we
> are a large funder of TPF. So we are in the position to throw
> resources at this problem if the circumstances were appropriate.
> 
> But it is very difficult to determine if the circumstances ARE
> appropriate because of the level of hysteria in your posts. I believe
> your intentions are good, and I am willing to assume you are
> technically correct about many of the things you comment on, but I
> must say that the tone and form in which you are discussing it is
> counter-productive.
> 
> The people you are talking to on this list by and large are software
> developers and DB types, a group of people who are very intelligent,
> experienced, and naturally conservative in their analytical approach.
> Hysterical comments are not going to impress people like this.
> Expecting people to blindly accept that your analysis is the one and
> only viable analysis when your debating style is high on rhetoric and
> low on facts or reasoned analysis is only going to frustrate yourself
> and annoy the people on this list.
> 
> I really think you need to step back, bring your frustration (however
> understandable that frustration is) into check and try a more
> dispassionate approach. Despite lurking in this thread for some time I
> am still lacking the precise specification of the problem at hand that
> drives some of your comments. While I would like to do a code review
> and see how much I agree with what you are saying, it is very
> difficult to extract the substance from the rhetoric.
> 
> I think if you can manage to tone your responses down a bit you will
> get a much more productive response from the list.
> 
> With respect,
> Yves

Hi! I'm sorry if you understood it as a hysteria.. above quoted parts
was just rhetoric question. Not offense or a bad tone. I'm not native
English speaker and foreign language could be here a problem. Sorry for
that.

I'm disappointed that it is now 3 or 4 months (or more) that development
of DBD::mysql is fully blocked and nothing is happening. On one side are
people wanting old DBD::mysql, on other side are people wanting support
for new MariaDB and everything is just stuck :-(

I just wanted to point that there are technical problems also between
libmysqlclient and DBD::mysql driver have similar impact as problems
between some perl applications and DBD::mysql.

Re: DBD::mysql path forward

[pali]

On Friday 10 November 2017 07:59:05 Michiel Beijen wrote:
> I'll stick with my earlier proposal - I'll propose to go back to the
> *current* latest DBD::mysql release which does not break backcompat
> for our users; add the patches that we discarded when we rolled back
> one by one, such as testing on many different lib/db options, memory
> leaks and so on, and make a new release so we can be on the move
> again.

Haven't we already done it? Haven't we already tested those Unicode
changes under different libraries and MySQL/MariaDB servers, including
releasing two trial versions on cpan with a long testing period without
any reported problem?

Answer is yes, we already did it.

But people who started complaining 1) fully ignores any development and
discussions; 2) fully ignores any testing release; 3) do not have own
integration tests; 4) update everything after release to last version;
5) complain if something is not how they imagine, even if they misuse
internals.

As already stated, if you want to have those users happy, then is no
other option as stop doing active development. If they still want to
misuse internals, you cannot do low level changes. Or at least you would
never be 100% sure that new changes does not break misusing of
internals.

And because for using new version of libmariadb or libmysql is needed
more robust changes, it has very high rate that something which misuse
internals would be broken.

So conclusion is: Either new version of MariaDB/MySQL or no guarantee
that misusing internals would lead to same results.

And beware of fact that C code of DBD::mysql misuse libmysqlclient
structures which leads to problems with compilation and also cause
segfaults for libmariadb. Very similar problem like between perl
application and DBD::mysql, but here it is between DBD::mysql and
libmysqlclient.

And if you are asking, then yes, MariaDB and MySQL decided to changes
their internals and application (like DBD::mysql) which misuse it,
starts crashing or not compiling. And they (MariaDB and MySQL) do that,
even without fact that DBD::mysql (as application) stop working.

So DBD::mysql is now in the same position as people who want to have old
behavior of DBD::mysql for their applications...

Should DBD::mysql maintainers now starting complaining to Oracle and
MariaDB, that they must revert their changes in MySQL and MariaDB, just
because DBD::mysql (as libmysqlclient application) stopped working,
because is misusing internals of C structures?

Re: DBD::mysql path forward

[pali]

I do not understand what (and how) your proposal with =2 solve. Probably
nothing and people would again start after final release again
complaining...

On Friday 10 November 2017 09:24:35 Night Light wrote:
> Forking would take away my concerns (thank you all for suggesting that) but
> one thing pops into my mind:
> A decision to fork a mature module is not a light decision and should only
> be considered as a last resort (when ran out of all options).
> So far it is all based on the opinion of only one developer who finds it
> hard or probably impossible to implement.
> 
> Can before forking at least a second person with C knowledge open dbdimp.c
> and measure if it is indeed unfeasible to make the module backward
> compatible by adding an option as suggested?
> 
> Michiel, do you know C? Can you do that 5-minute check?
> 
> The breaking is due to the fact that the prepared SQL statement and the
> bound parameters will be apart from what is returned from MySQL is encoded
> as well.
> The mysql_enable_utf8/mysql_enable_utf8mb4 flags already arrange logic to
> set if utf8 encoding/decoding should be done or not.
> I'm not a C expert but this is what I find in dbdimp.c of 4.042 about it:
> 
> dbd_st_prepare_sv function:
> bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> get_statement(aTHX_ statement_sv, enable_utf8, , _len);
> 
> dbd_bind_ph function:
> bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> bind_param(_sth->params[idx], value, sql_type, idx+1, enable_utf8);
> 
> 
> Would changing that into the following make the module backwards compatible?
> bool enable_utf8 = (imp_dbh->enable_utf8 == 2 || imp_dbh->enable_utf8mb4 ==
> 2);
> 
> enable_utf8 0 = no encoding of input and only decoding of output
> enable_utf8 1 = decode only output (4.043 mode)
> enable_utf8 2 = encoded input and decode output (4.042 mode)
> 
> On Fri, Nov 10, 2017 at 8:37 AM, Darren Duncan 
> wrote:
> 
> > Michael, why can't you accept moving forward under a new module name?  Why
> > does it have to be under the old name?  When people purposefully want to
> > upgrade they purposefully choose the new module name in order to do so.
> > What is the actual problem in that? -- Darren Duncan
> >
> > On 2017-11-09 10:59 PM, Michiel Beijen wrote:
> >
> >> On Fri, Nov 10, 2017 at 7:16 AM, Darren Duncan 
> >> wrote:
> >>
> >>> I agree with everything Dan said here.  Its what I proposed, in fewer
> >>> words.
> >>> Do all new development under a new name, including all of Pali's work,
> >>> and
> >>> leave the current name for a product with no further effort applied to
> >>> develop it. -- Darren Duncan
> >>>
> >>
> >> This is NOT an option to me - it simply can't because the world moves
> >> forward and because of bitrot. The 'old' version - the version that
> >> works for most people, the current version of DBD::mysql, the one
> >> which would then receive no more maintenance as it is no longer
> >> compiles with the latest version of libmysqlclient and it does not
> >> compile with libmariadb. This will only get worse in the future.
> >>
> >> I'll stick with my earlier proposal - I'll propose to go back to the
> >> *current* latest DBD::mysql release which does not break backcompat
> >> for our users; add the patches that we discarded when we rolled back
> >> one by one, such as testing on many different lib/db options, memory
> >> leaks and so on, and make a new release so we can be on the move
> >> again.
> >>
> >> If possible I'd want to add back the *breaking* unicode changes that
> >> were introduced but they should be either in a separate namespace OR
> >> under a specific configuration option.
> >> Currently this whole thing has cost us loosing MONTHS of progress and
> >> then MONTHS of nothing and that is simply not good.
> >>
> >> Patrick: let me know if you're OK with this and then let's get start
> >> again!
> >>
> >

Re: DBD::mysql path forward

[pali]

On Thursday 09 November 2017 14:26:01 Peter Rabbitson wrote:
> On 11/09/2017 01:46 PM, Noel Butler wrote:
> >On 09/11/2017 21:32, p...@cpan.org wrote:
> >
> >>
> >>>What the complaints in this thread are focused on is what the *users*
> >>>want.
> >and the users want now, or will need in the near future, to build with
> >latest, stable and recommended versions of MariaDB and MySQL.
> 
> Which is a great thing to want. I want this too. This is not what the thread
> is about.
> 
> The problem stems from Pali mashing together 3(!)* independent concerns, and
> presenting them as an "all or nothing" proposition:
>  1. Inability to build against latest libmysql libs
>  2. Several security fixes
>  3. Entirely redefining how unicode is handled throughout the stack

As I already wrote, doing active development on code of DBD::mysql is
hard or probably impossible if you want to have that old behavior.

So fixing more parts of code or adding new features would lead to
breaking that behavior or would be PITA for anybody who would need to
guarantee that it would not be broken again in future.

Moreover DBD::mysql itself (C part of code) misuse libmysql resp.
libmariadb API which leads to segfaults (or silent memory corruption)
when new mariadb is used. And this would be needed properly fixed if you
want to support mariadb.

So I guess it is better that compilation is failing. Users would not be
surprised that driver really damaged data. And not just double encoded
something to UTF-8 (due to misusing API) which has always 1:1 inverse
operation and can be fixed.

This part would not be simple to fix, there would be needed to of
changes in whole driver, find all places where driver touch mariadb C
structures... This is very good candidate which can break that old
DBD::mysql behavior of handling blobs.

Re: DBD::mysql path forward

[pali]

On Thursday 09 November 2017 13:01:19 H.Merijn Brand wrote:
> On Thu, 9 Nov 2017 12:32:00 +0100, p...@cpan.org wrote:
> 
> > > Satisfy the above - and you do get the privilege of being a maintainer
> > > of a central module with 15+ years of history.  
> > 
> > Why should I be interested in maintaining something from which users
> > want impossible things?
> 
>   "I can resist anything! (except temptation)"
> 
> I, like Peter, have backward compatibility high in my goals.
> 
> e.g. I dropped 5.005 supprt on Text::CSV_XS only in June 2012. The
> reasons to keep it working under 5.005 was that it was relatively
> little extra work. I dropped it, as it seriously blocked further
> development and support features my *user-base* asked for. Lucky me
> for being part of my own user-base, so I understand the requests.
> 
> In July 2016 I dropped 5.6.0 and 5.6.1 support. Not because it slows
> development, but because these two versions do not build anymore and
> do not support the modern toolchain. Not because the community or a
> part of the community thinks the users should upgrade to a more recent
> "stable" version of perl. Yes, I know 5.6.2 is from 2003 and 14 years
> old in 6 days and 5.8.0 is even older, but maintaining a module that
> serves almost 1000 other modules on CPAN (last time I counted, it was
> 997 direct or indirect dependencies) brings a burden of responsibility
> 
> This is how the river works: If I break this module, at least 997 other
> modules will break, and that is CPAN only. What about DarkPAN?

I fully understand backward compatibility and ability to use e.g. older
versions of perl with new modules on cpan. For example my module
Email::Address::XS is just reimplementation of the Email::Address module
in C to fix Algorithmic complexity vulnerability and has backward
compatible API. And because Email::Address worked with 5.6, I tried to
achieve also 5.6 compatibility with my Email::Address::XS module.

But DBD::mysql is different thing. Here we are facing two problems:

1) Passing perl Unicode strings is broken, even if some utf8 config
   option is enabled. This is changing data, see my identity examples
   which I sent to list.

2) Probably due to history reasons, API between DBD::mysql and perl
   applications is misused in way, that is atypical for perl itself and
   insane for pure perl applications. Moreover it is something which
   cannot be guaranteed to be "stable" as is affected also by modules
   not relevant to DBD::mysql. And is pseudo-random, but deterministic.

Fixing 1) implies breaking 2) and trying to achieve 2) leads to having
1) broken forever and leads to stopped development.

Similar/same problem as 2) had also DBD::Pg and DBD::SQLite and here
maintainers rather decided to fix Unicode and drop insanity in 2). In
previous emails I also send exact versions of DBD::Pg and DBD::SQLite
in which was behavior changed. (Modulo one problem in one specific
situation which I found in DBD::Pg.)

So now we are in situation when some DBI drivers doing one thing (from
perl POV correct) and DBD::mysql different.

If there are projects which uses both DBD::Pg and DBD::mysql, or
DBD::SQLite and DBD::mysql, then their are either broken (because
DBD::mysql differs how process non-ASCII due to 2)) or they needs
special hacks when MySQL is in use...

So what can be done now?

We have a DBD::mysql which behaves differently as others DBI drivers,
users misuse internals for BLOBs, driver has broken Unicode support
which makes it useless now when UTF-8 is mandatory for applications,
plus has pseudo-random behavior.

I already suggested, look at modules on cpan which uses DBD::mysql and
ask authors/maintainers which behavior they expects. Are they really
aware how non-ASCII data are handled in DBD::mysql and that it is
differently as e.g. DBD::SQLite?

We can also send patches to those modules to fix them. I already sent
documentation update for DBD::mysql which describe how to achieve
correct handling of the BLOB data in both old and Unicode-fixed version
of the DBD::mysql. I already said that I can help with fixing code, but
nobody was interested.

> If you do not want that burden, I'd strongly advice you *NOT* to take
> on the job of (co)maintaining DBD::mysql. Period.

I already expressed that I do not want to be maintainer of DBD::mysql.

IMO problems with 2) are not possible to maintain.

> I *see* your wish to improve it (for whatever value of improvement).
> This was exactly the reason I took maint on Text::CSV_XS: I wanted,
> maybe even required, new features. The original author/maintainer was
> not interested and/or motivated enough to do it and he passed the
> module on to me. By that time I HAD NO IDEA what the impact of doing so
> had. NONE! *I* wanted new feature and improvements, and he agreed.
> 
> It took me years to fully understand the implications of my actions.
> 
> So, currently I test a release on 7+ architectures and over 150
> versions of perl before I release and 

Re: DBD::mysql path forward

[pali]

On Thursday 09 November 2017 10:32:57 Peter Rabbitson wrote:
> On 11/09/2017 09:54 AM, p...@cpan.org wrote:
> >
> >As those people or projects misuse some internals of perl we cannot
> >guarantee anything such that which may be broken or changed by updating
> >any module from cpan or external source not related to DBD::mysql.
> >
> 
> Pali. This "argument" applies to a large portion of CPAN. All JSON variants,
> various web frameworks, even things as mundane as math libraries:
> https://rt.cpan.org/Public/Bug/Display.html?id=123268

Yes, it applies and people should be aware of it...

> >As already stated more times, if there is some code which depends on
> >some internals, it should stay conserved in tested version as it can
> >break by updating anything (also unrelated). There is no other option
> >how to achieve that code stay working and does not corrupt something
> >else.
> 
> Pali, this is *not* how CPAN works. Nor is this even how /usr/bin/perl
> works.

It does not matter if cpan works like this or not. More important is
that in any world (does not have to be perl) above statement is truth.
Once you are starting depending on some internals, then you must be
aware of it that internals may change at anytime if you update one
component in ecosystem.

> >Maintaining such thing is something what I believe nobody wants. For
> >sure I'm not.
> >
> 
> I will state several very obvious ( to me ) things, apologies if they seem
> condescending: I simply do not know whether the participants on this thread
> are talking about the same thing.
> 
> Pali's entire premise is flawed. In the grand scheme of things it makes
> absolutely no difference what "maintainers want". Zero.
> 
> What the complaints in this thread are focused on is what the *users* want.

As already wrote, if users want or need to depend on internals which are
subject to change or which can be changed by something irrelevant to
DBD::mysql, maintainers nor any other developers of DBD::mysql cannot do
anything with it. The only thing what users can do is not to updating
their ecosystem OR fixing their code to not use internals.

Basically if somebody is asking for something which is either not
possible or hard to achieve, then there would be few or zero people
ready to serve such thing.

So here I would ask you: Are you going to develop, maintain and ensure
these users requirements for DBD::mysql? If not, are you able to find
another people who will do it?

I spent non-trivial time on looking at current code and I could say that
above requirement is really really hard to achieve when DBD::mysql would
be actively developed.

And because current DBD::mysql is not possible to compile with last
MariaDB nor with last (beta) MySQL and fixing these parts needs active
development, it means that such requirement is even more hard.

So if somebody depends on internals, then is already bound to some MySQL
or MariaDB version (last which works with DBD::mysql) and is already
locked for upgrading MySQL/MariaDB.

> And every maintainer, Pali included, are irrevocably bound by the general
> direction of wishes of the userbase.

I think last time we did most what we could...

> The wishes are roughly ( and in order of inmportance ):
> 
> - Breakage is never silent and is rectified swiftly when reported ( and is
> ideally never argued about )

Michiel announced more times changes to list, wrote blog posts, released
two (or more?) trial releases to cpan and waited for a longer time
before releasing final version. Discussion about patches were on both RT
and github open...

What else could be done there?

> - Upgrades are noneventful as a whole, and are as modular as possible ( i.e.
> a DBD upgrade does not drag in Test2 or something similarly non-relevant )

DBD::mysql is just one module, there is not possible to make it modular.

> - The overall library seems to move in a "better direction" ( more
> performance, less memory use, support for newer protocols, etc )

New versions fixed some problems with both new and older MySQL and
MariaDB versions, added more automated tests for different versions.

> Satisfy the above - and you do get the privilege of being a maintainer of a
> central module with 15+ years of history.

Why should I be interested in maintaining something from which users
want impossible things?

> If one can't - then the user-base is better off with the module remaining
> "unmaintained" ( reusing Pali's terminology ), and Pali's brave-new-world
> plan ending up in a different namespace. DBD::MariaDB is an idea.
> 
> Cheers

Re: DBD::mysql path forward

[pali]

Hi! I'm responding below.

On Tuesday 07 November 2017 13:19:23 Darren Duncan wrote:
> Patrick and Pali, each of you please respond to the lists to confirm that
> what I say below is what you also understand is the primary plan, and if
> not, then say why not; in prior discussion I recall you agreed with it.
> 
> Assuming they agree, the concerns of Night Light and those he is speaking
> for should be handily satisfied.
> 
> The whole discussion on the mailing lists that I recall and participated in
> seemed to consensus on branching DBD::sqlite in order to best satisfy the
   ~~
   DBD::mysql

I hope you are talking about DBD::mysql there...

> needs of all stakeholders.
> 
> There would be one version control with 2 active release branches.
> 
> A maintenance branch would exist starting from the 4.041/3 release on which
> further stable releases named DBD::mysql 4.x would be made, and the primary
> goal of this branch is to not break/corrupt anything that relied on 4.041
> behavior. So people with legacy projects that just use DBD::mysql and made
> particular assumptions on its handling of Unicode or Blobs etc won't have
> corruption introduced because DBD::mysql changed its handling of those
> things.

As those people or projects misuse some internals of perl we cannot
guarantee anything such that which may be broken or changed by updating
any module from cpan or external source not related to DBD::mysql.

Maintaining such thing is something what I believe nobody wants. For
sure I'm not.

As already stated more times, if there is some code which depends on
some internals, it should stay conserved in tested version as it can
break by updating anything (also unrelated). There is no other option
how to achieve that code stay working and does not corrupt something
else.

> People without knowledge of CPAN or the development process will
> get something that just continues to work for them and doesn't corrupt.

As explained above (and also in past) it is not possible to guarantee.

> For
> all intents and purposes this branch would be frozen but could have select
> security or bug fixes that comply with the mandate and don't involve
> changing Unicode etc.

Are you going to maintain that branch with all problems which it brings?

Or is there anybody else who want to maintain such crap?

If not, then it does not make sense to talk about this. Because nobody
expressed that is going to do such thing for 6 months which is really
large amount of time.

> The master branch would then have all the short term breaking changes
> including the 4.042 Unicode fixes and would have all the significant feature
> changes including compatibility with newer MySQL major versions.  All of its
> releases would be under a new namespace such as DBD::mysql2 and start at
> version 5.0 to signify that large changes happened which might possibly
> break code or cause corruption if user code doesn't specifically account for
> the differences.  Being a different name this is strictly opt-in, so the
> only ones using DBD::mysql2 are those that explicitly opted in to it, and by
> doing so they also opted in to fixing anything with their code that needed
> fixing to not corrupt data while using it.
> 
> The concept of having a single driver with toggled behavior eg a
> mysql_enable_proper_unicode flag was already rejected as unwieldy to
> implement, especially for such deep rooted changes as properly handling
> Unicode.
> 
> Note that I expect that most other higher-level projects that use MySQL
> would NOT branch like this, instead having internal logic or their own
> toggle to work with DBD::mysql vs DBD::mysql2, or a few may branch, but the
> key thing is that branching DBD::mysql does NOT necessitate doing so in the
> rest of the ecosystem.
> 
> -- Darren Duncan
> 
> On 2017-11-07 12:07 PM, Night Light wrote:
> >Proposed, but no made decisions posted afterwards.
> >
> >The last proposal is to re-commit the rejected 4.042 changes into the 4.043
> >master branch and only work on fixes that came after June.
> >
> >The git issue that regards improperly encoding of BLOBs is opened on April 
> >6th
> >(hence me sending the message to prevent a recurring cycle).
> >
> >https://github.com/perl5-dbi/DBD-mysql/issues/117
> >
> >On Tue, Nov 7, 2017 at 8:57 PM, Michiel Beijen wrote:
> >
> >To me, the only real option is to make a new option in DBD::mysql;
> >mysql_enable_proper_unicode or the like which you would knowingly set
> >in your application code, which will expose the new behaviour. I
> >understand this is difficult, but I really think it's the only way.
> >
> >If in th

Re: Fork DBD::mysql

[pali]

On Wednesday 11 October 2017 15:09:49 Steven Lembark wrote:
> 
> > It is not as easy as it could appear. And also in some cases migration
> > from MySQL/MariaDB to Pg could be problematic from performance point of
> > view. One Pg developer already told us that for our use case is really
> > MySQL better then Pg.
> 
> Q: What about your use case is more adapted to MySQL? 
> 
> There is no part of SQL that Pg does not support that MySQL does; there
> should not be any serious performance issues with Pg that leave it 
> slower than MySQL. There are a variety of ways that Pg can be faster
> (e.g., partial indexes, exclusion constraints vs. triggers) and will
> usually be less error-prone. You may have to refactor some sloppy design
> that MySQL allowed but Pg does not, but that is also in your favor.
> 
> I really am curious to see any example of something in your database that
> can be handled more gracefully in MySQL than well-designed Pg.

Hi! The use case is massive updates of existing columns in huge table.
Updates of columns in Pg are done just in linked-list until autovacuum
cleanups older values. If there are too many updates, then select
queries are slower. I do not know if new version of Pg have better
performance for such use case, but year (or two?) I was told that by Pg
developer that for such usage is MySQL still better.

Re: DBD::mysql path forward

[pali]

On Tuesday 26 September 2017 14:20:33 Night Light wrote:
> That's a nifty function. Good to know that it can be reversed.

UTF-8 encode is a function which for any number from the range
0..1114111 assign unique sequence of the numbers 0..255.

Therefore this function has a well defined inverse - UTF-8 decode
function.

As a sequence of numbers from the range 0..1114111 via UTF-8 encode
function produce sequence of the numbers in range 0..255 (length of
sequence would be larger) it can be again used as as input for the UTF-8
encode function.

And because output from the UTF-8 encode has well defined inverse, you
can easily reconstruct also inverse of the composition of the more UTF-8
functions.

Take string $str and following pass:

decode('UTF-8', decode('UTF-8', encode('UTF-8', encode('UTF-8', $str eq 
$str;

To have exactly correct result, you just need to know how many times you
composed repeated call to UTF-8 encode function.

Re: DBD::mysql path forward

[pali]

On Tuesday 26 September 2017 14:20:33 Night Light wrote:
> I noticed that it was mentioned in DBD::mysql # 117 that DBD::mysql does
> know that a column in a returned resultset is a BLOB.

For returned values via MySQL protocol, there are two different and
independent things: SQL type and encoding. Which means that *returned*
data of type BLOB have encoding, and it can be encoded in UTF-8.

Moreover, MySQL server is known to send TEXT columns as type BLOB with
indication of the UTF-8 encoding.

> One should also consider the performance penalty of encoding/decoding
> BLOB's as they can be big.

Patches which were part of 4.042 release called UTF-8 decoding routine
for retrieved data only if MySQL server told that data are UTF-8
encoded. And independently of the type. Of course there is also "binary"
encoding in MySQL protocol and server can use it.

UTF-8 decoding has linear time complexity (in case of valid input data).
For large data it could be performance penalty, but returning *wrong*
and *invalid* data is worse as returning data slower.

Also note that MySQL SQL language has a functions for conversion, so it
is up to programmer to write SQL statement in correct way. There is a
MySQL SQL function to convert string into binary...

Also note that for non-served-side-prepared-statements the only
communication protocol between client and server is text based protocol.
And converting binary data to text on both side (client C library and
server library) would have probably bigger penalty as some UTF-8 decoder
in DBD driver.

Re: DBD::mysql path forward

[pali]

Hello, may I ask how this situation differs from upgrading DBD::Pg to 
version 3.3.0 or DBD::SQLite to version 1.43_04? What you are writing 
there basically affects also DBD::SQLite... And version 1.43_04 was 
already released...

On Tuesday 19 September 2017 14:46:09 Night Light wrote:
> Dear Perl gurus,
> 
> This is my first post. I'm using Perl with great joy, and I'd like to
> express my gratitude for all you are doing to keep Perl stable and
> fun to use.
> 
> I'd like to ask to object to re-releasing this version and discuss on
> how to make 4.043 backwards compatible instead.
> This change will with 100% certainty corrupt all BLOB data written to
> the database when the developer did not read the release notes
> before applying the latest version of DBD::mysql (and changed its
> code consequently). Knowing that sysadmins have the habit of not
> always reading the release notes of each updated package the
> likelihood that this will happen will therefore high.
> I myself wasn't even shown the release notes as it was a dependency
> of an updated package that I applied.
> The exposure of this change is big as DBD::mysql affects multiple
> applications and many user bases.
> I believe deliberately introducing industry wide database corruption
> is something that will significantly harm peoples confidence in
> using Perl. I believe that not providing backwards compatibility is
> not in line with the Perl policy that has been carefully put
> together by the community to maintain the quality of Perl as it is
> today.
> http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DE
> PRECATION
> 
> I therefore believe the only solution is an upgrade that is by
> default backwards compatible, and where it is the user who decides
> when to start UTF8 encode the input values of a SQL request instead.
> If it is too time consuming or too difficult it should be considered
> to park the UTF8-encoding "fix" and release a version with the
> security fix first.
> 
> I have the following objections against this release:
> 
> 1. the upgrade will corrupt more records than it fixes (it does more
> harm than good)
> 2. the reason given for not providing backward compatibility
> ("because it was hard to implement") is not plausible given the
> level of unwanted side effects.
>This especially knowing that there is already a mechanism in place
> to signal if its wants UTF8 encoding or not
> (mysql_enable_utf8/mysql_enable_utf8mb4).
> 3. it costs more resources to coordinate/discuss a "way forward" or
> options than to implement a solution that addresses backwards
> compatibility 4. it is unreasonable to ask for changing existing
> source knowing that depending modules may not be actively maintained
> or proprietary It can be argued that such module should always be
> maintained but it does not change the fact that a good running Perl
> program becomes unusable 5. it does not inform the user that after
> upgrading existing code will start write corrupt BLOB records
> 6. it does not inform the user about the fact that a code review of
> all existing code is necessary, and how it needs to be changed and
> tested 7. it does not give the user the option to decide how the
> BLOB's should be stored/encoded (opt in)
> 8. it does not provide backwards compatibility
>By doing so it does not respect the Perl policy that has been
> carefully put together by the community to maintain the quality of
> Perl as it is today.
> 
> http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DE
> PRECATION 9. it blocks users from using DBD::mysql upgrades as long
> as they have not rewritten their existing code
> 10. not all users from DBD::mysql can be warned beforehand about the
> side effects as it is not known which private parties have code that
> use DBD::mysql
> 12. I believe development will go faster when support for backwards
> compatibility is addressed
> 13. having to write 1 extra line for each SQL query value is a monks
> job that will make the module less attractive to use
> 
> About forking to DBD::mariadb?:
> The primary reason to create such a module is when the communication
> protocol of Mariadb has become incompatible with Mysql.
> To use this namespace to fix a bug in DBD::mysql does not meet that
> criteria and causes confusion for developers and unnecessary
> pollution of the DBD namespace.
> 
> ---
> 
> For people that do not know the impact of the change that is pending
> to be committed:
> (see Github issue that includes 3 reports of companies that suffered
> data loss https://github.com/perl5-dbi/DBD-mysql/issues/117 )
> 
> Issue: some UTF8 characters are not properly displayed after
> retrieval Cause: SQL query values are not UTF8 encoded when sent to
> the database but they are all decoded once retrieved.
> Occurence: Only records with string data that can only be written
> with UTF8. It can be considered rare as people haven't reported this
> issue after 10 years of usage.
> Regional 

Re: DBD::mysql path forward

[pali]

On Friday 15 September 2017 04:06:57 Patrick M. Galbraith wrote:
> On 9/13/17 9:31 AM, p...@cpan.org wrote:
> > On Tuesday 12 September 2017 11:32:36 Darren Duncan wrote:
> >> On 2017-09-12 11:05 AM, p...@cpan.org wrote:
> >>> On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:
> >>>> I strongly recommend that another thing happen, which is
> >>>> re-versioning DBD::mysql to 5.0.
> >>>> 
> >>>> 1. From now on, DBD::mysql versions 4.x would essentially be
> >>>> frozen at 4.041/4.043.
> >>>> 
> >>>> 2. From now on, DBD::mysql versions 5.x and above would be where
> >>>> all active development occurs, would be the only place where
> >>>> Unicode handling is fixed and new MySQL versions and features
> >>>> are supported, and where other features are added.
> >>> 
> >>> I'm fine with it. Basically it means reverting to pre-4.043 code
> >>> and continue *git* development there with increased version to
> >>> 5.x. And once code is ready it can be released to cpan as normal
> >>> (non-engineering) 5.x release.
> >> 
> >> Yes, as you say.  With respect to Git I propose doing this
> >> immediately:
> >> 
> >> 1. Create a Git tag/branch off of 4.043 which is the 4.x legacy
> >> support branch.
> > 
> > This is up to Patrick.
> 
> I agree that we can start there. I do like the discussion going on
> now and want to engage all those concerned and interested and have a
> plan.
> 
> >> 2. Revert Git master to the pre-4.043 code and then follow that
> >> with a commit to master that changes the DBD::mysql version to
> >> 5.0.
> > 
> > If everybody agree with this step, I can prepare pull request which
> > revert tree to this state, plus re-apply/rebase commits which were
> > newly accepted.

I prepared branch master-new, which is based on current DBD-mysql master 
branch and revert state to pre-4.043 version, including all changes done 
after 4.043 release to master branch. I have this master-new branch in 
my fork. If you want you can use it...

https://github.com/pali/DBD-mysql/tree/master-new

> >> Regardless, following point 2, mandate that all Git pull requests
> >> are made against the new 5.x master; the 4.x legacy branch would
> >> have no commits except minimal back-porting.
> > 
> > New pull requests are by default created against "master" branch.
> > So if 5.x development would happen in "master" and 4.x in e.g.
> > "legacy-4.0" then no changes is needed.
> > 
> > But on github it is not possible to disallow users to create new
> > pull requests for non-default branch. When creating pull request
> > there is a button which open dialog to change target branch.

Re: DBD::mysql path forward

[pali]

On Thursday 14 September 2017 08:21:42 H.Merijn Brand wrote:
> On Wed, 13 Sep 2017 13:27:58 -0700, Darren Duncan
>  wrote:
> 
> > On 2017-09-13 12:58 PM, Dan Book wrote:
> > > On Wed, Sep 13, 2017 at 3:53 AM, Peter Rabbitson wrote:
> > >
> > > On 09/12/2017 07:12 PM, p...@cpan.org wrote:
> > > And here is promised script:
> > > 
> > >
> > > The script side-steps showcasing the treatment of BLOB/BYTEA columns, 
> > > which
> > > was one of the main ( albeit not the only ) reason the userbase lost 
> > > data.
> > >
> > > Please extend the script with a BLOB/BYTEA test.
> > >
> > > I'm not sure how to usefully make such a script, since correct insertion 
> > > of BLOB
> > > data (binding with the SQL_BLOB type or similar) would work correctly both
> > > before and after the fix.  
> > 
> > Perhaps the requirement of the extra tests is to ensure that BLOB/BYTEA 
> > data is 
> > NOT mangled during input or output, that on input any strings with a true 
> > utf8 
> > flag are rejected and that on output any strings have a false utf8 flag.  
> > Part 
> > of the idea is to give regression testing that changes regarding Unicode 
> > handling with text don't inadvertently break blob handling. -- Darren Duncan
> 
> BYTE/BLOB/TEXT tests require three types of data
> 
> • Pure ASCII
> • Correct UTF-8 (with complex combinations)

subtest: Correct UTF-8 TEXT with only code points in range U+00 .. U+7F (ASCII 
subset)
subtest: Correct UTF-8 TEXT with only code points in range U+00 .. U+FF (Latin1 
subset)

> • Pure binary
>   - random bytes ranging 0x00..0xFF
>   - Images (png, jpg, gif, tiff)
> 
> None of those is hard to generate.
> 
> 1. create two tables with a field of the type to check
> 2. insert the data in table 1
> 3. use SQL to copy the data to table 2
> 4. extract data from table 2
> 5. compare to original data
> 6. drop tables
> 7. goto 1
> 
> If MySQL support different ways to do this, test all ways (in the
> string, placeholders, bind_columns, bind_params, other ...)

MySQL server and its databases has some limitations, so reflect it:

* it does not provide information if placeholder is TEXT, VARCHAR, VARBINARY or 
BLOB
* placeholder's bind value does not have to point to column, it can be also SQL 
function
  --> for caller/user all placeholders are equivalent and caller itself
  needs to know how to treat bind variable and needs to specify if
  it is TEXT or BLOB

* VARBINARY is right padded with 0x00
  --> there is no difference between binary "\x01\x02\x00" and "\x01\x02"

And note that perl itself does not distinguish between "binary bytes"
and "unicode string". User itself needs to know what he should expect in
perl scalar and how to handle it.

And if you are thinking about utf8 flag of scalar (utf8::is_utf8()) to
use as way how to distinguish character and binary data, then do not try
to do it.

In (updated) documentation in utf8.pm module is:
https://metacpan.org/pod/release/WOLFSAGE/perl-5.27.3/lib/utf8.pm

  Don't use this flag as a marker to distinguish character and binary
  data, that should be decided for each variable when you write your
  code.

Re: DBD::mysql path forward

[pali]

On Tuesday 12 September 2017 11:32:36 Darren Duncan wrote:
> On 2017-09-12 11:05 AM, p...@cpan.org wrote:
> >On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:
> >>I strongly recommend that another thing happen, which is
> >>re-versioning DBD::mysql to 5.0.
> >>
> >>1. From now on, DBD::mysql versions 4.x would essentially be frozen
> >>at 4.041/4.043.
> >>
> >>2. From now on, DBD::mysql versions 5.x and above would be where all
> >>active development occurs, would be the only place where Unicode
> >>handling is fixed and new MySQL versions and features are supported,
> >>and where other features are added.
> >
> >I'm fine with it. Basically it means reverting to pre-4.043 code and
> >continue *git* development there with increased version to 5.x. And once
> >code is ready it can be released to cpan as normal (non-engineering) 5.x
> >release.
> 
> Yes, as you say.  With respect to Git I propose doing this immediately:
> 
> 1. Create a Git tag/branch off of 4.043 which is the 4.x legacy support 
> branch.

This is up to Patrick.

> 2. Revert Git master to the pre-4.043 code and then follow that with a
> commit to master that changes the DBD::mysql version to 5.0.

If everybody agree with this step, I can prepare pull request which
revert tree to this state, plus re-apply/rebase commits which were newly
accepted.

> Regardless, following point 2, mandate that all Git pull requests are made
> against the new 5.x master; the 4.x legacy branch would have no commits
> except minimal back-porting.

New pull requests are by default created against "master" branch. So if
5.x development would happen in "master" and 4.x in e.g. "legacy-4.0"
then no changes is needed.

But on github it is not possible to disallow users to create new pull
requests for non-default branch. When creating pull request there is a
button which open dialog to change target branch.

Re: DBD::mysql path forward

[pali]

I would suggest to communicate with all people behind cpan modules which
uses mysql_enable_utf8.

https://grep.metacpan.org/search?size=20=mysql_enable_utf8==

They should know if their modules are expecting old broken or behavior
of DBD::mysql or not. I would expect that are modules which needs
correct behavior as not all perl people expect such bugs...

They can update their modules to be compatible with both old buggy
behavior and new correct. Months ago I wrote update for DBD::mysql POD
documentation which mention example how to achieve it:

https://github.com/perl5-dbi/DBD-mysql/pull/119/files

Meanwhile DBD::mysql can try to play with "conflicts" relationship in
META.json to prevent installing one of the above cpan module which
depends on broken DBD::mysql until maintainer of that module fixes it.

https://metacpan.org/pod/CPAN::Meta::Spec#Prereq-Spec

I have never uses "conflicts" in META.json, so I do not know if it is
working correctly in all cpan clients, but according to documentation it
is a way how to achieve that users would not upgrade DBD::mysql to new
version if they have installed another module which needs old
DBD::mysql.

Has anybody experience with "conflicts" in META.json?

Also this would mean to prepare list of all of those cpan modules and
add them into "conflicts" section in DBD::mysql's META.json.

This can be tested e.g. by releasing engineering/beta version on cpan.

On Tuesday 12 September 2017 21:45:11 Patrick M. Galbraith wrote:
> Darren,
> 
> I agree with this as well, with the exception of 4 and 5, keeping 5.0 "pure"
> to the new way of doing things.
> 
> For releases, I think I want to understand what this will mean. Sooner or
> later, a release shows up in as a distribution package, installed on the OS,
> and I want to know what the way of communicating that to the users and
> vendors so expectations are met. That's another question of "how do we
> handle that?" and "how do we inform an OS packager/vendor, what to do?
> 
> Thank you for the great discussion!
> 
> Patrick
> 
> On 9/12/17 2:05 PM, p...@cpan.org wrote:
> >On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:
> >>On 2017-09-12 8:54 AM, Dan Book wrote:
> >>>On Tue, Sep 12, 2017 at 11:04 AM, Patrick M. Galbraith wrote:
> >>> Pali,
> >>> Yes, I agree, we'll have to create a fork pre revert and stop
> >>> accepting PRs How might we allow people time to test the fixes
> >>> to give them time? Just have them use the fork, I would
> >>> assume?
> >>>
> >>>To be clear, this sounds like a branch not a fork. If your plan is
> >>>to reinstate the mysql_enable_utf8 behavior as in 4.042 rather
> >>>than adding a new option for this behavior, then branching from
> >>>4.042 seems reasonable to me; but you should be very clear if this
> >>>is your intended approach, as this is what led to many people
> >>>corrupting data as they send blobs to mysql with the same
> >>>mysql_enable_utf8 option, and expect them to accidentally not get
> >>>encoded.
> >>Assuming that broken Unicode handling has been in DBD::mysql for a
> >>long time and that users expect this broken behavior and that fixing
> >>DBD::mysql may break user code making those assumptions...
> >>
> >>I strongly recommend that another thing happen, which is
> >>re-versioning DBD::mysql to 5.0.
> >>
> >>A declared major version change would signify to a lot of people that
> >>there are significant changes from what came before and that they
> >>should be paying closer attention and expecting the possibility that
> >>their code might break unless they make some adjustments.  Without
> >>the major version change they can more easily and reasonably expect
> >>not having compatibility breaks.
> >>
> >>Part and parcel with this is that only DBD::mysql 5.0 would have the
> >>other changes required for compatibility with newer MySQL versions
> >>or features that would be the other more normal-sounding reasons to
> >>use it.
> >>
> >>So I specifically propose the following:
> >>
> >>1. From now on, DBD::mysql versions 4.x would essentially be frozen
> >>at 4.041/4.043.  They would expressly never receive any breaking
> >>changes (but see point 3) and in particular no Unicode handling
> >>changes.  They would expressly never receive any new features.  This
> >>is the option for people whose codebases and environments work now
> >>and want to leave it alone.
> >>
> >>2. From now on, DBD::mys

Re: DBD::mysql path forward

[pali]

On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:
> On 2017-09-12 8:54 AM, Dan Book wrote:
> > On Tue, Sep 12, 2017 at 11:04 AM, Patrick M. Galbraith wrote:
> > Pali,
> > Yes, I agree, we'll have to create a fork pre revert and stop
> > accepting PRs How might we allow people time to test the fixes
> > to give them time? Just have them use the fork, I would
> > assume?
> > 
> > To be clear, this sounds like a branch not a fork. If your plan is
> > to reinstate the mysql_enable_utf8 behavior as in 4.042 rather
> > than adding a new option for this behavior, then branching from
> > 4.042 seems reasonable to me; but you should be very clear if this
> > is your intended approach, as this is what led to many people
> > corrupting data as they send blobs to mysql with the same
> > mysql_enable_utf8 option, and expect them to accidentally not get
> > encoded.
> 
> Assuming that broken Unicode handling has been in DBD::mysql for a
> long time and that users expect this broken behavior and that fixing
> DBD::mysql may break user code making those assumptions...
> 
> I strongly recommend that another thing happen, which is
> re-versioning DBD::mysql to 5.0.
> 
> A declared major version change would signify to a lot of people that
> there are significant changes from what came before and that they
> should be paying closer attention and expecting the possibility that
> their code might break unless they make some adjustments.  Without
> the major version change they can more easily and reasonably expect
> not having compatibility breaks.
> 
> Part and parcel with this is that only DBD::mysql 5.0 would have the
> other changes required for compatibility with newer MySQL versions
> or features that would be the other more normal-sounding reasons to
> use it.
> 
> So I specifically propose the following:
> 
> 1. From now on, DBD::mysql versions 4.x would essentially be frozen
> at 4.041/4.043.  They would expressly never receive any breaking
> changes (but see point 3) and in particular no Unicode handling
> changes.  They would expressly never receive any new features.  This
> is the option for people whose codebases and environments work now
> and want to leave it alone.
> 
> 2. From now on, DBD::mysql versions 5.x and above would be where all
> active development occurs, would be the only place where Unicode
> handling is fixed and new MySQL versions and features are supported,
> and where other features are added.  Version 5.0 specifically would
> have all of the backwards-breaking changes at once that are
> currently planned or anticipated in the short term, in particular
> fixing the Unicode.  Anyone who is already making changes to their
> environment by moving to a newer MySQL version or want newer feature
> support will have to use DBD::mysql 5, and in the process they will
> have to go through their Perl codebase and fix anything that assumes
> Unicode is broken.
> 
> 3. As an exception to the complete freeze otherwise mentioned, there
> could be one or more DBD::mysql 4.x release whose sole purpose is to
> back-port security fixes or select other bug fixes from 5.x.  Also
> 4.x could gain minimalist documentation changes to indicate its new
> legacy status and point to 5.x.
> 
> 4. Optional bonus 1:  If it is reasonably possible to support both
> correct Unicode handling as well as the old broken handling in the
> same codebase, I suggest DBD::mysql 5.0 could also have a user
> config option where one could explicitly set it to make DBD::mysql
> use the old broken behavior, while the default would be to have the
> correct behavior.  This would be analogous to Perl's "use
> experimental" feature.  The idea is that it would provide a
> temporary stopgap for users migrating from 4.x where they could just
> make the minimal change of enabling that option but they otherwise
> wouldn't yet have to go through their codebase to fix all the
> assumptions of wrongness.  Using this option would cause deprecation
> warnings to be emitted.  Perhaps the feature could be lexical or
> handle-specific if appropriate to help support piecemeal migration
> to correct assumptions.  Alternately it may be be best to not have
> this option at all and people simply have to fix their code on
> moving to 5.x. This matter is probably its own discussion, but the
> key part is it only applies to 5.x, and meanwhile the 5.x default is
> correct Unicode while 4.x freezes the old bad Unicode behavior.
> 
> 5. Optional bonus 2:  A user config option could be turned on but all
> that it does is emit warnings in situations where the Unicode
> behavior was fixed, to prod users to check that particular spot in

Re: DBD::mysql path forward

[pali]

On Tuesday 12 September 2017 12:27:25 p...@cpan.org wrote:
> To prove fact that other DBI drivers (e.g. Pg or SQLite) had fixed
> similar/same UTF-8 issue as MySQL has and behave Perl-correctly, I
> would provide test cases so you would see difference between Pg,
> SQLite and mysql DBI drivers.

And here is promised script:

===

use strict;
use warnings;
use utf8;
binmode \*STDOUT, ':utf8';
binmode \*STDERR, ':utf8';
use DBI;

sub ords { $_[0] . ' (' . (join ', ', map ord, split //, $_[0]) . ')' }

# DBD::Pg enables Unicode <--> UTF-8 automatically
my $pg_dbh = DBI->connect("dbi:Pg:dbname=postgres", "", "");

# DBD::mysql needs mysql_enable_utf8 and mysql_enable_utf8mb4 for Unicode <--> 
UTF-8
my $mysql_dbh = DBI->connect("dbi:mysql:dbname=mysql", "root", "", { 
mysql_enable_utf8 => 1, mysql_enable_utf8mb4 => 1 });
# MySQL server by default uses latin1
# Without switch to utf8mb4, Unicode code points above U+FF cannot be stored (á 
is below U+FF, č not)
# Another option is to specify own charset in CREATE TABLE
$mysql_dbh->do("SET character_set_database='utf8mb4'");
$mysql_dbh->do("SET character_set_server='utf8mb4'");

# DBD::SQLite needs sqlite_unicode for Unicode <--> UTF-8
my $sqlite_dbh = DBI->connect("dbi:SQLite:dbname=:memory:", "", "", { 
sqlite_unicode => 1 });

for my $dbh ($pg_dbh, $mysql_dbh, $sqlite_dbh) {
  for my $ins ("\xC3\xA1", "\N{U+C3}\N{U+A1}", "á", "č", "\x{263A}", 
"\N{U+263A}", "☺") {
{
  $dbh->do("CREATE TEMPORARY TABLE t(s VARCHAR(10))");
  my $sth = $dbh->prepare("INSERT INTO t(s) VALUES('$ins')");
  $sth->execute();
  my $fetch = $dbh->selectrow_array("SELECT s FROM t");
  print $dbh->{Driver}->{Name} . ' without bind - insert: ' . ords($ins) . 
'; fetch: ' . ords($fetch) . ' - ' . ($ins eq $fetch ? 'OK' : 'FAIL') . "\n";
  $dbh->do("DROP TABLE t");
}
{
  $dbh->do("CREATE TEMPORARY TABLE t(s VARCHAR(10))");
  my $sth = $dbh->prepare("INSERT INTO t(s) VALUES(?)");
  $sth->execute($ins);
  my $fetch = $dbh->selectrow_array("SELECT s FROM t");
  print $dbh->{Driver}->{Name} . ' with bind - insert: ' . ords($ins) . '; 
fetch: ' . ords($fetch) . ' - ' . ($ins eq $fetch ? 'OK' : 'FAIL') . "\n";
  $dbh->do("DROP TABLE t");
}
  }
}

===

It is simple "identity" script, which stores value into database,
fetches it back and then compare results.

Look on results with DBD::mysql 4.041, 4.042 and 4.043.
Are all users of DBD::mysql aware of such results?

And specially look at results with DBD::Pg prior to version 3.3.0 and up.
Also DBD::SQLite prior to version 1.43_04.

Basically DBD::Pg (pg_enable_utf8 = -1) and DBD::SQLite
(via sqlite_unicode=1) had exactly same bug as DBD::mysql.

Re: DBD::mysql path forward

[pali]

Another option is to revert that "big revert commit" in git and continue 
development. Then people would be able to create pull request without 
problem...

Just need to be sure that new normal release would not be put to cpan. 
But it would be possible to put beta/engineering releases like it was 
between 4.041 and 4.042.

Any comments from other people about this idea?

On Tuesday 12 September 2017 17:04:18 Patrick M. Galbraith wrote:
> Pali,
> 
> Yes, I agree, we'll have to create a fork pre revert and stop
> accepting PRs
> 
> How might we allow people time to test the fixes to give them time?
> Just have them use the fork, I would assume?
> 
> Regards,
> 
> Patrick
> 
> On 9/12/17 6:27 AM, p...@cpan.org wrote:
> > On Tuesday 12 September 2017 05:40:31 Patrick M. Galbraith wrote:
> >> Hi all,
> >> 
> >> After talking to Pali and looking at how other drivers have
> >> handled the issue, the best way forward will be to deal with
> >> solving the UTF-8 issue correctly as was attempted in May. This
> >> will be a problem for some, but only due to having to been
> >> accustomed to a work-around that relies on a intermittent and
> >> buggy implementation. The plan is to give ample time to let
> >> people prepare and comment as well as give them a chance to try
> >> the changes and adjust prior to stable release. I'm going to look
> >> through the mailing list and find particularly those who had
> >> problems back in May and June when we tried this before and work
> >> with them in advance of the change.
> > 
> > Hi!
> > 
> > To prove fact that other DBI drivers (e.g. Pg or SQLite) had fixed
> > similar/same UTF-8 issue as MySQL has and behave Perl-correctly, I
> > would provide test cases so you would see difference between Pg,
> > SQLite and mysql DBI drivers.
> > 
> > There are also many modules on cpan which uses mysql DBI driver.
> > But I'm not sure if all those modules expects buggy behavior or
> > not...
> > 
> >> DBD::mysql should be on par with all the other DBD drivers and
> >> allow transparent usage regardless of backing RDBMS, and this
> >> really is the only way. It will also allow for other fixes to
> >> proceed and the driver to continue improving and supporting all
> >> versions and enhancements of MySQL and MariaDB.
> >> 
> >> Please do give us your thoughts on this as we want to be as
> >> helpful and transparent as possible.
> > 
> > If we are going to have UTF-8 fix in DBD::mysql, I would suggest
> > one thing right now:
> > 
> > STOP MERGING AND ACCEPTING PULL REQUEST TO GITHUB DBD::MYSQL
> > PROJECT!
> > 
> > Reason is simple: Current git master repository at version 4.043
> > (against which are creating new pull requests) is reset to version
> > 4.041 and couple of changes, including lot of bug fixes from 4.042
> > are missing
> > 
> > People in 4.043 (and git master) re-discovered 4.041 bugs, which
> > were fixed in 4.042 and started to fixing them again, either
> > wrongly or in different way. Or just started complaining about
> > them, that they are again present.
> > 
> > Those (reverted) UTF-8 commits changes couple of lines, plus other
> > fixes depends on it. Once git master repository starts to
> > divergent, it would be very hard to incorporate UTF-8 fixes again.
> > 
> > And speaking by myself, for me it would be easier to create a fork
> > of DBD::mysql from point before big revert as trying to re-apply
> > and rebase those patches on top of master with couple of
> > changes...

Re: DBD::mysql path forward

[pali]

On Tuesday 12 September 2017 05:40:31 Patrick M. Galbraith wrote:
> Hi all,
> 
> After talking to Pali and looking at how other drivers have handled the
> issue, the best way forward will be to deal with solving the UTF-8 issue
> correctly as was attempted in May. This will be a problem for some, but only
> due to having to been accustomed to a work-around that relies on a
> intermittent and buggy implementation. The plan is to give ample time to let
> people prepare and comment as well as give them a chance to try the changes
> and adjust prior to stable release. I'm going to look through the mailing
> list and find particularly those who had problems back in May and June when
> we tried this before and work with them in advance of the change.

Hi!

To prove fact that other DBI drivers (e.g. Pg or SQLite) had fixed
similar/same UTF-8 issue as MySQL has and behave Perl-correctly, I would
provide test cases so you would see difference between Pg, SQLite and
mysql DBI drivers.

There are also many modules on cpan which uses mysql DBI driver. But I'm
not sure if all those modules expects buggy behavior or not...

> DBD::mysql should be on par with all the other DBD drivers and allow
> transparent usage regardless of backing RDBMS, and this really is the only
> way. It will also allow for other fixes to proceed and the driver to
> continue improving and supporting all versions and enhancements of MySQL and
> MariaDB.
> 
> Please do give us your thoughts on this as we want to be as helpful and
> transparent as possible.

If we are going to have UTF-8 fix in DBD::mysql, I would suggest one
thing right now:

STOP MERGING AND ACCEPTING PULL REQUEST TO GITHUB DBD::MYSQL PROJECT!

Reason is simple: Current git master repository at version 4.043
(against which are creating new pull requests) is reset to version 4.041
and couple of changes, including lot of bug fixes from 4.042 are
missing

People in 4.043 (and git master) re-discovered 4.041 bugs, which were
fixed in 4.042 and started to fixing them again, either wrongly or in
different way. Or just started complaining about them, that they are
again present.

Those (reverted) UTF-8 commits changes couple of lines, plus other fixes
depends on it. Once git master repository starts to divergent, it would
be very hard to incorporate UTF-8 fixes again.

And speaking by myself, for me it would be easier to create a fork of
DBD::mysql from point before big revert as trying to re-apply and rebase
those patches on top of master with couple of changes...

Re: Fork DBD::mysql

[pali]

On Friday 01 September 2017 17:01:49 Patrick M. Galbraith wrote:
> Hi Pali,
> 
> I would very much like to address these issues and fix the UTF issue you so
> kindly had a patch for that people had problems with and do so in a way that
> doesn't require forking. I'm very sorry about this as I was in between jobs
> and distracted but am free next week to discuss this. I am also talking to
> my former Mysql colleagues at Oracle and MariaDB (Reggie, Matt, Georg) about
> moving this forward and get the driver MySQL 8 compatible. I would like to
> have a discussion to make this happen. How would you-- and everyone
> interested in this list-- like to work with me and others to solve this
> issue?
> 
> Kind regards,
> 
> Patrick

Hi! Welcome back! I would prefer discussion here on the dbi-dev mailing
list, so other people can watch current situation and also participate
in development discussion in open form.

Re: Fork DBD::mysql

[pali]

On Friday 01 September 2017 13:36:13 H.Merijn Brand wrote:
> On Fri, 1 Sep 2017 13:08:17 +0200, p...@cpan.org wrote:
> 
> > On Tuesday 29 August 2017 22:08:09 Dan Book wrote:
> > > Though the reversion was made with good intentions of preserving back
> > > compat in the short term, it was accompanied by a promise of reapplying 
> > > the
> > > many important fixes that were reverted (
> > > https://github.com/perl5-dbi/DBD-mysql/compare/4.041...4.042 ) as well as
> > > re-adding a correct utf8 encoding option, as it is still necessary to fix
> > > this broken behavior one way or another. Since nothing has happened,
> > > forking may be the only reasonable option. We are currently version pinned
> > > to 4.042 as "upgrading" to 4.043 would be a massive regression in our
> > > codebase.
> > > 
> > > -Dan  
> > 
> > Ok, I understood Dan's email as there are already users who cannot
> > upgrade to the last DBD::mysql version (where revert happened) as it
> > contains bugs...
> > 
> > And due to big silent, I would say that DBD::mysql is really dead.
> 
> I doubt that. Maybe the active maintainer is on holiday. It is the
> season you know.
> 
> June is indeed a few months ago, but it was still this year. NOT dead.
> Please try to contact Michiel first.

I already did it and also I CCed him in my first email in this thread.

> Author: Michiel Beijen <michiel.bei...@gmail.com>  2017-06-29 11:25:48
> Committer: Michiel Beijen <michiel.bei...@gmail.com>  2017-06-29 11:30:07
> Tags: 4.043
> Parent: 1ece6b4b8630a1cf348373aa41cbe5f748dcd62a (Merge pull request #137 
> from pali/versions)
> Branch: remotes/origin/master
> Follows: 4.042
> Precedes: 
> 
> New version 4.043 - the same as 4.041

Re: Fork DBD::mysql

[pali]

On Tuesday 29 August 2017 22:08:09 Dan Book wrote:
> Though the reversion was made with good intentions of preserving back
> compat in the short term, it was accompanied by a promise of reapplying the
> many important fixes that were reverted (
> https://github.com/perl5-dbi/DBD-mysql/compare/4.041...4.042 ) as well as
> re-adding a correct utf8 encoding option, as it is still necessary to fix
> this broken behavior one way or another. Since nothing has happened,
> forking may be the only reasonable option. We are currently version pinned
> to 4.042 as "upgrading" to 4.043 would be a massive regression in our
> codebase.
> 
> -Dan

Ok, I understood Dan's email as there are already users who cannot
upgrade to the last DBD::mysql version (where revert happened) as it
contains bugs...

And due to big silent, I would say that DBD::mysql is really dead.

Re: Fork DBD::mysql

[pali]

On Monday 28 August 2017 09:19:46 Darren Duncan wrote:
> While a fork may be the best short term fix, as keeping up with security
> issues is important, I honestly believe that the best fix is to migrate to
> Postgres as soon as you can.

It is not as easy as it could appear. And also in some cases migration
from MySQL/MariaDB to Pg could be problematic from performance point of
view. One Pg developer already told us that for our use case is really
MySQL better then Pg.

Re: [External] Fork DBD::mysql

[pali]

Then users of MySQL would stay with DBD::mysql and they would not have
fixed DBD driver. We were thinking about choosing DBD::mariadb name
for our fork. But without dropping MySQL support, so also MySQL users
could benefit from it.

On Monday 28 August 2017 18:08:20 Daniël van Eeden wrote:
> What about splitting it in DBD::mysql and DBD::mariadb ?
> 
> 
> On 28 August 2017 5:55:09 p.m. p...@cpan.org wrote:
> 
> >Hello, 2 months passed since two security problems related to DBD::mysql
> >were reported, namely CVE-2017-10789 and CVE-2017-10788. Looking at the
> >DBD::mysql github project page and RT bugtracker, it can be seen that
> >nothing has happened for 2 months. Some people are asking or waiting for
> >reported problems to be fixed... Without an answer.
> >
> >A long discussion was started in April by a group of people who didn't
> >like the improvements and didn't want to see the bugs fixed in
> >DBD::mysql driver due to the fact that they probably had old and buggy
> >legacy code which wouldn't work after DBD::mysql starts fixing bugs.
> >
> >From what happened it looks like that this group of people forced
> >DBD::mysql maintainers to revert all DBD::mysql changes (with all bug
> >fixes including security ones). And because nothing has happened for 2
> >months, that looks like fixing those bugs is not planned.
> >
> >This means just one thing:
> >
> >  DBD::mysql is dead right now :-(
> >
> >There are no new commit for 2 months, no evidence of any activity. And
> >it looks like this is what that group of people wanted. They haven't
> >written anything, neither any comment about open security issues, nor
> >any comment that they forced maintainers to revert a security related
> >bug fix. Probably they don't have to fix their own code and they can use
> >the last released version of DBD::mysql. Perfect for them, probably a
> >catastrophe for all others which don't want to use probably-vulnerable
> >code.
> >
> >But it also means that it isn't possible to compile the last DBD::mysql
> >with the new MySQL 8 or new MariaDB versions. More linux distributions
> >started to pack/bundle their own DBD::mysql patches just because they
> >need to build DBD::mysql into their repositories with upgraded MariaDB.
> >
> >I need to say that such situation is not acceptable for *any* future or
> >new development in Perl which uses either MySQL or MariaDB database.
> >
> >DBD::mysql has couple of bugs (some are security released) and fully
> >non-working UTF-8/Unicode support. It means that any internationalized
> >applications which use MySQL are hard to write in Perl.
> >
> >Because of the current situation we in the GoodData company are thinking
> >about forking DBD::mysql. We use DBI not only with DBD::mysql, but also
> >with DBD::Pg. The annoying bugs in DBD::mysql still require hacks and
> >workarounds, while similar bugs were fixed in DBD::Pg years ago. Also,
> >inability to use the new version of MySQL or MariaDB is a problem. Same
> >for the open security issues.
> >
> >I would like to ask, whether somebody else is interested in DBD::mysql
> >fork? Either as a user or as a developer? Maintaining a fork isn't a
> >simple task, but with supporting users and contributing developers it
> >should be easier.
> >
> >And specially, are MariaDB developers interested in fixing/improving
> >DBD::mysql fork, so Perl could work better with MariaDB servers? At
> >least it would be able to compile against the new MariaDB C-client?
> >

Fork DBD::mysql

[pali]

Hello, 2 months passed since two security problems related to DBD::mysql
were reported, namely CVE-2017-10789 and CVE-2017-10788. Looking at the
DBD::mysql github project page and RT bugtracker, it can be seen that
nothing has happened for 2 months. Some people are asking or waiting for
reported problems to be fixed... Without an answer.

A long discussion was started in April by a group of people who didn't
like the improvements and didn't want to see the bugs fixed in
DBD::mysql driver due to the fact that they probably had old and buggy
legacy code which wouldn't work after DBD::mysql starts fixing bugs.

>From what happened it looks like that this group of people forced
DBD::mysql maintainers to revert all DBD::mysql changes (with all bug
fixes including security ones). And because nothing has happened for 2
months, that looks like fixing those bugs is not planned.

This means just one thing:

  DBD::mysql is dead right now :-(

There are no new commit for 2 months, no evidence of any activity. And
it looks like this is what that group of people wanted. They haven't
written anything, neither any comment about open security issues, nor
any comment that they forced maintainers to revert a security related
bug fix. Probably they don't have to fix their own code and they can use
the last released version of DBD::mysql. Perfect for them, probably a
catastrophe for all others which don't want to use probably-vulnerable
code.

But it also means that it isn't possible to compile the last DBD::mysql
with the new MySQL 8 or new MariaDB versions. More linux distributions
started to pack/bundle their own DBD::mysql patches just because they
need to build DBD::mysql into their repositories with upgraded MariaDB.

I need to say that such situation is not acceptable for *any* future or
new development in Perl which uses either MySQL or MariaDB database.

DBD::mysql has couple of bugs (some are security released) and fully
non-working UTF-8/Unicode support. It means that any internationalized
applications which use MySQL are hard to write in Perl.

Because of the current situation we in the GoodData company are thinking
about forking DBD::mysql. We use DBI not only with DBD::mysql, but also
with DBD::Pg. The annoying bugs in DBD::mysql still require hacks and
workarounds, while similar bugs were fixed in DBD::Pg years ago. Also,
inability to use the new version of MySQL or MariaDB is a problem. Same
for the open security issues.

I would like to ask, whether somebody else is interested in DBD::mysql
fork? Either as a user or as a developer? Maintaining a fork isn't a
simple task, but with supporting users and contributing developers it
should be easier.

And specially, are MariaDB developers interested in fixing/improving
DBD::mysql fork, so Perl could work better with MariaDB servers? At
least it would be able to compile against the new MariaDB C-client?