Re: DBD::mysql path forward

[Elizabeth Mattijsen]

You cannot go further without either breaking existing code.

You *can* fix the code in a fork.  This gives people the of switching to the 
fork when *they* want to.  Or don’t switch.

Seems to me forking is the only way forward.


Just my 2c worth, having been exposed to these shenanigans of point #1 
extensively at former $work.


Liz

> On 10 Nov 2017, at 15:03, p...@cpan.org wrote:
> 
> On Friday 10 November 2017 10:09:59 demerphq wrote:
>> Pali is there a concise summary of what we are arguing about here?
> 
> Short summary:
> 
> 1. There are applications which misuse perl and DBD::mysql internals to
>   pass blob or utf-8 text data from perl to MySQL database. We were not
>   aware of them for a longer testing period of time as nobody
>   complained until version 4.042 of DBD::mysql was released. There were
>   trial releases, blog posts, asking testers for test, etc...
> 
> 2. Unicode support in DBD::mysql is broken and cause data corruption.
>   There are lot of reported bugs about it, e.g.:
>   https://rt.cpan.org/Public/Bug/Display.html?id=25590
>   https://rt.cpan.org/Public/Bug/Display.html?id=53130
>   https://rt.cpan.org/Public/Bug/Display.html?id=60987
>   https://rt.cpan.org/Public/Bug/Display.html?id=87428
>   https://rt.cpan.org/Public/Bug/Display.html?id=119079
>   https://rt.cpan.org/Public/Bug/Display.html?id=120141
>   All those problems were fixed, but breaking applications which misuse
>   internals. Basically Unicode support is opposite of the above.
>   All fixes from 4.042 were reverted and version 4.043 was released.
> 
> 3. DBD::mysql itself misuse libmysqlclient internals for a long time and
>   now when MariaDB changed something, it cannot be used with DBD::mysql
>   anymore.
> 
> 4. Due to how Oracle (not) reacted and (not) fixed problems related to
>   SSL, DBD::mysql has a security problems when SSL encryption is used.
>   There are also other crashing problems in DBD::mysql.
> 
> 5. Misusing internals of 3rd applications which use DBD::mysql is a
>   problem because basically any change of low level parts in DBD::mysql
>   can lead to breakage of those 3rd applications. And because nobody
>   was able to detect it would break something there is a really big
>   chance that bigger changes in DBD::mysql break something again.
>   Fixing 3. is expected to be big change.

Re: DBD::mysql path forward

[Alexander Hartmaier]

I followed the discussion silently so far as I‘m not a user of DBD:mysql.
If the changes are breaking I‘d also think a new distribution is the way to go.
What about DBD::MariaDB as that‘s the name of the OpenSource version these days?

Best regards, Alex

> Am 10.11.2017 um 07:16 schrieb Darren Duncan :
> 
>> On 2017-11-09 8:32 AM, Dan Book wrote:
>> It seems to me like the remaining option that can make everyone "happy" is 
>> the
>> previously-suggested option of maintaining a legacy branch and doing new
>> development (reinstating 4.042) in another branch which will be released as a
>> new distribution, like DBD::mysql2, by the same maintainers. (I would not 
>> favor
>> DBD::MariaDB as a name, since this new distribution would also be the favored
>> way to connect to MySQL.) After doing so DBD::mysql can be deprecated, with
>> migration instructions added to the docs, and only receive critical security
>> fixes if anything. If done this way the community should not be fractured; 
>> the
>> current module will simply be abandoned for future development, all support 
>> and
>> maintenance will move to the new one. Patrick and Michiel, what do you think?
> 
> I agree with everything Dan said here.  Its what I proposed, in fewer words.  
> Do all new development under a new name, including all of Pali's work, and 
> leave the current name for a product with no further effort applied to 
> develop it. -- Darren Duncan

Re: DBD::mysql path forward

[pali]

Technical: Reason why it does not help nor solve the main problem: Due
to internals, DBD::mysql pseudo-randomly decide if input bind variable
is encoded to UTF-8 or not. And do it independently of how is option
enable_utf8 configured. As we know there are applications which misuse
this internal. Calling perlapi functions may change this internal state
under some circumstances and IIRC perl (itself) can change. Therefore it
is not safe to touching code around. Fixing Unicode support needs to
totally replace this code.

Night Light wrote that enable_utf8=0 does no encoding of input and this
is not truth. It is also root of all problems and applications which
misuse it.

Also we have no idea what else applications misuse. And because we were
not already able to detect it in testing period, I really doubt that we
would be able to do it in future...

On Friday 10 November 2017 09:34:19 p...@cpan.org wrote:
> I do not understand what (and how) your proposal with =2 solve. Probably
> nothing and people would again start after final release again
> complaining...
> 
> On Friday 10 November 2017 09:24:35 Night Light wrote:
> > Forking would take away my concerns (thank you all for suggesting that) but
> > one thing pops into my mind:
> > A decision to fork a mature module is not a light decision and should only
> > be considered as a last resort (when ran out of all options).
> > So far it is all based on the opinion of only one developer who finds it
> > hard or probably impossible to implement.
> > 
> > Can before forking at least a second person with C knowledge open dbdimp.c
> > and measure if it is indeed unfeasible to make the module backward
> > compatible by adding an option as suggested?
> > 
> > Michiel, do you know C? Can you do that 5-minute check?
> > 
> > The breaking is due to the fact that the prepared SQL statement and the
> > bound parameters will be apart from what is returned from MySQL is encoded
> > as well.
> > The mysql_enable_utf8/mysql_enable_utf8mb4 flags already arrange logic to
> > set if utf8 encoding/decoding should be done or not.
> > I'm not a C expert but this is what I find in dbdimp.c of 4.042 about it:
> > 
> > dbd_st_prepare_sv function:
> > bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> > get_statement(aTHX_ statement_sv, enable_utf8, , _len);
> > 
> > dbd_bind_ph function:
> > bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> > bind_param(_sth->params[idx], value, sql_type, idx+1, enable_utf8);
> > 
> > 
> > Would changing that into the following make the module backwards compatible?
> > bool enable_utf8 = (imp_dbh->enable_utf8 == 2 || imp_dbh->enable_utf8mb4 ==
> > 2);
> > 
> > enable_utf8 0 = no encoding of input and only decoding of output
> > enable_utf8 1 = decode only output (4.043 mode)
> > enable_utf8 2 = encoded input and decode output (4.042 mode)
> > 
> > On Fri, Nov 10, 2017 at 8:37 AM, Darren Duncan 
> > wrote:
> > 
> > > Michael, why can't you accept moving forward under a new module name?  Why
> > > does it have to be under the old name?  When people purposefully want to
> > > upgrade they purposefully choose the new module name in order to do so.
> > > What is the actual problem in that? -- Darren Duncan
> > >
> > > On 2017-11-09 10:59 PM, Michiel Beijen wrote:
> > >
> > >> On Fri, Nov 10, 2017 at 7:16 AM, Darren Duncan 
> > >> wrote:
> > >>
> > >>> I agree with everything Dan said here.  Its what I proposed, in fewer
> > >>> words.
> > >>> Do all new development under a new name, including all of Pali's work,
> > >>> and
> > >>> leave the current name for a product with no further effort applied to
> > >>> develop it. -- Darren Duncan
> > >>>
> > >>
> > >> This is NOT an option to me - it simply can't because the world moves
> > >> forward and because of bitrot. The 'old' version - the version that
> > >> works for most people, the current version of DBD::mysql, the one
> > >> which would then receive no more maintenance as it is no longer
> > >> compiles with the latest version of libmysqlclient and it does not
> > >> compile with libmariadb. This will only get worse in the future.
> > >>
> > >> I'll stick with my earlier proposal - I'll propose to go back to the
> > >> *current* latest DBD::mysql release which does not break backcompat
> > >> for our users; add the patches that we discarded when we rolled back
> > >> one by one, such as testing on many different lib/db options, memory
> > >> leaks and so on, and make a new release so we can be on the move
> > >> again.
> > >>
> > >> If possible I'd want to add back the *breaking* unicode changes that
> > >> were introduced but they should be either in a separate namespace OR
> > >> under a specific configuration option.
> > >> Currently this whole thing has cost us loosing MONTHS of progress and
> > >> then MONTHS of nothing and that is simply not good.
> > >>
> > >> Patrick: let me know if you're OK with this 

Re: DBD::mysql path forward

[Night Light]

You repeatedly expressed that you are not willing to maintain this module.
This practically means that code is being asked to be released but somebody
else needs to maintain it.
Suggestions to change this code are next to that ignored. Questions to park
the blocking encoding issue and at least go through with security patches
are also ignored.
The code is simply bundled and presented as 'take it or leave it'.

The result is the project being stalled because that apart from the fact
that the code never will pass UAT (as it changes the interface and causes
database corruption) there is one fact of reality: you are simply the only
person that is actually willing to change the driver code (and I would like
to thank you for that).
In other words: there are currently no alternative resources available that
can assist overcoming these hurdles.

With this background I ask/asked if there is at least one other person
willing to review the driver code changes (248 changed lines in dbdimp.c)
to make sure that the judgement that forking is inevitable and is not based
on the opinion of one person but shared among others.

I believe your word and expect that the outcome is that forking is indeed
necessary, but this step has another advantage which is that somebody else
also understands the code which will help ensuring that DBD::mysql can be
maintained and remain stable.

Kind Regards,
Night Light


On Fri, Nov 10, 2017 at 9:34 AM,  wrote:

> I do not understand what (and how) your proposal with =2 solve. Probably
> nothing and people would again start after final release again
> complaining...
>
> On Friday 10 November 2017 09:24:35 Night Light wrote:
> > Forking would take away my concerns (thank you all for suggesting that)
> but
> > one thing pops into my mind:
> > A decision to fork a mature module is not a light decision and should
> only
> > be considered as a last resort (when ran out of all options).
> > So far it is all based on the opinion of only one developer who finds it
> > hard or probably impossible to implement.
> >
> > Can before forking at least a second person with C knowledge open
> dbdimp.c
> > and measure if it is indeed unfeasible to make the module backward
> > compatible by adding an option as suggested?
> >
> > Michiel, do you know C? Can you do that 5-minute check?
> >
> > The breaking is due to the fact that the prepared SQL statement and the
> > bound parameters will be apart from what is returned from MySQL is
> encoded
> > as well.
> > The mysql_enable_utf8/mysql_enable_utf8mb4 flags already arrange logic
> to
> > set if utf8 encoding/decoding should be done or not.
> > I'm not a C expert but this is what I find in dbdimp.c of 4.042 about it:
> >
> > dbd_st_prepare_sv function:
> > bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> > get_statement(aTHX_ statement_sv, enable_utf8, ,
> _len);
> >
> > dbd_bind_ph function:
> > bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> > bind_param(_sth->params[idx], value, sql_type, idx+1, enable_utf8);
> >
> >
> > Would changing that into the following make the module backwards
> compatible?
> > bool enable_utf8 = (imp_dbh->enable_utf8 == 2 || imp_dbh->enable_utf8mb4
> ==
> > 2);
> >
> > enable_utf8 0 = no encoding of input and only decoding of output
> > enable_utf8 1 = decode only output (4.043 mode)
> > enable_utf8 2 = encoded input and decode output (4.042 mode)
>
>

Re: DBD::mysql path forward

[pali]

On Friday 10 November 2017 10:09:59 demerphq wrote:
> Pali is there a concise summary of what we are arguing about here?

Short summary:

1. There are applications which misuse perl and DBD::mysql internals to
   pass blob or utf-8 text data from perl to MySQL database. We were not
   aware of them for a longer testing period of time as nobody
   complained until version 4.042 of DBD::mysql was released. There were
   trial releases, blog posts, asking testers for test, etc...

2. Unicode support in DBD::mysql is broken and cause data corruption.
   There are lot of reported bugs about it, e.g.:
   https://rt.cpan.org/Public/Bug/Display.html?id=25590
   https://rt.cpan.org/Public/Bug/Display.html?id=53130
   https://rt.cpan.org/Public/Bug/Display.html?id=60987
   https://rt.cpan.org/Public/Bug/Display.html?id=87428
   https://rt.cpan.org/Public/Bug/Display.html?id=119079
   https://rt.cpan.org/Public/Bug/Display.html?id=120141
   All those problems were fixed, but breaking applications which misuse
   internals. Basically Unicode support is opposite of the above.
   All fixes from 4.042 were reverted and version 4.043 was released.

3. DBD::mysql itself misuse libmysqlclient internals for a long time and
   now when MariaDB changed something, it cannot be used with DBD::mysql
   anymore.

4. Due to how Oracle (not) reacted and (not) fixed problems related to
   SSL, DBD::mysql has a security problems when SSL encryption is used.
   There are also other crashing problems in DBD::mysql.

5. Misusing internals of 3rd applications which use DBD::mysql is a
   problem because basically any change of low level parts in DBD::mysql
   can lead to breakage of those 3rd applications. And because nobody
   was able to detect it would break something there is a really big
   chance that bigger changes in DBD::mysql break something again.
   Fixing 3. is expected to be big change.

Re: DBD::mysql path forward

[pali]

On Friday 10 November 2017 13:40:40 demerphq wrote:
> On 10 November 2017 at 13:11,   wrote:
> > Should DBD::mysql maintainers now starting complaining to Oracle and
> > MariaDB, that they must revert their changes in MySQL and MariaDB, just
> > because DBD::mysql (as libmysqlclient application) stopped working,
> > because is misusing internals of C structures?
> 
> Pali, I wanted to give you some feedback on this thread. I have been
> monitoring this discussion for some time. My $workplace probably has
> one of the largest Mysql footprints in the world, and certainly in the
> Perl world. We have funded MySQL/DBI related work in the past, and we
> are a large funder of TPF. So we are in the position to throw
> resources at this problem if the circumstances were appropriate.
> 
> But it is very difficult to determine if the circumstances ARE
> appropriate because of the level of hysteria in your posts. I believe
> your intentions are good, and I am willing to assume you are
> technically correct about many of the things you comment on, but I
> must say that the tone and form in which you are discussing it is
> counter-productive.
> 
> The people you are talking to on this list by and large are software
> developers and DB types, a group of people who are very intelligent,
> experienced, and naturally conservative in their analytical approach.
> Hysterical comments are not going to impress people like this.
> Expecting people to blindly accept that your analysis is the one and
> only viable analysis when your debating style is high on rhetoric and
> low on facts or reasoned analysis is only going to frustrate yourself
> and annoy the people on this list.
> 
> I really think you need to step back, bring your frustration (however
> understandable that frustration is) into check and try a more
> dispassionate approach. Despite lurking in this thread for some time I
> am still lacking the precise specification of the problem at hand that
> drives some of your comments. While I would like to do a code review
> and see how much I agree with what you are saying, it is very
> difficult to extract the substance from the rhetoric.
> 
> I think if you can manage to tone your responses down a bit you will
> get a much more productive response from the list.
> 
> With respect,
> Yves

Hi! I'm sorry if you understood it as a hysteria.. above quoted parts
was just rhetoric question. Not offense or a bad tone. I'm not native
English speaker and foreign language could be here a problem. Sorry for
that.

I'm disappointed that it is now 3 or 4 months (or more) that development
of DBD::mysql is fully blocked and nothing is happening. On one side are
people wanting old DBD::mysql, on other side are people wanting support
for new MariaDB and everything is just stuck :-(

I just wanted to point that there are technical problems also between
libmysqlclient and DBD::mysql driver have similar impact as problems
between some perl applications and DBD::mysql.

Re: DBD::mysql path forward

[demerphq]

On 10 November 2017 at 13:11,   wrote:
> Should DBD::mysql maintainers now starting complaining to Oracle and
> MariaDB, that they must revert their changes in MySQL and MariaDB, just
> because DBD::mysql (as libmysqlclient application) stopped working,
> because is misusing internals of C structures?

Pali, I wanted to give you some feedback on this thread. I have been
monitoring this discussion for some time. My $workplace probably has
one of the largest Mysql footprints in the world, and certainly in the
Perl world. We have funded MySQL/DBI related work in the past, and we
are a large funder of TPF. So we are in the position to throw
resources at this problem if the circumstances were appropriate.

But it is very difficult to determine if the circumstances ARE
appropriate because of the level of hysteria in your posts. I believe
your intentions are good, and I am willing to assume you are
technically correct about many of the things you comment on, but I
must say that the tone and form in which you are discussing it is
counter-productive.

The people you are talking to on this list by and large are software
developers and DB types, a group of people who are very intelligent,
experienced, and naturally conservative in their analytical approach.
Hysterical comments are not going to impress people like this.
Expecting people to blindly accept that your analysis is the one and
only viable analysis when your debating style is high on rhetoric and
low on facts or reasoned analysis is only going to frustrate yourself
and annoy the people on this list.

I really think you need to step back, bring your frustration (however
understandable that frustration is) into check and try a more
dispassionate approach. Despite lurking in this thread for some time I
am still lacking the precise specification of the problem at hand that
drives some of your comments. While I would like to do a code review
and see how much I agree with what you are saying, it is very
difficult to extract the substance from the rhetoric.

I think if you can manage to tone your responses down a bit you will
get a much more productive response from the list.

With respect,
Yves




-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: DBD::mysql path forward

[pali]

On Friday 10 November 2017 07:59:05 Michiel Beijen wrote:
> I'll stick with my earlier proposal - I'll propose to go back to the
> *current* latest DBD::mysql release which does not break backcompat
> for our users; add the patches that we discarded when we rolled back
> one by one, such as testing on many different lib/db options, memory
> leaks and so on, and make a new release so we can be on the move
> again.

Haven't we already done it? Haven't we already tested those Unicode
changes under different libraries and MySQL/MariaDB servers, including
releasing two trial versions on cpan with a long testing period without
any reported problem?

Answer is yes, we already did it.

But people who started complaining 1) fully ignores any development and
discussions; 2) fully ignores any testing release; 3) do not have own
integration tests; 4) update everything after release to last version;
5) complain if something is not how they imagine, even if they misuse
internals.

As already stated, if you want to have those users happy, then is no
other option as stop doing active development. If they still want to
misuse internals, you cannot do low level changes. Or at least you would
never be 100% sure that new changes does not break misusing of
internals.

And because for using new version of libmariadb or libmysql is needed
more robust changes, it has very high rate that something which misuse
internals would be broken.

So conclusion is: Either new version of MariaDB/MySQL or no guarantee
that misusing internals would lead to same results.

And beware of fact that C code of DBD::mysql misuse libmysqlclient
structures which leads to problems with compilation and also cause
segfaults for libmariadb. Very similar problem like between perl
application and DBD::mysql, but here it is between DBD::mysql and
libmysqlclient.

And if you are asking, then yes, MariaDB and MySQL decided to changes
their internals and application (like DBD::mysql) which misuse it,
starts crashing or not compiling. And they (MariaDB and MySQL) do that,
even without fact that DBD::mysql (as application) stop working.

So DBD::mysql is now in the same position as people who want to have old
behavior of DBD::mysql for their applications...

Should DBD::mysql maintainers now starting complaining to Oracle and
MariaDB, that they must revert their changes in MySQL and MariaDB, just
because DBD::mysql (as libmysqlclient application) stopped working,
because is misusing internals of C structures?

Re: DBD::mysql path forward

[demerphq]

On 9 November 2017 at 15:45,   wrote:
> On Thursday 09 November 2017 14:26:01 Peter Rabbitson wrote:
>> On 11/09/2017 01:46 PM, Noel Butler wrote:
>> >On 09/11/2017 21:32, p...@cpan.org wrote:
>> >
>> >>
>> >>>What the complaints in this thread are focused on is what the *users*
>> >>>want.
>> >and the users want now, or will need in the near future, to build with
>> >latest, stable and recommended versions of MariaDB and MySQL.
>>
>> Which is a great thing to want. I want this too. This is not what the thread
>> is about.
>>
>> The problem stems from Pali mashing together 3(!)* independent concerns, and
>> presenting them as an "all or nothing" proposition:
>>  1. Inability to build against latest libmysql libs
>>  2. Several security fixes
>>  3. Entirely redefining how unicode is handled throughout the stack
>
> As I already wrote, doing active development on code of DBD::mysql is
> hard or probably impossible if you want to have that old behavior.
>
> So fixing more parts of code or adding new features would lead to
> breaking that behavior or would be PITA for anybody who would need to
> guarantee that it would not be broken again in future.
>
> Moreover DBD::mysql itself (C part of code) misuse libmysql resp.
> libmariadb API which leads to segfaults (or silent memory corruption)
> when new mariadb is used. And this would be needed properly fixed if you
> want to support mariadb.
>
> So I guess it is better that compilation is failing. Users would not be
> surprised that driver really damaged data. And not just double encoded
> something to UTF-8 (due to misusing API) which has always 1:1 inverse
> operation and can be fixed.
>
> This part would not be simple to fix, there would be needed to of
> changes in whole driver, find all places where driver touch mariadb C
> structures... This is very good candidate which can break that old
> DBD::mysql behavior of handling blobs.

Pali is there a concise summary of what we are arguing about here?

Thanks,
Yves

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: DBD::mysql path forward

[pali]

I do not understand what (and how) your proposal with =2 solve. Probably
nothing and people would again start after final release again
complaining...

On Friday 10 November 2017 09:24:35 Night Light wrote:
> Forking would take away my concerns (thank you all for suggesting that) but
> one thing pops into my mind:
> A decision to fork a mature module is not a light decision and should only
> be considered as a last resort (when ran out of all options).
> So far it is all based on the opinion of only one developer who finds it
> hard or probably impossible to implement.
> 
> Can before forking at least a second person with C knowledge open dbdimp.c
> and measure if it is indeed unfeasible to make the module backward
> compatible by adding an option as suggested?
> 
> Michiel, do you know C? Can you do that 5-minute check?
> 
> The breaking is due to the fact that the prepared SQL statement and the
> bound parameters will be apart from what is returned from MySQL is encoded
> as well.
> The mysql_enable_utf8/mysql_enable_utf8mb4 flags already arrange logic to
> set if utf8 encoding/decoding should be done or not.
> I'm not a C expert but this is what I find in dbdimp.c of 4.042 about it:
> 
> dbd_st_prepare_sv function:
> bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> get_statement(aTHX_ statement_sv, enable_utf8, , _len);
> 
> dbd_bind_ph function:
> bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> bind_param(_sth->params[idx], value, sql_type, idx+1, enable_utf8);
> 
> 
> Would changing that into the following make the module backwards compatible?
> bool enable_utf8 = (imp_dbh->enable_utf8 == 2 || imp_dbh->enable_utf8mb4 ==
> 2);
> 
> enable_utf8 0 = no encoding of input and only decoding of output
> enable_utf8 1 = decode only output (4.043 mode)
> enable_utf8 2 = encoded input and decode output (4.042 mode)
> 
> On Fri, Nov 10, 2017 at 8:37 AM, Darren Duncan 
> wrote:
> 
> > Michael, why can't you accept moving forward under a new module name?  Why
> > does it have to be under the old name?  When people purposefully want to
> > upgrade they purposefully choose the new module name in order to do so.
> > What is the actual problem in that? -- Darren Duncan
> >
> > On 2017-11-09 10:59 PM, Michiel Beijen wrote:
> >
> >> On Fri, Nov 10, 2017 at 7:16 AM, Darren Duncan 
> >> wrote:
> >>
> >>> I agree with everything Dan said here.  Its what I proposed, in fewer
> >>> words.
> >>> Do all new development under a new name, including all of Pali's work,
> >>> and
> >>> leave the current name for a product with no further effort applied to
> >>> develop it. -- Darren Duncan
> >>>
> >>
> >> This is NOT an option to me - it simply can't because the world moves
> >> forward and because of bitrot. The 'old' version - the version that
> >> works for most people, the current version of DBD::mysql, the one
> >> which would then receive no more maintenance as it is no longer
> >> compiles with the latest version of libmysqlclient and it does not
> >> compile with libmariadb. This will only get worse in the future.
> >>
> >> I'll stick with my earlier proposal - I'll propose to go back to the
> >> *current* latest DBD::mysql release which does not break backcompat
> >> for our users; add the patches that we discarded when we rolled back
> >> one by one, such as testing on many different lib/db options, memory
> >> leaks and so on, and make a new release so we can be on the move
> >> again.
> >>
> >> If possible I'd want to add back the *breaking* unicode changes that
> >> were introduced but they should be either in a separate namespace OR
> >> under a specific configuration option.
> >> Currently this whole thing has cost us loosing MONTHS of progress and
> >> then MONTHS of nothing and that is simply not good.
> >>
> >> Patrick: let me know if you're OK with this and then let's get start
> >> again!
> >>
> >

Re: DBD::mysql path forward

[Night Light]

Forking would take away my concerns (thank you all for suggesting that) but
one thing pops into my mind:
A decision to fork a mature module is not a light decision and should only
be considered as a last resort (when ran out of all options).
So far it is all based on the opinion of only one developer who finds it
hard or probably impossible to implement.

Can before forking at least a second person with C knowledge open dbdimp.c
and measure if it is indeed unfeasible to make the module backward
compatible by adding an option as suggested?

Michiel, do you know C? Can you do that 5-minute check?

The breaking is due to the fact that the prepared SQL statement and the
bound parameters will be apart from what is returned from MySQL is encoded
as well.
The mysql_enable_utf8/mysql_enable_utf8mb4 flags already arrange logic to
set if utf8 encoding/decoding should be done or not.
I'm not a C expert but this is what I find in dbdimp.c of 4.042 about it:

dbd_st_prepare_sv function:
bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
get_statement(aTHX_ statement_sv, enable_utf8, , _len);

dbd_bind_ph function:
bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
bind_param(_sth->params[idx], value, sql_type, idx+1, enable_utf8);


Would changing that into the following make the module backwards compatible?
bool enable_utf8 = (imp_dbh->enable_utf8 == 2 || imp_dbh->enable_utf8mb4 ==
2);

enable_utf8 0 = no encoding of input and only decoding of output
enable_utf8 1 = decode only output (4.043 mode)
enable_utf8 2 = encoded input and decode output (4.042 mode)

On Fri, Nov 10, 2017 at 8:37 AM, Darren Duncan 
wrote:

> Michael, why can't you accept moving forward under a new module name?  Why
> does it have to be under the old name?  When people purposefully want to
> upgrade they purposefully choose the new module name in order to do so.
> What is the actual problem in that? -- Darren Duncan
>
> On 2017-11-09 10:59 PM, Michiel Beijen wrote:
>
>> On Fri, Nov 10, 2017 at 7:16 AM, Darren Duncan 
>> wrote:
>>
>>> I agree with everything Dan said here.  Its what I proposed, in fewer
>>> words.
>>> Do all new development under a new name, including all of Pali's work,
>>> and
>>> leave the current name for a product with no further effort applied to
>>> develop it. -- Darren Duncan
>>>
>>
>> This is NOT an option to me - it simply can't because the world moves
>> forward and because of bitrot. The 'old' version - the version that
>> works for most people, the current version of DBD::mysql, the one
>> which would then receive no more maintenance as it is no longer
>> compiles with the latest version of libmysqlclient and it does not
>> compile with libmariadb. This will only get worse in the future.
>>
>> I'll stick with my earlier proposal - I'll propose to go back to the
>> *current* latest DBD::mysql release which does not break backcompat
>> for our users; add the patches that we discarded when we rolled back
>> one by one, such as testing on many different lib/db options, memory
>> leaks and so on, and make a new release so we can be on the move
>> again.
>>
>> If possible I'd want to add back the *breaking* unicode changes that
>> were introduced but they should be either in a separate namespace OR
>> under a specific configuration option.
>> Currently this whole thing has cost us loosing MONTHS of progress and
>> then MONTHS of nothing and that is simply not good.
>>
>> Patrick: let me know if you're OK with this and then let's get start
>> again!
>>
>

Re: DBD::mysql path forward

[Darren Duncan]

On 2017-11-09 10:50 PM, Alexander Hartmaier wrote:

What about DBD::MariaDB as that‘s the name of the OpenSource version these days?


No, MariaDB is simply a fork of MySQL.  Both MariaDB and MySQL are Open Source. 
Saying only MariaDB is open source is wrong.


I also disagree with using the MariaDB name as that sends the wrong message. 
Keeping the MySQL name says its for all the (API-compatible) forks while using 
the MariaDB name implies that the MariaDB fork is more special than the others.


-- Darren Duncan

Re: DBD::mysql path forward

[Darren Duncan]

Michael, why can't you accept moving forward under a new module name?  Why does 
it have to be under the old name?  When people purposefully want to upgrade they 
purposefully choose the new module name in order to do so.  What is the actual 
problem in that? -- Darren Duncan


On 2017-11-09 10:59 PM, Michiel Beijen wrote:

On Fri, Nov 10, 2017 at 7:16 AM, Darren Duncan  wrote:

I agree with everything Dan said here.  Its what I proposed, in fewer words.
Do all new development under a new name, including all of Pali's work, and
leave the current name for a product with no further effort applied to
develop it. -- Darren Duncan


This is NOT an option to me - it simply can't because the world moves
forward and because of bitrot. The 'old' version - the version that
works for most people, the current version of DBD::mysql, the one
which would then receive no more maintenance as it is no longer
compiles with the latest version of libmysqlclient and it does not
compile with libmariadb. This will only get worse in the future.

I'll stick with my earlier proposal - I'll propose to go back to the
*current* latest DBD::mysql release which does not break backcompat
for our users; add the patches that we discarded when we rolled back
one by one, such as testing on many different lib/db options, memory
leaks and so on, and make a new release so we can be on the move
again.

If possible I'd want to add back the *breaking* unicode changes that
were introduced but they should be either in a separate namespace OR
under a specific configuration option.
Currently this whole thing has cost us loosing MONTHS of progress and
then MONTHS of nothing and that is simply not good.

Patrick: let me know if you're OK with this and then let's get start again!

Re: DBD::mysql path forward

[Michiel Beijen]

Hi Darren, Dan,

On Fri, Nov 10, 2017 at 7:16 AM, Darren Duncan  wrote:
> On 2017-11-09 8:32 AM, Dan Book wrote:
>>
>> It seems to me like the remaining option that can make everyone "happy" is
>> the
>> previously-suggested option of maintaining a legacy branch and doing new
>> development (reinstating 4.042) in another branch which will be released
>> as a
>> new distribution, like DBD::mysql2, by the same maintainers. (I would not
>> favor
>> DBD::MariaDB as a name, since this new distribution would also be the
>> favored
>> way to connect to MySQL.) After doing so DBD::mysql can be deprecated,
>> with
>> migration instructions added to the docs, and only receive critical
>> security
>> fixes if anything. If done this way the community should not be fractured;
>> the
>> current module will simply be abandoned for future development, all
>> support and
>> maintenance will move to the new one. Patrick and Michiel, what do you
>> think?
>
> I agree with everything Dan said here.  Its what I proposed, in fewer words.
> Do all new development under a new name, including all of Pali's work, and
> leave the current name for a product with no further effort applied to
> develop it. -- Darren Duncan

This is NOT an option to me - it simply can't because the world moves
forward and because of bitrot. The 'old' version - the version that
works for most people, the current version of DBD::mysql, the one
which would then receive no more maintenance as it is no longer
compiles with the latest version of libmysqlclient and it does not
compile with libmariadb. This will only get worse in the future.

I'll stick with my earlier proposal - I'll propose to go back to the
*current* latest DBD::mysql release which does not break backcompat
for our users; add the patches that we discarded when we rolled back
one by one, such as testing on many different lib/db options, memory
leaks and so on, and make a new release so we can be on the move
again.

If possible I'd want to add back the *breaking* unicode changes that
were introduced but they should be either in a separate namespace OR
under a specific configuration option.
Currently this whole thing has cost us loosing MONTHS of progress and
then MONTHS of nothing and that is simply not good.

Patrick: let me know if you're OK with this and then let's get start again!

--
Michiel

Re: DBD::mysql path forward

[Darren Duncan]

On 2017-11-09 8:32 AM, Dan Book wrote:

It seems to me like the remaining option that can make everyone "happy" is the
previously-suggested option of maintaining a legacy branch and doing new
development (reinstating 4.042) in another branch which will be released as a
new distribution, like DBD::mysql2, by the same maintainers. (I would not favor
DBD::MariaDB as a name, since this new distribution would also be the favored
way to connect to MySQL.) After doing so DBD::mysql can be deprecated, with
migration instructions added to the docs, and only receive critical security
fixes if anything. If done this way the community should not be fractured; the
current module will simply be abandoned for future development, all support and
maintenance will move to the new one. Patrick and Michiel, what do you think?


I agree with everything Dan said here.  Its what I proposed, in fewer words.  Do 
all new development under a new name, including all of Pali's work, and leave 
the current name for a product with no further effort applied to develop it. -- 
Darren Duncan

Re: DBD::mysql path forward

[Darren Duncan]

Pali, there's a very simple solution to what you said.  The old DBD::mysql does 
not get further maintenance at all.  It is simply frozen at the 4.041/3 state 
forever.  This serves the primary reason for that to exist, which is that people 
whose package managers automatically upgrade to the highest number of a 
namespace don't introduce new corruption due to a higher version changing 
behavior being relied on.  All development can be under the new namespace full 
stop, assuming what you say is true that no one would want to backport anything 
to the otherwise frozen version. -- Darren Duncan


On 2017-11-09 12:54 AM, p...@cpan.org wrote:

On Tuesday 07 November 2017 13:19:23 Darren Duncan wrote:

A maintenance branch would exist starting from the 4.041/3 release on which
further stable releases named DBD::mysql 4.x would be made, and the primary
goal of this branch is to not break/corrupt anything that relied on 4.041
behavior. So people with legacy projects that just use DBD::mysql and made
particular assumptions on its handling of Unicode or Blobs etc won't have
corruption introduced because DBD::mysql changed its handling of those
things.


As those people or projects misuse some internals of perl we cannot
guarantee anything such that which may be broken or changed by updating
any module from cpan or external source not related to DBD::mysql.

Maintaining such thing is something what I believe nobody wants. For
sure I'm not.

As already stated more times, if there is some code which depends on
some internals, it should stay conserved in tested version as it can
break by updating anything (also unrelated). There is no other option
how to achieve that code stay working and does not corrupt something
else.


People without knowledge of CPAN or the development process will
get something that just continues to work for them and doesn't corrupt.


As explained above (and also in past) it is not possible to guarantee.


For
all intents and purposes this branch would be frozen but could have select
security or bug fixes that comply with the mandate and don't involve
changing Unicode etc.


Are you going to maintain that branch with all problems which it brings?

Or is there anybody else who want to maintain such crap?

If not, then it does not make sense to talk about this. Because nobody
expressed that is going to do such thing for 6 months which is really
large amount of time.

Re: DBD::mysql path forward

[Darren Duncan]

On 2017-11-09 12:54 AM, p...@cpan.org wrote:

On Tuesday 07 November 2017 13:19:23 Darren Duncan wrote:

The whole discussion on the mailing lists that I recall and participated in
seemed to consensus on branching DBD::sqlite in order to best satisfy the

   ~~
   DBD::mysql

I hope you are talking about DBD::mysql there...


Yes, that was a typo, I was talking exclusively about DBD::mysql here.

That being said, as an aside, DBD::SQLite (and yes I meant that) is still a 
valuable case study in that they also branched, back 14 years ago when SQLite 3 
came out, using in that case DBD::SQLite2 that bundled the SQLite version 2 and 
repurposing DBD::SQLite name, and bumping from version 0.33 to 1.0, to now 
bundle the incompatible SQLite 3.  In their case, it was a break for people that 
simply upgraded the usual CPAN manner, and the fork was for people that still 
needed to use SQLite 2 in Perl.  DBD::mysql doesn't want to follow their example.


-- Darren Duncan

Re: DBD::mysql path forward

[Dan Book]

It seems to me like the remaining option that can make everyone "happy" is
the previously-suggested option of maintaining a legacy branch and doing
new development (reinstating 4.042) in another branch which will be
released as a new distribution, like DBD::mysql2, by the same maintainers.
(I would not favor DBD::MariaDB as a name, since this new distribution
would also be the favored way to connect to MySQL.) After doing so
DBD::mysql can be deprecated, with migration instructions added to the
docs, and only receive critical security fixes if anything. If done this
way the community should not be fractured; the current module will simply
be abandoned for future development, all support and maintenance will move
to the new one. Patrick and Michiel, what do you think?

-Dan

Re: DBD::mysql path forward

[pali]

On Thursday 09 November 2017 14:26:01 Peter Rabbitson wrote:
> On 11/09/2017 01:46 PM, Noel Butler wrote:
> >On 09/11/2017 21:32, p...@cpan.org wrote:
> >
> >>
> >>>What the complaints in this thread are focused on is what the *users*
> >>>want.
> >and the users want now, or will need in the near future, to build with
> >latest, stable and recommended versions of MariaDB and MySQL.
> 
> Which is a great thing to want. I want this too. This is not what the thread
> is about.
> 
> The problem stems from Pali mashing together 3(!)* independent concerns, and
> presenting them as an "all or nothing" proposition:
>  1. Inability to build against latest libmysql libs
>  2. Several security fixes
>  3. Entirely redefining how unicode is handled throughout the stack

As I already wrote, doing active development on code of DBD::mysql is
hard or probably impossible if you want to have that old behavior.

So fixing more parts of code or adding new features would lead to
breaking that behavior or would be PITA for anybody who would need to
guarantee that it would not be broken again in future.

Moreover DBD::mysql itself (C part of code) misuse libmysql resp.
libmariadb API which leads to segfaults (or silent memory corruption)
when new mariadb is used. And this would be needed properly fixed if you
want to support mariadb.

So I guess it is better that compilation is failing. Users would not be
surprised that driver really damaged data. And not just double encoded
something to UTF-8 (due to misusing API) which has always 1:1 inverse
operation and can be fixed.

This part would not be simple to fix, there would be needed to of
changes in whole driver, find all places where driver touch mariadb C
structures... This is very good candidate which can break that old
DBD::mysql behavior of handling blobs.

Re: DBD::mysql path forward

[Peter Rabbitson]

On 11/09/2017 09:54 AM, p...@cpan.org wrote:


As those people or projects misuse some internals of perl we cannot
guarantee anything such that which may be broken or changed by updating
any module from cpan or external source not related to DBD::mysql.



Pali. This "argument" applies to a large portion of CPAN. All JSON 
variants, various web frameworks, even things as mundane as math 
libraries: https://rt.cpan.org/Public/Bug/Display.html?id=123268



As already stated more times, if there is some code which depends on
some internals, it should stay conserved in tested version as it can
break by updating anything (also unrelated). There is no other option
how to achieve that code stay working and does not corrupt something
else.


Pali, this is *not* how CPAN works. Nor is this even how /usr/bin/perl 
works.



Maintaining such thing is something what I believe nobody wants. For
sure I'm not.



I will state several very obvious ( to me ) things, apologies if they 
seem condescending: I simply do not know whether the participants on 
this thread are talking about the same thing.


Pali's entire premise is flawed. In the grand scheme of things it makes 
absolutely no difference what "maintainers want". Zero.


What the complaints in this thread are focused on is what the *users* 
want. And every maintainer, Pali included, are irrevocably bound by the 
general direction of wishes of the userbase.


The wishes are roughly ( and in order of inmportance ):

- Breakage is never silent and is rectified swiftly when reported ( and 
is ideally never argued about )


- Upgrades are noneventful as a whole, and are as modular as possible ( 
i.e. a DBD upgrade does not drag in Test2 or something similarly 
non-relevant )


- The overall library seems to move in a "better direction" ( more 
performance, less memory use, support for newer protocols, etc )



Satisfy the above - and you do get the privilege of being a maintainer 
of a central module with 15+ years of history.


If one can't - then the user-base is better off with the module 
remaining "unmaintained" ( reusing Pali's terminology ), and Pali's 
brave-new-world plan ending up in a different namespace. DBD::MariaDB is 
an idea.


Cheers

Re: DBD::mysql path forward

[pali]

On Thursday 09 November 2017 13:01:19 H.Merijn Brand wrote:
> On Thu, 9 Nov 2017 12:32:00 +0100, p...@cpan.org wrote:
> 
> > > Satisfy the above - and you do get the privilege of being a maintainer
> > > of a central module with 15+ years of history.  
> > 
> > Why should I be interested in maintaining something from which users
> > want impossible things?
> 
>   "I can resist anything! (except temptation)"
> 
> I, like Peter, have backward compatibility high in my goals.
> 
> e.g. I dropped 5.005 supprt on Text::CSV_XS only in June 2012. The
> reasons to keep it working under 5.005 was that it was relatively
> little extra work. I dropped it, as it seriously blocked further
> development and support features my *user-base* asked for. Lucky me
> for being part of my own user-base, so I understand the requests.
> 
> In July 2016 I dropped 5.6.0 and 5.6.1 support. Not because it slows
> development, but because these two versions do not build anymore and
> do not support the modern toolchain. Not because the community or a
> part of the community thinks the users should upgrade to a more recent
> "stable" version of perl. Yes, I know 5.6.2 is from 2003 and 14 years
> old in 6 days and 5.8.0 is even older, but maintaining a module that
> serves almost 1000 other modules on CPAN (last time I counted, it was
> 997 direct or indirect dependencies) brings a burden of responsibility
> 
> This is how the river works: If I break this module, at least 997 other
> modules will break, and that is CPAN only. What about DarkPAN?

I fully understand backward compatibility and ability to use e.g. older
versions of perl with new modules on cpan. For example my module
Email::Address::XS is just reimplementation of the Email::Address module
in C to fix Algorithmic complexity vulnerability and has backward
compatible API. And because Email::Address worked with 5.6, I tried to
achieve also 5.6 compatibility with my Email::Address::XS module.

But DBD::mysql is different thing. Here we are facing two problems:

1) Passing perl Unicode strings is broken, even if some utf8 config
   option is enabled. This is changing data, see my identity examples
   which I sent to list.

2) Probably due to history reasons, API between DBD::mysql and perl
   applications is misused in way, that is atypical for perl itself and
   insane for pure perl applications. Moreover it is something which
   cannot be guaranteed to be "stable" as is affected also by modules
   not relevant to DBD::mysql. And is pseudo-random, but deterministic.

Fixing 1) implies breaking 2) and trying to achieve 2) leads to having
1) broken forever and leads to stopped development.

Similar/same problem as 2) had also DBD::Pg and DBD::SQLite and here
maintainers rather decided to fix Unicode and drop insanity in 2). In
previous emails I also send exact versions of DBD::Pg and DBD::SQLite
in which was behavior changed. (Modulo one problem in one specific
situation which I found in DBD::Pg.)

So now we are in situation when some DBI drivers doing one thing (from
perl POV correct) and DBD::mysql different.

If there are projects which uses both DBD::Pg and DBD::mysql, or
DBD::SQLite and DBD::mysql, then their are either broken (because
DBD::mysql differs how process non-ASCII due to 2)) or they needs
special hacks when MySQL is in use...

So what can be done now?

We have a DBD::mysql which behaves differently as others DBI drivers,
users misuse internals for BLOBs, driver has broken Unicode support
which makes it useless now when UTF-8 is mandatory for applications,
plus has pseudo-random behavior.

I already suggested, look at modules on cpan which uses DBD::mysql and
ask authors/maintainers which behavior they expects. Are they really
aware how non-ASCII data are handled in DBD::mysql and that it is
differently as e.g. DBD::SQLite?

We can also send patches to those modules to fix them. I already sent
documentation update for DBD::mysql which describe how to achieve
correct handling of the BLOB data in both old and Unicode-fixed version
of the DBD::mysql. I already said that I can help with fixing code, but
nobody was interested.

> If you do not want that burden, I'd strongly advice you *NOT* to take
> on the job of (co)maintaining DBD::mysql. Period.

I already expressed that I do not want to be maintainer of DBD::mysql.

IMO problems with 2) are not possible to maintain.

> I *see* your wish to improve it (for whatever value of improvement).
> This was exactly the reason I took maint on Text::CSV_XS: I wanted,
> maybe even required, new features. The original author/maintainer was
> not interested and/or motivated enough to do it and he passed the
> module on to me. By that time I HAD NO IDEA what the impact of doing so
> had. NONE! *I* wanted new feature and improvements, and he agreed.
> 
> It took me years to fully understand the implications of my actions.
> 
> So, currently I test a release on 7+ architectures and over 150
> versions of perl before I release and 

Re: DBD::mysql path forward

[Peter Rabbitson]

On 11/09/2017 01:46 PM, Noel Butler wrote:

On 09/11/2017 21:32, p...@cpan.org wrote:



What the complaints in this thread are focused on is what the *users* 
want.
and the users want now, or will need in the near future, to build with 
latest, stable and recommended versions of MariaDB and MySQL.


Which is a great thing to want. I want this too. This is not what the 
thread is about.


The problem stems from Pali mashing together 3(!)* independent concerns, 
and presenting them as an "all or nothing" proposition:

 1. Inability to build against latest libmysql libs
 2. Several security fixes
 3. Entirely redefining how unicode is handled throughout the stack

If the ( artificial ) choice is the-above-3-together-or-nothing, then 
the logical answer is "of course we rather have nothing until someone 
else comes by and fixes 1 and 2, leaving 3 alone"


If Pali is willing to *permanently* back off from changing anything 
related to 3 ( regardless how displeasing it is ) - then this becomes a 
completely different conversation.


Not even a hint of such willingness has been demonstrated. This is what 
is being discussed.


Cheers

* There are actual 4 issues: there is also intentional breakage of 
bigint handling within the driver, which already shipped to CPAN at the 
start of 2017 and still has not been backed out, despite multiple pleas, 
including at least one large Perl shop declaring that DBD::mysql is 
unusable to them as it stands right now: 
https://github.com/perl5-dbi/DBD-mysql/pull/74

Re: DBD::mysql path forward

[Noel Butler]

On 09/11/2017 21:32, p...@cpan.org wrote:

>> What the complaints in this thread are focused on is what the *users* want.

and the users want now, or will need in the near future, to build with
latest, stable and recommended versions of MariaDB and MySQL. 

Code changes all the time with everything, people have to adjust if it
alters their expectations, why should an entire project stay in the dark
ages just to appease a tiny few. 

As a very wise man once said, you cant please everyone all of the time. 

-- 
Kind Regards, 

Noel Butler 

This Email, including any attachments, may contain legally 
privileged
information, therefore remains confidential and subject to copyright
protected under international law. You may not disseminate, discuss, or
reveal, any part, to anyone, without the authors express written
authority to do so. If you are not the intended recipient, please notify
the sender then delete all copies of this message including attachments,
immediately. Confidentiality, copyright, and legal privilege are not
waived or lost by reason of the mistaken delivery of this message. Only
PDF [1] and ODF [2] documents accepted, please do not send proprietary
formatted documents 

 

Links:
--
[1] http://www.adobe.com/
[2] http://en.wikipedia.org/wiki/OpenDocument

Re: DBD::mysql path forward

[H.Merijn Brand]

On Thu, 9 Nov 2017 12:32:00 +0100, p...@cpan.org wrote:

> > Satisfy the above - and you do get the privilege of being a maintainer
> > of a central module with 15+ years of history.  
> 
> Why should I be interested in maintaining something from which users
> want impossible things?

  "I can resist anything! (except temptation)"

I, like Peter, have backward compatibility high in my goals.

e.g. I dropped 5.005 supprt on Text::CSV_XS only in June 2012. The
reasons to keep it working under 5.005 was that it was relatively
little extra work. I dropped it, as it seriously blocked further
development and support features my *user-base* asked for. Lucky me
for being part of my own user-base, so I understand the requests.

In July 2016 I dropped 5.6.0 and 5.6.1 support. Not because it slows
development, but because these two versions do not build anymore and
do not support the modern toolchain. Not because the community or a
part of the community thinks the users should upgrade to a more recent
"stable" version of perl. Yes, I know 5.6.2 is from 2003 and 14 years
old in 6 days and 5.8.0 is even older, but maintaining a module that
serves almost 1000 other modules on CPAN (last time I counted, it was
997 direct or indirect dependencies) brings a burden of responsibility

This is how the river works: If I break this module, at least 997 other
modules will break, and that is CPAN only. What about DarkPAN?

If you do not want that burden, I'd strongly advice you *NOT* to take
on the job of (co)maintaining DBD::mysql. Period.

I *see* your wish to improve it (for whatever value of improvement).
This was exactly the reason I took maint on Text::CSV_XS: I wanted,
maybe even required, new features. The original author/maintainer was
not interested and/or motivated enough to do it and he passed the
module on to me. By that time I HAD NO IDEA what the impact of doing so
had. NONE! *I* wanted new feature and improvements, and he agreed.

It took me years to fully understand the implications of my actions.

So, currently I test a release on 7+ architectures and over 150
versions of perl before I release and additionally I test the latest
release against all of those 997 modules if the new release causes any
of those deps to show new failures (compared to the old release). (Some
of these modules are currently un(der)maintained and won't build on
5.24+ for whatever reason, so if that breaks, it is not my fault.

Additionally I run speed-tests: I compare the last 20 releases for
typical use and compare the speed. If it drops out of the noise range,
that is a blocker to release, even if the new feature request is wanted
by several requestors or communities.

If you're not up to this challenge, don't do it. There are companies
depending on DBD::mysql, and they won't be your friend if they need to
modify millions lines of code if you find a new API fit better to the
new features thereby breaking old code. Backward compatibility is
important. More than you think.

How angry would you be if we broke DBI?

-- 
H.Merijn Brand  http://tux.nl   Perl Monger  http://amsterdam.pm.org/
using perl5.00307 .. 5.27   porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/http://www.test-smoke.org/
http://qa.perl.org   http://www.goldmark.org/jeff/stupid-disclaimers/


pgppYaAfONAkn.pgp
Description: OpenPGP digital signature

Re: DBD::mysql path forward

[pali]

On Thursday 09 November 2017 10:32:57 Peter Rabbitson wrote:
> On 11/09/2017 09:54 AM, p...@cpan.org wrote:
> >
> >As those people or projects misuse some internals of perl we cannot
> >guarantee anything such that which may be broken or changed by updating
> >any module from cpan or external source not related to DBD::mysql.
> >
> 
> Pali. This "argument" applies to a large portion of CPAN. All JSON variants,
> various web frameworks, even things as mundane as math libraries:
> https://rt.cpan.org/Public/Bug/Display.html?id=123268

Yes, it applies and people should be aware of it...

> >As already stated more times, if there is some code which depends on
> >some internals, it should stay conserved in tested version as it can
> >break by updating anything (also unrelated). There is no other option
> >how to achieve that code stay working and does not corrupt something
> >else.
> 
> Pali, this is *not* how CPAN works. Nor is this even how /usr/bin/perl
> works.

It does not matter if cpan works like this or not. More important is
that in any world (does not have to be perl) above statement is truth.
Once you are starting depending on some internals, then you must be
aware of it that internals may change at anytime if you update one
component in ecosystem.

> >Maintaining such thing is something what I believe nobody wants. For
> >sure I'm not.
> >
> 
> I will state several very obvious ( to me ) things, apologies if they seem
> condescending: I simply do not know whether the participants on this thread
> are talking about the same thing.
> 
> Pali's entire premise is flawed. In the grand scheme of things it makes
> absolutely no difference what "maintainers want". Zero.
> 
> What the complaints in this thread are focused on is what the *users* want.

As already wrote, if users want or need to depend on internals which are
subject to change or which can be changed by something irrelevant to
DBD::mysql, maintainers nor any other developers of DBD::mysql cannot do
anything with it. The only thing what users can do is not to updating
their ecosystem OR fixing their code to not use internals.

Basically if somebody is asking for something which is either not
possible or hard to achieve, then there would be few or zero people
ready to serve such thing.

So here I would ask you: Are you going to develop, maintain and ensure
these users requirements for DBD::mysql? If not, are you able to find
another people who will do it?

I spent non-trivial time on looking at current code and I could say that
above requirement is really really hard to achieve when DBD::mysql would
be actively developed.

And because current DBD::mysql is not possible to compile with last
MariaDB nor with last (beta) MySQL and fixing these parts needs active
development, it means that such requirement is even more hard.

So if somebody depends on internals, then is already bound to some MySQL
or MariaDB version (last which works with DBD::mysql) and is already
locked for upgrading MySQL/MariaDB.

> And every maintainer, Pali included, are irrevocably bound by the general
> direction of wishes of the userbase.

I think last time we did most what we could...

> The wishes are roughly ( and in order of inmportance ):
> 
> - Breakage is never silent and is rectified swiftly when reported ( and is
> ideally never argued about )

Michiel announced more times changes to list, wrote blog posts, released
two (or more?) trial releases to cpan and waited for a longer time
before releasing final version. Discussion about patches were on both RT
and github open...

What else could be done there?

> - Upgrades are noneventful as a whole, and are as modular as possible ( i.e.
> a DBD upgrade does not drag in Test2 or something similarly non-relevant )

DBD::mysql is just one module, there is not possible to make it modular.

> - The overall library seems to move in a "better direction" ( more
> performance, less memory use, support for newer protocols, etc )

New versions fixed some problems with both new and older MySQL and
MariaDB versions, added more automated tests for different versions.

> Satisfy the above - and you do get the privilege of being a maintainer of a
> central module with 15+ years of history.

Why should I be interested in maintaining something from which users
want impossible things?

> If one can't - then the user-base is better off with the module remaining
> "unmaintained" ( reusing Pali's terminology ), and Pali's brave-new-world
> plan ending up in a different namespace. DBD::MariaDB is an idea.
> 
> Cheers

Re: DBD::mysql path forward

[pali]

Hi! I'm responding below.

On Tuesday 07 November 2017 13:19:23 Darren Duncan wrote:
> Patrick and Pali, each of you please respond to the lists to confirm that
> what I say below is what you also understand is the primary plan, and if
> not, then say why not; in prior discussion I recall you agreed with it.
> 
> Assuming they agree, the concerns of Night Light and those he is speaking
> for should be handily satisfied.
> 
> The whole discussion on the mailing lists that I recall and participated in
> seemed to consensus on branching DBD::sqlite in order to best satisfy the
   ~~
   DBD::mysql

I hope you are talking about DBD::mysql there...

> needs of all stakeholders.
> 
> There would be one version control with 2 active release branches.
> 
> A maintenance branch would exist starting from the 4.041/3 release on which
> further stable releases named DBD::mysql 4.x would be made, and the primary
> goal of this branch is to not break/corrupt anything that relied on 4.041
> behavior. So people with legacy projects that just use DBD::mysql and made
> particular assumptions on its handling of Unicode or Blobs etc won't have
> corruption introduced because DBD::mysql changed its handling of those
> things.

As those people or projects misuse some internals of perl we cannot
guarantee anything such that which may be broken or changed by updating
any module from cpan or external source not related to DBD::mysql.

Maintaining such thing is something what I believe nobody wants. For
sure I'm not.

As already stated more times, if there is some code which depends on
some internals, it should stay conserved in tested version as it can
break by updating anything (also unrelated). There is no other option
how to achieve that code stay working and does not corrupt something
else.

> People without knowledge of CPAN or the development process will
> get something that just continues to work for them and doesn't corrupt.

As explained above (and also in past) it is not possible to guarantee.

> For
> all intents and purposes this branch would be frozen but could have select
> security or bug fixes that comply with the mandate and don't involve
> changing Unicode etc.

Are you going to maintain that branch with all problems which it brings?

Or is there anybody else who want to maintain such crap?

If not, then it does not make sense to talk about this. Because nobody
expressed that is going to do such thing for 6 months which is really
large amount of time.

> The master branch would then have all the short term breaking changes
> including the 4.042 Unicode fixes and would have all the significant feature
> changes including compatibility with newer MySQL major versions.  All of its
> releases would be under a new namespace such as DBD::mysql2 and start at
> version 5.0 to signify that large changes happened which might possibly
> break code or cause corruption if user code doesn't specifically account for
> the differences.  Being a different name this is strictly opt-in, so the
> only ones using DBD::mysql2 are those that explicitly opted in to it, and by
> doing so they also opted in to fixing anything with their code that needed
> fixing to not corrupt data while using it.
> 
> The concept of having a single driver with toggled behavior eg a
> mysql_enable_proper_unicode flag was already rejected as unwieldy to
> implement, especially for such deep rooted changes as properly handling
> Unicode.
> 
> Note that I expect that most other higher-level projects that use MySQL
> would NOT branch like this, instead having internal logic or their own
> toggle to work with DBD::mysql vs DBD::mysql2, or a few may branch, but the
> key thing is that branching DBD::mysql does NOT necessitate doing so in the
> rest of the ecosystem.
> 
> -- Darren Duncan
> 
> On 2017-11-07 12:07 PM, Night Light wrote:
> >Proposed, but no made decisions posted afterwards.
> >
> >The last proposal is to re-commit the rejected 4.042 changes into the 4.043
> >master branch and only work on fixes that came after June.
> >
> >The git issue that regards improperly encoding of BLOBs is opened on April 
> >6th
> >(hence me sending the message to prevent a recurring cycle).
> >
> >https://github.com/perl5-dbi/DBD-mysql/issues/117
> >
> >On Tue, Nov 7, 2017 at 8:57 PM, Michiel Beijen wrote:
> >
> >To me, the only real option is to make a new option in DBD::mysql;
> >mysql_enable_proper_unicode or the like which you would knowingly set
> >in your application code, which will expose the new behaviour. I
> >understand this is difficult, but I really think it's the only way.
> >
> >If in the short term this is not feasible, it *could* be possible, in
> >my eyes, to release a DBD::mysql2 or similar that does *correct*
> >behaviour. Also in that case this is something the application
> >developer should set explicitly in his connection string.
> >
> >

Re: DBD::mysql path forward

[Patrick M. Galbraith]

Darren,

Thank you.

Yes, you are correct about prior discussions. How do we handle releases 
of having a DBD::mysql and a DBD::mysql2, in terms of people knowing 
"this is the old branch with no possibly breaking changes" vs. "this is 
the new branch" ? It's been pretty simple for the 14 years I've been 
working on DBD::mysql, my main concern being, how do the various 
distributions handle this sort of thing? I suppose deal with each OS 
distributor on a case-by-case basis.


Michiel and Pali too - what do you think of what Darren is saying?

My main concerns in all of these discussions are:

1. No fragmentation

2. Stability per Night Light's previous email

3. Choice for users to decide which driver to use

4. Single project under one roof

5. Contributors who work hard on this driver get credit

Let's do this ! :)

Regards,

Patrick


On 11/7/17 4:19 PM, Darren Duncan wrote:
Patrick and Pali, each of you please respond to the lists to confirm 
that what I say below is what you also understand is the primary plan, 
and if not, then say why not; in prior discussion I recall you agreed 
with it.


Assuming they agree, the concerns of Night Light and those he is 
speaking for should be handily satisfied.


The whole discussion on the mailing lists that I recall and 
participated in seemed to consensus on branching DBD::sqlite in order 
to best satisfy the needs of all stakeholders.


There would be one version control with 2 active release branches.

A maintenance branch would exist starting from the 4.041/3 release on 
which further stable releases named DBD::mysql 4.x would be made, and 
the primary goal of this branch is to not break/corrupt anything that 
relied on 4.041 behavior. So people with legacy projects that just use 
DBD::mysql and made particular assumptions on its handling of Unicode 
or Blobs etc won't have corruption introduced because DBD::mysql 
changed its handling of those things.  People without knowledge of 
CPAN or the development process will get something that just continues 
to work for them and doesn't corrupt.  For all intents and purposes 
this branch would be frozen but could have select security or bug 
fixes that comply with the mandate and don't involve changing Unicode 
etc.


The master branch would then have all the short term breaking changes 
including the 4.042 Unicode fixes and would have all the significant 
feature changes including compatibility with newer MySQL major 
versions.  All of its releases would be under a new namespace such as 
DBD::mysql2 and start at version 5.0 to signify that large changes 
happened which might possibly break code or cause corruption if user 
code doesn't specifically account for the differences.  Being a 
different name this is strictly opt-in, so the only ones using 
DBD::mysql2 are those that explicitly opted in to it, and by doing so 
they also opted in to fixing anything with their code that needed 
fixing to not corrupt data while using it.


The concept of having a single driver with toggled behavior eg a 
mysql_enable_proper_unicode flag was already rejected as unwieldy to 
implement, especially for such deep rooted changes as properly 
handling Unicode.


Note that I expect that most other higher-level projects that use 
MySQL would NOT branch like this, instead having internal logic or 
their own toggle to work with DBD::mysql vs DBD::mysql2, or a few may 
branch, but the key thing is that branching DBD::mysql does NOT 
necessitate doing so in the rest of the ecosystem.


-- Darren Duncan

On 2017-11-07 12:07 PM, Night Light wrote:

Proposed, but no made decisions posted afterwards.

The last proposal is to re-commit the rejected 4.042 changes into the 
4.043

master branch and only work on fixes that came after June.

The git issue that regards improperly encoding of BLOBs is opened on 
April 6th

(hence me sending the message to prevent a recurring cycle).

https://github.com/perl5-dbi/DBD-mysql/issues/117

On Tue, Nov 7, 2017 at 8:57 PM, Michiel Beijen wrote:

    To me, the only real option is to make a new option in DBD::mysql;
    mysql_enable_proper_unicode or the like which you would knowingly 
set

    in your application code, which will expose the new behaviour. I
    understand this is difficult, but I really think it's the only way.

    If in the short term this is not feasible, it *could* be 
possible, in

    my eyes, to release a DBD::mysql2 or similar that does *correct*
    behaviour. Also in that case this is something the application
    developer should set explicitly in his connection string.

    This DBD::mysql2 or similar could live in another git repository, 
but
    preferably in the same repo, and even in the same CPAN 
distribution as

    DBD::mysql, and eventually the goal should be that they re-unite and
    using DBD::mysql2 would really be the same as to use the
    'mysql_enable_proper_unicode' option in DBD::mysql.

    --
    Michiel

    On Tue, Nov 7, 2017 at 7:41 PM, Darren 

Re: DBD::mysql path forward

[Darren Duncan]

Patrick and Pali, each of you please respond to the lists to confirm that what I 
say below is what you also understand is the primary plan, and if not, then say 
why not; in prior discussion I recall you agreed with it.


Assuming they agree, the concerns of Night Light and those he is speaking for 
should be handily satisfied.


The whole discussion on the mailing lists that I recall and participated in 
seemed to consensus on branching DBD::sqlite in order to best satisfy the needs 
of all stakeholders.


There would be one version control with 2 active release branches.

A maintenance branch would exist starting from the 4.041/3 release on which 
further stable releases named DBD::mysql 4.x would be made, and the primary goal 
of this branch is to not break/corrupt anything that relied on 4.041 behavior. 
So people with legacy projects that just use DBD::mysql and made particular 
assumptions on its handling of Unicode or Blobs etc won't have corruption 
introduced because DBD::mysql changed its handling of those things.  People 
without knowledge of CPAN or the development process will get something that 
just continues to work for them and doesn't corrupt.  For all intents and 
purposes this branch would be frozen but could have select security or bug fixes 
that comply with the mandate and don't involve changing Unicode etc.


The master branch would then have all the short term breaking changes including 
the 4.042 Unicode fixes and would have all the significant feature changes 
including compatibility with newer MySQL major versions.  All of its releases 
would be under a new namespace such as DBD::mysql2 and start at version 5.0 to 
signify that large changes happened which might possibly break code or cause 
corruption if user code doesn't specifically account for the differences.  Being 
a different name this is strictly opt-in, so the only ones using DBD::mysql2 are 
those that explicitly opted in to it, and by doing so they also opted in to 
fixing anything with their code that needed fixing to not corrupt data while 
using it.


The concept of having a single driver with toggled behavior eg a 
mysql_enable_proper_unicode flag was already rejected as unwieldy to implement, 
especially for such deep rooted changes as properly handling Unicode.


Note that I expect that most other higher-level projects that use MySQL would 
NOT branch like this, instead having internal logic or their own toggle to work 
with DBD::mysql vs DBD::mysql2, or a few may branch, but the key thing is that 
branching DBD::mysql does NOT necessitate doing so in the rest of the ecosystem.


-- Darren Duncan

On 2017-11-07 12:07 PM, Night Light wrote:

Proposed, but no made decisions posted afterwards.

The last proposal is to re-commit the rejected 4.042 changes into the 4.043
master branch and only work on fixes that came after June.

The git issue that regards improperly encoding of BLOBs is opened on April 6th
(hence me sending the message to prevent a recurring cycle).

https://github.com/perl5-dbi/DBD-mysql/issues/117

On Tue, Nov 7, 2017 at 8:57 PM, Michiel Beijen wrote:

To me, the only real option is to make a new option in DBD::mysql;
mysql_enable_proper_unicode or the like which you would knowingly set
in your application code, which will expose the new behaviour. I
understand this is difficult, but I really think it's the only way.

If in the short term this is not feasible, it *could* be possible, in
my eyes, to release a DBD::mysql2 or similar that does *correct*
behaviour. Also in that case this is something the application
developer should set explicitly in his connection string.

This DBD::mysql2 or similar could live in another git repository, but
preferably in the same repo, and even in the same CPAN distribution as
DBD::mysql, and eventually the goal should be that they re-unite and
using DBD::mysql2 would really be the same as to use the
'mysql_enable_proper_unicode' option in DBD::mysql.

--
Michiel

On Tue, Nov 7, 2017 at 7:41 PM, Darren Duncan > wrote:
> My understanding from the last discussion on this matter is that it was
> agreed DBD::mysql would be forked such that the existing DBD::mysql name
> would be frozen at 4.041/3 indefinitely and that the 4.042 changes plus 
any
> further feature development would take place under a new name such as
> DBD::mysql2 version 5.0.

Re: DBD::mysql path forward

[Michiel Beijen]

I think we have not decided how we should fix the unicode issues.
Previously in this thread, it seemed to be pali and Dan Book had other
ideas about this than most others.

I wrote on this topic earlier to pali:

On Wed, Sep 13, 2017 at 9:21 AM,   wrote:
> I would suggest to communicate with all people behind cpan modules which
> uses mysql_enable_utf8.
>
> They should know if their modules are expecting old broken or behavior
> of DBD::mysql or not. I would expect that are modules which needs
> correct behavior as not all perl people expect such bugs...
>
> They can update their modules to be compatible with both old buggy
> behavior and new correct. Months ago I wrote update for DBD::mysql POD
> documentation which mention example how to achieve it:

I don't think this is a good idea. There is much more code using
DBD::mysql than what you see on CPAN. If we are going to introduce
changes in behavior, they should really be opt-in and not opt-out.
Because otherwise it can corrupt peoples data and we should not want
that.

> Meanwhile DBD::mysql can try to play with "conflicts" relationship in
> META.json to prevent installing one of the above cpan module which
> depends on broken DBD::mysql until maintainer of that module fixes it.
>
> https://metacpan.org/pod/CPAN::Meta::Spec#Prereq-Spec
>
> I have never uses "conflicts" in META.json, so I do not know if it is
> working correctly in all cpan clients, but according to documentation it
> is a way how to achieve that users would not upgrade DBD::mysql to new
> version if they have installed another module which needs old
> DBD::mysql.

Please understand that this is not a CPAN-only world. Many people
install DBD::mysql using their linux package managers, especially for
server installations of software. And some software which uses
DBD::mysql (for instance OTRS, or Bugzilla) is typically not installed
from CPAN. Of course the same goes for peoples internal systems.

And that is assuming the 'conflicts' in META.json would actually work
as expected.

To me, the only real option is to make a new option in DBD::mysql;
mysql_enable_proper_unicode or the like which you would knowingly set
in your application code, which will expose the new behaviour. I
understand this is difficult, but I really think it's the only way.

If in the short term this is not feasible, it *could* be possible, in
my eyes, to release a DBD::mysql2 or similar that does *correct*
behaviour. Also in that case this is something the application
developer should set explicitly in his connection string.

This DBD::mysql2 or similar could live in another git repository, but
preferably in the same repo, and even in the same CPAN distribution as
DBD::mysql, and eventually the goal should be that they re-unite and
using DBD::mysql2 would really be the same as to use the
'mysql_enable_proper_unicode' option in DBD::mysql.

--
Michiel

On Tue, Nov 7, 2017 at 7:41 PM, Darren Duncan  wrote:
> My understanding from the last discussion on this matter is that it was
> agreed DBD::mysql would be forked such that the existing DBD::mysql name
> would be frozen at 4.041/3 indefinitely and that the 4.042 changes plus any
> further feature development would take place under a new name such as
> DBD::mysql2 version 5.0.
>
> So those people using "DBD::mysql" would not experience any changes in
> behavior from what they were used to and BLOB etc should not break.  Whereas
> people that want the Unicode fixes and other features would use the
> DBD::mysql2 and fix their code to be compatible with its breaking changes.
>
> I thought this was settled, and takes care of your concerns, so it was just
> a matter of "make it so".
>
> (Anyone replying to this message/thread, please do NOT include my personal
> email address as an explicit recipient, only send it to the list addresses,
> I will get those.)
>
> -- Darren Duncan
>
>
> On 2017-11-07 3:41 AM, Night Light wrote:
>>
>> For the reason of "silence":
>> I've spoken to other users to hear that they have passively withdrawn from
>> this
>> discussion as they are not motivated to be part of a release where
>> concerns
>> about backwards compatibility are ignored. One of the users wrote earlier
>> (replace the word "sector" with "release"):
>>
>> "When you're dealing with software that's purpose in life is to not
>> corrupt
>> data, and have data there tomorrow, you go out of your way not to break
>> that
>> promise. There's no point in being involved in this sector if you don't
>> care to
>> deliver on that promise."
>>
>> Re-releasing the 4.042 changes will break the contract of a long-standing
>> interface and corrupt all BLOB data when using "do()". These changes do
>> therefore more harm than good.
>> Putting these utf8 changes in the freezer until a suggestion is made that
>> will
>> add the functionality instead of replacing it is not a sin. The PG driver
>> has
>> for instance also a similar issue open without plans to fix 

Re: DBD::mysql path forward

[Darren Duncan]

My understanding from the last discussion on this matter is that it was agreed 
DBD::mysql would be forked such that the existing DBD::mysql name would be 
frozen at 4.041/3 indefinitely and that the 4.042 changes plus any further 
feature development would take place under a new name such as DBD::mysql2 
version 5.0.


So those people using "DBD::mysql" would not experience any changes in behavior 
from what they were used to and BLOB etc should not break.  Whereas people that 
want the Unicode fixes and other features would use the DBD::mysql2 and fix 
their code to be compatible with its breaking changes.


I thought this was settled, and takes care of your concerns, so it was just a 
matter of "make it so".


(Anyone replying to this message/thread, please do NOT include my personal email 
address as an explicit recipient, only send it to the list addresses, I will get 
those.)


-- Darren Duncan

On 2017-11-07 3:41 AM, Night Light wrote:

For the reason of "silence":
I've spoken to other users to hear that they have passively withdrawn from this
discussion as they are not motivated to be part of a release where concerns
about backwards compatibility are ignored. One of the users wrote earlier
(replace the word "sector" with "release"):

"When you're dealing with software that's purpose in life is to not corrupt
data, and have data there tomorrow, you go out of your way not to break that
promise. There's no point in being involved in this sector if you don't care to
deliver on that promise."

Re-releasing the 4.042 changes will break the contract of a long-standing
interface and corrupt all BLOB data when using "do()". These changes do
therefore more harm than good.
Putting these utf8 changes in the freezer until a suggestion is made that will
add the functionality instead of replacing it is not a sin. The PG driver has
for instance also a similar issue open without plans to fix it anytime soon.
https://rt.cpan.org/Public/Bug/Display.html?id=122991

What is your objection against using the current 4.043 branch and work on
outstanding fixes, do a pre-release, a period of time for people to test/try
out, then release?

Re: DBD::mysql path forward

[Night Light]

For the reason of "silence":
I've spoken to other users to hear that they have passively withdrawn from
this discussion as they are not motivated to be part of a release where
concerns about backwards compatibility are ignored. One of the users wrote
earlier (replace the word "sector" with "release"):

"When you're dealing with software that's purpose in life is to not corrupt
data, and have data there tomorrow, you go out of your way not to break
that promise. There's no point in being involved in this sector if you
don't care to deliver on that promise."

Re-releasing the 4.042 changes will break the contract of a long-standing
interface and corrupt all BLOB data when using "do()". These changes do
therefore more harm than good.
Putting these utf8 changes in the freezer until a suggestion is made that
will add the functionality instead of replacing it is not a sin. The PG
driver has for instance also a similar issue open without plans to fix it
anytime soon.
https://rt.cpan.org/Public/Bug/Display.html?id=122991

What is your objection against using the current 4.043 branch and work on
outstanding fixes, do a pre-release, a period of time for people to
test/try out, then release?

On Mon, Nov 6, 2017 at 11:32 PM, Patrick Galbraith  wrote:

> Pali,
>
> I totally understand - timing timing - I started a new job and found out I
> had to prepare to talk about MySQL/Kubernetes at kubecon, so I apologize,
> and didn't plan it that way. Let me re-review those emails and give a
> decision, of course, your decision as well.
>
> As far as maintainer, you are an equal, and if you need to be seen as a
> decision maker and we work by consensus with you having equal decision. I'm
> glad with that as well because I do appreciate all of your contributions.
> I'm sorry for what you mentioned in the past though it was more of a
> reaction to deal with "customers" having issues and working panic mode, not
> a slight to your efforts. What would you prefer?
>
> So, what would be ideal for us to move forward is to get the utf8 fixes
> into the master branch, then work in the subsequent fixes that came after
> June, do a pre-release, a period of time for people to test/try out, then
> release. What sort of git gymnastics does this require?  What do you think?
>
> I think the key here to have a single driver that supports both Maria and
> MySQL both and not have splintering, and more than anything, provide
> stability as well as new features for users.
>
> Regards,
>
> Patrick
>
>
>
> On 11/05/2017 04:24 AM, p...@cpan.org wrote:
>
> Hi!
>
> Have you got replies from people who you informed/asked about that
> problems?
>
> What is needed from you is to decide how to handle problems with
> encoding and other bugs reported for DBD-mysql project [1].
>
> I sent in previous email some proposed solutions and I'm just waiting
> which would be chosen... I just do not like "silence" and no step
> forward. Just to note that Dan Book asked month ago for next steps.
>
> Also, I'm not very happy to see being co-maintainer of DBD::mysql,
> specially in this time when nothing happened for 5 months, plus some
> people opposed to me and wanted to revert bug fixes and contributions by
> me...
>
> [1] - https://rt.cpan.org/Public/Dist/Display.html?Name=DBD-mysql
>
> On Friday 03 November 2017 16:17:09 Patrick M. Galbraith wrote:
>
> Pali,
>
> Sorry, I wanted to give it time for people, plus have been very busy.
> I'm ready, just let me know when we can get code merged back in--
> what do you need me to do? I will need help with this.
>
> Regards,
>
> Patrick
>
> On 11/2/17 1:43 PM, p...@cpan.org wrote:
>
> On Tuesday 12 September 2017 11:40:31 Patrick M. Galbraith wrote:
>
> Hi all,
>
> After talking to Pali and looking at how other drivers have
> handled the issue, the best way forward will be to deal with
> solving the UTF-8 issue correctly as was attempted in May. This
> will be a problem for some, but only due to having to been
> accustomed to a work-around that relies on a intermittent and
> buggy implementation. The plan is to give ample time to let
> people prepare and comment as well as give them a chance to try
> the changes and adjust prior to stable release. I'm going to look
> through the mailing list and find particularly those who had
> problems back in May and June when we tried this before and work
> with them in advance of the change.
>
> DBD::mysql should be on par with all the other DBD drivers and
> allow transparent usage regardless of backing RDBMS, and this
> really is the only way. It will also allow for other fixes to
> proceed and the driver to continue improving and supporting all
> versions and enhancements of MySQL and MariaDB.
>
> Please do give us your thoughts on this as we want to be as
> helpful and transparent as possible.
>
> Thank you!
>
> Patrick
>
> Hi! Nearly another two months passed and there is no progress in
> unblocking development.
>
> Currently more people started reporting problems 

Re: DBD::mysql path forward

[Patrick Galbraith]

Pali,

I totally understand - timing timing - I started a new job and found out 
I had to prepare to talk about MySQL/Kubernetes at kubecon, so I 
apologize, and didn't plan it that way. Let me re-review those emails 
and give a decision, of course, your decision as well.


As far as maintainer, you are an equal, and if you need to be seen as a 
decision maker and we work by consensus with you having equal decision. 
I'm glad with that as well because I do appreciate all of your 
contributions.  I'm sorry for what you mentioned in the past though it 
was more of a reaction to deal with "customers" having issues and 
working panic mode, not a slight to your efforts. What would you prefer?


So, what would be ideal for us to move forward is to get the utf8 fixes 
into the master branch, then work in the subsequent fixes that came 
after June, do a pre-release, a period of time for people to test/try 
out, then release. What sort of git gymnastics does this require?  What 
do you think?


I think the key here to have a single driver that supports both Maria 
and MySQL both and not have splintering, and more than anything, provide 
stability as well as new features for users.


Regards,

Patrick



On 11/05/2017 04:24 AM, p...@cpan.org wrote:

Hi!

Have you got replies from people who you informed/asked about that
problems?

What is needed from you is to decide how to handle problems with
encoding and other bugs reported for DBD-mysql project [1].

I sent in previous email some proposed solutions and I'm just waiting
which would be chosen... I just do not like "silence" and no step
forward. Just to note that Dan Book asked month ago for next steps.

Also, I'm not very happy to see being co-maintainer of DBD::mysql,
specially in this time when nothing happened for 5 months, plus some
people opposed to me and wanted to revert bug fixes and contributions by
me...

[1] -https://rt.cpan.org/Public/Dist/Display.html?Name=DBD-mysql  


On Friday 03 November 2017 16:17:09 Patrick M. Galbraith wrote:

Pali,

Sorry, I wanted to give it time for people, plus have been very busy.
I'm ready, just let me know when we can get code merged back in--
what do you need me to do? I will need help with this.

Regards,

Patrick

On 11/2/17 1:43 PM,p...@cpan.org  wrote:

On Tuesday 12 September 2017 11:40:31 Patrick M. Galbraith wrote:

Hi all,

After talking to Pali and looking at how other drivers have
handled the issue, the best way forward will be to deal with
solving the UTF-8 issue correctly as was attempted in May. This
will be a problem for some, but only due to having to been
accustomed to a work-around that relies on a intermittent and
buggy implementation. The plan is to give ample time to let
people prepare and comment as well as give them a chance to try
the changes and adjust prior to stable release. I'm going to look
through the mailing list and find particularly those who had
problems back in May and June when we tried this before and work
with them in advance of the change.

DBD::mysql should be on par with all the other DBD drivers and
allow transparent usage regardless of backing RDBMS, and this
really is the only way. It will also allow for other fixes to
proceed and the driver to continue improving and supporting all
versions and enhancements of MySQL and MariaDB.

Please do give us your thoughts on this as we want to be as
helpful and transparent as possible.

Thank you!

Patrick

Hi! Nearly another two months passed and there is no progress in
unblocking development.

Currently more people started reporting problems that new versions
of both MySQL and MariaDB cannot be used for building DBD::mysql.

As I stated two months ago, we are thinking about forking
DBD::mysql as inability to build DBD::mysql with new version of
MySQL or MariaDB is a real problem which needs to be fixed ASAP --
and not waiting another 6 months.

Re: DBD::mysql path forward

[Patrick M. Galbraith]

Pali,

I have given you co-maintainership to DBD::mysql and permissions on the 
repo - let's go ahead and move forward.


Thank you,

Patrick

On 11/3/17 11:17 AM, Patrick M. Galbraith wrote:


Pali,

Sorry, I wanted to give it time for people, plus have been very busy. 
I'm ready, just let me know when we can get code merged back in-- what 
do you need me to do? I will need help with this.


Regards,

Patrick

On 11/2/17 1:43 PM, p...@cpan.org wrote:

On Tuesday 12 September 2017 11:40:31 Patrick M. Galbraith wrote:

Hi all,

After talking to Pali and looking at how other drivers have handled
the issue, the best way forward will be to deal with solving the
UTF-8 issue correctly as was attempted in May. This will be a
problem for some, but only due to having to been accustomed to a
work-around that relies on a intermittent and buggy implementation.
The plan is to give ample time to let people prepare and comment as
well as give them a chance to try the changes and adjust prior to
stable release. I'm going to look through the mailing list and find
particularly those who had problems back in May and June when we
tried this before and work with them in advance of the change.

DBD::mysql should be on par with all the other DBD drivers and allow
transparent usage regardless of backing RDBMS, and this really is the
only way. It will also allow for other fixes to proceed and the
driver to continue improving and supporting all versions and
enhancements of MySQL and MariaDB.

Please do give us your thoughts on this as we want to be as helpful
and transparent as possible.

Thank you!

Patrick

Hi! Nearly another two months passed and there is no progress in
unblocking development.

Currently more people started reporting problems that new versions of
both MySQL and MariaDB cannot be used for building DBD::mysql.

As I stated two months ago, we are thinking about forking DBD::mysql as
inability to build DBD::mysql with new version of MySQL or MariaDB is a
real problem which needs to be fixed ASAP -- and not waiting another 6
months.

Re: DBD::mysql path forward

[Patrick Galbraith]

Dan,

I think get the original work Pali did (I need to get the master-new he 
references below in) and also I need to figure out what it will 
inevitably mean for distribution packages. And test, and test. I need to 
also find the emails of people who had a problem last time and have a 
plan to get people able to adopt and use the driver changes and give 
them all the info they need to modify their code to work with the proper 
driver.


regards,

Patrick


On 10/04/2017 04:49 PM, Dan Book wrote:

How can we proceed from here?
-Dan

On Mon, Sep 18, 2017 at 1:17 PM, Patrick M. Galbraith > wrote:


Pali,

Great! Now we can start moving forward.

Sorry if my responses have been intermittent - first week at new job.

Regards,

Patrick

On 9/16/17 4:35 AM, p...@cpan.org  wrote:

I prepared branch master-new, which is based on current DBD-mysql master
branch and revert state to pre-4.043 version, including all changes done
after 4.043 release to master branch. I have this master-new branch in
my fork. If you want you can use it...

https://github.com/pali/DBD-mysql/tree/master-new

Re: DBD::mysql path forward

[Dan Book]

How can we proceed from here?
-Dan

On Mon, Sep 18, 2017 at 1:17 PM, Patrick M. Galbraith  wrote:

> Pali,
>
> Great! Now we can start moving forward.
>
> Sorry if my responses have been intermittent - first week at new job.
>
> Regards,
>
> Patrick
> On 9/16/17 4:35 AM, p...@cpan.org wrote:
>
> I prepared branch master-new, which is based on current DBD-mysql master
> branch and revert state to pre-4.043 version, including all changes done
> after 4.043 release to master branch. I have this master-new branch in
> my fork. If you want you can use it...
> https://github.com/pali/DBD-mysql/tree/master-new
>
>

Re: DBD::mysql path forward

[pali]

On Tuesday 26 September 2017 14:20:33 Night Light wrote:
> That's a nifty function. Good to know that it can be reversed.

UTF-8 encode is a function which for any number from the range
0..1114111 assign unique sequence of the numbers 0..255.

Therefore this function has a well defined inverse - UTF-8 decode
function.

As a sequence of numbers from the range 0..1114111 via UTF-8 encode
function produce sequence of the numbers in range 0..255 (length of
sequence would be larger) it can be again used as as input for the UTF-8
encode function.

And because output from the UTF-8 encode has well defined inverse, you
can easily reconstruct also inverse of the composition of the more UTF-8
functions.

Take string $str and following pass:

decode('UTF-8', decode('UTF-8', encode('UTF-8', encode('UTF-8', $str eq 
$str;

To have exactly correct result, you just need to know how many times you
composed repeated call to UTF-8 encode function.

Re: DBD::mysql path forward

[pali]

On Tuesday 26 September 2017 14:20:33 Night Light wrote:
> I noticed that it was mentioned in DBD::mysql # 117 that DBD::mysql does
> know that a column in a returned resultset is a BLOB.

For returned values via MySQL protocol, there are two different and
independent things: SQL type and encoding. Which means that *returned*
data of type BLOB have encoding, and it can be encoded in UTF-8.

Moreover, MySQL server is known to send TEXT columns as type BLOB with
indication of the UTF-8 encoding.

> One should also consider the performance penalty of encoding/decoding
> BLOB's as they can be big.

Patches which were part of 4.042 release called UTF-8 decoding routine
for retrieved data only if MySQL server told that data are UTF-8
encoded. And independently of the type. Of course there is also "binary"
encoding in MySQL protocol and server can use it.

UTF-8 decoding has linear time complexity (in case of valid input data).
For large data it could be performance penalty, but returning *wrong*
and *invalid* data is worse as returning data slower.

Also note that MySQL SQL language has a functions for conversion, so it
is up to programmer to write SQL statement in correct way. There is a
MySQL SQL function to convert string into binary...

Also note that for non-served-side-prepared-statements the only
communication protocol between client and server is text based protocol.
And converting binary data to text on both side (client C library and
server library) would have probably bigger penalty as some UTF-8 decoder
in DBD driver.

Re: DBD::mysql path forward

[demerphq]

On 27 September 2017 at 16:16,   wrote:
> On Tuesday 26 September 2017 14:20:33 Night Light wrote:
>> That's a nifty function. Good to know that it can be reversed.
>
> UTF-8 encode is a function which for any number from the range
> 0..1114111 assign unique sequence of the numbers 0..255.
>
> Therefore this function has a well defined inverse - UTF-8 decode
> function.
>
> As a sequence of numbers from the range 0..1114111 via UTF-8 encode
> function produce sequence of the numbers in range 0..255 (length of
> sequence would be larger) it can be again used as as input for the UTF-8
> encode function.
>
> And because output from the UTF-8 encode has well defined inverse, you
> can easily reconstruct also inverse of the composition of the more UTF-8
> functions.
>
> Take string $str and following pass:
>
> decode('UTF-8', decode('UTF-8', encode('UTF-8', encode('UTF-8', $str eq 
> $str;
>
> To have exactly correct result, you just need to know how many times you
> composed repeated call to UTF-8 encode function.

And in practice you don't need to know this at all, as once you
encounter a byte sequence that is not valid UTF8 you know you are
done. The "seekable" nature of the octets in utf8 means this type of
heuristic has a relatively low-error rate, all your data has to have
is two bytes with the top two bits set in a row, or a byte with the
top two bits next to a byte without the top bit set, or various other
combinations. Ending up with valid utf8 data without it being actual
utf8 data is IMO extremely unlikely in sane coding scenarios.

I have used that function to clean up multiply encoded data a number
of times on DB's that have been affected by this kind of encoding bug,
and I have never encountered a scenario where I needed to know how
many times it was encoded, nor where data was corrupted.

Of course I recognize that this is a *heuristic*, but unless you are
doing crazy things, or *mega*unlucky you are not going to corrupt data
with a function like that. UTF8 is too well designed for that.

Yves

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: DBD::mysql path forward

[Night Light]

The corruption is caused by the fact that the data type of a BLOB can not
be recognized and is therefore UTF8 encoded like a string before being sent
to MySQL.
I'd like to propose the introduction of an attribute to ensure correct UTF8
encoding while maintaining backwards compatibility.
Manual page text:


mysql_encode_utf8_input
Prior 4.044 versions did not UTF8 encode the SQL statement and its input
values before being sent to MySQL but did decode the returned output. This
caused inconsistency in certain occasions.
Setting this attribute will correct this but doing so makes it necessary to
specify the data type of BLOB parameters to prevent these from being
encoded as well.
The default value is 0 to ensure backwards compatibility. Setting of this
flag is recommended for correct UTF8 handling and for easier migrations to
other database backends (DBD's).

Example with mysql_encode_utf8_input set to 0:
$dbh = DBI->connect($dsn,$user,$password, {'RaiseError' => 1,
mysql_enable_utf8mb4 => 1});
$dbh->do("INSERT INTO authors (id,blob_column,first_name) VALUES (?,?,?)",
undef, 2,$content,"Jochen");

Example with mysql_encode_utf8_input set to 1 (recommended):
$dbh = DBI->connect($dsn,$user,$password, {'RaiseError' => 1,
mysql_enable_utf8mb4 => 1 ,mysql_encode_utf8_input => 1});
$sth = $dbh->prepare("INSERT INTO authors (id,blob_column,first_name)
VALUES (?,?,?)");
$sth->bind_param(1,123,SQL_BIGINT);
$sth->bind_param(2,$content,SQL_BLOB);
$sth->bind_param(3,$first_name,SQL_VARCHAR);
$sth->execute();


On Tue, Sep 26, 2017 at 11:55 AM, demerphq  wrote:

> On 19 September 2017 at 14:46, Night Light  wrote:
> > Dear Perl gurus,
> >
> > This is my first post. I'm using Perl with great joy, and I'd like to
> > express my gratitude for all you are doing to keep Perl stable and fun to
> > use.
> >
> > I'd like to ask to object to re-releasing this version and discuss on
> how to
> > make 4.043 backwards compatible instead.
> > This change will with 100% certainty corrupt all BLOB data written to the
> > database when the developer did not read the release notes before
> applying
> > the latest version of DBD::mysql (and changed its code consequently).
> > Knowing that sysadmins have the habit of not always reading the release
> > notes of each updated package the likelihood that this will happen will
> > therefore high.
> > I myself wasn't even shown the release notes as it was a dependency of an
> > updated package that I applied.
> > The exposure of this change is big as DBD::mysql affects multiple
> > applications and many user bases.
> > I believe deliberately introducing industry wide database corruption is
> > something that will significantly harm peoples confidence in using Perl.
> > I believe that not providing backwards compatibility is not in line with
> the
> > Perl policy that has been carefully put together by the community to
> > maintain the quality of Perl as it is today.
> > http://perldoc.perl.org/perlpolicy.html#BACKWARD-
> COMPATIBILITY-AND-DEPRECATION
> >
> > I therefore believe the only solution is an upgrade that is by default
> > backwards compatible, and where it is the user who decides when to start
> > UTF8 encode the input values of a SQL request instead.
> > If it is too time consuming or too difficult it should be considered to
> park
> > the UTF8-encoding "fix" and release a version with the security fix
> first.
> >
> > I have the following objections against this release:
> >
> > 1. the upgrade will corrupt more records than it fixes (it does more harm
> > than good)
> > 2. the reason given for not providing backward compatibility ("because it
> > was hard to implement") is not plausible given the level of unwanted side
> > effects.
> >This especially knowing that there is already a mechanism in place to
> > signal if its wants UTF8 encoding or not
> > (mysql_enable_utf8/mysql_enable_utf8mb4).
> > 3. it costs more resources to coordinate/discuss a "way forward" or
> options
> > than to implement a solution that addresses backwards compatibility
> > 4. it is unreasonable to ask for changing existing source knowing that
> > depending modules may not be actively maintained or proprietary
> >It can be argued that such module should always be maintained but it
> does
> > not change the fact that a good running Perl program becomes unusable
> > 5. it does not inform the user that after upgrading existing code will
> start
> > write corrupt BLOB records
> > 6. it does not inform the user about the fact that a code review of all
> > existing code is necessary, and how it needs to be changed and tested
> > 7. it does not give the user the option to decide how the BLOB's should
> be
> > stored/encoded (opt in)
> > 8. it does not provide backwards compatibility
> >By doing so it does not respect the Perl policy that has been
> carefully
> > put together by the community to maintain the quality of Perl as it is
> > today.
> >
> > 

Re: DBD::mysql path forward

[demerphq]

On 19 September 2017 at 14:46, Night Light  wrote:
> Dear Perl gurus,
>
> This is my first post. I'm using Perl with great joy, and I'd like to
> express my gratitude for all you are doing to keep Perl stable and fun to
> use.
>
> I'd like to ask to object to re-releasing this version and discuss on how to
> make 4.043 backwards compatible instead.
> This change will with 100% certainty corrupt all BLOB data written to the
> database when the developer did not read the release notes before applying
> the latest version of DBD::mysql (and changed its code consequently).
> Knowing that sysadmins have the habit of not always reading the release
> notes of each updated package the likelihood that this will happen will
> therefore high.
> I myself wasn't even shown the release notes as it was a dependency of an
> updated package that I applied.
> The exposure of this change is big as DBD::mysql affects multiple
> applications and many user bases.
> I believe deliberately introducing industry wide database corruption is
> something that will significantly harm peoples confidence in using Perl.
> I believe that not providing backwards compatibility is not in line with the
> Perl policy that has been carefully put together by the community to
> maintain the quality of Perl as it is today.
> http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
>
> I therefore believe the only solution is an upgrade that is by default
> backwards compatible, and where it is the user who decides when to start
> UTF8 encode the input values of a SQL request instead.
> If it is too time consuming or too difficult it should be considered to park
> the UTF8-encoding "fix" and release a version with the security fix first.
>
> I have the following objections against this release:
>
> 1. the upgrade will corrupt more records than it fixes (it does more harm
> than good)
> 2. the reason given for not providing backward compatibility ("because it
> was hard to implement") is not plausible given the level of unwanted side
> effects.
>This especially knowing that there is already a mechanism in place to
> signal if its wants UTF8 encoding or not
> (mysql_enable_utf8/mysql_enable_utf8mb4).
> 3. it costs more resources to coordinate/discuss a "way forward" or options
> than to implement a solution that addresses backwards compatibility
> 4. it is unreasonable to ask for changing existing source knowing that
> depending modules may not be actively maintained or proprietary
>It can be argued that such module should always be maintained but it does
> not change the fact that a good running Perl program becomes unusable
> 5. it does not inform the user that after upgrading existing code will start
> write corrupt BLOB records
> 6. it does not inform the user about the fact that a code review of all
> existing code is necessary, and how it needs to be changed and tested
> 7. it does not give the user the option to decide how the BLOB's should be
> stored/encoded (opt in)
> 8. it does not provide backwards compatibility
>By doing so it does not respect the Perl policy that has been carefully
> put together by the community to maintain the quality of Perl as it is
> today.
>
> http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
> 9. it blocks users from using DBD::mysql upgrades as long as they have not
> rewritten their existing code
> 10. not all users from DBD::mysql can be warned beforehand about the side
> effects as it is not known which private parties have code that use
> DBD::mysql
> 12. I believe development will go faster when support for backwards
> compatibility is addressed
> 13. having to write 1 extra line for each SQL query value is a monks job
> that will make the module less attractive to use
>
> About forking to DBD::mariadb?:
> The primary reason to create such a module is when the communication
> protocol of Mariadb has become incompatible with Mysql.
> To use this namespace to fix a bug in DBD::mysql does not meet that criteria
> and causes confusion for developers and unnecessary pollution of the DBD
> namespace.
>
> ---
>
> For people that do not know the impact of the change that is pending to be
> committed:
> (see Github issue that includes 3 reports of companies that suffered data
> loss https://github.com/perl5-dbi/DBD-mysql/issues/117 )
>
> Issue: some UTF8 characters are not properly displayed after retrieval
> Cause: SQL query values are not UTF8 encoded when sent to the database but
> they are all decoded once retrieved.
> Occurence: Only records with string data that can only be written with UTF8.
> It can be considered rare as people haven't reported this issue after 10
> years of usage.
> Regional impact: Only affects countries which characters need UTF8 encoding
> and only affects string values.
> Steps to recover from it: Read string data unencoded and write it encoded.
>
> Changes of upgrade pending to be re-released:
> SQL 

Re: DBD::mysql path forward

[demerphq]

On 26 September 2017 at 12:15, Night Light  wrote:
> The corruption is caused by the fact that the data type of a BLOB can not be
> recognized and is therefore UTF8 encoded like a string before being sent to
> MySQL.

Which means it can be fixed. And in a sane scenario will be detectable.

Corruption implies the inability to reverse the operation.

UTF8 encoding can be reversed.

Any sane use of BLOB for binary data that MIGHT be corrupted by this
bug would be repairable by something like recurse_decode_utf8().

Consider the only case where something like recurse_decode_utf8()
would fail is where you *deliberately* want to store multiply encoded
utf8 in the database as binary.

Any "sane" data that contained *any* sequence of bytes which was not a
valid UTF8 sequence would be trivially restored by a function like
recurse_decode_utf8().

I totally agree that the driver messing up data is undesirable. But it
is not "corrupted". It is transformed in a way that can be reversed
and fixed.

Yves

Re: DBD::mysql path forward

[pali]

Hello, may I ask how this situation differs from upgrading DBD::Pg to 
version 3.3.0 or DBD::SQLite to version 1.43_04? What you are writing 
there basically affects also DBD::SQLite... And version 1.43_04 was 
already released...

On Tuesday 19 September 2017 14:46:09 Night Light wrote:
> Dear Perl gurus,
> 
> This is my first post. I'm using Perl with great joy, and I'd like to
> express my gratitude for all you are doing to keep Perl stable and
> fun to use.
> 
> I'd like to ask to object to re-releasing this version and discuss on
> how to make 4.043 backwards compatible instead.
> This change will with 100% certainty corrupt all BLOB data written to
> the database when the developer did not read the release notes
> before applying the latest version of DBD::mysql (and changed its
> code consequently). Knowing that sysadmins have the habit of not
> always reading the release notes of each updated package the
> likelihood that this will happen will therefore high.
> I myself wasn't even shown the release notes as it was a dependency
> of an updated package that I applied.
> The exposure of this change is big as DBD::mysql affects multiple
> applications and many user bases.
> I believe deliberately introducing industry wide database corruption
> is something that will significantly harm peoples confidence in
> using Perl. I believe that not providing backwards compatibility is
> not in line with the Perl policy that has been carefully put
> together by the community to maintain the quality of Perl as it is
> today.
> http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DE
> PRECATION
> 
> I therefore believe the only solution is an upgrade that is by
> default backwards compatible, and where it is the user who decides
> when to start UTF8 encode the input values of a SQL request instead.
> If it is too time consuming or too difficult it should be considered
> to park the UTF8-encoding "fix" and release a version with the
> security fix first.
> 
> I have the following objections against this release:
> 
> 1. the upgrade will corrupt more records than it fixes (it does more
> harm than good)
> 2. the reason given for not providing backward compatibility
> ("because it was hard to implement") is not plausible given the
> level of unwanted side effects.
>This especially knowing that there is already a mechanism in place
> to signal if its wants UTF8 encoding or not
> (mysql_enable_utf8/mysql_enable_utf8mb4).
> 3. it costs more resources to coordinate/discuss a "way forward" or
> options than to implement a solution that addresses backwards
> compatibility 4. it is unreasonable to ask for changing existing
> source knowing that depending modules may not be actively maintained
> or proprietary It can be argued that such module should always be
> maintained but it does not change the fact that a good running Perl
> program becomes unusable 5. it does not inform the user that after
> upgrading existing code will start write corrupt BLOB records
> 6. it does not inform the user about the fact that a code review of
> all existing code is necessary, and how it needs to be changed and
> tested 7. it does not give the user the option to decide how the
> BLOB's should be stored/encoded (opt in)
> 8. it does not provide backwards compatibility
>By doing so it does not respect the Perl policy that has been
> carefully put together by the community to maintain the quality of
> Perl as it is today.
> 
> http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DE
> PRECATION 9. it blocks users from using DBD::mysql upgrades as long
> as they have not rewritten their existing code
> 10. not all users from DBD::mysql can be warned beforehand about the
> side effects as it is not known which private parties have code that
> use DBD::mysql
> 12. I believe development will go faster when support for backwards
> compatibility is addressed
> 13. having to write 1 extra line for each SQL query value is a monks
> job that will make the module less attractive to use
> 
> About forking to DBD::mariadb?:
> The primary reason to create such a module is when the communication
> protocol of Mariadb has become incompatible with Mysql.
> To use this namespace to fix a bug in DBD::mysql does not meet that
> criteria and causes confusion for developers and unnecessary
> pollution of the DBD namespace.
> 
> ---
> 
> For people that do not know the impact of the change that is pending
> to be committed:
> (see Github issue that includes 3 reports of companies that suffered
> data loss https://github.com/perl5-dbi/DBD-mysql/issues/117 )
> 
> Issue: some UTF8 characters are not properly displayed after
> retrieval Cause: SQL query values are not UTF8 encoded when sent to
> the database but they are all decoded once retrieved.
> Occurence: Only records with string data that can only be written
> with UTF8. It can be considered rare as people haven't reported this
> issue after 10 years of usage.
> Regional 

Re: DBD::mysql path forward

[Paul DuBois]

> On Sep 19, 2017, at 10:10 AM, Darren Duncan  wrote:
> 
> What Night Light's post says to me is that there is high risk of causing data 
> corruption if any changes are made under the DBD::mysql name where DBD::mysql 
> has not been exhaustively tested to guarantee that its behavior is backwards 
> compatible.
> 
> This makes a stronger case to me that the DBD::mysql Git master (that which 
> includes the 4.042 changes and any other default breaking changes) should 
> rename the Perl driver package name, I suggest DBD::mysql2 version 5.0, and 
> that any changes not guaranteed backwards compatible for whatever reason go 
> there.
> 
> If the Git legacy maintenance branch 4.041/3 can have careful security 
> patches applied that don't require any changes to user code to prevent 
> breakage, it gets them, and otherwise only DBD::mysql2 gets any changes.
> 
> By doing what I said, we can be guaranteed that users with no control over 
> how DBD::mysql gets upgraded for them will introduce corruption simply for 
> upgrading.

I don't think we want to guarantee that they will introduce corruption simply 
for upgrading. :-)

> 
> -- Darren Duncan
> 
> On 2017-09-19 5:46 AM, Night Light wrote:
>> Dear Perl gurus,
>> 
>> This is my first post. I'm using Perl with great joy, and I'd like to 
>> express my
>> gratitude for all you are doing to keep Perl stable and fun to use.
>> 
>> I'd like to ask to object to re-releasing this version and discuss on how to
>> make 4.043 backwards compatible instead.
>> This change will with 100% certainty corrupt all BLOB data written to the
>> database when the developer did not read the release notes before applying 
>> the
>> latest version of DBD::mysql (and changed its code consequently).
>> Knowing that sysadmins have the habit of not always reading the release 
>> notes of
>> each updated package the likelihood that this will happen will therefore 
>> high.
>> I myself wasn't even shown the release notes as it was a dependency of an
>> updated package that I applied.
>> The exposure of this change is big as DBD::mysql affects multiple 
>> applications
>> and many user bases.
>> I believe deliberately introducing industry wide database corruption is
>> something that will significantly harm peoples confidence in using Perl.
>> I believe that not providing backwards compatibility is not in line with the
>> Perl policy that has been carefully put together by the community to maintain
>> the quality of Perl as it is today.
>> http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
>> 
>> I therefore believe the only solution is an upgrade that is by default 
>> backwards
>> compatible, and where it is the user who decides when to start UTF8 encode 
>> the
>> input values of a SQL request instead.
>> If it is too time consuming or too difficult it should be considered to park 
>> the
>> UTF8-encoding "fix" and release a version with the security fix first.
>> 
>> I have the following objections against this release:
>> 
>> 1. the upgrade will corrupt more records than it fixes (it does more harm 
>> than good)
>> 2. the reason given for not providing backward compatibility ("because it was
>> hard to implement") is not plausible given the level of unwanted side 
>> effects.
>>   This especially knowing that there is already a mechanism in place to 
>> signal
>> if its wants UTF8 encoding or not (mysql_enable_utf8/mysql_enable_utf8mb4).
>> 3. it costs more resources to coordinate/discuss a "way forward" or options 
>> than
>> to implement a solution that addresses backwards compatibility
>> 4. it is unreasonable to ask for changing existing source knowing that 
>> depending
>> modules may not be actively maintained or proprietary
>>   It can be argued that such module should always be maintained but it does 
>> not
>> change the fact that a good running Perl program becomes unusable
>> 5. it does not inform the user that after upgrading existing code will start
>> write corrupt BLOB records
>> 6. it does not inform the user about the fact that a code review of all 
>> existing
>> code is necessary, and how it needs to be changed and tested
>> 7. it does not give the user the option to decide how the BLOB's should be
>> stored/encoded (opt in)
>> 8. it does not provide backwards compatibility
>>   By doing so it does not respect the Perl policy that has been carefully put
>> together by the community to maintain the quality of Perl as it is today.
>>   
>> http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
>> 9. it blocks users from using DBD::mysql upgrades as long as they have not
>> rewritten their existing code
>> 10. not all users from DBD::mysql can be warned beforehand about the side
>> effects as it is not known which private parties have code that use 
>> DBD::mysql
>> 12. I believe development will go faster when support for backwards
>> compatibility is addressed
>> 13. having to write 1 extra 

Re: DBD::mysql path forward

[Night Light]

Dear Perl gurus,

This is my first post. I'm using Perl with great joy, and I'd like to
express my gratitude for all you are doing to keep Perl stable and fun to
use.

I'd like to ask to object to re-releasing this version and discuss on how
to make 4.043 backwards compatible instead.
This change will with 100% certainty corrupt all BLOB data written to the
database when the developer did not read the release notes before applying
the latest version of DBD::mysql (and changed its code consequently).
Knowing that sysadmins have the habit of not always reading the release
notes of each updated package the likelihood that this will happen will
therefore high.
I myself wasn't even shown the release notes as it was a dependency of an
updated package that I applied.
The exposure of this change is big as DBD::mysql affects multiple
applications and many user bases.
I believe deliberately introducing industry wide database corruption is
something that will significantly harm peoples confidence in using Perl.
I believe that not providing backwards compatibility is not in line with
the Perl policy that has been carefully put together by the community to
maintain the quality of Perl as it is today.
http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION

I therefore believe the only solution is an upgrade that is by default
backwards compatible, and where it is the user who decides when to start
UTF8 encode the input values of a SQL request instead.
If it is too time consuming or too difficult it should be considered to
park the UTF8-encoding "fix" and release a version with the security fix
first.

I have the following objections against this release:

1. the upgrade will corrupt more records than it fixes (it does more harm
than good)
2. the reason given for not providing backward compatibility ("because it
was hard to implement") is not plausible given the level of unwanted side
effects.
   This especially knowing that there is already a mechanism in place to
signal if its wants UTF8 encoding or not
(mysql_enable_utf8/mysql_enable_utf8mb4).
3. it costs more resources to coordinate/discuss a "way forward" or options
than to implement a solution that addresses backwards compatibility
4. it is unreasonable to ask for changing existing source knowing that
depending modules may not be actively maintained or proprietary
   It can be argued that such module should always be maintained but it
does not change the fact that a good running Perl program becomes unusable
5. it does not inform the user that after upgrading existing code will
start write corrupt BLOB records
6. it does not inform the user about the fact that a code review of all
existing code is necessary, and how it needs to be changed and tested
7. it does not give the user the option to decide how the BLOB's should be
stored/encoded (opt in)
8. it does not provide backwards compatibility
   By doing so it does not respect the Perl policy that has been carefully
put together by the community to maintain the quality of Perl as it is
today.

http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
9. it blocks users from using DBD::mysql upgrades as long as they have not
rewritten their existing code
10. not all users from DBD::mysql can be warned beforehand about the side
effects as it is not known which private parties have code that use
DBD::mysql
12. I believe development will go faster when support for backwards
compatibility is addressed
13. having to write 1 extra line for each SQL query value is a monks job
that will make the module less attractive to use

About forking to DBD::mariadb?:
The primary reason to create such a module is when the communication
protocol of Mariadb has become incompatible with Mysql.
To use this namespace to fix a bug in DBD::mysql does not meet that
criteria and causes confusion for developers and unnecessary pollution of
the DBD namespace.

---

For people that do not know the impact of the change that is pending to be
committed:
(see Github issue that includes 3 reports of companies that suffered data
loss https://github.com/perl5-dbi/DBD-mysql/issues/117 )

Issue: some UTF8 characters are not properly displayed after retrieval
Cause: SQL query values are not UTF8 encoded when sent to the database but
they are all decoded once retrieved.
Occurence: Only records with string data that can only be written with
UTF8. It can be considered rare as people haven't reported this issue after
10 years of usage.
Regional impact: Only affects countries which characters need UTF8 encoding
and only affects string values.
Steps to recover from it: Read string data unencoded and write it encoded.

Changes of upgrade pending to be re-released:
SQL query values are both UTF8 encoded when sent to the database as when
its retrieved (including BLOB fields).
BLOB fields will be excluded from encoding only if you specify its data
type.

Side effects from installing upgrade:
- BLOB data will be written 

Re: DBD::mysql path forward

[Darren Duncan]

What Night Light's post says to me is that there is high risk of causing data 
corruption if any changes are made under the DBD::mysql name where DBD::mysql 
has not been exhaustively tested to guarantee that its behavior is backwards 
compatible.


This makes a stronger case to me that the DBD::mysql Git master (that which 
includes the 4.042 changes and any other default breaking changes) should rename 
the Perl driver package name, I suggest DBD::mysql2 version 5.0, and that any 
changes not guaranteed backwards compatible for whatever reason go there.


If the Git legacy maintenance branch 4.041/3 can have careful security patches 
applied that don't require any changes to user code to prevent breakage, it gets 
them, and otherwise only DBD::mysql2 gets any changes.


By doing what I said, we can be guaranteed that users with no control over how 
DBD::mysql gets upgraded for them will introduce corruption simply for upgrading.


-- Darren Duncan

On 2017-09-19 5:46 AM, Night Light wrote:

Dear Perl gurus,

This is my first post. I'm using Perl with great joy, and I'd like to express my
gratitude for all you are doing to keep Perl stable and fun to use.

I'd like to ask to object to re-releasing this version and discuss on how to
make 4.043 backwards compatible instead.
This change will with 100% certainty corrupt all BLOB data written to the
database when the developer did not read the release notes before applying the
latest version of DBD::mysql (and changed its code consequently).
Knowing that sysadmins have the habit of not always reading the release notes of
each updated package the likelihood that this will happen will therefore high.
I myself wasn't even shown the release notes as it was a dependency of an
updated package that I applied.
The exposure of this change is big as DBD::mysql affects multiple applications
and many user bases.
I believe deliberately introducing industry wide database corruption is
something that will significantly harm peoples confidence in using Perl.
I believe that not providing backwards compatibility is not in line with the
Perl policy that has been carefully put together by the community to maintain
the quality of Perl as it is today.
http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION

I therefore believe the only solution is an upgrade that is by default backwards
compatible, and where it is the user who decides when to start UTF8 encode the
input values of a SQL request instead.
If it is too time consuming or too difficult it should be considered to park the
UTF8-encoding "fix" and release a version with the security fix first.

I have the following objections against this release:

1. the upgrade will corrupt more records than it fixes (it does more harm than 
good)
2. the reason given for not providing backward compatibility ("because it was
hard to implement") is not plausible given the level of unwanted side effects.
   This especially knowing that there is already a mechanism in place to signal
if its wants UTF8 encoding or not (mysql_enable_utf8/mysql_enable_utf8mb4).
3. it costs more resources to coordinate/discuss a "way forward" or options than
to implement a solution that addresses backwards compatibility
4. it is unreasonable to ask for changing existing source knowing that depending
modules may not be actively maintained or proprietary
   It can be argued that such module should always be maintained but it does not
change the fact that a good running Perl program becomes unusable
5. it does not inform the user that after upgrading existing code will start
write corrupt BLOB records
6. it does not inform the user about the fact that a code review of all existing
code is necessary, and how it needs to be changed and tested
7. it does not give the user the option to decide how the BLOB's should be
stored/encoded (opt in)
8. it does not provide backwards compatibility
   By doing so it does not respect the Perl policy that has been carefully put
together by the community to maintain the quality of Perl as it is today.
   
http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
9. it blocks users from using DBD::mysql upgrades as long as they have not
rewritten their existing code
10. not all users from DBD::mysql can be warned beforehand about the side
effects as it is not known which private parties have code that use DBD::mysql
12. I believe development will go faster when support for backwards
compatibility is addressed
13. having to write 1 extra line for each SQL query value is a monks job that
will make the module less attractive to use

About forking to DBD::mariadb?:
The primary reason to create such a module is when the communication protocol of
Mariadb has become incompatible with Mysql.
To use this namespace to fix a bug in DBD::mysql does not meet that criteria and
causes confusion for developers and unnecessary pollution of the DBD namespace.

---

For people that do not know the impact of 

Re: DBD::mysql path forward

[Patrick M. Galbraith]

Pali,

Great! Now we can start moving forward.

Sorry if my responses have been intermittent - first week at new job.

Regards,

Patrick

On 9/16/17 4:35 AM, p...@cpan.org wrote:

On Friday 15 September 2017 04:06:57 Patrick M. Galbraith wrote:

On 9/13/17 9:31 AM, p...@cpan.org wrote:

On Tuesday 12 September 2017 11:32:36 Darren Duncan wrote:

On 2017-09-12 11:05 AM, p...@cpan.org wrote:

On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:

I strongly recommend that another thing happen, which is
re-versioning DBD::mysql to 5.0.

1. From now on, DBD::mysql versions 4.x would essentially be
frozen at 4.041/4.043.

2. From now on, DBD::mysql versions 5.x and above would be where
all active development occurs, would be the only place where
Unicode handling is fixed and new MySQL versions and features
are supported, and where other features are added.

I'm fine with it. Basically it means reverting to pre-4.043 code
and continue *git* development there with increased version to
5.x. And once code is ready it can be released to cpan as normal
(non-engineering) 5.x release.

Yes, as you say.  With respect to Git I propose doing this
immediately:

1. Create a Git tag/branch off of 4.043 which is the 4.x legacy
support branch.

This is up to Patrick.

I agree that we can start there. I do like the discussion going on
now and want to engage all those concerned and interested and have a
plan.


2. Revert Git master to the pre-4.043 code and then follow that
with a commit to master that changes the DBD::mysql version to
5.0.

If everybody agree with this step, I can prepare pull request which
revert tree to this state, plus re-apply/rebase commits which were
newly accepted.

I prepared branch master-new, which is based on current DBD-mysql master
branch and revert state to pre-4.043 version, including all changes done
after 4.043 release to master branch. I have this master-new branch in
my fork. If you want you can use it...

https://github.com/pali/DBD-mysql/tree/master-new


Regardless, following point 2, mandate that all Git pull requests
are made against the new 5.x master; the 4.x legacy branch would
have no commits except minimal back-porting.

New pull requests are by default created against "master" branch.
So if 5.x development would happen in "master" and 4.x in e.g.
"legacy-4.0" then no changes is needed.

But on github it is not possible to disallow users to create new
pull requests for non-default branch. When creating pull request
there is a button which open dialog to change target branch.

Re: DBD::mysql path forward

[pali]

On Friday 15 September 2017 04:06:57 Patrick M. Galbraith wrote:
> On 9/13/17 9:31 AM, p...@cpan.org wrote:
> > On Tuesday 12 September 2017 11:32:36 Darren Duncan wrote:
> >> On 2017-09-12 11:05 AM, p...@cpan.org wrote:
> >>> On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:
>  I strongly recommend that another thing happen, which is
>  re-versioning DBD::mysql to 5.0.
>  
>  1. From now on, DBD::mysql versions 4.x would essentially be
>  frozen at 4.041/4.043.
>  
>  2. From now on, DBD::mysql versions 5.x and above would be where
>  all active development occurs, would be the only place where
>  Unicode handling is fixed and new MySQL versions and features
>  are supported, and where other features are added.
> >>> 
> >>> I'm fine with it. Basically it means reverting to pre-4.043 code
> >>> and continue *git* development there with increased version to
> >>> 5.x. And once code is ready it can be released to cpan as normal
> >>> (non-engineering) 5.x release.
> >> 
> >> Yes, as you say.  With respect to Git I propose doing this
> >> immediately:
> >> 
> >> 1. Create a Git tag/branch off of 4.043 which is the 4.x legacy
> >> support branch.
> > 
> > This is up to Patrick.
> 
> I agree that we can start there. I do like the discussion going on
> now and want to engage all those concerned and interested and have a
> plan.
> 
> >> 2. Revert Git master to the pre-4.043 code and then follow that
> >> with a commit to master that changes the DBD::mysql version to
> >> 5.0.
> > 
> > If everybody agree with this step, I can prepare pull request which
> > revert tree to this state, plus re-apply/rebase commits which were
> > newly accepted.

I prepared branch master-new, which is based on current DBD-mysql master 
branch and revert state to pre-4.043 version, including all changes done 
after 4.043 release to master branch. I have this master-new branch in 
my fork. If you want you can use it...

https://github.com/pali/DBD-mysql/tree/master-new

> >> Regardless, following point 2, mandate that all Git pull requests
> >> are made against the new 5.x master; the 4.x legacy branch would
> >> have no commits except minimal back-porting.
> > 
> > New pull requests are by default created against "master" branch.
> > So if 5.x development would happen in "master" and 4.x in e.g.
> > "legacy-4.0" then no changes is needed.
> > 
> > But on github it is not possible to disallow users to create new
> > pull requests for non-default branch. When creating pull request
> > there is a button which open dialog to change target branch.

Re: DBD::mysql path forward

[Patrick M. Galbraith]

Hi Paul! (long time!)

we need to make sure we have these cases accounted for and test coverage.

regards,

Patrick

On 9/14/17 9:24 PM, Paul DuBois wrote:

On Sep 14, 2017, at 1:44 AM, p...@cpan.org wrote:

MySQL server and its databases has some limitations, so reflect it:

* it does not provide information if placeholder is TEXT, VARCHAR, VARBINARY or 
BLOB
* placeholder's bind value does not have to point to column, it can be also SQL 
function
  --> for caller/user all placeholders are equivalent and caller itself
  needs to know how to treat bind variable and needs to specify if
  it is TEXT or BLOB

* VARBINARY is right padded with 0x00
  --> there is no difference between binary "\x01\x02\x00" and "\x01\x02"

BINARY is padded (for storage), VARBINARY is not.

https://dev.mysql.com/doc/refman/5.7/en/binary-varbinary.html

Re: DBD::mysql path forward

[Patrick M. Galbraith]

On 9/13/17 9:31 AM, p...@cpan.org wrote:

On Tuesday 12 September 2017 11:32:36 Darren Duncan wrote:

On 2017-09-12 11:05 AM, p...@cpan.org wrote:

On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:

I strongly recommend that another thing happen, which is
re-versioning DBD::mysql to 5.0.

1. From now on, DBD::mysql versions 4.x would essentially be frozen
at 4.041/4.043.

2. From now on, DBD::mysql versions 5.x and above would be where all
active development occurs, would be the only place where Unicode
handling is fixed and new MySQL versions and features are supported,
and where other features are added.

I'm fine with it. Basically it means reverting to pre-4.043 code and
continue *git* development there with increased version to 5.x. And once
code is ready it can be released to cpan as normal (non-engineering) 5.x
release.

Yes, as you say.  With respect to Git I propose doing this immediately:

1. Create a Git tag/branch off of 4.043 which is the 4.x legacy support branch.

This is up to Patrick.


I agree that we can start there. I do like the discussion going on now 
and want to engage all those concerned and interested and have a plan.






2. Revert Git master to the pre-4.043 code and then follow that with a
commit to master that changes the DBD::mysql version to 5.0.

If everybody agree with this step, I can prepare pull request which
revert tree to this state, plus re-apply/rebase commits which were newly
accepted.


Regardless, following point 2, mandate that all Git pull requests are made
against the new 5.x master; the 4.x legacy branch would have no commits
except minimal back-porting.

New pull requests are by default created against "master" branch. So if
5.x development would happen in "master" and 4.x in e.g. "legacy-4.0"
then no changes is needed.

But on github it is not possible to disallow users to create new pull
requests for non-default branch. When creating pull request there is a
button which open dialog to change target branch.

Re: DBD::mysql path forward

[Paul DuBois]

> On Sep 14, 2017, at 1:44 AM, p...@cpan.org wrote:
> 

> MySQL server and its databases has some limitations, so reflect it:
> 
> * it does not provide information if placeholder is TEXT, VARCHAR, VARBINARY 
> or BLOB
> * placeholder's bind value does not have to point to column, it can be also 
> SQL function
>  --> for caller/user all placeholders are equivalent and caller itself
>  needs to know how to treat bind variable and needs to specify if
>  it is TEXT or BLOB
> 
> * VARBINARY is right padded with 0x00
>  --> there is no difference between binary "\x01\x02\x00" and "\x01\x02"

BINARY is padded (for storage), VARBINARY is not.

https://dev.mysql.com/doc/refman/5.7/en/binary-varbinary.html

Re: DBD::mysql path forward

[Darren Duncan]

On 2017-09-14 3:01 AM, H.Merijn Brand wrote:

On Thu, 14 Sep 2017 09:44:54 +0200, p...@cpan.org wrote:


BYTE/BLOB/TEXT tests require three types of data

• Pure ASCII
• Correct UTF-8 (with complex combinations)


subtest: Correct UTF-8 TEXT with only code points in range U+00 .. U+7F (ASCII 
subset)
subtest: Correct UTF-8 TEXT with only code points in range U+00 .. U+FF (Latin1 
subset)


ASCII:U+00 .. U+7F
iso-8859-*: + U+80 .. U+FF (includes cp1252)
iso-10646:  + U+000100 .. U+0007FF
+ U+000800 .. U+00D7FF
+ U+00E000 .. U+00
utf-8 1):   + U+01 .. U+10
+ surrogates
+ bidirectionality
+ normalization
+ collation (order by)

1) some iso-10646 implementations already support supplementary
   codepoints. Depends on the version of the standard


Regarding Unicode subtests I was going to respond to Pali's comment to say that 
there are more important ranges; H.Merijn addressed the main points I was going 
to raise, however I propose a simpler set of tests as being the main ones of 
importance for data being handled without corruption.


Note, these comments apply to ALL DBI drivers, not just DBD::mysql.

There are 7 main Unicode integer codepoint ranges of interest, representable by 
a signed 32-bit integer:


- negative integers rejected invalid data
- 0  ..0x7F - ASCII subset accepted
- 0x80   ..0xFF - non ASCII 8-bit subset accepted
- 0x100  ..0xD7FF   - middle Basic Multilingual Plane accepted
- 0xD800 ..0xDFFF   - UTF-16 surrogates rejected invalid data
- 0xE000 ..0x   - upper Basic Multilingual Plane accepted
- 0x1..0x10 - the 15 supplementary planes accepted
- 0x11000 and above rejected invalid data

I would argue strongly that a transit middleware like a DBI driver should 
strictly concern itself with the Unicode codepoint level and that it shuttles 
data back and forth preserving the exact valid codepoints given, while rejecting 
invalid codepoints for both input and output either with an error or use of the 
unicode substitution character 0xFFFD.


While Perl itself or MySQL itself can concern itself with other matters such as 
graphemes/normalization/collation/etc, a DBI driver should NOT.


Besides being logically correct, this means that DBI drivers can avoid needing 
code for the most complicated aspects of Unicode, they can avoid 99% of the 
complexity.


-- Darren Duncan

Re: DBD::mysql path forward

[H.Merijn Brand]

On Thu, 14 Sep 2017 09:44:54 +0200, p...@cpan.org wrote:

> > BYTE/BLOB/TEXT tests require three types of data
> > 
> > • Pure ASCII
> > • Correct UTF-8 (with complex combinations)  
> 
> subtest: Correct UTF-8 TEXT with only code points in range U+00 .. U+7F 
> (ASCII subset)
> subtest: Correct UTF-8 TEXT with only code points in range U+00 .. U+FF 
> (Latin1 subset)

ASCII:U+00 .. U+7F
iso-8859-*: + U+80 .. U+FF (includes cp1252)
iso-10646:  + U+000100 .. U+0007FF
+ U+000800 .. U+00D7FF
+ U+00E000 .. U+00
utf-8 1):   + U+01 .. U+10
+ surrogates
+ bidirectionality
+ normalization
+ collation (order by)

1) some iso-10646 implementations already support supplementary
   codepoints. Depends on the version of the standard

With 100% Unicode, data my go bust if stored in UTF-8 fields

Unify defines a "correct" order of combined characters. I don't know
exactly what the order is, but if a letter has more than one combined
characters in it, like

 ờ U01edd \N{LATIN SMALL LETTER O WITH HORN AND GRAVE}
 ȭ U0022d \N{LATIN SMALL LETTER O WITH TILDE AND MACRON}

inserting "LATIN SMALL LETTER O" "WITH GRAVE" "WITH HORN"
is allowed to return as "LATIN SMALL LETTER O" "WITH HORN" "WITH GRAVE"
or as "LATIN SMALL LETTER O WITH GRAVE" "WITH HORN" or
"LATIN SMALL LETTER O WITH HORN" "WITH GRAVE" or
"LATIN SMALL LETTER O WITH HORN AND GRAVE"

They all represent the same grapheme. From a user perpective when
dealing with Unicode, that is fine. From a testing purpose this is
not :(

So, *if* you test with combining characters (that do not represent in a
single codepoint) make sure it matches the Unicode defined order

FYI This is why I still don't support *real* binary in perl6' Text::CSV

-- 
H.Merijn Brand  http://tux.nl   Perl Monger  http://amsterdam.pm.org/
using perl5.00307 .. 5.27   porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/http://www.test-smoke.org/
http://qa.perl.org   http://www.goldmark.org/jeff/stupid-disclaimers/


pgpEYDrEx23aH.pgp
Description: OpenPGP digital signature

Re: DBD::mysql path forward

[pali]

On Thursday 14 September 2017 08:21:42 H.Merijn Brand wrote:
> On Wed, 13 Sep 2017 13:27:58 -0700, Darren Duncan
>  wrote:
> 
> > On 2017-09-13 12:58 PM, Dan Book wrote:
> > > On Wed, Sep 13, 2017 at 3:53 AM, Peter Rabbitson wrote:
> > >
> > > On 09/12/2017 07:12 PM, p...@cpan.org wrote:
> > > And here is promised script:
> > > 
> > >
> > > The script side-steps showcasing the treatment of BLOB/BYTEA columns, 
> > > which
> > > was one of the main ( albeit not the only ) reason the userbase lost 
> > > data.
> > >
> > > Please extend the script with a BLOB/BYTEA test.
> > >
> > > I'm not sure how to usefully make such a script, since correct insertion 
> > > of BLOB
> > > data (binding with the SQL_BLOB type or similar) would work correctly both
> > > before and after the fix.  
> > 
> > Perhaps the requirement of the extra tests is to ensure that BLOB/BYTEA 
> > data is 
> > NOT mangled during input or output, that on input any strings with a true 
> > utf8 
> > flag are rejected and that on output any strings have a false utf8 flag.  
> > Part 
> > of the idea is to give regression testing that changes regarding Unicode 
> > handling with text don't inadvertently break blob handling. -- Darren Duncan
> 
> BYTE/BLOB/TEXT tests require three types of data
> 
> • Pure ASCII
> • Correct UTF-8 (with complex combinations)

subtest: Correct UTF-8 TEXT with only code points in range U+00 .. U+7F (ASCII 
subset)
subtest: Correct UTF-8 TEXT with only code points in range U+00 .. U+FF (Latin1 
subset)

> • Pure binary
>   - random bytes ranging 0x00..0xFF
>   - Images (png, jpg, gif, tiff)
> 
> None of those is hard to generate.
> 
> 1. create two tables with a field of the type to check
> 2. insert the data in table 1
> 3. use SQL to copy the data to table 2
> 4. extract data from table 2
> 5. compare to original data
> 6. drop tables
> 7. goto 1
> 
> If MySQL support different ways to do this, test all ways (in the
> string, placeholders, bind_columns, bind_params, other ...)

MySQL server and its databases has some limitations, so reflect it:

* it does not provide information if placeholder is TEXT, VARCHAR, VARBINARY or 
BLOB
* placeholder's bind value does not have to point to column, it can be also SQL 
function
  --> for caller/user all placeholders are equivalent and caller itself
  needs to know how to treat bind variable and needs to specify if
  it is TEXT or BLOB

* VARBINARY is right padded with 0x00
  --> there is no difference between binary "\x01\x02\x00" and "\x01\x02"

And note that perl itself does not distinguish between "binary bytes"
and "unicode string". User itself needs to know what he should expect in
perl scalar and how to handle it.

And if you are thinking about utf8 flag of scalar (utf8::is_utf8()) to
use as way how to distinguish character and binary data, then do not try
to do it.

In (updated) documentation in utf8.pm module is:
https://metacpan.org/pod/release/WOLFSAGE/perl-5.27.3/lib/utf8.pm

  Don't use this flag as a marker to distinguish character and binary
  data, that should be decided for each variable when you write your
  code.

Re: DBD::mysql path forward

[H.Merijn Brand]

On Wed, 13 Sep 2017 13:27:58 -0700, Darren Duncan
 wrote:

> On 2017-09-13 12:58 PM, Dan Book wrote:
> > On Wed, Sep 13, 2017 at 3:53 AM, Peter Rabbitson wrote:
> >
> > On 09/12/2017 07:12 PM, p...@cpan.org wrote:
> > And here is promised script:
> > 
> >
> > The script side-steps showcasing the treatment of BLOB/BYTEA columns, 
> > which
> > was one of the main ( albeit not the only ) reason the userbase lost 
> > data.
> >
> > Please extend the script with a BLOB/BYTEA test.
> >
> > I'm not sure how to usefully make such a script, since correct insertion of 
> > BLOB
> > data (binding with the SQL_BLOB type or similar) would work correctly both
> > before and after the fix.  
> 
> Perhaps the requirement of the extra tests is to ensure that BLOB/BYTEA data 
> is 
> NOT mangled during input or output, that on input any strings with a true 
> utf8 
> flag are rejected and that on output any strings have a false utf8 flag.  
> Part 
> of the idea is to give regression testing that changes regarding Unicode 
> handling with text don't inadvertently break blob handling. -- Darren Duncan

BYTE/BLOB/TEXT tests require three types of data

• Pure ASCII
• Correct UTF-8 (with complex combinations)
• Pure binary
  - random bytes ranging 0x00..0xFF
  - Images (png, jpg, gif, tiff)

None of those is hard to generate.

1. create two tables with a field of the type to check
2. insert the data in table 1
3. use SQL to copy the data to table 2
4. extract data from table 2
5. compare to original data
6. drop tables
7. goto 1

If MySQL support different ways to do this, test all ways (in the
string, placeholders, bind_columns, bind_params, other ...)

-- 
H.Merijn Brand  http://tux.nl   Perl Monger  http://amsterdam.pm.org/
using perl5.00307 .. 5.27   porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/http://www.test-smoke.org/
http://qa.perl.org   http://www.goldmark.org/jeff/stupid-disclaimers/


pgpdXNk6g26nW.pgp
Description: OpenPGP digital signature

Re: DBD::mysql path forward

[Peter Rabbitson]

On 09/12/2017 07:12 PM, p...@cpan.org wrote:

On Tuesday 12 September 2017 12:27:25 p...@cpan.org wrote:

To prove fact that other DBI drivers (e.g. Pg or SQLite) had fixed
similar/same UTF-8 issue as MySQL has and behave Perl-correctly, I
would provide test cases so you would see difference between Pg,
SQLite and mysql DBI drivers.


And here is promised script:





The script side-steps showcasing the treatment of BLOB/BYTEA columns, 
which was one of the main ( albeit not the only ) reason the userbase 
lost data.


Please extend the script with a BLOB/BYTEA test.

Re: DBD::mysql path forward

[Peter Rabbitson]

On 09/13/2017 09:58 PM, Dan Book wrote:
On Wed, Sep 13, 2017 at 3:53 AM, Peter Rabbitson > wrote:


On 09/12/2017 07:12 PM, p...@cpan.org  wrote:

On Tuesday 12 September 2017 12:27:25 p...@cpan.org
 wrote:

To prove fact that other DBI drivers (e.g. Pg or SQLite) had
fixed
similar/same UTF-8 issue as MySQL has and behave
Perl-correctly, I
would provide test cases so you would see difference between Pg,
SQLite and mysql DBI drivers.


And here is promised script:




The script side-steps showcasing the treatment of BLOB/BYTEA
columns, which was one of the main ( albeit not the only ) reason
the userbase lost data.

Please extend the script with a BLOB/BYTEA test.


I'm not sure how to usefully make such a script, since correct insertion 
of BLOB data (binding with the SQL_BLOB type or similar) would work 
correctly both before and after the fix.


If you were to actually try to make the requested addition you'd have 
noticed that DBD::Pg *DOES NOT* require one to binding with SQL_BLOB. 
For more info please go back to the old ( 2013 ) RT threads on the 
"mysql unicode issue" and read again carefully.


Please try to actually implement this - it will give you additional 
perspective.

Re: DBD::mysql path forward

[Darren Duncan]

On 2017-09-13 12:58 PM, Dan Book wrote:

On Wed, Sep 13, 2017 at 3:53 AM, Peter Rabbitson wrote:

On 09/12/2017 07:12 PM, p...@cpan.org wrote:
And here is promised script:


The script side-steps showcasing the treatment of BLOB/BYTEA columns, which
was one of the main ( albeit not the only ) reason the userbase lost data.

Please extend the script with a BLOB/BYTEA test.

I'm not sure how to usefully make such a script, since correct insertion of BLOB
data (binding with the SQL_BLOB type or similar) would work correctly both
before and after the fix.


Perhaps the requirement of the extra tests is to ensure that BLOB/BYTEA data is 
NOT mangled during input or output, that on input any strings with a true utf8 
flag are rejected and that on output any strings have a false utf8 flag.  Part 
of the idea is to give regression testing that changes regarding Unicode 
handling with text don't inadvertently break blob handling. -- Darren Duncan

Re: DBD::mysql path forward

[Dan Book]

On Wed, Sep 13, 2017 at 3:53 AM, Peter Rabbitson  wrote:

> On 09/12/2017 07:12 PM, p...@cpan.org wrote:
>
>> On Tuesday 12 September 2017 12:27:25 p...@cpan.org wrote:
>>
>>> To prove fact that other DBI drivers (e.g. Pg or SQLite) had fixed
>>> similar/same UTF-8 issue as MySQL has and behave Perl-correctly, I
>>> would provide test cases so you would see difference between Pg,
>>> SQLite and mysql DBI drivers.
>>>
>>
>> And here is promised script:
>>
>>
> 
>
> The script side-steps showcasing the treatment of BLOB/BYTEA columns,
> which was one of the main ( albeit not the only ) reason the userbase lost
> data.
>
> Please extend the script with a BLOB/BYTEA test.
>

I'm not sure how to usefully make such a script, since correct insertion of
BLOB data (binding with the SQL_BLOB type or similar) would work correctly
both before and after the fix.

-Dan

Re: DBD::mysql path forward

[Darren Duncan]

On 2017-09-13 6:31 AM, p...@cpan.org wrote:

On Tuesday 12 September 2017 11:32:36 Darren Duncan wrote:

Regardless, following point 2, mandate that all Git pull requests are made
against the new 5.x master; the 4.x legacy branch would have no commits
except minimal back-porting.


New pull requests are by default created against "master" branch. So if
5.x development would happen in "master" and 4.x in e.g. "legacy-4.0"
then no changes is needed.

But on github it is not possible to disallow users to create new pull
requests for non-default branch. When creating pull request there is a
button which open dialog to change target branch.


When I said "mandate" I meant as a matter of project policy, not on what GitHub 
enforces.  Strictly speaking there are times where a pull request against the 
legacy branch is appropriate. -- Darren Duncan

Re: DBD::mysql path forward

[pali]

On Tuesday 12 September 2017 11:32:36 Darren Duncan wrote:
> On 2017-09-12 11:05 AM, p...@cpan.org wrote:
> >On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:
> >>I strongly recommend that another thing happen, which is
> >>re-versioning DBD::mysql to 5.0.
> >>
> >>1. From now on, DBD::mysql versions 4.x would essentially be frozen
> >>at 4.041/4.043.
> >>
> >>2. From now on, DBD::mysql versions 5.x and above would be where all
> >>active development occurs, would be the only place where Unicode
> >>handling is fixed and new MySQL versions and features are supported,
> >>and where other features are added.
> >
> >I'm fine with it. Basically it means reverting to pre-4.043 code and
> >continue *git* development there with increased version to 5.x. And once
> >code is ready it can be released to cpan as normal (non-engineering) 5.x
> >release.
> 
> Yes, as you say.  With respect to Git I propose doing this immediately:
> 
> 1. Create a Git tag/branch off of 4.043 which is the 4.x legacy support 
> branch.

This is up to Patrick.

> 2. Revert Git master to the pre-4.043 code and then follow that with a
> commit to master that changes the DBD::mysql version to 5.0.

If everybody agree with this step, I can prepare pull request which
revert tree to this state, plus re-apply/rebase commits which were newly
accepted.

> Regardless, following point 2, mandate that all Git pull requests are made
> against the new 5.x master; the 4.x legacy branch would have no commits
> except minimal back-porting.

New pull requests are by default created against "master" branch. So if
5.x development would happen in "master" and 4.x in e.g. "legacy-4.0"
then no changes is needed.

But on github it is not possible to disallow users to create new pull
requests for non-default branch. When creating pull request there is a
button which open dialog to change target branch.

Re: DBD::mysql path forward

[pali]

I would suggest to communicate with all people behind cpan modules which
uses mysql_enable_utf8.

https://grep.metacpan.org/search?size=20=mysql_enable_utf8==

They should know if their modules are expecting old broken or behavior
of DBD::mysql or not. I would expect that are modules which needs
correct behavior as not all perl people expect such bugs...

They can update their modules to be compatible with both old buggy
behavior and new correct. Months ago I wrote update for DBD::mysql POD
documentation which mention example how to achieve it:

https://github.com/perl5-dbi/DBD-mysql/pull/119/files

Meanwhile DBD::mysql can try to play with "conflicts" relationship in
META.json to prevent installing one of the above cpan module which
depends on broken DBD::mysql until maintainer of that module fixes it.

https://metacpan.org/pod/CPAN::Meta::Spec#Prereq-Spec

I have never uses "conflicts" in META.json, so I do not know if it is
working correctly in all cpan clients, but according to documentation it
is a way how to achieve that users would not upgrade DBD::mysql to new
version if they have installed another module which needs old
DBD::mysql.

Has anybody experience with "conflicts" in META.json?

Also this would mean to prepare list of all of those cpan modules and
add them into "conflicts" section in DBD::mysql's META.json.

This can be tested e.g. by releasing engineering/beta version on cpan.

On Tuesday 12 September 2017 21:45:11 Patrick M. Galbraith wrote:
> Darren,
> 
> I agree with this as well, with the exception of 4 and 5, keeping 5.0 "pure"
> to the new way of doing things.
> 
> For releases, I think I want to understand what this will mean. Sooner or
> later, a release shows up in as a distribution package, installed on the OS,
> and I want to know what the way of communicating that to the users and
> vendors so expectations are met. That's another question of "how do we
> handle that?" and "how do we inform an OS packager/vendor, what to do?
> 
> Thank you for the great discussion!
> 
> Patrick
> 
> On 9/12/17 2:05 PM, p...@cpan.org wrote:
> >On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:
> >>On 2017-09-12 8:54 AM, Dan Book wrote:
> >>>On Tue, Sep 12, 2017 at 11:04 AM, Patrick M. Galbraith wrote:
> >>> Pali,
> >>> Yes, I agree, we'll have to create a fork pre revert and stop
> >>> accepting PRs How might we allow people time to test the fixes
> >>> to give them time? Just have them use the fork, I would
> >>> assume?
> >>>
> >>>To be clear, this sounds like a branch not a fork. If your plan is
> >>>to reinstate the mysql_enable_utf8 behavior as in 4.042 rather
> >>>than adding a new option for this behavior, then branching from
> >>>4.042 seems reasonable to me; but you should be very clear if this
> >>>is your intended approach, as this is what led to many people
> >>>corrupting data as they send blobs to mysql with the same
> >>>mysql_enable_utf8 option, and expect them to accidentally not get
> >>>encoded.
> >>Assuming that broken Unicode handling has been in DBD::mysql for a
> >>long time and that users expect this broken behavior and that fixing
> >>DBD::mysql may break user code making those assumptions...
> >>
> >>I strongly recommend that another thing happen, which is
> >>re-versioning DBD::mysql to 5.0.
> >>
> >>A declared major version change would signify to a lot of people that
> >>there are significant changes from what came before and that they
> >>should be paying closer attention and expecting the possibility that
> >>their code might break unless they make some adjustments.  Without
> >>the major version change they can more easily and reasonably expect
> >>not having compatibility breaks.
> >>
> >>Part and parcel with this is that only DBD::mysql 5.0 would have the
> >>other changes required for compatibility with newer MySQL versions
> >>or features that would be the other more normal-sounding reasons to
> >>use it.
> >>
> >>So I specifically propose the following:
> >>
> >>1. From now on, DBD::mysql versions 4.x would essentially be frozen
> >>at 4.041/4.043.  They would expressly never receive any breaking
> >>changes (but see point 3) and in particular no Unicode handling
> >>changes.  They would expressly never receive any new features.  This
> >>is the option for people whose codebases and environments work now
> >>and want to leave it alone.
> >>
> >>2. From now on, DBD::mysql versions 5.x and above would be where all
> >>active development occurs, would be the only place where Unicode
> >>handling is fixed and new MySQL versions and features are supported,
> >>and where other features are added.  Version 5.0 specifically would
> >>have all of the backwards-breaking changes at once that are
> >>currently planned or anticipated in the short term, in particular
> >>fixing the Unicode.  Anyone who is already making changes to their
> >>environment by moving to a newer MySQL version or want newer feature
> >>support will have to use 

Re: DBD::mysql path forward

[Reggie Burnett]

Patrick M. Galbraith wrote:


Hi all,

After talking to Pali and looking at how other drivers have handled 
the issue, the best way forward will be to deal with solving the UTF-8 
issue correctly as was attempted in May. This will be a problem for 
some, but only due to having to been accustomed to a work-around that 
relies on a intermittent and buggy implementation. The plan is to give 
ample time to let people prepare and comment as well as give them a 
chance to try the changes and adjust prior to stable release. I'm 
going to look through the mailing list and find particularly those who 
had problems back in May and June when we tried this before and work 
with them in advance of the change.


DBD::mysql should be on par with all the other DBD drivers and allow 
transparent usage regardless of backing RDBMS, and this really is the 
only way. It will also allow for other fixes to proceed and the driver 
to continue improving and supporting all versions and enhancements of 
MySQL and MariaDB.


Please do give us your thoughts on this as we want to be as helpful 
and transparent as possible.


I completely agree with the direction.  It may affect users but fixing 
the behavior and making it correct has got to be the right approach.  
Giving users a heads up and time to adapt is also appropriate.  Good work!

Re: DBD::mysql path forward

[Darren Duncan]

On 2017-09-12 6:45 PM, Patrick M. Galbraith wrote:

Darren,

I agree with this as well, with the exception of 4 and 5, keeping 5.0 "pure" to
the new way of doing things.


I very much agree, pure is best.  I only suggested 4 and 5 as half-hearted 
options because I knew some projects in our community liked to do things like that.



For releases, I think I want to understand what this will mean. Sooner or later,
a release shows up in as a distribution package, installed on the OS, and I want
to know what the way of communicating that to the users and vendors so
expectations are met. That's another question of "how do we handle that?" and
"how do we inform an OS packager/vendor, what to do?

Thank you for the great discussion!


Based on my experience, the fact the major version number is being changed from 
4 to 5 will communicate a lot by itself.


Many software projects use semantic versioning, see http://semver.org for a 
formal definition adopted by SQLite among other projects, and all the common 
package managers and package repositories understand it.


Generally speaking, if the first number in a dot-separated version number is 
incremented, it is treated as being a major version, and then any version 
numbers having that first part in common are considered minor versions of the 
same major.


Given the default behavior of package managers, any users performing a generic 
"keep me up to date with all the patches and bug fixes" should receive the 
latest version sharing the major version they already have, and going to a new 
major version would require a more explicit "install this separate thing" on the 
user's behalf.


Tangentially to this, this DBD::mysql is a CPAN module, CPAN conventions can 
also be followed by the package managers.


As for giving explicit communication to repository maintainers eg Debian and 
CentOS and whatever, I think that's a matter of direct human communication. 
Perhaps have the person managing the packages for MySQL itself handle it.


This all being said, for people using the CPAN clients, I believe their rules 
are simpler and they simply always get the highest numbers, ignoring major/minor 
stuff, except for Perl itself.


For the sake of CPAN clients, another possible alternative is to fork the 
namespace.  The current DBD::mysql name either means the old legacy version or 
the new better version, and then you pick a new name for the complement.


A semi-relevant example of this is DBD::SQLite back in 2003 when SQLite 3 came 
out.  In that case, SQLite 3 was a complete break from 2, neither could read the 
others' files.  They renamed the old DBD::SQLite to DBD::SQLite2 (or some such) 
which was the one for SQLite 2, and the plain DBD::SQLite name then meant SQLite 
3.  That being said, all the SQLite 2 ones were 0.x versions, and then version 
1.0 was the first SQLite 3 version.  So the fact that prior to the break the 0.x 
semantics of (anything might break) made the decision easier.


I think the decision then of which MySQL client keeps the old name depends on 
what you want to happen when users simply "get the latest" without testing.  If 
the latest features and fixes stay with the old name, then the name for the 
legacy fork could be DBD::mysql::LegacyUnicode 4.044 AND DBD::mysql jumps to 
5.0.  If the latest features go to a new name, then you could have eg 
DBD::mysql2 version 1.0; it would be like a fork, but its official.  The former 
approach will mean users with broken code will have to update their code to the 
new module name but won't have to fix their Unicode.  The latter approach means 
people wanting the new stuff change the driver name they call.


Another option that lets you keep the same name for both but requires other 
parts of the ecosystem to at least update some code, is you stick with the 
originally discussed 4-5 move, and other middleware modules add explicit 
internal code paths for "if DBD::mysql version >= 5 then expect correct Unicode 
behavior and otherwise expect and compensate for broken Unicode"; I expect 
DBIx::Class and friends will all add such checks anyway, short of hard requiring 
the new version.


Basically, its all about tradeoffs, who you want to do what work and who you 
want to be able to do zero work.


-- Darren Duncan

Re: DBD::mysql path forward

[Patrick M. Galbraith]

Darren,

I agree with this as well, with the exception of 4 and 5, keeping 5.0 
"pure" to the new way of doing things.


For releases, I think I want to understand what this will mean. Sooner 
or later, a release shows up in as a distribution package, installed on 
the OS, and I want to know what the way of communicating that to the 
users and vendors so expectations are met. That's another question of 
"how do we handle that?" and "how do we inform an OS packager/vendor, 
what to do?


Thank you for the great discussion!

Patrick

On 9/12/17 2:05 PM, p...@cpan.org wrote:

On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:

On 2017-09-12 8:54 AM, Dan Book wrote:

On Tue, Sep 12, 2017 at 11:04 AM, Patrick M. Galbraith wrote:
 Pali,
 Yes, I agree, we'll have to create a fork pre revert and stop
 accepting PRs How might we allow people time to test the fixes
 to give them time? Just have them use the fork, I would
 assume?

To be clear, this sounds like a branch not a fork. If your plan is
to reinstate the mysql_enable_utf8 behavior as in 4.042 rather
than adding a new option for this behavior, then branching from
4.042 seems reasonable to me; but you should be very clear if this
is your intended approach, as this is what led to many people
corrupting data as they send blobs to mysql with the same
mysql_enable_utf8 option, and expect them to accidentally not get
encoded.

Assuming that broken Unicode handling has been in DBD::mysql for a
long time and that users expect this broken behavior and that fixing
DBD::mysql may break user code making those assumptions...

I strongly recommend that another thing happen, which is
re-versioning DBD::mysql to 5.0.

A declared major version change would signify to a lot of people that
there are significant changes from what came before and that they
should be paying closer attention and expecting the possibility that
their code might break unless they make some adjustments.  Without
the major version change they can more easily and reasonably expect
not having compatibility breaks.

Part and parcel with this is that only DBD::mysql 5.0 would have the
other changes required for compatibility with newer MySQL versions
or features that would be the other more normal-sounding reasons to
use it.

So I specifically propose the following:

1. From now on, DBD::mysql versions 4.x would essentially be frozen
at 4.041/4.043.  They would expressly never receive any breaking
changes (but see point 3) and in particular no Unicode handling
changes.  They would expressly never receive any new features.  This
is the option for people whose codebases and environments work now
and want to leave it alone.

2. From now on, DBD::mysql versions 5.x and above would be where all
active development occurs, would be the only place where Unicode
handling is fixed and new MySQL versions and features are supported,
and where other features are added.  Version 5.0 specifically would
have all of the backwards-breaking changes at once that are
currently planned or anticipated in the short term, in particular
fixing the Unicode.  Anyone who is already making changes to their
environment by moving to a newer MySQL version or want newer feature
support will have to use DBD::mysql 5, and in the process they will
have to go through their Perl codebase and fix anything that assumes
Unicode is broken.

3. As an exception to the complete freeze otherwise mentioned, there
could be one or more DBD::mysql 4.x release whose sole purpose is to
back-port security fixes or select other bug fixes from 5.x.  Also
4.x could gain minimalist documentation changes to indicate its new
legacy status and point to 5.x.

4. Optional bonus 1:  If it is reasonably possible to support both
correct Unicode handling as well as the old broken handling in the
same codebase, I suggest DBD::mysql 5.0 could also have a user
config option where one could explicitly set it to make DBD::mysql
use the old broken behavior, while the default would be to have the
correct behavior.  This would be analogous to Perl's "use
experimental" feature.  The idea is that it would provide a
temporary stopgap for users migrating from 4.x where they could just
make the minimal change of enabling that option but they otherwise
wouldn't yet have to go through their codebase to fix all the
assumptions of wrongness.  Using this option would cause deprecation
warnings to be emitted.  Perhaps the feature could be lexical or
handle-specific if appropriate to help support piecemeal migration
to correct assumptions.  Alternately it may be be best to not have
this option at all and people simply have to fix their code on
moving to 5.x. This matter is probably its own discussion, but the
key part is it only applies to 5.x, and meanwhile the 5.x default is
correct Unicode while 4.x freezes the old bad Unicode behavior.

5. Optional bonus 2:  A user config option could be turned on but all
that it does is emit warnings in situations where the 

Re: DBD::mysql path forward

[Noel Butler]

On 13/09/2017 03:00, Darren Duncan wrote:

> 1. From now on, DBD::mysql versions 4.x would essentially be frozen at 
> 4.041/4.043.  They would expressly never receive any breaking changes (but 
> see point 3) and in particular no Unicode handling changes.  They would 
> expressly never receive any new features.  This is the option for people 
> whose codebases and environments work now and want to leave it alone.
> 
> 2. From now on, DBD::mysql versions 5.x and above would be where all active 
> development occurs, would be the only place where Unicode handling is fixed 
> and new MySQL versions and features are supported, and where other features 
> are added.  Version 5.0 specifically would have all of the backwards-breaking 
> changes at once that are currently planned or anticipated in the short term, 
> in particular fixing the Unicode.  Anyone who is already making changes to 
> their environment by moving to a newer MySQL version or want newer feature 
> support will have to use DBD::mysql 5, and in the process they will have to 
> go through their Perl codebase and fix anything that assumes Unicode is 
> broken.
> 
> 3. As an exception to the complete freeze otherwise mentioned, there could be 
> one or more DBD::mysql 4.x release whose sole purpose is to back-port 
> security fixes or select other bug fixes from 5.x.  Also 4.x could gain 
> minimalist documentation changes to indicate its new legacy status and point 
> to 5.x.

I agree with and second all these three points. 

-- 
Kind Regards, 

Noel Butler 

This Email, including any attachments, may contain legally 
privileged
information, therefore remains confidential and subject to copyright
protected under international law. You may not disseminate, discuss, or
reveal, any part, to anyone, without the authors express written
authority to do so. If you are not the intended recipient, please notify
the sender then delete all copies of this message including attachments,
immediately. Confidentiality, copyright, and legal privilege are not
waived or lost by reason of the mistaken delivery of this message. Only
PDF [1] and ODF [2] documents accepted, please do not send proprietary
formatted documents 

 

Links:
--
[1] http://www.adobe.com/
[2] http://en.wikipedia.org/wiki/OpenDocument

Re: DBD::mysql path forward

[Darren Duncan]

On 2017-09-12 11:05 AM, p...@cpan.org wrote:

On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:

I strongly recommend that another thing happen, which is
re-versioning DBD::mysql to 5.0.

1. From now on, DBD::mysql versions 4.x would essentially be frozen
at 4.041/4.043.

2. From now on, DBD::mysql versions 5.x and above would be where all
active development occurs, would be the only place where Unicode
handling is fixed and new MySQL versions and features are supported,
and where other features are added.


I'm fine with it. Basically it means reverting to pre-4.043 code and
continue *git* development there with increased version to 5.x. And once
code is ready it can be released to cpan as normal (non-engineering) 5.x
release.


Yes, as you say.  With respect to Git I propose doing this immediately:

1. Create a Git tag/branch off of 4.043 which is the 4.x legacy support branch.

2. Revert Git master to the pre-4.043 code and then follow that with a commit to 
master that changes the DBD::mysql version to 5.0.


Optionally a DBD::mysql version 5.0 could then be released, from the new master, 
that matches 4.042 in content, or otherwise an additional round of planning or 
public consultation could happen first in case any other possible breaking 
changes want to be introduced first.


Regardless, following point 2, mandate that all Git pull requests are made 
against the new 5.x master; the 4.x legacy branch would have no commits except 
minimal back-porting.


-- Darren Duncan

Re: DBD::mysql path forward

[pali]

On Tuesday 12 September 2017 19:00:59 Darren Duncan wrote:
> On 2017-09-12 8:54 AM, Dan Book wrote:
> > On Tue, Sep 12, 2017 at 11:04 AM, Patrick M. Galbraith wrote:
> > Pali,
> > Yes, I agree, we'll have to create a fork pre revert and stop
> > accepting PRs How might we allow people time to test the fixes
> > to give them time? Just have them use the fork, I would
> > assume?
> > 
> > To be clear, this sounds like a branch not a fork. If your plan is
> > to reinstate the mysql_enable_utf8 behavior as in 4.042 rather
> > than adding a new option for this behavior, then branching from
> > 4.042 seems reasonable to me; but you should be very clear if this
> > is your intended approach, as this is what led to many people
> > corrupting data as they send blobs to mysql with the same
> > mysql_enable_utf8 option, and expect them to accidentally not get
> > encoded.
> 
> Assuming that broken Unicode handling has been in DBD::mysql for a
> long time and that users expect this broken behavior and that fixing
> DBD::mysql may break user code making those assumptions...
> 
> I strongly recommend that another thing happen, which is
> re-versioning DBD::mysql to 5.0.
> 
> A declared major version change would signify to a lot of people that
> there are significant changes from what came before and that they
> should be paying closer attention and expecting the possibility that
> their code might break unless they make some adjustments.  Without
> the major version change they can more easily and reasonably expect
> not having compatibility breaks.
> 
> Part and parcel with this is that only DBD::mysql 5.0 would have the
> other changes required for compatibility with newer MySQL versions
> or features that would be the other more normal-sounding reasons to
> use it.
> 
> So I specifically propose the following:
> 
> 1. From now on, DBD::mysql versions 4.x would essentially be frozen
> at 4.041/4.043.  They would expressly never receive any breaking
> changes (but see point 3) and in particular no Unicode handling
> changes.  They would expressly never receive any new features.  This
> is the option for people whose codebases and environments work now
> and want to leave it alone.
> 
> 2. From now on, DBD::mysql versions 5.x and above would be where all
> active development occurs, would be the only place where Unicode
> handling is fixed and new MySQL versions and features are supported,
> and where other features are added.  Version 5.0 specifically would
> have all of the backwards-breaking changes at once that are
> currently planned or anticipated in the short term, in particular
> fixing the Unicode.  Anyone who is already making changes to their
> environment by moving to a newer MySQL version or want newer feature
> support will have to use DBD::mysql 5, and in the process they will
> have to go through their Perl codebase and fix anything that assumes
> Unicode is broken.
> 
> 3. As an exception to the complete freeze otherwise mentioned, there
> could be one or more DBD::mysql 4.x release whose sole purpose is to
> back-port security fixes or select other bug fixes from 5.x.  Also
> 4.x could gain minimalist documentation changes to indicate its new
> legacy status and point to 5.x.
> 
> 4. Optional bonus 1:  If it is reasonably possible to support both
> correct Unicode handling as well as the old broken handling in the
> same codebase, I suggest DBD::mysql 5.0 could also have a user
> config option where one could explicitly set it to make DBD::mysql
> use the old broken behavior, while the default would be to have the
> correct behavior.  This would be analogous to Perl's "use
> experimental" feature.  The idea is that it would provide a
> temporary stopgap for users migrating from 4.x where they could just
> make the minimal change of enabling that option but they otherwise
> wouldn't yet have to go through their codebase to fix all the
> assumptions of wrongness.  Using this option would cause deprecation
> warnings to be emitted.  Perhaps the feature could be lexical or
> handle-specific if appropriate to help support piecemeal migration
> to correct assumptions.  Alternately it may be be best to not have
> this option at all and people simply have to fix their code on
> moving to 5.x. This matter is probably its own discussion, but the
> key part is it only applies to 5.x, and meanwhile the 5.x default is
> correct Unicode while 4.x freezes the old bad Unicode behavior.
> 
> 5. Optional bonus 2:  A user config option could be turned on but all
> that it does is emit warnings in situations where the Unicode
> behavior was fixed, to prod users to check that particular spot in
> their code that may have been impacted, this would be more like a
> typical "use warnings" maybe.  This is just a tool to help users
> convert their code and find impacted areas.
> 
> Note, 4.x versions have already been in use for a full decade, so
> there shouldn't be new version fatigue.
> 
> So Patrick 

Re: DBD::mysql path forward

[pali]

On Tuesday 12 September 2017 12:27:25 p...@cpan.org wrote:
> To prove fact that other DBI drivers (e.g. Pg or SQLite) had fixed
> similar/same UTF-8 issue as MySQL has and behave Perl-correctly, I
> would provide test cases so you would see difference between Pg,
> SQLite and mysql DBI drivers.

And here is promised script:

===

use strict;
use warnings;
use utf8;
binmode \*STDOUT, ':utf8';
binmode \*STDERR, ':utf8';
use DBI;

sub ords { $_[0] . ' (' . (join ', ', map ord, split //, $_[0]) . ')' }

# DBD::Pg enables Unicode <--> UTF-8 automatically
my $pg_dbh = DBI->connect("dbi:Pg:dbname=postgres", "", "");

# DBD::mysql needs mysql_enable_utf8 and mysql_enable_utf8mb4 for Unicode <--> 
UTF-8
my $mysql_dbh = DBI->connect("dbi:mysql:dbname=mysql", "root", "", { 
mysql_enable_utf8 => 1, mysql_enable_utf8mb4 => 1 });
# MySQL server by default uses latin1
# Without switch to utf8mb4, Unicode code points above U+FF cannot be stored (á 
is below U+FF, č not)
# Another option is to specify own charset in CREATE TABLE
$mysql_dbh->do("SET character_set_database='utf8mb4'");
$mysql_dbh->do("SET character_set_server='utf8mb4'");

# DBD::SQLite needs sqlite_unicode for Unicode <--> UTF-8
my $sqlite_dbh = DBI->connect("dbi:SQLite:dbname=:memory:", "", "", { 
sqlite_unicode => 1 });

for my $dbh ($pg_dbh, $mysql_dbh, $sqlite_dbh) {
  for my $ins ("\xC3\xA1", "\N{U+C3}\N{U+A1}", "á", "č", "\x{263A}", 
"\N{U+263A}", "☺") {
{
  $dbh->do("CREATE TEMPORARY TABLE t(s VARCHAR(10))");
  my $sth = $dbh->prepare("INSERT INTO t(s) VALUES('$ins')");
  $sth->execute();
  my $fetch = $dbh->selectrow_array("SELECT s FROM t");
  print $dbh->{Driver}->{Name} . ' without bind - insert: ' . ords($ins) . 
'; fetch: ' . ords($fetch) . ' - ' . ($ins eq $fetch ? 'OK' : 'FAIL') . "\n";
  $dbh->do("DROP TABLE t");
}
{
  $dbh->do("CREATE TEMPORARY TABLE t(s VARCHAR(10))");
  my $sth = $dbh->prepare("INSERT INTO t(s) VALUES(?)");
  $sth->execute($ins);
  my $fetch = $dbh->selectrow_array("SELECT s FROM t");
  print $dbh->{Driver}->{Name} . ' with bind - insert: ' . ords($ins) . '; 
fetch: ' . ords($fetch) . ' - ' . ($ins eq $fetch ? 'OK' : 'FAIL') . "\n";
  $dbh->do("DROP TABLE t");
}
  }
}

===

It is simple "identity" script, which stores value into database,
fetches it back and then compare results.

Look on results with DBD::mysql 4.041, 4.042 and 4.043.
Are all users of DBD::mysql aware of such results?

And specially look at results with DBD::Pg prior to version 3.3.0 and up.
Also DBD::SQLite prior to version 1.43_04.

Basically DBD::Pg (pg_enable_utf8 = -1) and DBD::SQLite
(via sqlite_unicode=1) had exactly same bug as DBD::mysql.

Re: DBD::mysql path forward

[Darren Duncan]

On 2017-09-12 8:54 AM, Dan Book wrote:

On Tue, Sep 12, 2017 at 11:04 AM, Patrick M. Galbraith wrote:
Pali,
Yes, I agree, we'll have to create a fork pre revert and stop accepting PRs
How might we allow people time to test the fixes to give them time? Just
have them use the fork, I would assume?

To be clear, this sounds like a branch not a fork. If your plan is to reinstate
the mysql_enable_utf8 behavior as in 4.042 rather than adding a new option for
this behavior, then branching from 4.042 seems reasonable to me; but you should
be very clear if this is your intended approach, as this is what led to many
people corrupting data as they send blobs to mysql with the same
mysql_enable_utf8 option, and expect them to accidentally not get encoded.


Assuming that broken Unicode handling has been in DBD::mysql for a long time and 
that users expect this broken behavior and that fixing DBD::mysql may break user 
code making those assumptions...


I strongly recommend that another thing happen, which is re-versioning 
DBD::mysql to 5.0.


A declared major version change would signify to a lot of people that there are 
significant changes from what came before and that they should be paying closer 
attention and expecting the possibility that their code might break unless they 
make some adjustments.  Without the major version change they can more easily 
and reasonably expect not having compatibility breaks.


Part and parcel with this is that only DBD::mysql 5.0 would have the other 
changes required for compatibility with newer MySQL versions or features that 
would be the other more normal-sounding reasons to use it.


So I specifically propose the following:

1. From now on, DBD::mysql versions 4.x would essentially be frozen at 
4.041/4.043.  They would expressly never receive any breaking changes (but see 
point 3) and in particular no Unicode handling changes.  They would expressly 
never receive any new features.  This is the option for people whose codebases 
and environments work now and want to leave it alone.


2. From now on, DBD::mysql versions 5.x and above would be where all active 
development occurs, would be the only place where Unicode handling is fixed and 
new MySQL versions and features are supported, and where other features are 
added.  Version 5.0 specifically would have all of the backwards-breaking 
changes at once that are currently planned or anticipated in the short term, in 
particular fixing the Unicode.  Anyone who is already making changes to their 
environment by moving to a newer MySQL version or want newer feature support 
will have to use DBD::mysql 5, and in the process they will have to go through 
their Perl codebase and fix anything that assumes Unicode is broken.


3. As an exception to the complete freeze otherwise mentioned, there could be 
one or more DBD::mysql 4.x release whose sole purpose is to back-port security 
fixes or select other bug fixes from 5.x.  Also 4.x could gain minimalist 
documentation changes to indicate its new legacy status and point to 5.x.


4. Optional bonus 1:  If it is reasonably possible to support both correct 
Unicode handling as well as the old broken handling in the same codebase, I 
suggest DBD::mysql 5.0 could also have a user config option where one could 
explicitly set it to make DBD::mysql use the old broken behavior, while the 
default would be to have the correct behavior.  This would be analogous to 
Perl's "use experimental" feature.  The idea is that it would provide a 
temporary stopgap for users migrating from 4.x where they could just make the 
minimal change of enabling that option but they otherwise wouldn't yet have to 
go through their codebase to fix all the assumptions of wrongness.  Using this 
option would cause deprecation warnings to be emitted.  Perhaps the feature 
could be lexical or handle-specific if appropriate to help support piecemeal 
migration to correct assumptions.  Alternately it may be be best to not have 
this option at all and people simply have to fix their code on moving to 5.x. 
This matter is probably its own discussion, but the key part is it only applies 
to 5.x, and meanwhile the 5.x default is correct Unicode while 4.x freezes the 
old bad Unicode behavior.


5. Optional bonus 2:  A user config option could be turned on but all that it 
does is emit warnings in situations where the Unicode behavior was fixed, to 
prod users to check that particular spot in their code that may have been 
impacted, this would be more like a typical "use warnings" maybe.  This is just 
a tool to help users convert their code and find impacted areas.


Note, 4.x versions have already been in use for a full decade, so there 
shouldn't be new version fatigue.


So Patrick and Pali and others, what think of my proposal?

-- Darren Duncan

Re: DBD::mysql path forward

[pali]

Another option is to revert that "big revert commit" in git and continue 
development. Then people would be able to create pull request without 
problem...

Just need to be sure that new normal release would not be put to cpan. 
But it would be possible to put beta/engineering releases like it was 
between 4.041 and 4.042.

Any comments from other people about this idea?

On Tuesday 12 September 2017 17:04:18 Patrick M. Galbraith wrote:
> Pali,
> 
> Yes, I agree, we'll have to create a fork pre revert and stop
> accepting PRs
> 
> How might we allow people time to test the fixes to give them time?
> Just have them use the fork, I would assume?
> 
> Regards,
> 
> Patrick
> 
> On 9/12/17 6:27 AM, p...@cpan.org wrote:
> > On Tuesday 12 September 2017 05:40:31 Patrick M. Galbraith wrote:
> >> Hi all,
> >> 
> >> After talking to Pali and looking at how other drivers have
> >> handled the issue, the best way forward will be to deal with
> >> solving the UTF-8 issue correctly as was attempted in May. This
> >> will be a problem for some, but only due to having to been
> >> accustomed to a work-around that relies on a intermittent and
> >> buggy implementation. The plan is to give ample time to let
> >> people prepare and comment as well as give them a chance to try
> >> the changes and adjust prior to stable release. I'm going to look
> >> through the mailing list and find particularly those who had
> >> problems back in May and June when we tried this before and work
> >> with them in advance of the change.
> > 
> > Hi!
> > 
> > To prove fact that other DBI drivers (e.g. Pg or SQLite) had fixed
> > similar/same UTF-8 issue as MySQL has and behave Perl-correctly, I
> > would provide test cases so you would see difference between Pg,
> > SQLite and mysql DBI drivers.
> > 
> > There are also many modules on cpan which uses mysql DBI driver.
> > But I'm not sure if all those modules expects buggy behavior or
> > not...
> > 
> >> DBD::mysql should be on par with all the other DBD drivers and
> >> allow transparent usage regardless of backing RDBMS, and this
> >> really is the only way. It will also allow for other fixes to
> >> proceed and the driver to continue improving and supporting all
> >> versions and enhancements of MySQL and MariaDB.
> >> 
> >> Please do give us your thoughts on this as we want to be as
> >> helpful and transparent as possible.
> > 
> > If we are going to have UTF-8 fix in DBD::mysql, I would suggest
> > one thing right now:
> > 
> > STOP MERGING AND ACCEPTING PULL REQUEST TO GITHUB DBD::MYSQL
> > PROJECT!
> > 
> > Reason is simple: Current git master repository at version 4.043
> > (against which are creating new pull requests) is reset to version
> > 4.041 and couple of changes, including lot of bug fixes from 4.042
> > are missing
> > 
> > People in 4.043 (and git master) re-discovered 4.041 bugs, which
> > were fixed in 4.042 and started to fixing them again, either
> > wrongly or in different way. Or just started complaining about
> > them, that they are again present.
> > 
> > Those (reverted) UTF-8 commits changes couple of lines, plus other
> > fixes depends on it. Once git master repository starts to
> > divergent, it would be very hard to incorporate UTF-8 fixes again.
> > 
> > And speaking by myself, for me it would be easier to create a fork
> > of DBD::mysql from point before big revert as trying to re-apply
> > and rebase those patches on top of master with couple of
> > changes...

Re: DBD::mysql path forward

[Dan Book]

On Tue, Sep 12, 2017 at 11:54 AM, Dan Book  wrote:

> On Tue, Sep 12, 2017 at 11:04 AM, Patrick M. Galbraith 
> wrote:
>
>> Pali,
>>
>> Yes, I agree, we'll have to create a fork pre revert and stop accepting
>> PRs
>>
>> How might we allow people time to test the fixes to give them time? Just
>> have them use the fork, I would assume?
>>
>> Regards,
>>
>> Patrick
>>
>
> To be clear, this sounds like a branch not a fork. If your plan is to
> reinstate the mysql_enable_utf8 behavior as in 4.042 rather than adding a
> new option for this behavior, then branching from 4.042 seems reasonable to
> me; but you should be very clear if this is your intended approach, as this
> is what led to many people corrupting data as they send blobs to mysql with
> the same mysql_enable_utf8 option, and expect them to accidentally not get
> encoded.
>
> -Dan
>

And one other note, if you do reinstate the 4.042 behavior, due to the
backcompat breakage at the very least this should be a 5.0 release.

-Dan

Re: DBD::mysql path forward

[Dan Book]

On Tue, Sep 12, 2017 at 11:04 AM, Patrick M. Galbraith 
wrote:

> Pali,
>
> Yes, I agree, we'll have to create a fork pre revert and stop accepting PRs
>
> How might we allow people time to test the fixes to give them time? Just
> have them use the fork, I would assume?
>
> Regards,
>
> Patrick
>

To be clear, this sounds like a branch not a fork. If your plan is to
reinstate the mysql_enable_utf8 behavior as in 4.042 rather than adding a
new option for this behavior, then branching from 4.042 seems reasonable to
me; but you should be very clear if this is your intended approach, as this
is what led to many people corrupting data as they send blobs to mysql with
the same mysql_enable_utf8 option, and expect them to accidentally not get
encoded.

-Dan

Re: DBD::mysql path forward

[Patrick M. Galbraith]

Pali,

Yes, I agree, we'll have to create a fork pre revert and stop accepting PRs

How might we allow people time to test the fixes to give them time? Just 
have them use the fork, I would assume?


Regards,

Patrick

On 9/12/17 6:27 AM, p...@cpan.org wrote:

On Tuesday 12 September 2017 05:40:31 Patrick M. Galbraith wrote:

Hi all,

After talking to Pali and looking at how other drivers have handled the
issue, the best way forward will be to deal with solving the UTF-8 issue
correctly as was attempted in May. This will be a problem for some, but only
due to having to been accustomed to a work-around that relies on a
intermittent and buggy implementation. The plan is to give ample time to let
people prepare and comment as well as give them a chance to try the changes
and adjust prior to stable release. I'm going to look through the mailing
list and find particularly those who had problems back in May and June when
we tried this before and work with them in advance of the change.

Hi!

To prove fact that other DBI drivers (e.g. Pg or SQLite) had fixed
similar/same UTF-8 issue as MySQL has and behave Perl-correctly, I would
provide test cases so you would see difference between Pg, SQLite and
mysql DBI drivers.

There are also many modules on cpan which uses mysql DBI driver. But I'm
not sure if all those modules expects buggy behavior or not...


DBD::mysql should be on par with all the other DBD drivers and allow
transparent usage regardless of backing RDBMS, and this really is the only
way. It will also allow for other fixes to proceed and the driver to
continue improving and supporting all versions and enhancements of MySQL and
MariaDB.

Please do give us your thoughts on this as we want to be as helpful and
transparent as possible.

If we are going to have UTF-8 fix in DBD::mysql, I would suggest one
thing right now:

STOP MERGING AND ACCEPTING PULL REQUEST TO GITHUB DBD::MYSQL PROJECT!

Reason is simple: Current git master repository at version 4.043
(against which are creating new pull requests) is reset to version 4.041
and couple of changes, including lot of bug fixes from 4.042 are
missing

People in 4.043 (and git master) re-discovered 4.041 bugs, which were
fixed in 4.042 and started to fixing them again, either wrongly or in
different way. Or just started complaining about them, that they are
again present.

Those (reverted) UTF-8 commits changes couple of lines, plus other fixes
depends on it. Once git master repository starts to divergent, it would
be very hard to incorporate UTF-8 fixes again.

And speaking by myself, for me it would be easier to create a fork of
DBD::mysql from point before big revert as trying to re-apply and rebase
those patches on top of master with couple of changes...

Re: DBD::mysql path forward

[pali]

On Tuesday 12 September 2017 05:40:31 Patrick M. Galbraith wrote:
> Hi all,
> 
> After talking to Pali and looking at how other drivers have handled the
> issue, the best way forward will be to deal with solving the UTF-8 issue
> correctly as was attempted in May. This will be a problem for some, but only
> due to having to been accustomed to a work-around that relies on a
> intermittent and buggy implementation. The plan is to give ample time to let
> people prepare and comment as well as give them a chance to try the changes
> and adjust prior to stable release. I'm going to look through the mailing
> list and find particularly those who had problems back in May and June when
> we tried this before and work with them in advance of the change.

Hi!

To prove fact that other DBI drivers (e.g. Pg or SQLite) had fixed
similar/same UTF-8 issue as MySQL has and behave Perl-correctly, I would
provide test cases so you would see difference between Pg, SQLite and
mysql DBI drivers.

There are also many modules on cpan which uses mysql DBI driver. But I'm
not sure if all those modules expects buggy behavior or not...

> DBD::mysql should be on par with all the other DBD drivers and allow
> transparent usage regardless of backing RDBMS, and this really is the only
> way. It will also allow for other fixes to proceed and the driver to
> continue improving and supporting all versions and enhancements of MySQL and
> MariaDB.
> 
> Please do give us your thoughts on this as we want to be as helpful and
> transparent as possible.

If we are going to have UTF-8 fix in DBD::mysql, I would suggest one
thing right now:

STOP MERGING AND ACCEPTING PULL REQUEST TO GITHUB DBD::MYSQL PROJECT!

Reason is simple: Current git master repository at version 4.043
(against which are creating new pull requests) is reset to version 4.041
and couple of changes, including lot of bug fixes from 4.042 are
missing

People in 4.043 (and git master) re-discovered 4.041 bugs, which were
fixed in 4.042 and started to fixing them again, either wrongly or in
different way. Or just started complaining about them, that they are
again present.

Those (reverted) UTF-8 commits changes couple of lines, plus other fixes
depends on it. Once git master repository starts to divergent, it would
be very hard to incorporate UTF-8 fixes again.

And speaking by myself, for me it would be easier to create a fork of
DBD::mysql from point before big revert as trying to re-apply and rebase
those patches on top of master with couple of changes...