Re: DBD::mysql path forward

[Darren Duncan]

Patrick and Pali, each of you please respond to the lists to confirm that what I 
say below is what you also understand is the primary plan, and if not, then say 
why not; in prior discussion I recall you agreed with it.


Assuming they agree, the concerns of Night Light and those he is speaking for 
should be handily satisfied.


The whole discussion on the mailing lists that I recall and participated in 
seemed to consensus on branching DBD::sqlite in order to best satisfy the needs 
of all stakeholders.


There would be one version control with 2 active release branches.

A maintenance branch would exist starting from the 4.041/3 release on which 
further stable releases named DBD::mysql 4.x would be made, and the primary goal 
of this branch is to not break/corrupt anything that relied on 4.041 behavior. 
So people with legacy projects that just use DBD::mysql and made particular 
assumptions on its handling of Unicode or Blobs etc won't have corruption 
introduced because DBD::mysql changed its handling of those things.  People 
without knowledge of CPAN or the development process will get something that 
just continues to work for them and doesn't corrupt.  For all intents and 
purposes this branch would be frozen but could have select security or bug fixes 
that comply with the mandate and don't involve changing Unicode etc.


The master branch would then have all the short term breaking changes including 
the 4.042 Unicode fixes and would have all the significant feature changes 
including compatibility with newer MySQL major versions.  All of its releases 
would be under a new namespace such as DBD::mysql2 and start at version 5.0 to 
signify that large changes happened which might possibly break code or cause 
corruption if user code doesn't specifically account for the differences.  Being 
a different name this is strictly opt-in, so the only ones using DBD::mysql2 are 
those that explicitly opted in to it, and by doing so they also opted in to 
fixing anything with their code that needed fixing to not corrupt data while 
using it.


The concept of having a single driver with toggled behavior eg a 
mysql_enable_proper_unicode flag was already rejected as unwieldy to implement, 
especially for such deep rooted changes as properly handling Unicode.


Note that I expect that most other higher-level projects that use MySQL would 
NOT branch like this, instead having internal logic or their own toggle to work 
with DBD::mysql vs DBD::mysql2, or a few may branch, but the key thing is that 
branching DBD::mysql does NOT necessitate doing so in the rest of the ecosystem.


-- Darren Duncan

On 2017-11-07 12:07 PM, Night Light wrote:

Proposed, but no made decisions posted afterwards.

The last proposal is to re-commit the rejected 4.042 changes into the 4.043
master branch and only work on fixes that came after June.

The git issue that regards improperly encoding of BLOBs is opened on April 6th
(hence me sending the message to prevent a recurring cycle).

https://github.com/perl5-dbi/DBD-mysql/issues/117

On Tue, Nov 7, 2017 at 8:57 PM, Michiel Beijen wrote:

To me, the only real option is to make a new option in DBD::mysql;
mysql_enable_proper_unicode or the like which you would knowingly set
in your application code, which will expose the new behaviour. I
understand this is difficult, but I really think it's the only way.

If in the short term this is not feasible, it *could* be possible, in
my eyes, to release a DBD::mysql2 or similar that does *correct*
behaviour. Also in that case this is something the application
developer should set explicitly in his connection string.

This DBD::mysql2 or similar could live in another git repository, but
preferably in the same repo, and even in the same CPAN distribution as
DBD::mysql, and eventually the goal should be that they re-unite and
using DBD::mysql2 would really be the same as to use the
'mysql_enable_proper_unicode' option in DBD::mysql.

--
Michiel

On Tue, Nov 7, 2017 at 7:41 PM, Darren Duncan > wrote:
> My understanding from the last discussion on this matter is that it was
> agreed DBD::mysql would be forked such that the existing DBD::mysql name
> would be frozen at 4.041/3 indefinitely and that the 4.042 changes plus 
any
> further feature development would take place under a new name such as
> DBD::mysql2 version 5.0.

Re: DBD::mysql path forward

[Michiel Beijen]

I think we have not decided how we should fix the unicode issues.
Previously in this thread, it seemed to be pali and Dan Book had other
ideas about this than most others.

I wrote on this topic earlier to pali:

On Wed, Sep 13, 2017 at 9:21 AM,   wrote:
> I would suggest to communicate with all people behind cpan modules which
> uses mysql_enable_utf8.
>
> They should know if their modules are expecting old broken or behavior
> of DBD::mysql or not. I would expect that are modules which needs
> correct behavior as not all perl people expect such bugs...
>
> They can update their modules to be compatible with both old buggy
> behavior and new correct. Months ago I wrote update for DBD::mysql POD
> documentation which mention example how to achieve it:

I don't think this is a good idea. There is much more code using
DBD::mysql than what you see on CPAN. If we are going to introduce
changes in behavior, they should really be opt-in and not opt-out.
Because otherwise it can corrupt peoples data and we should not want
that.

> Meanwhile DBD::mysql can try to play with "conflicts" relationship in
> META.json to prevent installing one of the above cpan module which
> depends on broken DBD::mysql until maintainer of that module fixes it.
>
> https://metacpan.org/pod/CPAN::Meta::Spec#Prereq-Spec
>
> I have never uses "conflicts" in META.json, so I do not know if it is
> working correctly in all cpan clients, but according to documentation it
> is a way how to achieve that users would not upgrade DBD::mysql to new
> version if they have installed another module which needs old
> DBD::mysql.

Please understand that this is not a CPAN-only world. Many people
install DBD::mysql using their linux package managers, especially for
server installations of software. And some software which uses
DBD::mysql (for instance OTRS, or Bugzilla) is typically not installed
from CPAN. Of course the same goes for peoples internal systems.

And that is assuming the 'conflicts' in META.json would actually work
as expected.

To me, the only real option is to make a new option in DBD::mysql;
mysql_enable_proper_unicode or the like which you would knowingly set
in your application code, which will expose the new behaviour. I
understand this is difficult, but I really think it's the only way.

If in the short term this is not feasible, it *could* be possible, in
my eyes, to release a DBD::mysql2 or similar that does *correct*
behaviour. Also in that case this is something the application
developer should set explicitly in his connection string.

This DBD::mysql2 or similar could live in another git repository, but
preferably in the same repo, and even in the same CPAN distribution as
DBD::mysql, and eventually the goal should be that they re-unite and
using DBD::mysql2 would really be the same as to use the
'mysql_enable_proper_unicode' option in DBD::mysql.

--
Michiel

On Tue, Nov 7, 2017 at 7:41 PM, Darren Duncan  wrote:
> My understanding from the last discussion on this matter is that it was
> agreed DBD::mysql would be forked such that the existing DBD::mysql name
> would be frozen at 4.041/3 indefinitely and that the 4.042 changes plus any
> further feature development would take place under a new name such as
> DBD::mysql2 version 5.0.
>
> So those people using "DBD::mysql" would not experience any changes in
> behavior from what they were used to and BLOB etc should not break.  Whereas
> people that want the Unicode fixes and other features would use the
> DBD::mysql2 and fix their code to be compatible with its breaking changes.
>
> I thought this was settled, and takes care of your concerns, so it was just
> a matter of "make it so".
>
> (Anyone replying to this message/thread, please do NOT include my personal
> email address as an explicit recipient, only send it to the list addresses,
> I will get those.)
>
> -- Darren Duncan
>
>
> On 2017-11-07 3:41 AM, Night Light wrote:
>>
>> For the reason of "silence":
>> I've spoken to other users to hear that they have passively withdrawn from
>> this
>> discussion as they are not motivated to be part of a release where
>> concerns
>> about backwards compatibility are ignored. One of the users wrote earlier
>> (replace the word "sector" with "release"):
>>
>> "When you're dealing with software that's purpose in life is to not
>> corrupt
>> data, and have data there tomorrow, you go out of your way not to break
>> that
>> promise. There's no point in being involved in this sector if you don't
>> care to
>> deliver on that promise."
>>
>> Re-releasing the 4.042 changes will break the contract of a long-standing
>> interface and corrupt all BLOB data when using "do()". These changes do
>> therefore more harm than good.
>> Putting these utf8 changes in the freezer until a suggestion is made that
>> will
>> add the functionality instead of replacing it is not a sin. The PG driver
>> has
>> for instance also a similar issue open without plans to fix 

Re: DBD::mysql path forward

[Darren Duncan]

My understanding from the last discussion on this matter is that it was agreed 
DBD::mysql would be forked such that the existing DBD::mysql name would be 
frozen at 4.041/3 indefinitely and that the 4.042 changes plus any further 
feature development would take place under a new name such as DBD::mysql2 
version 5.0.


So those people using "DBD::mysql" would not experience any changes in behavior 
from what they were used to and BLOB etc should not break.  Whereas people that 
want the Unicode fixes and other features would use the DBD::mysql2 and fix 
their code to be compatible with its breaking changes.


I thought this was settled, and takes care of your concerns, so it was just a 
matter of "make it so".


(Anyone replying to this message/thread, please do NOT include my personal email 
address as an explicit recipient, only send it to the list addresses, I will get 
those.)


-- Darren Duncan

On 2017-11-07 3:41 AM, Night Light wrote:

For the reason of "silence":
I've spoken to other users to hear that they have passively withdrawn from this
discussion as they are not motivated to be part of a release where concerns
about backwards compatibility are ignored. One of the users wrote earlier
(replace the word "sector" with "release"):

"When you're dealing with software that's purpose in life is to not corrupt
data, and have data there tomorrow, you go out of your way not to break that
promise. There's no point in being involved in this sector if you don't care to
deliver on that promise."

Re-releasing the 4.042 changes will break the contract of a long-standing
interface and corrupt all BLOB data when using "do()". These changes do
therefore more harm than good.
Putting these utf8 changes in the freezer until a suggestion is made that will
add the functionality instead of replacing it is not a sin. The PG driver has
for instance also a similar issue open without plans to fix it anytime soon.
https://rt.cpan.org/Public/Bug/Display.html?id=122991

What is your objection against using the current 4.043 branch and work on
outstanding fixes, do a pre-release, a period of time for people to test/try
out, then release?

Re: DBD::mysql path forward

[Night Light]

For the reason of "silence":
I've spoken to other users to hear that they have passively withdrawn from
this discussion as they are not motivated to be part of a release where
concerns about backwards compatibility are ignored. One of the users wrote
earlier (replace the word "sector" with "release"):

"When you're dealing with software that's purpose in life is to not corrupt
data, and have data there tomorrow, you go out of your way not to break
that promise. There's no point in being involved in this sector if you
don't care to deliver on that promise."

Re-releasing the 4.042 changes will break the contract of a long-standing
interface and corrupt all BLOB data when using "do()". These changes do
therefore more harm than good.
Putting these utf8 changes in the freezer until a suggestion is made that
will add the functionality instead of replacing it is not a sin. The PG
driver has for instance also a similar issue open without plans to fix it
anytime soon.
https://rt.cpan.org/Public/Bug/Display.html?id=122991

What is your objection against using the current 4.043 branch and work on
outstanding fixes, do a pre-release, a period of time for people to
test/try out, then release?

On Mon, Nov 6, 2017 at 11:32 PM, Patrick Galbraith  wrote:

> Pali,
>
> I totally understand - timing timing - I started a new job and found out I
> had to prepare to talk about MySQL/Kubernetes at kubecon, so I apologize,
> and didn't plan it that way. Let me re-review those emails and give a
> decision, of course, your decision as well.
>
> As far as maintainer, you are an equal, and if you need to be seen as a
> decision maker and we work by consensus with you having equal decision. I'm
> glad with that as well because I do appreciate all of your contributions.
> I'm sorry for what you mentioned in the past though it was more of a
> reaction to deal with "customers" having issues and working panic mode, not
> a slight to your efforts. What would you prefer?
>
> So, what would be ideal for us to move forward is to get the utf8 fixes
> into the master branch, then work in the subsequent fixes that came after
> June, do a pre-release, a period of time for people to test/try out, then
> release. What sort of git gymnastics does this require?  What do you think?
>
> I think the key here to have a single driver that supports both Maria and
> MySQL both and not have splintering, and more than anything, provide
> stability as well as new features for users.
>
> Regards,
>
> Patrick
>
>
>
> On 11/05/2017 04:24 AM, p...@cpan.org wrote:
>
> Hi!
>
> Have you got replies from people who you informed/asked about that
> problems?
>
> What is needed from you is to decide how to handle problems with
> encoding and other bugs reported for DBD-mysql project [1].
>
> I sent in previous email some proposed solutions and I'm just waiting
> which would be chosen... I just do not like "silence" and no step
> forward. Just to note that Dan Book asked month ago for next steps.
>
> Also, I'm not very happy to see being co-maintainer of DBD::mysql,
> specially in this time when nothing happened for 5 months, plus some
> people opposed to me and wanted to revert bug fixes and contributions by
> me...
>
> [1] - https://rt.cpan.org/Public/Dist/Display.html?Name=DBD-mysql
>
> On Friday 03 November 2017 16:17:09 Patrick M. Galbraith wrote:
>
> Pali,
>
> Sorry, I wanted to give it time for people, plus have been very busy.
> I'm ready, just let me know when we can get code merged back in--
> what do you need me to do? I will need help with this.
>
> Regards,
>
> Patrick
>
> On 11/2/17 1:43 PM, p...@cpan.org wrote:
>
> On Tuesday 12 September 2017 11:40:31 Patrick M. Galbraith wrote:
>
> Hi all,
>
> After talking to Pali and looking at how other drivers have
> handled the issue, the best way forward will be to deal with
> solving the UTF-8 issue correctly as was attempted in May. This
> will be a problem for some, but only due to having to been
> accustomed to a work-around that relies on a intermittent and
> buggy implementation. The plan is to give ample time to let
> people prepare and comment as well as give them a chance to try
> the changes and adjust prior to stable release. I'm going to look
> through the mailing list and find particularly those who had
> problems back in May and June when we tried this before and work
> with them in advance of the change.
>
> DBD::mysql should be on par with all the other DBD drivers and
> allow transparent usage regardless of backing RDBMS, and this
> really is the only way. It will also allow for other fixes to
> proceed and the driver to continue improving and supporting all
> versions and enhancements of MySQL and MariaDB.
>
> Please do give us your thoughts on this as we want to be as
> helpful and transparent as possible.
>
> Thank you!
>
> Patrick
>
> Hi! Nearly another two months passed and there is no progress in
> unblocking development.
>
> Currently more people started reporting problems