Re: DBD::mysql path forward

[Night Light]

You repeatedly expressed that you are not willing to maintain this module.
This practically means that code is being asked to be released but somebody
else needs to maintain it.
Suggestions to change this code are next to that ignored. Questions to park
the blocking encoding issue and at least go through with security patches
are also ignored.
The code is simply bundled and presented as 'take it or leave it'.

The result is the project being stalled because that apart from the fact
that the code never will pass UAT (as it changes the interface and causes
database corruption) there is one fact of reality: you are simply the only
person that is actually willing to change the driver code (and I would like
to thank you for that).
In other words: there are currently no alternative resources available that
can assist overcoming these hurdles.

With this background I ask/asked if there is at least one other person
willing to review the driver code changes (248 changed lines in dbdimp.c)
to make sure that the judgement that forking is inevitable and is not based
on the opinion of one person but shared among others.

I believe your word and expect that the outcome is that forking is indeed
necessary, but this step has another advantage which is that somebody else
also understands the code which will help ensuring that DBD::mysql can be
maintained and remain stable.

Kind Regards,
Night Light


On Fri, Nov 10, 2017 at 9:34 AM, <p...@cpan.org> wrote:

> I do not understand what (and how) your proposal with =2 solve. Probably
> nothing and people would again start after final release again
> complaining...
>
> On Friday 10 November 2017 09:24:35 Night Light wrote:
> > Forking would take away my concerns (thank you all for suggesting that)
> but
> > one thing pops into my mind:
> > A decision to fork a mature module is not a light decision and should
> only
> > be considered as a last resort (when ran out of all options).
> > So far it is all based on the opinion of only one developer who finds it
> > hard or probably impossible to implement.
> >
> > Can before forking at least a second person with C knowledge open
> dbdimp.c
> > and measure if it is indeed unfeasible to make the module backward
> > compatible by adding an option as suggested?
> >
> > Michiel, do you know C? Can you do that 5-minute check?
> >
> > The breaking is due to the fact that the prepared SQL statement and the
> > bound parameters will be apart from what is returned from MySQL is
> encoded
> > as well.
> > The mysql_enable_utf8/mysql_enable_utf8mb4 flags already arrange logic
> to
> > set if utf8 encoding/decoding should be done or not.
> > I'm not a C expert but this is what I find in dbdimp.c of 4.042 about it:
> >
> > dbd_st_prepare_sv function:
> > bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> > get_statement(aTHX_ statement_sv, enable_utf8, ,
> _len);
> >
> > dbd_bind_ph function:
> > bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
> > bind_param(_sth->params[idx], value, sql_type, idx+1, enable_utf8);
> >
> >
> > Would changing that into the following make the module backwards
> compatible?
> > bool enable_utf8 = (imp_dbh->enable_utf8 == 2 || imp_dbh->enable_utf8mb4
> ==
> > 2);
> >
> > enable_utf8 0 = no encoding of input and only decoding of output
> > enable_utf8 1 = decode only output (4.043 mode)
> > enable_utf8 2 = encoded input and decode output (4.042 mode)
>
>

Re: DBD::mysql path forward

[Night Light]

Forking would take away my concerns (thank you all for suggesting that) but
one thing pops into my mind:
A decision to fork a mature module is not a light decision and should only
be considered as a last resort (when ran out of all options).
So far it is all based on the opinion of only one developer who finds it
hard or probably impossible to implement.

Can before forking at least a second person with C knowledge open dbdimp.c
and measure if it is indeed unfeasible to make the module backward
compatible by adding an option as suggested?

Michiel, do you know C? Can you do that 5-minute check?

The breaking is due to the fact that the prepared SQL statement and the
bound parameters will be apart from what is returned from MySQL is encoded
as well.
The mysql_enable_utf8/mysql_enable_utf8mb4 flags already arrange logic to
set if utf8 encoding/decoding should be done or not.
I'm not a C expert but this is what I find in dbdimp.c of 4.042 about it:

dbd_st_prepare_sv function:
bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
get_statement(aTHX_ statement_sv, enable_utf8, , _len);

dbd_bind_ph function:
bool enable_utf8 = (imp_dbh->enable_utf8 || imp_dbh->enable_utf8mb4);
bind_param(_sth->params[idx], value, sql_type, idx+1, enable_utf8);


Would changing that into the following make the module backwards compatible?
bool enable_utf8 = (imp_dbh->enable_utf8 == 2 || imp_dbh->enable_utf8mb4 ==
2);

enable_utf8 0 = no encoding of input and only decoding of output
enable_utf8 1 = decode only output (4.043 mode)
enable_utf8 2 = encoded input and decode output (4.042 mode)

On Fri, Nov 10, 2017 at 8:37 AM, Darren Duncan 
wrote:

> Michael, why can't you accept moving forward under a new module name?  Why
> does it have to be under the old name?  When people purposefully want to
> upgrade they purposefully choose the new module name in order to do so.
> What is the actual problem in that? -- Darren Duncan
>
> On 2017-11-09 10:59 PM, Michiel Beijen wrote:
>
>> On Fri, Nov 10, 2017 at 7:16 AM, Darren Duncan 
>> wrote:
>>
>>> I agree with everything Dan said here.  Its what I proposed, in fewer
>>> words.
>>> Do all new development under a new name, including all of Pali's work,
>>> and
>>> leave the current name for a product with no further effort applied to
>>> develop it. -- Darren Duncan
>>>
>>
>> This is NOT an option to me - it simply can't because the world moves
>> forward and because of bitrot. The 'old' version - the version that
>> works for most people, the current version of DBD::mysql, the one
>> which would then receive no more maintenance as it is no longer
>> compiles with the latest version of libmysqlclient and it does not
>> compile with libmariadb. This will only get worse in the future.
>>
>> I'll stick with my earlier proposal - I'll propose to go back to the
>> *current* latest DBD::mysql release which does not break backcompat
>> for our users; add the patches that we discarded when we rolled back
>> one by one, such as testing on many different lib/db options, memory
>> leaks and so on, and make a new release so we can be on the move
>> again.
>>
>> If possible I'd want to add back the *breaking* unicode changes that
>> were introduced but they should be either in a separate namespace OR
>> under a specific configuration option.
>> Currently this whole thing has cost us loosing MONTHS of progress and
>> then MONTHS of nothing and that is simply not good.
>>
>> Patrick: let me know if you're OK with this and then let's get start
>> again!
>>
>

Re: DBD::mysql path forward

[Night Light]

For the reason of "silence":
I've spoken to other users to hear that they have passively withdrawn from
this discussion as they are not motivated to be part of a release where
concerns about backwards compatibility are ignored. One of the users wrote
earlier (replace the word "sector" with "release"):

"When you're dealing with software that's purpose in life is to not corrupt
data, and have data there tomorrow, you go out of your way not to break
that promise. There's no point in being involved in this sector if you
don't care to deliver on that promise."

Re-releasing the 4.042 changes will break the contract of a long-standing
interface and corrupt all BLOB data when using "do()". These changes do
therefore more harm than good.
Putting these utf8 changes in the freezer until a suggestion is made that
will add the functionality instead of replacing it is not a sin. The PG
driver has for instance also a similar issue open without plans to fix it
anytime soon.
https://rt.cpan.org/Public/Bug/Display.html?id=122991

What is your objection against using the current 4.043 branch and work on
outstanding fixes, do a pre-release, a period of time for people to
test/try out, then release?

On Mon, Nov 6, 2017 at 11:32 PM, Patrick Galbraith  wrote:

> Pali,
>
> I totally understand - timing timing - I started a new job and found out I
> had to prepare to talk about MySQL/Kubernetes at kubecon, so I apologize,
> and didn't plan it that way. Let me re-review those emails and give a
> decision, of course, your decision as well.
>
> As far as maintainer, you are an equal, and if you need to be seen as a
> decision maker and we work by consensus with you having equal decision. I'm
> glad with that as well because I do appreciate all of your contributions.
> I'm sorry for what you mentioned in the past though it was more of a
> reaction to deal with "customers" having issues and working panic mode, not
> a slight to your efforts. What would you prefer?
>
> So, what would be ideal for us to move forward is to get the utf8 fixes
> into the master branch, then work in the subsequent fixes that came after
> June, do a pre-release, a period of time for people to test/try out, then
> release. What sort of git gymnastics does this require?  What do you think?
>
> I think the key here to have a single driver that supports both Maria and
> MySQL both and not have splintering, and more than anything, provide
> stability as well as new features for users.
>
> Regards,
>
> Patrick
>
>
>
> On 11/05/2017 04:24 AM, p...@cpan.org wrote:
>
> Hi!
>
> Have you got replies from people who you informed/asked about that
> problems?
>
> What is needed from you is to decide how to handle problems with
> encoding and other bugs reported for DBD-mysql project [1].
>
> I sent in previous email some proposed solutions and I'm just waiting
> which would be chosen... I just do not like "silence" and no step
> forward. Just to note that Dan Book asked month ago for next steps.
>
> Also, I'm not very happy to see being co-maintainer of DBD::mysql,
> specially in this time when nothing happened for 5 months, plus some
> people opposed to me and wanted to revert bug fixes and contributions by
> me...
>
> [1] - https://rt.cpan.org/Public/Dist/Display.html?Name=DBD-mysql
>
> On Friday 03 November 2017 16:17:09 Patrick M. Galbraith wrote:
>
> Pali,
>
> Sorry, I wanted to give it time for people, plus have been very busy.
> I'm ready, just let me know when we can get code merged back in--
> what do you need me to do? I will need help with this.
>
> Regards,
>
> Patrick
>
> On 11/2/17 1:43 PM, p...@cpan.org wrote:
>
> On Tuesday 12 September 2017 11:40:31 Patrick M. Galbraith wrote:
>
> Hi all,
>
> After talking to Pali and looking at how other drivers have
> handled the issue, the best way forward will be to deal with
> solving the UTF-8 issue correctly as was attempted in May. This
> will be a problem for some, but only due to having to been
> accustomed to a work-around that relies on a intermittent and
> buggy implementation. The plan is to give ample time to let
> people prepare and comment as well as give them a chance to try
> the changes and adjust prior to stable release. I'm going to look
> through the mailing list and find particularly those who had
> problems back in May and June when we tried this before and work
> with them in advance of the change.
>
> DBD::mysql should be on par with all the other DBD drivers and
> allow transparent usage regardless of backing RDBMS, and this
> really is the only way. It will also allow for other fixes to
> proceed and the driver to continue improving and supporting all
> versions and enhancements of MySQL and MariaDB.
>
> Please do give us your thoughts on this as we want to be as
> helpful and transparent as possible.
>
> Thank you!
>
> Patrick
>
> Hi! Nearly another two months passed and there is no progress in
> unblocking development.
>
> Currently more people started reporting problems 

Re: DBD::mysql path forward

[Night Light]

The corruption is caused by the fact that the data type of a BLOB can not
be recognized and is therefore UTF8 encoded like a string before being sent
to MySQL.
I'd like to propose the introduction of an attribute to ensure correct UTF8
encoding while maintaining backwards compatibility.
Manual page text:


mysql_encode_utf8_input
Prior 4.044 versions did not UTF8 encode the SQL statement and its input
values before being sent to MySQL but did decode the returned output. This
caused inconsistency in certain occasions.
Setting this attribute will correct this but doing so makes it necessary to
specify the data type of BLOB parameters to prevent these from being
encoded as well.
The default value is 0 to ensure backwards compatibility. Setting of this
flag is recommended for correct UTF8 handling and for easier migrations to
other database backends (DBD's).

Example with mysql_encode_utf8_input set to 0:
$dbh = DBI->connect($dsn,$user,$password, {'RaiseError' => 1,
mysql_enable_utf8mb4 => 1});
$dbh->do("INSERT INTO authors (id,blob_column,first_name) VALUES (?,?,?)",
undef, 2,$content,"Jochen");

Example with mysql_encode_utf8_input set to 1 (recommended):
$dbh = DBI->connect($dsn,$user,$password, {'RaiseError' => 1,
mysql_enable_utf8mb4 => 1 ,mysql_encode_utf8_input => 1});
$sth = $dbh->prepare("INSERT INTO authors (id,blob_column,first_name)
VALUES (?,?,?)");
$sth->bind_param(1,123,SQL_BIGINT);
$sth->bind_param(2,$content,SQL_BLOB);
$sth->bind_param(3,$first_name,SQL_VARCHAR);
$sth->execute();


On Tue, Sep 26, 2017 at 11:55 AM, demerphq <demer...@gmail.com> wrote:

> On 19 September 2017 at 14:46, Night Light <nightligh...@gmail.com> wrote:
> > Dear Perl gurus,
> >
> > This is my first post. I'm using Perl with great joy, and I'd like to
> > express my gratitude for all you are doing to keep Perl stable and fun to
> > use.
> >
> > I'd like to ask to object to re-releasing this version and discuss on
> how to
> > make 4.043 backwards compatible instead.
> > This change will with 100% certainty corrupt all BLOB data written to the
> > database when the developer did not read the release notes before
> applying
> > the latest version of DBD::mysql (and changed its code consequently).
> > Knowing that sysadmins have the habit of not always reading the release
> > notes of each updated package the likelihood that this will happen will
> > therefore high.
> > I myself wasn't even shown the release notes as it was a dependency of an
> > updated package that I applied.
> > The exposure of this change is big as DBD::mysql affects multiple
> > applications and many user bases.
> > I believe deliberately introducing industry wide database corruption is
> > something that will significantly harm peoples confidence in using Perl.
> > I believe that not providing backwards compatibility is not in line with
> the
> > Perl policy that has been carefully put together by the community to
> > maintain the quality of Perl as it is today.
> > http://perldoc.perl.org/perlpolicy.html#BACKWARD-
> COMPATIBILITY-AND-DEPRECATION
> >
> > I therefore believe the only solution is an upgrade that is by default
> > backwards compatible, and where it is the user who decides when to start
> > UTF8 encode the input values of a SQL request instead.
> > If it is too time consuming or too difficult it should be considered to
> park
> > the UTF8-encoding "fix" and release a version with the security fix
> first.
> >
> > I have the following objections against this release:
> >
> > 1. the upgrade will corrupt more records than it fixes (it does more harm
> > than good)
> > 2. the reason given for not providing backward compatibility ("because it
> > was hard to implement") is not plausible given the level of unwanted side
> > effects.
> >This especially knowing that there is already a mechanism in place to
> > signal if its wants UTF8 encoding or not
> > (mysql_enable_utf8/mysql_enable_utf8mb4).
> > 3. it costs more resources to coordinate/discuss a "way forward" or
> options
> > than to implement a solution that addresses backwards compatibility
> > 4. it is unreasonable to ask for changing existing source knowing that
> > depending modules may not be actively maintained or proprietary
> >It can be argued that such module should always be maintained but it
> does
> > not change the fact that a good running Perl program becomes unusable
> > 5. it does not inform the user that after upgrading existing code will
> start
> > write corrupt BLOB records
> > 6. it does not inform the user about the fac

Re: DBD::mysql path forward

[Night Light]

Dear Perl gurus,

This is my first post. I'm using Perl with great joy, and I'd like to
express my gratitude for all you are doing to keep Perl stable and fun to
use.

I'd like to ask to object to re-releasing this version and discuss on how
to make 4.043 backwards compatible instead.
This change will with 100% certainty corrupt all BLOB data written to the
database when the developer did not read the release notes before applying
the latest version of DBD::mysql (and changed its code consequently).
Knowing that sysadmins have the habit of not always reading the release
notes of each updated package the likelihood that this will happen will
therefore high.
I myself wasn't even shown the release notes as it was a dependency of an
updated package that I applied.
The exposure of this change is big as DBD::mysql affects multiple
applications and many user bases.
I believe deliberately introducing industry wide database corruption is
something that will significantly harm peoples confidence in using Perl.
I believe that not providing backwards compatibility is not in line with
the Perl policy that has been carefully put together by the community to
maintain the quality of Perl as it is today.
http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION

I therefore believe the only solution is an upgrade that is by default
backwards compatible, and where it is the user who decides when to start
UTF8 encode the input values of a SQL request instead.
If it is too time consuming or too difficult it should be considered to
park the UTF8-encoding "fix" and release a version with the security fix
first.

I have the following objections against this release:

1. the upgrade will corrupt more records than it fixes (it does more harm
than good)
2. the reason given for not providing backward compatibility ("because it
was hard to implement") is not plausible given the level of unwanted side
effects.
   This especially knowing that there is already a mechanism in place to
signal if its wants UTF8 encoding or not
(mysql_enable_utf8/mysql_enable_utf8mb4).
3. it costs more resources to coordinate/discuss a "way forward" or options
than to implement a solution that addresses backwards compatibility
4. it is unreasonable to ask for changing existing source knowing that
depending modules may not be actively maintained or proprietary
   It can be argued that such module should always be maintained but it
does not change the fact that a good running Perl program becomes unusable
5. it does not inform the user that after upgrading existing code will
start write corrupt BLOB records
6. it does not inform the user about the fact that a code review of all
existing code is necessary, and how it needs to be changed and tested
7. it does not give the user the option to decide how the BLOB's should be
stored/encoded (opt in)
8. it does not provide backwards compatibility
   By doing so it does not respect the Perl policy that has been carefully
put together by the community to maintain the quality of Perl as it is
today.

http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
9. it blocks users from using DBD::mysql upgrades as long as they have not
rewritten their existing code
10. not all users from DBD::mysql can be warned beforehand about the side
effects as it is not known which private parties have code that use
DBD::mysql
12. I believe development will go faster when support for backwards
compatibility is addressed
13. having to write 1 extra line for each SQL query value is a monks job
that will make the module less attractive to use

About forking to DBD::mariadb?:
The primary reason to create such a module is when the communication
protocol of Mariadb has become incompatible with Mysql.
To use this namespace to fix a bug in DBD::mysql does not meet that
criteria and causes confusion for developers and unnecessary pollution of
the DBD namespace.

---

For people that do not know the impact of the change that is pending to be
committed:
(see Github issue that includes 3 reports of companies that suffered data
loss https://github.com/perl5-dbi/DBD-mysql/issues/117 )

Issue: some UTF8 characters are not properly displayed after retrieval
Cause: SQL query values are not UTF8 encoded when sent to the database but
they are all decoded once retrieved.
Occurence: Only records with string data that can only be written with
UTF8. It can be considered rare as people haven't reported this issue after
10 years of usage.
Regional impact: Only affects countries which characters need UTF8 encoding
and only affects string values.
Steps to recover from it: Read string data unencoded and write it encoded.

Changes of upgrade pending to be re-released:
SQL query values are both UTF8 encoded when sent to the database as when
its retrieved (including BLOB fields).
BLOB fields will be excluded from encoding only if you specify its data
type.

Side effects from installing upgrade:
- BLOB data will be written