Hello, may I ask how this situation differs from upgrading DBD::Pg to version 3.3.0 or DBD::SQLite to version 1.43_04? What you are writing there basically affects also DBD::SQLite... And version 1.43_04 was already released...
On Tuesday 19 September 2017 14:46:09 Night Light wrote: > Dear Perl gurus, > > This is my first post. I'm using Perl with great joy, and I'd like to > express my gratitude for all you are doing to keep Perl stable and > fun to use. > > I'd like to ask to object to re-releasing this version and discuss on > how to make 4.043 backwards compatible instead. > This change will with 100% certainty corrupt all BLOB data written to > the database when the developer did not read the release notes > before applying the latest version of DBD::mysql (and changed its > code consequently). Knowing that sysadmins have the habit of not > always reading the release notes of each updated package the > likelihood that this will happen will therefore high. > I myself wasn't even shown the release notes as it was a dependency > of an updated package that I applied. > The exposure of this change is big as DBD::mysql affects multiple > applications and many user bases. > I believe deliberately introducing industry wide database corruption > is something that will significantly harm peoples confidence in > using Perl. I believe that not providing backwards compatibility is > not in line with the Perl policy that has been carefully put > together by the community to maintain the quality of Perl as it is > today. > http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DE > PRECATION > > I therefore believe the only solution is an upgrade that is by > default backwards compatible, and where it is the user who decides > when to start UTF8 encode the input values of a SQL request instead. > If it is too time consuming or too difficult it should be considered > to park the UTF8-encoding "fix" and release a version with the > security fix first. > > I have the following objections against this release: > > 1. the upgrade will corrupt more records than it fixes (it does more > harm than good) > 2. the reason given for not providing backward compatibility > ("because it was hard to implement") is not plausible given the > level of unwanted side effects. > This especially knowing that there is already a mechanism in place > to signal if its wants UTF8 encoding or not > (mysql_enable_utf8/mysql_enable_utf8mb4). > 3. it costs more resources to coordinate/discuss a "way forward" or > options than to implement a solution that addresses backwards > compatibility 4. it is unreasonable to ask for changing existing > source knowing that depending modules may not be actively maintained > or proprietary It can be argued that such module should always be > maintained but it does not change the fact that a good running Perl > program becomes unusable 5. it does not inform the user that after > upgrading existing code will start write corrupt BLOB records > 6. it does not inform the user about the fact that a code review of > all existing code is necessary, and how it needs to be changed and > tested 7. it does not give the user the option to decide how the > BLOB's should be stored/encoded (opt in) > 8. it does not provide backwards compatibility > By doing so it does not respect the Perl policy that has been > carefully put together by the community to maintain the quality of > Perl as it is today. > > http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DE > PRECATION 9. it blocks users from using DBD::mysql upgrades as long > as they have not rewritten their existing code > 10. not all users from DBD::mysql can be warned beforehand about the > side effects as it is not known which private parties have code that > use DBD::mysql > 12. I believe development will go faster when support for backwards > compatibility is addressed > 13. having to write 1 extra line for each SQL query value is a monks > job that will make the module less attractive to use > > About forking to DBD::mariadb?: > The primary reason to create such a module is when the communication > protocol of Mariadb has become incompatible with Mysql. > To use this namespace to fix a bug in DBD::mysql does not meet that > criteria and causes confusion for developers and unnecessary > pollution of the DBD namespace. > > --- > > For people that do not know the impact of the change that is pending > to be committed: > (see Github issue that includes 3 reports of companies that suffered > data loss https://github.com/perl5-dbi/DBD-mysql/issues/117 ) > > Issue: some UTF8 characters are not properly displayed after > retrieval Cause: SQL query values are not UTF8 encoded when sent to > the database but they are all decoded once retrieved. > Occurence: Only records with string data that can only be written > with UTF8. It can be considered rare as people haven't reported this > issue after 10 years of usage. > Regional impact: Only affects countries which characters need UTF8 > encoding and only affects string values. > Steps to recover from it: Read string data unencoded and write it > encoded. > > Changes of upgrade pending to be re-released: > SQL query values are both UTF8 encoded when sent to the database as > when its retrieved (including BLOB fields). > BLOB fields will be excluded from encoding only if you specify its > data type. > > Side effects from installing upgrade: > - BLOB data will be written after UTF8 encoding and will therefore be > corrupt > - no possibility to detect if a BLOB field is corrupt or not. Only > when known when the INSERT/UPDATE took place, and when the upgrade > was installed - existing data will still display incorrect > > Occurence: every INSERT/UPDATE statement will start writing corrupted > BLOB data > Regional impact: worldwide > Steps to recover from it corrupted BLOBs? You cannot. Your binary > blobs are encoded as if they were UTF8 strings. Your binary data is > unrecoverable (as in "gone forever"). > If you are a dentist you have to ask your customers to come back to > make another x-ray as the made photo's are gone. > > What is asked from the developer to prevent this from happening? > - do not miss reading the release notes before upgrading > - review all source code (including written by other included > modules) and specify the data type of each SQL parameter value > before: $dbh->do('INSERT INTO test (BLOB1,BLOB2,BLOB3,BLOB4) > VALUES(?,?,?,?)',undef,$col1,$col2,$col3); > after: $dbh->do('INSERT INTO test (BLOB1,BLOB2,BLOB3,BLOB4) > VALUES(?,?,?,?)'); > $sth->bind_param(1, $file, SQL_BLOB); > $sth->bind_param(2, $file, SQL_BLOB); > $sth->bind_param(3, $file, SQL_BLOB); > ... > One line more for each SQL statement. This will be a time consuming > monks task during which the user will ask why this is necessary > while it worked before. > - upgrade scripts need to be written to UTF8 encode existing string > data - retest all source code
