Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Friday 31 July 2015 10:34:50 Knoll Lars wrote: On 31/07/15 10:15, Thiago Macieira thiago.macie...@intel.com wrote: [snip] I will try to submit a patch tomorrow. Thanks Thiago! Done. Change: https://codereview.qt-project.org/122598 Doc update: https://codereview.qt-project.org/121992 -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Friday 31 July 2015 10:34:50 Knoll Lars wrote: On 31/07/15 10:15, Thiago Macieira thiago.macie...@intel.com wrote: [snip] I will try to submit a patch tomorrow. Thanks Thiago! Without changing QtTest. That is, I'd like to keep the QString escaping in QtTest exactly as it is today. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On 28/07/15 21:24, Thiago Macieira thiago.macie...@intel.com wrote: On Tuesday 28 July 2015 21:58:08 NIkolai Marchenko wrote: Fact is : you are severly overestimating the amount of cases where homoglyph is a problem at the same time severly underestimating the amount of code you broke and inconvenience caused by the change. The amount of people that has since agreed with my point of view in this thread is a testament to that. Want homoglyph debugging? turn on environment variable. Simple as that. At the same time peopel with existing code won't be tearing at their hair on the prospect of retyping every qdebug instance I don't think I am underestimating the annoyance to those people nor that I am overestimating the benefit. You have also take into account that a great majority of source code is written in English and for those cases this feature is a net win. Anyway, I'm waiting for Lars to make a decision. Lars, you can either make your choices below or you may request consensus and then make your decision: 1) qDebug for QStrings should a) escape everything non-US-ASCII (5.5.0 behaviour) b) escape everything that isn't QChar::isPrint c) escape only US-ASCII control characters (0 to 31), backslash and quote d) escape only NUL, backslash and quote e) escape nothing (5.0 through 5.4 behaviour) [I haven't included option f) mangle irrevocably (4.x behaviour)] I think (b) is the most sane option (including escaping of backslash and quote). If you’re worried about isPrint performance, feel free to optimise the ascii (or latin1) case. In practice, I don’t think this will matter currently, as qDebug is using QTextStream behind the scenes anyway. 2) ditto for qDebug for QByteArrays, except that for: b1) escape everything that isn't isprint() from ctype.h (locale dependent) b2) escape everything that isn't printable under Latin 1 Note that QByteArray::toUpper and toLower do not and have never used ctype.h and are instead fixed to Latin 1. QByteArray should probably use (a), as the encoding isn’t clear (we’re unfortunately not really consistent in our usage neither. Constructing a QString from a QByteArray assumes utf8, toUpper/toLower as you say assumes latin1). 3) QtTest should a) follow qDebug and be consistent b) do it differently, in which case we need to select which of the options Above Are there arguments against being consistent? Apart from us not wanting to change it in 5.5.1? 4) when to apply those changes, which are technically behaviour changes a) 5.5.1 b) 5.6.0 5.5.1 has my vote. Yes, we’re changing behaviour once again, but this brings it a lot closer to 5.4, while avoiding most of the issues that initially triggered the change. Note that the QtTest QString change happened in 5.4.0, so I advise against changing QTest::toString(QString) in 5.5.1, if a change is to happen. Agree. Obviously my preference has already been stated. If we eliminate 1a, my preferences are: 1c and 2a. I do not want QChar::isPrint to be used because it's expensive. As said above, I don’t think it matters, especially if we inline the ascii (and maybe latin1) case. Anything except 1e will still change C:\Users to C:\\Users in the output, which was one of the complaints listed. Yes, but this is unavoidable. We need to escape non printable characters and quote, and this implies we also need to escape the backslash. My goal is to keep the string easily readable in all languages, which is what IMO is needed to make debugging and logging easy. Cheers, Lars ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
Op 31-7-2015 om 09:20 schreef Knoll Lars: Anything except 1e will still change C:\Users to C:\\Users in the output, which was one of the complaints listed. Yes, but this is unavoidable. We need to escape non printable characters and quote, and this implies we also need to escape the backslash. My goal is to keep the string easily readable in all languages, which is what IMO is needed to make debugging and logging easy. Does it? The output is not aimed at being C++ again. So, why is is needed that the backslash is always escaped? Would it be possible to only escape \ if it is followed by 0x in the actual string, and then make escaped characters print as \0xAB01 or something like that (and just leave the other occurences alone?) Or only escape them if there is something else to escape? Or even choose a different escape character completely, one that is not used as often? Paths end up in debug statements quite often (at least, in the code I work on) and I do find it confusing to have them escaped. André ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Friday 31 July 2015 09:29:36 André Somers wrote: Does it? The output is not aimed at being C++ again. So, why is is needed that the backslash is always escaped? Would it be possible to only escape \ if it is followed by 0x in the actual string, and then make escaped characters print as \0xAB01 or something like that (and just leave the other occurences alone?) Or only escape them if there is something else to escape? So you're proposing that qDebug should output: C:\\Users C:\\users C:\\01.org C:\\newdir C:\Windows Sorry, no, that's a total veto from me. Or even choose a different escape character completely, one that is not used as often? I don't think that's a good idea. What character can we use that is among the 94 from US-ASCII that aren't control characters already or the space? It's only going to punt the problem elsewhere and cause more confusion because such character is not used as often to escape things. It has to be a non-control character because this thread has shown that qDebug output is read by people. A control character that isn't visible can't be seen by people. The only alternative is to not escape the backslash and the quote at all, in which case these two completely different strings would be printed the same way: QString(C:\\u00a0); QString::fromUtf8(C:\\u\xc3\xa0); as C:\u00a0 -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Friday 31 July 2015 07:20:00 Knoll Lars wrote: 1) qDebug for QStrings should a) escape everything non-US-ASCII (5.5.0 behaviour) b) escape everything that isn't QChar::isPrint c) escape only US-ASCII control characters (0 to 31), backslash and quote d) escape only NUL, backslash and quote e) escape nothing (5.0 through 5.4 behaviour) [I haven't included option f) mangle irrevocably (4.x behaviour)] I think (b) is the most sane option (including escaping of backslash and quote). If you’re worried about isPrint performance, feel free to optimise the ascii (or latin1) case. In practice, I don’t think this will matter currently, as qDebug is using QTextStream behind the scenes anyway. No optimisation because, as this thread has shown, non-ASCII text is common. Therefore, optimising for a particular case is not necessary and might just hurt the whole scenario. Besides, it's *debugging*, which implies slow code anyway. I'd rather make this simpler. 2) ditto for qDebug for QByteArrays, except that for: b1) escape everything that isn't isprint() from ctype.h (locale dependent) b2) escape everything that isn't printable under Latin 1 Note that QByteArray::toUpper and toLower do not and have never used ctype.h and are instead fixed to Latin 1. QByteArray should probably use (a), as the encoding isn’t clear (we’re unfortunately not really consistent in our usage neither. Constructing a QString from a QByteArray assumes utf8, toUpper/toLower as you say assumes latin1). Ok, so no change for QByteArray. 3) QtTest should a) follow qDebug and be consistent b) do it differently, in which case we need to select which of the options Above Are there arguments against being consistent? Apart from us not wanting to change it in 5.5.1? Same as before: we're trying to figure out why two strings are not the same. I'd rather *see* the evidence than try to guess. 4) when to apply those changes, which are technically behaviour changes a) 5.5.1 b) 5.6.0 5.5.1 has my vote. Yes, we’re changing behaviour once again, but this brings it a lot closer to 5.4, while avoiding most of the issues that initially triggered the change. Ok. Anything except 1e will still change C:\Users to C:\\Users in the output, which was one of the complaints listed. Yes, but this is unavoidable. We need to escape non printable characters and quote, and this implies we also need to escape the backslash. My goal is to keep the string easily readable in all languages, which is what IMO is needed to make debugging and logging easy. Ok. I will try to submit a patch tomorrow. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
Agree with Thiago. Backslash is a good escape character, as it’s used in pretty much all programming languages for that. If you want unescaped paths, you can use noquote, or print the paths using slashes (using QDir::fromNativeSeparators()). Cheers, Lars On 31/07/15 10:23, Thiago Macieira thiago.macie...@intel.com wrote: On Friday 31 July 2015 09:29:36 André Somers wrote: Does it? The output is not aimed at being C++ again. So, why is is needed that the backslash is always escaped? Would it be possible to only escape \ if it is followed by 0x in the actual string, and then make escaped characters print as \0xAB01 or something like that (and just leave the other occurences alone?) Or only escape them if there is something else to escape? So you're proposing that qDebug should output: C:\\Users C:\\users C:\\01.org C:\\newdir C:\Windows Sorry, no, that's a total veto from me. Or even choose a different escape character completely, one that is not used as often? I don't think that's a good idea. What character can we use that is among the 94 from US-ASCII that aren't control characters already or the space? It's only going to punt the problem elsewhere and cause more confusion because such character is not used as often to escape things. It has to be a non-control character because this thread has shown that qDebug output is read by people. A control character that isn't visible can't be seen by people. The only alternative is to not escape the backslash and the quote at all, in which case these two completely different strings would be printed the same way: QString(C:\\u00a0); QString::fromUtf8(C:\\u\xc3\xa0); as C:\u00a0 -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On 31/07/15 10:15, Thiago Macieira thiago.macie...@intel.com wrote: [snip] I will try to submit a patch tomorrow. Thanks Thiago! Cheers, Lars ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Tuesday 28. July 2015 12:24:16 Thiago Macieira wrote: I don't think I am underestimating the annoyance to those people nor that I am overestimating the benefit. You have also take into account that a great majority of source code is written in English and for those cases this feature is a net win. Anyway, I'm waiting for Lars to make a decision. Lars, you can either make your choices below or you may request consensus and then make your decision: I know this is not a vote. But here is my opinion: 1) qDebug for QStrings should a) escape everything non-US-ASCII (5.5.0 behaviour) b) escape everything that isn't QChar::isPrint c) escape only US-ASCII control characters (0 to 31), backslash and quote d) escape only NUL, backslash and quote e) escape nothing (5.0 through 5.4 behaviour) [I haven't included option f) mangle irrevocably (4.x behaviour)] I'd say b. (or c if you think b is too slow, but maybe you can optimize and not call isPrint for ascii) 2) ditto for qDebug for QByteArrays, except that for: b1) escape everything that isn't isprint() from ctype.h (locale dependent) b2) escape everything that isn't printable under Latin 1 Note that QByteArray::toUpper and toLower do not and have never used ctype.h and are instead fixed to Latin 1. For QByteArray I think a. is ok (escape everything non ascii) We don't know if the encoding is utf-8 or latin-1 so we can just escape everything. 3) QtTest should a) follow qDebug and be consistent b) do it differently, in which case we need to select which of the options above QtTest printer do not have to be consistant. But both behaviour would be acceptable as far as I am concerned. 4) when to apply those changes, which are technically behaviour changes a) 5.5.1 b) 5.6.0 5.5.1 for qDebug because it is a regression. Note that the QtTest QString change happened in 5.4.0, so I advise against changing QTest::toString(QString) in 5.5.1, if a change is to happen. I agree. Obviously my preference has already been stated. If we eliminate 1a, my preferences are: 1c and 2a. I do not want QChar::isPrint to be used because it's expensive. Anything except 1e will still change C:\Users to C:\\Users in the output, which was one of the complaints listed. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Monday 27. July 2015 10:03:25 Thiago Macieira wrote: The whole thinking is that the use of operator for QString implies you're trying to figure out why that string is the way it is, as opposed to trying to convey a message. I think that's where the disagreement is. I would think the use of operator for QString is just trying to figure out WHAT is that string. Most of the code looks like qDebug() There was an error processing XYZ: job-errorString(); qDebug() Error parsing file: fileName; qDebug() User entered: searchLineEdit-text(); Imagine that in a app written in russian for russian, or in other languages. You really want to know what is the error message or the file name. I use qDebug a lot, and I don't recall a single time I had problems with homoglyph. However I already was annoyed by the Qt 5.5's escaping. So if you ask my opinion, I think this regression should be fixed in Qt 5.5.x. -- Olivier Woboq - Qt services and support - http://woboq.com - http://code.woboq.org ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Tuesday 28. July 2015 11:47:09 Tasuku Suzuki wrote: Hi Thiago, 2015-07-28 1:34 GMT+09:00 Thiago Macieira thiago.macie...@intel.com: On Monday 27 July 2015 19:27:44 NIkolai Marchenko wrote: Or just use qPrintable() around your strings. you do realize, that it will require editing thousands of instances of qDebug in the old code? Yes. grepped QDebug\s*?operator\s*\\(QDebug under qtbase/src, and checked if QDebug operator(QDebug, const QString) is used in it for strings that can be utf8 very roughly. Do we fix those functions? QtCore: QDebug operator(QDebug debug, const QDir dir) QDebug operator(QDebug, const QJsonValue ) QDebug operator(QDebug, const QVariant ) QtSql: QDebug operator(QDebug dbg, const QSqlDatabase d) QDebug operator(QDebug dbg, const QSqlError s) QDebug operator(QDebug dbg, const QSqlField f) QDebug operator(QDebug dbg, const QSqlRecord r) QtGui: QDebug operator(QDebug d, const QAccessibleInterface *iface) QDebug operator(QDebug stream, const QFont font) QDebug operator(QDebug dbg, const QPageSize pageSize) QtDBus: QDebug operator(QDebug, const QDBusError ) Qt itself uses QDebug operator(QDebug, const QString) to get warning messages useful. e.g. http://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/io/qiodevice.cpp#n83 Do we apply qPrintable() everywhere in Qt? Will it be possible to apply qPrintable() to QString in containers? This is another very good point is that Qt itself uses operator with filenames or potentially translated error messages a lot. More than 4000 usages to check: http://code.woboq.org/data/symbol.html?root=../qt5ref=_ZN6QDebuglsERK7QString Not to mention that qPrintable would only make the problem worse since it converts the string to latin1 then interpret it as utf8! qUtf8Printable should be used instead. -- Olivier Woboq - Qt services and support - http://woboq.com - http://code.woboq.org ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On 28.07.2015 09:28, Olivier Goffart wrote: On Monday 27. July 2015 10:03:25 Thiago Macieira wrote: The whole thinking is that the use of operator for QString implies you're trying to figure out why that string is the way it is, as opposed to trying to convey a message. I think that's where the disagreement is. I would think the use of operator for QString is just trying to figure out WHAT is that string. [...] Imagine that in a app written in russian for russian, or in other languages. You really want to know what is the error message or the file name. This is exactly how I (and most of our customers) use qDebug() on a daily basis. I appreciate having the possibility to get an unambiguous output for those tricky test cases, but please make that an opt-in. So if you ask my opinion, I think this regression should be fixed in Qt 5.5.x. +1 cu Robert -- Robert Griebl SENIOR SOFTWARE ENGINEER Pelagicore AG Balanstr. 55, 81541 Munich, Germany robert.gri...@pelagicore.com www.pelagicore.com ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Tuesday 28 July 2015 09:28:08 Olivier Goffart wrote: qDebug() There was an error processing XYZ: job-errorString(); qDebug() Error parsing file: fileName; qDebug() User entered: searchLineEdit-text(); Imagine that in a app written in russian for russian, or in other languages. You really want to know what is the error message or the file name. At the same time, the error might be in the file name and you'd want to inspect why the file name is failing. A very concrete case is the NFC / NFD mess of filenames on OS X. NFC and NFD *look* identitcal on purpose, but have different binary representations. Qt API converts from NFD to NFC, but suppose you got the NFD form from elsewhere and then this job class tried to do a compare to a Qt- formatted filename. This is not so far-fetched. You'll spend at least half an hour trying to figure out why /tmp/Résumé.txt doesn't behave as you expect it to. So damned if you do and damned if you don't. I opt for leaving the default as the option that gives the most information, unambiguously, with no apparent or real data loss. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
Fact is : you are severly overestimating the amount of cases where homoglyph is a problem at the same time severly underestimating the amount of code you broke and inconvenience caused by the change. The amount of people that has since agreed with my point of view in this thread is a testament to that. Want homoglyph debugging? turn on environment variable. Simple as that. At the same time peopel with existing code won't be tearing at their hair on the prospect of retyping every qdebug instance On Tue, Jul 28, 2015 at 7:25 PM, Thiago Macieira thiago.macie...@intel.com wrote: On Tuesday 28 July 2015 09:28:08 Olivier Goffart wrote: qDebug() There was an error processing XYZ: job-errorString(); qDebug() Error parsing file: fileName; qDebug() User entered: searchLineEdit-text(); Imagine that in a app written in russian for russian, or in other languages. You really want to know what is the error message or the file name. At the same time, the error might be in the file name and you'd want to inspect why the file name is failing. A very concrete case is the NFC / NFD mess of filenames on OS X. NFC and NFD *look* identitcal on purpose, but have different binary representations. Qt API converts from NFD to NFC, but suppose you got the NFD form from elsewhere and then this job class tried to do a compare to a Qt- formatted filename. This is not so far-fetched. You'll spend at least half an hour trying to figure out why /tmp/Résumé.txt doesn't behave as you expect it to. So damned if you do and damned if you don't. I opt for leaving the default as the option that gives the most information, unambiguously, with no apparent or real data loss. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Tuesday 28 July 2015 21:58:08 NIkolai Marchenko wrote: Fact is : you are severly overestimating the amount of cases where homoglyph is a problem at the same time severly underestimating the amount of code you broke and inconvenience caused by the change. The amount of people that has since agreed with my point of view in this thread is a testament to that. Want homoglyph debugging? turn on environment variable. Simple as that. At the same time peopel with existing code won't be tearing at their hair on the prospect of retyping every qdebug instance I don't think I am underestimating the annoyance to those people nor that I am overestimating the benefit. You have also take into account that a great majority of source code is written in English and for those cases this feature is a net win. Anyway, I'm waiting for Lars to make a decision. Lars, you can either make your choices below or you may request consensus and then make your decision: 1) qDebug for QStrings should a) escape everything non-US-ASCII (5.5.0 behaviour) b) escape everything that isn't QChar::isPrint c) escape only US-ASCII control characters (0 to 31), backslash and quote d) escape only NUL, backslash and quote e) escape nothing (5.0 through 5.4 behaviour) [I haven't included option f) mangle irrevocably (4.x behaviour)] 2) ditto for qDebug for QByteArrays, except that for: b1) escape everything that isn't isprint() from ctype.h (locale dependent) b2) escape everything that isn't printable under Latin 1 Note that QByteArray::toUpper and toLower do not and have never used ctype.h and are instead fixed to Latin 1. 3) QtTest should a) follow qDebug and be consistent b) do it differently, in which case we need to select which of the options above 4) when to apply those changes, which are technically behaviour changes a) 5.5.1 b) 5.6.0 Note that the QtTest QString change happened in 5.4.0, so I advise against changing QTest::toString(QString) in 5.5.1, if a change is to happen. Obviously my preference has already been stated. If we eliminate 1a, my preferences are: 1c and 2a. I do not want QChar::isPrint to be used because it's expensive. Anything except 1e will still change C:\Users to C:\\Users in the output, which was one of the complaints listed. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Monday 27 July 2015 08:44:30 Knoll Lars wrote: I can understand the issues that non latin speakers are facing with this well. I’ve often used qDebug() to debug non latin use cases, and in 90% of the cases I just want to know what the string reads. The unicode content of it is interesting to me in only 10% of the cases. For most users not debugging Qt’s internals I would assume that ratio to be more like 99 to 1. As I said in my email, the problem is that you will not turn this on for those 10% of the cases until you've already lost data, either visibly or really. I'm counting ambiguity and homographs as apparent data loss because it will send you down the wrong debugging path. Thiago, which backends can’t handle utf8 or utf16 output these days? Android log, slog2, syslog and the regular stderr output if the system locale isn't UTF-8. In particular, on Windows the system locale is never UTF-8 unless you're using Vietnamese Windows. On the other hand, the OS X system locale is always UTF-8 unless you've messed up, including with setCodecForLocale. Or, if I list those that always do: WIndows OutputDebugString and journald. None of those are binary-safe, which means they will not work with a QString containing a NUL character. One idea could be a slightly modified approach that is more compatible: * We always encode QStrings as utf8 (maybe utf16 for the windows console) when sending them to qDebug() That would change behaviour for people with non-UTF-8 systems, as they'd now see mojibake. * We quote every character that is not printable (ie. !QChar::isPrint()) This is what I've done, except that I used ctype.h's isprint(), under the C locale. I considered using QChar::isPrint here, but it would be very expensive and it would not solve the homograph and ambiguity problem anyway. Better to just do nothing than use QChar::isPrint. * We add a flag that would give fully quoted strings so you can get a quoted version if required (e.g. qDebug() quoteStrings) I'm ok with a flag, as long as it's the default. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
Or just use qPrintable() around your strings. you do realize, that it will require editing thousands of instances of qDebug in the old code? On Mon, Jul 27, 2015 at 7:26 PM, Thiago Macieira thiago.macie...@intel.com wrote: On Monday 27 July 2015 19:20:34 NIkolai Marchenko wrote: b) getting the old behaviour back requires using noquote everywhere which breaks the code if you need to build it with earlier qt version Or just use qPrintable() around your strings. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Monday 27 July 2015 19:27:44 NIkolai Marchenko wrote: Or just use qPrintable() around your strings. you do realize, that it will require editing thousands of instances of qDebug in the old code? Yes. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
If we are to believe Thiago, every developer that needs to see trace information from an old code, now needs to edit all of his qDebug()'ed instances to use qPrintable. Yes, all of them. However much there is, manually, one by one. On Mon, Jul 27, 2015 at 7:34 PM, Thiago Macieira thiago.macie...@intel.com wrote: On Monday 27 July 2015 19:27:44 NIkolai Marchenko wrote: Or just use qPrintable() around your strings. you do realize, that it will require editing thousands of instances of qDebug in the old code? Yes. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Monday 27 July 2015 19:20:34 NIkolai Marchenko wrote: b) getting the old behaviour back requires using noquote everywhere which breaks the code if you need to build it with earlier qt version Or just use qPrintable() around your strings. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Monday 27 July 2015 16:54:49 Knoll Lars wrote: QChar::isPrint() is not that expensive, and this is about debug output. Your code escaping all these chars is a more expensive if it gets hit. I don't think so. The code is pretty efficient and it's just a simple stateful loop. QChar::isPrint requires an out-of-line function call for every character and then looking it up in our Unicode tables. That's a lot of data for the L1 cache: qstring.o's .rodata section is 188200 bytes. And I don't see how homographs are a problem for debug output. Unless you're debugging certain rather special cases in which case you should be able to turn full escaping on. I would guess that 98% of our users are using qDebug differently than you are. That is again what I am trying to avoid: having to know that you need this before you actually see the problem. In any case, if we want to escape characters 32, we need to escape the backslash too, which also was a source of complaints in the bug report, as C:\Users became C:\\Users Nikolai didn't include this in his mail to the list. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
you do realize, that it will require editing thousands of instances of qDebug in the old code? Yes. And here, ladies and gentlemen, is exactly the reason this discussion is even happening. This.Should. Not. Be. The. Case. This is just wrong on so many levels I cannot express it in words. On Mon, Jul 27, 2015 at 7:34 PM, Thiago Macieira thiago.macie...@intel.com wrote: On Monday 27 July 2015 19:27:44 NIkolai Marchenko wrote: Or just use qPrintable() around your strings. you do realize, that it will require editing thousands of instances of qDebug in the old code? Yes. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On 27/07/15 18:13, development-bounces+lars.knoll=theqtcompany@qt-project.org on behalf of Thiago Macieira development-bounces+lars.knoll=theqtcompany@qt-project.org on behalf of thiago.macie...@intel.com wrote: On Monday 27 July 2015 08:44:30 Knoll Lars wrote: I can understand the issues that non latin speakers are facing with this well. I’ve often used qDebug() to debug non latin use cases, and in 90% of the cases I just want to know what the string reads. The unicode content of it is interesting to me in only 10% of the cases. For most users not debugging Qt’s internals I would assume that ratio to be more like 99 to 1. As I said in my email, the problem is that you will not turn this on for those 10% of the cases until you've already lost data, either visibly or really. I'm counting ambiguity and homographs as apparent data loss because it will send you down the wrong debugging path. That's a wrong argument. This happens very rarely in real world use cases. And you don't loose data because of debug output. In most cases, you want to quickly see whether the string says bøh or bæh, and I loose a lot of time as a developer if I don't see that at a glance. Thiago, which backends can’t handle utf8 or utf16 output these days? Android log, slog2, syslog and the regular stderr output if the system locale isn't UTF-8. Well, Android systems are always utf8 afaik. And all modern linux systems are utf8 as well. In particular, on Windows the system locale is never UTF-8 unless you're using Vietnamese Windows. On the other hand, the OS X system locale is always UTF-8 unless you've messed up, including with setCodecForLocale. On Windows we use OutputDebugString by default, which takes utf16. So Mac and Windows are ok as well. That leaves QNX/slog2, and if the system charset there is not utf8, I'm happy if you escape all chars that it can't show on that platform (and in this case QNX should fix their OS). But we should penalize everybody because of one platform that very few people use. Or, if I list those that always do: WIndows OutputDebugString and journald. None of those are binary-safe, which means they will not work with a QString containing a NUL character. Yes, that's why I said that we should encode non printable characters. One idea could be a slightly modified approach that is more compatible: * We always encode QStrings as utf8 (maybe utf16 for the windows console) when sending them to qDebug() That would change behaviour for people with non-UTF-8 systems, as they'd now see mojibake. Well, I think there are no non utf8 systems left apart from Windows, where this isn't a problem (since we use OutputDebugString). * We quote every character that is not printable (ie. !QChar::isPrint()) This is what I've done, except that I used ctype.h's isprint(), under the C locale. I considered using QChar::isPrint here, but it would be very expensive and it would not solve the homograph and ambiguity problem anyway. Better to just do nothing than use QChar::isPrint. QChar::isPrint() is not that expensive, and this is about debug output. Your code escaping all these chars is a more expensive if it gets hit. And I don't see how homographs are a problem for debug output. Unless you're debugging certain rather special cases in which case you should be able to turn full escaping on. I would guess that 98% of our users are using qDebug differently than you are. * We add a flag that would give fully quoted strings so you can get a quoted version if required (e.g. qDebug() quoteStrings) I'm ok with a flag, as long as it's the default. Well, having to turn this on on every qDebug() line is a major issue for large existing code bases. You can say that this was never really supported by qDebug, but in practice, it worked on Windows, any modern Linux, Mac OS X (and I'd assume also iOS and Android). So people were using it. Suddenly breaking it is a major issue for anybody developing programs in non ascii (esp non latin) languages and trying to debug these. Just think of the use case of trying to figure out what a string contains at a certain point in your program. If the output is human readable it's immediately obvious. If I just get escaped data, I need to spend lots of time figuring it out. So IMO the default behaviour should be to show unicode strings in a form that is readable but at the same time doesn't hide things like zero width spaces (ie. It should escape non printable chars). Cheers, Lars ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On 27/07/15 18:52, development-bounces+lars.knoll=theqtcompany@qt-project.org on behalf of Thiago Macieira development-bounces+lars.knoll=theqtcompany@qt-project.org on behalf of thiago.macie...@intel.com wrote: On Monday 27 July 2015 19:49:15 NIkolai Marchenko wrote: If we are to believe Thiago, every developer that needs to see trace information from an old code, now needs to edit all of his qDebug()'ed instances to use qPrintable. Yes, all of them. However much there is, manually, one by one. You need to qualify this as every developer who wants to see the string printed without the quotes and escaping. This reduces your every developer to maybe 5% of them, probably less. I think you underestimate this. We have a huge amount of developers in non latin countries and for everybody doing development in these countries this is a very common problem. Cheers, Lars ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Monday 27 July 2015 16:57:23 Knoll Lars wrote: On 27/07/15 18:52, development-bounces+lars.knoll=theqtcompany.com@qt- project.org on behalf of Thiago Macieira development- bounces+lars.knoll=theqtcompany@qt-project.org on behalf of thiago.macie...@intel.com wrote: On Monday 27 July 2015 19:49:15 NIkolai Marchenko wrote: If we are to believe Thiago, every developer that needs to see trace information from an old code, now needs to edit all of his qDebug()'ed instances to use qPrintable. Yes, all of them. However much there is, manually, one by one. You need to qualify this as every developer who wants to see the string printed without the quotes and escaping. This reduces your every developer to maybe 5% of them, probably less. I think you underestimate this. We have a huge amount of developers in non latin countries and for everybody doing development in these countries this is a very common problem. I'm not thinking of people's languages. I'm thinking of the intent to read the string in the output, as opposed to inspecting it for why it contains the content it contains. The whole thinking is that the use of operator for QString implies you're trying to figure out why that string is the way it is, as opposed to trying to convey a message. If you wanted the latter case, you'd have used operator for const char*, which wasn't affected. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On Monday 27 July 2015 19:49:15 NIkolai Marchenko wrote: If we are to believe Thiago, every developer that needs to see trace information from an old code, now needs to edit all of his qDebug()'ed instances to use qPrintable. Yes, all of them. However much there is, manually, one by one. You need to qualify this as every developer who wants to see the string printed without the quotes and escaping. This reduces your every developer to maybe 5% of them, probably less. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
I'm ok with a flag, as long as it's the default. I am ok with a flag too. As long as escaping can be turned off, preferably in one place instead of editing qDebug()'s everywhere it is ok. I am so vehemently against the change as it is now primarily because it a) cannot be contained in a simple #ifdef block b) getting the old behaviour back requires using noquote everywhere which breaks the code if you need to build it with earlier qt version On Mon, Jul 27, 2015 at 7:13 PM, Thiago Macieira thiago.macie...@intel.com wrote: On Monday 27 July 2015 08:44:30 Knoll Lars wrote: I can understand the issues that non latin speakers are facing with this well. I’ve often used qDebug() to debug non latin use cases, and in 90% of the cases I just want to know what the string reads. The unicode content of it is interesting to me in only 10% of the cases. For most users not debugging Qt’s internals I would assume that ratio to be more like 99 to 1. As I said in my email, the problem is that you will not turn this on for those 10% of the cases until you've already lost data, either visibly or really. I'm counting ambiguity and homographs as apparent data loss because it will send you down the wrong debugging path. Thiago, which backends can’t handle utf8 or utf16 output these days? Android log, slog2, syslog and the regular stderr output if the system locale isn't UTF-8. In particular, on Windows the system locale is never UTF-8 unless you're using Vietnamese Windows. On the other hand, the OS X system locale is always UTF-8 unless you've messed up, including with setCodecForLocale. Or, if I list those that always do: WIndows OutputDebugString and journald. None of those are binary-safe, which means they will not work with a QString containing a NUL character. One idea could be a slightly modified approach that is more compatible: * We always encode QStrings as utf8 (maybe utf16 for the windows console) when sending them to qDebug() That would change behaviour for people with non-UTF-8 systems, as they'd now see mojibake. * We quote every character that is not printable (ie. !QChar::isPrint()) This is what I've done, except that I used ctype.h's isprint(), under the C locale. I considered using QChar::isPrint here, but it would be very expensive and it would not solve the homograph and ambiguity problem anyway. Better to just do nothing than use QChar::isPrint. * We add a flag that would give fully quoted strings so you can get a quoted version if required (e.g. qDebug() quoteStrings) I'm ok with a flag, as long as it's the default. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
Not to mention, this is cumbersome to the extreme. On Mon, Jul 27, 2015 at 7:26 PM, Thiago Macieira thiago.macie...@intel.com wrote: On Monday 27 July 2015 19:20:34 NIkolai Marchenko wrote: b) getting the old behaviour back requires using noquote everywhere which breaks the code if you need to build it with earlier qt version Or just use qPrintable() around your strings. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
Hi Thiago, 2015-07-28 1:34 GMT+09:00 Thiago Macieira thiago.macie...@intel.com: On Monday 27 July 2015 19:27:44 NIkolai Marchenko wrote: Or just use qPrintable() around your strings. you do realize, that it will require editing thousands of instances of qDebug in the old code? Yes. grepped QDebug\s*?operator\s*\\(QDebug under qtbase/src, and checked if QDebug operator(QDebug, const QString) is used in it for strings that can be utf8 very roughly. Do we fix those functions? QtCore: QDebug operator(QDebug debug, const QDir dir) QDebug operator(QDebug, const QJsonValue ) QDebug operator(QDebug, const QVariant ) QtSql: QDebug operator(QDebug dbg, const QSqlDatabase d) QDebug operator(QDebug dbg, const QSqlError s) QDebug operator(QDebug dbg, const QSqlField f) QDebug operator(QDebug dbg, const QSqlRecord r) QtGui: QDebug operator(QDebug d, const QAccessibleInterface *iface) QDebug operator(QDebug stream, const QFont font) QDebug operator(QDebug dbg, const QPageSize pageSize) QtDBus: QDebug operator(QDebug, const QDBusError ) Qt itself uses QDebug operator(QDebug, const QString) to get warning messages useful. e.g. http://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/io/qiodevice.cpp#n83 Do we apply qPrintable() everywhere in Qt? Will it be possible to apply qPrintable() to QString in containers? Best Regards, -- Tasuku Suzuki ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] FW: Backwards compatibiltiy break in Qt 5.5
On 27-7-2015 18:13, Thiago Macieira wrote: On Monday 27 July 2015 08:44:30 Knoll Lars wrote: I can understand the issues that non latin speakers are facing with this well. I’ve often used qDebug() to debug non latin use cases, and in 90% of the cases I just want to know what the string reads. The unicode content of it is interesting to me in only 10% of the cases. For most users not debugging Qt’s internals I would assume that ratio to be more like 99 to 1. As I said in my email, the problem is that you will not turn this on for those 10% of the cases until you've already lost data, either visibly or really. I'm counting ambiguity and homographs as apparent data loss because it will send you down the wrong debugging path. In that case, I think your solution _causes_ data loss when debugging path-related issues in Windows. If paths get their separators doubled all of a sudden, then that is certainly going to send people down the wrong debugging track. André ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
[Development] FW: Backwards compatibiltiy break in Qt 5.5
Hi, I can understand the issues that non latin speakers are facing with this well. I’ve often used qDebug() to debug non latin use cases, and in 90% of the cases I just want to know what the string reads. The unicode content of it is interesting to me in only 10% of the cases. For most users not debugging Qt’s internals I would assume that ratio to be more like 99 to 1. Thiago, which backends can’t handle utf8 or utf16 output these days? One idea could be a slightly modified approach that is more compatible: * We always encode QStrings as utf8 (maybe utf16 for the windows console) when sending them to qDebug() * We quote every character that is not printable (ie. !QChar::isPrint()) * We add a flag that would give fully quoted strings so you can get a quoted version if required (e.g. qDebug() quoteStrings) Could such an approach work in practice? Cheers, Lars On 27/07/15 00:25, NIkolai Marchenko enmarantis...@gmail.com wrote: (I am not entirely sure why I haven't received at least a notification of why this post has not appeared the first time I tried to send it, so here we go again. Btw, it was Thiago Macieira himself who actually suggested I mail this list) Hi! We (Russian Qt community) have a situation we could not resolve with Thiago Macieira in bug tracker, so have to ask for Chief Maintainer's attention. The bug in question is there: https://bugreports.qt.io/browse/QTBUG-47316 In version 5.5 Thiago Macieira introduced a compatibility break with older qt versions. This break has come in the form of completely changing the way qDebug() prints non-english characters to console. In essense - his fix to qDebug() will make it so that all non-english output from QString variable to qDebug() will be passed as a sequence of unicode escaped characters along the lines of: \u043F\u0440\u043E\u0432\u0435\u0440\u043A\u0430 043A\u0438\u0440\u0438\u043B\u043B\u0438\u0446\u044B Previously, in _some_ cases, but not reliably cross platform it would actually appear in console as readable output in user's language. I have to admit that Thiago is right about it being non-reliable contradicts qt's goal. Still... What Thiago completely disregards as unimportant - qDebug()'s been around for a long time. Even during times of qt4, users, through the use of setCodec functions, achieved readable output from qDebug in the console in their native language. Note that I say qt4. It's been around for almost a decade and there being no warning qDebug ever changing, it's become a standard on a lot of non-english installations, that actually allowed getting readable output, to use qDebug for tracing variables.. After all, opening debugger to inspect every little thing is very counterproductive. So here is where we are now, there are hundreds (probably thousands actually) codebases, where there is code like that: qDebug() someLabel; // skip qDebug() someVariableFromDatabase; // skip qDebug() someTextFromNetworkService; This is kinda normal and completely adhering to Kai Kohne's message he posted on youtube in qt-dev days video of using qDebug() to log program execution to console. Now, we arrive to Qt5.5. From here on now, instead of reading the trace of program execution everyone using the old code will be subjected to strings like \u043F\u0440\u043E\u0432\u0435\u0440\u043A\u0430 043A\u0438\u0440\u0438\u043B\u043B\u0438\u0446\u044B Note, that there are dozens to hundreds of such lines per application. And in SUCH a format, the trace posted above becomes completely useless for the purposes of debugging, because people, generaly, don't have unicode parser plugged into their brains. Thiago argues that his way, user will always get exactly which type of A the symbol is. Our point - now user will first need to pass the output of qDebug to unicode converter to even understand what's he looking at. Now, again, I do not dispute that qDebug in a way it was being used was correct or reliable across all platforms or allowed to check exactly which symbol was contained in a string. But BETTER way to fix that, in my opinion, and all of the Russian users who are already aware of the problem is to introduce a qDebug().escape() or something along those lines, in those RARE cases when instead of just tracing variables, user NEEDS to see unicode. (note I say Russian, because that is the community I come from, but in reality the situation is the same for greek, japanese, whatever non-english) Here we have a conflict of interest: Do we fix qDebug() in a way it is SUPPOSED to work, breaking code for users, or do we make a different flavour of qDebug that actually works as we intended it to, cross platform and escapes everything and let people who need it switch to that? Personally, from what I know of application development, breaking codebase is always bad. breaking a LARGE codebase, is horrible. So, I am at a loss of why Thiago decided to go the first way. We've tried to convince him he underestimates the
