Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
> -Original Message- > From: J-P Nurmi > Sent: Wednesday, 31 October 2018 12:41 PM > To: Mitch Curtis > Cc: development@qt-project.org > Subject: Re: [Development] Serialising UI state in QML via QSettings and > JSON: QByteArray vs QString > > On Wed, Oct 31, 2018 at 8:25 AM Mitch Curtis wrote: > > > Pass the state as a base64 string. Where does it turn to a byte > > > array if > > > Qt.btoa() returns a string? > > > > As of the previous patch set, saveState() returned a QByteArray, so the > user starts off with a byte array. If I somehow manage to make Qt.atob() and > Qt.btoa() accept and return a byte array (it wouldn't make sense to accept a > byte array and return a string), this prevents writing such state to JSON due > to its lack of support for byte arrays. > > > > The only way to serialise SplitView's state to JSON if we changed these > functions to use byte arrays would be to go through each value in the > QVariantMap that is exposed to QML for it to set the UI state (QJsonObject > can't be exposed to QML) and check if it's a byte array and then convert it to > a Base64 string. That seems very hacky for users. > > > > > > What do you think about the proposed solution of saveState() > > > > returning a > > > Base64 QString? > > > > > > -1 :) > > > > Can you explain why? It's a negligible difference in binary vs text for the > amount of data we're talking about. > > What you do inside saveState() and restoreState() is your business. > Whether you use JSON, CBOR, QDataStream, or something else, doesn't > really matter. But I don't see any compelling reason to deviate from the > "standard" QByteArray-based save/restoreState() pattern used in Qt > Widgets, including QSplitter. Don't let a silly bug in Qt.btoa() drive the API > design. If you want to store a binary blob in JSON, base64 is probably a good > option, but why enforce it for those who don't need it? For C++ users, > QByteArray provides the tools to encode and decode base64. Since base64 is > ASCII, you can safely convert to QString and store it in JSON. In QML, you are > supposed to be able to do the very same with Qt.atob(), Qt.btoa(), and > toString(). Looking closer into the code and tests, it seems that they are indeed completely broken, and somehow nobody ever noticed or complained: http://code.qt.io/cgit/qt/qtdeclarative.git/tree/src/qml/qml/v8/qqmlbuiltinfunctions.cpp?id=4a886753a75c7c4d66f1fa9cab5a6c5a03240df3#n992 http://code.qt.io/cgit/qt/qtdeclarative.git/tree/tests/auto/qml/qqmlqt/tst_qqmlqt.cpp?id=4a886753a75c7c4d66f1fa9cab5a6c5a03240df3#n937 http://code.qt.io/cgit/qt/qtdeclarative.git/tree/tests/auto/qml/qqmlqt/data/atob.qml?id=4a886753a75c7c4d66f1fa9cab5a6c5a03240df3 http://code.qt.io/cgit/qt/qtdeclarative.git/tree/tests/auto/qml/qqmlqt/data/btoa.qml?id=4a886753a75c7c4d66f1fa9cab5a6c5a03240df3 I'll look into fixing them and then go back to using QByteArray in the API. > -- > J-P Nurmi ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
On Wed, Oct 31, 2018 at 8:25 AM Mitch Curtis wrote: > > Pass the state as a base64 string. Where does it turn to a byte array if > > Qt.btoa() returns a string? > > As of the previous patch set, saveState() returned a QByteArray, so the user > starts off with a byte array. If I somehow manage to make Qt.atob() and > Qt.btoa() accept and return a byte array (it wouldn't make sense to accept a > byte array and return a string), this prevents writing such state to JSON due > to its lack of support for byte arrays. > > The only way to serialise SplitView's state to JSON if we changed these > functions to use byte arrays would be to go through each value in the > QVariantMap that is exposed to QML for it to set the UI state (QJsonObject > can't be exposed to QML) and check if it's a byte array and then convert it > to a Base64 string. That seems very hacky for users. > > > > What do you think about the proposed solution of saveState() returning a > > Base64 QString? > > > > -1 :) > > Can you explain why? It's a negligible difference in binary vs text for the > amount of data we're talking about. What you do inside saveState() and restoreState() is your business. Whether you use JSON, CBOR, QDataStream, or something else, doesn't really matter. But I don't see any compelling reason to deviate from the "standard" QByteArray-based save/restoreState() pattern used in Qt Widgets, including QSplitter. Don't let a silly bug in Qt.btoa() drive the API design. If you want to store a binary blob in JSON, base64 is probably a good option, but why enforce it for those who don't need it? For C++ users, QByteArray provides the tools to encode and decode base64. Since base64 is ASCII, you can safely convert to QString and store it in JSON. In QML, you are supposed to be able to do the very same with Qt.atob(), Qt.btoa(), and toString(). -- J-P Nurmi ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
> -Original Message- > From: J-P Nurmi > Sent: Tuesday, 30 October 2018 3:52 PM > To: Mitch Curtis > Cc: development@qt-project.org > Subject: Re: [Development] Serialising UI state in QML via QSettings and > JSON: QByteArray vs QString > > On Tue, Oct 30, 2018 at 3:04 PM Mitch Curtis wrote: > > Their documentation doesn't claim that they operate on a specific type, so > I'm not sure it's a matter of not doing what they claim to do. > > They do promise to handle binary data, though. Representing binary data > with null-terminated strings makes them useless. The sole purpose of base64 > is to encode binary data to text that can be represented as a string. > > > Whether or not they could be extended to work with byte arrays is a > question I don't have the answer to, though presumably it would involve > modifying QV4::Value and who knows what else in the guts of QML. > > The QML engine gained better QByteArray support recently (5.10?): > http://doc.qt.io/qt-5/qtqml-cppintegration-data.html#qbytearray-to- > javascript-arraybuffer > > > Unfortunately it wouldn't help the use case I'm trying to support though, > because there is still the problem of Qt's JSON implementation not > supporting byte arrays. > > Pass the state as a base64 string. Where does it turn to a byte array if > Qt.btoa() returns a string? As of the previous patch set, saveState() returned a QByteArray, so the user starts off with a byte array. If I somehow manage to make Qt.atob() and Qt.btoa() accept and return a byte array (it wouldn't make sense to accept a byte array and return a string), this prevents writing such state to JSON due to its lack of support for byte arrays. The only way to serialise SplitView's state to JSON if we changed these functions to use byte arrays would be to go through each value in the QVariantMap that is exposed to QML for it to set the UI state (QJsonObject can't be exposed to QML) and check if it's a byte array and then convert it to a Base64 string. That seems very hacky for users. > > What do you think about the proposed solution of saveState() returning a > Base64 QString? > > -1 :) Can you explain why? It's a negligible difference in binary vs text for the amount of data we're talking about. > -- > J-P Nurmi ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
On Tuesday, 30 October 2018 06:13:54 PDT Mitch Curtis wrote: > Since QCborValue also uses the Base64 string approach and it what is what I > was planning on doing if no other solution came up, I think I will just go > ahead and return a Base64 QString from QQuickSplitView::saveState() instead > of a QVariant containing a QByteArray. I will use the CBOR stuff to > serialise it though, as it's much nicer than using QDataStream. Sounds like a good plan! -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
On Tue, Oct 30, 2018 at 3:04 PM Mitch Curtis wrote: > Their documentation doesn't claim that they operate on a specific type, so > I'm not sure it's a matter of not doing what they claim to do. They do promise to handle binary data, though. Representing binary data with null-terminated strings makes them useless. The sole purpose of base64 is to encode binary data to text that can be represented as a string. > Whether or not they could be extended to work with byte arrays is a question > I don't have the answer to, though presumably it would involve modifying > QV4::Value and who knows what else in the guts of QML. The QML engine gained better QByteArray support recently (5.10?): http://doc.qt.io/qt-5/qtqml-cppintegration-data.html#qbytearray-to-javascript-arraybuffer > Unfortunately it wouldn't help the use case I'm trying to support though, > because there is still the problem of Qt's JSON implementation not supporting > byte arrays. Pass the state as a base64 string. Where does it turn to a byte array if Qt.btoa() returns a string? > What do you think about the proposed solution of saveState() returning a > Base64 QString? -1 :) -- J-P Nurmi ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
> -Original Message- > From: Development project.org> On Behalf Of Mitch Curtis > Sent: Tuesday, 30 October 2018 3:04 PM > To: J-P Nurmi > Cc: development@qt-project.org > Subject: Re: [Development] Serialising UI state in QML via QSettings and > JSON: QByteArray vs QString > > > -Original Message- > > From: J-P Nurmi > > Sent: Tuesday, 30 October 2018 2:49 PM > > To: Mitch Curtis > > Cc: development@qt-project.org > > Subject: Re: [Development] Serialising UI state in QML via QSettings > > and > > JSON: QByteArray vs QString > > > > On Mon, Oct 29, 2018 at 5:42 PM Mitch Curtis wrote: > > > I thought that I could get around this by using Qt.atob() and > > > Qt.btoa() in > > QML to convert the byte array into a Base64 QString, but it turns out > > that those functions expect a string: > > > > > > http://code.qt.io/cgit/qt/qtdeclarative.git/tree/src/qml/qml/v8/qqml > > > bu > > > > > > iltinfunctions.cpp?h=5.12=475c74a9926efcd968572563e678988e53804603# > > > n996 > > > > Hi Mitch, > > Hey. :) > > > So Qt.atob() and Qt.btoa() are not doing what they claim to do? Would > > it be feasible to fix them? > > Their documentation doesn't claim that they operate on a specific type, so > I'm not sure it's a matter of not doing what they claim to do. Correcting myself here; the docs say "string": "ASCII to binary - this function decodes the base64 encoded data string and returns it." - http://doc.qt.io/qt-5/qml-qtqml-qt.html#atob-method > Whether or not they could be extended to work with byte arrays is a > question I don't have the answer to, though presumably it would involve > modifying QV4::Value and who knows what else in the guts of QML. > > Unfortunately it wouldn't help the use case I'm trying to support though, > because there is still the problem of Qt's JSON implementation not > supporting byte arrays. > > What do you think about the proposed solution of saveState() returning a > Base64 QString? > > > -- > > J-P Nurmi > ___ > Development mailing list > Development@qt-project.org > http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
> -Original Message- > From: J-P Nurmi > Sent: Tuesday, 30 October 2018 2:49 PM > To: Mitch Curtis > Cc: development@qt-project.org > Subject: Re: [Development] Serialising UI state in QML via QSettings and > JSON: QByteArray vs QString > > On Mon, Oct 29, 2018 at 5:42 PM Mitch Curtis wrote: > > I thought that I could get around this by using Qt.atob() and Qt.btoa() in > QML to convert the byte array into a Base64 QString, but it turns out that > those functions expect a string: > > > > http://code.qt.io/cgit/qt/qtdeclarative.git/tree/src/qml/qml/v8/qqmlbu > > > iltinfunctions.cpp?h=5.12=475c74a9926efcd968572563e678988e53804603# > > n996 > > Hi Mitch, Hey. :) > So Qt.atob() and Qt.btoa() are not doing what they claim to do? Would it be > feasible to fix them? Their documentation doesn't claim that they operate on a specific type, so I'm not sure it's a matter of not doing what they claim to do. Whether or not they could be extended to work with byte arrays is a question I don't have the answer to, though presumably it would involve modifying QV4::Value and who knows what else in the guts of QML. Unfortunately it wouldn't help the use case I'm trying to support though, because there is still the problem of Qt's JSON implementation not supporting byte arrays. What do you think about the proposed solution of saveState() returning a Base64 QString? > -- > J-P Nurmi ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
On Mon, Oct 29, 2018 at 5:42 PM Mitch Curtis wrote: > I thought that I could get around this by using Qt.atob() and Qt.btoa() in > QML to convert the byte array into a Base64 QString, but it turns out that > those functions expect a string: > > http://code.qt.io/cgit/qt/qtdeclarative.git/tree/src/qml/qml/v8/qqmlbuiltinfunctions.cpp?h=5.12=475c74a9926efcd968572563e678988e53804603#n996 Hi Mitch, So Qt.atob() and Qt.btoa() are not doing what they claim to do? Would it be feasible to fix them? -- J-P Nurmi ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
> -Original Message- > From: Development project.org> On Behalf Of Mitch Curtis > Sent: Tuesday, 30 October 2018 12:30 PM > To: Edward Welbourne > Cc: Qt development mailing list > Subject: Re: [Development] Serialising UI state in QML via QSettings and > JSON: QByteArray vs QString > > > -Original Message- > > From: Edward Welbourne > > Sent: Tuesday, 30 October 2018 11:37 AM > > To: Mitch Curtis > > Cc: Qt development mailing list > > Subject: Re: [Development] Serialising UI state in QML via QSettings > > and > > JSON: QByteArray vs QString > > > > Edward Welbourne (Monday, 29 October 2018 7:31 PM) wrote > > >> I'm not sure where JSON got involved in that > > > > Mitch Curtis (30 October 2018 08:37) replied > > > What do you mean? It's in the title of the email; it's a popular > > > choice for storing data like this, so I want SplitView's > > > serialisation to be compatible with it. > > > > Perhaps I need to rephrase: to what extent is the choice of JSON, in > > the present discussion, necessary or incidental ? If what you're > > trying to do has to use JSON with QSettings, then it's necessary. If > > JSON just looked, at first sight, like it would solve your primary > > problem, then it's incidental and can be replaced by something compatible. > > > > Eddy. > > I want users to be able to call SplitView's saveState() function and store the > result in a JSON file. To do that, it needs to be able be stored in a QVariant > because QJsonObject isn't exposed to QML. The next issue for users would > be the lack of ability to convert that QVariant into a QJsonObject. > > By the way, QCborValue and friends is working quite well so far. :) I ran into an issue. An approach that uses QCborArray/Map/Value to serialise SplitView's state into a byte array still runs into the same problem when it comes to JSON files: https://doc-snapshots.qt.io/qt5-5.12/qcborvalue.html#toJsonValue This table says that byte arrays will be "converted to a lossless encoding like Base64url, but the distinction between strings and byte arrays is lost". So even if I were to document that QCborValue is used to save the state (which I don't really want to do), someone wanting to serialise their UI state to a JSON file would need to use QCbor* API to do so, and at that point the type information that identifies the data as a byte array is lost anyway. Since QCborValue also uses the Base64 string approach and it what is what I was planning on doing if no other solution came up, I think I will just go ahead and return a Base64 QString from QQuickSplitView::saveState() instead of a QVariant containing a QByteArray. I will use the CBOR stuff to serialise it though, as it's much nicer than using QDataStream. > ___ > Development mailing list > Development@qt-project.org > http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
> -Original Message- > From: Edward Welbourne > Sent: Tuesday, 30 October 2018 11:37 AM > To: Mitch Curtis > Cc: Qt development mailing list > Subject: Re: [Development] Serialising UI state in QML via QSettings and > JSON: QByteArray vs QString > > Edward Welbourne (Monday, 29 October 2018 7:31 PM) wrote > >> I'm not sure where JSON got involved in that > > Mitch Curtis (30 October 2018 08:37) replied > > What do you mean? It's in the title of the email; it's a popular > > choice for storing data like this, so I want SplitView's serialisation > > to be compatible with it. > > Perhaps I need to rephrase: to what extent is the choice of JSON, in the > present discussion, necessary or incidental ? If what you're trying to do has > to use JSON with QSettings, then it's necessary. If JSON just looked, at > first > sight, like it would solve your primary problem, then it's incidental and can > be > replaced by something compatible. > > Eddy. I want users to be able to call SplitView's saveState() function and store the result in a JSON file. To do that, it needs to be able be stored in a QVariant because QJsonObject isn't exposed to QML. The next issue for users would be the lack of ability to convert that QVariant into a QJsonObject. By the way, QCborValue and friends is working quite well so far. :) ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
Edward Welbourne (Monday, 29 October 2018 7:31 PM) wrote >> I'm not sure where JSON got involved in that Mitch Curtis (30 October 2018 08:37) replied > What do you mean? It's in the title of the email; it's a popular > choice for storing data like this, so I want SplitView's serialisation > to be compatible with it. Perhaps I need to rephrase: to what extent is the choice of JSON, in the present discussion, necessary or incidental ? If what you're trying to do has to use JSON with QSettings, then it's necessary. If JSON just looked, at first sight, like it would solve your primary problem, then it's incidental and can be replaced by something compatible. Eddy. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
> -Original Message- > From: Edward Welbourne > Sent: Monday, 29 October 2018 7:31 PM > To: Mitch Curtis > Cc: Qt development mailing list > Subject: Re: [Development] Serialising UI state in QML via QSettings and > JSON: QByteArray vs QString > > Mitch Curtis (29 October 2018 17:42) > > To solve #2, I first tried simply saving a QVariant containing a > > QByteArray (the contents of which were QDataStream's output). This > > didn't work because Qt's JSON implementation doesn't support > > QByteArray; it only supports QString. That is, calling > > QJsonObject::fromVariantMap(map) where "map" contains a QVariant > > containing a QByteArray will result in the corresponding JSON value > > being invalid/null. > > I'm not sure where JSON got involved in that What do you mean? It's in the title of the email; it's a popular choice for storing data like this, so I want SplitView's serialisation to be compatible with it. >but if you can manage to use > CBOR instead, it's just as happy with QByteArray as with QString (and round- > trips them to QByteArray, IIRC). For details, see > qtbase/src/corelib/serialization/qcborstream.h > > I've also no idea how well it plays with QSettings; hopefully you know enough > to work it out from the QCbor API ... Thanks, I'll take a look. > Eddy. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Serialising UI state in QML via QSettings and JSON: QByteArray vs QString
Mitch Curtis (29 October 2018 17:42) > To solve #2, I first tried simply saving a QVariant containing a > QByteArray (the contents of which were QDataStream's output). This > didn't work because Qt's JSON implementation doesn't support > QByteArray; it only supports QString. That is, calling > QJsonObject::fromVariantMap(map) where "map" contains a QVariant > containing a QByteArray will result in the corresponding JSON value > being invalid/null. I'm not sure where JSON got involved in that, but if you can manage to use CBOR instead, it's just as happy with QByteArray as with QString (and round-trips them to QByteArray, IIRC). For details, see qtbase/src/corelib/serialization/qcborstream.h I've also no idea how well it plays with QSettings; hopefully you know enough to work it out from the QCbor API ... Eddy. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
