Re: [Development] QVariant container API
If you are going to review the proposed approach I don't mind making a voice call. If someone says that this is alright, I will spend time and create a gerrit patchset. On Mon, Mar 23, 2020 at 11:45 AM Ulf Hermann wrote: > Hi, > > We have the same problem in QML: We want to support user-defined > containers as transparently as possible, with as little "registration" > overhead as possible. Currently there is a fixed set of pre-defined > containers you pass between QML and C++ code. See [1] for the relevant > rant. I've been experimenting with a QML variant of container meta types > [2], but I haven't gotten very far, yet. I also haven't looked into the > code suggested by Vasily, yet. > > I'm pretty sure we cannot just remove QSequentialIterable and > QAssociativeIterable in Qt6. That would be too much of a source > compatibility break, especially since we haven't deprecated them in Qt5. > Any solution to container meta types needs to retain those two classes > as compatibility wrappers. > > [1] https://bugreports.qt.io/browse/QTBUG-71574 > [2] https://codereview.qt-project.org/c/qt/qtdeclarative/+/284171 > > Ulf > ___ > Development mailing list > Development@qt-project.org > https://lists.qt-project.org/listinfo/development > ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
Hi, We have the same problem in QML: We want to support user-defined containers as transparently as possible, with as little "registration" overhead as possible. Currently there is a fixed set of pre-defined containers you pass between QML and C++ code. See [1] for the relevant rant. I've been experimenting with a QML variant of container meta types [2], but I haven't gotten very far, yet. I also haven't looked into the code suggested by Vasily, yet. I'm pretty sure we cannot just remove QSequentialIterable and QAssociativeIterable in Qt6. That would be too much of a source compatibility break, especially since we haven't deprecated them in Qt5. Any solution to container meta types needs to retain those two classes as compatibility wrappers. [1] https://bugreports.qt.io/browse/QTBUG-71574 [2] https://codereview.qt-project.org/c/qt/qtdeclarative/+/284171 Ulf ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
Hi. It has been almost a year since I have raised the topic. A quick recap: I was playing with serializaion and stumbled upon a problem with container reflection. In Qt there is an API to introspect container elements dynamically via the meta type system. But there is no way to populate a container. Jedrzej Nowacki discussed a way to extend QSequentialIterable. Thiago Macieira expressed a point that QSequentialIterable seems unsuitable for the task and will lead to lossy conversions without error reporting. I took my time and created a proof of concept project. https://github.com/windymindy/rializer I want to clarify that I am not pushing any serialization API into Qt. Everyone can write his own serialization/deserialization library if necessary. But I do propose an API for a meta container class. The project is just a playground for it. https://github.com/windymindy/rializer/blob/master/sources/metatypesystem/rmetacontainer.h In my opinion it could be of great use, for example, in QML, QWebChannel or JSON-RPC 2.0 plugin for QHttpServer. There is already a feature request for the functionality. https://bugreports.qt.io/browse/QTBUG-79170 Please, tell if you mind removing QSequentialIterable and QAssociativeIterable completely in Qt 6 in favour of a QMetaContainer-like class. And also adding IsContainer flag to QMetaTyle::TypeFlag. P.S. Feel free to share any feedback. I still need to investigate automatic registration. Need to spare some time to look into /mkspecs/features/metatypes.prf . I also wonder why std::deque doesn't have built-in support like std::vector or list. https://doc.qt.io/qt-5/qmetatype.html#Q_DECLARE_SEQUENTIAL_CONTAINER_METATYPE On Mon, Apr 1, 2019 at 4:10 PM Thiago Macieira wrote: > But I'm not going to stop you. Make a proof of concept and we may be able > to > discuss. Worst case is that we don't add it to QtCore, but it gets > released > elsewhere. > On Mon, Apr 1, 2019 at 1:33 PM Jedrzej Nowacki wrote: > Sure, try, you can add me to review :-) ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
In your examples, you know the destination types statically so my suggestion did not seem off-topic. Cheers From: Vasily Pupkin To: Arnaud Clère > Using QVariant as an intermediate data structure is not optimal to > read/write your QList > It also prevents from directly using the QList type to guide the > deserialization. In QBind you do not need to register types at runtime and > you avoid losing data silently (instead you have static_asserts complaining > at compile time). The whole idea, is to provide (de)serialization mechanism in scenarios, when one cannot know the type staticaly, but can get it dynamicaly in a statically typed language. In general one has to provide a void pointer and type enumeration. But Qt already happens to have rich type information, while C++ trails a bit. And that is QVariant+QMetaType. The only thing in cannot do right now is operating on containers. And that's what I wanted to discuss. ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
> Using QVariant as an intermediate data structure is not optimal to read/write your QList > It also prevents from directly using the QList type to guide the deserialization. In QBind you do not need to register types at runtime and you avoid losing data silently (instead you have static_asserts complaining at compile time). The whole idea, is to provide (de)serialization mechanism in scenarios, when one cannot know the type staticaly, but can get it dynamicaly in a statically typed language. In general one has to provide a void pointer and type enumeration. But Qt already happens to have rich type information, while C++ trails a bit. And that is QVariant+QMetaType. The only thing in cannot do right now is operating on containers. And that's what I wanted to discuss. ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
Hi,
I have a proposal to drastically reduce this kind of boilerplate code which I
call "QBind". When I presented it last year to QtCS it was not mature enough
but it know exhibits very good performance and convenience for this kind of use
case.
Maybe you can have a look at QBind benchmark's main.cpp to see if the approach
can suit your need :
https://gricad-gitlab.univ-grenoble-alpes.fr/modmed/modmedLog/blob/master/tests/QBind/main.cpp
#L67 defines a struct Person with internal bind method (it can be external too)
>From them on, you can Read/Write a Person in various ways in a single line of
>code:
#L431 to QVariant container
#L436 to Cbor buffer (3x faster)
#L581 from Cbor buffer (with read errors)
#L693 to QTreeView or QTableView using a QStandardItemModel and providing some
metadata
https://gricad-gitlab.univ-grenoble-alpes.fr/modmed/modmedLog/blob/master/tests/QBind/qstandardmodel.PNG
Using QVariant as an intermediate data structure is not optimal to read/write
your QList:
- dynamic allocations will significantly limit the write performance, see my
benchmark that writes various tracepoint payloads to a QVariant container vs
Cbor and Json buffers
https://gricad-gitlab.univ-grenoble-alpes.fr/modmed/modmedLog/blob/master/tests/QBind/write.PNG
- it also prevents from directly using the QList type to guide the
deserialization. In QBind you do not need to register types at runtime and you
avoid losing data silently (instead you have static_asserts complaining at
compile time).
Arnaud Clère
From: Vasily Pupkin
Sent: 01 April 2019 15:44
To: Thiago Macieira
Cc: development@qt-project.org
Subject: Re: [Development] QVariant container API
> By the way, conversions through QVariant are *exactly* what I implemented in
> the examples/corelib/serialization/convert tool.
Right, I've seen these to be helpful.
The global task would allow to code like this.
~
struct MyStruct
{
int property1;
QString property2;
};
class MyObject
{
Q_OBJECT
...
public slots:
void mySlot(const QList& value);
}
QJsonDocument doc(...);
QMetaMethod mySlot(...);
JsonDeserializer deserializer;
QVariant parameter;
if(deserializer.deserialize(doc, mySlot.paremeterType(0), parameter)
invoke(...);
else
{
//error
}
~
The only difference from examples, is that library user specifies, what
datatype should be produced. This would alllow to significantly reduce the
amount of boilerplate code.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
> By the way, conversions through QVariant are *exactly* what I implemented
in
> the examples/corelib/serialization/convert tool.
Right, I've seen these to be helpful.
The global task would allow to code like this.
~
struct MyStruct
{
int property1;
QString property2;
};
class MyObject
{
Q_OBJECT
...
public slots:
void mySlot(const QList& value);
}
QJsonDocument doc(...);
QMetaMethod mySlot(...);
JsonDeserializer deserializer;
QVariant parameter;
if(deserializer.deserialize(doc, mySlot.paremeterType(0), parameter)
invoke(...);
else
{
//error
}
~
The only difference from examples, is that library user specifies, what
datatype should be produced. This would alllow to significantly reduce the
amount of boilerplate code.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
On Monday, 1 April 2019 12:41:59 CEST Vasily Pupkin wrote: > The API would be QByteArray serialize(const QVariant& value), and QVariant > might be a gadget or a collection of any depth. The only thing it does, is > peaking inside QVariant and recursively (de)serializes properties or items. By the way, conversions through QVariant are *exactly* what I implemented in the examples/corelib/serialization/convert tool. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
On Monday, 1 April 2019 12:41:59 CEST Vasily Pupkin wrote: > > is too fragile and not compatible enough with standards. I recommend you > > reconsider and think about a more standard format. > > Just wanted to clarify, that I am working on a solution, which is totaly > different from datastream serialization (and opposes it) and is format > agnostic. It has an abstract module, which can be implemented for XML, JSON > and so on. > The API would be QByteArray serialize(const QVariant& value), and QVariant > might be a gadget or a collection of any depth. The only thing it does, is > peaking inside QVariant and recursively (de)serializes properties or items. My first reaction is that I don't like it. We already have too many of those and we've had proposals for such API before. But I'm not going to stop you. Make a proof of concept and we may be able to discuss. Worst case is that we don't add it to QtCore, but it gets released elsewhere. > > Nor I. That code is black magic to me and I don't touch it. > > Is it going to be left like that forever :( ? Yes, unless someone steps up and maintains it. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
01.04.2019, 13:43, "Vasily Pupkin" : > Thanks for the answers. > >> I'm not sure I want to see more Datastream-based serialisation. The code >> there >> is too fragile and not compatible enough with standards. I recommend you >> reconsider and think about a more standard format. > > Just wanted to clarify, that I am working on a solution, which is totaly > different from datastream serialization (and opposes it) and is format > agnostic. It has an abstract module, which can be implemented for XML, JSON > and so on. > The API would be QByteArray serialize(const QVariant& value), and QVariant > might be a gadget or a collection of any depth. The only thing it does, is > peaking inside QVariant and recursively (de)serializes properties or items. > >> This is a very lossy conversion without sufficient error reporting. That's an >> example of why I don't think we should invest more time in this type of API. > > Thanks for pointing this out. > >> Nor I. That code is black magic to me and I don't touch it. > > Is it going to be left like that forever :( ? Someone has to step up as a maintainer for that code, otherwise it's entirely up to Thiago to decide its fate. -- Regards, Konstantin ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
> You can be sure that the conversion from QList to QSequentialIterable > will be successful, but the other way around is not given. QSequentialIterable > can contain QVariants of different types. I'm curious what is the use case for > the feature? Imagine one could call a Q_INVOKABLE or a slot given QJsonDocument. The approach is to get parameter type ids, to make deserializations into QVariant instances and to invoke. This becomes non-trivial for container types of arbitary element types. > I think it could be implemented by extending: > https://code.woboq.org/qt5/qtbase/src/corelib/kernel/qmetatype.h.html#2306 > which registers conversion from a container to QSequentialIterable. You could > add the opposite function. That way no binary compatibility would be broken > nor major re-factoring would be needed. That's right. This solution works exactly like you suggest https://github.com/Skycoder42/QtJsonSerializer . I am seeking for a way of eliminating a need of registering types and converters. > Sure, try, you can add me to review :-) Great! Thanks! ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
Thanks for the answers. > I'm not sure I want to see more Datastream-based serialisation. The code there > is too fragile and not compatible enough with standards. I recommend you > reconsider and think about a more standard format. Just wanted to clarify, that I am working on a solution, which is totaly different from datastream serialization (and opposes it) and is format agnostic. It has an abstract module, which can be implemented for XML, JSON and so on. The API would be QByteArray serialize(const QVariant& value), and QVariant might be a gadget or a collection of any depth. The only thing it does, is peaking inside QVariant and recursively (de)serializes properties or items. > This is a very lossy conversion without sufficient error reporting. That's an > example of why I don't think we should invest more time in this type of API. Thanks for pointing this out. > Nor I. That code is black magic to me and I don't touch it. Is it going to be left like that forever :( ? ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
On Monday, April 1, 2019 11:09:27 AM CEST Vasily Pupkin wrote: > Hi. > > I would like to submit a patch. Since it is probably going to break binary > compatibility and is mostly about coding conventions, I would like to have > some feedback before investing time. Great! > The general idea is to create a playground project for automatic > (de)serialization for Qt (not manual, i.e. overriding operators). The only > stopper is container deserialization. > To operate containers without staticaly knowing their types, one should be > able to make conversions from QVariant containing a container to container > of QVariants and vice versa. > Seriazation is quite straightforward via QSequentialIterable and > QAssociativeIterable. But deserialization is currently possible only by > QMetaType::registerConverter. I think, that this is would be so nice to > have an ability to create a container via QVariant from collection of > QVariant instances without a need for custom converters. > ~ > QList variantContainer; > variantContainer.append(QVariant(123)); > variantContainer.append(QVariant(321)); > QVariant variant(QVariant::fromValue(variantContainer)); > QVariant containerVariant(variant.convert(qMetaTypeId >())); > ~ You can be sure that the conversion from QList to QSequentialIterable will be successful, but the other way around is not given. QSequentialIterable can contain QVariants of different types. I'm curious what is the use case for the feature? I think it could be implemented by extending: https://code.woboq.org/qt5/qtbase/src/corelib/kernel/qmetatype.h.html#2306 which registers conversion from a container to QSequentialIterable. You could add the opposite function. That way no binary compatibility would be broken nor major re-factoring would be needed. > And so I have stumbled into a problem. QSequentialIterable and > QAssociativeIterable related code is scattered across and > . All related conversions are conveyed in templates, instead > of > https://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/kernel/qvariant.cpp#n > 387 . > I just don't feel, that further > https://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/kernel/qvariant.h#n78 > 2 and > https://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/kernel/qmetatype.h#n1 > 121 development > is the way to go. Well, it is a bit of a mess. Conceptually, conversion code should be in qmetatype.cpp/qmetatype.h. QVariant should use QMetaType to access / modify / create / convert instances of different types. The code is not well split because QMetaType class is a late addition, so as usual historical reasons :-). If you want to cleanup it a bit I would start on moving QSequentialIterable and QAssociativeIterable out of qvariant.h to own headers. The fact that the conversion happens through QVariantValueHelperInterface in my opinion is a mistake, this code should be de-inlined as it is hard to extend it. > And so the question is: should I try to refactor sequential and associative > iterators, so templates are used only to staticaly gather information about > containers and conversions are made in qvariant.cpp, where they belong? Sure, try, you can add me to review :-) > It also would be nice to hear, if someone acknowledges this conversion > feature to be of use. Yeah, there are not so many users of QXXXIterable. So as long as we can keep cost of it low it is fine. > Thanks for reading this far. Thanks! Jędrek ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] QVariant container API
On Monday, 1 April 2019 11:09:27 CEST Vasily Pupkin wrote: > Hi. > > I would like to submit a patch. Since it is probably going to break binary > compatibility and is mostly about coding conventions, I would like to have > some feedback before investing time. If it's going to break BC, it's a Qt6 API only. Please design your change so that it exists inside an #ifdef alongside the existing code. > The general idea is to create a playground project for automatic > (de)serialization for Qt (not manual, i.e. overriding operators). The only > stopper is container deserialization. I'm not sure I want to see more Datastream-based serialisation. The code there is too fragile and not compatible enough with standards. I recommend you reconsider and think about a more standard format. > Seriazation is quite straightforward via QSequentialIterable and > QAssociativeIterable. But deserialization is currently possible only by > QMetaType::registerConverter. I think, that this is would be so nice to > have an ability to create a container via QVariant from collection of > QVariant instances without a need for custom converters. > ~ > QList variantContainer; > variantContainer.append(QVariant(123)); > variantContainer.append(QVariant(321)); > QVariant variant(QVariant::fromValue(variantContainer)); > QVariant containerVariant(variant.convert(qMetaTypeId >())); > ~ This is a very lossy conversion without sufficient error reporting. That's an example of why I don't think we should invest more time in this type of API. > And so I have stumbled into a problem. QSequentialIterable and > QAssociativeIterable related code is scattered across and > . All related conversions are conveyed in templates, instead > of > https://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/kernel/qvariant.cpp#n > 387 . > I just don't feel, that further > https://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/kernel/qvariant.h#n78 > 2 and > https://code.qt.io/cgit/qt/qtbase.git/tree/src/corelib/kernel/qmetatype.h#n1 > 121 development > is the way to go. Nor I. That code is black magic to me and I don't touch it. > And so the question is: should I try to refactor sequential and associative > iterators, so templates are used only to staticaly gather information about > containers and conversions are made in qvariant.cpp, where they belong? I suggest you leave it alone. > It also would be nice to hear, if someone acknowledges this conversion > feature to be of use. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel System Software Products ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
