Re: [Development] QDataStream: blackbox or document all versions?

2016-10-03 Thread Bernhard Lindner 
> You could choose to turn QDataStream into a black-box, but I think there
> should be a white-box alternative which has to be
> 1/ as efficient : binary format
> 2/ as easy to use : QDataStream is able to serialize any type with the help
> of qRegisterMetaTypeStreamOperators
> 3/ as generic : it should be able to use any QIODevice for transport/storage

+1

> 1/ discards XML and JSON, and 3/ discards QSettings and D-Bus.

Thanks for pointing this out!

P.S.: https://bugreports.qt.io/browse/QTBUG-25237

-- 
Regards, Bernhard

signature.asc
Description: This is a digitally signed message part.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Thiago Macieira 
On segunda-feira, 26 de setembro de 2016 10:39:51 PDT Konstantin Tokarev 
wrote:
> 26.09.2016, 10:15, "Thiago Macieira" :
> > On segunda-feira, 26 de setembro de 2016 00:42:12 PDT Konstantin Tokarev
> > 
> > wrote:
> >>  Option 3: restore documentation of version 12 and declare that it's the
> >> only format version that is meant for use with 3rd party encoders and
> >> decoders.> 
> > Why 12? It can't transmit QDateTime timezones.
> 
> Just because it is already documented, so no extra work is needed.

Version 13 is documented:

http://doc.qt.io/qt-5/datastreamformat.html

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Thiago Macieira 
On segunda-feira, 26 de setembro de 2016 12:00:46 PDT Stottlemyer, Brett 
(B.S.) wrote:
> Not really sure which reply to respond to with my thoughts, so I’m going
> back to the beginning.
 
> When you talk about documenting QDataStream, you are talking about using it
> with IPC, either to other processes on the same machine or different
> devices.  What you care about depends very much on what you are trying to
> do, which brings up the question of what should Qt support.  Questions off
> the top of my head: Which types are supported?  What errors are
> checked/caught?  Are they concerned with speed?  Speed on which platform? 
> Or are they concerned with packet size?  Which languages are supported?

Good questions all of them.

Right now, QDataStream does not catch errors, is not very speedy (big endian 
by default), is more concerned for easy marshalling and demarshalling of Qt 
types rather than compatibility or packet size.
 
> Which types?
> I’m going to assume that when Thiago talks about documenting QDataStream, he
> is talking about the operators in qtbase only?  That is, all of the custom
> QDataStream operators in other modules, whether they get an internal index
> or not?  Or is it every internal type, not matter which module?  I guess I
> should ask, rather than assume.  As I understand it, Qt stores an internal
> list of known types, each with an index.  When data is written, the index
> is written first, then the element itself, in whatever format.  Custom
> types won’t have an index by default, so if you pass one of those,
> QDataStream sends the name as well, and I’d assume, a size.  Even if the
> type has the data stream operator in both processes (or on both devices),
> it likely won’t be the same index, thus the name provides the way to look
> up the index.

You're describing how QVariant's operator<< works. Outside of QVariant, 
there's no type index: the data is written as-is, no prefix, no header. You can 
write one quint32 and read two quint16. You can even write one QString and 
read a quint32 (then throw away the data).
 
> I think we should assume that people will want to pass custom types, which
> means there needs to be a documented way to describe them.  You know people
> will want to pass Q_ENUM values, right?

They write their operator<< and operator>>. That's all.
 
> For my part, I’m very interested in this discussion.  Thiago’s point that
> JSON/DBus/etc provide alternatives to QDataStream is correct, but isn’t
> great for my use case.  As a user (and developer of) Qt Remote Objects[1],
> communication is between Qt and Qt, so it uses QDataStream.  To be able to
> also support interaction with non-Qt processes would great, but only if it
> can re-use the QDataStream.  I’m just not interested in the overhead (work
> and processing) of other formats when the focus is Qt.

Ok, this is a good point. Just like Rich mentioned for DCOP, we'd need to 
document at least one QDataStream version for Qt Remote Objects to be usable 
with non-Qt implementations.
 
-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Thiago Macieira 
On segunda-feira, 26 de setembro de 2016 10:53:51 PDT Richard Moore wrote:
> On 26 September 2016 at 10:38, Giuseppe D'Angelo 
> wrote:
> > Il 25/09/2016 05:58, Thiago Macieira ha scritto:
> > anyhow). Plus, the mere existence of that page means that someone is
> > relying on the binary format.
> 
> ​IIRC it was added to allow DCOP to be used in non-Qt contexts.

Makes sense, but then it should have stayed the Qt 3.3 documentation (or 3.0?) 
and not updated. DCOP had a fixed QDataStream version number.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Thiago Macieira 
On segunda-feira, 26 de setembro de 2016 10:29:07 PDT Shawn Rutledge wrote:
> On Sep 26, 2016, at 11:34, Simon Hausmann
> > wrote:
> 
> Hi,
> 
> I'm very much in favor of using a proper schema based system such as
> protocol buffers if we decide to remove the black box from serialization.
> They don't appear to be connected, but the moment you need to deal with
> changes in the format, the protobuf approach wins IMO. The other advantage
> is interoperability with the world outside of Qt.

Interoperability would be a plus, indeed. But if you want Protobuf, why not 
use Protobuf? Why do you need QDataStream to do it?

> There are many alternatives to protocol buffers, some of them faster and/or
> more compressed.  https://capnproto.org/ claims to be a really good one,
> for example.  It has an MIT license.  Zero-copy is a nice feature to have. 
> After what I read, I’d probably never choose to use protocol buffers if it
> can be avoided.  You have to use the code generator, and it’s not
> efficient.  But I never tried, either.

Zero-copy is useless for Qt due to the way our strings are stored in memory 
anyway (UTF-16 host-endian) and due to our implicit sharing.

If I were to choose and implement this, I'd choose CBOR for these reasons:
* I personally already know the format
* I maintain a library to encode and decode it (whose unit tests are written 
   with QtTest)
* it's in an RFC (7049), which none other besides JSON is
* it's got its own MIME type (application/cbor) and CoAP content-format (60)

> Being able to mmap the file and immediately treat it as the data structure
> that you really wanted is a nice feature to have; that kind of
> implementation is not necessarily the same thing as a serialization
> protocol, although the ideal serialization protocol could be designed so
> that you can deal with it either way.  Doesn’t fit the QDataStream API,
> anyway.

Again, not so useful to us. If we need that, we already have one: the 
QJsonDocument binary format. If we need another, I'd recommend the dconf 
format, which is the same as the D-Bus wire format plus indexing, is already 
very much in use in Linux and has a Qt implementation already written.

And as you said, mmap() ability is usually at odds with streaming.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Thiago Macieira 
On segunda-feira, 26 de setembro de 2016 10:40:02 PDT Michael Winkelmann 
wrote:
> I think what Qt is lacking is a good serialization module like cereal or
> boost serialization.
> With this pattern, QDataStream would be just another archive format,
> like JSON or XML.
> Cereal has portable and unportable binary for endianness.

Having worked with Cereal on another project, I don't think that's a good 
approach. Serialising to different protocols and formats is not something that 
can be properly abstracted. Not if you want to produce a schema that the other 
side expects.

If you're going to serialise something that only you will read, then 
QDataStream is already sufficient.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Stottlemyer, Brett (B.S.) 
On 9/24/16, 11:58 PM, "Development on behalf of Thiago Macieira" 
 wrote:



>A thread[1] on the interest mailing list started when someone asked for the 
>docs for the current format of the QDataStream wire protocol, to which I 
>replied that it doesn't exist as we don't maintain such docs.

Not really sure which reply to respond to with my thoughts, so I’m going back 
to the beginning.

When you talk about documenting QDataStream, you are talking about using it 
with IPC, either to other processes on the same machine or different devices.  
What you care about depends very much on what you are trying to do, which 
brings up the question of what should Qt support.  Questions off the top of my 
head: Which types are supported?  What errors are checked/caught?  Are they 
concerned with speed?  Speed on which platform?  Or are they concerned with 
packet size?  Which languages are supported?

Which types?
I’m going to assume that when Thiago talks about documenting QDataStream, he is 
talking about the operators in qtbase only?  That is, all of the custom 
QDataStream operators in other modules, whether they get an internal index or 
not?  Or is it every internal type, not matter which module?  I guess I should 
ask, rather than assume.  As I understand it, Qt stores an internal list of 
known types, each with an index.  When data is written, the index is written 
first, then the element itself, in whatever format.  Custom types won’t have an 
index by default, so if you pass one of those, QDataStream sends the name as 
well, and I’d assume, a size.  Even if the type has the data stream operator in 
both processes (or on both devices), it likely won’t be the same index, thus 
the name provides the way to look up the index.

I think we should assume that people will want to pass custom types, which 
means there needs to be a documented way to describe them.  You know people 
will want to pass Q_ENUM values, right?

Hopefully the question of custom types is valid, even if my attempted 
description of the internals is faulty.

For all of the other questions.
Different answers would likely favor different protocols.  There simply isn’t 
one best answer.  If we selected a specific use-case, there may be a best 
answer for that, but is that the best solution for Qt?

For my part, I’m very interested in this discussion.  Thiago’s point that 
JSON/DBus/etc provide alternatives to QDataStream is correct, but isn’t great 
for my use case.  As a user (and developer of) Qt Remote Objects[1], 
communication is between Qt and Qt, so it uses QDataStream.  To be able to also 
support interaction with non-Qt processes would great, but only if it can 
re-use the QDataStream.  I’m just not interested in the overhead (work and 
processing) of other formats when the focus is Qt.

My ideal option (specific to the use-case I care most about) would be a 
documented QDataStream format that supported one-time passing of custom type 
information (rather than passing each instance of a type), with minimal 
overhead on embedded targets and enough information to ignore unknown types.

[1] https://codereview.qt-project.org/p/playground/qtremoteobjects.git
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Konstantin Tokarev 


26.09.2016, 13:29, "Shawn Rutledge" :
> On Sep 26, 2016, at 11:34, Simon Hausmann  wrote:
>
>> Hi,
>>
>> I'm very much in favor of using a proper schema based system such as 
>> protocol buffers if we decide
>> to remove the black box from serialization. They don't appear to be 
>> connected, but the moment you
>> need to deal with changes in the format, the protobuf approach wins IMO. The 
>> other advantage is interoperability
>> with the world outside of Qt.
>
> There are many alternatives to protocol buffers, some of them faster and/or 
> more compressed.  https://capnproto.org/ claims to be a really good one, for 
> example.  It has an MIT license.  Zero-copy is a nice feature to have.  After 
> what I read, I’d probably never choose to use protocol buffers if it can be 
> avoided.  You have to use the code generator, and it’s not efficient.  But I 
> never tried, either.

Note that you probably won't probably be able to get zero-copy for Qt types, as 
canonical use of Cap'n'Proto assumes that you use Cap'n'Proto data types for 
encoded data. Though it still should be more efficient that protobuf as it 
doesn't require you to convert data to std containers and then back, and you'll 
probably be able to take more shortcuts when constructing Qt types from 
"zero-copy" Cap'n'Proto types.

>
> Being able to mmap the file and immediately treat it as the data structure 
> that you really wanted is a nice feature to have; that kind of implementation 
> is not necessarily the same thing as a serialization protocol, although the 
> ideal serialization protocol could be designed so that you can deal with it 
> either way.  Doesn’t fit the QDataStream API, anyway.
>
> Wikipedia has a comparison:  
> https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats
>
> There are even several that claim to be both binary and human-readable.  i 
> always thought that a binary format which doesn’t require a schema, uses tags 
> like XML, but can represent the common data types natively (various types of 
> numbers, bools, enums, opaque byte arrays, and strings) and for which a tool 
> is available to format it to be human-readable ought to be better than things 
> like XML and JSON.  An early example is wbxml: WAP Binary XML (but it doesn’t 
> have any representation for floating-point numbers AFAICT).  I also wrote one 
> which uses a string table for the tags, and binary representation for the 
> tree structure and the data.  But then I realized maybe I should have 
> designed it for mmapping rather than only for bytewise streaming.  But doing 
> alignment wastes some space.
>
> If you like schemas or IDL, there’s the DDS serialization protocol.  We’ve 
> already tossed around the idea of having a Qt DDS wrapper because it would be 
> useful in certain known industries.  But it requires more structure and 
> discipline compared to tag-based formats.  It resembles CORBA because it 
> comes from OMG.  Like CORBA, it’s not worth the effort for rapid prototyping, 
> only for larger-scale projects where the need for robustness outweighs the 
> need to have a short development effort.
>
> QDataStream is just a sequence - you have to know what to expect when you are 
> deserializing, rather than checking tag names or using a schema.  Unless you 
> just have a convention that the stream will consist of alternating tags and 
> values.
>
> Is it actually that useful?  Maybe we should deprecate it and come up with 
> something better?
>
> ,
>
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development


-- 
Regards,
Konstantin
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Bogdan Vatra 
Hi,

What about Option 3 :) ?
Use another binary format for serialization: flatbuffers [1] which is super 
fast, has a stable binary format and can be used by lots of other languages 
[2].

Cheers,
BogDan.

P.S. On my personal fork [3] I even added Qt support, and it can be used from 
QML as well ;-)

[1] http://google.github.io/flatbuffers/
[2] http://google.github.io/flatbuffers/flatbuffers_support.html
[3] https://github.com/bog-dan-ro/flatbuffers

On sâmbătă, 24 septembrie 2016 20:58:38 EEST Thiago Macieira wrote:
> A thread[1] on the interest mailing list started when someone asked for the
> docs for the current format of the QDataStream wire protocol, to which I
> replied that it doesn't exist as we don't maintain such docs.
> 
> Long story short, we have two options:
> 
> Option 1: claim QDataStream is a blackbox and tell people that the only
> thing that can read QDataStream is QDataStream. That means removing the
> documentation file src/corelib/doc/src/datastreamformat.qdoc, as we don't
> want people getting any ideas that they could write their own decoders or
> encoders.
> 
> Option 2: the opposite, saying that reading QDataStream's output is fine
> from non-Qt code and it's fine to write data that QDataStream should read.
> This means extending the documentation to cover ALL 17 wire formats
> (including bugs) and keeping it up to date whenever someone modifies the
> format.
> 
> 
> I am in favour of Option 1 because I really don't think QDataStream is a
> good format for exchanging data with non-Qt code. It's designed first and
> foremost for Qt's own internal data formats (sometimes even depending on
> internal details), the marshalling of certain types in certain versions is
> buggy and lossy. Instead, people should find a better transport format for
> their data, of which we already have in Qt:
> 
>   XML
>   JSON
>   D-Bus
>   QSettings (to an extent)
> 
> And I can add CBOR support if we want to.
> 
> [1]
> http://lists.qt-project.org/pipermail/interest/2016-September/024387.html


___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Shawn Rutledge 

On Sep 26, 2016, at 11:34, Simon Hausmann 
> wrote:

Hi,

I'm very much in favor of using a proper schema based system such as protocol 
buffers if we decide
to remove the black box from serialization. They don't appear to be connected, 
but the moment you
need to deal with changes in the format, the protobuf approach wins IMO. The 
other advantage is interoperability
with the world outside of Qt.

There are many alternatives to protocol buffers, some of them faster and/or 
more compressed.  https://capnproto.org/ claims to be a really good one, for 
example.  It has an MIT license.  Zero-copy is a nice feature to have.  After 
what I read, I’d probably never choose to use protocol buffers if it can be 
avoided.  You have to use the code generator, and it’s not efficient.  But I 
never tried, either.

Being able to mmap the file and immediately treat it as the data structure that 
you really wanted is a nice feature to have; that kind of implementation is not 
necessarily the same thing as a serialization protocol, although the ideal 
serialization protocol could be designed so that you can deal with it either 
way.  Doesn’t fit the QDataStream API, anyway.

Wikipedia has a comparison:  
https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats

There are even several that claim to be both binary and human-readable.  i 
always thought that a binary format which doesn’t require a schema, uses tags 
like XML, but can represent the common data types natively (various types of 
numbers, bools, enums, opaque byte arrays, and strings) and for which a tool is 
available to format it to be human-readable ought to be better than things like 
XML and JSON.  An early example is wbxml: WAP Binary XML (but it doesn’t have 
any representation for floating-point numbers AFAICT).  I also wrote one which 
uses a string table for the tags, and binary representation for the tree 
structure and the data.  But then I realized maybe I should have designed it 
for mmapping rather than only for bytewise streaming.  But doing alignment 
wastes some space.

If you like schemas or IDL, there’s the DDS serialization protocol.  We’ve 
already tossed around the idea of having a Qt DDS wrapper because it would be 
useful in certain known industries.  But it requires more structure and 
discipline compared to tag-based formats.  It resembles CORBA because it comes 
from OMG.  Like CORBA, it’s not worth the effort for rapid prototyping, only 
for larger-scale projects where the need for robustness outweighs the need to 
have a short development effort.

QDataStream is just a sequence - you have to know what to expect when you are 
deserializing, rather than checking tag names or using a schema.  Unless you 
just have a convention that the stream will consist of alternating tags and 
values.

Is it actually that useful?  Maybe we should deprecate it and come up with 
something better?

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Richard Moore 
On 26 September 2016 at 10:38, Giuseppe D'Angelo 
wrote:

> Il 25/09/2016 05:58, Thiago Macieira ha scritto:
> anyhow). Plus, the mere existence of that page means that someone is
> relying on the binary format.
>

​IIRC it was added to allow DCOP to be used in non-Qt contexts.

Rich.​
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Konstantin Tokarev 


26.09.2016, 12:35, "Simon Hausmann" <simon.hausm...@qt.io>:
> Hi,
>
> I'm very much in favor of using a proper schema based system such as protocol 
> buffers if we decide
>
> to remove the black box from serialization. They don't appear to be 
> connected, but the moment you
>
> need to deal with changes in the format, the protobuf approach wins IMO. The 
> other advantage is interoperability
>
> with the world outside of Qt.

Protobuf itself is kinda outdated format, there are a lot of newer alternatives 
which incur much lower overhead. But it certainly has advantage of wider 
interoperability.

>
> But this isn't so much a question of what to do, I I think it's more of a 
> question of man power. Until then I'm in favor
>
> of QDataStream remaining a black box.
>
> Simon
> 
> From: Development <development-bounces+simon.hausmann=qt...@qt-project.org> 
> on behalf of Michael Winkelmann <michael.winkelm...@qt.io>
> Sent: Monday, September 26, 2016 10:40:02 AM
> To: development@qt-project.org
> Subject: Re: [Development] QDataStream: blackbox or document all versions?
>
> I think what Qt is lacking is a good serialization module like cereal or
> boost serialization.
> With this pattern, QDataStream would be just another archive format,
> like JSON or XML.
> Cereal has portable and unportable binary for endianness.
>
> The developer should be able specify how the data is serialized and
> select an arbitrary archive type that takes streams operators as default
> serialization method.
>
> PS: For wire protocols, I would also consider Protobuf as an option.
>
> On 26.09.2016 10:11, Ulf Hermann wrote:
>> On 09/25/2016 05:58 AM, Thiago Macieira wrote:
>>> A thread[1] on the interest mailing list started when someone asked
>>> for the
>>> docs for the current format of the QDataStream wire protocol, to which I
>>> replied that it doesn't exist as we don't maintain such docs.
>>>
>>> Long story short, we have two options: [...]
>>
>> I see another option: Make QDataStream a good interchange format.
>> From your reply on the interest mailing list:
>>
>>> I also don't think the QDataStream binary format is particularly good
>>> for non-
>>> Qt purposes. It passes internals that are really about how Qt works,
>>> not meant
>>> for interchange of information. For example, QDate passes the Julian
>>> Day in
>>> 64-bit format -- who in the world exchanges data like that? One would
>>> expect
>>> the serialisation format of a date and time to be something like
>>> milliseconds
>>> since 1970, milliseconds since 1601 or a string format, like ISO
>>> 3337. It
>>> passes strings as UTF-16 encoded instead of encoding with UTF-8,
>>> which would
>>> in turn allow interchanging QString and QByteArray serialisation
>>> formats.
>>>
>>> It's also by default quite inefficient since it marshalls everything
>>> as big-
>>> endian, while most CPUs are little-endian. It has no support for
>>> indexing or
>>> random-access seeking. It can't be used as a DOM storage, like
>>> QJsonDocument
>>> can.
>>
>> All of that, except for the default endianness, could be fixed on a
>> per operator base. Changing the default endianness can also be done
>> with minimal effort and would only require incrementing the data
>> stream version. Considering this, QDataStream may at the moment not be
>> very efficient, but it does have some potential. We could probably
>> even change all the operators to read and write CBOR, and that would
>> also only require a version change. The versioning allows us to adapt
>> the wire format to whatever is needed now or in the future. I don't
>> know any other data interchange format which can do this.
>>
>> This doesn't change the implicit conversion problem that Marc
>> mentioned, but at the moment I don't quite see a way to fix it without
>> removing a lot of convenience for the user (that is, have the user
>> specify the exact intended type for each serialization operation).
>>
>> regards,
>> Ulf
>>
>>
>>
>> ___
>> Development mailing list
>> Development@qt-project.org
>> http://lists.qt-project.org/mailman/listinfo/development
>
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development
> ,
>
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development


-- 
Regards,
Konstantin
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Giuseppe D'Angelo 
Il 25/09/2016 05:58, Thiago Macieira ha scritto:
> 
> Option 2: the opposite, saying that reading QDataStream's output is fine from 
> non-Qt code and it's fine to write data that QDataStream should read. This 
> means extending the documentation to cover ALL 17 wire formats (including 
> bugs) and keeping it up to date whenever someone modifies the format.
> 
> 
> I am in favour of Option 1 because I really don't think QDataStream is a good 
> format for exchanging data with non-Qt code. It's designed first and foremost 
> for Qt's own internal data formats (sometimes even depending on internal 
> details), the marshalling of certain types in certain versions is buggy and 
> lossy. Instead, people should find a better transport format for their data, 
> of 
> which we already have in Qt:

For the record, I'm for option 2, as I don't think it poses a huge
burden (apart from an initial task to add older protocol versions to it,
then it's just a matter of maintaining documentation, and we must do it
anyhow). Plus, the mere existence of that page means that someone is
relying on the binary format.

That having being said, I agree that QDataStream hasn't got the best
wire format or the best implementation, but this is totally orthogonal
to the matter at hand.

Cheers,
-- 
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (UK) Ltd., a KDAB Group company | Tel: UK +44-1625-809908
KDAB - Qt, C++ and OpenGL Experts



smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Simon Hausmann 
Hi,


I'm very much in favor of using a proper schema based system such as protocol 
buffers if we decide

to remove the black box from serialization. They don't appear to be connected, 
but the moment you

need to deal with changes in the format, the protobuf approach wins IMO. The 
other advantage is interoperability

with the world outside of Qt.


But this isn't so much a question of what to do, I I think it's more of a 
question of man power. Until then I'm in favor

of QDataStream remaining a black box.



Simon


From: Development <development-bounces+simon.hausmann=qt...@qt-project.org> on 
behalf of Michael Winkelmann <michael.winkelm...@qt.io>
Sent: Monday, September 26, 2016 10:40:02 AM
To: development@qt-project.org
Subject: Re: [Development] QDataStream: blackbox or document all versions?

I think what Qt is lacking is a good serialization module like cereal or
boost serialization.
With this pattern, QDataStream would be just another archive format,
like JSON or XML.
Cereal has portable and unportable binary for endianness.

The developer should be able specify how the data is serialized and
select an arbitrary archive type that takes streams operators as default
serialization method.

PS: For wire protocols, I would also consider Protobuf as an option.



On 26.09.2016 10:11, Ulf Hermann wrote:
> On 09/25/2016 05:58 AM, Thiago Macieira wrote:
>> A thread[1] on the interest mailing list started when someone asked
>> for the
>> docs for the current format of the QDataStream wire protocol, to which I
>> replied that it doesn't exist as we don't maintain such docs.
>>
>> Long story short, we have two options: [...]
>
> I see another option: Make QDataStream a good interchange format.
> From your reply on the interest mailing list:
>
>> I also don't think the QDataStream binary format is particularly good
>> for non-
>> Qt purposes. It passes internals that are really about how Qt works,
>> not meant
>> for interchange of information. For example, QDate passes the Julian
>> Day in
>> 64-bit format -- who in the world exchanges data like that? One would
>> expect
>> the serialisation format of a date and time to be something like
>> milliseconds
>> since 1970, milliseconds since 1601 or a string format, like ISO
>> 3337. It
>> passes strings as UTF-16 encoded instead of encoding with UTF-8,
>> which would
>> in turn allow interchanging QString and QByteArray serialisation
>> formats.
>>
>> It's also by default quite inefficient since it marshalls everything
>> as big-
>> endian, while most CPUs are little-endian. It has no support for
>> indexing or
>> random-access seeking. It can't be used as a DOM storage, like
>> QJsonDocument
>> can.
>
> All of that, except for the default endianness, could be fixed on a
> per operator base. Changing the default endianness can also be done
> with minimal effort and would only require incrementing the data
> stream version. Considering this, QDataStream may at the moment not be
> very efficient, but it does have some potential. We could probably
> even change all the operators to read and write CBOR, and that would
> also only require a version change. The versioning allows us to adapt
> the wire format to whatever is needed now or in the future. I don't
> know any other data interchange format which can do this.
>
> This doesn't change the implicit conversion problem that Marc
> mentioned, but at the moment I don't quite see a way to fix it without
> removing a lot of convenience for the user (that is, have the user
> specify the exact intended type for each serialization operation).
>
> regards,
> Ulf
>
>
>
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Ulf Hermann 

On 09/26/2016 10:40 AM, Michael Winkelmann wrote:

I think what Qt is lacking is a good serialization module like cereal or
boost serialization.
With this pattern, QDataStream would be just another archive format,
like JSON or XML.
Cereal has portable and unportable binary for endianness.


Btw, QDataStream has a method setByteOrder(). Big endian is just the default.

regards,
Ulf
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Michael Winkelmann 
I think what Qt is lacking is a good serialization module like cereal or
boost serialization.
With this pattern, QDataStream would be just another archive format,
like JSON or XML.
Cereal has portable and unportable binary for endianness.

The developer should be able specify how the data is serialized and
select an arbitrary archive type that takes streams operators as default
serialization method.

PS: For wire protocols, I would also consider Protobuf as an option.



On 26.09.2016 10:11, Ulf Hermann wrote:
> On 09/25/2016 05:58 AM, Thiago Macieira wrote:
>> A thread[1] on the interest mailing list started when someone asked
>> for the
>> docs for the current format of the QDataStream wire protocol, to which I
>> replied that it doesn't exist as we don't maintain such docs.
>>
>> Long story short, we have two options: [...]
>
> I see another option: Make QDataStream a good interchange format.
> From your reply on the interest mailing list:
>
>> I also don't think the QDataStream binary format is particularly good
>> for non-
>> Qt purposes. It passes internals that are really about how Qt works,
>> not meant
>> for interchange of information. For example, QDate passes the Julian
>> Day in
>> 64-bit format -- who in the world exchanges data like that? One would
>> expect
>> the serialisation format of a date and time to be something like
>> milliseconds
>> since 1970, milliseconds since 1601 or a string format, like ISO
>> 3337. It
>> passes strings as UTF-16 encoded instead of encoding with UTF-8,
>> which would
>> in turn allow interchanging QString and QByteArray serialisation
>> formats.
>>
>> It's also by default quite inefficient since it marshalls everything
>> as big-
>> endian, while most CPUs are little-endian. It has no support for
>> indexing or
>> random-access seeking. It can't be used as a DOM storage, like
>> QJsonDocument
>> can.
>
> All of that, except for the default endianness, could be fixed on a
> per operator base. Changing the default endianness can also be done
> with minimal effort and would only require incrementing the data
> stream version. Considering this, QDataStream may at the moment not be
> very efficient, but it does have some potential. We could probably
> even change all the operators to read and write CBOR, and that would
> also only require a version change. The versioning allows us to adapt
> the wire format to whatever is needed now or in the future. I don't
> know any other data interchange format which can do this.
>
> This doesn't change the implicit conversion problem that Marc
> mentioned, but at the moment I don't quite see a way to fix it without
> removing a lot of convenience for the user (that is, have the user
> specify the exact intended type for each serialization operation).
>
> regards,
> Ulf
>
>
>
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Marc Mutz 
On Monday 26 September 2016 10:11:09 Ulf Hermann wrote:
> This doesn't change the implicit conversion problem that Marc mentioned,
> but at the moment I don't quite see a way to fix it without removing a lot
> of convenience for the user (that is, have the user specify the exact
> intended type for each serialization operation).

The solution is to have just one (templated) op<> and (partially) 
specialise a struct that contains a static function (or a function call 
operator). This is how std::hash works (except it lacks the convenience 
function so you have to be explicit about the type), and why it's not 
susceptible to problems such a qHash(badly-designed smart-pointer) invoking 
qHash(bool) (if we had qHash(bool), which we don't have, for precisely that 
reason).

Another solution is to write every op<> as a constrained template, but 
that is decidedly not user-friendly (where user here is the user that extends 
op<> to her own types).

Thanks,
Marc

-- 
Marc Mutz  | Senior Software Engineer
KDAB (Deutschland) GmbH & Co.KG, a KDAB Group Company
Tel: +49-30-521325470
KDAB - Qt, C++ and OpenGL Experts
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Ulf Hermann 

On 09/25/2016 05:58 AM, Thiago Macieira wrote:

A thread[1] on the interest mailing list started when someone asked for the
docs for the current format of the QDataStream wire protocol, to which I
replied that it doesn't exist as we don't maintain such docs.

Long story short, we have two options: [...]


I see another option: Make QDataStream a good interchange format.
From your reply on the interest mailing list:


I also don't think the QDataStream binary format is particularly good for non-
Qt purposes. It passes internals that are really about how Qt works, not meant
for interchange of information. For example, QDate passes the Julian Day in
64-bit format -- who in the world exchanges data like that? One would expect
the serialisation format of a date and time to be something like milliseconds
since 1970, milliseconds since 1601 or a string format, like ISO 3337. It
passes strings as UTF-16 encoded instead of encoding with UTF-8, which would
in turn allow interchanging QString and QByteArray serialisation formats.

It's also by default quite inefficient since it marshalls everything as big-
endian, while most CPUs are little-endian. It has no support for indexing or
random-access seeking. It can't be used as a DOM storage, like QJsonDocument
can.


All of that, except for the default endianness, could be fixed on a per 
operator base. Changing the default endianness can also be done with minimal 
effort and would only require incrementing the data stream version. Considering 
this, QDataStream may at the moment not be very efficient, but it does have 
some potential. We could probably even change all the operators to read and 
write CBOR, and that would also only require a version change. The versioning 
allows us to adapt the wire format to whatever is needed now or in the future. 
I don't know any other data interchange format which can do this.

This doesn't change the implicit conversion problem that Marc mentioned, but at 
the moment I don't quite see a way to fix it without removing a lot of 
convenience for the user (that is, have the user specify the exact intended 
type for each serialization operation).

regards,
Ulf



___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Marc Mutz 
On Sunday 25 September 2016 19:40:06 Thiago Macieira wrote:
> But that brings us to QDataStream's biggest advantage: the wealth of
> available  operator<<.

Which is also one of its major drawbacks: Streaming out may invoke any 
implicit conversions, and while the worst of these are probably caught be the 
streaming-in code, which disables most implicit conversion by way of taking an 
lvalue reference, the derived-to-base class conversion is still performed, 
with bad potential side effects for both Qt and the user (cf. the recent 
QDBusArgument::op<<(QList) issue[1]).

Thanks,
Marc

[1] QtBase commits d55f2b1fb9c910bc118f75967a0e6273f8aa98d1 and 
5f542f3cca13f2da58b82aee2efbaffefeee00a7

-- 
Marc Mutz  | Senior Software Engineer
KDAB (Deutschland) GmbH & Co.KG, a KDAB Group Company
Tel: +49-30-521325470
KDAB - Qt, C++ and OpenGL Experts
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Konstantin Tokarev 


26.09.2016, 10:15, "Thiago Macieira" :
> On segunda-feira, 26 de setembro de 2016 00:42:12 PDT Konstantin Tokarev
> wrote:
>>  Option 3: restore documentation of version 12 and declare that it's the only
>>  format version that is meant for use with 3rd party encoders and decoders.
>
> Why 12? It can't transmit QDateTime timezones.

Just because it is already documented, so no extra work is needed.

>
> --
> Thiago Macieira - thiago.macieira (AT) intel.com
>   Software Architect - Intel Open Source Technology Center
>
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

-- 
Regards,
Konstantin
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Thiago Macieira 
On segunda-feira, 26 de setembro de 2016 06:59:01 PDT Martin Smith wrote:
> Do we need a way to tie A specific qdoc comment to the result of a specific
> autotest?

I don't think we need to go that far.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Thiago Macieira 
On segunda-feira, 26 de setembro de 2016 00:42:12 PDT Konstantin Tokarev 
wrote:
> Option 3: restore documentation of version 12 and declare that it's the only
> format version that is meant for use with 3rd party encoders and decoders.

Why 12? It can't transmit QDateTime timezones.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Thiago Macieira 
On segunda-feira, 26 de setembro de 2016 00:12:13 PDT Konstantin Tokarev 
wrote:
> In case you are going to implement CBOR from scratch, please support string
> reference extension [1] which is needed to encode repeating strings (most
> importantly, data keys) efficiently.

Please create a feature suggestion for me at
https://github.com/01org/tinycbor/
I'll implement it once it becomes an RFC (not before).

This looks like a nice suggestion, but requires the decoder to support it and 
may not be supported by all. TinyCBOR is meant to work on very constrained 
systems and keeping track of what an atom/quark is supposed to be could be 
expensive if all the decoder does is do a stream decoding and operate as it 
reads.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Martin Smith 
Do we need a way to tie A specific qdoc comment to the result of a specific 
autotest?


martin


From: Development <development-bounces+martin.smith=qt...@qt-project.org> on 
behalf of Thiago Macieira <thiago.macie...@intel.com>
Sent: Sunday, September 25, 2016 5:58:38 AM
To: development@qt-project.org
Subject: [Development] QDataStream: blackbox or document all versions?

A thread[1] on the interest mailing list started when someone asked for the
docs for the current format of the QDataStream wire protocol, to which I
replied that it doesn't exist as we don't maintain such docs.

Long story short, we have two options:

Option 1: claim QDataStream is a blackbox and tell people that the only thing
that can read QDataStream is QDataStream. That means removing the
documentation file src/corelib/doc/src/datastreamformat.qdoc, as we don't want
people getting any ideas that they could write their own decoders or encoders.

Option 2: the opposite, saying that reading QDataStream's output is fine from
non-Qt code and it's fine to write data that QDataStream should read. This
means extending the documentation to cover ALL 17 wire formats (including
bugs) and keeping it up to date whenever someone modifies the format.


I am in favour of Option 1 because I really don't think QDataStream is a good
format for exchanging data with non-Qt code. It's designed first and foremost
for Qt's own internal data formats (sometimes even depending on internal
details), the marshalling of certain types in certain versions is buggy and
lossy. Instead, people should find a better transport format for their data, of
which we already have in Qt:

XML
JSON
D-Bus
QSettings (to an extent)

And I can add CBOR support if we want to.

[1] http://lists.qt-project.org/pipermail/interest/2016-September/024387.html
--
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-26 Thread Jędrzej Nowacki 
On lørdag 24. september 2016 20.58.38 CEST Thiago Macieira wrote:
> A thread[1] on the interest mailing list started when someone asked for the
> docs for the current format of the QDataStream wire protocol, to which I
> replied that it doesn't exist as we don't maintain such docs.
> 
> Long story short, we have two options:
> 
> Option 1: claim QDataStream is a blackbox and tell people that the only
> thing that can read QDataStream is QDataStream. That means removing the
> documentation file src/corelib/doc/src/datastreamformat.qdoc, as we don't
> want people getting any ideas that they could write their own decoders or
> encoders.
> 
> Option 2: the opposite, saying that reading QDataStream's output is fine
> from non-Qt code and it's fine to write data that QDataStream should read.
> This means extending the documentation to cover ALL 17 wire formats
> (including bugs) and keeping it up to date whenever someone modifies the
> format.
> 
> 
> I am in favour of Option 1 because I really don't think QDataStream is a
> good format for exchanging data with non-Qt code. It's designed first and
> foremost for Qt's own internal data formats (sometimes even depending on
> internal details), the marshalling of certain types in certain versions is
> buggy and lossy. Instead, people should find a better transport format for
> their data, of which we already have in Qt:
> 
>   XML
>   JSON
>   D-Bus
>   QSettings (to an extent)
> 
> And I can add CBOR support if we want to.
> 
> [1]
> http://lists.qt-project.org/pipermail/interest/2016-September/024387.html

Hi,

 Option 4: Document only basic types, like char, int, maybe QByteArray and 
QString, QVector. Everything above is tight to Qt implementation and therefore 
is super hard to use from 3rd party perspective as it is changing often.

  But I agree with you QDataStream is not a good format for exchanging data 
with non-Qt code so option 1 is Ok too.

Cheers,
  Jędrek
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-25 Thread Giuseppe D'Angelo 
Il 25/09/2016 23:42, Konstantin Tokarev ha scritto:
> I think there is also
> 
> Option 3: restore documentation of version 12 and declare that it's the only 
> format version that is meant for use with 3rd party encoders and decoders.
> 

What is the rationale be for this?

Cheers,
-- 
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (UK) Ltd., a KDAB Group company | Tel: UK +44-1625-809908
KDAB - Qt, C++ and OpenGL Experts



smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-25 Thread Konstantin Tokarev 


25.09.2016, 06:58, "Thiago Macieira" :
> A thread[1] on the interest mailing list started when someone asked for the
> docs for the current format of the QDataStream wire protocol, to which I
> replied that it doesn't exist as we don't maintain such docs.
>
> Long story short, we have two options:
>
> Option 1: claim QDataStream is a blackbox and tell people that the only thing
> that can read QDataStream is QDataStream. That means removing the
> documentation file src/corelib/doc/src/datastreamformat.qdoc, as we don't want
> people getting any ideas that they could write their own decoders or encoders.
>
> Option 2: the opposite, saying that reading QDataStream's output is fine from
> non-Qt code and it's fine to write data that QDataStream should read. This
> means extending the documentation to cover ALL 17 wire formats (including
> bugs) and keeping it up to date whenever someone modifies the format.

I think there is also

Option 3: restore documentation of version 12 and declare that it's the only 
format version that is meant for use with 3rd party encoders and decoders.

>
> I am in favour of Option 1 because I really don't think QDataStream is a good
> format for exchanging data with non-Qt code. It's designed first and foremost
> for Qt's own internal data formats (sometimes even depending on internal
> details), the marshalling of certain types in certain versions is buggy and
> lossy. Instead, people should find a better transport format for their data, 
> of
> which we already have in Qt:
>
> XML
> JSON
> D-Bus
> QSettings (to an extent)
>
> And I can add CBOR support if we want to.
>
> [1] http://lists.qt-project.org/pipermail/interest/2016-September/024387.html
> --
> Thiago Macieira - thiago.macieira (AT) intel.com
>   Software Architect - Intel Open Source Technology Center
>
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

-- 
Regards,
Konstantin
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-25 Thread Konstantin Tokarev 


26.09.2016, 00:12, "Konstantin Tokarev" :
> 25.09.2016, 06:58, "Thiago Macieira" :
>>  A thread[1] on the interest mailing list started when someone asked for the
>>  docs for the current format of the QDataStream wire protocol, to which I
>>  replied that it doesn't exist as we don't maintain such docs.
>>
>>  Long story short, we have two options:
>>
>>  Option 1: claim QDataStream is a blackbox and tell people that the only 
>> thing
>>  that can read QDataStream is QDataStream. That means removing the
>>  documentation file src/corelib/doc/src/datastreamformat.qdoc, as we don't 
>> want
>>  people getting any ideas that they could write their own decoders or 
>> encoders.
>>
>>  Option 2: the opposite, saying that reading QDataStream's output is fine 
>> from
>>  non-Qt code and it's fine to write data that QDataStream should read. This
>>  means extending the documentation to cover ALL 17 wire formats (including
>>  bugs) and keeping it up to date whenever someone modifies the format.
>>
>>  I am in favour of Option 1 because I really don't think QDataStream is a 
>> good
>>  format for exchanging data with non-Qt code. It's designed first and 
>> foremost
>>  for Qt's own internal data formats (sometimes even depending on internal
>>  details), the marshalling of certain types in certain versions is buggy and
>>  lossy. Instead, people should find a better transport format for their 
>> data, of
>>  which we already have in Qt:
>>
>>  XML
>>  JSON
>>  D-Bus
>>  QSettings (to an extent)
>>
>>  And I can add CBOR support if we want to.
>
> In case you are going to implement CBOR from scratch, please support string
> reference extension [1] which is needed to encode repeating strings (most
> importantly, data keys) efficiently.
>
> [1] http://cbor.schmorp.de/stringref

BTW, newly emerged SuperPack format goes further and introduces repeated keyset
optimization

https://github.com/shapesecurity/superpack-spec
https://www.infoq.com/news/2016/08/superpack



>
>>  [1] 
>> http://lists.qt-project.org/pipermail/interest/2016-September/024387.html
>>  --
>>  Thiago Macieira - thiago.macieira (AT) intel.com
>>    Software Architect - Intel Open Source Technology Center
>>
>>  ___
>>  Development mailing list
>>  Development@qt-project.org
>>  http://lists.qt-project.org/mailman/listinfo/development
>
> --
> Regards,
> Konstantin
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

-- 
Regards,
Konstantin
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-25 Thread Konstantin Tokarev 


25.09.2016, 06:58, "Thiago Macieira" :
> A thread[1] on the interest mailing list started when someone asked for the
> docs for the current format of the QDataStream wire protocol, to which I
> replied that it doesn't exist as we don't maintain such docs.
>
> Long story short, we have two options:
>
> Option 1: claim QDataStream is a blackbox and tell people that the only thing
> that can read QDataStream is QDataStream. That means removing the
> documentation file src/corelib/doc/src/datastreamformat.qdoc, as we don't want
> people getting any ideas that they could write their own decoders or encoders.
>
> Option 2: the opposite, saying that reading QDataStream's output is fine from
> non-Qt code and it's fine to write data that QDataStream should read. This
> means extending the documentation to cover ALL 17 wire formats (including
> bugs) and keeping it up to date whenever someone modifies the format.
>
> I am in favour of Option 1 because I really don't think QDataStream is a good
> format for exchanging data with non-Qt code. It's designed first and foremost
> for Qt's own internal data formats (sometimes even depending on internal
> details), the marshalling of certain types in certain versions is buggy and
> lossy. Instead, people should find a better transport format for their data, 
> of
> which we already have in Qt:
>
> XML
> JSON
> D-Bus
> QSettings (to an extent)
>
> And I can add CBOR support if we want to.

In case you are going to implement CBOR from scratch, please support string
reference extension [1] which is needed to encode repeating strings (most
importantly, data keys) efficiently.

[1] http://cbor.schmorp.de/stringref

>
> [1] http://lists.qt-project.org/pipermail/interest/2016-September/024387.html
> --
> Thiago Macieira - thiago.macieira (AT) intel.com
>   Software Architect - Intel Open Source Technology Center
>
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

-- 
Regards,
Konstantin
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-25 Thread Thiago Macieira 
On domingo, 25 de setembro de 2016 14:24:29 PDT Eric Lemanisser wrote:
> 2/ as easy to use : QDataStream is able to serialize any type with the help
> of qRegisterMetaTypeStreamOperators

You only need that if you use QVariant. Regular QDataStream with non-type-
erased types doesn't require that call.

But that brings us to QDataStream's biggest advantage: the wealth of available 
operator<<.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QDataStream: blackbox or document all versions?

2016-09-25 Thread Eric Lemanisser 
You could choose to turn QDataStream into a black-box, but I think there
should be a white-box alternative which has to be
1/ as efficient : binary format
2/ as easy to use : QDataStream is able to serialize any type with the help
of qRegisterMetaTypeStreamOperators
3/ as generic : it should be able to use any QIODevice for transport/storage

1/ discards XML and JSON, and 3/ discards QSettings and D-Bus. I don't know
CBOR, but it seems an interesting alternative.

Eric Lemanissier

Le dim. 25 sept. 2016 à 05:59, Thiago Macieira 
a écrit :

> A thread[1] on the interest mailing list started when someone asked for the
> docs for the current format of the QDataStream wire protocol, to which I
> replied that it doesn't exist as we don't maintain such docs.
>
> Long story short, we have two options:
>
> that can read QDataStream is QDataStream. That means removing the
> Option 1: claim QDataStream is a blackbox and tell people that the only
> thing
> that can read QDataStream is QDataStream. That means removing the
> documentation file src/corelib/doc/src/datastreamformat.qdoc, as we don't
> want
> people getting any ideas that they could write their own decoders or
> encoders.
>
> Option 2: the opposite, saying that reading QDataStream's output is fine
> from
> non-Qt code and it's fine to write data that QDataStream should read. This
> means extending the documentation to cover ALL 17 wire formats (including
> bugs) and keeping it up to date whenever someone modifies the format.
>
>
> I am in favour of Option 1 because I really don't think QDataStream is a
> good
> format for exchanging data with non-Qt code. It's designed first and
> foremost
> for Qt's own internal data formats (sometimes even depending on internal
> details), the marshalling of certain types in certain versions is buggy and
> lossy. Instead, people should find a better transport format for their
> data, of
> which we already have in Qt:
>
> XML
> JSON
> D-Bus
> QSettings (to an extent)
>
> And I can add CBOR support if we want to.
>
> [1]
> http://lists.qt-project.org/pipermail/interest/2016-September/024387.html
> --
> Thiago Macieira - thiago.macieira (AT) intel.com
>   Software Architect - Intel Open Source Technology Center
>
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development
>
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] QDataStream: blackbox or document all versions?

2016-09-24 Thread Thiago Macieira 
A thread[1] on the interest mailing list started when someone asked for the 
docs for the current format of the QDataStream wire protocol, to which I 
replied that it doesn't exist as we don't maintain such docs.

Long story short, we have two options:

Option 1: claim QDataStream is a blackbox and tell people that the only thing 
that can read QDataStream is QDataStream. That means removing the 
documentation file src/corelib/doc/src/datastreamformat.qdoc, as we don't want 
people getting any ideas that they could write their own decoders or encoders.

Option 2: the opposite, saying that reading QDataStream's output is fine from 
non-Qt code and it's fine to write data that QDataStream should read. This 
means extending the documentation to cover ALL 17 wire formats (including 
bugs) and keeping it up to date whenever someone modifies the format.


I am in favour of Option 1 because I really don't think QDataStream is a good 
format for exchanging data with non-Qt code. It's designed first and foremost 
for Qt's own internal data formats (sometimes even depending on internal 
details), the marshalling of certain types in certain versions is buggy and 
lossy. Instead, people should find a better transport format for their data, of 
which we already have in Qt:

XML
JSON
D-Bus
QSettings (to an extent)

And I can add CBOR support if we want to.

[1] http://lists.qt-project.org/pipermail/interest/2016-September/024387.html
-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development