Re: [Interest] Interest Digest, Vol 80, Issue 9

[Roland Hughes]

On 05/14/2018 10:54 AM, interest-requ...@qt-project.org wrote:

Finally, Qt isn't used in mainframes. So mainframes are not an argument for
this discussion.

It's on LinuxOne.

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Wdf4d0ae77421_40d3_9e0e_ad0889993334/page/ClefOS7.1%20Disc1

search for qt.

--
Roland Hughes, President
Logikal Solutions
(630)-205-1593

http://www.theminimumyouneedtoknow.com
http://www.infiniteexposure.net
http://www.johnsmith-book.com
http://www.logikalblog.com
http://www.interestingauthors.com/blog
http://lesedi.us/
http://onedollarcontentstore.com

___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest

Re: [Interest] QFile::write(const QByteArray&) does not, write, all data?

[Thiago Macieira]

On Monday, 14 May 2018 09:17:42 PDT Roland Hughes wrote:
> > ?That is simply incorrect. It might not be how you would like it to
> > behave,
> > however it is the norm that short writes are allowed and should be handled
> > by the callee. This is the case in POSIX, the C standard library etc. I'm
> > afraid, you'll simply have to learn to live with it - an easy solution
> > would be to write a tiny wrapper function of your own however if you do
> > you'll need to be very careful with your error handling if you always want
> > to be able to retry.
> 
> Your statement would be patently false.
> =
> FREAD(3) Linux Programmer's Manual FREAD(3)

Ok, so not the standard C library. But it is still the case for POSIX and 
Win32.

In fact, the man page for write(2) says specifically 

   [...] It is not an error if this number is smaller
   than the number of bytes requested; this may happen for example
   because the disk device was filled.

See http://man7.org/linux/man-pages/man2/write.2.html

Looping trying to write more after short writes is a good idea in some 
circumstances, but could be bad in others. We can't tell why the short write 
happened: it may have been interrupted by a signal, which means we could 
easily resume without further harm. But trying to write more could trigger an 
actual error, which means we need a path to handling it. You can't count on 
subsequent writes producing the same error.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center



___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest

Re: [Interest] Interest Digest, Vol 80, Issue 9

[Roland Hughes]

On 05/14/2018 10:54 AM, interest-requ...@qt-project.org wrote:

On Saturday, 12 May 2018 07:11:57 PDT Roland Hughes wrote:

On 05/12/2018 08:57 AM, Thiago Macieira wrote:

I'm claiming that Qt is run 1% of the time in big-endian mode. Explain to
me why the 99% should perform the byte swap.

Because that class was created to communicate with the big-endian world
of real computers.

No, it was not. It was created to communicate with other Qt applications.
On that we will perpetually disagree. Us old people were actually around 
back then...



Do not change the default. Feel free to create a
different class,

I won't change the default, but mostly because I want people to stop using
QDataStream altogether and switch to CBOR. So I won't invest any time in
updating QDataStream so the problems you've raised in terms of compatibility
would be addressed. (yes, the  problems would be real)

Thank you.

--
Roland Hughes, President
Logikal Solutions
(630)-205-1593

http://www.theminimumyouneedtoknow.com
http://www.infiniteexposure.net
http://www.johnsmith-book.com
http://www.logikalblog.com
http://www.interestingauthors.com/blog
http://lesedi.us/
http://onedollarcontentstore.com

___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest

Re: [Interest] Interest Digest, Vol 80, Issue 9

[Roland Hughes]

On 05/14/2018 10:54 AM, Thiago Macieira wrote:

On Saturday, 12 May 2018 07:18:03 PDT Roland Hughes wrote:

On 05/12/2018 08:57 AM,interest-requ...@qt-project.org  wrote:

On Thursday, 10 May 2018 23:11:04 PDT alexander golks wrote:

i think it "silently" breaks here without setting error condition on the
QFile when hitting errors on all chunks but the first one, thus returning
a
size lower the input array size, which is the behaviour i see.

It's not an error to not write everything.

It most certainly should be. Either everything got written or there was
a failure, device buffer full, whatever.
The only reason to not write everything is some error happened.

But how would you report that some of it got written successfully? It may be
sufficient for the upper layer.

Imagine the case of disk full. The application may be able to free up space or
it could decide to wait until some other process did. But it needs to know how
much of its data it did write.

No, partial writes are success, so long as they report how much was
successful. This is the case as well for write(2) and WriteFile.
A partial read/write is a failure. You need to stop consuming the error, 
making it available via some method in the class.


=
FREAD(3) Linux Programmer's Manual FREAD(3)

NAME
   fread, fwrite - binary stream input/output

SYNOPSIS
   #include 

   size_t fread(void *ptr, size_t size, size_t nmemb, FILE *stream);

   size_t fwrite(const void *ptr, size_t size, size_t nmemb,
 FILE *stream);

DESCRIPTION
   The function fread() reads nmemb items of data, each size bytes 
long, from the stream pointed to by stream, storing them at the location 
given by ptr.


   The function fwrite() writes nmemb items of data, each size 
bytes long, to the stream pointed to by stream, obtaining them from the 
location given by ptr.


   For nonlocking counterparts, see unlocked_stdio(3).

RETURN VALUE
   On  success,  fread()  and fwrite() return the number of items 
read or written.  This number equals the number of bytes transferred 
only when size is 1.  If an error occurs, or the end of the file is

   reached, the return value is a short item count (or zero).

   fread() does not distinguish between end-of-file and error, and 
callers must use feof(3) and ferror(3) to determine which occurred.


=

Look at that last line. Both fread() and fwrite() set an error value. 
The application can retrieve this value when a partial read/write happens.


The original poster's issue is that Qt is consuming the error. They have 
no method of determining what the error was so the application can make 
an informed decision about how to proceed.


--
Roland Hughes, President
Logikal Solutions
(630)-205-1593

http://www.theminimumyouneedtoknow.com
http://www.infiniteexposure.net
http://www.johnsmith-book.com
http://www.logikalblog.com
http://www.interestingauthors.com/blog
http://lesedi.us/
http://onedollarcontentstore.com

___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest

Re: [Interest] QFile::write(const QByteArray&) does not, write, all data?

[Roland Hughes]

On 05/14/2018 10:54 AM, Richard Moore wrote:
On Sat, 12 May 2018 at 15:45, Roland Hughes 
 wrote:

Any time a write comes up short, it is an error.


?That is simply incorrect. It might not be how you would like it to behave,
however it is the norm that short writes are allowed and should be handled
by the callee. This is the case in POSIX, the C standard library etc. I'm
afraid, you'll simply have to learn to live with it - an easy solution
would be to write a tiny wrapper function of your own however if you do
you'll need to be very careful with your error handling if you always want
to be able to retry.

Your statement would be patently false.
=
FREAD(3) Linux Programmer's Manual FREAD(3)

NAME
   fread, fwrite - binary stream input/output

SYNOPSIS
   #include 

   size_t fread(void *ptr, size_t size, size_t nmemb, FILE *stream);

   size_t fwrite(const void *ptr, size_t size, size_t nmemb,
 FILE *stream);

DESCRIPTION
   The function fread() reads nmemb items of data, each size bytes 
long, from the stream pointed to by stream, storing them at the location 
given by ptr.


   The function fwrite() writes nmemb items of data, each size 
bytes long, to the stream pointed to by stream, obtaining them from the 
location given by ptr.


   For nonlocking counterparts, see unlocked_stdio(3).

RETURN VALUE
   On  success,  fread()  and fwrite() return the number of items 
read or written.  This number equals the number of bytes transferred 
only when size is 1.  If an error occurs, or the end of the file is

   reached, the return value is a short item count (or zero).

   fread() does not distinguish between end-of-file and error, and 
callers must use feof(3) and ferror(3) to determine which occurred.


=

Pay ___special___ attention to that last line, take straight from the 
man page of KDE linux.


The original poster's issue is that Qt is consuming the error. It is 
unreachable by the application so the application has no idea what to do.


--
Roland Hughes, President
Logikal Solutions
(630)-205-1593

http://www.theminimumyouneedtoknow.com
http://www.infiniteexposure.net
http://www.johnsmith-book.com
http://www.logikalblog.com
http://www.interestingauthors.com/blog
http://lesedi.us/
http://onedollarcontentstore.com

___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest

Re: [Interest] Interest Digest, Vol 80, Issue 8

[Thiago Macieira]

On Monday, 14 May 2018 08:14:24 PDT Jason H wrote:
> Is there going to be some kind of serialization operator/function for CBOR?

Yes.

https://codereview.qt-project.org/107465 - QCborStreamWriter
https://codereview.qt-project.org/107466 - QCborStreamReader
https://codereview.qt-project.org/214286 - QCborValue, QCborMap, QCborArray

The integration is stuck while we determine what the CPU endianness is when 
compiling with the INTEGRITY compiler.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center



___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest

Re: [Interest] QFile::write(const QByteArray&) does not write, all data?

[Thiago Macieira]

On Monday, 14 May 2018 00:32:38 PDT alexander golks wrote:
> > int QDataStream::writeRawData(const char *s, int len)
> > {
> > 
> > CHECK_STREAM_WRITE_PRECOND(-1)
> > int ret = dev->write(s, len);
> > if (ret != len)
> > q_status = WriteFailed;
> > return ret;
> > 
> > }
> 
> it does, partly. the write has not failed, it simply has not written all
> data.

It's a matter of perspective. As seen in this thread, in some situations 
failing to write everything is an unrecoverable situation. That is the case 
here: QDataStream cannot recover from a partial write.

We could loop retrying the write so long as ret > 0. This is probably 
acceptable here, since the alternative is to go into WriteFailed state anyway.

Would you submit an update?

> > > otherwise, as i still don't know the "no-error" conditions, i would
> > > always
> > > need to use something like QDataStream::writeRawData, which is barely
> > > more
> > > then a QFile::write, and no other operator<< methods.
> > 
> > it will tell you if there was a write failure. See above.
> 
> yes, but all other operator<< methods not.

The other methods do the same:

QDataStream ::operator<<(qint16 i)
{
CHECK_STREAM_WRITE_PRECOND(*this)
if (!noswap) {
i = qbswap(i);
}
if (dev->write((char *), sizeof(qint16)) != sizeof(qint16))
q_status = WriteFailed;
return *this;
}

Inserting the loop requires a bit of refactoring to centralise the writing to 
the device.

> what about something like:
> 
> int QDataStream::writeRawData(const char *s, int len)
> {
> CHECK_STREAM_WRITE_PRECOND(-1)
> qint64 bytesToWrite = len;
> qint64 totalWritten = 0;
> do {
> int ret = dev->write(s + totalWritten, bytesToWrite);
> if(ret < 0)
>   break;
> totalWritten += ret;
> bytesToWrite -= ret;
> } while (totalWritten < len);
> if(totalWritten != len)
> q_status = WriteFailed;
> return totalWritten;
> }
> 
> and appropriate modifications for all other related methods.

Please submit via the code review system.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center



___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest

Re: [Interest] Interest Digest, Vol 80, Issue 8

[Jason H]

> Sent: Saturday, May 12, 2018 at 1:06 PM
> From: "Thiago Macieira" 
> To: interest@qt-project.org
> Subject: Re: [Interest] Interest Digest, Vol 80, Issue 8
>
> On Saturday, 12 May 2018 07:11:57 PDT Roland Hughes wrote:
> > On 05/12/2018 08:57 AM, Thiago Macieira wrote:
> > > I'm claiming that Qt is run 1% of the time in big-endian mode. Explain to
> > > me why the 99% should perform the byte swap.
> > 
> > Because that class was created to communicate with the big-endian world
> > of real computers.
> 
> No, it was not. It was created to communicate with other Qt applications.
> 
> > Do not change the default. Feel free to create a
> > different class,
> 
> I won't change the default, but mostly because I want people to stop using 
> QDataStream altogether and switch to CBOR. So I won't invest any time in 
> updating QDataStream so the problems you've raised in terms of compatibility 
> would be addressed. (yes, the  problems would be real)


Is there going to be some kind of serialization operator/function for CBOR?
___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest

Re: [Interest] QFile::write(const QByteArray&) does not write, all data?

[Elvis Stansvik]

2018-05-14 1:29 GMT+02:00 Thiago Macieira :
> On Sunday, 13 May 2018 09:32:23 PDT Elvis Stansvik wrote:
>> QDataStream can operate
>> fine on an QIODevice that does partial writes (like QFile).
>
> No, it cannot. If there's a failure to write, the stream is corrupt and
> unrecoverable.

Yes of course. I only meant that this is an corruption that the user
can check for (status() == QDataStream::WriteFailed), so QDataStream
works as intended (for some definition of "works").

But yes, my "operate fine" was an exaggeration and a little tounge in
cheek, sorry about that.

I still think that bit of advise to subclassers in the docs is a
little too demanding, how about instead:

When reimplementing this function it is important that this function
_tries to write_
  all the data available before returning. This is required in order
for QDataStream
  to be able to operate on the class. QDataStream assumes all the information
  was written and therefore does not retry writing if there was a problem.

The emphasis shows what I changed.

Because a device obviously can't write all data in some circumstances
(e.g. out of disk space).

I think it's the way it is written is what confused the OP and some
others, because it almost makes it sound as if all current subclasses
(like QFile) _will_ write all data (heed the advice in that
paragraph).

Elvis

>
> --
> Thiago Macieira - thiago.macieira (AT) intel.com
>   Software Architect - Intel Open Source Technology Center
>
>
>
> ___
> Interest mailing list
> Interest@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/interest
___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest

Re: [Interest] QFile::write(const QByteArray&) does not write, all data?

[alexander golks]

Am Sun, 13 May 2018 23:50:20 -0700
schrieb Thiago Macieira :

> On Sunday, 13 May 2018 23:28:47 PDT alexander golks wrote:
> > > In fact, QDataStream can't recover from any errors.  
> > 
> > as you've said, my origin problem is no error.
> > wouldn't it be possible for QDataStream to handle this no-errors, too?  
> 
> It already does.
> 
> int QDataStream::writeRawData(const char *s, int len)
> {
> CHECK_STREAM_WRITE_PRECOND(-1)
> int ret = dev->write(s, len);
> if (ret != len)
> q_status = WriteFailed;
> return ret;
> }

it does, partly. the write has not failed, it simply has not written all data.
 
> > otherwise, as i still don't know the "no-error" conditions, i would always
> > need to use something like QDataStream::writeRawData, which is barely more
> > then a QFile::write, and no other operator<< methods.  
> 
> it will tell you if there was a write failure. See above.

yes, but all other operator<< methods not.

what about something like:

int QDataStream::writeRawData(const char *s, int len)
{
CHECK_STREAM_WRITE_PRECOND(-1)
qint64 bytesToWrite = len;
qint64 totalWritten = 0;
do {
int ret = dev->write(s + totalWritten, bytesToWrite);
if(ret < 0)
break;
totalWritten += ret;
bytesToWrite -= ret;
} while (totalWritten < len);
if(totalWritten != len)
q_status = WriteFailed;
return totalWritten;
}

and appropriate modifications for all other related methods.

-- 
/*
 *  All generalizations are false, including this one.
 *  -- Mark Twain
 */
___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest

Re: [Interest] QFile::write(const QByteArray&) does not write, all data?

[Thiago Macieira]

On Sunday, 13 May 2018 23:28:47 PDT alexander golks wrote:
> > In fact, QDataStream can't recover from any errors.
> 
> as you've said, my origin problem is no error.
> wouldn't it be possible for QDataStream to handle this no-errors, too?

It already does.

int QDataStream::writeRawData(const char *s, int len)
{
CHECK_STREAM_WRITE_PRECOND(-1)
int ret = dev->write(s, len);
if (ret != len)
q_status = WriteFailed;
return ret;
}

> otherwise, as i still don't know the "no-error" conditions, i would always
> need to use something like QDataStream::writeRawData, which is barely more
> then a QFile::write, and no other operator<< methods.

it will tell you if there was a write failure. See above.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center



___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest