Re: [Interest] How to properly show progress for a signal processing pipeline

[alexander golks]

Am Mon, 8 Apr 2019 16:02:10 +
schrieb "Murphy, Sean" :

> > I would just show one progress set to 3 * raw count.  Each stage
> > periodically signals to the ui thread its progress, and whether it has
> > processed the end-of-data signal from the previous stage.  In the ui
> > thread slot:
> > 
> > int totalCount, rawCount;
> > 
> > QVector stageProgress;
> > 
> > void stageProgressSlot(int stageIndex, int count, bool completed) {
> > 
> >      Q_ASSERT(count <= rawCount);
> > 
> >      auto thisCount = (if completed ? rawCount : count);
> > 
> >      Q_ASSERT(thisCount >= stageProgress[stageIndex]);
> > 
> >      totalCount += thisCount - stageProgress[stageIndex];
> > 
> >      stageProgress[stageIndex] = thisCount;
> > 
> >      ui->progress->setValue(totalCount);
> > 
> > }
> > 
> > It will always appear to start slowly then jump forward depending on the
> > degree of filtering.  I can't see a way to avoid that other than Bill's
> > idea.
> > 
> > Hope that helps,  Tony  
> 
> I think it does, thank you. I can't really see any other way to do it to 
> provide a somewhat meaningful progress bar - I'd like to avoid the
> "busy" progress bar if possible since that doesn't offer the user any sort
> of sense of how much longer they have to wait and your idea at least
> gives them some sense of status, even if it's not 100% accurate at all times.
> 
> Sean

what about 2 progress bars? one for the count of filter processes, and the 
other for the steps at each process?
i use this when loading several files (your filter processes) and display the 
current line in a file (your steps).

-- 
/*
 *printk(KERN_ERR "Danger Will Robinson: failed to re-trigger IRQ%d\n", irq);
 *linux-2.6.6/arch/arm/common/sa.c
 */


pgpK1HwGeq1gQ.pgp
Description: Digitale Signatur von OpenPGP
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Parsing data from serialport

[Martin Marmsoler]

What is, when I ignore the ready read? Has the internal buffer a max size?
Because at the moment I have a memory leak and maybe it comes from that?

Martin

Am Fr., 5. Apr. 2019 um 21:12 Uhr schrieb Konstantin Shegunov <
kshegu...@gmail.com>:

> On Thu, Apr 4, 2019 at 6:42 PM Jérôme Godbout  wrote:
>
>> You jus did the same loop into your slot, the function I did made can be
>> called as slot to the connect you just told, I agree the
>> serial_port->waitForReadyRead(5); is not necessary, just often the
>> transmission have partial or bunch of data by small burst (USB VCOM) and
>> was handy for me, but it is not ncessary in any way.
>>
> A similar loop yes, however your loop goes until there's something to read
> on the port, mine is run a few times only when the port already has some
> information pending in its buffer. The QSerialPort class already knows if
> data has arrived and buffers it*, so what would be the point to loop over
> the port when there's no data? When you talk with someone do you ask every
> 5ms if [s]he has something to say to you? I really doubt it. You'd normally
> just wait for something to be said to decide if and how to respond to it.
>
>> Side note: ReadLines made it less general purpose, you will need a
>> protocol that use endline to signal end of message which is not always the
>> case.
>>
> Involving realistic protocol handling is another kettle of fish. It would
> require much more thought and effort and is beyond the scope of what was
> asked initially.
>
> * Unless it's in unbuffered mode, as Thiago already noted.
>
>> ___
> Interest mailing list
> Interest@qt-project.org
> https://lists.qt-project.org/listinfo/interest
>
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] How to properly show progress for a signal processing pipeline

[Konstantin Tokarev]

  08.04.2019, 16:16, "Murphy, Sean" :> We have a data processing pipeline that looks like so:> Raw Data objects -> Type "A" data objects -> Type "B" data objects -> Final Data objects>> Each step can be considered a filtering process where the following mathematical relationship holds for the quantities of each type of object:> Raw count >= A count >= B count >= Final count> Technically all of those quantity relationships can be >=, but in practice they almost never equal, the left hand side is almost always greater than right hand side, but it's technically possible that some could be equal with the right input data.>>  So the issue I'm having here is that I want to show the user some sort of progress indicator (this whole pipeline takes a bit of time), but at the start I only know the raw count, not any of the intermediate or final counts. And because the A->B->Final portion of the pipeline takes a noticeable amount of time, I can't just run the progress bar from 0 to Raw count, because then from the user's standpoint the progress bar would reach 100% once the number of raw samples is exhausted, not when processing is fully complete.>> Is my only strategy to use the "undetermined state" of QProgress bar to just show "something" is happening, but not how far along we are in the process? To make the issue even a little more complicated, each processing node is running in a separate thread, so these conversions are happening in parallel as much as possible, making tracking a little more difficult. For example, at any given point in time, the node that converts type A objects to B objects, might be empty. It may have processed all of the A objects it has received into all the B objects it can make and passed them on to the next node, while the node ahead of it might not have finished its work on the next chunk of raw data to emit the next A object. So at that instant in time, the A->B node thinks it's "done" until it receives the next A object at some time later. Assuming that the main chunk of work is done on A->B step, you can increment progress bar when each thread is started and finished, plus fixed amounts of progress for the first and the last step. Approximate thread number as 2*Raw when it starts, and when amount is known, adjust progress bar settings accordingly.  -- Regards,Konstantin ___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] How to properly show progress for a signal processing pipeline

[Murphy, Sean]

> I would just show one progress set to 3 * raw count.  Each stage
> periodically signals to the ui thread its progress, and whether it has
> processed the end-of-data signal from the previous stage.  In the ui
> thread slot:
> 
> int totalCount, rawCount;
> 
> QVector stageProgress;
> 
> void stageProgressSlot(int stageIndex, int count, bool completed) {
> 
>      Q_ASSERT(count <= rawCount);
> 
>      auto thisCount = (if completed ? rawCount : count);
> 
>      Q_ASSERT(thisCount >= stageProgress[stageIndex]);
> 
>      totalCount += thisCount - stageProgress[stageIndex];
> 
>      stageProgress[stageIndex] = thisCount;
> 
>      ui->progress->setValue(totalCount);
> 
> }
> 
> It will always appear to start slowly then jump forward depending on the
> degree of filtering.  I can't see a way to avoid that other than Bill's
> idea.
> 
> Hope that helps,  Tony

I think it does, thank you. I can't really see any other way to do it to 
provide a somewhat meaningful progress bar - I'd like to avoid the
"busy" progress bar if possible since that doesn't offer the user any sort
of sense of how much longer they have to wait and your idea at least
gives them some sense of status, even if it's not 100% accurate at all times.

Sean


This message has been scanned for malware by Forcepoint. www.forcepoint.com
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] How to properly show progress for a signal processing pipeline

[Igor Mironchik]

Hi,

Have a look at Qt Maintenance Tool download meta information progress bar...

On 08.04.2019 18:04, Tony Rietwyk wrote:

Hi Sean,

I would just show one progress set to 3 * raw count.  Each stage 
periodically signals to the ui thread its progress, and whether it has 
processed the end-of-data signal from the previous stage.  In the ui 
thread slot:


int totalCount, rawCount;

QVector stageProgress;

void stageProgressSlot(int stageIndex, int count, bool completed) {

    Q_ASSERT(count <= rawCount);

    auto thisCount = (if completed ? rawCount : count);

    Q_ASSERT(thisCount >= stageProgress[stageIndex]);

    totalCount += thisCount - stageProgress[stageIndex];

    stageProgress[stageIndex] = thisCount;

    ui->progress->setValue(totalCount);

}

It will always appear to start slowly then jump forward depending on 
the degree of filtering.  I can't see a way to avoid that other than 
Bill's idea.


Hope that helps,  Tony


On 9/04/2019 12:16 am, william.croc...@analog.com wrote:




  So the issue I'm having here is that I want to show the user some 
sort of progress indicator (this whole pipeline takes a bit of 
time), but at the start I only know the raw count, not any of the 
intermediate or final counts. And because the A->B->Final portion of 
the pipeline takes a noticeable amount of time, I can't just run the 
progress bar from 0 to Raw count, because then from the user's 
standpoint the progress bar would reach 100% once the number of raw 
samples is exhausted, not when processing is fully complete.




You could show three progress bars, one for each stage.
Assume each item will pass through each stage (which is pessimistic)
and show progress based on that. In most cases the whole
process will finish early based on the bars, but progress
will be shown.

Bill
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] How to properly show progress for a signal processing pipeline

[Tony Rietwyk]

Hi Sean,

I would just show one progress set to 3 * raw count.  Each stage 
periodically signals to the ui thread its progress, and whether it has 
processed the end-of-data signal from the previous stage.  In the ui 
thread slot:


int totalCount, rawCount;

QVector stageProgress;

void stageProgressSlot(int stageIndex, int count, bool completed) {

    Q_ASSERT(count <= rawCount);

    auto thisCount = (if completed ? rawCount : count);

    Q_ASSERT(thisCount >= stageProgress[stageIndex]);

    totalCount += thisCount - stageProgress[stageIndex];

    stageProgress[stageIndex] = thisCount;

    ui->progress->setValue(totalCount);

}

It will always appear to start slowly then jump forward depending on the 
degree of filtering.  I can't see a way to avoid that other than Bill's 
idea.


Hope that helps,  Tony


On 9/04/2019 12:16 am, william.croc...@analog.com wrote:




  So the issue I'm having here is that I want to show the user some 
sort of progress indicator (this whole pipeline takes a bit of time), 
but at the start I only know the raw count, not any of the 
intermediate or final counts. And because the A->B->Final portion of 
the pipeline takes a noticeable amount of time, I can't just run the 
progress bar from 0 to Raw count, because then from the user's 
standpoint the progress bar would reach 100% once the number of raw 
samples is exhausted, not when processing is fully complete.




You could show three progress bars, one for each stage.
Assume each item will pass through each stage (which is pessimistic)
and show progress based on that. In most cases the whole
process will finish early based on the bars, but progress
will be shown.

Bill
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] How to properly show progress for a signal processing pipeline

[william.croc...@analog.com]

  So the issue I'm having here is that I want to show the user some sort of progress 
indicator (this whole pipeline takes a bit of time), but at the start I only know the 
raw count, not any of the intermediate or final counts. And because the 
A->B->Final portion of the pipeline takes a noticeable amount of time, I can't 
just run the progress bar from 0 to Raw count, because then from the user's 
standpoint the progress bar would reach 100% once the number of raw samples is 
exhausted, not when processing is fully complete.



You could show three progress bars, one for each stage.
Assume each item will pass through each stage (which is pessimistic)
and show progress based on that. In most cases the whole
process will finish early based on the bars, but progress
will be shown.

Bill
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] How to properly show progress for a signal processing pipeline

[william.croc...@analog.com]

We have a data processing pipeline that looks like so:
 Raw Data objects -> Type "A" data objects -> Type "B" data objects -> 
Final Data objects

Each step can be considered a filtering process where the following 
mathematical relationship holds for the quantities of each type of object:
 Raw count >= A count >= B count >= Final count
Technically all of those quantity relationships can be >=, but in practice they 
almost never equal, the left hand side is almost always greater than right hand 
side, but it's technically possible that some could be equal with the right input 
data.

  So the issue I'm having here is that I want to show the user some sort of 
progress indicator (this whole pipeline takes a bit of time), but at the start 
I only know the raw count, not any of the intermediate or final counts.


You can do what Microsoft does when copying files from one place to another, for 
example...

Take an unreasonable amount of additional time to study the problem
for the sole purposes of presenting an approximate progress bar.

:-)

Bill
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

[Interest] How to properly show progress for a signal processing pipeline

[Murphy, Sean]

We have a data processing pipeline that looks like so:
Raw Data objects -> Type "A" data objects -> Type "B" data objects -> Final 
Data objects

Each step can be considered a filtering process where the following 
mathematical relationship holds for the quantities of each type of object:
Raw count >= A count >= B count >= Final count
Technically all of those quantity relationships can be >=, but in practice they 
almost never equal, the left hand side is almost always greater than right hand 
side, but it's technically possible that some could be equal with the right 
input data.

 So the issue I'm having here is that I want to show the user some sort of 
progress indicator (this whole pipeline takes a bit of time), but at the start 
I only know the raw count, not any of the intermediate or final counts. And 
because the A->B->Final portion of the pipeline takes a noticeable amount of 
time, I can't just run the progress bar from 0 to Raw count, because then from 
the user's standpoint the progress bar would reach 100% once the number of raw 
samples is exhausted, not when processing is fully complete.

Is my only strategy to use the "undetermined state" of QProgress bar to just 
show "something" is happening, but not how far along we are in the process? To 
make the issue even a little more complicated, each processing node is running 
in a separate thread, so these conversions are happening in parallel as much as 
possible, making tracking a little more difficult. For example, at any given 
point in time, the node that converts type A objects to B objects, might be 
empty. It may have processed all of the A objects it has received into all the 
B objects it can make and passed them on to the next node, while the node ahead 
of it might not have finished its work on the next chunk of raw data to emit 
the next A object. So at that instant in time, the A->B node thinks it's "done" 
until it receives the next A object at some time later.

Sean


This message has been scanned for malware by Forcepoint. www.forcepoint.com
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] QWebEngine, custom URL schemes and content-types

[Steve Atkins]

> On Apr 8, 2019, at 12:46 PM, Andy  wrote:
> 
> Not sure if it's the same situation, but I just had a similar problem - SVGs 
> weren't showing up properly because I had the wrong MIME type.
> 
> I fixed it by using the QMimeDatabase like this:
> 
>   const auto  cMIMEType = QMimeDatabase().mimeTypeForData( data );
> 
>   inRequest->reply( cMIMEType.name().toLatin1(), buffer );
> 
> (where "data" is the contents of a file)
> 

Thanks! That's pretty much what I'm doing. Knowing it seems to be working for
someone is a very useful data point.

Cheers,
  Steve


> ---
> Andy Maloney  //  https://asmaloney.com
> twitter ~ @asmaloney
> 
> 
> 
> On Mon, Apr 8, 2019 at 7:40 AM Steve Atkins  wrote:
> I'm using a custom URL scheme by registering the scheme with 
> QEbEngineUrlScheme::registerScheme() at app startup, creating a handler 
> ("Assets")  that inherits from QWebEngineUrlSchemeHandler, installing that 
> scheme handler in the QWebEngineProfile passed to the QWebPage.
> 
> Assets::requestStarted(QWebEngineUrlRequestJob *job) handles the request and 
> finishes by doing job->reply() with a content type of application/javascript 
> and an open QFile.
> 
> Everything works fine. The page loads, it loads scripts from the header from 
> URLs asset:whatever.js, the content is correct and the js runs fine.
> 
> Until I try and load the asset: scripts as ES6 modules. Then I get "Error: 
> Failed to load module script: The server responded with a non-JavaScript MIME 
> type of \"\". Strict MIME type checking is enforced for module scripts per 
> HTML spec. (asset:/ticket.js:0)".
> 
> It seems from that that job->reply() is not setting the content-type of the 
> response successfully. I've tried to confirm that, but the chrome inspector 
> shows no response headers, and fetch() errors out because the custom URL 
> scheme isn't supported.
> 
> Does this look familiar to anyone? And has anyone used custom schemes with 
> content-types successfully?
> 
> Cheers,
>  Steve
> 
> ___
> Interest mailing list
> Interest@qt-project.org
> https://lists.qt-project.org/listinfo/interest

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] QWebEngine, custom URL schemes and content-types

[Andy]

Not sure if it's the same situation, but I just had a similar problem -
SVGs weren't showing up properly because I had the wrong MIME type.

I fixed it by using the QMimeDatabase like this:

  const auto  cMIMEType = QMimeDatabase().mimeTypeForData( data );

  inRequest->reply( cMIMEType.name().toLatin1(), buffer );

(where "data" is the contents of a file)

---
Andy Maloney  //  https://asmaloney.com
twitter ~ @asmaloney 



On Mon, Apr 8, 2019 at 7:40 AM Steve Atkins  wrote:

> I'm using a custom URL scheme by registering the scheme with
> QEbEngineUrlScheme::registerScheme() at app startup, creating a handler
> ("Assets")  that inherits from QWebEngineUrlSchemeHandler, installing that
> scheme handler in the QWebEngineProfile passed to the QWebPage.
>
> Assets::requestStarted(QWebEngineUrlRequestJob *job) handles the request
> and finishes by doing job->reply() with a content type of
> application/javascript and an open QFile.
>
> Everything works fine. The page loads, it loads scripts from the header
> from URLs asset:whatever.js, the content is correct and the js runs fine.
>
> Until I try and load the asset: scripts as ES6 modules. Then I get "Error:
> Failed to load module script: The server responded with a non-JavaScript
> MIME type of \"\". Strict MIME type checking is enforced for module scripts
> per HTML spec. (asset:/ticket.js:0)".
>
> It seems from that that job->reply() is not setting the content-type of
> the response successfully. I've tried to confirm that, but the chrome
> inspector shows no response headers, and fetch() errors out because the
> custom URL scheme isn't supported.
>
> Does this look familiar to anyone? And has anyone used custom schemes with
> content-types successfully?
>
> Cheers,
>  Steve
>
> ___
> Interest mailing list
> Interest@qt-project.org
> https://lists.qt-project.org/listinfo/interest
>
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

[Interest] QWebEngine, custom URL schemes and content-types

[Steve Atkins]

I'm using a custom URL scheme by registering the scheme with 
QEbEngineUrlScheme::registerScheme() at app startup, creating a handler 
("Assets")  that inherits from QWebEngineUrlSchemeHandler, installing that 
scheme handler in the QWebEngineProfile passed to the QWebPage.

Assets::requestStarted(QWebEngineUrlRequestJob *job) handles the request and 
finishes by doing job->reply() with a content type of application/javascript 
and an open QFile.

Everything works fine. The page loads, it loads scripts from the header from 
URLs asset:whatever.js, the content is correct and the js runs fine.

Until I try and load the asset: scripts as ES6 modules. Then I get "Error: 
Failed to load module script: The server responded with a non-JavaScript MIME 
type of \"\". Strict MIME type checking is enforced for module scripts per HTML 
spec. (asset:/ticket.js:0)".

It seems from that that job->reply() is not setting the content-type of the 
response successfully. I've tried to confirm that, but the chrome inspector 
shows no response headers, and fetch() errors out because the custom URL scheme 
isn't supported.

Does this look familiar to anyone? And has anyone used custom schemes with 
content-types successfully?

Cheers,
 Steve

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest