Re: [PySide] PySide Digest, Vol 83, Issue 13

Tue, 10 Dec 2019 07:17:49 -0800

Not really.
 
Maybe you need to flip your application in-side out? Make it a C++ program with all the threads it needs and just embed a python interpreter for your processing functions? Not sure how you'll efficiently move data between the two sides though. For OpenCV (c++), it's lucky that Qt and OpenCV support a few same memory formats so that you can just take the data as an opaque pointer.
 
As long as you want to code your processing of data in Python, you'll have the GIL, with will have this problem. C++ Qt can deal with it with no problem.
 
Maybe you could use some Async python (Python 3 only) and integrate Qt event loop with the python aync event loop and maybe do it that way? But I'm not a fan of Python async (Qt does it better, and the event loop integration isn't offically supported. Python's event loop  is nvery new and API isn't stable.)
 
Sent: Monday, December 09, 2019 at 6:14 PM
From: "Ognyan Moore" <ognyan.mo...@gmail.com>
To: "Jason H" <jh...@gmx.com>
Cc: "Israel Brewster" <ijbrews...@alaska.edu>, pyside@qt-project.org
Subject: Re: [PySide] PySide Digest, Vol 83, Issue 13
In my case, I'm trying to do very rapid spectrogram calculations, ....lots of them, they each take ~40-100ms but the fact that I do these calculations very rapidly makes it problematic for them occurring in the GUI thread.  These calculations are triggered by GUI events in pyqtgraph (user adjusts a plot window in some way); and I attempt to redraw a spectrogram calculation in as close to real-time as possible.  Now, I can make it so that the function running on a separate thread _never_ has to access some reference in the main thread.  Clearly there must be some way to spawn up a longish numpy task that does not effect the GUI.
 
 
 
On Mon, Dec 9, 2019 at 3:07 PM Jason H <jh...@gmx.com> wrote:
Before anyone points it out, I'm not suggesting Qt is the problem ;-) Just that if it is a data-transform-heavy task shuffeling between Qt and Python might be ok for display and let the server handle the processing. 
 
 
 
Sent: Monday, December 09, 2019 at 3:47 PM
From: "Israel Brewster" <ijbrews...@alaska.edu>
To: "Jason H" <jh...@gmx.com>
Cc: "Ognyan Moore" <ognyan.mo...@gmail.com>, pyside@qt-project.org
Subject: Re: [PySide] PySide Digest, Vol 83, Issue 13
On Dec 9, 2019, at 11:34 AM, Jason H <jh...@gmx.com> wrote:
 
"The Pandas data frames are pure Python classes, so they are not easy to construct from C/C++, but the underlying data storage of the individual columns (see class Series source) is numpy 1D array. Numpy has a nice C API and you can construct the numpy arrays from C and then pass it to your Python code." - https://stackoverflow.com/questions/11607387/is-there-a-c-c-api-for-python-pandas
 
 
Ugh, it's not pretty. But I bet you could wrap the C api or target the Python API directly.
 
If it were me though, having no idea what you are actually doing, I would problably write a json-RPC server in python and just relegate Qt to a pretty UI that makes json-RPC requests.
 
Interesting thought. Separate out the UI and the backend in completely separate processes that don’t interfere with each other. Definitely something to consider, although given that I need to graph the data, it might still result in a lot of data needing to be shuffled around.
 
As far as what I’m actually doing, it’s a data processing app that pulls in data from gas data collection flights, allows the user to select a bunch of different areas of interest, processes the data, and creates pretty graphs.
 
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory 
Geophysical Institute - UAF 
2156 Koyukuk Drive 
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145
 
The other me would just figure out a way to bypass pandas and work out how to do it entirely in C++ (Eigen?) except for the storage format. IIRC, Numpy arrays are just c arrays at the lowest level (don't quote me on that)
 
 
 
 
Sent: Monday, December 09, 2019 at 1:56 PM
From: "Israel Brewster" <ijbrews...@alaska.edu>
To: "Jason H" <jh...@gmx.com>
Cc: "Ognyan Moore" <ognyan.mo...@gmail.com>, pyside@qt-project.org
Subject: Re: [PySide] PySide Digest, Vol 83, Issue 13
On Dec 9, 2019, at 9:37 AM, Jason H <jh...@gmx.com> wrote:
 
That article does not take into account that the QThread logic is in python. That is the problem
The problem is the processing function is in Python. Moving it to be C++ only would work fine. Python would invoke a compiled function that would start the multithreaded processing and complete immediately, then the Qt event loop could be used via signals/slots to inform completion.
 
Exactly. If the function could be moved to be C++ only, there wouldn’t be an issue. If the function could be executed in a different process rather than a thread, there wouldn’t be an issue. Unfortunately, the sheer volume of data involved makes moving the data between python and anything else a non-starter - the function *has* to be able to modify pandas data frames in the calling class directly - unless there is some shortcut that could make it work based on the fact that the data is in pandas data frame, and at its core pandas/numpy *is* C++ already.
 
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory 
Geophysical Institute - UAF 
2156 Koyukuk Drive 
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145
 
 
 
 
Sent: Monday, December 09, 2019 at 1:05 PM
From: "Ognyan Moore" <ognyan.mo...@gmail.com>
To: pyside@qt-project.org
Subject: Re: [PySide] PySide Digest, Vol 83, Issue 13
I think it would be great if we can have some guidance regarding multi-threading with CPU heavy tasks, not just IO bound tasks.  Say I wanted to multi-thread a function
 
def cpu_intensive_function(a: int, b: int, c: int) -> np.ndarray:
    return some_numpy_function_calls(a, b, c)
 
How would I run this function in such a way that the computation does not hang the GUI thread?  I have attempted to follow the guidance on this blog post here:  https://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/ . Where I utilize QObject.moveToThread(some_thread) as that article suggests, the function still executes in the GUI thread.
 
Some caveats here being that I am making an attempt to have no references to mutable objects in the GUI thread and the thread doing the computation.
 
Thanks!
Ogi
 
 
On Mon, Dec 9, 2019 at 3:00 AM <pyside-requ...@qt-project.org> wrote:
Send PySide mailing list submissions to
        pyside@qt-project.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.qt-project.org/listinfo/pyside
or, via email, send a message with subject or body 'help' to
        pyside-requ...@qt-project.org

You can reach the person managing the list at
        pyside-ow...@qt-project.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of PySide digest..."


Today's Topics:

   1. Re: Keeping GUI responsive (Sivan Greenberg)


----------------------------------------------------------------------

Message: 1
Date: Sun, 8 Dec 2019 17:54:10 +0200
From: Sivan Greenberg <si...@omniqueue.com>
To: Israel Brewster <ijbrews...@alaska.edu>
Cc: pyside@qt-project.org
Subject: Re: [PySide] Keeping GUI responsive
Message-ID:
        <cajncf-av+xygpo6gv+kr-neifnedtiwg3gyylyg16njqjbr...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Israel,

 If your non main function is mostly IO bound, you could try implement your
processing function with an interface that allows for polling of results
using co-routines which is very slim and very robust (I used that in the
past for several PyQt & PySide small apps but also for a full blown project
for Ubuntu back in 2006[0] , though with PyGTK - but principle is same) and
will allow your GUI main loop to be in charge, even if you ever need to use
a spawned shell command or program! (see links below)

 It seems that nowadays, there are even ready made libs for that (this one
seems very interesting): https://github.com/reclosedev/async_gui and also
supports abstraction over multiprocessing which would support you even if
the the subtask is indeed CPU intensive (like data cleansing , validation
etc).

 If you're using python 3 , the whole coroutines interface has been greatly
simplified, and could be easier to use, however the first guide will
explain in deep and show how to achieve cooperative multitasking using
plain Python 2.x interface:
-
http://masnun.com/2015/11/13/python-generators-coroutines-native-coroutines-and-async-await.html
-
https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb

HTH,

-Sivan

[0]:
http://www.ubuntugeek.com/hubackup-backup-application-for-ubuntu-home-users.html

On Thu, Dec 5, 2019 at 1:41 AM Israel Brewster <ijbrews...@alaska.edu>
wrote:

> I know this is a FAQ, however I haven’t been able to make any of the
> standard answers work for me. Here’s the situation:
>
> - using PySide2 5.12.2
> - I have an object (QMainWindow subclass) that contains most of the code
> for my application
> - One of the functions that runs in response to user input takes around 2
> seconds to run. The GUI obviously freezes during this time (BAD!)
> - Said function needs to access and modify several large variables (pandas
> data frames) from the main object
>
> So here’s the problem: If I run this function as a separate (python)
> thread, that doesn’t help - the GUI is still frozen. I’m thinking this is
> due to the GIL, but I could be wrong about that. Running under the
> multiprocessing module, however, doesn’t appear to be an option due to the
> number and size of the data structures that the function needs to modify,
> and if I try just to see what happens, the process actually crashes.
>
> So what are my options here? How can I keep the GUI responsive while this
> function runs, without being able to spin it off as a separate process? Or
> is the only option going to be to completely rip apart the function and try
> to re-build it in such a way that it can, somehow, still access the memory
> from the main thread, while doing the processing in a separate function?
> ---
> Israel Brewster
> Software Engineer
> Alaska Volcano Observatory
> Geophysical Institute - UAF
> 2156 Koyukuk Drive
> Fairbanks AK 99775-7320
> Work: 907-474-5172
> cell:  907-328-9145
>
> _______________________________________________
> PySide mailing list
> PySide@qt-project.org
> https://lists.qt-project.org/listinfo/pyside
>


--
-Sivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/pyside/attachments/20191208/fc69df57/attachment-0001.html>

------------------------------

Subject: Digest Footer

_______________________________________________
PySide mailing list
PySide@qt-project.org
https://lists.qt-project.org/listinfo/pyside


------------------------------

End of PySide Digest, Vol 83, Issue 13
**************************************
_______________________________________________ PySide mailing list PySide@qt-project.org https://lists.qt-project.org/listinfo/pyside
_______________________________________________
PySide mailing list
PySide@qt-project.org
https://lists.qt-project.org/listinfo/pyside
_______________________________________________
PySide mailing list
PySide@qt-project.org
https://lists.qt-project.org/listinfo/pyside

Reply via email to