In my case, I'm trying to do very rapid spectrogram calculations, ....lots of them, they each take ~40-100ms but the fact that I do these calculations very rapidly makes it problematic for them occurring in the GUI thread. These calculations are triggered by GUI events in pyqtgraph (user adjusts a plot window in some way); and I attempt to redraw a spectrogram calculation in as close to real-time as possible. Now, I can make it so that the function running on a separate thread _never_ has to access some reference in the main thread. Clearly there must be some way to spawn up a longish numpy task that does not effect the GUI.
On Mon, Dec 9, 2019 at 3:07 PM Jason H <jh...@gmx.com> wrote: > Before anyone points it out, I'm not suggesting Qt is the problem ;-) Just > that if it is a data-transform-heavy task shuffeling between Qt and Python > might be ok for display and let the server handle the processing. > > > > *Sent:* Monday, December 09, 2019 at 3:47 PM > *From:* "Israel Brewster" <ijbrews...@alaska.edu> > *To:* "Jason H" <jh...@gmx.com> > *Cc:* "Ognyan Moore" <ognyan.mo...@gmail.com>, pyside@qt-project.org > *Subject:* Re: [PySide] PySide Digest, Vol 83, Issue 13 > > On Dec 9, 2019, at 11:34 AM, Jason H <jh...@gmx.com> wrote: > > "The Pandas data frames are pure Python classes, so they are not easy to > construct from C/C++, but the underlying data storage of the individual > columns (see class Series source) > <https://github.com/pydata/pandas/blob/master/pandas/core/series.py#L88> is > numpy 1D array. Numpy has a nice C API > <http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html> and you can > construct the numpy arrays from C and then pass it to your Python code." - > https://stackoverflow.com/questions/11607387/is-there-a-c-c-api-for-python-pandas > > https://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html > > Ugh, it's not pretty. But I bet you could wrap the C api or target the > Python API directly. > > If it were me though, having no idea what you are actually doing, I would > problably write a json-RPC server in python and just relegate Qt to a > pretty UI that makes json-RPC requests. > > > Interesting thought. Separate out the UI and the backend in completely > separate processes that don’t interfere with each other. Definitely > something to consider, although given that I need to graph the data, it > might still result in a lot of data needing to be shuffled around. > > As far as what I’m actually doing, it’s a data processing app that pulls > in data from gas data collection flights, allows the user to select a bunch > of different areas of interest, processes the data, and creates pretty > graphs. > > --- > Israel Brewster > Software Engineer > Alaska Volcano Observatory > Geophysical Institute - UAF > 2156 Koyukuk Drive > Fairbanks AK 99775-7320 > Work: 907-474-5172 > cell: 907-328-9145 > > > The other me would just figure out a way to bypass pandas and work out how > to do it entirely in C++ (Eigen?) except for the storage format. IIRC, > Numpy arrays are just c arrays at the lowest level (don't quote me on that) > > > > > *Sent:* Monday, December 09, 2019 at 1:56 PM > *From:* "Israel Brewster" <ijbrews...@alaska.edu> > *To:* "Jason H" <jh...@gmx.com> > *Cc:* "Ognyan Moore" <ognyan.mo...@gmail.com>, pyside@qt-project.org > *Subject:* Re: [PySide] PySide Digest, Vol 83, Issue 13 > > On Dec 9, 2019, at 9:37 AM, Jason H <jh...@gmx.com> wrote: > > That article does not take into account that the QThread logic is in > python. That is the problem > The problem is the processing function is in Python. Moving it to be C++ > only would work fine. Python would invoke a compiled function that would > start the multithreaded processing and complete immediately, then the Qt > event loop could be used via signals/slots to inform completion. > > > Exactly. If the function could be moved to be C++ only, there wouldn’t be > an issue. If the function could be executed in a different process rather > than a thread, there wouldn’t be an issue. Unfortunately, the sheer volume > of data involved makes moving the data between python and anything else a > non-starter - the function *has* to be able to modify pandas data frames in > the calling class directly - unless there is some shortcut that could make > it work based on the fact that the data is in pandas data frame, and at its > core pandas/numpy *is* C++ already. > > --- > Israel Brewster > Software Engineer > Alaska Volcano Observatory > Geophysical Institute - UAF > 2156 Koyukuk Drive > Fairbanks AK 99775-7320 > Work: 907-474-5172 > cell: 907-328-9145 > > > > > > *Sent:* Monday, December 09, 2019 at 1:05 PM > *From:* "Ognyan Moore" <ognyan.mo...@gmail.com> > *To:* pyside@qt-project.org > *Subject:* Re: [PySide] PySide Digest, Vol 83, Issue 13 > I think it would be great if we can have some guidance regarding > multi-threading with CPU heavy tasks, not just IO bound tasks. Say I > wanted to multi-thread a function > > > def cpu_intensive_function(a: int, b: int, c: int) -> np.ndarray: > return some_numpy_function_calls(a, b, c) > > > How would I run this function in such a way that the computation does not > hang the GUI thread? I have attempted to follow the guidance on this blog > post here: > https://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/ > . > Where I utilize QObject.moveToThread(some_thread) as that article suggests, > the function still executes in the GUI thread. > > Some caveats here being that I am making an attempt to have no references > to mutable objects in the GUI thread and the thread doing the computation. > > Thanks! > Ogi > > > On Mon, Dec 9, 2019 at 3:00 AM <pyside-requ...@qt-project.org> wrote: > >> Send PySide mailing list submissions to >> pyside@qt-project.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://lists.qt-project.org/listinfo/pyside >> or, via email, send a message with subject or body 'help' to >> pyside-requ...@qt-project.org >> >> You can reach the person managing the list at >> pyside-ow...@qt-project.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of PySide digest..." >> >> >> Today's Topics: >> >> 1. Re: Keeping GUI responsive (Sivan Greenberg) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Sun, 8 Dec 2019 17:54:10 +0200 >> From: Sivan Greenberg <si...@omniqueue.com> >> To: Israel Brewster <ijbrews...@alaska.edu> >> Cc: pyside@qt-project.org >> Subject: Re: [PySide] Keeping GUI responsive >> Message-ID: >> < >> cajncf-av+xygpo6gv+kr-neifnedtiwg3gyylyg16njqjbr...@mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Hi Israel, >> >> If your non main function is mostly IO bound, you could try implement >> your >> processing function with an interface that allows for polling of results >> using co-routines which is very slim and very robust (I used that in the >> past for several PyQt & PySide small apps but also for a full blown >> project >> for Ubuntu back in 2006[0] , though with PyGTK - but principle is same) >> and >> will allow your GUI main loop to be in charge, even if you ever need to >> use >> a spawned shell command or program! (see links below) >> >> It seems that nowadays, there are even ready made libs for that (this one >> seems very interesting): https://github.com/reclosedev/async_gui and also >> supports abstraction over multiprocessing which would support you even if >> the the subtask is indeed CPU intensive (like data cleansing , validation >> etc). >> >> If you're using python 3 , the whole coroutines interface has been >> greatly >> simplified, and could be easier to use, however the first guide will >> explain in deep and show how to achieve cooperative multitasking using >> plain Python 2.x interface: >> - >> >> http://masnun.com/2015/11/13/python-generators-coroutines-native-coroutines-and-async-await.html >> - >> >> https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb >> >> HTH, >> >> -Sivan >> >> [0]: >> >> http://www.ubuntugeek.com/hubackup-backup-application-for-ubuntu-home-users.html >> >> On Thu, Dec 5, 2019 at 1:41 AM Israel Brewster <ijbrews...@alaska.edu> >> wrote: >> >> > I know this is a FAQ, however I haven’t been able to make any of the >> > standard answers work for me. Here’s the situation: >> > >> > - using PySide2 5.12.2 >> > - I have an object (QMainWindow subclass) that contains most of the code >> > for my application >> > - One of the functions that runs in response to user input takes around >> 2 >> > seconds to run. The GUI obviously freezes during this time (BAD!) >> > - Said function needs to access and modify several large variables >> (pandas >> > data frames) from the main object >> > >> > So here’s the problem: If I run this function as a separate (python) >> > thread, that doesn’t help - the GUI is still frozen. I’m thinking this >> is >> > due to the GIL, but I could be wrong about that. Running under the >> > multiprocessing module, however, doesn’t appear to be an option due to >> the >> > number and size of the data structures that the function needs to >> modify, >> > and if I try just to see what happens, the process actually crashes. >> > >> > So what are my options here? How can I keep the GUI responsive while >> this >> > function runs, without being able to spin it off as a separate process? >> Or >> > is the only option going to be to completely rip apart the function and >> try >> > to re-build it in such a way that it can, somehow, still access the >> memory >> > from the main thread, while doing the processing in a separate function? >> > --- >> > Israel Brewster >> > Software Engineer >> > Alaska Volcano Observatory >> > Geophysical Institute - UAF >> > 2156 Koyukuk Drive >> > Fairbanks AK 99775-7320 >> > Work: 907-474-5172 >> > cell: 907-328-9145 >> > >> > _______________________________________________ >> > PySide mailing list >> > PySide@qt-project.org >> > https://lists.qt-project.org/listinfo/pyside >> > >> >> >> -- >> -Sivan >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: < >> http://lists.qt-project.org/pipermail/pyside/attachments/20191208/fc69df57/attachment-0001.html >> > >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> PySide mailing list >> PySide@qt-project.org >> https://lists.qt-project.org/listinfo/pyside >> >> >> ------------------------------ >> >> End of PySide Digest, Vol 83, Issue 13 >> ************************************** > > _______________________________________________ PySide mailing list > PySide@qt-project.org https://lists.qt-project.org/listinfo/pyside > _______________________________________________ > PySide mailing list > PySide@qt-project.org > https://lists.qt-project.org/listinfo/pyside > >
_______________________________________________ PySide mailing list PySide@qt-project.org https://lists.qt-project.org/listinfo/pyside
