Thank you everyone for your input. I have tried threads now and it seems to not work well as I have 6 threads apart from the main thread and each thread has a matplotlib graph window with a context menu that allows the user to select data to plot. Data is generally 1 million to 500k x 50 dict or sometimes 1 million x 50 dict. Each thread has a mutex lock with a QWaitCondition so it only runs when I call wakeAll(). Despite that, my app crashes anytime randomly and sometimes it never crashes for the same data and all the steps. Crashing varies depending on how many apps are open on my laptop at a given time and I can verify that it is taking too much of my memory when it is just about to crash using task manager. I also ensured that there are no variables or pointers that I am keeping undefined or any other reasons which can lead to the app crashing, and it seems to be my RAM limitations. My RAM is 8 GB which I thought should work but apparently not for 6 matplotlib graphs each with 500k x 50 data size. I hope using a pandas data frame doesn't result in a crash as I know that pandas are slower than a dict and I use that for some parts of my code. I am trying to get rid of that and just have a NumPy array. Each thread receives a copy of the big data from the main thread by means of a signal.
It seems that having all 6 plot windows as different processes would be better and so I guess I need to use the multiprocessing module... Though I don't know if I would be able to integrate all the plots (which are in different processes) still in the main GUI with dockWidgets, as I am currently doing. Your thoughts would be greatly appreciated, thanks in advance! On Fri, Oct 21, 2022 at 9:39 AM [email protected] < [email protected]> wrote: > I have used multiprocessing for reading data because the library that I > used was not thread safe so it couldn't be used inside a thread (they are > experimental data, not data written on a disk, so I needed an external > compiled library to access them). To share the read data between the > processes I used shared memory. I used multiprocessing RawArray given the > Python version I had. There may be newer alternatives, but if you want to > share big array between multiple process you need or shared memory or MPI > or some other thing. The standard Python way may be too slow. > > > Il giorno venerdì 14 ottobre 2022 alle 05:10:03 UTC+2 [email protected] > ha scritto: > >> Thank you all for your thoughts and responses. I think I will use >> QThreads (instead of threading.thread for Python) if I choose threads over >> multiprocessing. Now, after looking at some of the posts, I see that memory >> is not shared in multiprocessing, whereas threads have a shared memory. I >> have like 10 plots to be continuously plotting real time data, and that >> data would be read from a file that is being written by another Python >> script on the disk. Yet, whether or not I should switch to pyqtgraph is >> still a question as I was able to do real time plotting of a million full >> data points by threading with Tkinter when I last used matplotlib.And that >> is what I need now as well, however, using PyQt gives me advantage of >> possibly putting in QThreads to call out pyqtsignals which would be >> otherwise complex with just Python threads. Since I am rethinking my >> approach while using PyQt, I am also thinking whether I should use threads >> or not and just use multiprocessing (which is not something I am >> experienced with). I know Martin already answered about multiprocessing vs >> threading but I am still confused whether what I did earlier with Tkinter >> GUIs is correct or should I use multiprocessing now for PyQt GUIs. My >> software needs to get the data read continuously from a file on disk, and >> plot it continuously (real time) with 10 plot figures. I understand that >> multiprocessing is for heavy number crunching, but why would it not be a >> good candidate for data fetching from the disk (which could be one process) >> parallely to other 10 plot processes? My old approach consisted of data >> fetching as a thread and plotting as another thread different from the main >> thread. But, I didn't know multiprocessing capabilities then and that it >> can distribute the processing load to multiple cores. So, I am wondering if >> I can deploy multiprocessing for data fetching processes and data plotting >> processes, and there may be additional processes added later that have some >> mathematical calculations but not complicated ones (meaning scaling data >> vector or addition and subtraction of data vectors), OR if that is not >> correct use of multiprocessing and that I should stick with >> threading instead? >> Another reason why I am procrastinating to try one of the two, and >> research first, is because I am making a software that should run real time >> forever until interrupted, and that plots should be able to plot data of >> let's say 3 days ago where a day full data is like 10 million plot points >> (unlike some real time GUIs which would work as real time but for sometime >> only).Of Course I can read the data from disk if user needs to plot a 3 day >> or a week old data, but I don't know how plausible that is, as that comes >> with a overhead of read time for data, so I instead store data in a numpy >> array of length 10 million (beyond which software really struggles). At the >> same time, having all the week-long data in memory (using numpy or >> whatever) isn't possible or maybe also inefficient. Any thoughts on this? >> Your thoughts would be greatly appreciated. Thanks again! >> >> >> On Thu, Oct 13, 2022 at 1:38 PM Ognyan Moore <[email protected]> wrote: >> >>> I just want to add that the real advantage of QThreads (when talking >>> about pyqtgraph) is that you can do CPU-heavy processing in a non-GUI >>> thread, so the GUI doesn't (temporarily) freeze/lag as number crunching is >>> happening. I use a QThread to do some log-based calculations here, and I >>> use the Signal/Slot mechanism to relay the information I want pyqtgraph to >>> plot: >>> >>> >>> https://github.com/j9ac9k/barney/blob/main/src/barney/controllers/SubPlotControllers/SpectrogramController.py >>> >>> The calculation is based on an interactive element in the GUI, so when I >>> first had this setup, when the user would adjust a linear region item, the >>> GUI would temporarily freeze while I was doing a log calculation on a >>> ~500x2000 numpy array. Moving to a QThread meant the resulting plot was a >>> little late to update vs. the others, but the GUI didn't lag or freeze. >>> >>> Adding QThreads does add complexity, and I hope to roll an example in >>> the pyqtgraph example library on how to do this; but the general advice is >>> don't go down that route unless you have an actual need. >>> >>> On Thu, Oct 13, 2022 at 5:27 AM Edmondo Giovannozzi < >>> [email protected]> wrote: >>> >>>> Personally I want speed in a GUI application. So I always use >>>> pyqtgraph, even sometime modifying the pyqtgraph source in order to have >>>> what I need. It is well written, so modifying it is easy. >>>> If a calculation is slowing me down I can transform it in a Fortran or >>>> C code and use ctypes to link it to Python. >>>> >>>> Il Mer 12 Ott 2022, 23:36 Martin Chase <[email protected]> ha >>>> scritto: >>>> >>>>> Heya, >>>>> >>>>> There isn't any performance difference between QThreads and python >>>>> Threads; that's mostly a difference of how you want to integrate with Qt >>>>> (QThreads fit into the rest of the Qt lifecycle better). Otherwise, the >>>>> question that I ask to determine thread v. multiprocess is: will my worker >>>>> mostly be sitting around waiting? If that's the case (e.g. waiting on >>>>> i/o), >>>>> then using a thread is great. Otherwise, if the worker needs to do a bunch >>>>> of heavy work before the data is ready to display (e.g. it's the output of >>>>> a difficult analysis), put that work on a different core. >>>>> >>>>> Only matplotlib is as good at plotting as matplotlib; it's the best >>>>> python plotting library. If that's your minimum bar, I agree that you'll >>>>> be >>>>> writing everything yourself from scratch. PyQtGraph is built for >>>>> performance, though, so you're going to need the best of both libraries if >>>>> that's your use-case. >>>>> >>>>> Good luck, >>>>> - Martin >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "pyqtgraph" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/pyqtgraph/CAD_p8v3668bj82zYmWi%2Bns_DfHK4JHAbwvb8oSmitU-RkArFpA%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/pyqtgraph/CAD_p8v3668bj82zYmWi%2Bns_DfHK4JHAbwvb8oSmitU-RkArFpA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "pyqtgraph" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/pyqtgraph/CACwj3kUb7toyty5sAE4vP6mV3YsRxQ%2BjGMxRh-hecLFvy9w2DA%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/pyqtgraph/CACwj3kUb7toyty5sAE4vP6mV3YsRxQ%2BjGMxRh-hecLFvy9w2DA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "pyqtgraph" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/pyqtgraph/CA%2BnduTFi5wBSuqB9_g43HC_yYhwVSunEf7qz4Km4fjUMLnLbXw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/pyqtgraph/CA%2BnduTFi5wBSuqB9_g43HC_yYhwVSunEf7qz4Km4fjUMLnLbXw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "pyqtgraph" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pyqtgraph/e3b41dea-5854-4dc7-8174-ddf3b67d32den%40googlegroups.com > <https://groups.google.com/d/msgid/pyqtgraph/e3b41dea-5854-4dc7-8174-ddf3b67d32den%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "pyqtgraph" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/CAKhctBHPG%3Davs3R%3Dc43GSnyA5TOBXRZfzoXU9REf%3DeL1855k1g%40mail.gmail.com.
