Hello everyone,
here the background story ;-):
------------------------------
I'm writing an application which handles a lot of nested data. Imagine a
QTreeView with 10-20 levels and up to 50000 lines.
The standard use case would be to expand one level after the other in
the GUI. For each level I would load the data from the database. This
can be very slow, as I have a lot of network traffic and DB requests
with little amounts of data.
On the other hand, I am able to identify all the data of the tree in the
database. Therefore I can load all items to the client with one request.
But this is not always necessary.
My idea is to load all the data to the client (as bulk) on certain user
requests only, e.g. if the user clicks "expand all". In the meantime,
the user should be able to keep on working with the app.
Here the question:
------------------
I've read about multithreading and multiprocessing, but have no
experience with it yet. I've done a lot of MPI programming with Fortran,
so multiprocessing is more familiar to me. Right now I have these
options in my mind:
1) Multiprocessing: Loading the bulk data from the DB with a seperate
process and merge this data with the exisiting data in my main process.
The data is in a huge dict with the item ids as keys. So merging should
not be a problem. That sounds clean to me, but I'm concerned about the
fact, that multiprocessing is copying the whole app (with all the data
already loaded from the DB) in memory.
2) Multithreading: I guess trying to write to the same dict with my
items with two threads could be messy. Therefore I would create a
temporary dict in the secondary thread that loads the bulk from the DB
and merge with the main dict later. The question here would be the
responsiveness of the GUI as Python can't perform "real" multithreading.
3) Creating a Subprocess: I would create a helper app that loads the
data from the DB and saves it temporarly on the local drive as pickle.
The main app would read the data from the file and merge with the main dict.
What are your recommendations and experiences? Are there best practices?
Any comments are welcome!
Cheers,
Jan
_______________________________________________
PyQt mailing list [email protected]
http://www.riverbankcomputing.com/mailman/listinfo/pyqt
