this is a very interesting question, I would like to see some suggestions!!!
On Thu, Dec 30, 2010 at 4:37 AM, Knacktus <[email protected]> wrote: > Hello everyone, > > here the background story ;-): > ------------------------------ > > I'm writing an application which handles a lot of nested data. Imagine a > QTreeView with 10-20 levels and up to 50000 lines. > The standard use case would be to expand one level after the other in the > GUI. For each level I would load the data from the database. This can be > very slow, as I have a lot of network traffic and DB requests with little > amounts of data. > On the other hand, I am able to identify all the data of the tree in the > database. Therefore I can load all items to the client with one request. But > this is not always necessary. > My idea is to load all the data to the client (as bulk) on certain user > requests only, e.g. if the user clicks "expand all". In the meantime, the > user should be able to keep on working with the app. > > Here the question: > ------------------ > > I've read about multithreading and multiprocessing, but have no experience > with it yet. I've done a lot of MPI programming with Fortran, so > multiprocessing is more familiar to me. Right now I have these options in my > mind: > > 1) Multiprocessing: Loading the bulk data from the DB with a seperate > process and merge this data with the exisiting data in my main process. The > data is in a huge dict with the item ids as keys. So merging should not be a > problem. That sounds clean to me, but I'm concerned about the fact, that > multiprocessing is copying the whole app (with all the data already loaded > from the DB) in memory. > > 2) Multithreading: I guess trying to write to the same dict with my items > with two threads could be messy. Therefore I would create a temporary dict > in the secondary thread that loads the bulk from the DB and merge with the > main dict later. The question here would be the responsiveness of the GUI as > Python can't perform "real" multithreading. > > 3) Creating a Subprocess: I would create a helper app that loads the data > from the DB and saves it temporarly on the local drive as pickle. The main > app would read the data from the file and merge with the main dict. > > > What are your recommendations and experiences? Are there best practices? > > Any comments are welcome! > > Cheers, > > Jan > _______________________________________________ > PyQt mailing list [email protected] > http://www.riverbankcomputing.com/mailman/listinfo/pyqt > -- Lic. José M. Rodriguez Bacallao Centro de Biofisica Medica ----------------------------------------------------------------- Todos somos muy ignorantes, lo que ocurre es que no todos ignoramos lo mismo. Recuerda: El arca de Noe fue construida por aficionados, el titanic por profesionales ----------------------------------------------------------------- _______________________________________________ PyQt mailing list [email protected] http://www.riverbankcomputing.com/mailman/listinfo/pyqt
