On Feb 17, 2009, at 10:18 AM, Barak, Ron wrote:
I have a wxPython application that builds an internal database from
a list of files and then displays various aspects of that data,
in response to user's requests.
I want to add a module that finds events in a set of log files
(LogManager).
These log files are potentially huge, and the initial processing is
lengthy (several minutes).
Thus, when the user will choose LogManager, it would be unacceptable
to block the other parts of the program, and so - the initial
LogManager processing
would need to be done separately from the normal run of the program.
Once the initial processing is done, the main program would be
notified and could display the results of LogManager processing.
I was thinking of either using threads, or using separate processes,
for the main programs and LogManager.
What would you suggest I should consider in choosing between the two
options ?
Are there other options besides threads and multi-processing ?
Hi Ron,
The general rule is that it is a lot easier to share data between
threads than between processes. The multiprocessing library makes the
latter easier but is only part of the standard library in Python >=
2.6. The design of your application matters a lot. For instance, will
the processing code write its results to a database, ping the GUI code
and then exit, allowing the GUI to read the database? That sounds like
an excellent setup for processes.
In addition, there's the GIL to consider. Multi-process applications
aren't affected by it while multi-threaded applications may be. In
these days where multi-processor/multi-core machines are more common,
this fact is ever more important. Torrents of words have been written
about the GIL on this list and elsewhere and I have nothing useful to
add to the torrents. I encourage you to read some of those
conversations.
FWIW, when I was faced with a similar setup, I went with multiple
processes rather than threads.
Last but not least, since you asked about alternatives to threads and
multiprocessing, I'll point you to some low level libraries I wrote
for doing interprocess communication:
http://semanchuk.com/philip/posix_ipc/
http://semanchuk.com/philip/sysv_ipc/
Good luck
Philip
--
http://mail.python.org/mailman/listinfo/python-list
