Greetings everybody, I've tried to come up with this message for a couple of weeks now and it doesn't look like I'm getting any clearer in my thoughts so I decided that it's probably best to take the plunge and ask you guys to kindly throw me a rope...
What I'm trying to come up with is some kind of simple, dynamically scalable, domain-agnostic, multi-processor capable, tasking system. Maybe I'm already giving myself too many constraints here, i.e. it might all be possible except... the "simple" part. Clearly a good (the?) starting point is to think in terms of tasks, queues and worker threads. Once synchronization issues are properly attended this methodology feels, at least in theory, neat and intuitive. The problem is of course to put in practice. For example in my context I can think of tasks with very heterogeneous computing requirements, i.e. quick tasks such as UI interactions, slow tasks such as I/O with large files, and pretty much everything in- between. In this context, and given that I cannot know a priory a task's execution time, how can I create a system that is responsive on things like UI, takes its time on things such as I/O but gives the application the opportunity to further customize things? I.e. a user might want to limit the number of processors/core used to a subset of those available. Alternatively an application might want to create more threads than the optimal number simply because interleaving the executions of a few tasks is preferable to simply queuing them. And all this might have to happen at runtime. Currently on one end of the spectrum I'm considering a purely functional approach, with one queue for UI-related tasks, one for I/O- related tasks and one for everything else. On a single processor/core machine they'd simply be serviced by one thread each, all sharing one processor. On a two processors/cores machine UI and I/O queues might be serviced by two separate threads on the same processor while everything else is serviced by a worker thread on the other processor. As the number of cores/processors increases more threads/processor would be attached to each queue, but how to establish what should get more resources first? Alternatively I've read about queues with threads that "steal" from other queues. But that seems to assumes more or less homogeneous task execution or applications where responsiveness might drop to zero, i.e. because slow I/O tasks are being attended by all available threads/processors. In these cases, should the UI-related thread/queue never ever steal from the other queues, to maintain responsiveness at all times, effectively creating an hybrid approach with one purely functional thread/queue and (potentially) many other self-balancing threads/queues? Thoughts, hints, tips, tricks, directions, reading material of any kind will all be appreciated! =) Sincerely, Manu -- http://mail.python.org/mailman/listinfo/python-list