On 28 Oct 2011, at 15:21, Daryl Herzmann wrote:
> Hi,
>
> I have an application that ingests data and does a reactor.spawnProcess()
> for each chunk of data (a product) to pass it as STDIN to a compiled binary
> and then I harvest the STDOUT. It has been working well, but I have an issue
> when
def parallel(iterable, count, callable, *args, **named):
#Copy/paste from http://jcalderone.livejournal.com/24285.html
#
http://oubiwann.blogspot.com/2008/06/async-batching-with-twisted-walkthrough.html
coop = task.Cooperator()
work = (callable(elem, *args, **named) for elem in iter
On Sat, Oct 29, 2011 at 2:10 AM, Nicolas Toper wrote:
> For a similar use case, we're using a combination of cooperator with
> defertoprocess.
>
> I can explain more if anyone's interested.
I would certainly be interested :) I'm having a difficult time
understanding how the pieces fit together.
For a similar use case, we're using a combination of cooperator with
defertoprocess.
I can explain more if anyone's interested.
Le 29 oct. 2011 00:41, "Donal McMullan" a écrit :
> I think twisted.internet.defer.DeferredSemaphore is designed for this. John
> Paul Calderone answers a similar quest
I think twisted.internet.defer.DeferredSemaphore is designed for this. John
Paul Calderone answers a similar question in detail:
http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-twisted-perspective-broker
Donal McMullan
On 29/10/2011, at 11:21 AM, Daryl Herzmann wrote:
Hi,
I have an application that ingests data and does a reactor.spawnProcess()
for each chunk of data (a product) to pass it as STDIN to a compiled binary
and then I harvest the STDOUT. It has been working well, but I have an issue
when my data rates get too high and the machine gets overloaded wit
