Thank you for all for all the answers so far, particularly to Ilya and
Jean-Paul who provided some very helpful code samples.
It's interesting to realise that, by avoiding locking, we can end up
with a much more efficient implementation. I'll have to figure out how
widely we can apply this technique - and how often it's going to be
worth rewriting things to allow that. Thanks for some useful pointers!
On 12/03/18 20:00, Jean-Paul Calderone wrote:
On Mon, Mar 12, 2018 at 3:52 PM, Ilya Skriblovsky
<ilyaskriblov...@gmail.com <mailto:ilyaskriblov...@gmail.com>> wrote:
I've used class like this to cache the result of Expensive
pending = None
result = None
failure = None
def __init__(self, expensive_func):
self.expensive_func = expensive_func
if self.pending is None:
self.result = result
self.failure = failure
This seems like basically a correct answer to me. However, I suggest
one small change.
You probably want to create and return a new Deferred for each
result. If you don't, then your internal `pending` Deferred is now
reachable by application code.
As written, an application might (very, very reasonably):
d = getResource()
Now `pending` has `long_async_operation` as a callback on its chain.
This will prevent anyone else from getting a result until
`long_async_operation` is done.
You can fix this by:
result = Deferred()
Now the application can only reach `result`. Nothing they do to
`result` will make much difference to `pending` because
`chainDeferred` only puts `callback` (and `errback`) onto `pending`'s
callback chain. `callback` and `errback` don't wait on anything.
You have to be a little careful with `chainDeferred` because it
doesn't have the recursion-avoidance logic that implicit chaining
has. However, that doesn't matter in this particular case because the
chain depth is fixed at two (`pending` and `result`). The problems
only arise if you extend the chain out in this direction without bound.
def _return_result(self, _):
return self.failure or self.result
Using it you can get rid of DeferredLocks:
deferred_cache = DeferredCache(do_expensive_calculation)
It will start `expensive_func` on the first call. The second and
consequtive calls will return deferreds that resolves with the
result when expensive_func is done. If you call it when result is
already here, it will return alread-fired deferred.
Of course, it will require some more work if you need to pass
arguments to `expensive_func` and memoize results per arguments
Twisted-Python mailing list