> On Oct 18, 2016, at 4:05 PM, Kevin Conway <kevinjacobcon...@gmail.com> wrote:
> > This is why twisted.python.context came to exist in the first place; I
> > always wanted to attach it to Deferred somehow
> Well, it's not something we've announced yet through any official channel,
> but we had to solve the context propagation problem at Atlassian to
> instrument our services with traceable logging. We open sourced our solution
> at https://bitbucket.org/hipchat/txlocal
> <https://bitbucket.org/hipchat/txlocal>. The answer for us was an extension
> for the reactor, thread pool, and inline callbacks that maintain the needed
> state. There's a readme with some insight into how we tool our services.
This is (A) very cool, and (B) making such aggressive use of private APIs that
it could win a contest about how to ensure that you break on every new release
of Twisted :). I'm super impressed that you tracked the introduction of
twisted._threads and support both old- and new-style thread pools!
> We've had it on our backlog to address the mailing list and, possibly even,
> discuss what it would take to put this into Twisted. I guess now is as good
> if a time as any. Feel free to spin off a another thread or reach out to me
> off list with any questions or feedback
This seems like as good a time to talk about it as any! Integrating this into
the core in some fashion would be good, but I imagine that it's a non-trivial
impact to performance, so it would be worthwhile to track that.
Speaking of performance - I found the long digression on "Don't Switch In The
Core" interesting, since we actually _do_ tracking this kind of context
already, for logging. I was a little surprised you didn't integrate with this
To be fair, this is something that our friends over at PyPy have been bugging
us about since forever; when you're benchmarking raw wire speed it does tend to
show up in profiling.
I also take it from the performance notes that you're not using PyPy?
__slots__ shouldn't make much of a difference there. (In fact I'm given to
believe it's a slight _decrease_ in performance on pypy...)
> On Tue, Oct 18, 2016, 14:47 Glyph Lefkowitz <gl...@twistedmatrix.com
> <mailto:gl...@twistedmatrix.com>> wrote:
>> On Oct 18, 2016, at 5:50 AM, Itamar Turner-Trauring <ita...@itamarst.org
>> <mailto:ita...@itamarst.org>> wrote:
>> Not been doing much Twisted lately, but have been doing async stuff
>> elsewhere, and I've learned some useful things.
> Thanks for writing these up, Itamar! This sort of reflection is rare and
> it's always helpful :).
>> 1. Callbacks should be sync or async, but never
>> sometimes-one-sometimes-the-other. For details go read
>> For example, Deferred.addCallback(f) really should never run f()
> This has come up a lot in a compare-and-contrast of Twisted vs. asyncio.
> I agree that the problems with synchronous callbacks are not insignificant
> (reentrancy is a degenerate form of preemption, and as we all know preemption
> is the corrupt wellspring of all bugs). However, the benefit, i.e.
> consistency of behavior with respect to reentrancy, comes with a cost: tight
> coupling to an event loop. In asyncio, Future's tight coupling to call_soon
> is a source of problems; it makes it hard to write a test without setting up
> an elaborate scheduling trampoline, whereas successResultOf/failureResultOf
> are quite simple to work with.
> I think Deferred as it is today is a pretty good compromise between the two
> positions. On the one hand it is decoupled from the event loop. On the
> other - and this is important - no Deferred-returning API will ever call your
> callbacks synchronously. Deferred.addCallback will, of course, but savvy
> Twisted programmers can (and should) do this, if they have dependent state
> d = doSomethingPotentiallySynchronous()
> As a caller, you can always decide whether you can safely be re-entered or
> not. In most cases, simply moving the 'addCallback' to the end of the
> function (a-la Go's "defer", oddly enough) is fine. In more complex cases
> where you really need to unwind reentrancy completely, you can do your own
> callLater(0) or callFromThread() from an object with a reference to a reactor.
> What happened to '2'? :)
>> By instrumenting all callbacks it manages, which may or may not
>> require item #1, Twisted can have a context that automatically follows
>> callbacks. Node has this and it is extremely useful.
>> is best summary I've found with a bit of searching.
> This was _always_ supposed to be the way that Twisted worked, but frankly I
> just wasn't smart enough to figure it out. This is why
> twisted.python.context came to exist in the first place; I always wanted to
> attach it to Deferred somehow. I will watch this talk intently; if #1 really
> is required to address this, my opinion might change. A PR would be
> intensely appreciated.
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com>
> Twisted-Python mailing list
Twisted-Python mailing list