On Jan 5, 2014, at 12:20 AM, Guido van Rossum <[email protected]> wrote:
> On Sat, Jan 4, 2014 at 9:32 PM, Glyph <[email protected]> wrote: >> Endpoints are really very simple. They're two interfaces for listening and >> connecting; .listen(factory) for servers and .connect(factory) for clients. >> They each return a Deferred, which fires with an IListeningPort and a >> connected IProtocol instance, respectively. >> >> As the documentation puts it, "there's not a lot to it": >> <https://twistedmatrix.com/documents/current/core/howto/endpoints.html#auto2> > > After reading Tobias' description and some example code referenced in > those docs I agree, but these docs and your first paragraph above > couch everything in too-abstract terms. (Like those consulting > companies whose website claims "we help you be more effective." :-) If you have suggestions for those docs, I'd be happy to hear them. Believe it or not that documentation was what we came up with after several drafts that were more uselessly abstract; it's a bit hard to have a comprehensible, concrete introduction when the interface is basically "a method that, when you call it, does something arbitrary". Besides, endpoints *will* make you more effective! They promote synergy, and dynamic creativity, and serendipity and so on. >> They're cool because they decouple the part of the program which figures out >> what to listen on or connect to from the part of the program that figures >> out when to listen or connect. > > Basically, it's a mechanism that calls the right variant of > create_connection() or create_server() for you. > >> This allows you to transparently support running a protocol over an outbound >> TCP connection, outbound TLS connection, or something fancier, like an SSH >> command tunnel. > > Of course, you can already do that, just by having an if-statement > that chooses between different available ways to create connections. > What you really mean is that the mechanism is extensible so you don't > have to keep adding clauses to that if-statement in every program. Precisely. >> Similarly, since 'listen()' returns a Deferred, you can do >> asynchronous set-up when your program opens a new listening port (like, for >> example, executing firewall rules or manipulating a load balancer's >> configuration). > > This I don't quite understand. I suspect it addresses a specific > deficiency in the old way of setting up servers in Twisted? It's nothing particularly specific to Twisted, it just enables a very flexible way of setting up listening servers automatically. > When would > that specific Deferred fire in relationship to the actual sequence of > socket()/bind()/listen()/accept() syscalls? The basic TCP one, which does what create_server does, just fires synchronously (i.e. after "listen" returns). The one I'm talking about with the firewall/load-balancer stuff is more complex than that and would make a whole lot more syscalls than socket()/bind()/listen()/accept() ;-). But the idea is also pretty vague, so let me be specific: Let's say you have a service that is behind a TCP load balancer that has a JSON API for adding new back-end servers. You could have a LoadBalancedTCPEndpoint(loadBalancerURI, localPort) whose listen() method first listens locally, then makes the relevant API call to the load balancer to add the local host to the load balancer's pool, and only when that succeeds actually fire the returned Deferred. If that endpoint also had a plugin, then you could do 'twistd -n web --port loadbalanced:http\://loadbalancer/control-plane/add-port:8080' to add an existing server to the pool rather than write any new code. But of course you wouldn't want the server to register as "started" if the LB is down and won't recognize your new server, hence the Deferred. Does that explanation make sense? >> Also, there's the thing that Tobias was referring to, the ability to parse a >> string into an endpoint. This involves a plug-in interface so that third >> party packages can add new endpoint types, so you can run things over new >> types of transports without writing any new code at all (as long as >> applications use serverFromString or clientFromString to parse their >> command-line arguments, and an increasing number of the servers built in to >> Twisted do). > > It feels like the extensibility through a plug-in is the key feature > here, and the rest is just shaping the API to enable the > extensibility. Right? Yeah, although the API shape is basically the same kind of customizability as a library rather than as a plug-in, which can be just as important. >> The thing that wasn't ready was >> <https://twistedmatrix.com/trac/ticket/4859>, which is an example of how to >> implement the 'happy eyeballs' RFC, in terms of the endpoint API. It's done >> in the most recent Twisted release :-). > > Ah, I now know what this is. We don't quite have it in Tulip (we just > try the choices returned by getaddrinfo() in turn) but we should, and > I think we can do it transparently in create_connection(). I'm > tracking it in http://code.google.com/p/tulip/issues/detail?id=86, > feel free to add to it. I don't think I have lots to add; create_connection's signature is obvious enough that I think it's clear what to do :). >> As I recall, when I said they weren't quite ready, I was trying to suggest >> that tulip not even have an API like create_connection, and just have >> connect() that only took IP addresses, not hostnames, and put the >> name-resolution bits exclusively into the endpoint, but we hadn't proven >> that concept yet with Twisted. We still haven't quite, because the >> "low-level" API that HostnameEndpoint.connect() calls is still connectTCP(), >> which can in fact take hostnames (although it does something a lot less >> useful, TCP4 resolution only). > > Oh well. Tulip is pretty fixed now that it's in the Python 3.4 beta > release, but the PEP is provisional so we can change things for Python > 3.5, and of course we can certainly add new stuff then. Small tweaks > are still allowable before the first 3.4 release candidate (but this > seems to go beyond small). > >> Indeed, none of this is terribly complicated. It would be useful to put >> them into Tulip proper though, even if no functionality beyond what >> create_connection offers is available, so that application code doesn't use >> low-level loop interfaces like create_connection and create_server, since >> those aren't extensible. If applications habitually use this simple little >> bit of indirection, you can get the same sort of no-new-code flexibility for >> new transports that Twisted offers, at least at the library-code level. > > Unfortunately I think it's too late to add this for 3.4. But there > also are no other options yet besides create_connection() and > create_server(), so it's all mostly important for very forward-looking > code. Tulip doesn't even have UNIX domain sockets > (http://code.google.com/p/tulip/issues/detail?id=81). Most current > Tulip users seem to be creating frameworks themselves, so they can add > this at their end. :-) Twisted took like 9 years to add this feature so I'm pretty sure that asyncio will be just fine without it for 18 months ;-). -glyph
