> On Dec 1, 2017, at 1:30 AM, Chris Lattner <clatt...@nondot.org> wrote: > >> On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgre...@apple.com >> <mailto:dgre...@apple.com>> wrote: >>>> Philosophy > >>> More problematically for your argument: your preferred approach requires >>> the introduction of (something like) DynamicMemberLookupProtocol or >>> something like AnyObject-For-Python, so your proposal would be additive on >>> top of solving the core problem I’m trying to solve. It isn’t an >>> alternative approach at all. >> >> I wouldn’t say that’s my preferred approach. My preferred approach involves >> taking the method/property/etc. declarations that already exist in Python >> and mapping them into corresponding Swift declarations so we have something >> to find with name lookup. One could put all of these declarations on some >> PyVal struct or PythonObject and there would be no need for >> AnyObject-for-Python or DynamicMemberLookupProtocol. > > You’re suggesting that the transitive closure of all Python methods and > properties be preprocessed into a single gigantic Swift PyVal type? I guess > something like that could be done.
That’s effectively what Swift is already doing with AnyObject. > I would be concerned because there are many N^2 or worse algorithms in the > Swift compiler would probably explode. As noted above, we already do this for AnyObject with every @objc entity everywhere. The # of overloads for a given name is usually not so high that it’s a problem for the compiler. > It also doesn’t provide the great tooling experience that you’re seeking, > given that code completion would show everything in the Python universe, > which is not helpful. It’s better for code completion to provide too much than to provide nothing at all. If code completion provides too much, typing a small number of characters will reduce the completion set down to something manageable quite fast. > Further, it doesn’t provide a *better* experience than what I’m suggesting, > it seems strictly worse. >From my prior message, having the declarations of all Python methods and >properties available on PyVal makes Swift tooling work: For the same working Swift code dog.add_trick(rollOver) that calls into Python, the solution I’m proposing: * Gave you the add_trick(<#trick#>) code completion with the number of parameters and their names * Supports goto definition to jump to the wrapper method on PyVal * Supports “quick help” to show the declaration of that wrapper method on PyVal, showing the documentation (docstring) and in which Python class it was declared. * Supports indexing functionality so we can find the likely uses of “add_trick" DynamicMemberLookup doesn’t gave any of those, because “add_trick” is just a string that looks like an identifier. Only the Python runtime can resolve. > A preprocessing step prevents users from playfully importing random Python > modules into the Swift repl and playgrounds. *At worst*, you invoke the tool from the command line to build the Swift module that corresponds to a given Python module. py2swift <pythonmodulename> We could absolutely introduce tooling hooks to make the compiler initiate that step. > It also seems worse for implementors (who will get a stream of new bugs > about compiler scalability). To my knowledge, AnyObject lookup has not been a serious source of performance problems for the compiler. The global lookup table it requires was annoying to implement, but it’s just a symbol table. > >>>> Whenever we discuss adding more dynamic features to Swift, there’s a >>>> strong focus on maintaining that strong static type system. >>> >>> Which this does, by providing full type safety - unlike AnyObject lookup. >> >> You get dynamic safety because it goes into the Python interpreter; fair >> enough. You get no help from your tools to form a correct invocation of any >> method provided by Python. > > Sure, that’s status quo for Python APIs. That’s not the status quo for Python IDEs. They will give you the basic shape of method calls. > >>>> Tooling >>>> The Python interoperability enabled by this proposal *does* look fairly >>>> nice when you look at a small, correctly-written example. However, >>>> absolutely none of the tooling assistance we rely on when writing such >>>> code will work for Python interoperability. Examples: >>>> >>>> * As noted earlier, if you typo’d a name of a Python entity or passed the >>>> wrong number of arguments to it, the compiler will not tell you: it’ll be >>>> a runtime failure in the Python interpreter. I guess that’s what you’d get >>>> if you were writing the code in Python, but Swift is supposed to be >>>> *better* than Python if we’re to convince a community to use Swift instead. >>>> * Code completion won’t work, because Swift has no visibility into >>>> declarations written in Python >>>> * Indexing/jump-to-definition/lookup documentation/generated interface >>>> won’t ever work. None of the IDE features supported by SourceKit will >>>> work, which will be a significant regression for users coming from a >>>> Python-capable IDE. >>> >>> Yes, welcome to the realities of modern Python development! >> >> Python plugins for IDEs (e.g., for Atom) provide code completion, goto >> definition, and other related features. None of the Swift tooling will work >> if Swift’s interoperability with Python is entirely based on >> DynamicMemberLookupProtocol. > > I don’t understand your rationale here. I think you agree that we need to > support the fully dynamic case (which is what I’m proposing). That said, > this step does not preclude introducing importer magic (either through > compiler hackery or a theoretical "type providers” style of feature). Doing > so would provide the functionality you’re describing. I don’t agree that we need language support for the fully-dynamic case. There has to be a way to handle the fully-dynamic case, but it can be string-based APIs vended by an Python interoperability library. I believe that the majority of property and method accesses in Python programs will be resolved to a definition in the current model or an imported module, i.e., a definition that is visible to the compiler at compile-time. There are exceptional cases involving programmatically creating properties from JSON, a database, etc., but while prominent I suspect they account for a relatively small fraction of use sites. Do you agree with this statement? >>>> How Should Python Interoperability Work? >>>> Going back to the central motivator for this proposal, I think that >>>> providing something akin to the Clang Importer provides the best >>>> interoperability experience: it would turn Python declarations into *real* >>>> Swift declarations, so that we get the various tooling benefits of having >>>> a strong statically-typed language. >>> >>> This could be an theoretically interesting refinement to this proposal but >>> I’m personally very skeptical that this is every going to happen. I’ve put >>> the rationale into the alternatives section of the proposal. I don’t >>> explain it in the proposal in this way directly, but I believe it is far >>> more likely for a Pythonista transplant into Swift to rewrite their code in >>> Swift than it is to use Python type annotations. >> >> I assume that this belief is based on type annotations lack of traction in >> the Python community thus far? > > There are many parts to this, which have to do with the ObjC<->Swift > situation being very different than the Python<->Swift situation: > > 1) The annotations don’t have significant traction in the Python community. > 2) The Python annotations are not as powerful as ObjC generics are, and thus > lack important expressive capability. > 3) Many Python APIs are wrappers for C APIs. “Swiftizing” a Python API in > this case means writing a new Swift wrapper for the API, not adding type > annotations. > 4) The Python community doesn’t care about Swift, and are not motivated to do > things to make Swift succeed. > 5) There is no “clang equivalent” for Python (that I’m aware of) which close > enough to the way Clang does for us to directly use. The owners of the > existing Python compiler/interpreter implementations are not going to be > strongly motivated to change their stuff for us. > > Finally, just MHO, but I don’t expect a lot of “mix and match" Python/Swift > apps to exist (where the developer owns both the Python and the Swift code), > which is one case where type annotations are super awesome in ObjC. IMO, the > most important use-case is a Swift program that uses some Python APIs. Let’s consider Python type annotations to be useless for our purposes. None of my argument hinges on it. >>>> Sure, the argument types will all by PyObject or PyVal, >>> >>> That’s the root of the problem. Python has the “fully dynamic” equivalent >>> of “id” in Objective-C, so we need to represent that somehow. Even if we >>> followed the implementation approach of the Clang importer, we would need >>> some way to represent this dynamic case. That type needs features like >>> DynamicMemberLookup or AnyObject. In my opinion, the DynamicMemberLookup >>> approach is better in every way than AnyObject is. >> >> The AnyObject approach has the advantage of knowing the set of declared, >> reachable APIs: >> >> * Code completion shows all of the APIs that are possible to use via dynamic >> dispatch, with their signatures so can fill in the right # of arguments, see >> the names of the parameters, see documentation, etc. >> * Indexing/refactoring/goto definition all point you to the declarations >> that could be the targets of dynamic dispatch >> >> The DynamicMemberLookup approach is better for cases where you don’t have a >> declaration of the member you want to access. I suspect that’s not the >> common case. > > Your points are valid, but the advantages for Objective-C don’t obviously > translate to Swift. Note that ObjC (due to its heritage) has very long > method names that are perhaps arguably designed to not conflict with each > other often. Python doesn’t have this heritage, and it has much shorter > names, which means that we’ll get a lot more conflicts and a lot less > “safety" out of this. AnyObject lookup resolves conflicts by collapsing similar overloads, and that’s okay: we don’t really care which class contained the declaration we found; we just care about the shape of the declaration. The lack of this magic will be a problem for the wrapper approach I’m describing, but that’s fixable. > AnyObject lookup also depends on a strange set of scoping heuristics that was > designed to be similar to Clang’s “header import” scope. It isn’t clear that > this approach will work in Python, given that it doesn’t have an analogue of > umbrella headers that import things that cross frameworks. Not sure what you think the scoping heuristics are. AnyObject lookup is fairly simple to define: it finds all @objc members of every class and protocol in your current module and any modules you imported. >>> Which approach do you think is the best way to handle the untyped “actually >>> dynamic” case? >> >> AnyObject already exists in the language, and it fits the untyped “actually >> dynamic” case well. It does require having a declaration for the thing you >> want to reference, which I consider to be important: we can code-complete >> those declarations, goto-definition to see those declarations, >> index/refactor/look up documentation based on those declarations. >> >> I’d be more inclined to push for the ImplicitlyUnwrappedOptional -> Optional >> change if we did something to make AnyObject more prominent in Swift. > > I didn’t realize that you were thinking we would literally use AnyObject > itself. I haven’t thought fully through it, but I think this will provide > several problems: > > 1) You’re mushing all of the ObjC and Python world’s together, making the > ObjC interop worse just because you’re doing some Python stuff too. > 2) You’re introducing ambiguity: does “ao = [1,2,3]” create an NSArray or a > Python array? How do string literals work? (The answer is obvious, Python > loses). Maybe there is some really complicated bridging solution to these > problems, but that causes its own massive complexity spiral. > 3) You can’t realistically overload the Python operator set on AnyObject, > which means you get a worse python experience. > 4) AnyObject magic is currently limited to Apple platforms. This would bring > its problems to other platforms like Linux. > > There are probably other issues, but I haven’t thought through it. You are correct that literally using AnyObject has these issues. There appears to be a lot of confusion about what the AnyObject model *is*, so let’s sort through that. I think it is reasonable to take the AnyObject model and replicate it for PythonObject. >>>> In truth, you don’t even need the compiler to be involved. The dynamic >>>> “subscript” operation could be implemented in a Swift library, and one >>>> could write a Python program to process a Python module and emit Swift >>>> wrappers that call into that subscript operation. You’ll get all of the >>>> tooling benefits with no compiler changes, and can tweak the wrapper >>>> generation however much you want, using typing annotations or other >>>> Python-specific information to create better wrappers over time. >>> >>> I’d love for you to sketch out how any of this works with an acceptable >>> user experience, because I don't see anything concrete here. >> >> We don’t need the basic dynamic case in the language to do this experiment. >> Take the PyVal struct from the proposal. Now, write a Python script that >> loads some module Foo and uses Python’s inspect >> <https://docs.python.org/3/library/inspect.html> module to go find the >> classes, methods, etc., and pretty-print Swift code that uses PyVal. So this: >> >> def add_trick(self, trick): >> >> turns into >> >> extension PyVal { >> func add_trick(_ trick: PyVal) -> PyVal { >> /* do the magic to call into Python */ >> } >> } >> >> Using the inspect module, you can extract parameter names, default >> arguments, docstrings, and more to reflect the existing Python API as Swift >> API, packed into a bridging module. >> >> Note that we have a “flat” namespace of all Python methods on PyVal, which >> is basically what you get with AnyObject today. Swift tooling will provide >> code completion for member accesses into PyVal. Goto definition will jump to >> the pretty-printed declarations, which could have the docstrings formatted >> in comments and would show up in QuickHelp. The types are weak (everything >> is PyVal), but that’s what we expect from importing a dynamically-typed >> language. > > As I mention above, I expect this to expose significant scalability problems > in the Swift compiler and it also defeats REPL/Playgrounds. Being able to > use the Swift REPL is really important for Python programmers. I’ve addressed both of these issues above. The approach I am proposing requires no language or compiler support, works with existing Swift tooling, fits with an existing notion in the language (AnyObject), and can be implemented via a Python script. - Doug
_______________________________________________ swift-evolution mailing list swift-evolution@swift.org https://lists.swift.org/mailman/listinfo/swift-evolution