> On Dec 1, 2017, at 1:30 AM, Chris Lattner <clatt...@nondot.org> wrote:
>> On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgre...@apple.com 
>> <mailto:dgre...@apple.com>> wrote:
>>>> Philosophy
>>> More problematically for your argument: your preferred approach requires 
>>> the introduction of (something like) DynamicMemberLookupProtocol or 
>>> something like AnyObject-For-Python, so your proposal would be additive on 
>>> top of solving the core problem I’m trying to solve.  It isn’t an 
>>> alternative approach at all.
>> I wouldn’t say that’s my preferred approach. My preferred approach involves 
>> taking the method/property/etc. declarations that already exist in Python 
>> and mapping them into corresponding Swift declarations so we have something 
>> to find with name lookup. One could put all of these declarations on some 
>> PyVal struct or PythonObject and there would be no need for 
>> AnyObject-for-Python or DynamicMemberLookupProtocol.
> You’re suggesting that the transitive closure of all Python methods and 
> properties be preprocessed into a single gigantic Swift PyVal type?  I guess 
> something like that could be done.

That’s effectively what Swift is already doing with AnyObject.

> I would be concerned because there are many N^2 or worse algorithms in the 
> Swift compiler would probably explode.

As noted above, we already do this for AnyObject with every @objc entity 
everywhere. The # of overloads for a given name is usually not so high that 
it’s a problem for the compiler.

>  It also doesn’t provide the great tooling experience that you’re seeking, 
> given that code completion would show everything in the Python universe, 
> which is not helpful.

It’s better for code completion to provide too much than to provide nothing at 
all. If code completion provides too much, typing a small number of characters 
will reduce the completion set down to something manageable quite fast.

> Further, it doesn’t provide a *better* experience than what I’m suggesting, 
> it seems strictly worse.  

>From my prior message, having the declarations of all Python methods and 
>properties available on PyVal makes Swift tooling work:

For the same working Swift code


that calls into Python, the solution I’m proposing:

* Gave you the add_trick(<#trick#>) code completion with the number of 
parameters and their names
* Supports goto definition to jump to the wrapper method on PyVal
* Supports “quick help” to show the declaration of that wrapper method on 
PyVal, showing the documentation (docstring) and in which Python class it was 
* Supports indexing functionality so we can find the likely uses of “add_trick"

DynamicMemberLookup doesn’t gave any of those, because “add_trick” is just a 
string that looks like an identifier. Only the Python runtime can resolve.

> A preprocessing step prevents users from playfully importing random Python 
> modules into the Swift repl and playgrounds.

*At worst*, you invoke the tool from the command line to build the Swift module 
that corresponds to a given Python module.

        py2swift <pythonmodulename>

We could absolutely introduce tooling hooks to make the compiler initiate that 

>  It also seems worse for implementors (who will get a stream of new bugs 
> about compiler scalability).

To my knowledge, AnyObject lookup has not been a serious source of performance 
problems for the compiler. The global lookup table it requires was annoying to 
implement, but it’s just a symbol table.

>>>> Whenever we discuss adding more dynamic features to Swift, there’s a 
>>>> strong focus on maintaining that strong static type system.
>>> Which this does, by providing full type safety - unlike AnyObject lookup.
>> You get dynamic safety because it goes into the Python interpreter; fair 
>> enough. You get no help from your tools to form a correct invocation of any 
>> method provided by Python.
> Sure, that’s status quo for Python APIs.

That’s not the status quo for Python IDEs. They will give you the basic shape 
of method calls.

>>>> Tooling
>>>> The Python interoperability enabled by this proposal *does* look fairly 
>>>> nice when you look at a small, correctly-written example. However, 
>>>> absolutely none of the tooling assistance we rely on when writing such 
>>>> code will work for Python interoperability. Examples:
>>>> * As noted earlier, if you typo’d a name of a Python entity or passed the 
>>>> wrong number of arguments to it, the compiler will not tell you: it’ll be 
>>>> a runtime failure in the Python interpreter. I guess that’s what you’d get 
>>>> if you were writing the code in Python, but Swift is supposed to be 
>>>> *better* than Python if we’re to convince a community to use Swift instead.
>>>> * Code completion won’t work, because Swift has no visibility into 
>>>> declarations written in Python
>>>> * Indexing/jump-to-definition/lookup documentation/generated interface 
>>>> won’t ever work. None of the IDE features supported by SourceKit will 
>>>> work, which will be a significant regression for users coming from a 
>>>> Python-capable IDE.
>>> Yes, welcome to the realities of modern Python development!
>> Python plugins for IDEs (e.g., for Atom) provide code completion, goto 
>> definition, and other related features. None of the Swift tooling will work 
>> if Swift’s interoperability with Python is entirely based on 
>> DynamicMemberLookupProtocol.
> I don’t understand your rationale here.  I think you agree that we need to 
> support the fully dynamic case (which is what I’m proposing).  That said, 
> this step does not preclude introducing importer magic (either through 
> compiler hackery or a theoretical "type providers” style of feature).  Doing 
> so would provide the functionality you’re describing.

I don’t agree that we need language support for the fully-dynamic case. There 
has to be a way to handle the fully-dynamic case, but it can be string-based 
APIs vended by an Python interoperability library.

I believe that the majority of property and method accesses in Python programs 
will be resolved to a definition in the current model or an imported module, 
i.e., a definition that is visible to the compiler at compile-time. There are 
exceptional cases involving programmatically creating properties from JSON, a 
database, etc., but while prominent I suspect they account for a relatively 
small fraction of use sites. Do you agree with this statement?

>>>> How Should Python Interoperability Work?
>>>> Going back to the central motivator for this proposal, I think that 
>>>> providing something akin to the Clang Importer provides the best 
>>>> interoperability experience: it would turn Python declarations into *real* 
>>>> Swift declarations, so that we get the various tooling benefits of having 
>>>> a strong statically-typed language.
>>> This could be an theoretically interesting refinement to this proposal but 
>>> I’m personally very skeptical that this is every going to happen.  I’ve put 
>>> the rationale into the alternatives section of the proposal.  I don’t 
>>> explain it in the proposal in this way directly, but I believe it is far 
>>> more likely for a Pythonista transplant into Swift to rewrite their code in 
>>> Swift than it is to use Python type annotations.
>> I assume that this belief is based on type annotations lack of traction in 
>> the Python community thus far?
> There are many parts to this, which have to do with the ObjC<->Swift 
> situation being very different than the Python<->Swift situation:
> 1) The annotations don’t have significant traction in the Python community. 
> 2) The Python annotations are not as powerful as ObjC generics are, and thus 
> lack important expressive capability.
> 3) Many Python APIs are wrappers for C APIs.  “Swiftizing” a Python API in 
> this case means writing a new Swift wrapper for the API, not adding type 
> annotations.
> 4) The Python community doesn’t care about Swift, and are not motivated to do 
> things to make Swift succeed.
> 5) There is no “clang equivalent” for Python (that I’m aware of) which close 
> enough to the way Clang does for us to directly use.  The owners of the 
> existing Python compiler/interpreter implementations are not going to be 
> strongly motivated to change their stuff for us.
> Finally, just MHO, but I don’t expect a lot of “mix and match" Python/Swift 
> apps to exist (where the developer owns both the Python and the Swift code), 
> which is one case where type annotations are super awesome in ObjC.  IMO, the 
> most important use-case is a Swift program that uses some Python APIs.

Let’s consider Python type annotations to be useless for our purposes. None of 
my argument hinges on it.

>>>> Sure, the argument types will all by PyObject or PyVal,
>>> That’s the root of the problem.  Python has the “fully dynamic” equivalent 
>>> of “id” in Objective-C, so we need to represent that somehow.  Even if we 
>>> followed the implementation approach of the Clang importer, we would need 
>>> some way to represent this dynamic case.  That type needs features like 
>>> DynamicMemberLookup or AnyObject.  In my opinion, the DynamicMemberLookup 
>>> approach is better in every way than AnyObject is.  
>> The AnyObject approach has the advantage of knowing the set of declared, 
>> reachable APIs:
>> * Code completion shows all of the APIs that are possible to use via dynamic 
>> dispatch, with their signatures so can fill in the right # of arguments, see 
>> the names of the parameters, see documentation, etc.
>> * Indexing/refactoring/goto definition all point you to the declarations 
>> that could be the targets of dynamic dispatch
>> The DynamicMemberLookup approach is better for cases where you don’t have a 
>> declaration of the member you want to access. I suspect that’s not the 
>> common case.
> Your points are valid, but the advantages for Objective-C don’t obviously 
> translate to Swift.  Note that ObjC (due to its heritage) has very long 
> method names that are perhaps arguably designed to not conflict with each 
> other often.  Python doesn’t have this heritage, and it has much shorter 
> names, which means that we’ll get a lot more conflicts and a lot less 
> “safety" out of this.

AnyObject lookup resolves conflicts by collapsing similar overloads, and that’s 
okay: we don’t really care which class contained the declaration we found; we 
just care about the shape of the declaration. The lack of this magic will be a 
problem for the wrapper approach I’m describing, but that’s fixable.

> AnyObject lookup also depends on a strange set of scoping heuristics that was 
> designed to be similar to Clang’s “header import” scope.  It isn’t clear that 
> this approach will work in Python, given that it doesn’t have an analogue of 
> umbrella headers that import things that cross frameworks.

Not sure what you think the scoping heuristics are. AnyObject lookup is fairly 
simple to define: it finds all @objc members of every class and protocol in 
your current module and any modules you imported.

>>> Which approach do you think is the best way to handle the untyped “actually 
>>> dynamic” case? 
>> AnyObject already exists in the language, and it fits the untyped “actually 
>> dynamic” case well. It does require having a declaration for the thing you 
>> want to reference, which I consider to be important: we can code-complete 
>> those declarations, goto-definition to see those declarations, 
>> index/refactor/look up documentation based on those declarations.
>> I’d be more inclined to push for the ImplicitlyUnwrappedOptional -> Optional 
>> change if we did something to make AnyObject more prominent in Swift.
> I didn’t realize that you were thinking we would literally use AnyObject 
> itself.  I haven’t thought fully through it, but I think this will provide 
> several problems:
> 1) You’re mushing all of the ObjC and Python world’s together, making the 
> ObjC interop worse just because you’re doing some Python stuff too.
> 2) You’re introducing ambiguity: does “ao = [1,2,3]” create an NSArray or a 
> Python array?  How do string literals work? (The answer is obvious, Python 
> loses). Maybe there is some really complicated bridging solution to these 
> problems, but that causes its own massive complexity spiral.
> 3) You can’t realistically overload the Python operator set on AnyObject, 
> which means you get a worse python experience.
> 4) AnyObject magic is currently limited to Apple platforms.  This would bring 
> its problems to other platforms like Linux.
> There are probably other issues, but I haven’t thought through it.

You are correct that literally using AnyObject has these issues. There appears 
to be a lot of confusion about what the AnyObject model *is*, so let’s sort 
through that. I think it is reasonable to take the AnyObject model and 
replicate it for PythonObject.

>>>> In truth, you don’t even need the compiler to be involved. The dynamic 
>>>> “subscript” operation could be implemented in a Swift library, and one 
>>>> could write a Python program to process a Python module and emit Swift 
>>>> wrappers that call into that subscript operation. You’ll get all of the 
>>>> tooling benefits with no compiler changes, and can tweak the wrapper 
>>>> generation however much you want, using typing annotations or other 
>>>> Python-specific information to create better wrappers over time.
>>> I’d love for you to sketch out how any of this works with an acceptable 
>>> user experience, because I don't see anything concrete here.  
>> We don’t need the basic dynamic case in the language to do this experiment. 
>> Take the PyVal struct from the proposal. Now, write a Python script that 
>> loads some module Foo and uses Python’s inspect 
>> <https://docs.python.org/3/library/inspect.html> module to go find the 
>> classes, methods, etc., and pretty-print Swift code that uses PyVal. So this:
>>      def add_trick(self, trick):
>> turns into
>>      extension PyVal {
>>        func add_trick(_ trick: PyVal) -> PyVal {
>>          /* do the magic to call into Python */
>>        }
>>      }
>> Using the inspect module, you can extract parameter names, default 
>> arguments, docstrings, and more to reflect the existing Python API as Swift 
>> API, packed into a bridging module.
>> Note that we have a “flat” namespace of all Python methods on PyVal, which 
>> is basically what you get with AnyObject today. Swift tooling will provide 
>> code completion for member accesses into PyVal. Goto definition will jump to 
>> the pretty-printed declarations, which could have the docstrings formatted 
>> in comments and would show up in QuickHelp. The types are weak (everything 
>> is PyVal), but that’s what we expect from importing a dynamically-typed 
>> language.
> As I mention above, I expect this to expose significant scalability problems 
> in the Swift compiler and it also defeats REPL/Playgrounds.  Being able to 
> use the Swift REPL is really important for Python programmers.

I’ve addressed both of these issues above. The approach I am proposing requires 
no language or compiler support, works with existing Swift tooling, fits with 
an existing notion in the language (AnyObject), and can be implemented via a 
Python script.

        - Doug

swift-evolution mailing list

Reply via email to