> On Dec 1, 2017, at 1:30 AM, Chris Lattner <clatt...@nondot.org> wrote:
>
>> On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgre...@apple.com
>> <mailto:dgre...@apple.com>> wrote:
>>>> Philosophy
>
>>> More problematically for your argument: your preferred approach requires
>>> the introduction of (something like) DynamicMemberLookupProtocol or
>>> something like AnyObject-For-Python, so your proposal would be additive on
>>> top of solving the core problem I’m trying to solve. It isn’t an
>>> alternative approach at all.
>>
>> I wouldn’t say that’s my preferred approach. My preferred approach involves
>> taking the method/property/etc. declarations that already exist in Python
>> and mapping them into corresponding Swift declarations so we have something
>> to find with name lookup. One could put all of these declarations on some
>> PyVal struct or PythonObject and there would be no need for
>> AnyObject-for-Python or DynamicMemberLookupProtocol.
>
> You’re suggesting that the transitive closure of all Python methods and
> properties be preprocessed into a single gigantic Swift PyVal type? I guess
> something like that could be done.
That’s effectively what Swift is already doing with AnyObject.
> I would be concerned because there are many N^2 or worse algorithms in the
> Swift compiler would probably explode.
As noted above, we already do this for AnyObject with every @objc entity
everywhere. The # of overloads for a given name is usually not so high that
it’s a problem for the compiler.
> It also doesn’t provide the great tooling experience that you’re seeking,
> given that code completion would show everything in the Python universe,
> which is not helpful.
It’s better for code completion to provide too much than to provide nothing at
all. If code completion provides too much, typing a small number of characters
will reduce the completion set down to something manageable quite fast.
> Further, it doesn’t provide a *better* experience than what I’m suggesting,
> it seems strictly worse.
>From my prior message, having the declarations of all Python methods and
>properties available on PyVal makes Swift tooling work:
For the same working Swift code
dog.add_trick(rollOver)
that calls into Python, the solution I’m proposing:
* Gave you the add_trick(<#trick#>) code completion with the number of
parameters and their names
* Supports goto definition to jump to the wrapper method on PyVal
* Supports “quick help” to show the declaration of that wrapper method on
PyVal, showing the documentation (docstring) and in which Python class it was
declared.
* Supports indexing functionality so we can find the likely uses of “add_trick"
DynamicMemberLookup doesn’t gave any of those, because “add_trick” is just a
string that looks like an identifier. Only the Python runtime can resolve.
> A preprocessing step prevents users from playfully importing random Python
> modules into the Swift repl and playgrounds.
*At worst*, you invoke the tool from the command line to build the Swift module
that corresponds to a given Python module.
py2swift <pythonmodulename>
We could absolutely introduce tooling hooks to make the compiler initiate that
step.
> It also seems worse for implementors (who will get a stream of new bugs
> about compiler scalability).
To my knowledge, AnyObject lookup has not been a serious source of performance
problems for the compiler. The global lookup table it requires was annoying to
implement, but it’s just a symbol table.
>
>>>> Whenever we discuss adding more dynamic features to Swift, there’s a
>>>> strong focus on maintaining that strong static type system.
>>>
>>> Which this does, by providing full type safety - unlike AnyObject lookup.
>>
>> You get dynamic safety because it goes into the Python interpreter; fair
>> enough. You get no help from your tools to form a correct invocation of any
>> method provided by Python.
>
> Sure, that’s status quo for Python APIs.
That’s not the status quo for Python IDEs. They will give you the basic shape
of method calls.
>
>>>> Tooling
>>>> The Python interoperability enabled by this proposal *does* look fairly
>>>> nice when you look at a small, correctly-written example. However,
>>>> absolutely none of the tooling assistance we rely on when writing such
>>>> code will work for Python interoperability. Examples:
>>>>
>>>> * As noted earlier, if you typo’d a name of a Python entity or passed the
>>>> wrong number of arguments to it, the compiler will not tell you: it’ll be
>>>> a runtime failure in the Python interpreter. I guess that’s what you’d get
>>>> if you were writing the code in Python, but Swift is supposed to be
>>>> *better* than Python if we’re to convince a community to use Swift instead.
>>>> * Code completion won’t work, because Swift has no visibility into
>>>> declarations written in Python
>>>> * Indexing/jump-to-definition/lookup documentation/generated interface
>>>> won’t ever work. None of the IDE features supported by SourceKit will
>>>> work, which will be a significant regression for users coming from a
>>>> Python-capable IDE.
>>>
>>> Yes, welcome to the realities of modern Python development!
>>
>> Python plugins for IDEs (e.g., for Atom) provide code completion, goto
>> definition, and other related features. None of the Swift tooling will work
>> if Swift’s interoperability with Python is entirely based on
>> DynamicMemberLookupProtocol.
>
> I don’t understand your rationale here. I think you agree that we need to
> support the fully dynamic case (which is what I’m proposing). That said,
> this step does not preclude introducing importer magic (either through
> compiler hackery or a theoretical "type providers” style of feature). Doing
> so would provide the functionality you’re describing.
I don’t agree that we need language support for the fully-dynamic case. There
has to be a way to handle the fully-dynamic case, but it can be string-based
APIs vended by an Python interoperability library.
I believe that the majority of property and method accesses in Python programs
will be resolved to a definition in the current model or an imported module,
i.e., a definition that is visible to the compiler at compile-time. There are
exceptional cases involving programmatically creating properties from JSON, a
database, etc., but while prominent I suspect they account for a relatively
small fraction of use sites. Do you agree with this statement?
>>>> How Should Python Interoperability Work?
>>>> Going back to the central motivator for this proposal, I think that
>>>> providing something akin to the Clang Importer provides the best
>>>> interoperability experience: it would turn Python declarations into *real*
>>>> Swift declarations, so that we get the various tooling benefits of having
>>>> a strong statically-typed language.
>>>
>>> This could be an theoretically interesting refinement to this proposal but
>>> I’m personally very skeptical that this is every going to happen. I’ve put
>>> the rationale into the alternatives section of the proposal. I don’t
>>> explain it in the proposal in this way directly, but I believe it is far
>>> more likely for a Pythonista transplant into Swift to rewrite their code in
>>> Swift than it is to use Python type annotations.
>>
>> I assume that this belief is based on type annotations lack of traction in
>> the Python community thus far?
>
> There are many parts to this, which have to do with the ObjC<->Swift
> situation being very different than the Python<->Swift situation:
>
> 1) The annotations don’t have significant traction in the Python community.
> 2) The Python annotations are not as powerful as ObjC generics are, and thus
> lack important expressive capability.
> 3) Many Python APIs are wrappers for C APIs. “Swiftizing” a Python API in
> this case means writing a new Swift wrapper for the API, not adding type
> annotations.
> 4) The Python community doesn’t care about Swift, and are not motivated to do
> things to make Swift succeed.
> 5) There is no “clang equivalent” for Python (that I’m aware of) which close
> enough to the way Clang does for us to directly use. The owners of the
> existing Python compiler/interpreter implementations are not going to be
> strongly motivated to change their stuff for us.
>
> Finally, just MHO, but I don’t expect a lot of “mix and match" Python/Swift
> apps to exist (where the developer owns both the Python and the Swift code),
> which is one case where type annotations are super awesome in ObjC. IMO, the
> most important use-case is a Swift program that uses some Python APIs.
Let’s consider Python type annotations to be useless for our purposes. None of
my argument hinges on it.
>>>> Sure, the argument types will all by PyObject or PyVal,
>>>
>>> That’s the root of the problem. Python has the “fully dynamic” equivalent
>>> of “id” in Objective-C, so we need to represent that somehow. Even if we
>>> followed the implementation approach of the Clang importer, we would need
>>> some way to represent this dynamic case. That type needs features like
>>> DynamicMemberLookup or AnyObject. In my opinion, the DynamicMemberLookup
>>> approach is better in every way than AnyObject is.
>>
>> The AnyObject approach has the advantage of knowing the set of declared,
>> reachable APIs:
>>
>> * Code completion shows all of the APIs that are possible to use via dynamic
>> dispatch, with their signatures so can fill in the right # of arguments, see
>> the names of the parameters, see documentation, etc.
>> * Indexing/refactoring/goto definition all point you to the declarations
>> that could be the targets of dynamic dispatch
>>
>> The DynamicMemberLookup approach is better for cases where you don’t have a
>> declaration of the member you want to access. I suspect that’s not the
>> common case.
>
> Your points are valid, but the advantages for Objective-C don’t obviously
> translate to Swift. Note that ObjC (due to its heritage) has very long
> method names that are perhaps arguably designed to not conflict with each
> other often. Python doesn’t have this heritage, and it has much shorter
> names, which means that we’ll get a lot more conflicts and a lot less
> “safety" out of this.
AnyObject lookup resolves conflicts by collapsing similar overloads, and that’s
okay: we don’t really care which class contained the declaration we found; we
just care about the shape of the declaration. The lack of this magic will be a
problem for the wrapper approach I’m describing, but that’s fixable.
> AnyObject lookup also depends on a strange set of scoping heuristics that was
> designed to be similar to Clang’s “header import” scope. It isn’t clear that
> this approach will work in Python, given that it doesn’t have an analogue of
> umbrella headers that import things that cross frameworks.
Not sure what you think the scoping heuristics are. AnyObject lookup is fairly
simple to define: it finds all @objc members of every class and protocol in
your current module and any modules you imported.
>>> Which approach do you think is the best way to handle the untyped “actually
>>> dynamic” case?
>>
>> AnyObject already exists in the language, and it fits the untyped “actually
>> dynamic” case well. It does require having a declaration for the thing you
>> want to reference, which I consider to be important: we can code-complete
>> those declarations, goto-definition to see those declarations,
>> index/refactor/look up documentation based on those declarations.
>>
>> I’d be more inclined to push for the ImplicitlyUnwrappedOptional -> Optional
>> change if we did something to make AnyObject more prominent in Swift.
>
> I didn’t realize that you were thinking we would literally use AnyObject
> itself. I haven’t thought fully through it, but I think this will provide
> several problems:
>
> 1) You’re mushing all of the ObjC and Python world’s together, making the
> ObjC interop worse just because you’re doing some Python stuff too.
> 2) You’re introducing ambiguity: does “ao = [1,2,3]” create an NSArray or a
> Python array? How do string literals work? (The answer is obvious, Python
> loses). Maybe there is some really complicated bridging solution to these
> problems, but that causes its own massive complexity spiral.
> 3) You can’t realistically overload the Python operator set on AnyObject,
> which means you get a worse python experience.
> 4) AnyObject magic is currently limited to Apple platforms. This would bring
> its problems to other platforms like Linux.
>
> There are probably other issues, but I haven’t thought through it.
You are correct that literally using AnyObject has these issues. There appears
to be a lot of confusion about what the AnyObject model *is*, so let’s sort
through that. I think it is reasonable to take the AnyObject model and
replicate it for PythonObject.
>>>> In truth, you don’t even need the compiler to be involved. The dynamic
>>>> “subscript” operation could be implemented in a Swift library, and one
>>>> could write a Python program to process a Python module and emit Swift
>>>> wrappers that call into that subscript operation. You’ll get all of the
>>>> tooling benefits with no compiler changes, and can tweak the wrapper
>>>> generation however much you want, using typing annotations or other
>>>> Python-specific information to create better wrappers over time.
>>>
>>> I’d love for you to sketch out how any of this works with an acceptable
>>> user experience, because I don't see anything concrete here.
>>
>> We don’t need the basic dynamic case in the language to do this experiment.
>> Take the PyVal struct from the proposal. Now, write a Python script that
>> loads some module Foo and uses Python’s inspect
>> <https://docs.python.org/3/library/inspect.html> module to go find the
>> classes, methods, etc., and pretty-print Swift code that uses PyVal. So this:
>>
>> def add_trick(self, trick):
>>
>> turns into
>>
>> extension PyVal {
>> func add_trick(_ trick: PyVal) -> PyVal {
>> /* do the magic to call into Python */
>> }
>> }
>>
>> Using the inspect module, you can extract parameter names, default
>> arguments, docstrings, and more to reflect the existing Python API as Swift
>> API, packed into a bridging module.
>>
>> Note that we have a “flat” namespace of all Python methods on PyVal, which
>> is basically what you get with AnyObject today. Swift tooling will provide
>> code completion for member accesses into PyVal. Goto definition will jump to
>> the pretty-printed declarations, which could have the docstrings formatted
>> in comments and would show up in QuickHelp. The types are weak (everything
>> is PyVal), but that’s what we expect from importing a dynamically-typed
>> language.
>
> As I mention above, I expect this to expose significant scalability problems
> in the Swift compiler and it also defeats REPL/Playgrounds. Being able to
> use the Swift REPL is really important for Python programmers.
I’ve addressed both of these issues above. The approach I am proposing requires
no language or compiler support, works with existing Swift tooling, fits with
an existing notion in the language (AnyObject), and can be implemented via a
Python script.
- Doug
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution
