On Fri, Dec 1, 2017 at 11:03 AM, Benjamin G via swift-evolution < swift-evolution@swift.org> wrote:
> I think the fear most of us , poor developers working with developers of > various skills, is the potential for abuse. I've heard many times that it > isn't a guiding principle for swift, so i'll just reformulate the concern > in a more acceptable way : > Could your proposal explain how the swift language will still keep > encouraging developers to use static / compile-time type checking *when > it's possible* ? (because i think we can all argue that a compile-time > checks is better than a runtime one). Obviously, python interop isn't my > concern here. I'm talking about the impact of your proposal on pure swift > code. > > As example, calling some feature "unsafe", or "force", makes every > developer pause and wonder if there isn't a better alternative. I'd like to > see the same kind of things for dynamic calls. > A few examples of the kind of things that i would think about : - use different file names for code using dynamic features. - have every dynamic code included in a dynamic{} block - have a different syntax for dynamic function calls, using ! or !!! - Make the dynamic calls work only of subclasses of a base "PythonObject". That would make the goal clear. etc. Now, i think those are all bad ideas (and that's why i don't like the proposal in the first place). But i really think the concern should be addressed. > > > On Fri, Dec 1, 2017 at 10:30 AM, Chris Lattner via swift-evolution < > swift-evolution@swift.org> wrote: > >> On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgre...@apple.com> wrote: >> >> *Philosophy* >> Swift is, unabashedly, a strong statically-typed language. >> >> >> That’s factually incorrect. >> >> >> You’re going to have to explain that statement without reference to >> AnyObject (we’ll discuss that case below). >> >> >> Perhaps it depends on what you mean by “strong”: I interpreted that as >> meaning that it provides type safety, with a level of strength akin to what >> Java provides: a level that could support mobile code deployment. >> >> Swift certainly does not provide that strong of a static type system, >> because it gives people explicit ways to opt out of that. >> UnsafeMutableRawPointer, unsafe bitcast, and many other facilities support >> this. It also allows calling into non-type safe code, so it isn’t very >> strong that way. There are also race conditions and other holes in the >> type system. >> >> All that said, I think it is correct that a subset of Swift exists that >> does provide strong type safety, but particularly when bridging to C/ObjC >> is involved, that quickly goes away. >> >> >> More problematically for your argument: your preferred approach requires >> the introduction of (something like) DynamicMemberLookupProtocol or >> something like AnyObject-For-Python, so your proposal would be additive on >> top of solving the core problem I’m trying to solve. It isn’t an >> alternative approach at all. >> >> >> I wouldn’t say that’s my preferred approach. My preferred approach >> involves taking the method/property/etc. declarations that already exist in >> Python and mapping them into corresponding Swift declarations so we have >> something to find with name lookup. One could put all of these declarations >> on some PyVal struct or PythonObject and there would be no need for >> AnyObject-for-Python or DynamicMemberLookupProtocol. >> >> >> You’re suggesting that the transitive closure of all Python methods and >> properties be preprocessed into a single gigantic Swift PyVal type? I >> guess something like that could be done. >> >> I would be concerned because there are many N^2 or worse algorithms in >> the Swift compiler would probably explode. It also doesn’t provide the >> great tooling experience that you’re seeking, given that code completion >> would show everything in the Python universe, which is not helpful. >> >> Further, it doesn’t provide a *better* experience than what I’m >> suggesting, it seems strictly worse. A preprocessing step prevents users >> from playfully importing random Python modules into the Swift repl and >> playgrounds. It also seems worse for implementors (who will get a stream >> of new bugs about compiler scalability). >> >> Whenever we discuss adding more dynamic features to Swift, there’s a >> strong focus on maintaining that strong static type system. >> >> >> Which this does, by providing full type safety - unlike AnyObject lookup. >> >> >> You get dynamic safety because it goes into the Python interpreter; fair >> enough. You get no help from your tools to form a correct invocation of any >> method provided by Python. >> >> >> Sure, that’s status quo for Python APIs. >> >> IMO, this proposal is a significant departure from the fundamental >> character of Swift, because it allows access to possibly-nonexistent >> members (as well as calls with incorrect arguments, in the related >> proposal) without any indication that the operation might fail. >> >> >> The only way your claim is correct is if someone implements the protocol >> wrong. What you describe is true of AnyObject lookup though, so I >> understand how you could be confused by that. >> >> >> AnyObject lookup still requires you to find an actual declaration with a >> type signature. Yes, there are still failure cases. The dynamic type might >> be totally unrelated to the class in which you found the declaration you’re >> supposedly calling, which is a typical “unrecognized selector” failure and >> will always be an issue with dynamic typing. The actual type safety hole >> you’re presumably referring to is that the selector could be overloaded >> with a different type signature, and we don’t proactively check that the >> signature we type-checked against matches the signature found at runtime. >> It’s doable with the Objective-C method encodings, but has never been >> considered worthwhile. >> >> >> To be clear, I’m not suggesting a change to AnyObject lookup. I don’t >> think that it is worth changing at this point in time. My point was to >> observe that the proposed DynamicMemberLookupProtocol proposal does not >> suffer from these problems. It really is type / memory safe, assuming a >> sane implementation. >> >> >> It’s easy to fall through these cracks for any type that supports >> DynamicMemberLookupProtocol—a single-character typo when using a >> DynamicMemberLookupProtocol-capable type means you’ve fallen out of the >> safety that Swift provides. >> >> >> Since you seem to be latching on to this, I’ll say it again: the proposal >> is fully type safe :-) >> >> That said, the “simple typo” problem is fundamental to the problem of >> working with a dynamically typed language: unless you can eradicate all >> “fundamental dynamism” from the language, you cannot prevent this. >> >> That said, since this is a problem inherent with these languages, it is >> something people are already very familiar with, and something that >> everyone using those APIs has had to deal with. This is also not our >> problem to solve, and people in the Python community have been discussing >> and working on it over a large part of its 25 year history. I find your >> belief that we could solve this for Python better than the Python community >> itself has both idealistic and a bit naive. >> >> >> I’m not pretending we can fully solve the problem. I’m pointing out that >> depending entirely on DynamicMemberLookupProtocol throws away information >> about method declarations that is present in Python and used by Python >> tooling to good effect. >> >> >> I’m not opposed to going further over time, but I’d like to get started >> at some point :-). I’m not in a super urgent hurry to get this in in the >> next week or month or anything like that, but I also don’t want to wait >> until Swift 10. >> >> >> *Tooling* >> The Python interoperability enabled by this proposal *does* look fairly >> nice when you look at a small, correctly-written example. However, >> absolutely none of the tooling assistance we rely on when writing such code >> will work for Python interoperability. Examples: >> >> * As noted earlier, if you typo’d a name of a Python entity or passed the >> wrong number of arguments to it, the compiler will not tell you: it’ll be a >> runtime failure in the Python interpreter. I guess that’s what you’d get if >> you were writing the code in Python, but Swift is supposed to be *better* >> than Python if we’re to convince a community to use Swift instead. >> * Code completion won’t work, because Swift has no visibility into >> declarations written in Python >> * Indexing/jump-to-definition/lookup documentation/generated interface >> won’t ever work. None of the IDE features supported by SourceKit will work, >> which will be a significant regression for users coming from a >> Python-capable IDE. >> >> >> Yes, welcome to the realities of modern Python development! >> >> >> Python plugins for IDEs (e.g., for Atom) provide code completion, goto >> definition, and other related features. None of the Swift tooling will work >> if Swift’s interoperability with Python is entirely based on >> DynamicMemberLookupProtocol. >> >> >> I don’t understand your rationale here. I think you agree that we need >> to support the fully dynamic case (which is what I’m proposing). That >> said, this step does not preclude introducing importer magic (either >> through compiler hackery or a theoretical "type providers” style of >> feature). Doing so would provide the functionality you’re describing. >> >> >> Statically-typed languages should be a boon for tooling, but if a user >> coming from Python to Swift *because* it’s supposed to be a better >> development experience actually sees a significantly worse development >> experience, we’re not going to win them over. It’ll just feel inconsistent. >> >> >> By your argument we should ban AnyObject lookup as well, given its >> inconsistency with the rest of the language. >> >> >> By my argument, we should at least replace the >> ImplicitlyUnwrappedOptional result with a true Optional (so one has to >> acknowledge that the method shouldn’t be there). We’ve seriously considered >> it, but it’s a source-breaking change, and it hasn’t seemed worth the >> engineering effort to pursue it. >> >> >> Ok, that would be a nice step, but doesn’t fix the type safety hole. >> >> In any case, my proposal allows the use of strong optional results as >> well, so the fate of AnyObject isn’t really bound up with it. >> >> I don’t think that removing AnyObject dispatch entirely is possible at >> this point in Swift’s lifetime. While AnyObject has become much less >> prominent than it was in the Swift 1.0 days ([AnyObject] and [NSObject : >> AnyObject], oh my!), there is still a significant amount of code using it >> in the wild. >> >> >> Completely agreed. The major advantage I see of changing it now is if >> there is some small mostly-user-invisible-change that allows a dramatic >> simplification to the compiler implementation. >> >> *Dynamic Typing Features* >> It’s possible that the right evolutionary path for Swift involves some >> notion of dynamic typing, which would have a lot of the properties sought >> by this proposal (and the DynamicCallableProtocol one). If that is true—and >> I’m not at all convinced that it is—we shouldn’t accidentally fall into a >> suboptimal design by taking small, easy, steps. >> >> >> Given that you haven’t followed the discussion on the many threads we’ve >> had on this, and haven’t proposed a workable approach to this problem, I’m >> not sure upon what basis your fears and uncertainty and doubt are founded. >> >> >> A few meta-comments here. First of all, following all previous threads on >> a discussion is not realistic. >> >> >> I understand that, but you’re also accusing the proposal of being a >> suboptimal design made by looking at a series of small easy steps, instead >> of the right design for the long term. I’m pointing out that it is hard to >> see the rationale for that sort of claim. >> >> This is part of the reason why we have different stages in a proposal’s >> lifetime, and is the responsibility of the proposal’s authors to capture >> alternatives and rationale in the proposal to make it self-contained. >> >> >> Agreed. As I mentioned in my previous email, I definitely screwed that >> up by not capturing this discussion in the proposal. Thank you again for >> pulling this perspective to the front of the discussion so I could fix that >> oversight. >> >> This proposal went through rapid iteration in the pitch phase and has now >> been elevated to a pull request to ask for formal review—you should expect >> more people to come on board having not read those threads. I appreciate >> that you have now captured more alternatives and rationale in the proposal. >> >> >> Agreed. This is why I’ve been proactive about starting threads and >> trying to keep visibility on the proposal each time there is a significant >> change. I really do value the discussion and feedback (both on the >> proposed direction but also the writing itself). >> >> Second, it is absolutely reasonable to disagree with the technical >> direction of a proposal without providing a complete solution to the >> problem that the proposal is attempting to solve. Some problems aren’t >> worth solving at all, or fully. >> >> >> Ok, but at some point, if there is no alternative proposed, then a strong >> opposition has the appearance of saying “we shouldn’t solve this problem”. >> It was my understanding that thought that this was a worthwhile problem to >> solve. >> >> >> *How Should Python Interoperability Work?* >> Going back to the central motivator for this proposal, I think that >> providing something akin to the Clang Importer provides the best >> interoperability experience: it would turn Python declarations into *real* >> Swift declarations, so that we get the various tooling benefits of having a >> strong statically-typed language. >> >> >> This could be an theoretically interesting refinement to this proposal >> but I’m personally very skeptical that this is every going to happen. I’ve >> put the rationale into the alternatives section of the proposal. I don’t >> explain it in the proposal in this way directly, but I believe it is far >> more likely for a Pythonista transplant into Swift to rewrite their code in >> Swift than it is to use Python type annotations. >> >> >> I assume that this belief is based on type annotations lack of traction >> in the Python community thus far? >> >> >> There are many parts to this, which have to do with the ObjC<->Swift >> situation being very different than the Python<->Swift situation: >> >> 1) The annotations don’t have significant traction in the Python >> community. >> 2) The Python annotations are not as powerful as ObjC generics are, and >> thus lack important expressive capability. >> 3) Many Python APIs are wrappers for C APIs. “Swiftizing” a Python API >> in this case means writing a new Swift wrapper for the API, not adding type >> annotations. >> 4) The Python community doesn’t care about Swift, and are not motivated >> to do things to make Swift succeed. >> 5) There is no “clang equivalent” for Python (that I’m aware of) which >> close enough to the way Clang does for us to directly use. The owners of >> the existing Python compiler/interpreter implementations are not going to >> be strongly motivated to change their stuff for us. >> >> Finally, just MHO, but I don’t expect a lot of “mix and match" >> Python/Swift apps to exist (where the developer owns both the Python and >> the Swift code), which is one case where type annotations are super awesome >> in ObjC. IMO, the most important use-case is a Swift program that uses >> some Python APIs. >> >> >> Sure, the argument types will all by PyObject or PyVal, >> >> >> That’s the root of the problem. Python has the “fully dynamic” >> equivalent of “id” in Objective-C, so we need to represent that somehow. >> Even if we followed the implementation approach of the Clang importer, we >> would need some way to represent this dynamic case. That type needs >> features like DynamicMemberLookup or AnyObject. In my opinion, the >> DynamicMemberLookup approach is better in every way than AnyObject is. >> >> >> The AnyObject approach has the advantage of knowing the set of declared, >> reachable APIs: >> >> * Code completion shows all of the APIs that are possible to use via >> dynamic dispatch, with their signatures so can fill in the right # of >> arguments, see the names of the parameters, see documentation, etc. >> * Indexing/refactoring/goto definition all point you to the declarations >> that could be the targets of dynamic dispatch >> >> The DynamicMemberLookup approach is better for cases where you don’t have >> a declaration of the member you want to access. I suspect that’s not the >> common case. >> >> >> Your points are valid, but the advantages for Objective-C don’t obviously >> translate to Swift. Note that ObjC (due to its heritage) has very long >> method names that are perhaps arguably designed to not conflict with each >> other often. Python doesn’t have this heritage, and it has much shorter >> names, which means that we’ll get a lot more conflicts and a lot less >> “safety" out of this. >> >> AnyObject lookup also depends on a strange set of scoping heuristics that >> was designed to be similar to Clang’s “header import” scope. It isn’t >> clear that this approach will work in Python, given that it doesn’t have an >> analogue of umbrella headers that import things that cross frameworks. >> >> >> Which approach do you think is the best way to handle the untyped >> “actually dynamic” case? >> >> >> AnyObject already exists in the language, and it fits the untyped >> “actually dynamic” case well. It does require having a declaration for the >> thing you want to reference, which I consider to be important: we can >> code-complete those declarations, goto-definition to see those >> declarations, index/refactor/look up documentation based on those >> declarations. >> >> I’d be more inclined to push for the ImplicitlyUnwrappedOptional -> >> Optional change if we did something to make AnyObject more prominent in >> Swift. >> >> >> I didn’t realize that you were thinking we would literally use AnyObject >> itself. I haven’t thought fully through it, but I think this will provide >> several problems: >> >> 1) You’re mushing all of the ObjC and Python world’s together, making the >> ObjC interop worse just because you’re doing some Python stuff too. >> 2) You’re introducing ambiguity: does “ao = [1,2,3]” create an NSArray or >> a Python array? How do string literals work? (The answer is obvious, >> Python loses). Maybe there is some really complicated bridging solution to >> these problems, but that causes its own massive complexity spiral. >> 3) You can’t realistically overload the Python operator set on AnyObject, >> which means you get a worse python experience. >> 4) AnyObject magic is currently limited to Apple platforms. This would >> bring its problems to other platforms like Linux. >> >> There are probably other issues, but I haven’t thought through it. >> >> >> but the names are there for code completion (and indexing, etc.) to work, >> and one could certainly imagine growing the importer to support Python’s >> typing >> annotations <https://docs.python.org/3/library/typing.html>. >> >> >> You’re basing this on the flawed assumption that local variables will >> pervasively have types, which I can’t imagine being the case. Even on >> "typable” API, I wouldn’t expect people to commonly get code completion >> results for reasons now explained in the proposal. >> >> >> Remember that one *does* get code completion results for member access >> into an AnyObject… lots of them… but the list filters down pretty fast when >> you type a few characters, and then you get a member access that’ll fill in >> stubs for (say) the arguments to the method you were trying to call. But >> you can’t get those code completion results without having declarations to >> complete to. >> >> >> Fair point, it’s unclear to me how useful this would be with python’s >> style of naming, but it could work. >> >> In truth, you don’t even need the compiler to be involved. The dynamic >> “subscript” operation could be implemented in a Swift library, and one >> could write a Python program to process a Python module and emit Swift >> wrappers that call into that subscript operation. You’ll get all of the >> tooling benefits with no compiler changes, and can tweak the wrapper >> generation however much you want, using typing annotations or other >> Python-specific information to create better wrappers over time. >> >> >> I’d love for you to sketch out how any of this works with an acceptable >> user experience, because I don't see anything concrete here. >> >> >> We don’t need the basic dynamic case in the language to do this >> experiment. Take the PyVal struct from the proposal. Now, write a Python >> script that loads some module Foo and uses Python’s inspect >> <https://docs.python.org/3/library/inspect.html> module to go find the >> classes, methods, etc., and pretty-print Swift code that uses PyVal. So >> this: >> >> def add_trick(self, trick): >> >> turns into >> >> extension PyVal { >> func add_trick(_ trick: PyVal) -> PyVal { >> /* do the magic to call into Python */ >> } >> } >> >> Using the inspect module, you can extract parameter names, default >> arguments, docstrings, and more to reflect the existing Python API as Swift >> API, packed into a bridging module. >> >> Note that we have a “flat” namespace of all Python methods on PyVal, >> which is basically what you get with AnyObject today. Swift tooling will >> provide code completion for member accesses into PyVal. Goto definition >> will jump to the pretty-printed declarations, which could have the >> docstrings formatted in comments and would show up in QuickHelp. The types >> are weak (everything is PyVal), but that’s what we expect from importing a >> dynamically-typed language. >> >> >> As I mention above, I expect this to expose significant scalability >> problems in the Swift compiler and it also defeats REPL/Playgrounds. Being >> able to use the Swift REPL is really important for Python programmers. >> >> -Chris >> >> >> _______________________________________________ >> swift-evolution mailing list >> swift-evolution@swift.org >> https://lists.swift.org/mailman/listinfo/swift-evolution >> >> > > _______________________________________________ > swift-evolution mailing list > swift-evolution@swift.org > https://lists.swift.org/mailman/listinfo/swift-evolution > >
_______________________________________________ swift-evolution mailing list swift-evolution@swift.org https://lists.swift.org/mailman/listinfo/swift-evolution