Re: [Python-Dev] typeshed for 3rd party packages
On Wed, Apr 22, 2015 at 11:26:14AM -0500, Ian Cordasco wrote: On a separate thread Cory provided an example of what the hints would look like for *part* of one function in the requests public functional API. While our API is outwardly simple, the values we accept in certain cases are actually non-trivially represented. Getting the hints *exactly* correct would be extraordinarily difficult. I don't think you need to get them exactly correct. The type-checker does two things: (1) catch type errors involving types which should not be allowed; (2) allow code which involves types which should be allowed. If the type hints are wrong, there are two errors: false positives, when code which should be allowed is flagged as a type error; and false negatives, when code which should be flagged as an error is not. Ideally, there should be no false positives. But false negatives are not so important, since you will still be doing runtime checks. All that means is that the static type-checker will be a little less capable of picking up type errors at compile time. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages
On Fri, Apr 24, 2015 at 03:44:45PM +0100, Cory Benfield wrote: On 24 April 2015 at 15:21, Steven D'Aprano st...@pearwood.info wrote: If the type hints are wrong, there are two errors: false positives, when code which should be allowed is flagged as a type error; and false negatives, when code which should be flagged as an error is not. Ideally, there should be no false positives. But false negatives are not so important, since you will still be doing runtime checks. All that means is that the static type-checker will be a little less capable of picking up type errors at compile time. I think that's a rational view that will not be shared as widely as I'd like. I can't tell if you are agreeing with me, or disagreeing. The above sentence seems to be agreeing with me, but you later end your message with do it properly or not at all which disagrees. So I'm confused. Given that the purpose of a type checker is to catch bugs caused by passing incorrectly typed objects to a function, it seems entirely reasonable to me to raise a bug against a type hint that allows code that was of an incorrect type where that incorrectness *could* have been caught by the type hint. Of course it is reasonable for people to submit bug reports to do with the type hints. And it is also reasonable for the package maintainer to reject the bug report as Won't Fix if it makes the type hint too complex. The beauty of gradual typing is that unlike Java or Haskell, you can choose to have as little or as much type checking as works for you. You don't have to satisfy the type checker over the entire program before the code will run, you only need check the parts you want to check. Extending from that into the general ratio of reports that are actually bugs versus reports that are errors on the part of the reporter, I can assume that plenty of people will raise bug reports for incorrect cases as well. Okay. Do you get many false positive bug reports for your tests too? From the perspective of sustainable long-term maintenance, I think the only way to do type hints is to have them be sufficiently exhaustive that a user would have to actively *try* to hit an edge case false negative. I believe that requests' API is too dynamically-typed to fit into that category at this time. I think we agree that, static type checks or no static type checks, requests is going to need to do runtime type checks. So why does it matter if it misses a few type errors at compile time? I think we're all in agreement that for extremely dynamic code like requests, you may not get as much value from static type checks as some other libraries or applications. You might even decide that you get no value at all. Okay, that's fine. I'm just suggesting that you don't have just two choices, all or nothing. The whole point of gradual typing is to give developers more options. PS: I should mention that, as Gary Bernhardt pointed out at PyCon, people often believe (incorrectly) that types are a replacement for tests. They *can* be a replacement for tests. You don't see Java or Haskell programmers writing unit tests to check that their code never tries to add a string to a float. Even if they could write such as test, they don't bother because the type checker will catch that sort of error. The situation in Python is a bit different, and as Antoine points out, libraries cannot rely on their callers obeying the type restrictions of the public API. (Private functions are different -- if you call my private function with the wrong type and blow up your computer, it's your own fault.) For libraries, I see type checks as complementing tests, not replacing them. But for application code, type checks may replace unit tests, provided that nobody checks in production code until both the type checker and the unit tests pass. If you work under that rule, there's no point in having the unit tests check what the type checker already tested. For that reason I feel like underspecified type hints are something of an attractive nuisance. Again, I really think this is a case of do it properly or not at all. In my opinion, underspecified type hints are no more of an attractive nuisance than a test suite which doesn't test enough. Full coverage is great, but 10% coverage is better than 5% coverage, which is better than nothing. That applies whether we are talking about tests, type checks, or documentation. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages
On 24 April 2015 at 15:21, Steven D'Aprano st...@pearwood.info wrote: If the type hints are wrong, there are two errors: false positives, when code which should be allowed is flagged as a type error; and false negatives, when code which should be flagged as an error is not. Ideally, there should be no false positives. But false negatives are not so important, since you will still be doing runtime checks. All that means is that the static type-checker will be a little less capable of picking up type errors at compile time. I think that's a rational view that will not be shared as widely as I'd like. Given that the purpose of a type checker is to catch bugs caused by passing incorrectly typed objects to a function, it seems entirely reasonable to me to raise a bug against a type hint that allows code that was of an incorrect type where that incorrectness *could* have been caught by the type hint. Extending from that into the general ratio of reports that are actually bugs versus reports that are errors on the part of the reporter, I can assume that plenty of people will raise bug reports for incorrect cases as well. From the perspective of sustainable long-term maintenance, I think the only way to do type hints is to have them be sufficiently exhaustive that a user would have to actively *try* to hit an edge case false negative. I believe that requests' API is too dynamically-typed to fit into that category at this time. PS: I should mention that, as Gary Bernhardt pointed out at PyCon, people often believe (incorrectly) that types are a replacement for tests. For that reason I feel like underspecified type hints are something of an attractive nuisance. Again, I really think this is a case of do it properly or not at all. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages
On Wed, Apr 22, 2015 at 10:43 AM, Guido van Rossum gu...@python.org wrote: For Requests, it looks like it may be better not to have stubs at all. Can you expand on this? Why would Requests be any different than any other module/package? As for versioning, I think stub files would absolutely have to declare the appropriate version(s) to which they apply (probably via embedded directives), so type checkers can ignore them when necessary. That also means that type checkers must be able to figure out the version of the package used by the application being analyzed. Not sure I'm being too clear, so I will provide an example. I have app myapp which imports module yourmod v 1.2.7. The yourmod author hasn't yet provided type annotations, so someone else contributed a stub to the typeshed. Time passes and a new version of yourmod comes out, v 2.0.0. (Semantic versioning tells us the API has changed in an incompatible way because of the major version bump.) I decide I need some of its new features and update myapp. There is no new stub file in the typeshed yet. When I run my fancy type checker (someone suggested I will shortly have 50 to choose from!), it needs to recognize that the stub no longer matches the version of yourmod I am using, and must ignore it. Does that suggest the typeshed needs some sort of structure which allows all versions of stubs for the same package to be gathered together? My apologies if I'm following along way behind the curve. Skip ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages
On Wed, Apr 22, 2015 at 11:12 AM, Skip Montanaro skip.montan...@gmail.com wrote: On Wed, Apr 22, 2015 at 10:43 AM, Guido van Rossum gu...@python.org wrote: For Requests, it looks like it may be better not to have stubs at all. Can you expand on this? Why would Requests be any different than any other module/package? On a separate thread Cory provided an example of what the hints would look like for *part* of one function in the requests public functional API. While our API is outwardly simple, the values we accept in certain cases are actually non-trivially represented. Getting the hints *exactly* correct would be extraordinarily difficult. As for versioning, I think stub files would absolutely have to declare the appropriate version(s) to which they apply (probably via embedded directives), so type checkers can ignore them when necessary. That also means that type checkers must be able to figure out the version of the package used by the application being analyzed. Not sure I'm being too clear, so I will provide an example. I have app myapp which imports module yourmod v 1.2.7. The yourmod author hasn't yet provided type annotations, so someone else contributed a stub to the typeshed. Time passes and a new version of yourmod comes out, v 2.0.0. (Semantic versioning tells us the API has changed in an incompatible way because of the major version bump.) I decide I need some of its new features and update myapp. There is no new stub file in the typeshed yet. When I run my fancy type checker (someone suggested I will shortly have 50 to choose from!), it needs to recognize that the stub no longer matches the version of yourmod I am using, and must ignore it. Which of course also assumes that the author of that library is even using Semantic Versioning (which is not a universal release strategy, even in the Ruby community). I understand where you're coming from, but I think this is a reason as to why typeshed as a catch-all for third party type-hints may not be feasible. Does that suggest the typeshed needs some sort of structure which allows all versions of stubs for the same package to be gathered together? My apologies if I'm following along way behind the curve. No need to apologize. =) As the other maintainer of requests, I think having hints *might* help some developers, but looking at what Cory generated (which looks to be valid), I'm wondering about something else with Type Hints. I've heard several people say Just create an aliased type for the hint so it's shorter! but doesn't that mean we then have to document that alias for our users? I mean if the IDE suggests that the developer use XYZ for this parameter and there's no explanation for XYZ actually is (in the IDE), doesn't this just add a lot more maintenance to adding hints? Maintainers now have to: - Keep the stubs up-to-date - Document the stubs (and if the stubs are in typeshed, does $MyPackage link to the docs in typeshed to avoid users filing bugs on $MyPackage's issue tracker?) - Version the stubs (assuming they're maintained in a third-party location, e.g., typeshed) Don't get me wrong. I really like the idea of moving towards Type Hints. I'm not even particularly against adding type hints for Requests to typeshed. I'm just hesitant that it will be easy to make them entirely accurate. Cheers, Ian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages
On Wed, Apr 22, 2015 at 11:30 AM, Skip Montanaro skip.montan...@gmail.com wrote: On Wed, Apr 22, 2015 at 11:26 AM, Ian Cordasco graffatcolmin...@gmail.com wrote: On a separate thread Cory provided an example of what the hints would look like for *part* of one function in the requests public functional API. Thanks. That encouraged me to look around for recent posts from Cory. Wow... You're welcome! And yeah. That union that Cory posted was for *one* parameter if I remember correctly. I won't speak for Cory, but I'm not against the type hints in 484 but they will be difficult for us as a project. They'll be marginally less difficult for me in a different project of mine. I also wonder about importing type definitions from other packages. The Requests-Toolbelt adds a few features that are enhanced versions of what's already in Requests. I can think of a few type hints that we might create to represent certain parameters, but I don't want to have to copy those for the features in the Requests-Toolbelt. I would expect this to Just Work, but I wonder if anyone else has considered the possibility of this being a need. Cheers, Ian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages
I definitely think that we shouldn't jump the gun here and tread carefully. Both Marc-André and Cory brought up good things to watch out for. For closed-source software the only way to obtain stubs is presumably from the author, if they care. As Gregory Smith said in another thread, the tooling will have to prove itself to the point where library developers *want* to use it. For Requests, it looks like it may be better not to have stubs at all. You can assign any and all tracker issues asking for stubs to me, I'll gladly explain why in Requests' case it's better to live without stubs. On Wed, Apr 22, 2015 at 4:22 AM, Cory Benfield c...@lukasa.co.uk wrote: On 22 April 2015 at 11:46, M.-A. Lemburg m...@egenix.com wrote: Unlike with translations, where missing or poor ones don't have much effect on the usefulness of the software, a type checker would complain loudly and probably show lots of false positives (if you read a type bug as positive), causing the usual complaints by users to the software authors. I don't really think that users would instead complain to the type checker authors or find the actual source of the problem which are the broken stub files. This is my expectation as well. Requests receives bug reports for bugs that were introduced by downstream packagers, or for bugs that are outright in unrelated projects. I field IRC questions about 'requests bugs' that are actually bugs in the web application on the other end of the HTTP connection! I can *guarantee* that if a stub file is bad, I'll get told, not the author of the stub file. OTOH, if the type checkers are written in a way where they can detect authoritative stubs compared to non-authoritative ones and point users to the possible type stub file problem, this could be resolved, I guess. The stub files would then need an authoritative flag and probably also be versioned to get this working. This would be great: +1. As I've explained above, in my experience, people (*) often first go to the authors of the software and not do research to find out that the tool they were using has a problem (via the non-authoritative stub files it's using). (*) More experienced users of pylint like tools will probably think twice due to the many false positives these tools tend to generate. I'm not sure whether people using type checkers would have the same approach, though, esp. not if they are coming from the land of statically typed languages. I can back this up, as can a search through Requests' past GitHub issues. pylint in particular has caused me pain due to at least one GitHub issue that was nothing more than a dump of pylint output when run over Requests', where *every single line* was a false positive. This ends up having negative network effects as well, because the next time someone opens a GitHub issue with the word 'pylint' in it I'm simply going to close it, rather than waste another 45 minutes visually confirming that every line is a false positive. I worry that stub files will have the same problems. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages
On Wed, Apr 22, 2015 at 9:12 AM, Skip Montanaro skip.montan...@gmail.com wrote: On Wed, Apr 22, 2015 at 10:43 AM, Guido van Rossum gu...@python.org wrote: For Requests, it looks like it may be better not to have stubs at all. Can you expand on this? Why would Requests be any different than any other module/package? Did you see the crazy union that Cory came up with for one of the parameters of get() in another thread? Requests assumes duck typing for most of its arguments *and* it handles different forms of many arguments (e.g. you can specify an iterable or a mapping). Requests is different for a number of reasons; it has evolved for years to support many different use cases with a few methods, so it is using every trick in the book from dynamic typing. Type checking (using the current state of the tools) is probably more useful for business logic code than for library code. We'll iterate over the next few Python versions (and even while Python 3.5 is out we can tweak typing.py because it's provisional in the PEP 411 sense) and eventually it may be possible to provide stubs for Requests. But since PEP 484 doesn't address duck typing it's simply too early. (As I said elsewhere, duck typing is an important next step, but we need PEP 484 as the foundation first). As for versioning, I think stub files would absolutely have to declare the appropriate version(s) to which they apply (probably via embedded directives), so type checkers can ignore them when necessary. That also means that type checkers must be able to figure out the version of the package used by the application being analyzed. Not sure I'm being too clear, so I will provide an example. I have app myapp which imports module yourmod v 1.2.7. The yourmod author hasn't yet provided type annotations, so someone else contributed a stub to the typeshed. Time passes and a new version of yourmod comes out, v 2.0.0. (Semantic versioning tells us the API has changed in an incompatible way because of the major version bump.) I decide I need some of its new features and update myapp. There is no new stub file in the typeshed yet. When I run my fancy type checker (someone suggested I will shortly have 50 to choose from!), it needs to recognize that the stub no longer matches the version of yourmod I am using, and must ignore it. Does that suggest the typeshed needs some sort of structure which allows all versions of stubs for the same package to be gathered together? My apologies if I'm following along way behind the curve. No, I think you can start filing (or adding your view to) issues with the typeshed tracker: https://github.com/JukkaL/typeshed/issues -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages
On Wed, Apr 22, 2015 at 9:26 AM, Ian Cordasco graffatcolmin...@gmail.com wrote: As the other maintainer of requests, I think having hints *might* help some developers, but looking at what Cory generated (which looks to be valid), I'm wondering about something else with Type Hints. I've heard several people say Just create an aliased type for the hint so it's shorter! but doesn't that mean we then have to document that alias for our users? I mean if the IDE suggests that the developer use XYZ for this parameter and there's no explanation for XYZ actually is (in the IDE), doesn't this just add a lot more maintenance to adding hints? Maintainers now have to: - Keep the stubs up-to-date - Document the stubs (and if the stubs are in typeshed, does $MyPackage link to the docs in typeshed to avoid users filing bugs on $MyPackage's issue tracker?) - Version the stubs (assuming they're maintained in a third-party location, e.g., typeshed) Don't get me wrong. I really like the idea of moving towards Type Hints. I'm not even particularly against adding type hints for Requests to typeshed. I'm just hesitant that it will be easy to make them entirely accurate. To be useful for the users of a package, type aliases need to be exported by the package, which means that the package itself grows a dependency on typing.py. You could probably make that a conditional dependency, e.g. try: from typing import Union, Tuple, AnyStr, Optional HeaderTuple = Union[Tuple[AnyStr, AnyStr], Tuple[AnyStr, AnyStr, Optional[AnyStr]]] # etc. except ImportError: pass # Don't define type aliases and use a stub file for the actual signatures. User code that itself has a hard dependency on typing could import and use the type aliases unconditionally; user code with a conditional dependency on typing should stick to stubs (or similar import hacks). If you use type hints this way you should probably maintain the stubs as part of your package (as .pyi files living alongside the .py files) so that you don't have to deal with typeshed being out of date. There are many other possible workflows; we haven't discovered the best one(s) yet. It's a work in progress. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages
On 21.04.2015 18:08, Guido van Rossum wrote: On Tue, Apr 21, 2015 at 12:33 AM, M.-A. Lemburg m...@egenix.com wrote: On 21.04.2015 05:37, Guido van Rossum wrote: On Mon, Apr 20, 2015 at 4:41 PM, Jack Diederich jackd...@gmail.com wrote: * Uploading stubs for other people's code is a terrible idea. Who do I contact when I update the interface to my library? The random Joe who helped by uploading annotations three months ago and then quit the internet? I don't even want to think about people maliciously adding stubs to PyPI. We're certainly not planning to let arbitrary people upload stubs for arbitrary code to PyPI that will automatically be used by the type checkers. (We can't stop people from publishing their stubs, just as you can't stop people from writing blog posts or stackoverflow answers with examples for your library.) The actual plan is to have a common repository of stubs (a prototype exists at https://github.com/JukkaL/typeshed) but we also plan some kind of submission review. I've proposed that when submitting stubs for a package you don't own, the typeshed owners ask the package owner what their position is, and we expect the answers to fall on the following spectrum: - I don't want stubs uploaded for my package - I'll write the stubs myself - I want to review all stubs that are uploaded for my package before they are accepted - Please go ahead and add stubs for my package and let me know when they're ready - Go ahead, I trust you This seems a reasonable due diligence policy that avoids the scenarios you're worried about. (Of course if you refuse stubs a black market for stubs might spring into existence. That sounds kind of exciting... :-) Hmm, that's the first time I've heard about this. I agree with Jack that it's a terrible idea to allow this for 3rd party packages. If people want to contribute stubs, they should contribute them to the package in the usual ways, not in a side channel. The important part missing in the above setup is maintenance and to some extent an external change of the API definitions. Both require active participation in the package project, not the separated setup proposed above, to be effective and actually work out in the long run. For the stdlib, typeshed looks like a nice idea to spread the workload. I hesitate to speak for others, but IIUC the reason why typeshed was started is that companies like PyCharm and Google (and maybe others) are *already* creating their own stubs for 3rd party packages, because they have a need to type-check code that *uses* 3rd party packages. Their use cases are otherwise quite different (the user code type-checked by PyCharm is that of PyCharm users, and the code type-checked by Google is their own proprietary code) but they both find themselves needing stubs for commonly used 3rd party packages. mypy finds itself in a similar position. Think of it this way. Suppose you wrote an app that downloaded some files from the web using the popular Requests package. Now suppose you wanted to run mypy over your app. You're willing to do the work of adding signatures to your own app, and presumably there are stubs for those parts of the stdlib that you're using, but without stubs for Requests, mypy won't do a very good job type-checking your calls into Requests. It's not rocket science to come up with stubs for Requests (there aren't that many classes and methods) but the Requests package is in maintenance mode, and while they respond quickly to security issues they might take their time to release a new version that includes your stub files, and until there are a lot of people clamoring for stubs for Requests, they might not care at all. For projects in maintenance mode, it does make sense indeed. For active ones, I think you'd quickly run into a situation similar to translation projects: there are always parts which haven't been translated yet or where the translation no longer matches the original intent. Unlike with translations, where missing or poor ones don't have much effect on the usefulness of the software, a type checker would complain loudly and probably show lots of false positives (if you read a type bug as positive), causing the usual complaints by users to the software authors. I don't really think that users would instead complain to the type checker authors or find the actual source of the problem which are the broken stub files. OTOH, if the type checkers are written in a way where they can detect authoritative stubs compared to non-authoritative ones and point users to the possible type stub file problem, this could be resolved, I guess. The stub files would then need an authoritative flag and probably also be versioned to get this working. So what does Requests have to lose if, instead of including the stubs in Requests, they let the typeshed people distribute stubs for Requests? Presumably having the stubs in typeshed means that PyCharm
Re: [Python-Dev] typeshed for 3rd party packages
On 22 April 2015 at 11:46, M.-A. Lemburg m...@egenix.com wrote: Unlike with translations, where missing or poor ones don't have much effect on the usefulness of the software, a type checker would complain loudly and probably show lots of false positives (if you read a type bug as positive), causing the usual complaints by users to the software authors. I don't really think that users would instead complain to the type checker authors or find the actual source of the problem which are the broken stub files. This is my expectation as well. Requests receives bug reports for bugs that were introduced by downstream packagers, or for bugs that are outright in unrelated projects. I field IRC questions about 'requests bugs' that are actually bugs in the web application on the other end of the HTTP connection! I can *guarantee* that if a stub file is bad, I'll get told, not the author of the stub file. OTOH, if the type checkers are written in a way where they can detect authoritative stubs compared to non-authoritative ones and point users to the possible type stub file problem, this could be resolved, I guess. The stub files would then need an authoritative flag and probably also be versioned to get this working. This would be great: +1. As I've explained above, in my experience, people (*) often first go to the authors of the software and not do research to find out that the tool they were using has a problem (via the non-authoritative stub files it's using). (*) More experienced users of pylint like tools will probably think twice due to the many false positives these tools tend to generate. I'm not sure whether people using type checkers would have the same approach, though, esp. not if they are coming from the land of statically typed languages. I can back this up, as can a search through Requests' past GitHub issues. pylint in particular has caused me pain due to at least one GitHub issue that was nothing more than a dump of pylint output when run over Requests', where *every single line* was a false positive. This ends up having negative network effects as well, because the next time someone opens a GitHub issue with the word 'pylint' in it I'm simply going to close it, rather than waste another 45 minutes visually confirming that every line is a false positive. I worry that stub files will have the same problems. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages (was: Type hints -- a mediocre programmer's reaction)
On 21.04.2015 05:37, Guido van Rossum wrote: On Mon, Apr 20, 2015 at 4:41 PM, Jack Diederich jackd...@gmail.com wrote: * Uploading stubs for other people's code is a terrible idea. Who do I contact when I update the interface to my library? The random Joe who helped by uploading annotations three months ago and then quit the internet? I don't even want to think about people maliciously adding stubs to PyPI. We're certainly not planning to let arbitrary people upload stubs for arbitrary code to PyPI that will automatically be used by the type checkers. (We can't stop people from publishing their stubs, just as you can't stop people from writing blog posts or stackoverflow answers with examples for your library.) The actual plan is to have a common repository of stubs (a prototype exists at https://github.com/JukkaL/typeshed) but we also plan some kind of submission review. I've proposed that when submitting stubs for a package you don't own, the typeshed owners ask the package owner what their position is, and we expect the answers to fall on the following spectrum: - I don't want stubs uploaded for my package - I'll write the stubs myself - I want to review all stubs that are uploaded for my package before they are accepted - Please go ahead and add stubs for my package and let me know when they're ready - Go ahead, I trust you This seems a reasonable due diligence policy that avoids the scenarios you're worried about. (Of course if you refuse stubs a black market for stubs might spring into existence. That sounds kind of exciting... :-) Hmm, that's the first time I've heard about this. I agree with Jack that it's a terrible idea to allow this for 3rd party packages. If people want to contribute stubs, they should contribute them to the package in the usual ways, not in a side channel. The important part missing in the above setup is maintenance and to some extent an external change of the API definitions. Both require active participation in the package project, not the separated setup proposed above, to be effective and actually work out in the long run. For the stdlib, typesched looks like a nice idea to spread the workload. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 21 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] typeshed for 3rd party packages (was: Type hints -- a mediocre programmer's reaction)
On Tue, Apr 21, 2015 at 12:33 AM, M.-A. Lemburg m...@egenix.com wrote: On 21.04.2015 05:37, Guido van Rossum wrote: On Mon, Apr 20, 2015 at 4:41 PM, Jack Diederich jackd...@gmail.com wrote: * Uploading stubs for other people's code is a terrible idea. Who do I contact when I update the interface to my library? The random Joe who helped by uploading annotations three months ago and then quit the internet? I don't even want to think about people maliciously adding stubs to PyPI. We're certainly not planning to let arbitrary people upload stubs for arbitrary code to PyPI that will automatically be used by the type checkers. (We can't stop people from publishing their stubs, just as you can't stop people from writing blog posts or stackoverflow answers with examples for your library.) The actual plan is to have a common repository of stubs (a prototype exists at https://github.com/JukkaL/typeshed) but we also plan some kind of submission review. I've proposed that when submitting stubs for a package you don't own, the typeshed owners ask the package owner what their position is, and we expect the answers to fall on the following spectrum: - I don't want stubs uploaded for my package - I'll write the stubs myself - I want to review all stubs that are uploaded for my package before they are accepted - Please go ahead and add stubs for my package and let me know when they're ready - Go ahead, I trust you This seems a reasonable due diligence policy that avoids the scenarios you're worried about. (Of course if you refuse stubs a black market for stubs might spring into existence. That sounds kind of exciting... :-) Hmm, that's the first time I've heard about this. I agree with Jack that it's a terrible idea to allow this for 3rd party packages. If people want to contribute stubs, they should contribute them to the package in the usual ways, not in a side channel. The important part missing in the above setup is maintenance and to some extent an external change of the API definitions. Both require active participation in the package project, not the separated setup proposed above, to be effective and actually work out in the long run. For the stdlib, typeshed looks like a nice idea to spread the workload. I hesitate to speak for others, but IIUC the reason why typeshed was started is that companies like PyCharm and Google (and maybe others) are *already* creating their own stubs for 3rd party packages, because they have a need to type-check code that *uses* 3rd party packages. Their use cases are otherwise quite different (the user code type-checked by PyCharm is that of PyCharm users, and the code type-checked by Google is their own proprietary code) but they both find themselves needing stubs for commonly used 3rd party packages. mypy finds itself in a similar position. Think of it this way. Suppose you wrote an app that downloaded some files from the web using the popular Requests package. Now suppose you wanted to run mypy over your app. You're willing to do the work of adding signatures to your own app, and presumably there are stubs for those parts of the stdlib that you're using, but without stubs for Requests, mypy won't do a very good job type-checking your calls into Requests. It's not rocket science to come up with stubs for Requests (there aren't that many classes and methods) but the Requests package is in maintenance mode, and while they respond quickly to security issues they might take their time to release a new version that includes your stub files, and until there are a lot of people clamoring for stubs for Requests, they might not care at all. So what does Requests have to lose if, instead of including the stubs in Requests, they let the typeshed people distribute stubs for Requests? Presumably having the stubs in typeshed means that PyCharm and mypy (and the 50 other type-checkers that are being written right now :-) can give better diagnostics for code using Requests, and once in a while this may save a user of Requests from doing something dumb and blaming Requests. The only downside would be if something was missing in the stubs and users would get incorrect error messages from their favorite type checker. But it's a long stretch to see this rain down on Requests' reputation -- more likely the type checker will be blamed, so type checker authors/distributors will be vigilant before distributing stubs. OTOH if you prefer to make and distribute your own stubs, type checkers will use those, and there won't be a need to include stubs in typeshed when a package already provides stubs. And if you really don't want anything to do with stubs for your package, just tell the typeshed owners and your wish will be respected. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev
