[Freeipa-devel] Re: [RFC] Static type checking for FreeIPA (Mypy)
On 2017-08-10 10:24, Tomas Krizek wrote: > I like the idea of specifying types and it can also be helpful when > reading the code, because it serves as additional documentation. For > this reason, I also think it's important not to use stub files. > > The Python 3.6 syntax looks awesome and I'd like to use it. However, > since we have to support Python 2, we'd have to use the comments. I > don't really like that, because it's not a part of the in-line code. > There's also a big downside -- you can't use partial annotations with > comments. I'm sorry for being the bringer of bad news. Although the Python 3 syntax (supported since Python 3.0!) looks awesome, you don't want to use them in FreeIPA. Trust me. :) Type annotation have a negative impact on both memory consumption and import time. After all you end up with thousands of additional Python objects in memory. The objects have to be constructed, validated and attached to __annotations__ attribute. Christian -- Christian Heimes Senior Software Engineer, Identity Management and Platform Security Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander signature.asc Description: OpenPGP digital signature ___ FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org
[Freeipa-devel] Re: [RFC] Static type checking for FreeIPA (Mypy)
On 2017-08-10 03:26, Fraser Tweedale wrote: > On Wed, Aug 09, 2017 at 10:18:33AM +0200, Christian Heimes via FreeIPA-devel > wrote: >> On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote: >>> Hi team, >>> >>> At PyCon Australia on the weekend I was reminded of PEP-484 type >>> hinting** and the Mypy type checker for Python. >>> >>> With focus of FreeIPA project shifting more towards stability, >>> quality and maintainability, and with Python 3 porting work nearly >>> wrapped up, now is the time to think about how we can get more >>> confidence in our code not just from tests, but from the code >>> itself. Static checking of annotated types can help us there, and >>> Mypy can let us begin to do this when writing new code or >>> refactoring old code. Furthermore there is a benefit for IDE-users >>> where plugins can use type annotations to provide better completion >>> suggestions, etc. For an overview of Mypy please see the PyCon AU >>> talk[1] or the docs[2]. >>> >>> [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ >>> [2] http://mypy.readthedocs.io/en/latest/index.html >>> >>> So, what's the plan? Alongside my other tasks, I'm going to start >>> looking at how we could use Mypy in FreeIPA CI, and see what it is >>> like using types in some of the areas I'm familiar with e.g. >>> ipalib.x509. Based on my findings I'll update the team on the wins >>> and challenges and we can decide how to proceed from there. >> >> Felipe ask me about typing and Mypy a couple of weeks ago. It's a good >> idea and we should do it. But I advise against typing information in the >> source code. FreeIPA should use external stub files for two reasons. >> First of all it is required to stay compatible with Python 2. And more >> importantly it's faster. FreeIPA's CLI scripts already take several >> hundred milliseconds to execute. Typing would slow them down even further. >> > I disagree with using stub files. Types should be declared where > the functions are defined. Types are documentation and proximity is > important (for humans). >> > Fortunately, Mypy supports "type comments" in addition to PEP 3107 > function annotations. Mypy groks them but CPython will treat them > as comments and discard. This allows us to use type hints with no > runtime cost for the CLI scripts. This should be an issue with the right editor. For example PyCharm supports stub files and displays type annotations. But I'm ok with both external stub files and full inline comments. Since the FreeIPA team is going to be both author and consumer of type annotations, the inline comment style might be more appropriate. Unfortunately inline coments can't be automated with stubgen. :/ Christian -- Christian Heimes Senior Software Engineer, Identity Management and Platform Security Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander signature.asc Description: OpenPGP digital signature ___ FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org
[Freeipa-devel] Re: [RFC] Static type checking for FreeIPA (Mypy)
On Wed, Aug 09, 2017 at 10:18:33AM +0200, Christian Heimes via FreeIPA-devel wrote: > On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote: > > Hi team, > > > > At PyCon Australia on the weekend I was reminded of PEP-484 type > > hinting** and the Mypy type checker for Python. > > > > With focus of FreeIPA project shifting more towards stability, > > quality and maintainability, and with Python 3 porting work nearly > > wrapped up, now is the time to think about how we can get more > > confidence in our code not just from tests, but from the code > > itself. Static checking of annotated types can help us there, and > > Mypy can let us begin to do this when writing new code or > > refactoring old code. Furthermore there is a benefit for IDE-users > > where plugins can use type annotations to provide better completion > > suggestions, etc. For an overview of Mypy please see the PyCon AU > > talk[1] or the docs[2]. > > > > [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ > > [2] http://mypy.readthedocs.io/en/latest/index.html > > > > So, what's the plan? Alongside my other tasks, I'm going to start > > looking at how we could use Mypy in FreeIPA CI, and see what it is > > like using types in some of the areas I'm familiar with e.g. > > ipalib.x509. Based on my findings I'll update the team on the wins > > and challenges and we can decide how to proceed from there. > > Felipe ask me about typing and Mypy a couple of weeks ago. It's a good > idea and we should do it. But I advise against typing information in the > source code. FreeIPA should use external stub files for two reasons. > First of all it is required to stay compatible with Python 2. And more > importantly it's faster. FreeIPA's CLI scripts already take several > hundred milliseconds to execute. Typing would slow them down even further. > I disagree with using stub files. Types should be declared where the functions are defined. Types are documentation and proximity is important (for humans). Fortunately, Mypy supports "type comments" in addition to PEP 3107 function annotations. Mypy groks them but CPython will treat them as comments and discard. This allows us to use type hints with no runtime cost for the CLI scripts. > It's rather easy to auto-generate stub files -- assuming you are running > on Fedora and have all Python 3 dependencies installed: > > $ sudo dnf install python3-mypy > $ echo "api.bootstrap(ra_plugin='dogtag')" >> ipalib/__init__.py > $ mkdir out > $ PYTHONPATH=. stubgen --recursive ipaclient ipalib ipaplatform > ipapython ipaserver > > The api.bootstrap() call is required. Otherwise stubgen cannot import a > bunch of plugin files. > Thanks for this additional info. Cheers, Fraser ___ FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org
[Freeipa-devel] Re: [RFC] Static type checking for FreeIPA (Mypy)
On 2017-08-09 10:18, Christian Heimes via FreeIPA-devel wrote: > On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote: >> Hi team, >> >> At PyCon Australia on the weekend I was reminded of PEP-484 type >> hinting** and the Mypy type checker for Python. >> >> With focus of FreeIPA project shifting more towards stability, >> quality and maintainability, and with Python 3 porting work nearly >> wrapped up, now is the time to think about how we can get more >> confidence in our code not just from tests, but from the code >> itself. Static checking of annotated types can help us there, and >> Mypy can let us begin to do this when writing new code or >> refactoring old code. Furthermore there is a benefit for IDE-users >> where plugins can use type annotations to provide better completion >> suggestions, etc. For an overview of Mypy please see the PyCon AU >> talk[1] or the docs[2]. >> >> [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ >> [2] http://mypy.readthedocs.io/en/latest/index.html >> >> So, what's the plan? Alongside my other tasks, I'm going to start >> looking at how we could use Mypy in FreeIPA CI, and see what it is >> like using types in some of the areas I'm familiar with e.g. >> ipalib.x509. Based on my findings I'll update the team on the wins >> and challenges and we can decide how to proceed from there. > > Felipe ask me about typing and Mypy a couple of weeks ago. It's a good > idea and we should do it. But I advise against typing information in the > source code. FreeIPA should use external stub files for two reasons. > First of all it is required to stay compatible with Python 2. And more > importantly it's faster. FreeIPA's CLI scripts already take several > hundred milliseconds to execute. Typing would slow them down even further. > > It's rather easy to auto-generate stub files -- assuming you are running > on Fedora and have all Python 3 dependencies installed: Small correction:, stubgen in Fedora 26 is broken [1]. Try this: https://gist.github.com/tiran/a281aa3baf9ea39e1d02800c1a7f8ea6 Christian [1] https://bugzilla.redhat.com/show_bug.cgi?id=1479697 -- Christian Heimes Senior Software Engineer, Identity Management and Platform Security Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander signature.asc Description: OpenPGP digital signature ___ FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org
[Freeipa-devel] Re: [RFC] Static type checking for FreeIPA (Mypy)
On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote: > Hi team, > > At PyCon Australia on the weekend I was reminded of PEP-484 type > hinting** and the Mypy type checker for Python. > > With focus of FreeIPA project shifting more towards stability, > quality and maintainability, and with Python 3 porting work nearly > wrapped up, now is the time to think about how we can get more > confidence in our code not just from tests, but from the code > itself. Static checking of annotated types can help us there, and > Mypy can let us begin to do this when writing new code or > refactoring old code. Furthermore there is a benefit for IDE-users > where plugins can use type annotations to provide better completion > suggestions, etc. For an overview of Mypy please see the PyCon AU > talk[1] or the docs[2]. > > [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ > [2] http://mypy.readthedocs.io/en/latest/index.html > > So, what's the plan? Alongside my other tasks, I'm going to start > looking at how we could use Mypy in FreeIPA CI, and see what it is > like using types in some of the areas I'm familiar with e.g. > ipalib.x509. Based on my findings I'll update the team on the wins > and challenges and we can decide how to proceed from there. Felipe ask me about typing and Mypy a couple of weeks ago. It's a good idea and we should do it. But I advise against typing information in the source code. FreeIPA should use external stub files for two reasons. First of all it is required to stay compatible with Python 2. And more importantly it's faster. FreeIPA's CLI scripts already take several hundred milliseconds to execute. Typing would slow them down even further. It's rather easy to auto-generate stub files -- assuming you are running on Fedora and have all Python 3 dependencies installed: $ sudo dnf install python3-mypy $ echo "api.bootstrap(ra_plugin='dogtag')" >> ipalib/__init__.py $ mkdir out $ PYTHONPATH=. stubgen --recursive ipaclient ipalib ipaplatform ipapython ipaserver The api.bootstrap() call is required. Otherwise stubgen cannot import a bunch of plugin files. Christian -- Christian Heimes Senior Software Engineer, Identity Management and Platform Security Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander signature.asc Description: OpenPGP digital signature ___ FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org
