[Freeipa-devel] Re: [RFC] Static type checking for FreeIPA (Mypy)

2017-08-09 Thread Fraser Tweedale via FreeIPA-devel 
On Wed, Aug 09, 2017 at 10:18:33AM +0200, Christian Heimes via FreeIPA-devel 
wrote:
> On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote:
> > Hi team,
> > 
> > At PyCon Australia on the weekend I was reminded of PEP-484 type
> > hinting** and the Mypy type checker for Python.
> > 
> > With focus of FreeIPA project shifting more towards stability,
> > quality and maintainability, and with Python 3 porting work nearly
> > wrapped up, now is the time to think about how we can get more
> > confidence in our code not just from tests, but from the code
> > itself.  Static checking of annotated types can help us there, and
> > Mypy can let us begin to do this when writing new code or
> > refactoring old code.  Furthermore there is a benefit for IDE-users
> > where plugins can use type annotations to provide better completion
> > suggestions, etc.  For an overview of Mypy please see the PyCon AU
> > talk[1] or the docs[2].
> > 
> > [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ
> > [2] http://mypy.readthedocs.io/en/latest/index.html
> > 
> > So, what's the plan?  Alongside my other tasks, I'm going to start
> > looking at how we could use Mypy in FreeIPA CI, and see what it is
> > like using types in some of the areas I'm familiar with e.g.
> > ipalib.x509.  Based on my findings I'll update the team on the wins
> > and challenges and we can decide how to proceed from there.
> 
> Felipe ask me about typing and Mypy a couple of weeks ago. It's a good
> idea and we should do it. But I advise against typing information in the
> source code. FreeIPA should use external stub files for two reasons.
> First of all it is required to stay compatible with Python 2. And more
> importantly it's faster. FreeIPA's CLI scripts already take several
> hundred milliseconds to execute. Typing would slow them down even further.
> 
I disagree with using stub files.  Types should be declared where
the functions are defined.  Types are documentation and proximity is
important (for humans).

Fortunately, Mypy supports "type comments" in addition to PEP 3107
function annotations.  Mypy groks them but CPython will treat them
as comments and discard.  This allows us to use type hints with no
runtime cost for the CLI scripts.

> It's rather easy to auto-generate stub files -- assuming you are running
> on Fedora and have all Python 3 dependencies installed:
> 
> $ sudo dnf install python3-mypy
> $ echo "api.bootstrap(ra_plugin='dogtag')" >> ipalib/__init__.py
> $ mkdir out
> $ PYTHONPATH=. stubgen --recursive ipaclient ipalib ipaplatform
> ipapython ipaserver
> 
> The api.bootstrap() call is required. Otherwise stubgen cannot import a
> bunch of plugin files.
> 
Thanks for this additional info.

Cheers,
Fraser
___
FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org
To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org

[Freeipa-devel] [freeipa PR#963][opened] Include the CA basic constraint in CSRs when renewing a CA

2017-08-09 Thread rcritten via FreeIPA-devel 
   URL: https://github.com/freeipa/freeipa/pull/963
Author: rcritten
 Title: #963: Include the CA basic constraint in CSRs when renewing a CA
Action: opened

PR body:
"""
The CSR generated by `ipa-cacert-manage renew --external-ca` did
not include the CA basic constraint:

  X509v3 Basic Constraints: critical
  CA:TRUE

Add a flag to certmonger::resubmit_request to specify that a
CA is being requested.

Note that this also sets pathlen to -1 which means an unlimited
pathlen. Leave it up to the issuing CA to set this.

https://pagure.io/freeipa/issue/7088
"""

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/963/head:pr963
git checkout pr963
From fb672013196f465ff42a155420f4ff92742dafa0 Mon Sep 17 00:00:00 2001
From: Rob Crittenden 
Date: Wed, 9 Aug 2017 17:28:35 -0400
Subject: [PATCH] Include the CA basic constraint in CSRs when renewing a CA

The CSR generated by `ipa-cacert-manage renew --external-ca` did
not include the CA basic constraint:

  X509v3 Basic Constraints: critical
  CA:TRUE

Add a flag to certmonger::resubmit_request to specify that a
CA is being requested.

Note that this also sets pathlen to -1 which means an unlimited
pathlen. Leave it up to the issuing CA to set this.

https://pagure.io/freeipa/issue/7088
---
 ipalib/install/certmonger.py   | 13 +++--
 ipaserver/install/ipa_cacert_manage.py |  3 ++-
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/ipalib/install/certmonger.py b/ipalib/install/certmonger.py
index 13c68894d6..7c3f16 100644
--- a/ipalib/install/certmonger.py
+++ b/ipalib/install/certmonger.py
@@ -519,16 +519,25 @@ def modify(request_id, ca=None, profile=None):
 request.obj_if.modify(update)
 
 
-def resubmit_request(request_id, ca=None, profile=None):
+def resubmit_request(request_id, ca=None, profile=None, is_ca=False):
+"""
+:param request_id: the certmonger numeric request ID
+:param ca: the nickname for the certmonger CA, e.g. IPA or SelfSign
+:param profile: the dogtag template profile to use, e.g. SubCA
+:param is_ca: boolean that if True adds the CA basic constraint
+"""
 request = _get_request({'nickname': request_id})
 if request:
-if ca or profile:
+if ca or profile or is_ca:
 update = {}
 if ca is not None:
 cm = _certmonger()
 update['CA'] = cm.obj_if.find_ca_by_nickname(ca)
 if profile is not None:
 update['template-profile'] = profile
+if is_ca:
+update['template-is-ca'] = True
+update['template-ca-path-length'] = -1  # no path length
 request.obj_if.modify(update)
 request.obj_if.resubmit()
 
diff --git a/ipaserver/install/ipa_cacert_manage.py b/ipaserver/install/ipa_cacert_manage.py
index 24ef86fe1d..86243d342b 100644
--- a/ipaserver/install/ipa_cacert_manage.py
+++ b/ipaserver/install/ipa_cacert_manage.py
@@ -303,7 +303,8 @@ def resubmit_request(self, ca='dogtag-ipa-ca-renew-agent', profile=''):
 timeout = api.env.startup_timeout + 60
 
 logger.debug("resubmitting certmonger request '%s'", self.request_id)
-certmonger.resubmit_request(self.request_id, ca=ca, profile=profile)
+certmonger.resubmit_request(self.request_id, ca=ca, profile=profile,
+is_ca=True)
 try:
 state = certmonger.wait_for_request(self.request_id, timeout)
 except RuntimeError:
___
FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org
To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org

[Freeipa-devel] [freeipa PR#961][opened] [WIP] Vault testcase improvement

2017-08-09 Thread Akasurde via FreeIPA-devel 
   URL: https://github.com/freeipa/freeipa/pull/961
Author: Akasurde
 Title: #961: [WIP] Vault testcase improvement
Action: opened

PR body:
"""
* Collect logs for install KRA

fixes: #7098

Signed-off-by: Abhijeet Kasurde 
"""

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/961/head:pr961
git checkout pr961
From e6b20185a6c93f598d47a3aad2f146a4d619c57c Mon Sep 17 00:00:00 2001
From: Abhijeet Kasurde 
Date: Wed, 9 Aug 2017 18:12:32 +0530
Subject: [PATCH] [WIP] Vault testcase improvement

* Collect logs for install KRA

fixes: #7098

Signed-off-by: Abhijeet Kasurde 
---
 ipatests/pytest_plugins/integration/tasks.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/ipatests/pytest_plugins/integration/tasks.py b/ipatests/pytest_plugins/integration/tasks.py
index 9341c2910f..6038e08ab1 100644
--- a/ipatests/pytest_plugins/integration/tasks.py
+++ b/ipatests/pytest_plugins/integration/tasks.py
@@ -1163,7 +1163,9 @@ def install_kra(host, domain_level=None, first_instance=False, raiseonerr=True):
 if domain_level == DOMAIN_LEVEL_0 and not first_instance:
 replica_file = get_replica_filename(host)
 command.append(replica_file)
-return host.run_command(command, raiseonerr=raiseonerr)
+result = host.run_command(command, raiseonerr=raiseonerr)
+setup_server_logs_collecting(host)
+return result
 
 
 def install_ca(host, domain_level=None, first_instance=False, raiseonerr=True):
___
FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org
To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org

[Freeipa-devel] [freeipa PR#404][closed] tests: Add LDAP URI to ldappasswd explicitly

2017-08-09 Thread davidkupka via FreeIPA-devel 
   URL: https://github.com/freeipa/freeipa/pull/404
Author: davidkupka
 Title: #404: tests: Add LDAP URI to ldappasswd explicitly
Action: closed

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/404/head:pr404
git checkout pr404
___
FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org
To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org

[Freeipa-devel] [freeipa PR#960][opened] tests: Add LDAP URI to ldappasswd explicitelly

2017-08-09 Thread davidkupka via FreeIPA-devel 
   URL: https://github.com/freeipa/freeipa/pull/960
Author: davidkupka
 Title: #960: tests: Add LDAP URI to ldappasswd explicitelly
Action: opened

PR body:
"""
https://fedorahosted.org/freeipa/ticket/6622

This PR replaces PR #404 which is orphaned and I'm no longer able to push into 
it.
"""

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/960/head:pr960
git checkout pr960
From 052a248329e6b08051149c70fb414f490b1e27bd Mon Sep 17 00:00:00 2001
From: David Kupka 
Date: Thu, 19 Jan 2017 09:18:32 +0100
Subject: [PATCH] tests: Add LDAP URI to ldappasswd explicitelly

https://fedorahosted.org/freeipa/ticket/6622
---
 ipatests/pytest_plugins/integration/tasks.py | 3 ++-
 ipatests/util.py | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/ipatests/pytest_plugins/integration/tasks.py b/ipatests/pytest_plugins/integration/tasks.py
index 9341c2910f..4cd993520b 100644
--- a/ipatests/pytest_plugins/integration/tasks.py
+++ b/ipatests/pytest_plugins/integration/tasks.py
@@ -1314,7 +1314,8 @@ def ldappasswd_user_change(user, oldpw, newpw, master):
 basedn = master.domain.basedn
 
 userdn = "uid={},{},{}".format(user, container_user, basedn)
+master_ldap_uri = "ldap://{}:389".format(master.external_hostname)
 
 args = [paths.LDAPPASSWD, '-D', userdn, '-w', oldpw, '-a', oldpw,
-'-s', newpw, '-x']
+'-s', newpw, '-x', '-H', master_ldap_uri]
 master.run_command(args)
diff --git a/ipatests/util.py b/ipatests/util.py
index 575d5cc36b..3727490bac 100644
--- a/ipatests/util.py
+++ b/ipatests/util.py
@@ -748,7 +748,7 @@ def unlock_principal_password(user, oldpw, newpw):
 user, api.env.container_user, api.env.basedn)
 
 args = [paths.LDAPPASSWD, '-D', userdn, '-w', oldpw, '-a', oldpw,
-'-s', newpw, '-x']
+'-s', newpw, '-x', '-H', api.env.ldap_uri]
 return run(args)
 
 
___
FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org
To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org

[Freeipa-devel] Re: [RFC] Static type checking for FreeIPA (Mypy)

2017-08-09 Thread Christian Heimes via FreeIPA-devel 
On 2017-08-09 10:18, Christian Heimes via FreeIPA-devel wrote:
> On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote:
>> Hi team,
>>
>> At PyCon Australia on the weekend I was reminded of PEP-484 type
>> hinting** and the Mypy type checker for Python.
>>
>> With focus of FreeIPA project shifting more towards stability,
>> quality and maintainability, and with Python 3 porting work nearly
>> wrapped up, now is the time to think about how we can get more
>> confidence in our code not just from tests, but from the code
>> itself.  Static checking of annotated types can help us there, and
>> Mypy can let us begin to do this when writing new code or
>> refactoring old code.  Furthermore there is a benefit for IDE-users
>> where plugins can use type annotations to provide better completion
>> suggestions, etc.  For an overview of Mypy please see the PyCon AU
>> talk[1] or the docs[2].
>>
>> [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ
>> [2] http://mypy.readthedocs.io/en/latest/index.html
>>
>> So, what's the plan?  Alongside my other tasks, I'm going to start
>> looking at how we could use Mypy in FreeIPA CI, and see what it is
>> like using types in some of the areas I'm familiar with e.g.
>> ipalib.x509.  Based on my findings I'll update the team on the wins
>> and challenges and we can decide how to proceed from there.
> 
> Felipe ask me about typing and Mypy a couple of weeks ago. It's a good
> idea and we should do it. But I advise against typing information in the
> source code. FreeIPA should use external stub files for two reasons.
> First of all it is required to stay compatible with Python 2. And more
> importantly it's faster. FreeIPA's CLI scripts already take several
> hundred milliseconds to execute. Typing would slow them down even further.
> 
> It's rather easy to auto-generate stub files -- assuming you are running
> on Fedora and have all Python 3 dependencies installed:

Small correction:, stubgen in Fedora 26 is broken [1]. Try this:
https://gist.github.com/tiran/a281aa3baf9ea39e1d02800c1a7f8ea6

Christian

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1479697

-- 
Christian Heimes
Senior Software Engineer, Identity Management and Platform Security

Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander



signature.asc
Description: OpenPGP digital signature
___
FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org
To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org

[Freeipa-devel] Re: [RFC] Static type checking for FreeIPA (Mypy)

2017-08-09 Thread Christian Heimes via FreeIPA-devel 
On 2017-08-08 08:04, Fraser Tweedale via FreeIPA-devel wrote:
> Hi team,
> 
> At PyCon Australia on the weekend I was reminded of PEP-484 type
> hinting** and the Mypy type checker for Python.
> 
> With focus of FreeIPA project shifting more towards stability,
> quality and maintainability, and with Python 3 porting work nearly
> wrapped up, now is the time to think about how we can get more
> confidence in our code not just from tests, but from the code
> itself.  Static checking of annotated types can help us there, and
> Mypy can let us begin to do this when writing new code or
> refactoring old code.  Furthermore there is a benefit for IDE-users
> where plugins can use type annotations to provide better completion
> suggestions, etc.  For an overview of Mypy please see the PyCon AU
> talk[1] or the docs[2].
> 
> [1] https://www.youtube.com/watch?v=mXfsMDM3LwQ
> [2] http://mypy.readthedocs.io/en/latest/index.html
> 
> So, what's the plan?  Alongside my other tasks, I'm going to start
> looking at how we could use Mypy in FreeIPA CI, and see what it is
> like using types in some of the areas I'm familiar with e.g.
> ipalib.x509.  Based on my findings I'll update the team on the wins
> and challenges and we can decide how to proceed from there.

Felipe ask me about typing and Mypy a couple of weeks ago. It's a good
idea and we should do it. But I advise against typing information in the
source code. FreeIPA should use external stub files for two reasons.
First of all it is required to stay compatible with Python 2. And more
importantly it's faster. FreeIPA's CLI scripts already take several
hundred milliseconds to execute. Typing would slow them down even further.

It's rather easy to auto-generate stub files -- assuming you are running
on Fedora and have all Python 3 dependencies installed:

$ sudo dnf install python3-mypy
$ echo "api.bootstrap(ra_plugin='dogtag')" >> ipalib/__init__.py
$ mkdir out
$ PYTHONPATH=. stubgen --recursive ipaclient ipalib ipaplatform
ipapython ipaserver

The api.bootstrap() call is required. Otherwise stubgen cannot import a
bunch of plugin files.

Christian

-- 
Christian Heimes
Senior Software Engineer, Identity Management and Platform Security

Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Eric Shander



signature.asc
Description: OpenPGP digital signature
___
FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org
To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org