Re: [Freeipa-devel] [PATCH] 0020..0022 pki-related upgrade fixes
On 30/06/15 18:02, Fraser Tweedale wrote: On Mon, Jun 29, 2015 at 05:56:11PM +0200, Martin Basti wrote: On 29/06/15 16:03, Fraser Tweedale wrote: On Thu, Jun 25, 2015 at 11:23:01AM +0200, Martin Basti wrote: On 19/06/15 09:28, Fraser Tweedale wrote: The attached patches fix upgrade issues when pki is also updated from pre 10.2.4. pki dependency is bumped to 10.2.5 - the official builds should be done Friday (US time) but it is available from my copr[1]. If someone wants to add to official freeipa COPR in meantime the SRPM is here[2]. [1] https://copr.fedoraproject.org/coprs/ftweedal/freeipa/ [2] https://ftweedal.fedorapeople.org/pki-core-10.2.5-0.2.fc21.src.rpm Thanks, Fraser Thank you. 1) I cannot apply patches. Rebased patches attached. 2) IMO patch 0020 was fixed with my patch 266 It seems we are hitting another case of LDAP disconnection during upgrade; without 0020 the upgrade fails. There might be a better way so let me know if you have ideas. 3) This print should not be there + +print cs_cfg +for profile_id in profile_ids: Thakns; removed. 4) This is unused variable, it is defined later + cs_cfg = None Thanks; removed. 5) Can you add there log.error or log.debug instead of pass please? +# enable the profile +try: +profile_api.enable_profile(profile_id) +except errors.RemoteRetrieveError: +pass You've got it. Also did this a few lines up where the profile is disabled. I will test it later. -- Martin Basti Thank you, Fraser PATCH 0020 - NACK see my patch 269, it fixes root cause. (IMO with reworked patch 21 it is not needed) PATCH 0021 - NACK, it runs whole upgrade machinery again. Patch how to fix it is attached. Sorry I didn't notice it last time. PATCH 0022 - LGTM -- Martin Basti Thank you very much! Your patch to my patch works perfectly. I squashed it into 0021. Patch 0020 rescinded. Rebased patches attached. Cheers, Fraser Thank you, ACK for both patches. -- Martin Basti -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH] 0020..0022 pki-related upgrade fixes
On Mon, Jun 29, 2015 at 05:56:11PM +0200, Martin Basti wrote:
On 29/06/15 16:03, Fraser Tweedale wrote:
On Thu, Jun 25, 2015 at 11:23:01AM +0200, Martin Basti wrote:
On 19/06/15 09:28, Fraser Tweedale wrote:
The attached patches fix upgrade issues when pki is also updated
from pre 10.2.4.
pki dependency is bumped to 10.2.5 - the official builds should be
done Friday (US time) but it is available from my copr[1]. If
someone wants to add to official freeipa COPR in meantime the SRPM
is here[2].
[1] https://copr.fedoraproject.org/coprs/ftweedal/freeipa/
[2] https://ftweedal.fedorapeople.org/pki-core-10.2.5-0.2.fc21.src.rpm
Thanks,
Fraser
Thank you.
1)
I cannot apply patches.
Rebased patches attached.
2)
IMO patch 0020 was fixed with my patch 266
It seems we are hitting another case of LDAP disconnection during
upgrade; without 0020 the upgrade fails. There might be a better
way so let me know if you have ideas.
3)
This print should not be there
+
+print cs_cfg
+for profile_id in profile_ids:
Thakns; removed.
4)
This is unused variable, it is defined later
+ cs_cfg = None
Thanks; removed.
5)
Can you add there log.error or log.debug instead of pass please?
+# enable the profile
+try:
+profile_api.enable_profile(profile_id)
+except errors.RemoteRetrieveError:
+pass
You've got it. Also did this a few lines up where the profile is
disabled.
I will test it later.
--
Martin Basti
Thank you,
Fraser
PATCH 0020 - NACK see my patch 269, it fixes root cause. (IMO with reworked
patch 21 it is not needed)
PATCH 0021 - NACK, it runs whole upgrade machinery again. Patch how to fix
it is attached. Sorry I didn't notice it last time.
PATCH 0022 - LGTM
--
Martin Basti
Thank you very much!
Your patch to my patch works perfectly. I squashed it into 0021.
Patch 0020 rescinded. Rebased patches attached.
Cheers,
Fraser
From 8daaed33cf06b5f940195d08038dbaadc562f880 Mon Sep 17 00:00:00 2001
From: Fraser Tweedale fr...@frase.id.au
Date: Tue, 16 Jun 2015 07:40:36 -0400
Subject: [PATCH 21/22] Upgrade CA schema during upgrade
New schema (for LDAP-based profiles) was introduced in Dogtag, but
Dogtag does not yet have a reliable method for upgrading its schema.
Use FreeIPA's schema update machinery to add the new attributeTypes
and objectClasses defined by Dogtag.
Also update the pki dependencies to 10.2.5, which provides the
schema update file.
---
freeipa.spec.in | 6 +++---
ipaserver/install/server/upgrade.py | 23 +++
2 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/freeipa.spec.in b/freeipa.spec.in
index
4f08db9f693318c6f4bfaf5e634ccffa78a4a28c..de250d8843506acd6109525c0630132fe60e2268
100644
--- a/freeipa.spec.in
+++ b/freeipa.spec.in
@@ -96,7 +96,7 @@ BuildRequires: python-backports-ssl_match_hostname
BuildRequires: softhsm-devel = 2.0.0rc1-1
BuildRequires: openssl-devel
BuildRequires: p11-kit-devel
-BuildRequires: pki-base = 10.2.4-1
+BuildRequires: pki-base = 10.2.5
BuildRequires: python-pytest-multihost = 0.5
BuildRequires: python-pytest-sourceorder
BuildRequires: python-kdcproxy = 0.3
@@ -141,8 +141,8 @@ Requires(post): systemd-units
Requires: selinux-policy = %{selinux_policy_version}
Requires(post): selinux-policy-base
Requires: slapi-nis = 0.54.2-1
-Requires: pki-ca = 10.2.4-1
-Requires: pki-kra = 10.2.4-1
+Requires: pki-ca = 10.2.5
+Requires: pki-kra = 10.2.5
Requires(preun): python systemd-units
Requires(postun): python systemd-units
Requires: python-dns = 1.11.1
diff --git a/ipaserver/install/server/upgrade.py
b/ipaserver/install/server/upgrade.py
index
822f746222bd3cb491901205af862a68ec464bbb..4a9f0128aed901e21a1fb57d3f72aecf954df478
100644
--- a/ipaserver/install/server/upgrade.py
+++ b/ipaserver/install/server/upgrade.py
@@ -31,6 +31,7 @@ from ipaserver.install import service
from ipaserver.install import cainstance
from ipaserver.install import certs
from ipaserver.install import otpdinstance
+from ipaserver.install import schemaupdate
from ipaserver.install import sysupgrade
from ipaserver.install import dnskeysyncinstance
from ipaserver.install.upgradeinstance import IPAUpgrade
@@ -1254,6 +1255,27 @@ def update_mod_nss_protocol(http):
sysupgrade.set_upgrade_state('nss.conf', 'protocol_updated_tls12', True)
+def ca_upgrade_schema(ca):
+root_logger.info('[Upgrading CA schema]')
+if not ca.is_configured():
+root_logger.info('CA is not configured')
+return False
+
+schema_files=['/usr/share/pki/server/conf/schema-certProfile.ldif']
+try:
+modified = schemaupdate.update_schema(schema_files, ldapi=True)
+except Exception as e:
+root_logger.error(%s, e)
+raise RuntimeError('CA schema upgrade failed.', 1)
+else:
+if modified:
+root_logger.info('CA schema update complete')
+
Re: [Freeipa-devel] [PATCH] 0020..0022 pki-related upgrade fixes
On Thu, Jun 25, 2015 at 11:23:01AM +0200, Martin Basti wrote:
On 19/06/15 09:28, Fraser Tweedale wrote:
The attached patches fix upgrade issues when pki is also updated
from pre 10.2.4.
pki dependency is bumped to 10.2.5 - the official builds should be
done Friday (US time) but it is available from my copr[1]. If
someone wants to add to official freeipa COPR in meantime the SRPM
is here[2].
[1] https://copr.fedoraproject.org/coprs/ftweedal/freeipa/
[2] https://ftweedal.fedorapeople.org/pki-core-10.2.5-0.2.fc21.src.rpm
Thanks,
Fraser
Thank you.
1)
I cannot apply patches.
Rebased patches attached.
2)
IMO patch 0020 was fixed with my patch 266
It seems we are hitting another case of LDAP disconnection during
upgrade; without 0020 the upgrade fails. There might be a better
way so let me know if you have ideas.
3)
This print should not be there
+
+print cs_cfg
+for profile_id in profile_ids:
Thakns; removed.
4)
This is unused variable, it is defined later
+ cs_cfg = None
Thanks; removed.
5)
Can you add there log.error or log.debug instead of pass please?
+# enable the profile
+try:
+profile_api.enable_profile(profile_id)
+except errors.RemoteRetrieveError:
+pass
You've got it. Also did this a few lines up where the profile is
disabled.
I will test it later.
--
Martin Basti
Thank you,
Fraser
From e2ee2584a683c7a25a90df9bd5d70cabfc448a21 Mon Sep 17 00:00:00 2001
From: Fraser Tweedale ftwee...@redhat.com
Date: Fri, 19 Jun 2015 01:37:26 -0400
Subject: [PATCH 20/22] Upgrade: disconnect ldap2 after adding CA DNS records
Non-disconnection of ldap2 backend in 'add_ca_dns_records' seems to
be causing problems with later uses. Avoid the problem by
disconnecting it before returning.
---
ipaserver/install/server/upgrade.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/ipaserver/install/server/upgrade.py
b/ipaserver/install/server/upgrade.py
index
784a03b195ab99c865935b6e51cc86a3b81842ee..b9e809f314bfb83eafe26f92f359a0539b98c2f0
100644
--- a/ipaserver/install/server/upgrade.py
+++ b/ipaserver/install/server/upgrade.py
@@ -1038,6 +1038,7 @@ def add_ca_dns_records():
if not ret['result']:
root_logger.info('DNS is not configured')
sysupgrade.set_upgrade_state('dns', 'ipa_ca_records', True)
+api.Backend.ldap2.disconnect()
return
bind = bindinstance.BindInstance()
@@ -1050,6 +1051,7 @@ def add_ca_dns_records():
ca_configured=None)
sysupgrade.set_upgrade_state('dns', 'ipa_ca_records', True)
+api.Backend.ldap2.disconnect()
def find_subject_base():
--
2.1.0
From 8ec6bca25e71bd41bd422a9010feddc14d5ea77a Mon Sep 17 00:00:00 2001
From: Fraser Tweedale fr...@frase.id.au
Date: Tue, 16 Jun 2015 07:40:36 -0400
Subject: [PATCH 21/22] Upgrade CA schema during upgrade
New schema (for LDAP-based profiles) was introduced in Dogtag, but
Dogtag does not yet have a reliable method for upgrading its schema.
Use FreeIPA's schema update machinery to add the new attributeTypes
and objectClasses defined by Dogtag.
Also update the pki dependencies to 10.2.5, which provides the
schema update file.
---
freeipa.spec.in | 6 +++---
ipaserver/install/server/upgrade.py | 26 ++
2 files changed, 29 insertions(+), 3 deletions(-)
diff --git a/freeipa.spec.in b/freeipa.spec.in
index
4f08db9f693318c6f4bfaf5e634ccffa78a4a28c..de250d8843506acd6109525c0630132fe60e2268
100644
--- a/freeipa.spec.in
+++ b/freeipa.spec.in
@@ -96,7 +96,7 @@ BuildRequires: python-backports-ssl_match_hostname
BuildRequires: softhsm-devel = 2.0.0rc1-1
BuildRequires: openssl-devel
BuildRequires: p11-kit-devel
-BuildRequires: pki-base = 10.2.4-1
+BuildRequires: pki-base = 10.2.5
BuildRequires: python-pytest-multihost = 0.5
BuildRequires: python-pytest-sourceorder
BuildRequires: python-kdcproxy = 0.3
@@ -141,8 +141,8 @@ Requires(post): systemd-units
Requires: selinux-policy = %{selinux_policy_version}
Requires(post): selinux-policy-base
Requires: slapi-nis = 0.54.2-1
-Requires: pki-ca = 10.2.4-1
-Requires: pki-kra = 10.2.4-1
+Requires: pki-ca = 10.2.5
+Requires: pki-kra = 10.2.5
Requires(preun): python systemd-units
Requires(postun): python systemd-units
Requires: python-dns = 1.11.1
diff --git a/ipaserver/install/server/upgrade.py
b/ipaserver/install/server/upgrade.py
index
b9e809f314bfb83eafe26f92f359a0539b98c2f0..0d24e03f96ebc465df90dede1ff44cd609ea7592
100644
--- a/ipaserver/install/server/upgrade.py
+++ b/ipaserver/install/server/upgrade.py
@@ -1256,6 +1256,31 @@ def update_mod_nss_protocol(http):
sysupgrade.set_upgrade_state('nss.conf', 'protocol_updated_tls12', True)
+def ca_upgrade_schema(ca):
+root_logger.info('[Upgrading CA schema]')
+if not ca.is_configured():
+root_logger.info('CA is not configured')
+return False
+
+realm =
Re: [Freeipa-devel] [PATCH] 0020..0022 pki-related upgrade fixes
On 29/06/15 16:03, Fraser Tweedale wrote:
On Thu, Jun 25, 2015 at 11:23:01AM +0200, Martin Basti wrote:
On 19/06/15 09:28, Fraser Tweedale wrote:
The attached patches fix upgrade issues when pki is also updated
from pre 10.2.4.
pki dependency is bumped to 10.2.5 - the official builds should be
done Friday (US time) but it is available from my copr[1]. If
someone wants to add to official freeipa COPR in meantime the SRPM
is here[2].
[1] https://copr.fedoraproject.org/coprs/ftweedal/freeipa/
[2] https://ftweedal.fedorapeople.org/pki-core-10.2.5-0.2.fc21.src.rpm
Thanks,
Fraser
Thank you.
1)
I cannot apply patches.
Rebased patches attached.
2)
IMO patch 0020 was fixed with my patch 266
It seems we are hitting another case of LDAP disconnection during
upgrade; without 0020 the upgrade fails. There might be a better
way so let me know if you have ideas.
3)
This print should not be there
+
+print cs_cfg
+for profile_id in profile_ids:
Thakns; removed.
4)
This is unused variable, it is defined later
+ cs_cfg = None
Thanks; removed.
5)
Can you add there log.error or log.debug instead of pass please?
+# enable the profile
+try:
+profile_api.enable_profile(profile_id)
+except errors.RemoteRetrieveError:
+pass
You've got it. Also did this a few lines up where the profile is
disabled.
I will test it later.
--
Martin Basti
Thank you,
Fraser
PATCH 0020 - NACK see my patch 269, it fixes root cause. (IMO with
reworked patch 21 it is not needed)
PATCH 0021 - NACK, it runs whole upgrade machinery again. Patch how to
fix it is attached. Sorry I didn't notice it last time.
PATCH 0022 - LGTM
--
Martin Basti
From 2c5e865357b20adff2636be5e5a9723777dc0131 Mon Sep 17 00:00:00 2001
From: Martin Basti mba...@redhat.com
Date: Mon, 29 Jun 2015 17:38:46 +0200
Subject: [PATCH] fix fraser ca schema
---
ipaserver/install/server/upgrade.py | 15 ++-
1 file changed, 6 insertions(+), 9 deletions(-)
diff --git a/ipaserver/install/server/upgrade.py b/ipaserver/install/server/upgrade.py
index 2376b86d105984b97ab0e0709328242b49828069..395b762ff41404763e8f796192ce5ba537c2a1bf 100644
--- a/ipaserver/install/server/upgrade.py
+++ b/ipaserver/install/server/upgrade.py
@@ -31,6 +31,7 @@ from ipaserver.install import service
from ipaserver.install import cainstance
from ipaserver.install import certs
from ipaserver.install import otpdinstance
+from ipaserver.install import schemaupdate
from ipaserver.install import sysupgrade
from ipaserver.install import dnskeysyncinstance
from ipaserver.install.upgradeinstance import IPAUpgrade
@@ -1260,18 +1261,14 @@ def ca_upgrade_schema(ca):
root_logger.info('CA is not configured')
return False
-realm = krbV.default_context().default_realm
-upgrade = IPAUpgrade(realm,
-schema_files=['/usr/share/pki/server/conf/schema-certProfile.ldif'])
+schema_files=['/usr/share/pki/server/conf/schema-certProfile.ldif']
try:
-upgrade.create_instance()
-except BadSyntax:
-raise RuntimeError(
-'Bad syntax detected in CA schema file(s).', 1)
-except RuntimeError:
+modified = schemaupdate.update_schema(schema_files, ldapi=True)
+except Exception as e:
+root_logger.error(%s, e)
raise RuntimeError('CA schema upgrade failed.', 1)
else:
-if upgrade.modified:
+if modified:
root_logger.info('CA schema update complete')
return True
else:
--
2.4.3
--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH] 0020..0022 pki-related upgrade fixes
On 19/06/15 09:28, Fraser Tweedale wrote: The attached patches fix upgrade issues when pki is also updated from pre 10.2.4. pki dependency is bumped to 10.2.5 - the official builds should be done Friday (US time) but it is available from my copr[1]. If someone wants to add to official freeipa COPR in meantime the SRPM is here[2]. [1] https://copr.fedoraproject.org/coprs/ftweedal/freeipa/ [2] https://ftweedal.fedorapeople.org/pki-core-10.2.5-0.2.fc21.src.rpm Thanks, Fraser Thank you. 1) I cannot apply patches. 2) IMO patch 0020 was fixed with my patch 266 3) This print should not be there + +print cs_cfg +for profile_id in profile_ids: 4) This is unused variable, it is defined later + cs_cfg = None 5) Can you add there log.error or log.debug instead of pass please? +# enable the profile +try: +profile_api.enable_profile(profile_id) +except errors.RemoteRetrieveError: +pass I will test it later. -- Martin Basti -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH] 0020..0022 pki-related upgrade fixes
On 06/19/2015 09:44 AM, Fraser Tweedale wrote: On Fri, Jun 19, 2015 at 09:38:01AM +0200, Martin Kosek wrote: On 06/19/2015 09:28 AM, Fraser Tweedale wrote: The attached patches fix upgrade issues when pki is also updated from pre 10.2.4. pki dependency is bumped to 10.2.5 - the official builds should be done Friday (US time) but it is available from my copr[1]. If someone wants to add to official freeipa COPR in meantime the SRPM is here[2]. [1] https://copr.fedoraproject.org/coprs/ftweedal/freeipa/ [2] https://ftweedal.fedorapeople.org/pki-core-10.2.5-0.2.fc21.src.rpm Thanks, Fraser This may work, it would be of course cleaner to do this via context manager, in lines with: @contextmanager def ldap_connect_autobind(): if not api.Backend.ldap2.isconnected(): api.Backend.ldap2.connect(autobind=True) try: yield finally: if api.Backend.ldap2.isconnected(): api.Backend.ldap2.disconnect() ... try: with ldap_connect_autobind(): # do API stuff except PublicError, e: ... as that would also unbind it if exception is raised for example. Unless you or others feel strongly about this, let us do it as a cleanup later, as there are several places in upgrade.py that do this sort of thing. Alternatively (or in addition) we need to make ldap2 be able to attempt to reconnect on failure, as Simo (I think it was Simo) suggested recently in another thread. Cheers, Fraser Sure. Whatever is more systematic approach to this solution. We need to have as less hacks in the code as possible. -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH] 0020..0022 pki-related upgrade fixes
On Fri, Jun 19, 2015 at 09:38:01AM +0200, Martin Kosek wrote: On 06/19/2015 09:28 AM, Fraser Tweedale wrote: The attached patches fix upgrade issues when pki is also updated from pre 10.2.4. pki dependency is bumped to 10.2.5 - the official builds should be done Friday (US time) but it is available from my copr[1]. If someone wants to add to official freeipa COPR in meantime the SRPM is here[2]. [1] https://copr.fedoraproject.org/coprs/ftweedal/freeipa/ [2] https://ftweedal.fedorapeople.org/pki-core-10.2.5-0.2.fc21.src.rpm Thanks, Fraser This may work, it would be of course cleaner to do this via context manager, in lines with: @contextmanager def ldap_connect_autobind(): if not api.Backend.ldap2.isconnected(): api.Backend.ldap2.connect(autobind=True) try: yield finally: if api.Backend.ldap2.isconnected(): api.Backend.ldap2.disconnect() ... try: with ldap_connect_autobind(): # do API stuff except PublicError, e: ... as that would also unbind it if exception is raised for example. Unless you or others feel strongly about this, let us do it as a cleanup later, as there are several places in upgrade.py that do this sort of thing. Alternatively (or in addition) we need to make ldap2 be able to attempt to reconnect on failure, as Simo (I think it was Simo) suggested recently in another thread. Cheers, Fraser -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH] 0020..0022 pki-related upgrade fixes
On 06/19/2015 09:28 AM, Fraser Tweedale wrote: The attached patches fix upgrade issues when pki is also updated from pre 10.2.4. pki dependency is bumped to 10.2.5 - the official builds should be done Friday (US time) but it is available from my copr[1]. If someone wants to add to official freeipa COPR in meantime the SRPM is here[2]. [1] https://copr.fedoraproject.org/coprs/ftweedal/freeipa/ [2] https://ftweedal.fedorapeople.org/pki-core-10.2.5-0.2.fc21.src.rpm Thanks, Fraser This may work, it would be of course cleaner to do this via context manager, in lines with: @contextmanager def ldap_connect_autobind(): if not api.Backend.ldap2.isconnected(): api.Backend.ldap2.connect(autobind=True) try: yield finally: if api.Backend.ldap2.isconnected(): api.Backend.ldap2.disconnect() ... try: with ldap_connect_autobind(): # do API stuff except PublicError, e: ... as that would also unbind it if exception is raised for example. -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
