Re: [Freeipa-devel] [PATCHES 0204-0207] Server upgrade: Make LDAP data upgrade deterministic

2015-03-12 Thread David Kupka

On 03/06/2015 04:50 PM, Martin Basti wrote:

The patchset ensure, the upgrade order will respect ordering of entries
in *.update files.

Required for: https://fedorahosted.org/freeipa/ticket/4904

Patch 205 also fixes https://fedorahosted.org/freeipa/ticket/3560

Required patch mbasti-0203

Patches attached.




Changes in code looks good and the upgrade process still works, ACK.

--
David Kupka

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code


Re: [Freeipa-devel] [PATCHES 0204-0207] Server upgrade: Make LDAP data upgrade deterministic

2015-03-12 Thread Rob Crittenden
Martin Basti wrote:
 The patchset ensure, the upgrade order will respect ordering of entries
 in *.update files.
 
 Required for: https://fedorahosted.org/freeipa/ticket/4904
 
 Patch 205 also fixes https://fedorahosted.org/freeipa/ticket/3560
 
 Required patch mbasti-0203
 
 Patches attached.
 
 
 

Just reading the patches, untested.

I think ordered should default to True in the update() method of
ldapupdater to keep in spirit with the design.

Otherwise LGTM that it implements what was designed.

rob


-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code


[Freeipa-devel] [PATCHES 0204-0207] Server upgrade: Make LDAP data upgrade deterministic

2015-03-06 Thread Martin Basti
The patchset ensure, the upgrade order will respect ordering of entries 
in *.update files.


Required for: https://fedorahosted.org/freeipa/ticket/4904

Patch 205 also fixes https://fedorahosted.org/freeipa/ticket/3560

Required patch mbasti-0203

Patches attached.

--
Martin Basti

From 9dd32e80f37b852feb980fd4ef2ec7c082ffc1a5 Mon Sep 17 00:00:00 2001
From: Martin Basti mba...@redhat.com
Date: Thu, 26 Feb 2015 12:01:19 +0100
Subject: [PATCH 2/5] Server Upgrade: do not sort updates by DN

Ticket: https://fedorahosted.org/freeipa/ticket/4904
---
 ipaserver/install/ldapupdate.py | 15 ++-
 1 file changed, 2 insertions(+), 13 deletions(-)

diff --git a/ipaserver/install/ldapupdate.py b/ipaserver/install/ldapupdate.py
index 53d5407d5e8a15abe13f2f6d8b3df74ca100ea5a..e8516ff86a951f828c4213f8e70db613b99ed8c4 100644
--- a/ipaserver/install/ldapupdate.py
+++ b/ipaserver/install/ldapupdate.py
@@ -784,22 +784,11 @@ class LDAPUpdate:
 raise RuntimeError(Offline updates are not supported.)
 
 def _run_updates(self, all_updates):
-# For adds and updates we want to apply updates from shortest
-# to greatest length of the DN.
-# For deletes we want the reverse
-def update_sort_key(dn_update):
-dn, update = dn_update
-assert isinstance(dn, DN)
-return len(dn)
 
-sorted_updates = sorted(all_updates.iteritems(), key=update_sort_key)
-
-for dn, update in sorted_updates:
+for dn, update in all_updates.iteritems():
 self._update_record(update)
 
-# Now run the deletes in reversed order
-sorted_updates.reverse()
-for dn, update in sorted_updates:
+for dn, update in all_updates.iteritems():
 self._delete_record(update)
 
 def update(self, files, ordered=False):
-- 
2.1.0

From 689caca4d322a18108756b530522be625f5b0964 Mon Sep 17 00:00:00 2001
From: Martin Basti mba...@redhat.com
Date: Thu, 5 Mar 2015 16:56:02 +0100
Subject: [PATCH 3/5] Server Upgrade: Upgrade one file per time

* Files are sorted alphabetically, no numbering required anymore
* One file updated per time

Ticket: https://fedorahosted.org/freeipa/ticket/3560
---
 ipaserver/install/ldapupdate.py | 54 ++---
 1 file changed, 18 insertions(+), 36 deletions(-)

diff --git a/ipaserver/install/ldapupdate.py b/ipaserver/install/ldapupdate.py
index e8516ff86a951f828c4213f8e70db613b99ed8c4..3b4aa58d9e84006510c23e4aa7d52a84c205f79c 100644
--- a/ipaserver/install/ldapupdate.py
+++ b/ipaserver/install/ldapupdate.py
@@ -793,44 +793,27 @@ class LDAPUpdate:
 
 def update(self, files, ordered=False):
 Execute the update. files is a list of the update files to use.
+:param ordered: Update files are executed in alphabetical order
 
-   If ordered is True then the updates the file must be of the form
-   ##-name.update where ## is an integer between 10 and 89. The
-   changes are applied to LDAP at the end of each value divisible
-   by 10, so after 20, 30, etc.
-
-   returns True if anything was changed, otherwise False
+returns True if anything was changed, otherwise False
 
 
-pat = re.compile(r'(\d+)-.*\.update')
 all_updates = {}
-r = 20
-if self.plugins:
-self.info('PRE_UPDATE')
-updates = api.Backend.updateclient.update(PRE_UPDATE, self.dm_password, self.ldapi, self.live_run)
-self.merge_updates(all_updates, updates)
 try:
 self.create_connection()
-if ordered and all_updates:
+if self.plugins:
+self.info('PRE_UPDATE')
+updates = api.Backend.updateclient.update(PRE_UPDATE, self.dm_password, self.ldapi, self.live_run)
+self.merge_updates(all_updates, updates)
 # flush out PRE_UPDATE plugin updates before we begin
 self._run_updates(all_updates)
 all_updates = {}
 
-for f in files:
-name = os.path.basename(f)
-if ordered:
-m = pat.match(name)
-if not m:
-raise RuntimeError(Filename does not match format #-name.update: %s % f)
-index = int(m.group(1))
-if index  10 or index  90:
-raise RuntimeError(Index not legal range: %d % index)
-
-if index = r:
-self._run_updates(all_updates)
-all_updates = {}
-r += 10
+upgrade_files = files
+if ordered:
+upgrade_files = sorted(files)
 
+for f in upgrade_files:
 try:
 self.info(Parsing update file '%s' % f)
 data = self.read_file(f)
@@ -839,17 +822,16 @@ class LDAPUpdate: