Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
JR Aquino wrote: On Oct 7, 2011, at 11:14 AM, Simo Sorce wrote: On Mon, 2011-10-03 at 18:17 -0400, Simo Sorce wrote: On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote: Newer 389ds servers have a new option to have a different set of filtered attributes from normal replication. This has been added in order to allow DS to replicate memberof attributes only during a total update so that we do not need to run a fixup memberof task on a replica at install time. This task is quite inefficient for big database and can take a long time. By replicating memberof while the DB is locked we are guaranteed the memberof list is consistent so we do not need a fixup. This patch allows to enable this feature dynamically. If the server does not yet support the new option it falls back to the previous behavior. Fixes: https://fedorahosted.org/freeipa/ticket/1794 I am sending the patch but it has been jointly developed at various stages by Nathan, JR, and me. Simo. After some thinking I found out that we cannot commit this patch until the memberof plugin is converted to use the new transaction interfaces for plugins, as otherwise it is possible to run into race conditions where the member/memberof relations are not settled if a new replica is installed while member attributes are being changed. Granted the race is quite small and unlikely but real. So please test and ack it, but we need to defer pushing to stable branches until ds copes. I think it is ok to push to master for testing, DS should have the necessary support by the time we make another stable release from master and in our test environments I am sure we will never hit the race. After some more testing I found a small bug that can cause issues in some conditions, new patch attached. Simo. ACK with 389-ds-base-1.2.10-0.4.a4 Pushed to master ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
On Oct 7, 2011, at 11:14 AM, Simo Sorce wrote: > On Mon, 2011-10-03 at 18:17 -0400, Simo Sorce wrote: >> On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote: >>> Newer 389ds servers have a new option to have a different set of >>> filtered attributes from normal replication. >>> >>> This has been added in order to allow DS to replicate memberof >>> attributes only during a total update so that we do not need to run a >>> fixup memberof task on a replica at install time. >>> This task is quite inefficient for big database and can take a long >>> time. By replicating memberof while the DB is locked we are guaranteed >>> the memberof list is consistent so we do not need a fixup. >>> >>> This patch allows to enable this feature dynamically. If the server does >>> not yet support the new option it falls back to the previous behavior. >>> >>> Fixes: https://fedorahosted.org/freeipa/ticket/1794 >>> >>> I am sending the patch but it has been jointly developed at various >>> stages by Nathan, JR, and me. >>> >>> Simo. >> >> After some thinking I found out that we cannot commit this patch until >> the memberof plugin is converted to use the new transaction interfaces >> for plugins, as otherwise it is possible to run into race conditions >> where the member/memberof relations are not settled if a new replica is >> installed while member attributes are being changed. >> >> Granted the race is quite small and unlikely but real. >> So please test and ack it, but we need to defer pushing to stable >> branches until ds copes. >> I think it is ok to push to master for testing, DS should have the >> necessary support by the time we make another stable release from master >> and in our test environments I am sure we will never hit the race. > > After some more testing I found a small bug that can cause issues in > some conditions, new patch attached. > > Simo. ACK with 389-ds-base-1.2.10-0.4.a4 ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
On Mon, 2011-10-03 at 18:17 -0400, Simo Sorce wrote:
> On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote:
> > Newer 389ds servers have a new option to have a different set of
> > filtered attributes from normal replication.
> >
> > This has been added in order to allow DS to replicate memberof
> > attributes only during a total update so that we do not need to run a
> > fixup memberof task on a replica at install time.
> > This task is quite inefficient for big database and can take a long
> > time. By replicating memberof while the DB is locked we are guaranteed
> > the memberof list is consistent so we do not need a fixup.
> >
> > This patch allows to enable this feature dynamically. If the server does
> > not yet support the new option it falls back to the previous behavior.
> >
> > Fixes: https://fedorahosted.org/freeipa/ticket/1794
> >
> > I am sending the patch but it has been jointly developed at various
> > stages by Nathan, JR, and me.
> >
> > Simo.
>
> After some thinking I found out that we cannot commit this patch until
> the memberof plugin is converted to use the new transaction interfaces
> for plugins, as otherwise it is possible to run into race conditions
> where the member/memberof relations are not settled if a new replica is
> installed while member attributes are being changed.
>
> Granted the race is quite small and unlikely but real.
> So please test and ack it, but we need to defer pushing to stable
> branches until ds copes.
> I think it is ok to push to master for testing, DS should have the
> necessary support by the time we make another stable release from master
> and in our test environments I am sure we will never hit the race.
After some more testing I found a small bug that can cause issues in
some conditions, new patch attached.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
>From 2a3d985d704d5a18d256cca1afe4f040c46478b6 Mon Sep 17 00:00:00 2001
From: JR Aquino
Date: Fri, 16 Sep 2011 10:23:02 -0700
Subject: [PATCH] Replication: Adjust replica installation to omit processing
memberof computations
https://fedorahosted.org/freeipa/ticket/1794
If the master does not yet support the total update list feature we still run
the memberof fixup task and not fail to replicate due to the new attribute not
being settable.
Jointly-developed-with: Simo Sorce
Jointly-developed-with: Nathank Kinder
---
install/tools/ipa-replica-install |1 -
install/ui/test/data/json_metadata.json |1 +
ipaserver/install/dsinstance.py |6 ++
ipaserver/install/replication.py| 28 +++-
4 files changed, 30 insertions(+), 6 deletions(-)
diff --git a/install/tools/ipa-replica-install b/install/tools/ipa-replica-install
index c2018f4d5ef992879f3aef6937db87c58e23f9ec..9fddb75844b3996c8ecf8443b0ded412e1339e82 100755
--- a/install/tools/ipa-replica-install
+++ b/install/tools/ipa-replica-install
@@ -469,7 +469,6 @@ def main():
raise RuntimeError("Failed to configure the client")
ds.replica_populate()
-ds.init_memberof()
#Everything installed properly, activate ipa service.
ipaservices.knownservices.ipa.enable()
diff --git a/install/ui/test/data/json_metadata.json b/install/ui/test/data/json_metadata.json
index 482eb6c3b8e502ee81714fe611822f2593bfd8f9..f87ab114c48330d6af791214d462fc9fc6562ffa 100644
--- a/install/ui/test/data/json_metadata.json
+++ b/install/ui/test/data/json_metadata.json
@@ -3273,6 +3273,7 @@
"nsds5replicaroot",
"nsds5replicasessionpausetime",
"nsds5replicatedattributelist",
+"nsds5replicatedattributelisttotal",
"nsds5replicatimeout",
"nsds5replicatombstonepurgeinterval",
"nsds5replicatransportinfo",
diff --git a/ipaserver/install/dsinstance.py b/ipaserver/install/dsinstance.py
index f2b16dfe822d757eaab6fe3d28a455d56af3cef9..36a386619850c6152b7f401a342e77b41f6835e1 100644
--- a/ipaserver/install/dsinstance.py
+++ b/ipaserver/install/dsinstance.py
@@ -169,6 +169,7 @@ class DsInstance(service.Service):
self.idmax = None
self.subject_base = None
self.open_ports = []
+self.run_init_memberof = True
if realm_name:
self.suffix = util.realm_to_suffix(self.realm_name)
self.__setup_sub_dict()
@@ -295,6 +296,7 @@ class DsInstance(service.Service):
repl.setup_replication(self.master_fqdn,
r_binddn="cn=Directory Manager",
r_bindpw=self.dm_password)
+self.run_init_memberof = repl.needs_memberof_fixup()
def __enable(self):
self.backup_state("enabled", self.is_enabled())
@@ -433,6 +435,10 @@ class DsInstance(service.Service):
self._ldap_mod("memberof-conf.ldif")
def init_memberof(self):
+
+if not self.run_init_memberof:
+return
+
self._ldap_mod("
Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
On Oct 3, 2011, at 3:17 PM, Simo Sorce wrote: > On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote: >> Newer 389ds servers have a new option to have a different set of >> filtered attributes from normal replication. >> >> This has been added in order to allow DS to replicate memberof >> attributes only during a total update so that we do not need to run a >> fixup memberof task on a replica at install time. >> This task is quite inefficient for big database and can take a long >> time. By replicating memberof while the DB is locked we are guaranteed >> the memberof list is consistent so we do not need a fixup. >> >> This patch allows to enable this feature dynamically. If the server does >> not yet support the new option it falls back to the previous behavior. >> >> Fixes: https://fedorahosted.org/freeipa/ticket/1794 >> >> I am sending the patch but it has been jointly developed at various >> stages by Nathan, JR, and me. >> >> Simo. > > After some thinking I found out that we cannot commit this patch until > the memberof plugin is converted to use the new transaction interfaces > for plugins, as otherwise it is possible to run into race conditions > where the member/memberof relations are not settled if a new replica is > installed while member attributes are being changed. > > Granted the race is quite small and unlikely but real. > So please test and ack it, but we need to defer pushing to stable > branches until ds copes. > I think it is ok to push to master for testing, DS should have the > necessary support by the time we make another stable release from master > and in our test environments I am sure we will never hit the race. Do we know which 389-ds-base incorporates the new option? I would like to test and ack, but I'm not sure if I have a fixed 389-ds-base installed. ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote: > Newer 389ds servers have a new option to have a different set of > filtered attributes from normal replication. > > This has been added in order to allow DS to replicate memberof > attributes only during a total update so that we do not need to run a > fixup memberof task on a replica at install time. > This task is quite inefficient for big database and can take a long > time. By replicating memberof while the DB is locked we are guaranteed > the memberof list is consistent so we do not need a fixup. > > This patch allows to enable this feature dynamically. If the server does > not yet support the new option it falls back to the previous behavior. > > Fixes: https://fedorahosted.org/freeipa/ticket/1794 > > I am sending the patch but it has been jointly developed at various > stages by Nathan, JR, and me. > > Simo. After some thinking I found out that we cannot commit this patch until the memberof plugin is converted to use the new transaction interfaces for plugins, as otherwise it is possible to run into race conditions where the member/memberof relations are not settled if a new replica is installed while member attributes are being changed. Granted the race is quite small and unlikely but real. So please test and ack it, but we need to defer pushing to stable branches until ds copes. I think it is ok to push to master for testing, DS should have the necessary support by the time we make another stable release from master and in our test environments I am sure we will never hit the race. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
On 10/03/2011 04:39 PM, JR Aquino wrote: On Oct 3, 2011, at 3:17 PM, Simo Sorce wrote: On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote: Newer 389ds servers have a new option to have a different set of filtered attributes from normal replication. This has been added in order to allow DS to replicate memberof attributes only during a total update so that we do not need to run a fixup memberof task on a replica at install time. This task is quite inefficient for big database and can take a long time. By replicating memberof while the DB is locked we are guaranteed the memberof list is consistent so we do not need a fixup. This patch allows to enable this feature dynamically. If the server does not yet support the new option it falls back to the previous behavior. Fixes: https://fedorahosted.org/freeipa/ticket/1794 I am sending the patch but it has been jointly developed at various stages by Nathan, JR, and me. Simo. After some thinking I found out that we cannot commit this patch until the memberof plugin is converted to use the new transaction interfaces for plugins, as otherwise it is possible to run into race conditions where the member/memberof relations are not settled if a new replica is installed while member attributes are being changed. Granted the race is quite small and unlikely but real. So please test and ack it, but we need to defer pushing to stable branches until ds copes. I think it is ok to push to master for testing, DS should have the necessary support by the time we make another stable release from master and in our test environments I am sure we will never hit the race. Do we know which 389-ds-base incorporates the new option? I would like to test and ack, but I'm not sure if I have a fixed 389-ds-base installed. 1.2.10.a1 - in updates-testing ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
[Freeipa-devel] [PATCH] #1794 - Speed up replica setup
Newer 389ds servers have a new option to have a different set of
filtered attributes from normal replication.
This has been added in order to allow DS to replicate memberof
attributes only during a total update so that we do not need to run a
fixup memberof task on a replica at install time.
This task is quite inefficient for big database and can take a long
time. By replicating memberof while the DB is locked we are guaranteed
the memberof list is consistent so we do not need a fixup.
This patch allows to enable this feature dynamically. If the server does
not yet support the new option it falls back to the previous behavior.
Fixes: https://fedorahosted.org/freeipa/ticket/1794
I am sending the patch but it has been jointly developed at various
stages by Nathan, JR, and me.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
>From 1ce9bd802d0cc59057d89ef2a2a7f0605c582308 Mon Sep 17 00:00:00 2001
From: JR Aquino
Date: Fri, 16 Sep 2011 10:23:02 -0700
Subject: [PATCH] Replication: Adjust replica installation to omit processing
memberof computations
https://fedorahosted.org/freeipa/ticket/1794
If the master does not yet support the total update list feature we still run
the memberof fixup task and not fail to replicate due to the new attribute not
being settable.
Jointly-developed-with: Simo Sorce
Jointly-developed-with: Nathank Kinder
---
install/tools/ipa-replica-install |1 -
install/ui/test/data/json_metadata.json |1 +
ipaserver/install/dsinstance.py |6 ++
ipaserver/install/replication.py| 28 +++-
4 files changed, 30 insertions(+), 6 deletions(-)
diff --git a/install/tools/ipa-replica-install b/install/tools/ipa-replica-install
index af317cde5bfbcfccde8d768b35c369207221e636..c77c21327b22326230c1587b932da5dd1bab3a76 100755
--- a/install/tools/ipa-replica-install
+++ b/install/tools/ipa-replica-install
@@ -466,7 +466,6 @@ def main():
raise RuntimeError("Failed to configure the client")
ds.replica_populate()
-ds.init_memberof()
#Everything installed properly, activate ipa service.
ipaservices.knownservices.ipa.enable()
diff --git a/install/ui/test/data/json_metadata.json b/install/ui/test/data/json_metadata.json
index 482eb6c3b8e502ee81714fe611822f2593bfd8f9..f87ab114c48330d6af791214d462fc9fc6562ffa 100644
--- a/install/ui/test/data/json_metadata.json
+++ b/install/ui/test/data/json_metadata.json
@@ -3273,6 +3273,7 @@
"nsds5replicaroot",
"nsds5replicasessionpausetime",
"nsds5replicatedattributelist",
+"nsds5replicatedattributelisttotal",
"nsds5replicatimeout",
"nsds5replicatombstonepurgeinterval",
"nsds5replicatransportinfo",
diff --git a/ipaserver/install/dsinstance.py b/ipaserver/install/dsinstance.py
index 790b560b92a28897a8f7e4ad076669241b168c3f..858bb892cb39fc36c7512cfb5e6c74b68fafb37f 100644
--- a/ipaserver/install/dsinstance.py
+++ b/ipaserver/install/dsinstance.py
@@ -169,6 +169,7 @@ class DsInstance(service.Service):
self.idmax = None
self.subject_base = None
self.open_ports = []
+self.run_init_memberof = True
if realm_name:
self.suffix = util.realm_to_suffix(self.realm_name)
self.__setup_sub_dict()
@@ -295,6 +296,7 @@ class DsInstance(service.Service):
repl.setup_replication(self.master_fqdn,
r_binddn="cn=Directory Manager",
r_bindpw=self.dm_password)
+self.run_init_memberof = repl.needs_memberof_fixup()
def __enable(self):
self.backup_state("enabled", self.is_enabled())
@@ -433,6 +435,10 @@ class DsInstance(service.Service):
self._ldap_mod("memberof-conf.ldif")
def init_memberof(self):
+
+if not self.run_init_memberof:
+return
+
self._ldap_mod("memberof-task.ldif", self.sub_dict)
# Note, keep dn in sync with dn in install/share/memberof-task.ldif
dn = "cn=IPA install %s,cn=memberof task,cn=tasks,cn=config" % self.sub_dict["TIME"]
diff --git a/ipaserver/install/replication.py b/ipaserver/install/replication.py
index 986fb5e83e9296ad415f54fa106170391904b3c0..a048cac4d070110b2071ec04b3880f812eb129c4 100644
--- a/ipaserver/install/replication.py
+++ b/ipaserver/install/replication.py
@@ -107,6 +107,7 @@ class ReplicationManager(object):
self.starttls = starttls
tmp = util.realm_to_suffix(realm)
self.suffix = str(DN(tmp)).lower()
+self.need_memberof_fixup = False
# If we are passed a password we'll use it as the DM password
# otherwise we'll do a GSSAPI bind.
@@ -433,6 +434,7 @@ class ReplicationManager(object):
which use a different name on each side. If master is None then
isn't a dogtag replication agreement.
"""
+
cn, dn = self.agreem
