Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
JR Aquino wrote: On Oct 7, 2011, at 11:14 AM, Simo Sorce wrote: On Mon, 2011-10-03 at 18:17 -0400, Simo Sorce wrote: On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote: Newer 389ds servers have a new option to have a different set of filtered attributes from normal replication. This has been added in order to allow DS to replicate memberof attributes only during a total update so that we do not need to run a fixup memberof task on a replica at install time. This task is quite inefficient for big database and can take a long time. By replicating memberof while the DB is locked we are guaranteed the memberof list is consistent so we do not need a fixup. This patch allows to enable this feature dynamically. If the server does not yet support the new option it falls back to the previous behavior. Fixes: https://fedorahosted.org/freeipa/ticket/1794 I am sending the patch but it has been jointly developed at various stages by Nathan, JR, and me. Simo. After some thinking I found out that we cannot commit this patch until the memberof plugin is converted to use the new transaction interfaces for plugins, as otherwise it is possible to run into race conditions where the member/memberof relations are not settled if a new replica is installed while member attributes are being changed. Granted the race is quite small and unlikely but real. So please test and ack it, but we need to defer pushing to stable branches until ds copes. I think it is ok to push to master for testing, DS should have the necessary support by the time we make another stable release from master and in our test environments I am sure we will never hit the race. After some more testing I found a small bug that can cause issues in some conditions, new patch attached. Simo. ACK with 389-ds-base-1.2.10-0.4.a4 Pushed to master ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
On Oct 7, 2011, at 11:14 AM, Simo Sorce wrote: On Mon, 2011-10-03 at 18:17 -0400, Simo Sorce wrote: On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote: Newer 389ds servers have a new option to have a different set of filtered attributes from normal replication. This has been added in order to allow DS to replicate memberof attributes only during a total update so that we do not need to run a fixup memberof task on a replica at install time. This task is quite inefficient for big database and can take a long time. By replicating memberof while the DB is locked we are guaranteed the memberof list is consistent so we do not need a fixup. This patch allows to enable this feature dynamically. If the server does not yet support the new option it falls back to the previous behavior. Fixes: https://fedorahosted.org/freeipa/ticket/1794 I am sending the patch but it has been jointly developed at various stages by Nathan, JR, and me. Simo. After some thinking I found out that we cannot commit this patch until the memberof plugin is converted to use the new transaction interfaces for plugins, as otherwise it is possible to run into race conditions where the member/memberof relations are not settled if a new replica is installed while member attributes are being changed. Granted the race is quite small and unlikely but real. So please test and ack it, but we need to defer pushing to stable branches until ds copes. I think it is ok to push to master for testing, DS should have the necessary support by the time we make another stable release from master and in our test environments I am sure we will never hit the race. After some more testing I found a small bug that can cause issues in some conditions, new patch attached. Simo. ACK with 389-ds-base-1.2.10-0.4.a4 ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
On 10/03/2011 04:39 PM, JR Aquino wrote: On Oct 3, 2011, at 3:17 PM, Simo Sorce wrote: On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote: Newer 389ds servers have a new option to have a different set of filtered attributes from normal replication. This has been added in order to allow DS to replicate memberof attributes only during a total update so that we do not need to run a fixup memberof task on a replica at install time. This task is quite inefficient for big database and can take a long time. By replicating memberof while the DB is locked we are guaranteed the memberof list is consistent so we do not need a fixup. This patch allows to enable this feature dynamically. If the server does not yet support the new option it falls back to the previous behavior. Fixes: https://fedorahosted.org/freeipa/ticket/1794 I am sending the patch but it has been jointly developed at various stages by Nathan, JR, and me. Simo. After some thinking I found out that we cannot commit this patch until the memberof plugin is converted to use the new transaction interfaces for plugins, as otherwise it is possible to run into race conditions where the member/memberof relations are not settled if a new replica is installed while member attributes are being changed. Granted the race is quite small and unlikely but real. So please test and ack it, but we need to defer pushing to stable branches until ds copes. I think it is ok to push to master for testing, DS should have the necessary support by the time we make another stable release from master and in our test environments I am sure we will never hit the race. Do we know which 389-ds-base incorporates the new option? I would like to test and ack, but I'm not sure if I have a fixed 389-ds-base installed. 1.2.10.a1 - in updates-testing ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote: Newer 389ds servers have a new option to have a different set of filtered attributes from normal replication. This has been added in order to allow DS to replicate memberof attributes only during a total update so that we do not need to run a fixup memberof task on a replica at install time. This task is quite inefficient for big database and can take a long time. By replicating memberof while the DB is locked we are guaranteed the memberof list is consistent so we do not need a fixup. This patch allows to enable this feature dynamically. If the server does not yet support the new option it falls back to the previous behavior. Fixes: https://fedorahosted.org/freeipa/ticket/1794 I am sending the patch but it has been jointly developed at various stages by Nathan, JR, and me. Simo. After some thinking I found out that we cannot commit this patch until the memberof plugin is converted to use the new transaction interfaces for plugins, as otherwise it is possible to run into race conditions where the member/memberof relations are not settled if a new replica is installed while member attributes are being changed. Granted the race is quite small and unlikely but real. So please test and ack it, but we need to defer pushing to stable branches until ds copes. I think it is ok to push to master for testing, DS should have the necessary support by the time we make another stable release from master and in our test environments I am sure we will never hit the race. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] #1794 - Speed up replica setup
On Oct 3, 2011, at 3:17 PM, Simo Sorce wrote: On Mon, 2011-10-03 at 16:20 -0400, Simo Sorce wrote: Newer 389ds servers have a new option to have a different set of filtered attributes from normal replication. This has been added in order to allow DS to replicate memberof attributes only during a total update so that we do not need to run a fixup memberof task on a replica at install time. This task is quite inefficient for big database and can take a long time. By replicating memberof while the DB is locked we are guaranteed the memberof list is consistent so we do not need a fixup. This patch allows to enable this feature dynamically. If the server does not yet support the new option it falls back to the previous behavior. Fixes: https://fedorahosted.org/freeipa/ticket/1794 I am sending the patch but it has been jointly developed at various stages by Nathan, JR, and me. Simo. After some thinking I found out that we cannot commit this patch until the memberof plugin is converted to use the new transaction interfaces for plugins, as otherwise it is possible to run into race conditions where the member/memberof relations are not settled if a new replica is installed while member attributes are being changed. Granted the race is quite small and unlikely but real. So please test and ack it, but we need to defer pushing to stable branches until ds copes. I think it is ok to push to master for testing, DS should have the necessary support by the time we make another stable release from master and in our test environments I am sure we will never hit the race. Do we know which 389-ds-base incorporates the new option? I would like to test and ack, but I'm not sure if I have a fixed 389-ds-base installed. ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
