Re: [Freeipa-devel] [PATCH] 142 Do not skip SSSD known hosts in ipa-client-install --ssh-trust-dns
On 06/26/2013 09:43 PM, Jan Pazdziora wrote: On Tue, Jun 25, 2013 at 10:54:55AM +0200, Jan Cholasta wrote: the attached patch fixes https://fedorahosted.org/freeipa/ticket/3705. Ack. Pushed to master, ipa-3-2. Martin ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH 0075] Change group ownership of CRL publish directory
On 06/21/2013 02:18 PM, Tomas Babej wrote:
On 06/21/2013 02:15 PM, Martin Kosek wrote:
On 06/21/2013 02:11 PM, Tomas Babej wrote:
On 06/20/2013 06:00 PM, Simo Sorce wrote:
On Thu, 2013-06-20 at 17:47 +0200, Martin Kosek wrote:
On 06/20/2013 05:44 PM, Simo Sorce wrote:
On Thu, 2013-06-20 at 17:33 +0200, Martin Kosek wrote:
On 06/20/2013 05:15 PM, Tomas Babej wrote:
Hi,
Spec file modified so that /var/lib/ipa/pki-ca/publish/ is owned
by pkiuser group.
https://fedorahosted.org/freeipa/ticket/3727
Tomas
NACK. This won't fly. pkiuser is created by FreeIPA when server is
installed,
thus you cannot just simply change ownership in our spec file because in
the
time when package is installed or updated, pkiuser may not exist.
I think you need to delete the %attr from spec file and set the correct
ownership during ipa-{server,ca}-install. When CA is configured, we
should
also
probably let ipa-upgradeconfig check this directory and amend when
necessary
(to fix affected IPA CA instances).
Probably even better to not create the directory via rpm at all, but
make ipa-ca-install create it and remove it when --uninstall is run.
Simo.
This could also work, sure. Could we then at least mark this directory in
our
spec file as %ghost? So that rpm -qf /var/lib/ipa/pki-ca/publish/ gives
some
information?
I guess so.
Simo.
Updated version attached.
Tomas
Looks good by reading (I did not test it), maybe just one nitpick:
+root_logger.warning(Error while removing CRL publish
+directory: %s % str(e))
This should read:
+root_logger.warning(Error while removing CRL publish
+directory: %s, e)
We do not need to format the string before it is really logged and we also do
not need to convert it to str as we already requested the conversion to
string by %s.
Martin
Fixed.
Tomas
The patch itself works fine, but there are still SELinux related questions and
concerns which may also affect the patch (currently it does not work with
enforced SELinux).
I posted them to the relevant Bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=976308
Martin
___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] 426 Create Firefox configuration extension on CA-less install
On 06/27/2013 11:48 AM, Petr Vobornik wrote:
On 06/27/2013 12:26 AM, Ana Krivokapic wrote:
On 06/26/2013 08:59 AM, Petr Vobornik wrote:
Create:
* kerberosauth.xpi
* krb.js
even when --http_pkcs12 option is used.
https://fedorahosted.org/freeipa/ticket/3747
In the logging statement in httpinstance.py, 'configure.jar' should be
in lower case:
root_logger.warning('Object-signing certificate was not
found. '
'Configure.jar was not created.')
--
Regards,
Ana Krivokapic
Thanks.
I've also changed the sentence structure to avoid starting the sentence with
lower case letter.
___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel
ACK
--
Regards,
Ana Krivokapic
Associate Software Engineer
FreeIPA team
Red Hat Inc.
___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] 412 Remove entitlement support
On 26.6.2013 14:03, Tomas Babej wrote:
On 06/19/2013 10:31 AM, Petr Vobornik wrote:
On 06/19/2013 10:13 AM, Martin Kosek wrote:
Entitlements code was not tested nor supported upstream since
version 3.0. Remove the associated code.
https://fedorahosted.org/freeipa/ticket/3739
As agreed on Triage meeting, I plan to push this patch to ipa-3-2 and
master
branches.
Martin
ACK on Web UI part.
ACK on the IPA part
Tomas
ipa-upgradeconfig fails for me when upgrading from version with
entitlement plugin to version without entitlement plugin:
2013-06-26T22:22:43Z DEBUG /usr/sbin/ipa-upgradeconfig was invoked with
options: {'debug': False, 'quiet': True}
2013-06-26T22:22:43Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2013-06-26T22:22:43Z DEBUG importing all plugin modules in
'/usr/lib/python2.7/site-packages/ipalib/plugins'...
snip
2013-06-26T22:22:43Z DEBUG importing plugin module
'/usr/lib/python2.7/site-packages/ipalib/plugins/entitle.py'
2013-06-26T22:22:43Z DEBUG File
/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py,
line 614, in run_script
return_value = main_function()
File /usr/sbin/ipa-upgradeconfig, line 872, in main
api.finalize()
File /usr/lib/python2.7/site-packages/ipalib/plugable.py, line 674,
in finalize
self.__do_if_not_done('load_plugins')
File /usr/lib/python2.7/site-packages/ipalib/plugable.py, line 454,
in __do_if_not_done
getattr(self, name)()
File /usr/lib/python2.7/site-packages/ipalib/plugable.py, line 613,
in load_plugins
self.import_plugins('ipalib')
File /usr/lib/python2.7/site-packages/ipalib/plugable.py, line 655,
in import_plugins
__import__(fullname)
File /usr/lib/python2.7/site-packages/ipalib/plugins/entitle.py,
line 180, in module
class entitle(LDAPObject):
File /usr/lib/python2.7/site-packages/ipalib/plugins/entitle.py,
line 184, in entitle
container_dn = api.env.container_entitlements
2013-06-26T22:22:43Z DEBUG The ipa-upgradeconfig command failed,
exception: AttributeError: 'Env' object has no attribute
'container_entitlements'
Honza
--
Jan Cholasta
___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] 426 Create Firefox configuration extension on CA-less install
On 06/27/2013 12:08 PM, Ana Krivokapic wrote:
On 06/27/2013 11:48 AM, Petr Vobornik wrote:
On 06/27/2013 12:26 AM, Ana Krivokapic wrote:
On 06/26/2013 08:59 AM, Petr Vobornik wrote:
Create:
* kerberosauth.xpi
* krb.js
even when --http_pkcs12 option is used.
https://fedorahosted.org/freeipa/ticket/3747
In the logging statement in httpinstance.py, 'configure.jar' should be
in lower case:
root_logger.warning('Object-signing certificate was not
found. '
'Configure.jar was not created.')
--
Regards,
Ana Krivokapic
Thanks.
I've also changed the sentence structure to avoid starting the
sentence with lower case letter.
___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel
ACK
Pushed to master, ipa-3-2.
master: f5bc155f56a3673a419f921db18e64f8647065ec
ipa-3-2: 2f3d918d749eb6341fd6269435d49fc569ee3d5c
--
Petr³
___
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel
[Freeipa-devel] [PATCHES] 143-147 Improve performance with large groups
Hi,
the attached patches are an attempt to solve
https://fedorahosted.org/freeipa/ticket/3706 without actually removing
ipausers.
I have done some basic timing on IPA with 10k users, the results are:
ipa user-add: 18 s originally, 4 s with the patches
ipa user-del: 54 s originally, 7 s with the patches
Other commands should be affected as well, especially del commands
(deleting an entry triggers a originally unindexed search in the
referint plugin) and member manipulation commands (full member list is
no longer fetched and stored back when adding/removing members).
Patch 147 fixes https://fedorahosted.org/freeipa/ticket/3743.
Honza
--
Jan Cholasta
From ddca9fbf73e985fb8a6e5ea43b0e2e68c957377b Mon Sep 17 00:00:00 2001
From: Jan Cholasta jchol...@redhat.com
Date: Tue, 25 Jun 2013 12:58:37 +
Subject: [PATCH 1/5] Use LDAP search instead of *group_show to check if a
group exists.
https://fedorahosted.org/freeipa/ticket/3706
---
ipalib/plugins/aci.py | 9 +
ipalib/plugins/baseldap.py | 5 +
ipalib/plugins/config.py| 2 +-
ipalib/plugins/hostgroup.py | 4 ++--
ipalib/plugins/netgroup.py | 2 +-
ipalib/plugins/user.py | 2 +-
6 files changed, 15 insertions(+), 9 deletions(-)
diff --git a/ipalib/plugins/aci.py b/ipalib/plugins/aci.py
index dab209e..a7f85dd 100644
--- a/ipalib/plugins/aci.py
+++ b/ipalib/plugins/aci.py
@@ -252,7 +252,8 @@ def _make_aci(ldap, current, aciname, kw):
elif group:
# Not so friendly with groups. This will raise
try:
-entry_attrs = api.Command['group_show'](kw['group'])['result']
+group_dn = api.Object['group'].get_dn_if_exists(kw['group'])
+entry_attrs = {'dn': group_dn}
except errors.NotFound:
raise errors.NotFound(reason=_(Group '%s' does not exist) % kw['group'])
@@ -269,7 +270,7 @@ def _make_aci(ldap, current, aciname, kw):
a.set_target_attr(kw['attrs'])
if valid['memberof']:
try:
-api.Command['group_show'](kw['memberof'])
+api.Object['group'].get_dn_if_exists(kw['memberof'])
except errors.NotFound:
api.Object['group'].handle_not_found(kw['memberof'])
groupdn = _group_from_memberof(kw['memberof'])
@@ -291,8 +292,8 @@ def _make_aci(ldap, current, aciname, kw):
a.set_target(target)
if valid['targetgroup']:
# Purposely no try here so we'll raise a NotFound
-entry_attrs = api.Command['group_show'](kw['targetgroup'])['result']
-target = 'ldap:///%s' % entry_attrs['dn']
+group_dn = api.Object['group'].get_dn_if_exists(kw['targetgroup'])
+target = 'ldap:///%s' % group_dn
a.set_target(target)
if valid['subtree']:
# See if the subtree is a full URI
diff --git a/ipalib/plugins/baseldap.py b/ipalib/plugins/baseldap.py
index bb0de98..1312107 100644
--- a/ipalib/plugins/baseldap.py
+++ b/ipalib/plugins/baseldap.py
@@ -493,6 +493,11 @@ class LDAPObject(Object):
assert isinstance(parent_dn, DN)
return parent_dn
+def get_dn_if_exists(self, *keys, **kwargs):
+dn = self.get_dn(*keys, **kwargs)
+entry = self.backend.get_entry(dn, [''])
+return entry.dn
+
def get_primary_key_from_dn(self, dn):
assert isinstance(dn, DN)
try:
diff --git a/ipalib/plugins/config.py b/ipalib/plugins/config.py
index 33eb174..b9cf050 100644
--- a/ipalib/plugins/config.py
+++ b/ipalib/plugins/config.py
@@ -213,7 +213,7 @@ class config_mod(LDAPUpdate):
if 'ipadefaultprimarygroup' in entry_attrs:
group=entry_attrs['ipadefaultprimarygroup']
try:
-api.Command['group_show'](group)
+api.Object['group'].get_dn_if_exists(group)
except errors.NotFound:
raise errors.NotFound(message=_(The group doesn't exist))
kw = {}
diff --git a/ipalib/plugins/hostgroup.py b/ipalib/plugins/hostgroup.py
index 9fb1029..bc10994 100644
--- a/ipalib/plugins/hostgroup.py
+++ b/ipalib/plugins/hostgroup.py
@@ -122,7 +122,7 @@ class hostgroup_add(LDAPCreate):
assert isinstance(dn, DN)
try:
# check duplicity with hostgroups first to provide proper error
-netgroup = api.Command['hostgroup_show'](keys[-1])
+api.Object['hostgroup'].get_dn_if_exists(keys[-1])
self.obj.handle_duplicate_entry(*keys)
except errors.NotFound:
pass
@@ -130,7 +130,7 @@ class hostgroup_add(LDAPCreate):
try:
# when enabled, a managed netgroup is created for every hostgroup
# make sure that the netgroup can be created
-netgroup = api.Command['netgroup_show'](keys[-1])
+api.Object['netgroup'].get_dn_if_exists(keys[-1])
raise errors.DuplicateEntry(message=unicode(_(\
u'netgroup
Re: [Freeipa-devel] [PATCH] 412 Remove entitlement support
On 06/27/2013 12:32 PM, Jan Cholasta wrote:
On 26.6.2013 14:03, Tomas Babej wrote:
On 06/19/2013 10:31 AM, Petr Vobornik wrote:
On 06/19/2013 10:13 AM, Martin Kosek wrote:
Entitlements code was not tested nor supported upstream since
version 3.0. Remove the associated code.
https://fedorahosted.org/freeipa/ticket/3739
As agreed on Triage meeting, I plan to push this patch to ipa-3-2 and
master
branches.
Martin
ACK on Web UI part.
ACK on the IPA part
Tomas
ipa-upgradeconfig fails for me when upgrading from version with entitlement
plugin to version without entitlement plugin:
2013-06-26T22:22:43Z DEBUG /usr/sbin/ipa-upgradeconfig was invoked with
options: {'debug': False, 'quiet': True}
2013-06-26T22:22:43Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2013-06-26T22:22:43Z DEBUG importing all plugin modules in
'/usr/lib/python2.7/site-packages/ipalib/plugins'...
snip
2013-06-26T22:22:43Z DEBUG importing plugin module
'/usr/lib/python2.7/site-packages/ipalib/plugins/entitle.py'
2013-06-26T22:22:43Z DEBUG File
/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py, line
614,
in run_script
return_value = main_function()
File /usr/sbin/ipa-upgradeconfig, line 872, in main
api.finalize()
File /usr/lib/python2.7/site-packages/ipalib/plugable.py, line 674, in
finalize
self.__do_if_not_done('load_plugins')
File /usr/lib/python2.7/site-packages/ipalib/plugable.py, line 454, in
__do_if_not_done
getattr(self, name)()
File /usr/lib/python2.7/site-packages/ipalib/plugable.py, line 613, in
load_plugins
self.import_plugins('ipalib')
File /usr/lib/python2.7/site-packages/ipalib/plugable.py, line 655, in
import_plugins
__import__(fullname)
File /usr/lib/python2.7/site-packages/ipalib/plugins/entitle.py, line 180,
in module
class entitle(LDAPObject):
File /usr/lib/python2.7/site-packages/ipalib/plugins/entitle.py, line 184,
in entitle
container_dn = api.env.container_entitlements
2013-06-26T22:22:43Z DEBUG The ipa-upgradeconfig command failed, exception:
AttributeError: 'Env' object has no attribute 'container_entitlements'
Honza
This happens because we run ipa-upgradeconfig in %post while there was still
entitlements plugin. I think that clean solution for this plugin (and also for
other future occurrences of this issue) is to run upgrade/server restart
process only in %posttrans.
In the end, I iterated to the attached patch. With this spec change, I was able
to upgrade from FreeIPA 3.2 to current master version without any entitlements
related upgrade error.
Adding Alexander and Rob to CC to double-check this upgrade-related change, I
want to be sure I didn't do something stupid.
Martin
From 4c0f2dafdd24941c560ec463d92c44ff6a772196 Mon Sep 17 00:00:00 2001
From: Martin Kosek mko...@redhat.com
Date: Thu, 27 Jun 2013 15:37:05 +0200
Subject: [PATCH] Run server upgrade and restart in posttrans
Running server upgrade or restart in %post or %postun may cause issues
when there are still parts of old FreeIPA software (like entitlements
plugin).
https://fedorahosted.org/freeipa/ticket/3739
---
freeipa.spec.in | 25 -
1 file changed, 16 insertions(+), 9 deletions(-)
diff --git a/freeipa.spec.in b/freeipa.spec.in
index fcbad3e975108ec5b9265a05600fc3f36b6a2cd6..9f7146e4ae371cd3c55d1a9c2be7b2eb10c1aefe 100644
--- a/freeipa.spec.in
+++ b/freeipa.spec.in
@@ -467,13 +467,22 @@ rm -rf %{buildroot}
# END
if [ $1 -gt 1 ] ; then
/bin/systemctl condrestart certmonger.service 21 || :
-/usr/sbin/ipa-upgradeconfig --quiet /dev/null || :
fi
%posttrans server
# This must be run in posttrans so that updates from previous
# execution that may no longer be shipped are not applied.
/usr/sbin/ipa-ldap-updater --upgrade --quiet /dev/null || :
+/usr/sbin/ipa-upgradeconfig --quiet /dev/null || :
+
+# Restart IPA processes. This must be also run in postrans so that plugins
+# and software is in consistent state
+python -c import sys; from ipaserver.install import installutils; sys.exit(0 if installutils. is_ipa_configured() else 1); /dev/null 21
+# NOTE: systemd specific section
+if [ $? -eq 0 ]; then
+/bin/systemctl try-restart ipa.service /dev/null 21 || :
+fi
+# END
%preun server
if [ $1 = 0 ]; then
@@ -483,14 +492,6 @@ if [ $1 = 0 ]; then
# END
fi
-%postun server
-if [ $1 -ge 1 ]; then
-# NOTE: systemd specific section
-/bin/systemctl --quiet is-active ipa.service /dev/null \
-/bin/systemctl try-restart ipa.service /dev/null 21 || :
-# END
-fi
-
%pre server
# Stop ipa_kpasswd if it exists before upgrading so we don't have a
# zombie process when we're done.
@@ -510,6 +511,8 @@ fi
%post server-trust-ad
%{_sbindir}/update-alternatives --install %{_libdir}/krb5/plugins/libkrb5/winbind_krb5_locator.so \
winbind_krb5_locator.so /dev/null 90
+
+%posttrans server-trust-ad
python -c import
Re: [Freeipa-devel] [PATCH] 122 Enable SASL mapping fallback
On 06/26/2013 03:02 PM, Jan Cholasta wrote: On 26.6.2013 13:42, Martin Kosek wrote: As 389-ds-base 1.3.1.1 requested in the ticket is already out, I think we should revive these patches. Martin Rebased patch attached. Honza ACK. Pushed to master, ipa-3-2 (rebased). Martin ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCHES] 143-147 Improve performance with large groups
On 06/27/2013 04:55 PM, Jan Cholasta wrote: Hi, the attached patches are an attempt to solve https://fedorahosted.org/freeipa/ticket/3706 without actually removing ipausers. I have done some basic timing on IPA with 10k users, the results are: ipa user-add: 18 s originally, 4 s with the patches ipa user-del: 54 s originally, 7 s with the patches Other commands should be affected as well, especially del commands (deleting an entry triggers a originally unindexed search in the referint plugin) and member manipulation commands (full member list is no longer fetched and stored back when adding/removing members). Patch 147 fixes https://fedorahosted.org/freeipa/ticket/3743. Honza Thanks for this effort! I quickly went through the patches, they mostly look harmless. Except the following: Subject: [PATCH 4/5] Add missing substring indices for attributes managed by the referint plugin. AFAIK, sub index is a very expensive index - as we discussed offline - adding Rich to advise and confirm this. I think you added it because some plugin was doing substring/wildcard search when an LDAP entry was being deleted - did you identify which one it is? Because I would rather get rid of the bad search than adding so many sub indices. Secondly, did you also check Web UI performance? I think we could noticeable improve user/group lists performance if we added a new (hidden) option to suppress loading membership information which could then be utilized by Web UI. Adding Petr Vobornik to CC to consider this. Martin ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCHES] 143-147 Improve performance with large groups
On 06/27/2013 09:31 AM, Jan Cholasta wrote: On 27.6.2013 17:23, Martin Kosek wrote: Thanks for this effort! I quickly went through the patches, they mostly look harmless. Except the following: Subject: [PATCH 4/5] Add missing substring indices for attributes managed by the referint plugin. AFAIK, sub index is a very expensive index - as we discussed offline - adding Rich to advise and confirm this. I think you added it because some plugin was doing substring/wildcard search when an LDAP entry was being deleted - did you identify which one it is? Because I would rather get rid of the bad search than adding so many sub indices. The search is hard-coded in the referint plugin, see https://git.fedorahosted.org/cgit/389/ds.git/tree/ldap/servers/plugins/referint/referint.c#n745. Not sure if it makes sense to do a wildcard/substr search here - please file a ticket with 389 to investigate. sub index isn't necessarily a bad thing - in this case it may be more beneficial than harmful - if you have enough nsslapd-idlistscanlimit to hold the entire candidate list in a single id list without hurting performance (i.e. a list of 1 entries is probably ok - a list of 100 entries is not) Secondly, did you also check Web UI performance? I think we could noticeable improve user/group lists performance if we added a new (hidden) option to suppress loading membership information which could then be utilized by Web UI. Adding Petr Vobornik to CC to consider this. No, not yet. Honza ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCHES] 143-147 Improve performance with large groups
On 27.6.2013 17:34, Rich Megginson wrote: On 06/27/2013 09:31 AM, Jan Cholasta wrote: The search is hard-coded in the referint plugin, see https://git.fedorahosted.org/cgit/389/ds.git/tree/ldap/servers/plugins/referint/referint.c#n745. Not sure if it makes sense to do a wildcard/substr search here - please file a ticket with 389 to investigate. https://fedorahosted.org/389/ticket/47411 -- Jan Cholasta ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] DNSSEC support design considerations: migration to RBTDB
On 21.6.2013 16:19, Simo Sorce wrote: On Thu, 2013-06-20 at 14:30 +0200, Petr Spacek wrote: On 23.5.2013 16:32, Simo Sorce wrote: On Thu, 2013-05-23 at 14:35 +0200, Petr Spacek wrote: It looks that we agree on nearly all points (I apologize if overlooked something). I will prepare a design document for transition to RBTDB and then another design document for DNSSEC implementation. The current version of the design is available at: https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/Design/RBTDB Great write-up, thanks. There are several questions inside (search for text Question, it should find all of them). I would like to get your opinion about the problems. Note that 389 DS team decided to implement RFC 4533 (syncrepl), so persistent search is definitely obsolete and we can do synchronization in some clever way. Answering inline here after quoting the questions for the doc: Periodical re-synchronization Questions * Do we still need periodical re-synchronization if 389 DS team implements RFC 4533 (syncrepl)? It wasn't considered in the initial design. We probably do. We have to be especially careful of the case when a replica is re-initialized. We should either automatically detect that this is happening or change ipa-replica-manage to kick named some how. We also need a tool or maybe a special attribute in LDAP that is monitored so that we can tell bind-dyndb-ldap to do a full rebuild of the cache on demand. This way admins can force a rebuild if they end up noticing something wrong. Is it acceptable to let admin to delete files restart named manually? I don't wont to overcomplicate things at the beginning ... * What about dynamic updates during re-synchronization? Should we return a temporary error ? Or maybe just queue up the change and apply it right after the resync operation has finished ? Unfortunately, the only reasonable error code is SERVFAIL. It is completely up to client if it tries to do update again or not. I personally don't like queuing of updates because it confuses clients: Update is accepted by server but the client still can see an old value (for limited period of time). * How to get sorted list of entries from LDAP? Use LDAP server-side sorting? Do we have necessary indices? We can do client side sorting as well I guess, I do not have a strong opinion here. The main reason why you need ordering is to detect delete records right ? Exactly. I realized that server-side sorting doesn't make sense because we plan to use syncrepl, so there is nothing to sort - only the flow of incremental updates. Is thee a way to mark rdtdb records as updated instead (with a generation number) and then do a second pass on the rbtdb tree and remove any record that was not updated with the generation number ? There is no 'generation' number, but we can extend the auxiliary database (i.e. database with UUID=DNS name mapping) with generation number. We will get UUID along with each update from LDAP, so we can simply use UUID for database lookup. Then we can go though the UUID database and delete all records which don't have generation == expected_value. This would also allow us to keep accepting dynamic updates by simply marking records as generation+1 so that the resync will not overwrite records that are updated during the resync phase. I agree. The simplest variant can solve the basic case where 1 update was received during re-synchronization. Proposed (simple) solution: 1) At the beginning of re-synchronization, set curr_gen = prev_gen+1 2) For each entry in LDAP do (via syncrepl): - Only if entry['gen'] curr_gen: -- Overwrite data in local RBTDB with data from LDAP -- Overwrite entry['gen'] = curr_gen - Else: Do nothing In parallel: 1) Update request received from a client 2) Write new data to LDAP (syncrepl should cope with this) 3) Read UUID from LDAP (via RFC 4527 controls) 4) Write curr_gen to UUID database 5) Write data to local RBTDB 6) Reply 'update accepted' to the client Crash at any time should not hurt: Curr_gen will be incremented on restart and re-sychronization will be restarted. The worst case is that update will be stored in LDAP but client will not get reply because of crash (i.e. client times out). There is a drawback: Two or more successive updates to a single entry can create race condition, as described at https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/Design/RBTDB#Raceconditions1 . The reason is that generation number is not incremented each time, but only overwritten with current global value (i.e. old + 1). I don't like the other option with incrementing generation number. It could create nasty corner cases during re-synchronization and handling updates made directly in LDAP/by other DNS server. It is not nice, but I think that we can live with it. The important fact is that
Re: [Freeipa-devel] DNSSEC support design considerations: migration to RBTDB
On Thu, 2013-06-27 at 18:23 +0200, Petr Spacek wrote: On 21.6.2013 16:19, Simo Sorce wrote: On Thu, 2013-06-20 at 14:30 +0200, Petr Spacek wrote: On 23.5.2013 16:32, Simo Sorce wrote: On Thu, 2013-05-23 at 14:35 +0200, Petr Spacek wrote: It looks that we agree on nearly all points (I apologize if overlooked something). I will prepare a design document for transition to RBTDB and then another design document for DNSSEC implementation. The current version of the design is available at: https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/Design/RBTDB Great write-up, thanks. There are several questions inside (search for text Question, it should find all of them). I would like to get your opinion about the problems. Note that 389 DS team decided to implement RFC 4533 (syncrepl), so persistent search is definitely obsolete and we can do synchronization in some clever way. Answering inline here after quoting the questions for the doc: Periodical re-synchronization Questions * Do we still need periodical re-synchronization if 389 DS team implements RFC 4533 (syncrepl)? It wasn't considered in the initial design. We probably do. We have to be especially careful of the case when a replica is re-initialized. We should either automatically detect that this is happening or change ipa-replica-manage to kick named some how. We also need a tool or maybe a special attribute in LDAP that is monitored so that we can tell bind-dyndb-ldap to do a full rebuild of the cache on demand. This way admins can force a rebuild if they end up noticing something wrong. Is it acceptable to let admin to delete files restart named manually? I don't wont to overcomplicate things at the beginning ... Sure, probably fine, we can have a tool that simply just does that for starters, and later on we can make it do more complex things if needed. * What about dynamic updates during re-synchronization? Should we return a temporary error ? Or maybe just queue up the change and apply it right after the resync operation has finished ? Unfortunately, the only reasonable error code is SERVFAIL. It is completely up to client if it tries to do update again or not. I personally don't like queuing of updates because it confuses clients: Update is accepted by server but the client still can see an old value (for limited period of time). Another option is to mark fields so that they are not updated with older values, and just allow the thing to succeed. * How to get sorted list of entries from LDAP? Use LDAP server-side sorting? Do we have necessary indices? We can do client side sorting as well I guess, I do not have a strong opinion here. The main reason why you need ordering is to detect delete records right ? Exactly. I realized that server-side sorting doesn't make sense because we plan to use syncrepl, so there is nothing to sort - only the flow of incremental updates. Syncrepl includes notice of deletions too, right ? Is thee a way to mark rdtdb records as updated instead (with a generation number) and then do a second pass on the rbtdb tree and remove any record that was not updated with the generation number ? There is no 'generation' number, but we can extend the auxiliary database (i.e. database with UUID=DNS name mapping) with generation number. We will get UUID along with each update from LDAP, so we can simply use UUID for database lookup. Then we can go though the UUID database and delete all records which don't have generation == expected_value. Yes, something like this should work. This would also allow us to keep accepting dynamic updates by simply marking records as generation+1 so that the resync will not overwrite records that are updated during the resync phase. I agree. The simplest variant can solve the basic case where 1 update was received during re-synchronization. Proposed (simple) solution: 1) At the beginning of re-synchronization, set curr_gen = prev_gen+1 2) For each entry in LDAP do (via syncrepl): - Only if entry['gen'] curr_gen: -- Overwrite data in local RBTDB with data from LDAP -- Overwrite entry['gen'] = curr_gen - Else: Do nothing In parallel: 1) Update request received from a client 2) Write new data to LDAP (syncrepl should cope with this) 3) Read UUID from LDAP (via RFC 4527 controls) 4) Write curr_gen to UUID database 5) Write data to local RBTDB 6) Reply 'update accepted' to the client Crash at any time should not hurt: Curr_gen will be incremented on restart and re-sychronization will be restarted. Yep. The worst case is that update will be stored in LDAP but client will not get reply because of crash (i.e. client times out). Not a big deal. This can always happen for clients, as the
