Re: [Freeipa-devel] [PATCH] 810 speed up indirect member processing
Dne 20.4.2015 v 11:59 Petr Vobornik napsal(a): On 04/20/2015 09:51 AM, Jan Cholasta wrote: Dne 9.4.2015 v 13:56 Petr Vobornik napsal(a): On 04/08/2015 10:21 AM, Jan Cholasta wrote: 4) The processing of memberof should be done even when memberofindirect is not requested, otherwise its value will depend on whether memberofindirect was requested or not. True, but it's the same behavior as before. Could be changed in other patch. OK. Should we file a ticket? AFAIK, memberof and memberofindirect are requested always together atm. Do we have a use case for this change? It's a bug. Use case is when someone requests memberof only, they must get direct memberof values only. In any case, I've opened a ticket about more finer control of fetching members (as was discussed previously in triage and dev mtgs), it might be part of it. https://fedorahosted.org/freeipa/ticket/4995 OK. 5) I would prefer if all membership processing (.convert_attribute_members() and .get_indirect_members()) was done in a single LDAPObject method. Now, as before, get_indirect_members is called before post callbacks and convert_attribute_members after. If it should be combined, it should be done separately. OK, but at least move get_indirect_members to LDAPObject. Moved Thanks, ACK. Pushed to master: 4364ac08c538e3a4253804f523707092b34c2ed2 -- Jan Cholasta -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH] 810 speed up indirect member processing
Dne 9.4.2015 v 13:56 Petr Vobornik napsal(a):
On 04/08/2015 10:21 AM, Jan Cholasta wrote:
Hi,
Dne 31.3.2015 v 12:11 Petr Vobornik napsal(a):
the old implementation tried to get all entries which are member of
group. That means also user. User can't have any members therefore this
costly processing was unnecessary.
New implementation reduces the search only to entries which can have
entries.
Also page size was removed to avoid paging by small pages(default size:
100) which is very slow for many members.
https://fedorahosted.org/freeipa/ticket/4947
Useful to test with #809
1) To search for entries with members, you should search for entries
with the member attribute set ('(member=*)'), not for entries with some
arbitrary object class.
Replaced, new presence index added
2) I don't like how the search in get_memberindirect is limited to an
arbitrary hard-coded subtree. You should go through the object's
attribute_members to figure out which subtrees to search.
The subtree search was removed.
3) Since memberindirect and memberofindirect are not real attributes,
you must define their syntax in ipaldap before you cat set them using
.raw[], otherwise they will be decoded to wrong type.
Added.
4) The processing of memberof should be done even when memberofindirect
is not requested, otherwise its value will depend on whether
memberofindirect was requested or not.
True, but it's the same behavior as before. Could be changed in other
patch.
OK. Should we file a ticket?
5) I would prefer if all membership processing
(.convert_attribute_members() and .get_indirect_members()) was done in a
single LDAPObject method.
Now, as before, get_indirect_members is called before post callbacks and
convert_attribute_members after. If it should be combined, it should be
done separately.
OK, but at least move get_indirect_members to LDAPObject.
Honza
--
Jan Cholasta
--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH] 810 speed up indirect member processing
On 04/20/2015 09:51 AM, Jan Cholasta wrote:
Dne 9.4.2015 v 13:56 Petr Vobornik napsal(a):
On 04/08/2015 10:21 AM, Jan Cholasta wrote:
Hi,
Dne 31.3.2015 v 12:11 Petr Vobornik napsal(a):
the old implementation tried to get all entries which are member of
group. That means also user. User can't have any members therefore this
costly processing was unnecessary.
New implementation reduces the search only to entries which can have
entries.
Also page size was removed to avoid paging by small pages(default size:
100) which is very slow for many members.
https://fedorahosted.org/freeipa/ticket/4947
Useful to test with #809
1) To search for entries with members, you should search for entries
with the member attribute set ('(member=*)'), not for entries with some
arbitrary object class.
Replaced, new presence index added
2) I don't like how the search in get_memberindirect is limited to an
arbitrary hard-coded subtree. You should go through the object's
attribute_members to figure out which subtrees to search.
The subtree search was removed.
3) Since memberindirect and memberofindirect are not real attributes,
you must define their syntax in ipaldap before you cat set them using
.raw[], otherwise they will be decoded to wrong type.
Added.
4) The processing of memberof should be done even when memberofindirect
is not requested, otherwise its value will depend on whether
memberofindirect was requested or not.
True, but it's the same behavior as before. Could be changed in other
patch.
OK. Should we file a ticket?
AFAIK, memberof and memberofindirect are requested always together atm.
Do we have a use case for this change? In any case, I've opened a ticket
about more finer control of fetching members (as was discussed
previously in triage and dev mtgs), it might be part of it.
https://fedorahosted.org/freeipa/ticket/4995
5) I would prefer if all membership processing
(.convert_attribute_members() and .get_indirect_members()) was done in a
single LDAPObject method.
Now, as before, get_indirect_members is called before post callbacks and
convert_attribute_members after. If it should be combined, it should be
done separately.
OK, but at least move get_indirect_members to LDAPObject.
Moved
--
Petr Vobornik
From 68164aa2a5acba14dbfd6bb6d0f2b45cadf6a503 Mon Sep 17 00:00:00 2001
From: Petr Vobornik pvobo...@redhat.com
Date: Tue, 31 Mar 2015 10:59:37 +0200
Subject: [PATCH] speed up indirect member processing
the old implementation tried to get all entries which are member of group.
That means also user. User can't have any members therefore this costly
processing was unnecessary.
New implementation reduces the search only to entries which have members.
Also page size was removed to avoid paging by small pages(default size: 100)
which is very slow for many members.
https://fedorahosted.org/freeipa/ticket/4947
---
install/updates/20-indices.update | 2 +-
ipalib/plugins/baseldap.py| 72 +++
ipalib/plugins/host.py| 2 +-
ipalib/plugins/role.py| 8 ++--
ipapython/ipaldap.py | 2 +
ipaserver/plugins/ldap2.py| 90 ---
6 files changed, 81 insertions(+), 95 deletions(-)
diff --git a/install/updates/20-indices.update b/install/updates/20-indices.update
index a8a432d9c28a7fb4ca74582e36d4c39fd98df2cf..a9ec9f9eb9bcc228dcbb7eba99879ce5a8251e80 100644
--- a/install/updates/20-indices.update
+++ b/install/updates/20-indices.update
@@ -27,7 +27,7 @@ default:nsSystemIndex: false
only:nsIndexType: eq,pres,sub
dn: cn=member,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
-only:nsIndexType: eq,sub
+only:nsIndexType: eq,pres,sub
dn: cn=uniquemember,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
only:nsIndexType: eq,sub
diff --git a/ipalib/plugins/baseldap.py b/ipalib/plugins/baseldap.py
index ca4e54fd269aba9fdcf234f6450382fb7daabe42..b06b570cbc353aa903e5b2218932b8c6e59ce31b 100644
--- a/ipalib/plugins/baseldap.py
+++ b/ipalib/plugins/baseldap.py
@@ -663,6 +663,67 @@ class LDAPObject(Object):
new_attr.append(new_value)
break
+def get_indirect_members(self, entry_attrs, attrs_list):
+if 'memberindirect' in attrs_list:
+self.get_memberindirect(entry_attrs)
+if 'memberofindirect' in attrs_list:
+self.get_memberofindirect(entry_attrs)
+
+def get_memberindirect(self, group_entry):
+
+Get indirect members
+
+
+mo_filter = self.backend.make_filter({'memberof': group_entry.dn})
+filter = self.backend.combine_filters(
+('(member=*)', mo_filter), self.backend.MATCH_ALL)
+try:
+result, truncated = self.backend.find_entries(
+base_dn=self.api.env.basedn,
+filter=filter,
+attrs_list=['member'],
+size_limit=-1, # paged
Re: [Freeipa-devel] [PATCH] 810 speed up indirect member processing
On 04/08/2015 10:21 AM, Jan Cholasta wrote:
Hi,
Dne 31.3.2015 v 12:11 Petr Vobornik napsal(a):
the old implementation tried to get all entries which are member of
group. That means also user. User can't have any members therefore this
costly processing was unnecessary.
New implementation reduces the search only to entries which can have
entries.
Also page size was removed to avoid paging by small pages(default size:
100) which is very slow for many members.
https://fedorahosted.org/freeipa/ticket/4947
Useful to test with #809
1) To search for entries with members, you should search for entries
with the member attribute set ('(member=*)'), not for entries with some
arbitrary object class.
Replaced, new presence index added
2) I don't like how the search in get_memberindirect is limited to an
arbitrary hard-coded subtree. You should go through the object's
attribute_members to figure out which subtrees to search.
The subtree search was removed.
3) Since memberindirect and memberofindirect are not real attributes,
you must define their syntax in ipaldap before you cat set them using
.raw[], otherwise they will be decoded to wrong type.
Added.
4) The processing of memberof should be done even when memberofindirect
is not requested, otherwise its value will depend on whether
memberofindirect was requested or not.
True, but it's the same behavior as before. Could be changed in other patch.
5) I would prefer if all membership processing
(.convert_attribute_members() and .get_indirect_members()) was done in a
single LDAPObject method.
Now, as before, get_indirect_members is called before post callbacks and
convert_attribute_members after. If it should be combined, it should be
done separately.
Honza
--
Petr Vobornik
From 27dfa6d0e2a1815d496f47b4f10b5a6307f51fda Mon Sep 17 00:00:00 2001
From: Petr Vobornik pvobo...@redhat.com
Date: Tue, 31 Mar 2015 10:59:37 +0200
Subject: [PATCH] speed up indirect member processing
the old implementation tried to get all entries which are member of group.
That means also user. User can't have any members therefore this costly
processing was unnecessary.
New implementation reduces the search only to entries which have members.
Also page size was removed to avoid paging by small pages(default size: 100)
which is very slow for many members.
https://fedorahosted.org/freeipa/ticket/4947
---
install/updates/20-indices.update | 2 +-
ipalib/plugins/baseldap.py| 72 +++
ipalib/plugins/host.py| 2 +-
ipalib/plugins/role.py| 8 ++--
ipapython/ipaldap.py | 2 +
ipaserver/plugins/ldap2.py| 90 ---
6 files changed, 81 insertions(+), 95 deletions(-)
diff --git a/install/updates/20-indices.update b/install/updates/20-indices.update
index a8a432d9c28a7fb4ca74582e36d4c39fd98df2cf..a9ec9f9eb9bcc228dcbb7eba99879ce5a8251e80 100644
--- a/install/updates/20-indices.update
+++ b/install/updates/20-indices.update
@@ -27,7 +27,7 @@ default:nsSystemIndex: false
only:nsIndexType: eq,pres,sub
dn: cn=member,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
-only:nsIndexType: eq,sub
+only:nsIndexType: eq,pres,sub
dn: cn=uniquemember,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
only:nsIndexType: eq,sub
diff --git a/ipalib/plugins/baseldap.py b/ipalib/plugins/baseldap.py
index c9f98ed12d3a584597af9b0be08361bceca620e7..9c75bc048283546902a89825ba489d42b26f10fd 100644
--- a/ipalib/plugins/baseldap.py
+++ b/ipalib/plugins/baseldap.py
@@ -663,6 +663,61 @@ class LDAPObject(Object):
new_attr.append(new_value)
break
+def get_memberindirect(self, group_entry):
+
+Get indirect members
+
+
+mo_filter = self.backend.make_filter({'memberof': group_entry.dn})
+filter = self.backend.combine_filters(
+('(member=*)', mo_filter), self.backend.MATCH_ALL)
+try:
+result, truncated = self.backend.find_entries(
+base_dn=self.api.env.basedn,
+filter=filter,
+attrs_list=['member'],
+size_limit=-1, # paged search will get everything anyway
+paged_search=True)
+if truncated:
+raise errors.LimitsExceeded()
+except errors.NotFound:
+result = []
+
+indirect = set()
+for entry in result:
+indirect.update(entry.raw.get('member', []))
+indirect.difference_update(group_entry.raw.get('member', []))
+
+if indirect:
+group_entry.raw['memberindirect'] = list(indirect)
+
+def get_memberofindirect(self, entry):
+
+dn = entry.dn
+filter = self.backend.make_filter(
+{'member': dn, 'memberuser': dn, 'memberhost': dn})
+try:
+result, truncated = self.backend.find_entries(
+
Re: [Freeipa-devel] [PATCH] 810 speed up indirect member processing
Hi,
Dne 31.3.2015 v 12:11 Petr Vobornik napsal(a):
the old implementation tried to get all entries which are member of
group. That means also user. User can't have any members therefore this
costly processing was unnecessary.
New implementation reduces the search only to entries which can have
entries.
Also page size was removed to avoid paging by small pages(default size:
100) which is very slow for many members.
https://fedorahosted.org/freeipa/ticket/4947
Useful to test with #809
1) To search for entries with members, you should search for entries
with the member attribute set ('(member=*)'), not for entries with some
arbitrary object class.
2) I don't like how the search in get_memberindirect is limited to an
arbitrary hard-coded subtree. You should go through the object's
attribute_members to figure out which subtrees to search.
3) Since memberindirect and memberofindirect are not real attributes,
you must define their syntax in ipaldap before you cat set them using
.raw[], otherwise they will be decoded to wrong type.
4) The processing of memberof should be done even when memberofindirect
is not requested, otherwise its value will depend on whether
memberofindirect was requested or not.
5) I would prefer if all membership processing
(.convert_attribute_members() and .get_indirect_members()) was done in a
single LDAPObject method.
Honza
--
Jan Cholasta
--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
[Freeipa-devel] [PATCH] 810 speed up indirect member processing
the old implementation tried to get all entries which are member of
group. That means also user. User can't have any members therefore this
costly processing was unnecessary.
New implementation reduces the search only to entries which can have
entries.
Also page size was removed to avoid paging by small pages(default size:
100) which is very slow for many members.
https://fedorahosted.org/freeipa/ticket/4947
Useful to test with #809
--
Petr Vobornik
From 2d07532771a4a542cb220f87cdc7c1ce115014f6 Mon Sep 17 00:00:00 2001
From: Petr Vobornik pvobo...@redhat.com
Date: Tue, 31 Mar 2015 10:59:37 +0200
Subject: [PATCH] speed up indirect member processing
the old implementation tried to get all entries which are member of group.
That means also user. User can't have any members therefore this costly
processing was unnecessary.
New implementation reduces the search only to entries which can have entries.
Also page size was removed to avoid paging by small pages(default size: 100)
which is very slow for many members.
https://fedorahosted.org/freeipa/ticket/4947
---
ipalib/plugins/baseldap.py | 91
ipalib/plugins/group.py | 1 +
ipalib/plugins/host.py | 2 +-
ipalib/plugins/hostgroup.py | 1 +
ipalib/plugins/netgroup.py | 1 +
ipalib/plugins/permission.py | 1 +
ipalib/plugins/role.py | 8 ++--
ipaserver/plugins/ldap2.py | 90 ---
8 files changed, 101 insertions(+), 94 deletions(-)
diff --git a/ipalib/plugins/baseldap.py b/ipalib/plugins/baseldap.py
index 42509d63de6078b01cc794a005a528d27a168f26..56f871ac309b562130f52abce11816d811f9c89c 100644
--- a/ipalib/plugins/baseldap.py
+++ b/ipalib/plugins/baseldap.py
@@ -538,6 +538,7 @@ class LDAPObject(Object):
rdn_attribute = ''
uuid_attribute = ''
attribute_members = {}
+memberindirect_config = ('*', None)
rdn_is_primary_key = False # Do we need RDN change to do a rename?
password_attributes = []
# Can bind as this entry (has userPassword or krbPrincipalKey)
@@ -663,6 +664,79 @@ class LDAPObject(Object):
break
del entry_attrs[attr]
+def get_memberindirect(self, group_entry):
+
+Get indirect members
+
+Other objects with memberindirect attribute should define
+`memberindirect_config`. This object class filter is used to limit the
+number of members returned to only the members which can contain other
+members.
+
+
+(member_oc, basedn) = self.memberindirect_config
+
+if not member_oc:
+self.log.error(
+('Missing `memberindirect_object_class` IPA object property.\n'
+ 'Skipping indirect member processing.'))
+return
+
+if not basedn:
+basedn = self.api.env.basedn
+
+filter = self.backend.make_filter(
+{'memberof': group_entry.dn, 'objectclass': member_oc},
+None, self.backend.MATCH_ALL
+)
+
+try:
+result, truncated = self.backend.find_entries(
+base_dn=basedn,
+filter=filter,
+attrs_list=['member'],
+size_limit=-1, # paged search will get everything anyway
+paged_search=True)
+if truncated:
+raise errors.LimitsExceeded()
+except errors.NotFound:
+result = []
+
+indirect = set()
+for entry in result:
+indirect.update(entry.raw.get('member', []))
+indirect.difference_update(group_entry.raw.get('member', []))
+
+if indirect:
+group_entry.raw['memberindirect'] = list(indirect)
+
+def get_memberofindirect(self, entry):
+
+dn = entry.dn
+filter = self.backend.make_filter(
+{'member': dn, 'memberuser': dn, 'memberhost': dn})
+try:
+result, truncated = self.backend.find_entries(
+base_dn=self.api.env.basedn,
+filter=filter,
+attrs_list=[''])
+if truncated:
+raise errors.LimitsExceeded()
+except errors.NotFound:
+result = []
+
+direct = set()
+indirect = set(entry.raw.get('memberof', []))
+for group_entry in result:
+dn = str(group_entry.dn)
+if dn in indirect:
+indirect.remove(dn)
+direct.add(dn)
+
+entry.raw['memberof'] = list(direct)
+if indirect:
+entry.raw['memberofindirect'] = list(indirect)
+
def get_password_attributes(self, ldap, dn, entry_attrs):
Search on the entry to determine if it has a password or
@@ -1106,6 +1180,12 @@ last, after all sets and adds.),
)
break
+def get_indirect_members(self, entry_attrs, attrs_list):
+if 'memberindirect' in
