Re: [Freeipa-devel] user-* commands performance issues
On 21.03.2016 15:50, Petr Vobornik wrote: On 03/21/2016 01:02 PM, Martin Basti wrote: On 21.03.2016 12:44, Jan Cholasta wrote: On 21.3.2016 11:00, Petr Vobornik wrote: On 03/17/2016 04:09 PM, Martin Basti wrote: Hello all, I would like to discuss the way how we should improve the speed of user-find commands (and other commands too if possible): 0) Do not do extra search for ipasshpubkey. This is clear, patch posted for review. https://fedorahosted.org/freeipa/ticket/3376 commands: user, stageuser, host, idview 1) make --no-members option visible in CLI https://fedorahosted.org/freeipa/ticket/4995 There was a discussion around devconf that --no-members should be a default behavior of xxx-find commands and I'm for it. +1, although we should be backward compatible with old clients which expect the attributes to be there. Ok, I agree to have --no-members as default for *-find commands, but it doesn't contradict to exposing --no-member option for all commands. +1, xxx-show can have --no-members Reasoning: use case: 'find me all groups which satisfy this filter'. Showing members clutters the output(one group with >500 member makes it unusable) and makes things slow(both on server and CLI side). For xxx-show commands it is a question where I don't have a strong opinion. I think it shouldn't hurt to keep them in -show commands, as there is always only a single entry to process. +1 I don't think we should implement also --no-indirect-members, I think that this kind of granularity is not needed. If --no-members is used, then indirect members will be ignored too. +1 +1 commands: all which use members 2) Limit the amount of searches for memberof[indirect] (group, netgroup, role, hbacrule, sudorule) and search for each dn only once in find commands. We can have configurable option in default.conf (for example memberof_search_limit=100 (0 unlimited)). Find commands will get members only for specified amount and if this limit is exceeded a warning message is shown. I do not like this idea much, I think it should be all or nothing, I prefer to not do this. +1 I'd also avoid anything special here. But there are sometimes cases when the behavior is not good. E.g. a command fails because something is not able to get members and you actually don't care about the members. Not sure if it is was "fixed"(sizelimit=0). But with new member handling it might not be a big issue. However I like the idea of temporary caching inside find commands, where each memberof DN is resolved just once and results are cached in a map and reused in current context of command. This should be improvement mainly for indirect searches, but cache should be faster for direct members than doing internal calls of framework objects. This part is backward compatible, the first part is not. https://fedorahosted.org/freeipa/ticket/5282 What parts of the ticket can be solved with deref plugin? I guess we can get the CNs, but not what is a direct member. Maybe it should be discussed separately. Indirect members are already resolved by a single LDAP search. What kind of additional optimization would you like to do for them? We can use deref plugin to get pkeys from one search in case that pkeys are not part of DN. (I have to investigate if it is worth to do for user-find, I'm not sure if any memberof attributes have pkey that is not part of DN) sudo rule, hbac rule For indirect members, it is one search per entry, but for 1000 users, it is 1000 searches and I would like to have just one for the particular indirect member. are we talking about user-find? If so then it is mostly solved with default --no-member style behavior. But if a user or a group is directly/indirectly a member of a lot of groups(1000) then it might become slow. But caching won't probably help much, not sure. In case that you have one user in 1000 indirect groups, it will not help, but if you have 500 users in the same 1000 indirect groups, it will be ~500 times faster than without caching. Also we have to find out how to solve case that a user wants to show members for *-find commands. is enough to show members when --all flag is used, or should we add separate option to show members in find command (--members, --get-members)? Martin^2 commands: user-find, stageuser-find, possibly all find commands 3) Remove userPassword, krbPrincipalKey from search results This change is not backward compatible, can we do this? https://fedorahosted.org/freeipa/ticket/5281 commands: user-find I'm for it, would like to hear other opinions. Note: it should be only in user-find commands. 'show' has to display it. +1 -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] user-* commands performance issues
On 03/21/2016 01:02 PM, Martin Basti wrote: On 21.03.2016 12:44, Jan Cholasta wrote: On 21.3.2016 11:00, Petr Vobornik wrote: On 03/17/2016 04:09 PM, Martin Basti wrote: Hello all, I would like to discuss the way how we should improve the speed of user-find commands (and other commands too if possible): 0) Do not do extra search for ipasshpubkey. This is clear, patch posted for review. https://fedorahosted.org/freeipa/ticket/3376 commands: user, stageuser, host, idview 1) make --no-members option visible in CLI https://fedorahosted.org/freeipa/ticket/4995 There was a discussion around devconf that --no-members should be a default behavior of xxx-find commands and I'm for it. +1, although we should be backward compatible with old clients which expect the attributes to be there. Ok, I agree to have --no-members as default for *-find commands, but it doesn't contradict to exposing --no-member option for all commands. +1, xxx-show can have --no-members Reasoning: use case: 'find me all groups which satisfy this filter'. Showing members clutters the output(one group with >500 member makes it unusable) and makes things slow(both on server and CLI side). For xxx-show commands it is a question where I don't have a strong opinion. I think it shouldn't hurt to keep them in -show commands, as there is always only a single entry to process. +1 I don't think we should implement also --no-indirect-members, I think that this kind of granularity is not needed. If --no-members is used, then indirect members will be ignored too. +1 +1 commands: all which use members 2) Limit the amount of searches for memberof[indirect] (group, netgroup, role, hbacrule, sudorule) and search for each dn only once in find commands. We can have configurable option in default.conf (for example memberof_search_limit=100 (0 unlimited)). Find commands will get members only for specified amount and if this limit is exceeded a warning message is shown. I do not like this idea much, I think it should be all or nothing, I prefer to not do this. +1 I'd also avoid anything special here. But there are sometimes cases when the behavior is not good. E.g. a command fails because something is not able to get members and you actually don't care about the members. Not sure if it is was "fixed"(sizelimit=0). But with new member handling it might not be a big issue. However I like the idea of temporary caching inside find commands, where each memberof DN is resolved just once and results are cached in a map and reused in current context of command. This should be improvement mainly for indirect searches, but cache should be faster for direct members than doing internal calls of framework objects. This part is backward compatible, the first part is not. https://fedorahosted.org/freeipa/ticket/5282 What parts of the ticket can be solved with deref plugin? I guess we can get the CNs, but not what is a direct member. Maybe it should be discussed separately. Indirect members are already resolved by a single LDAP search. What kind of additional optimization would you like to do for them? We can use deref plugin to get pkeys from one search in case that pkeys are not part of DN. (I have to investigate if it is worth to do for user-find, I'm not sure if any memberof attributes have pkey that is not part of DN) sudo rule, hbac rule For indirect members, it is one search per entry, but for 1000 users, it is 1000 searches and I would like to have just one for the particular indirect member. are we talking about user-find? If so then it is mostly solved with default --no-member style behavior. But if a user or a group is directly/indirectly a member of a lot of groups(1000) then it might become slow. But caching won't probably help much, not sure. commands: user-find, stageuser-find, possibly all find commands 3) Remove userPassword, krbPrincipalKey from search results This change is not backward compatible, can we do this? https://fedorahosted.org/freeipa/ticket/5281 commands: user-find I'm for it, would like to hear other opinions. Note: it should be only in user-find commands. 'show' has to display it. +1 -- Petr Vobornik -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] user-* commands performance issues
On 21.03.2016 12:44, Jan Cholasta wrote: On 21.3.2016 11:00, Petr Vobornik wrote: On 03/17/2016 04:09 PM, Martin Basti wrote: Hello all, I would like to discuss the way how we should improve the speed of user-find commands (and other commands too if possible): 0) Do not do extra search for ipasshpubkey. This is clear, patch posted for review. https://fedorahosted.org/freeipa/ticket/3376 commands: user, stageuser, host, idview 1) make --no-members option visible in CLI https://fedorahosted.org/freeipa/ticket/4995 There was a discussion around devconf that --no-members should be a default behavior of xxx-find commands and I'm for it. +1, although we should be backward compatible with old clients which expect the attributes to be there. Ok, I agree to have --no-members as default for *-find commands, but it doesn't contradict to exposing --no-member option for all commands. Reasoning: use case: 'find me all groups which satisfy this filter'. Showing members clutters the output(one group with >500 member makes it unusable) and makes things slow(both on server and CLI side). For xxx-show commands it is a question where I don't have a strong opinion. I think it shouldn't hurt to keep them in -show commands, as there is always only a single entry to process. +1 I don't think we should implement also --no-indirect-members, I think that this kind of granularity is not needed. If --no-members is used, then indirect members will be ignored too. +1 +1 commands: all which use members 2) Limit the amount of searches for memberof[indirect] (group, netgroup, role, hbacrule, sudorule) and search for each dn only once in find commands. We can have configurable option in default.conf (for example memberof_search_limit=100 (0 unlimited)). Find commands will get members only for specified amount and if this limit is exceeded a warning message is shown. I do not like this idea much, I think it should be all or nothing, I prefer to not do this. +1 However I like the idea of temporary caching inside find commands, where each memberof DN is resolved just once and results are cached in a map and reused in current context of command. This should be improvement mainly for indirect searches, but cache should be faster for direct members than doing internal calls of framework objects. This part is backward compatible, the first part is not. https://fedorahosted.org/freeipa/ticket/5282 What parts of the ticket can be solved with deref plugin? I guess we can get the CNs, but not what is a direct member. Maybe it should be discussed separately. Indirect members are already resolved by a single LDAP search. What kind of additional optimization would you like to do for them? We can use deref plugin to get pkeys from one search in case that pkeys are not part of DN. (I have to investigate if it is worth to do for user-find, I'm not sure if any memberof attributes have pkey that is not part of DN) For indirect members, it is one search per entry, but for 1000 users, it is 1000 searches and I would like to have just one for the particular indirect member. commands: user-find, stageuser-find, possibly all find commands 3) Remove userPassword, krbPrincipalKey from search results This change is not backward compatible, can we do this? https://fedorahosted.org/freeipa/ticket/5281 commands: user-find I'm for it, would like to hear other opinions. Note: it should be only in user-find commands. 'show' has to display it. +1 -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] user-* commands performance issues
On 21.3.2016 11:00, Petr Vobornik wrote: On 03/17/2016 04:09 PM, Martin Basti wrote: Hello all, I would like to discuss the way how we should improve the speed of user-find commands (and other commands too if possible): 0) Do not do extra search for ipasshpubkey. This is clear, patch posted for review. https://fedorahosted.org/freeipa/ticket/3376 commands: user, stageuser, host, idview 1) make --no-members option visible in CLI https://fedorahosted.org/freeipa/ticket/4995 There was a discussion around devconf that --no-members should be a default behavior of xxx-find commands and I'm for it. +1, although we should be backward compatible with old clients which expect the attributes to be there. Reasoning: use case: 'find me all groups which satisfy this filter'. Showing members clutters the output(one group with >500 member makes it unusable) and makes things slow(both on server and CLI side). For xxx-show commands it is a question where I don't have a strong opinion. I think it shouldn't hurt to keep them in -show commands, as there is always only a single entry to process. I don't think we should implement also --no-indirect-members, I think that this kind of granularity is not needed. If --no-members is used, then indirect members will be ignored too. +1 +1 commands: all which use members 2) Limit the amount of searches for memberof[indirect] (group, netgroup, role, hbacrule, sudorule) and search for each dn only once in find commands. We can have configurable option in default.conf (for example memberof_search_limit=100 (0 unlimited)). Find commands will get members only for specified amount and if this limit is exceeded a warning message is shown. I do not like this idea much, I think it should be all or nothing, I prefer to not do this. +1 However I like the idea of temporary caching inside find commands, where each memberof DN is resolved just once and results are cached in a map and reused in current context of command. This should be improvement mainly for indirect searches, but cache should be faster for direct members than doing internal calls of framework objects. This part is backward compatible, the first part is not. https://fedorahosted.org/freeipa/ticket/5282 What parts of the ticket can be solved with deref plugin? I guess we can get the CNs, but not what is a direct member. Maybe it should be discussed separately. Indirect members are already resolved by a single LDAP search. What kind of additional optimization would you like to do for them? commands: user-find, stageuser-find, possibly all find commands 3) Remove userPassword, krbPrincipalKey from search results This change is not backward compatible, can we do this? https://fedorahosted.org/freeipa/ticket/5281 commands: user-find I'm for it, would like to hear other opinions. Note: it should be only in user-find commands. 'show' has to display it. +1 -- Jan Cholasta -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] user-* commands performance issues
On 03/17/2016 04:09 PM, Martin Basti wrote: Hello all, I would like to discuss the way how we should improve the speed of user-find commands (and other commands too if possible): 0) Do not do extra search for ipasshpubkey. This is clear, patch posted for review. https://fedorahosted.org/freeipa/ticket/3376 commands: user, stageuser, host, idview 1) make --no-members option visible in CLI https://fedorahosted.org/freeipa/ticket/4995 There was a discussion around devconf that --no-members should be a default behavior of xxx-find commands and I'm for it. Reasoning: use case: 'find me all groups which satisfy this filter'. Showing members clutters the output(one group with >500 member makes it unusable) and makes things slow(both on server and CLI side). For xxx-show commands it is a question where I don't have a strong opinion. I don't think we should implement also --no-indirect-members, I think that this kind of granularity is not needed. If --no-members is used, then indirect members will be ignored too. +1 commands: all which use members 2) Limit the amount of searches for memberof[indirect] (group, netgroup, role, hbacrule, sudorule) and search for each dn only once in find commands. We can have configurable option in default.conf (for example memberof_search_limit=100 (0 unlimited)). Find commands will get members only for specified amount and if this limit is exceeded a warning message is shown. I do not like this idea much, I think it should be all or nothing, I prefer to not do this. However I like the idea of temporary caching inside find commands, where each memberof DN is resolved just once and results are cached in a map and reused in current context of command. This should be improvement mainly for indirect searches, but cache should be faster for direct members than doing internal calls of framework objects. This part is backward compatible, the first part is not. https://fedorahosted.org/freeipa/ticket/5282 What parts of the ticket can be solved with deref plugin? I guess we can get the CNs, but not what is a direct member. Maybe it should be discussed separately. commands: user-find, stageuser-find, possibly all find commands 3) Remove userPassword, krbPrincipalKey from search results This change is not backward compatible, can we do this? https://fedorahosted.org/freeipa/ticket/5281 commands: user-find I'm for it, would like to hear other opinions. Note: it should be only in user-find commands. 'show' has to display it. Martin^2 -- Petr Vobornik -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
