Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Jakub Hrozek]

On Mon, Jul 18, 2016 at 01:36:30PM +, Sullivan, Daniel [AAA] wrote:
> > Are also users that are not part of this group misbehaving?
> 
> Not that I am aware of.  I’ll get you a real answer though.  Are there any 
> known workarounds to the @ problem used to transform group names (i.e. a more 
> robust ‘override_space’ option)?  I looked a the doc briefly but can’t find 
> anything. 

The override_space really just concerns spaces, not @-characters.

> 
> I was thinking maybe could use re_expression to tokenize group names by 
> taking the last token parsed by @ for the domain portion, although this seems 
> kind of hacky, also not sure if it would work.

Yes, I guess this should work.

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

> Are also users that are not part of this group misbehaving?

Not that I am aware of.  I’ll get you a real answer though.  Are there any 
known workarounds to the @ problem used to transform group names (i.e. a more 
robust ‘override_space’ option)?  I looked a the doc briefly but can’t find 
anything. 

I was thinking maybe could use re_expression to tokenize group names by taking 
the last token parsed by @ for the domain portion, although this seems kind of 
hacky, also not sure if it would work.

Dan



> On Jul 18, 2016, at 8:23 AM, Jakub Hrozek  wrote:
> 
> On Mon, Jul 18, 2016 at 11:56:24AM +, Sullivan, Daniel [AAA] wrote:
>> Hi, Jakub,
>> 
>> In line with your performance tuning document referenced prior in this
>> thread, I’ve actually already implemented the three configuration changes
>> you specified (prior to identifying this issue).  Right now I am focusing on
>> the use case documented below, because as of right now I am unable to get
>> that user populated into a client cache with sssd 1.14, at all.  In other
>> cases for individual users (prior to implementing tmpfs for example), it
>> seemed like an initial lookup on a client failed, then subsequent lookups
>> would succeed, presumably as a result of the DC eventually looking up and
>> caching the user.  This user (the one I can’t seem to lookup on a client)
>> is a member of a large number of groups, and also some of these groups
>> have longer names with spaces and special characters in them (i.e. $ and
>> . @)   I haven’t gone through and checked if one of these groups has a
>> large number of users, primarily because I am able to lookup users that
>> are members of groups with a large number of members (over 1000) already.
>> This is an actual group that this user is a member of, for example:
>> 
>> 788658174(members of this group will have full mailbox access and send as 
>> rights to 
>> urbj...@health.bsd.uchicago.edu 
>> mailbox)
>> 
>> Right now my theory is that the @ in this group name is causing the lookup 
>> to fail, as it is used as a character to specify the actual domain of a 
>> trusted group, although that has yet to be verified.
> 
> Yes, I would say this is the issue, because sssd tries to split the
> input string into name,domain components according to the re_expression
> value which tries to match anything before the first @ as a groupname
> and the rest as the username.
> 
> Are also users that are not part of this group misbehaving?
> 
>> 
>> NSS - https://gist.github.com/dsulli99/01715234efab09772e8236a13e4f4ef5
>> IPA - https://gist.github.com/dsulli99/f3cc92d7c32061fd4676a83a039c31b1
>> 
>> Here is the full list of groups the user is a member of, from the output of 
>> the id command on a DC:
>> 
>> uid=339741696(hah...@bsdad.uchicago.edu) 
>> gid=339741696(hah...@bsdad.uchicago.edu) 
>> groups=339741696(hah...@bsdad.uchicago.edu),788655857(hsd$
>>  kcbd 6260 conference room freebusy 
>> r...@bsdad.uchicago.edu),788668882(phs 
>> phsapps remoteapp default 
>> a...@bsdad.uchicago.edu),788670425(phs 
>> phsapps notepad2 
>> us...@bsdad.uchicago.edu),788670429(phs 
>> phsapps cmd 
>> us...@bsdad.uchicago.edu),339797692(cri-hpc_allus...@bsdad.uchicago.edu),788670440(phs
>>  phsapps r v3.2.0 32-bit 
>> us...@bsdad.uchicago.edu),788672389(phs 
>> phsapps remote desktop 
>> us...@bsdad.uchicago.edu),788655856(hsd$ 
>> w230 conference room freebusy 
>> r...@bsdad.uchicago.edu),788670441(phs 
>> phsapps r v3.2.0 64-bit 
>> us...@bsdad.uchicago.edu),788672413(phs 
>> phsapps r v3.2.3 64-bit 
>> us...@bsdad.uchicago.edu),788670431(phs 
>> phsapps file explorer 
>> us...@bsdad.uchicago.edu),788670428(phs 
>> phsapps adobe reader xi 
>> us...@bsdad.uchicago.edu),788609545(adm-trackitus...@bsdad.uchicago.edu),788615356(hsd$
>>  workstation local 
>> lo...@bsdad.uchicago.edu),339794097(cri-lmem_cri_us...@bsdad.uchicago.edu),788670445(phs
>>  phsapps taskmgr 
>> us...@bsdad.uchicago.edu),788624309(hsd$ 
>> pr...@bsdad.uchicago.edu),788670436(phs 
>> phsapps notepadplusplus 
>> us...@bsdad.uchicago.edu),788654299(cri-all_gro...@bsdad.uchicago.edu),788670434(phs
>>  phsapps notepad users@bsdad

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Jakub Hrozek]

On Mon, Jul 18, 2016 at 11:56:24AM +, Sullivan, Daniel [AAA] wrote:
> Hi, Jakub,
> 
> In line with your performance tuning document referenced prior in this
> thread, I’ve actually already implemented the three configuration changes
> you specified (prior to identifying this issue).  Right now I am focusing on
> the use case documented below, because as of right now I am unable to get
> that user populated into a client cache with sssd 1.14, at all.  In other
> cases for individual users (prior to implementing tmpfs for example), it
> seemed like an initial lookup on a client failed, then subsequent lookups
> would succeed, presumably as a result of the DC eventually looking up and
> caching the user.  This user (the one I can’t seem to lookup on a client)
> is a member of a large number of groups, and also some of these groups
> have longer names with spaces and special characters in them (i.e. $ and
> . @)   I haven’t gone through and checked if one of these groups has a
> large number of users, primarily because I am able to lookup users that
> are members of groups with a large number of members (over 1000) already.
> This is an actual group that this user is a member of, for example:
> 
> 788658174(members of this group will have full mailbox access and send as 
> rights to 
> urbj...@health.bsd.uchicago.edu 
> mailbox)
> 
> Right now my theory is that the @ in this group name is causing the lookup to 
> fail, as it is used as a character to specify the actual domain of a trusted 
> group, although that has yet to be verified.

Yes, I would say this is the issue, because sssd tries to split the
input string into name,domain components according to the re_expression
value which tries to match anything before the first @ as a groupname
and the rest as the username.

Are also users that are not part of this group misbehaving?

> 
> NSS - https://gist.github.com/dsulli99/01715234efab09772e8236a13e4f4ef5
> IPA - https://gist.github.com/dsulli99/f3cc92d7c32061fd4676a83a039c31b1
> 
> Here is the full list of groups the user is a member of, from the output of 
> the id command on a DC:
> 
> uid=339741696(hah...@bsdad.uchicago.edu) 
> gid=339741696(hah...@bsdad.uchicago.edu) 
> groups=339741696(hah...@bsdad.uchicago.edu),788655857(hsd$
>  kcbd 6260 conference room freebusy 
> r...@bsdad.uchicago.edu),788668882(phs 
> phsapps remoteapp default 
> a...@bsdad.uchicago.edu),788670425(phs 
> phsapps notepad2 
> us...@bsdad.uchicago.edu),788670429(phs 
> phsapps cmd 
> us...@bsdad.uchicago.edu),339797692(cri-hpc_allus...@bsdad.uchicago.edu),788670440(phs
>  phsapps r v3.2.0 32-bit 
> us...@bsdad.uchicago.edu),788672389(phs 
> phsapps remote desktop 
> us...@bsdad.uchicago.edu),788655856(hsd$ 
> w230 conference room freebusy 
> r...@bsdad.uchicago.edu),788670441(phs 
> phsapps r v3.2.0 64-bit 
> us...@bsdad.uchicago.edu),788672413(phs 
> phsapps r v3.2.3 64-bit 
> us...@bsdad.uchicago.edu),788670431(phs 
> phsapps file explorer 
> us...@bsdad.uchicago.edu),788670428(phs 
> phsapps adobe reader xi 
> us...@bsdad.uchicago.edu),788609545(adm-trackitus...@bsdad.uchicago.edu),788615356(hsd$
>  workstation local 
> lo...@bsdad.uchicago.edu),339794097(cri-lmem_cri_us...@bsdad.uchicago.edu),788670445(phs
>  phsapps taskmgr 
> us...@bsdad.uchicago.edu),788624309(hsd$ 
> pr...@bsdad.uchicago.edu),788670436(phs 
> phsapps notepadplusplus 
> us...@bsdad.uchicago.edu),788654299(cri-all_gro...@bsdad.uchicago.edu),788670434(phs
>  phsapps notepad 
> us...@bsdad.uchicago.edu),788670438(phs 
> phsapps plink 1.90 
> us...@bsdad.uchicago.edu),788670427(phs 
> phsapps office access 2013 
> us...@bsdad.uchicago.edu),788655855(hsd$ 
> w229 conference room freebusy 
> r...@bsdad.uchicago.edu),788635799(adm-sde-clie...@bsdad.uchicago.edu),788670439(phs
>  phsapps office powerpoint 2013 
> u...@bsdad.uchicago.edu),788610792(hsd$ all 
> health 
> stud...@bsdad.uchicago.edu),788655854(hsd$ 
> n102 conference 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Hi, Jakub,

In line with your performance tuning document referenced prior in this thread, 
I’ve actually already implemented the three configuration changes you specified 
(prior to identifying this issue).  Right now I am focusing on the use case 
documented below, because as of right now I am unable to get that user 
populated into a client cache with sssd 1.14, at all.  In other cases for 
individual users (prior to implementing tmpfs for example), it seemed like an 
initial lookup on a client failed, then subsequent lookups would succeed, 
presumably as a result of the DC eventually looking up and caching the user.  
This user (the one I can’t seem to lookup on a client) is a member of a large 
number of groups, and also some of these groups have longer names with spaces 
and special characters in them (i.e. $ and . @)   I haven’t gone through and 
checked if one of these groups has a large number of users, primarily because I 
am able to lookup users that are members of groups with a large number of 
members (over 1000) already.  This is an actual group that this user is a 
member of, for example:

788658174(members of this group will have full mailbox access and send as 
rights to 
urbj...@health.bsd.uchicago.edu mailbox)

Right now my theory is that the @ in this group name is causing the lookup to 
fail, as it is used as a character to specify the actual domain of a trusted 
group, although that has yet to be verified.

NSS - https://gist.github.com/dsulli99/01715234efab09772e8236a13e4f4ef5
IPA - https://gist.github.com/dsulli99/f3cc92d7c32061fd4676a83a039c31b1

Here is the full list of groups the user is a member of, from the output of the 
id command on a DC:

uid=339741696(hah...@bsdad.uchicago.edu) 
gid=339741696(hah...@bsdad.uchicago.edu) 
groups=339741696(hah...@bsdad.uchicago.edu),788655857(hsd$
 kcbd 6260 conference room freebusy 
r...@bsdad.uchicago.edu),788668882(phs phsapps 
remoteapp default 
a...@bsdad.uchicago.edu),788670425(phs phsapps 
notepad2 
us...@bsdad.uchicago.edu),788670429(phs 
phsapps cmd 
us...@bsdad.uchicago.edu),339797692(cri-hpc_allus...@bsdad.uchicago.edu),788670440(phs
 phsapps r v3.2.0 32-bit 
us...@bsdad.uchicago.edu),788672389(phs 
phsapps remote desktop 
us...@bsdad.uchicago.edu),788655856(hsd$ w230 
conference room freebusy 
r...@bsdad.uchicago.edu),788670441(phs phsapps 
r v3.2.0 64-bit 
us...@bsdad.uchicago.edu),788672413(phs 
phsapps r v3.2.3 64-bit 
us...@bsdad.uchicago.edu),788670431(phs 
phsapps file explorer 
us...@bsdad.uchicago.edu),788670428(phs 
phsapps adobe reader xi 
us...@bsdad.uchicago.edu),788609545(adm-trackitus...@bsdad.uchicago.edu),788615356(hsd$
 workstation local 
lo...@bsdad.uchicago.edu),339794097(cri-lmem_cri_us...@bsdad.uchicago.edu),788670445(phs
 phsapps taskmgr 
us...@bsdad.uchicago.edu),788624309(hsd$ 
pr...@bsdad.uchicago.edu),788670436(phs 
phsapps notepadplusplus 
us...@bsdad.uchicago.edu),788654299(cri-all_gro...@bsdad.uchicago.edu),788670434(phs
 phsapps notepad us...@bsdad.uc
hicago.edu),788670438(phs phsapps plink 1.90 
us...@bsdad.uchicago.edu),788670427(phs 
phsapps office access 2013 
us...@bsdad.uchicago.edu),788655855(hsd$ w229 
conference room freebusy 
r...@bsdad.uchicago.edu),788635799(adm-sde-clie...@bsdad.uchicago.edu),788670439(phs
 phsapps office powerpoint 2013 
u...@bsdad.uchicago.edu),788610792(hsd$ all 
health 
stud...@bsdad.uchicago.edu),788655854(hsd$ 
n102 conference room freebusy 
r...@bsdad.uchicago.edu),339793627(cri-galaxy_web_us...@bsdad.uchicago.edu),788670444(phs
 phsapps statamp 14 
us...@bsdad.uchicago.edu),339792922(cri-all_us...@bsdad.uchicago.edu),788670442(phs
 phsapps rstudio 
us...@bsdad.uchicago.edu),788655852(hsd$ 
freebusy read for all conference 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Jakub Hrozek]

On Fri, Jul 15, 2016 at 04:35:54PM +, Sullivan, Daniel [AAA] wrote:
> 
> Jakub,
> 
> Thank you for replying to me.  Before I forget I will say that I am still on 
> sssd 1.13 on the domain controller; I didn’t upgrade it because I haven’t had 
> any problems logging into that system yet.  That being said:
> 
> Thank you, but did this command return "No such user” ?
> 
> Yes.  Whenever this occurs "No such user" is the result from the id command 
> executed on the client.
> 
> If it did, was the user cached previously (iow, was there a successfull
> lookup before) ?
> 
> No, this is the first time the user has ever been looked up.  As far as I 
> know the user has never been successfully entered into the cache.  Similarly, 
> the user has never logged in to the IPA server via an SSSD client.

Ah, thank you, if the user has not been cached before, then it's
expected that the lookup has nothing to fall back to if the client fails
to look up information from the server.

> 
> Here is an example of a failed lookup from a client:
> 
> [root@cri-kcriwebgdp1 problem]# id hahsan
> id: hahsan: No such user
> 
> The DC logs for this operation are
> NSS - https://gist.github.com/dsulli99/01715234efab09772e8236a13e4f4ef5
> IPA - https://gist.github.com/dsulli99/f3cc92d7c32061fd4676a83a039c31b1

Thank you, I see that there is quite a lot of groups and the lookup
takes a bit of time. I wonder if any of the groups the user is a member
of are large?

If yes (and since moving the cache to tmpfs had helped), I wonder if
also using ignore_group_members would mitigate the issue further, like
this:

subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout
ignore_group_members = True
ldap_purge_cache_timeout = 0

These would go into the domain section on the server itself.

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Jakub,

Thank you for replying to me.  Before I forget I will say that I am still on 
sssd 1.13 on the domain controller; I didn’t upgrade it because I haven’t had 
any problems logging into that system yet.  That being said:

Thank you, but did this command return "No such user” ?

Yes.  Whenever this occurs "No such user" is the result from the id command 
executed on the client.

If it did, was the user cached previously (iow, was there a successfull
lookup before) ?

No, this is the first time the user has ever been looked up.  As far as I know 
the user has never been successfully entered into the cache.  Similarly, the 
user has never logged in to the IPA server via an SSSD client.

Here is an example of a failed lookup from a client:

[root@cri-kcriwebgdp1 problem]# id hahsan
id: hahsan: No such user

The DC logs for this operation are
NSS - https://gist.github.com/dsulli99/01715234efab09772e8236a13e4f4ef5
IPA - https://gist.github.com/dsulli99/f3cc92d7c32061fd4676a83a039c31b1

I can lookup this user fine on the DC:

[root@cri-ksysipadcp2 sssd]# id  hahsan
uid=339741696(hah...@bsdad.uchicago.edu) 
gid=339741696(hah...@bsdad.uchicago.edu) 
groups=339741696(hah...@bsdad.uchicago.edu),788655857
.

I appreciate your help with this.

Best,

Dan


On Jul 15, 2016, at 10:20 AM, Jakub Hrozek 
> wrote:

On Fri, Jul 15, 2016 at 01:22:07PM +, Sullivan, Daniel [AAA] wrote:
Jakub,

Sure, no problem, I am happy to provide the output that you are requesting.  
Thank you for taking the time to help me.

To answer your question, no record is returned (not missing groups). For 
example, the output of the failure was:

[root@cri-kcriwebgdp1 log]# id mjarsulic
id: mjarsulic: No such user

As per your request I have attached domain and nss logs for a lookup on the
user ‘spott’ (command invoked ‘id spott’ on the client). (immediately
after executing 'sss_cache -E; service sssd stop ; rm -rf /var/log/sssd/*;
service sssd start;’ on the client):

Thank you, but did this command return "No such user" ?

If it did, was the user cached previously (iow, was there a successfull
lookup before) ? The thing I'm confused about is that even if the back
end request failed (indicated by the "s2n exop request failed" message),
I would expect the NSS process to still return data from the cache.

As per why the request failed, you need to look into the matching logs
on the server side around that time the s2n request failed to see if
there was some issue with lookups.

The double-qualification is just an annoying debug message.
In 1.14, we store all usernames fully qualified (that's the first one
you see) but also append the domain name in some functions when printing
debug messages (that's the second one).


IPA - https://gist.github.com/dsulli99/4e45faa39474b9131be811e4a0779c40
NSS - https://gist.github.com/dsulli99/e2e10da34ff860ec15e56ea521eb8315

Not every record fails, and the behavior is inconsistent between lookups (i.e. 
sometimes a user will lookup correctly, sometimes it will not), but it appears 
that in some situations a timeout is occurring in the nss logs (not in the 
failure above).   In these situations it looks to me like the query is 
dispatched to the DC, and the lookup times out.  If I wait a little bit and 
perform the lookup on the same user again,  the record is returned (presumably 
because the DC eventually resolved and cached the query?).  We are migrating 
from CentrifyDC and have loaded 2000+ custom ID overrides into our Default 
Trust ID View; perhaps we will need to implement the tempfs caching for the 
/var/lib/sss/db on the DC as described in your performance tuning document 
(https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/).
  These timeouts look like:

(Fri Jul 15 07:21:04 2016) [sssd[nss]] [get_dp_name_and_id] (0x0400): Not a 
LOCAL view, continuing with provided values.
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing 
request for 
[0x41e750:1:b...@bsdad.uchicago.edu@bsdad.uchicago.edu]
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): 
Creating request for 
[bsdad.uchicago.edu][0x1][BE_REQ_USER][1][name=b...@bsdad.uchicago.edu:-]
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sbus_add_timeout] (0x2000): 0x1fa9020
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): 
Entering request 
[0x41e750:1:b...@bsdad.uchicago.edu@bsdad.uchicago.edu]
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sbus_remove_timeout] (0x2000): 0x1fa9020
(Fri 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Jakub Hrozek]

On Fri, Jul 15, 2016 at 01:22:07PM +, Sullivan, Daniel [AAA] wrote:
> Jakub,
> 
> Sure, no problem, I am happy to provide the output that you are requesting.  
> Thank you for taking the time to help me.
> 
> To answer your question, no record is returned (not missing groups). For 
> example, the output of the failure was:
> 
> [root@cri-kcriwebgdp1 log]# id mjarsulic
> id: mjarsulic: No such user
> 
> As per your request I have attached domain and nss logs for a lookup on the
> user ‘spott’ (command invoked ‘id spott’ on the client). (immediately
> after executing 'sss_cache -E; service sssd stop ; rm -rf /var/log/sssd/*;
> service sssd start;’ on the client):

Thank you, but did this command return "No such user" ?

If it did, was the user cached previously (iow, was there a successfull
lookup before) ? The thing I'm confused about is that even if the back
end request failed (indicated by the "s2n exop request failed" message),
I would expect the NSS process to still return data from the cache.

As per why the request failed, you need to look into the matching logs
on the server side around that time the s2n request failed to see if
there was some issue with lookups.

The double-qualification is just an annoying debug message.
In 1.14, we store all usernames fully qualified (that's the first one
you see) but also append the domain name in some functions when printing
debug messages (that's the second one).

> 
> IPA - https://gist.github.com/dsulli99/4e45faa39474b9131be811e4a0779c40
> NSS - https://gist.github.com/dsulli99/e2e10da34ff860ec15e56ea521eb8315
> 
> Not every record fails, and the behavior is inconsistent between lookups 
> (i.e. sometimes a user will lookup correctly, sometimes it will not), but it 
> appears that in some situations a timeout is occurring in the nss logs (not 
> in the failure above).   In these situations it looks to me like the query is 
> dispatched to the DC, and the lookup times out.  If I wait a little bit and 
> perform the lookup on the same user again,  the record is returned 
> (presumably because the DC eventually resolved and cached the query?).  We 
> are migrating from CentrifyDC and have loaded 2000+ custom ID overrides into 
> our Default Trust ID View; perhaps we will need to implement the tempfs 
> caching for the /var/lib/sss/db on the DC as described in your performance 
> tuning document 
> (https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/).
>   These timeouts look like:
> 
> (Fri Jul 15 07:21:04 2016) [sssd[nss]] [get_dp_name_and_id] (0x0400): Not a 
> LOCAL view, continuing with provided values.
> (Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_issue_request] (0x0400): 
> Issuing request for 
> [0x41e750:1:b...@bsdad.uchicago.edu@bsdad.uchicago.edu]
> (Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): 
> Creating request for 
> [bsdad.uchicago.edu][0x1][BE_REQ_USER][1][name=b...@bsdad.uchicago.edu:-]
> (Fri Jul 15 07:21:04 2016) [sssd[nss]] [sbus_add_timeout] (0x2000): 0x1fa9020
> (Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): 
> Entering request 
> [0x41e750:1:b...@bsdad.uchicago.edu@bsdad.uchicago.edu]
> (Fri Jul 15 07:21:17 2016) [sssd[nss]] [sbus_remove_timeout] (0x2000): 
> 0x1fa9020
> (Fri Jul 15 07:21:17 2016) [sssd[nss]] [sbus_dispatch] (0x4000): dbus conn: 
> 0x1fa0730
> (Fri Jul 15 07:21:17 2016) [sssd[nss]] [sbus_dispatch] (0x4000): Dispatching.
> (Fri Jul 15 07:21:17 2016) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply 
> from Data Provider - DP error code: 3 errno: 110 error message: Connection 
> timed out
> (Fri Jul 15 07:21:17 2016) [sssd[nss]] [nss_cmd_getby_dp_callback] (0x0040): 
> Unable to get information from Data Provider
> Error: 3, 110, Connection timed out
> Will try to return what we have in cache
> (Fri Jul 15 07:21:17 2016) [sssd[nss]] [sss_dp_req_destructor] (0x0400): 
> Deleting request: 
> [0x41e750:1:b...@bsdad.uchicago.edu@bsdad.uchicago.edu]
> (Fri Jul 15 07:21:17 2016) [sssd[nss]] [reset_idle_timer] (0x4000): Idle 
> timer re-set for client [0x1fa7fc0][22]
> (Fri Jul 15 07:21:17 2016) [sssd[nss]] [reset_idle_timer] (0x4000): Idle 
> timer re-set for client [0x1fa7fc0][22]
> (Fri Jul 15 07:21:17 2016) [sssd[nss]] [client_recv] (0x0200): Client 
> disconnected!
> (Fri Jul 15 07:21:17 2016) [sssd[nss]] [client_close_fn] (0x2000): Terminated 
> client [0x1fa7fc0][22]
> 
> I’m going to implement tmpfs caching on the DC, hopefully this will address 
> at least a subset of these lookup failures.  I’ll report back with my 
> findings.
> 
> Thank you again for your help.
> 
> Best,
> 
> Dan Sullivan
> 
> 
> 
> 
> On Jul 15, 2016, at 7:12 AM, Jakub Hrozek 
> > wrote:
> 
> On Fri, Jul 15, 2016 at 12:00:56PM +, 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Jakub Hrozek]

On Fri, Jul 15, 2016 at 02:04:43PM +, Sullivan, Daniel [AAA] wrote:
> Hi,
> 
> Changing pam_id_timeout = 60 and krb5_auth_timeout = 60 on the client in 
> conjunction with enabling tmpfs caching for /var/lib/sss/db on the DC appears 
> to have helped significantly.  

pam_id_timeout and krb5_auth_timeout are only applied during login, not
when id is invoked. So I think the piece that helped in your environment
was the tmpfs on the server.

Still, I think there are two issues:
1) why does the s2n operation fail at all? We should look into the
server logs around the time the s2n operation fails to find the
reason

2) why doesn't sssd on the client return cached data if the s2n
request fails? See my other mail, I'm interested if the data was
cached from a previous lookup.

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Hi,

Changing pam_id_timeout = 60 and krb5_auth_timeout = 60 on the client in 
conjunction with enabling tmpfs caching for /var/lib/sss/db on the DC appears 
to have helped significantly.  This issue is becoming much more difficult to 
reproduce, although I can still reproduce it.  Now, it appears that rapid 
successive invocations of the id command will yield a returned record. The 
timeout for the output specified below (i.e. the time it took the first command 
to return) was definitely less than 60 seconds, probably 10-20.  I am going to 
look into the tuning options for sssd, and would of course be interested in any 
advisement you could provide this regard.  AFAIK this issue currently only 
impacts users with a large number of groups (in fact I have only been able to 
cause this issue one one user after tuning as described above).  I am going to 
script a test and do a lookup for every single ID Override user in our 
environment to see what kind of a hit rate I get.  I’ll report back.  Thank you 
again for your help.

[root@cri-kcriwebgdp1 log]# id rcrist

id: rcrist: No such user
[root@cri-kcriwebgdp1 log]# id rcrist
uid=339748142(rcrist) gid=339748142(rcrist) 
groups=339748142(rcrist),339801232(cri-aaa_static_hosting),788635799(adm-sde-clients),788600520(group
 policy creator owners),788602710(bsd exchange view only 
administrators),339792922(cri-all_users),788659064(aaa-static_hosting_groups),788601114(bsd$
 dns 
read),788609545(adm-trackitusers),339806103(cri-ciscat),788609528(adm-bsd-mis),788619855(adm-oua-dl),788615498(adm-himss),788637726(adm-dstmlist-dl),788600513(domain
 users),788601110(bsd$ all 
oua),788654299(cri-all_groups),788658170(ocr-sharepoint ocr 
members),788619946(adm-trackitreports),788638566(ocr-coi),788633650(#ocr-office-dl),788644425(ocr
 velos 
email),788609542(adm-testgroup1),788638733(ocr-dfc-users),788665477(med-section_shares-clinical
 trials (only)),788609532(adm-bsdis-print),788634332(ocr-clinical 
research),788609546(adm-tss),788658806(ocr-hiro),788672525(ocr-bsdvpn-allow),788640103(adm
 shpt srp 
contributors),788659092(ocr-sharepoint-velosupgrade),788639053(ocr-velos-tickets),788610719(adm-premigration-proofpoint),788635798(adm-sde-techs),788635657(adm-www-clinres),788653680(ocr-email-management),788663575(ocr-bsdirb),788658171(ocr-sharepoint
 irb members),788650124(ocr it),788609567(ors-teleform),788653595(ocr$ 
oua),788609341(ic),788646237(adm shpt ocr 
visitors),788609544(adm-trackittech),788671562(ocr-ocrepic),788652940(dma 
management)

Dan


On Jul 15, 2016, at 8:22 AM, Sullivan, Daniel [AAA] 
> wrote:

Jakub,

Sure, no problem, I am happy to provide the output that you are requesting.  
Thank you for taking the time to help me.

To answer your question, no record is returned (not missing groups). For 
example, the output of the failure was:

[root@cri-kcriwebgdp1 log]# id mjarsulic
id: mjarsulic: No such user

As per your request I have attached domain and nss logs for a lookup on the 
user ‘spott’ (command invoked ‘id spott’ on the client). (immediately after 
executing 'sss_cache -E; service sssd stop ; rm -rf /var/log/sssd/*; service 
sssd start;’ on the client):

IPA - https://gist.github.com/dsulli99/4e45faa39474b9131be811e4a0779c40
NSS - https://gist.github.com/dsulli99/e2e10da34ff860ec15e56ea521eb8315

Not every record fails, and the behavior is inconsistent between lookups (i.e. 
sometimes a user will lookup correctly, sometimes it will not), but it appears 
that in some situations a timeout is occurring in the nss logs (not in the 
failure above).   In these situations it looks to me like the query is 
dispatched to the DC, and the lookup times out.  If I wait a little bit and 
perform the lookup on the same user again,  the record is returned (presumably 
because the DC eventually resolved and cached the query?).  We are migrating 
from CentrifyDC and have loaded 2000+ custom ID overrides into our Default 
Trust ID View; perhaps we will need to implement the tempfs caching for the 
/var/lib/sss/db on the DC as described in your performance tuning document 
(https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/).
  These timeouts look like:

(Fri Jul 15 07:21:04 2016) [sssd[nss]] [get_dp_name_and_id] (0x0400): Not a 
LOCAL view, continuing with provided values.
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing 
request for 
[0x41e750:1:b...@bsdad.uchicago.edu@bsdad.uchicago.edu]
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): 
Creating request for 
[bsdad.uchicago.edu][0x1][BE_REQ_USER][1][name=b...@bsdad.uchicago.edu:-]
(Fri Jul 15 07:21:04 2016) [sssd[nss]] 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Jakub,

Sure, no problem, I am happy to provide the output that you are requesting.  
Thank you for taking the time to help me.

To answer your question, no record is returned (not missing groups). For 
example, the output of the failure was:

[root@cri-kcriwebgdp1 log]# id mjarsulic
id: mjarsulic: No such user

As per your request I have attached domain and nss logs for a lookup on the 
user ‘spott’ (command invoked ‘id spott’ on the client). (immediately after 
executing 'sss_cache -E; service sssd stop ; rm -rf /var/log/sssd/*; service 
sssd start;’ on the client):

IPA - https://gist.github.com/dsulli99/4e45faa39474b9131be811e4a0779c40
NSS - https://gist.github.com/dsulli99/e2e10da34ff860ec15e56ea521eb8315

Not every record fails, and the behavior is inconsistent between lookups (i.e. 
sometimes a user will lookup correctly, sometimes it will not), but it appears 
that in some situations a timeout is occurring in the nss logs (not in the 
failure above).   In these situations it looks to me like the query is 
dispatched to the DC, and the lookup times out.  If I wait a little bit and 
perform the lookup on the same user again,  the record is returned (presumably 
because the DC eventually resolved and cached the query?).  We are migrating 
from CentrifyDC and have loaded 2000+ custom ID overrides into our Default 
Trust ID View; perhaps we will need to implement the tempfs caching for the 
/var/lib/sss/db on the DC as described in your performance tuning document 
(https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/).
  These timeouts look like:

(Fri Jul 15 07:21:04 2016) [sssd[nss]] [get_dp_name_and_id] (0x0400): Not a 
LOCAL view, continuing with provided values.
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing 
request for 
[0x41e750:1:b...@bsdad.uchicago.edu@bsdad.uchicago.edu]
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): 
Creating request for 
[bsdad.uchicago.edu][0x1][BE_REQ_USER][1][name=b...@bsdad.uchicago.edu:-]
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sbus_add_timeout] (0x2000): 0x1fa9020
(Fri Jul 15 07:21:04 2016) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): 
Entering request 
[0x41e750:1:b...@bsdad.uchicago.edu@bsdad.uchicago.edu]
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sbus_remove_timeout] (0x2000): 0x1fa9020
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sbus_dispatch] (0x4000): dbus conn: 
0x1fa0730
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sbus_dispatch] (0x4000): Dispatching.
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply 
from Data Provider - DP error code: 3 errno: 110 error message: Connection 
timed out
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [nss_cmd_getby_dp_callback] (0x0040): 
Unable to get information from Data Provider
Error: 3, 110, Connection timed out
Will try to return what we have in cache
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [sss_dp_req_destructor] (0x0400): 
Deleting request: 
[0x41e750:1:b...@bsdad.uchicago.edu@bsdad.uchicago.edu]
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer 
re-set for client [0x1fa7fc0][22]
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer 
re-set for client [0x1fa7fc0][22]
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [client_recv] (0x0200): Client 
disconnected!
(Fri Jul 15 07:21:17 2016) [sssd[nss]] [client_close_fn] (0x2000): Terminated 
client [0x1fa7fc0][22]

I’m going to implement tmpfs caching on the DC, hopefully this will address at 
least a subset of these lookup failures.  I’ll report back with my findings.

Thank you again for your help.

Best,

Dan Sullivan




On Jul 15, 2016, at 7:12 AM, Jakub Hrozek 
> wrote:

On Fri, Jul 15, 2016 at 12:00:56PM +, Sullivan, Daniel [AAA] wrote:
Lukas,

Thank you for your reply and inquiry.

First, to answer your question; yes, we have been using the 
default_domain_suffix for some time.  I am not sure what you mean by 
previously, but it is currently implemented and has been implemented prior to 
our 1.13 -> 1.14 upgrade.

And yes, I am assessing a possible software regression at the
current moment. It might be related to the default_domain_suffix
you are inquiring about.  Basically I am getting inconsistent
results on invocation of the id command with specifying the username
as ‘username’ or ‘username@fqdn’ on a client running 1.14
against a DC running 1.13 (i.e. no way to reliably invoke id against a
trusted domain account).  Sometimes the command will return a result,
and sometimes it will not.

No result or missing groups?

Looking at nss debug logs it appears that
a duplicate fqdn is being appended to the nss query as show here (as

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Jakub Hrozek]

On Fri, Jul 15, 2016 at 12:00:56PM +, Sullivan, Daniel [AAA] wrote:
> Lukas,
> 
> Thank you for your reply and inquiry.
> 
> First, to answer your question; yes, we have been using the 
> default_domain_suffix for some time.  I am not sure what you mean by 
> previously, but it is currently implemented and has been implemented prior to 
> our 1.13 -> 1.14 upgrade.
> 
> And yes, I am assessing a possible software regression at the
> current moment. It might be related to the default_domain_suffix
> you are inquiring about.  Basically I am getting inconsistent
> results on invocation of the id command with specifying the username
> as ‘username’ or ‘username@fqdn’ on a client running 1.14
> against a DC running 1.13 (i.e. no way to reliably invoke id against a
> trusted domain account).  Sometimes the command will return a result,
> and sometimes it will not.  

No result or missing groups?

> Looking at nss debug logs it appears that
> a duplicate fqdn is being appended to the nss query as show here (as
> @bsdad.uchicago@bsdad.uchicago.edu).
> This lookup fails.

Yes, this is wrong, can you send me the full NSS and domain logs please?

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Lukas,

Also, I would be interested to have high-level knowledge of known regressions 
you describe so that we can more quickly identify that we are being impacted by 
a known issue as we move forward with testing and evaluation of our IPA 
implementation, particularly if they are missing from the 1.14 section of (6 
tickets open):

https://fedorahosted.org/sssd/report/2

Best,

Dan


This e-mail is intended only for the use of the individual or entity to which
it is addressed and may contain information that is privileged and confidential.
If the reader of this e-mail message is not the intended recipient, you are 
hereby notified that any dissemination, distribution or copying of this
communication is prohibited. If you have received this e-mail in error, please 
notify the sender and destroy all copies of the transmittal. 

Thank you
University of Chicago Medicine and Biological Sciences 


-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Lukas,

Thank you for your reply and inquiry.

First, to answer your question; yes, we have been using the 
default_domain_suffix for some time.  I am not sure what you mean by 
previously, but it is currently implemented and has been implemented prior to 
our 1.13 -> 1.14 upgrade.

And yes, I am assessing a possible software regression at the current moment. 
It might be related to the default_domain_suffix you are inquiring about.  
Basically I am getting inconsistent results on invocation of the id command 
with specifying the username as ‘username’ or ‘username@fqdn’ on a client 
running 1.14 against a DC running 1.13 (i.e. no way to reliably invoke id 
against a trusted domain account).  Sometimes the command will return a result, 
and sometimes it will not.  Looking at nss debug logs it appears that a 
duplicate fqdn is being appended to the nss query as show here (as 
@bsdad.uchicago@bsdad.uchicago.edu).
  This lookup fails.

(Fri Jul 15 06:53:07 2016) [sssd[nss]] [get_dp_name_and_id] (0x0400): Not a 
LOCAL view, continuing with provided values.
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing 
request for 
[0x41e750:1:mjarsu...@bsdad.uchicago.edu@bsdad.uchicago.edu]
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): 
Creating request for 
[bsdad.uchicago.edu][0x1][BE_REQ_USER][1][name=mjarsu...@bsdad.uchicago.edu:-]
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [sbus_add_timeout] (0x2000): 0x7d0860
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): 
Entering request 
[0x41e750:1:mjarsu...@bsdad.uchicago.edu@bsdad.uchicago.edu]
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [sbus_remove_timeout] (0x2000): 0x7d0860
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [sbus_dispatch] (0x4000): dbus conn: 
0x7a1730
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [sbus_dispatch] (0x4000): Dispatching.
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply 
from Data Provider - DP error code: 0 errno: 0 error message: Success
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [sss_ncache_check_str] (0x2000): 
Checking negative cache for 
[NCE/USER/bsdad.uchicago.edu/mjarsu...@bsdad.uchicago.edu]
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [nss_cmd_getpwnam_search] (0x0100): 
Requesting info for 
[mjarsu...@bsdad.uchicago.edu]
(Fri Jul 15 06:53:07 2016) [sssd[nss]] [ldb] (0x4000): Added timed event 
"ltdb_callback": 0x7bb560

Right now I am considering snapshotting our DC, upgrading the sssd to 1.14 on 
it, flushing the cache on DC and client (both 1.14), and re-testing.

If you have any insight on resolving this issue I’d be interested in hearing 
your thoughts.

Best,

Dan

On Jul 15, 2016, at 6:13 AM, Lukas Slebodnik 
> wrote:

On (14/07/16 21:23), Sullivan, Daniel [AAA] wrote:
Justin,

Thank you for taking the time to reply to me; I really appreciate your 
willingness to help.

Upgrading to sssd1.14 (from the copr repo) on the client seems to have fixed 
this problem across the board.  I don’t have a system that is currently broken 
to capture this data, but if it is important for you to have the log data to 
try and resolve this bug I could try to obtain it for you by purposely try to 
induce the issue by upgrading another system and hoping the bug presents 
itself, and then capture the data.  Please advise if you would like me to 
attempt this.

I was really frustrated by this bug and am happy that I can consider this issue 
resolved.  Please let me know if you would like me to try and capture the data 
as you described.

I am glad that 1.14 works for you :-)
But there might be other bugs. I know about few regressions.

BTW about the HBAC issue, did you use the default_domain_suffix previously?

LS



This e-mail is intended only for the use of the individual or entity to which
it is addressed and may contain information that is privileged and confidential.
If the reader of this e-mail message is not the intended recipient, you are 
hereby notified that any dissemination, distribution or copying of this
communication is prohibited. If you have received this e-mail in error, please 
notify the sender and destroy all copies of the transmittal. 

Thank you
University of Chicago Medicine and Biological Sciences 


-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Lukas Slebodnik]

On (14/07/16 21:23), Sullivan, Daniel [AAA] wrote:
>Justin,
>
>Thank you for taking the time to reply to me; I really appreciate your 
>willingness to help.
>
>Upgrading to sssd1.14 (from the copr repo) on the client seems to have fixed 
>this problem across the board.  I don’t have a system that is currently broken 
>to capture this data, but if it is important for you to have the log data to 
>try and resolve this bug I could try to obtain it for you by purposely try to 
>induce the issue by upgrading another system and hoping the bug presents 
>itself, and then capture the data.  Please advise if you would like me to 
>attempt this.
>
>I was really frustrated by this bug and am happy that I can consider this 
>issue resolved.  Please let me know if you would like me to try and capture 
>the data as you described.
>
I am glad that 1.14 works for you :-)
But there might be other bugs. I know about few regressions.

BTW about the HBAC issue, did you use the default_domain_suffix previously?

LS

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Hi,

I wanted to follow up on this thread in case others are experiencing this 
problem.  Installing SSSD 1.14 from the copr repository seems to have 
completely eliminated the HBAC issue on all systems that were exhibiting the 
problem as previously described.

https://copr.fedorainfracloud.org/coprs/g/sssd/sssd-1-14/

Also, FWIW thank you for fixing this (unrelated):

https://fedorahosted.org/sssd/ticket/2838

Thank you to everybody who helped with this.

Dan




This e-mail is intended only for the use of the individual or entity to which
it is addressed and may contain information that is privileged and confidential.
If the reader of this e-mail message is not the intended recipient, you are 
hereby notified that any dissemination, distribution or copying of this
communication is prohibited. If you have received this e-mail in error, please 
notify the sender and destroy all copies of the transmittal. 

Thank you
University of Chicago Medicine and Biological Sciences 


-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Justin Stephenson]

Hello Daniel,

Just to clarify the issue:

user 'a.cri.dsulli...@bsdad.uchicago.edu' is a member of IDM POSIX group 
'cri-cri_server_administrators_ipa' which is linked to the external 
group used for the AD trust.


The following HBAC rule is not working to allow SSH access

[root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show 
'cri-cri_server_administrators_allow_all'

  Rule name: cri-cri_server_administrators_allow_all
  Host category: all
  Service category: all
  Description: Allow anyone in 
cri-cri_server_administrat...@bsdad.uchicago.edu 
to login to any machine

  Enabled: TRUE
  User Groups: cri-cri_server_administrators_ipa

==

During the HBAC processing, sssd checks which groups are associated with 
the HBAC rule and adds those groups(just the 
'cri-cri_server_administrators_ipa' group in this case) to the 
memberUser attribute of the HBAC rule sysdb entry.


(Wed Jul 13 12:07:13 2016) [sssd[be[ipa.cri.uchicago.edu]]] 
[hbac_attrs_to_rule] (0x1000): Processing rule 
[cri-cri_server_administrators_allow_all]
(Wed Jul 13 12:07:13 2016) [sssd[be[ipa.cri.uchicago.edu]]] 
[sysdb_search_groups] (0x2000): Search groups with filter: 
(&(objectclass=group)(originalDN=cn=cri-cri_server_administrators_ipa,cn=groups,cn=accounts,dc=ipa,dc=cri,dc=uchicago,dc=edu))
(Wed Jul 13 12:07:13 2016) [sssd[be[ipa.cri.uchicago.edu]]] 
[hbac_user_attrs_to_rule] (0x2000): Added POSIX group 
[cri-cri_server_administrators_ipa] to rule 
[cri-cri_server_administrators_allow_all]


The hbac evaluation reads the originalMemberof attribute of 
'a.cri.dsulli...@bsdad.uchicago.edu' to assess the groups which HBAC 
should match with


(Wed Jul 13 12:07:13 2016) [sssd[be[ipa.cri.uchicago.edu]]] 
[hbac_eval_user_element] (0x1000): [30] groups for 
[a.cri.dsulli...@bsdad.uchicago.edu]


With debug logging enabled we should see a message such as the following 
when the group is found, but we don't see that in your log output:


[hbac_eval_user_element]  Added group 
[cri-cri_server_administrators_ipa] for user 
[a.cri.dsulli...@bsdad.uchicago.edu]


We need to understand why this group is being removed from the sysdb, as 
you said the group does not return in id command output after the SSH 
attempt.


It would be great to get full sssd debug logs and a dump of the sssd 
sysdb cache after the first 'id' command is run, and then after the SSH 
attempt is made when the group no longer shows. Note the ldbsearch 
command is included in the 'ldb-tools' rpm


For example:

  ldbsearch -H /var/lib/sss/db/cache_.ldb > 
ldbsearch-first-id-command.ldb


  ldbsearch -H /var/lib/sss/db/cache_.ldb > 
ldbsearch-after-ssh-attempt.ldb


Kind regards,
Justin Stephenson


On 07/13/2016 03:14 PM, Sullivan, Daniel [AAA] wrote:

Jakub, Justin,

Thank you both very much for taking the time to continue helping me resolve 
this issue.  I apologize for not replying right away; I’ve been dealing with a 
production issue for most of the morning.

An invocation of ‘id 
a.cri.dsulli...@bsdad.uchicago.edu’ on the IPA DC 
shows me as a member of the POSIX IPA group 
(cri_server_administrators_...@ipa.cri.uchicago.edu)
 as well as the AD domain group in the trusted domain 
(cri-aaa_server_administrat...@bsdad.uchicago.edu).
  This remains consistent across any number of successful sshd logins into the DC using my 
a.cri.dsullivan@bsdad.uchicago.edu account, including 
after clearing the cache on the DC.

On the client, I am seeing some unusual behavior.  If I run the commands 'sss_cache -E; 
service sssd stop ; rm -rf /var/log/sssd/* ; service sssd start’ , then run ‘id 
a.cri.dsulli...@bsdad.uchicago.edu’, I see 
the POSIX IPA group as well as the AD domain group as described above (what I presumably 
want and expect).  However (and this is the unusual part), if I attempt to login via SSH 
(it will fail with HBAC validation), and then run the ‘id 
a.cri.dsulli...@bsdad.uchicago.edu’ 
command again , the POSIX IPA group disappears from the list of groups output by the id 
command.  From what I can tell, this group will not reappear in the group list on the 
client until the client cache is cleared.  Presumably this behavior is related to the HBAC 
authentication errors I am experiencing.  I have cleared the cache on both the DC and the 
client using ssh_cache -E and this behavior is still exhibited.  With respect to output 
from testing:

1) The sssd domain log from from the client of the initial id invocation (both groups 
appear) after clearing the cache (on the client) can be found here (this output 
contains both 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Hi,

I have a brief follow up question regarding this issue; 

I’m actually not bent on using HBAC; it is a nice feature and I’d like to use 
it, but at the end of the day I’m not married to the idea of managing this type 
of policy centrally; in theory, group or user based access control using 
AllowGroups/AllowUsers in sshd_config should work, as long as 
GSSAPIAuthentication and UsePAM are enabled, right? I’ve seen a couple of 
threads that suggest this is possible, although I haven’t seen it explicitly 
mentioned anywhere in the documentation.

I’ve made a brief failed attempt at getting sshd authentication working using 
AllowGroups in sshd_config, however I haven’t spent a whole lot of time on it 
yet (I’m running into some issues with PAM, presumably to pre-existing problems 
with group enumeration).

I’m growing concerned about our upcoming IPA implementation because as of now I 
don’t have a known workaround to the issue described in this thread (it is 
impacting more than one client).  Any advice with respect to a viable 
workaround to this issue would be appreciated.

Thank you so much for your ongoing support.

Best,

Dan

> On Jul 13, 2016, at 2:14 PM, Sullivan, Daniel [AAA] 
>  wrote:
> 
> Jakub, Justin,
> 
> Thank you both very much for taking the time to continue helping me resolve 
> this issue.  I apologize for not replying right away; I’ve been dealing with 
> a production issue for most of the morning.
> 
> An invocation of ‘id 
> a.cri.dsulli...@bsdad.uchicago.edu’
>  on the IPA DC shows me as a member of the POSIX IPA group 
> (cri_server_administrators_...@ipa.cri.uchicago.edu)
>  as well as the AD domain group in the trusted domain 
> (cri-aaa_server_administrat...@bsdad.uchicago.edu).
>   This remains consistent across any number of successful sshd logins into 
> the DC using my 
> a.cri.dsullivan@bsdad.uchicago.edu 
> account, including after clearing the cache on the DC.
> 
> On the client, I am seeing some unusual behavior.  If I run the commands 
> 'sss_cache -E; service sssd stop ; rm -rf /var/log/sssd/* ; service sssd 
> start’ , then run ‘id 
> a.cri.dsulli...@bsdad.uchicago.edu’,
>  I see the POSIX IPA group as well as the AD domain group as described above 
> (what I presumably want and expect).  However (and this is the unusual part), 
> if I attempt to login via SSH (it will fail with HBAC validation), and then 
> run the ‘id 
> a.cri.dsulli...@bsdad.uchicago.edu’
>  command again , the POSIX IPA group disappears from the list of groups 
> output by the id command.  From what I can tell, this group will not reappear 
> in the group list on the client until the client cache is cleared.  
> Presumably this behavior is related to the HBAC authentication errors I am 
> experiencing.  I have cleared the cache on both the DC and the client using 
> ssh_cache -E and this behavior is still exhibited.  With respect to output 
> from testing:
> 
> 1) The sssd domain log from from the client of the initial id invocation 
> (both groups appear) after clearing the cache (on the client) can be found 
> here (this output contains both groups): 
> https://gist.github.com/dsulli99/7117f8d567cc7cdf727d474b0aeab8da
> 2) The sssd domain log from the client for the failed sshd login (similar to 
> the output I provided yesterday, however re-captured) can be found here 
> (after this operation the IPA group disappears from the list of groups from 
> the id command): 
> https://gist.github.com/dsulli99/668a8799709ff0cd311b321206591124
> 3) The DC log (after the client cache is cleared) of my running the id 
> invocation (from the client) can be found here (this is the DC side of 1) 
> from above. https://gist.github.com/dsulli99/a2a5e80b6a8b143afa20024aa40a7b39
> 4) The DC log of the the failed sshd login into the client (this is the DC 
> side of 2) from above is 
> https://gist.github.com/dsulli99/4e3ba53c942ad78d7487ae51da92007e
> 
> I really appreciate your help with looking at this issue.  As I said I have 
> another machine built from the same image that this is working fine on.  I am 
> going to keep plugging away at this, I will let you know if I come up with 
> anything.
> 
> Dan
> 
> 
> 
> 
> This e-mail is intended only for the use of the individual or entity to which
> it is addressed and may contain information that is privileged and 
> confidential.
> If the reader of this e-mail message is not the intended recipient, you are 
> hereby notified that any dissemination, distribution or copying of this
> communication is prohibited. If 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Jakub, Justin,

Thank you both very much for taking the time to continue helping me resolve 
this issue.  I apologize for not replying right away; I’ve been dealing with a 
production issue for most of the morning.

An invocation of ‘id 
a.cri.dsulli...@bsdad.uchicago.edu’ 
on the IPA DC shows me as a member of the POSIX IPA group 
(cri_server_administrators_...@ipa.cri.uchicago.edu)
 as well as the AD domain group in the trusted domain 
(cri-aaa_server_administrat...@bsdad.uchicago.edu).
  This remains consistent across any number of successful sshd logins into the 
DC using my 
a.cri.dsullivan@bsdad.uchicago.edu 
account, including after clearing the cache on the DC.

On the client, I am seeing some unusual behavior.  If I run the commands 
'sss_cache -E; service sssd stop ; rm -rf /var/log/sssd/* ; service sssd start’ 
, then run ‘id 
a.cri.dsulli...@bsdad.uchicago.edu’, 
I see the POSIX IPA group as well as the AD domain group as described above 
(what I presumably want and expect).  However (and this is the unusual part), 
if I attempt to login via SSH (it will fail with HBAC validation), and then run 
the ‘id 
a.cri.dsulli...@bsdad.uchicago.edu’ 
command again , the POSIX IPA group disappears from the list of groups output 
by the id command.  From what I can tell, this group will not reappear in the 
group list on the client until the client cache is cleared.  Presumably this 
behavior is related to the HBAC authentication errors I am experiencing.  I 
have cleared the cache on both the DC and the client using ssh_cache -E and 
this behavior is still exhibited.  With respect to output from testing:

1) The sssd domain log from from the client of the initial id invocation (both 
groups appear) after clearing the cache (on the client) can be found here (this 
output contains both groups): 
https://gist.github.com/dsulli99/7117f8d567cc7cdf727d474b0aeab8da
2) The sssd domain log from the client for the failed sshd login (similar to 
the output I provided yesterday, however re-captured) can be found here (after 
this operation the IPA group disappears from the list of groups from the id 
command): 
https://gist.github.com/dsulli99/668a8799709ff0cd311b321206591124
3) The DC log (after the client cache is cleared) of my running the id 
invocation (from the client) can be found here (this is the DC side of 1) from 
above. https://gist.github.com/dsulli99/a2a5e80b6a8b143afa20024aa40a7b39
4) The DC log of the the failed sshd login into the client (this is the DC side 
of 2) from above is 
https://gist.github.com/dsulli99/4e3ba53c942ad78d7487ae51da92007e

I really appreciate your help with looking at this issue.  As I said I have 
another machine built from the same image that this is working fine on.  I am 
going to keep plugging away at this, I will let you know if I come up with 
anything.

Dan




This e-mail is intended only for the use of the individual or entity to which
it is addressed and may contain information that is privileged and confidential.
If the reader of this e-mail message is not the intended recipient, you are 
hereby notified that any dissemination, distribution or copying of this
communication is prohibited. If you have received this e-mail in error, please 
notify the sender and destroy all copies of the transmittal. 

Thank you
University of Chicago Medicine and Biological Sciences 


-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Hi, Lachlan,

Yes, I see that from here 
(https://www.redhat.com/archives/freeipa-users/2016-May/msg00322.html).  
Unfortunately clearing the cache and restarting SSSD is not proving to help us. 
 I’d be interested to know any progress you make on this issue.

Thank you for responding to me.

Best,

Dan Sullivan


On Jul 12, 2016, at 8:04 PM, Lachlan Musicman 
> wrote:

This is exactly the issue I'm seeing too, various differences, but the symptoms 
are the same.

Main diff would be that sometimes stopping sssd, clearing cache and restarting 
sssd works, but only if individual AD domain members are added to the external 
group - not AD domain groups.

Cheers
L.

--
The most dangerous phrase in the language is, "We've always done it this way."

- Grace Hopper

On 13 July 2016 at 08:07, Sullivan, Daniel [AAA] 
> wrote:
Justin,

I really appreciate you taking the time to respond to me.  This problem is 
driving me crazy and I will certainly take any help I can get. My suspicion is 
that the external user group in the policy below was causing the log entry you 
specified, removing it from the policy does not remediate the problem, even 
after flushing the client cache.

The way I have this setup is as follows:

1) I created a POSIX group in IPA named 
'cri-cri_server_administrators_ipa’
 and allowed IPA to assign the GID.
2) I created an external group in IPA named 
'cri-cri_server_administrators_external’
 and added the AD group in the trusted domain as an external member to this 
group 
(cri-cri_server_administrat...@bsdad.uchicago.edu>).
3) I added the group 
cri-cri_server_administrators_external
 as a member of 
'cri-cri_server_administrators_ipa’

The HBAC rule is configured as (removing the external group does not seem to 
make a difference).

[root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show 
'cri-cri_server_administrators_allow_all'
  Rule name: cri-cri_server_administrators_allow_all
  Host category: all
  Service category: all
  Description: Allow anyone in 
cri-cri_server_administrat...@bsdad.uchicago.edu>
 to login to any machine
  Enabled: TRUE
  User Groups: cri-cri_server_administrators_external, 
cri-cri_server_administrators_ipa
[root@cri-ksysipadcp2 a.cri.dsullivan]#

For example, the problem still persists when the policy is configured in this 
manner:

[root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show 
'cri-cri_server_administrators_allow_all'
  Rule name: cri-cri_server_administrators_allow_all
  Host category: all
  Service category: all
  Description: Allow anyone in 
cri-cri_server_administrat...@bsdad.uchicago.edu>
 to login to any machine
  Enabled: TRUE
  User Groups: cri-cri_server_administrators_ipa

And my login validates against the host in question as follows:

As I said I have this working consistently (i.e. can flush the cash) on another 
host with the same exact version of IPA and SSSD.  Here is a validation of 
hbactest (works with either of the two policy configurations above).

[root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbactest
User name: 
a.cri.dsulli...@bsdad.uchicago.edu>
Target host: 
cri-kcriwebgdp1.cri.uchicago.edu>
Service: sshd

Access granted: True

  Matched rules: cri-cri_server_administrators_allow_all
  Not matched rules: cri-hpc_server_administration
  Not matched rules: Gardner_cluster_login_no_ssh
  Not matched rules: s.cri.ipa-idprovisioner_domain_controllers
[root@cri-ksysipadcp2 a.cri.dsullivan]#

Thank you again for your response.

Best,

Dan

On Jul 12, 2016, at 4:12 PM, Justin Stephenson 
>>
 wrote:


Hello,

I am assuming this is the AD trust user that is having the problem with HBAC, 
in my 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Sumit,

Thank you for getting back to me  I really appreciate you taking the time to 
help me assess this problem (I am not authorized to view this bug).  In order 
to test I upgraded to ipa-server 4.2.0-15.el7_2.17 and flushed the cache on 
both the client and the server; the problem still presents itself. 

I’ve seen some threads around that seem related to what I am experiencing, i.e. 

https://www.redhat.com/archives/freeipa-users/2016-May/msg00354.html

https://www.redhat.com/archives/freeipa-users/2015-December/msg00046.html

Based on my reading I think that the version of the server I upgraded to would 
have fixed this problem, though (it did not).

Dan Sullivan

> On Jul 13, 2016, at 3:20 AM, Sumit Bose  wrote:
> 
> On Wed, Jul 13, 2016 at 08:37:44AM +0200, Jakub Hrozek wrote:
>> On Wed, Jul 13, 2016 at 09:10:07AM +0300, Alexander Bokovoy wrote:
>>> On Tue, 12 Jul 2016, Sullivan, Daniel [AAA] wrote:
 Justin,
 
 I really appreciate you taking the time to respond to me.  This problem
 is driving me crazy and I will certainly take any help I can get. My
 suspicion is that the external user group in the policy below was
 causing the log entry you specified, removing it from the policy does
 not remediate the problem, even after flushing the client cache.
 
 The way I have this setup is as follows:
 
 1) I created a POSIX group in IPA named
 'cri-cri_server_administrators_ipa' and allowed IPA to assign the GID.
 2) I created an external group in IPA named
 'cri-cri_server_administrators_external’ and added the AD group in the
 trusted domain as an external member to this group
 (cri-cri_server_administrat...@bsdad.uchicago.edu).
 3) I added the group cri-cri_server_administrators_external' as a
 member of 'cri-cri_server_administrators_ipa’
 
 The HBAC rule is configured as (removing the external group does not
 seem to make a difference).
 
 [root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show 
 'cri-cri_server_administrators_allow_all'
 Rule name: cri-cri_server_administrators_allow_all
 Host category: all
 Service category: all
 Description: Allow anyone in 
 cri-cri_server_administrat...@bsdad.uchicago.edu
  to login to any machine
 Enabled: TRUE
 User Groups: cri-cri_server_administrators_external, 
 cri-cri_server_administrators_ipa
 [root@cri-ksysipadcp2 a.cri.dsullivan]#
 
 For example, the problem still persists when the policy is configured in 
 this manner:
 
 [root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show 
 'cri-cri_server_administrators_allow_all'
 Rule name: cri-cri_server_administrators_allow_all
 Host category: all
 Service category: all
 Description: Allow anyone in 
 cri-cri_server_administrat...@bsdad.uchicago.edu to login to any machine
 Enabled: TRUE
 User Groups: cri-cri_server_administrators_ipa
 
 And my login validates against the host in question as follows:
 
 As I said I have this working consistently (i.e. can flush the cash) on
 another host with the same exact version of IPA and SSSD.  Here is a
 validation of hbactest (works with either of the two policy
 configurations above).
>>> I think you problems are related to this snippet of your domain log
>>> where SSSD on IPA client was unable to add membership of your user to
>>> any of these groups:
>>> 
> 
> ...
> 
>>> 
>>> as result, the user is viewed by SSSD on this IPA client as not
>>> belonging to the cri-cri_server_administrat...@bsdad.uchicago.edu group
>>> and thus, HBAC rule validation on this client fails.
>> 
>> First, we have some debug messages in this part of sssd that can really
>> use some improvement. That is, some debug messages are expected to
>> report failures and we recover from them later.
>> 
>> But in general Alexander is right. Does 'id
>> a.cri.dsulli...@bsdad.uchicago.edu' report the user as a member of the
>> group that should be allowing access?
>> 
>> If not, I would suggest to run:
>>1) sss_cache -E on both server and client (don't remove the cache,
>>please)
>>2) truncate the logs on server and client
>>3) run id a.cri.dsulli...@bsdad.uchicago.edu on the client
>> then send us the logs from that single id run..
> 
> If some of the IPA group memberships are missing you might hit
> https://bugzilla.redhat.com/show_bug.cgi?id=1304333 which is not fixed
> in the IPA version you use (ipa-4.2.0-15.el7_2.6.1).
> 
> Mabye upgrading the server helps?
> 
> bye,
> Sumit
> 
> -- 
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project



This e-mail is intended only for the use of the 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sumit Bose]

On Wed, Jul 13, 2016 at 08:37:44AM +0200, Jakub Hrozek wrote:
> On Wed, Jul 13, 2016 at 09:10:07AM +0300, Alexander Bokovoy wrote:
> > On Tue, 12 Jul 2016, Sullivan, Daniel [AAA] wrote:
> > > Justin,
> > > 
> > > I really appreciate you taking the time to respond to me.  This problem
> > > is driving me crazy and I will certainly take any help I can get. My
> > > suspicion is that the external user group in the policy below was
> > > causing the log entry you specified, removing it from the policy does
> > > not remediate the problem, even after flushing the client cache.
> > > 
> > > The way I have this setup is as follows:
> > > 
> > > 1) I created a POSIX group in IPA named
> > > 'cri-cri_server_administrators_ipa' and allowed IPA to assign the GID.
> > > 2) I created an external group in IPA named
> > > 'cri-cri_server_administrators_external’ and added the AD group in the
> > > trusted domain as an external member to this group
> > > (cri-cri_server_administrat...@bsdad.uchicago.edu).
> > > 3) I added the group cri-cri_server_administrators_external' as a
> > > member of 'cri-cri_server_administrators_ipa’
> > > 
> > > The HBAC rule is configured as (removing the external group does not
> > > seem to make a difference).
> > > 
> > > [root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show 
> > > 'cri-cri_server_administrators_allow_all'
> > >  Rule name: cri-cri_server_administrators_allow_all
> > >  Host category: all
> > >  Service category: all
> > >  Description: Allow anyone in 
> > > cri-cri_server_administrat...@bsdad.uchicago.edu
> > >  to login to any machine
> > >  Enabled: TRUE
> > >  User Groups: cri-cri_server_administrators_external, 
> > > cri-cri_server_administrators_ipa
> > > [root@cri-ksysipadcp2 a.cri.dsullivan]#
> > > 
> > > For example, the problem still persists when the policy is configured in 
> > > this manner:
> > > 
> > > [root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show 
> > > 'cri-cri_server_administrators_allow_all'
> > >  Rule name: cri-cri_server_administrators_allow_all
> > >  Host category: all
> > >  Service category: all
> > >  Description: Allow anyone in 
> > > cri-cri_server_administrat...@bsdad.uchicago.edu to login to any machine
> > >  Enabled: TRUE
> > >  User Groups: cri-cri_server_administrators_ipa
> > > 
> > > And my login validates against the host in question as follows:
> > > 
> > > As I said I have this working consistently (i.e. can flush the cash) on
> > > another host with the same exact version of IPA and SSSD.  Here is a
> > > validation of hbactest (works with either of the two policy
> > > configurations above).
> > I think you problems are related to this snippet of your domain log
> > where SSSD on IPA client was unable to add membership of your user to
> > any of these groups:
> > 

...

> > 
> > as result, the user is viewed by SSSD on this IPA client as not
> > belonging to the cri-cri_server_administrat...@bsdad.uchicago.edu group
> > and thus, HBAC rule validation on this client fails.
> 
> First, we have some debug messages in this part of sssd that can really
> use some improvement. That is, some debug messages are expected to
> report failures and we recover from them later.
> 
> But in general Alexander is right. Does 'id
> a.cri.dsulli...@bsdad.uchicago.edu' report the user as a member of the
> group that should be allowing access?
> 
> If not, I would suggest to run:
> 1) sss_cache -E on both server and client (don't remove the cache,
> please)
> 2) truncate the logs on server and client
> 3) run id a.cri.dsulli...@bsdad.uchicago.edu on the client
> then send us the logs from that single id run..

If some of the IPA group memberships are missing you might hit
https://bugzilla.redhat.com/show_bug.cgi?id=1304333 which is not fixed
in the IPA version you use (ipa-4.2.0-15.el7_2.6.1).

Mabye upgrading the server helps?

bye,
Sumit

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Jakub Hrozek]

On Wed, Jul 13, 2016 at 09:10:07AM +0300, Alexander Bokovoy wrote:
> On Tue, 12 Jul 2016, Sullivan, Daniel [AAA] wrote:
> > Justin,
> > 
> > I really appreciate you taking the time to respond to me.  This problem
> > is driving me crazy and I will certainly take any help I can get. My
> > suspicion is that the external user group in the policy below was
> > causing the log entry you specified, removing it from the policy does
> > not remediate the problem, even after flushing the client cache.
> > 
> > The way I have this setup is as follows:
> > 
> > 1) I created a POSIX group in IPA named
> > 'cri-cri_server_administrators_ipa' and allowed IPA to assign the GID.
> > 2) I created an external group in IPA named
> > 'cri-cri_server_administrators_external’ and added the AD group in the
> > trusted domain as an external member to this group
> > (cri-cri_server_administrat...@bsdad.uchicago.edu).
> > 3) I added the group cri-cri_server_administrators_external' as a
> > member of 'cri-cri_server_administrators_ipa’
> > 
> > The HBAC rule is configured as (removing the external group does not
> > seem to make a difference).
> > 
> > [root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show 
> > 'cri-cri_server_administrators_allow_all'
> >  Rule name: cri-cri_server_administrators_allow_all
> >  Host category: all
> >  Service category: all
> >  Description: Allow anyone in 
> > cri-cri_server_administrat...@bsdad.uchicago.edu
> >  to login to any machine
> >  Enabled: TRUE
> >  User Groups: cri-cri_server_administrators_external, 
> > cri-cri_server_administrators_ipa
> > [root@cri-ksysipadcp2 a.cri.dsullivan]#
> > 
> > For example, the problem still persists when the policy is configured in 
> > this manner:
> > 
> > [root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show 
> > 'cri-cri_server_administrators_allow_all'
> >  Rule name: cri-cri_server_administrators_allow_all
> >  Host category: all
> >  Service category: all
> >  Description: Allow anyone in 
> > cri-cri_server_administrat...@bsdad.uchicago.edu to login to any machine
> >  Enabled: TRUE
> >  User Groups: cri-cri_server_administrators_ipa
> > 
> > And my login validates against the host in question as follows:
> > 
> > As I said I have this working consistently (i.e. can flush the cash) on
> > another host with the same exact version of IPA and SSSD.  Here is a
> > validation of hbactest (works with either of the two policy
> > configurations above).
> I think you problems are related to this snippet of your domain log
> where SSSD on IPA client was unable to add membership of your user to
> any of these groups:
> 
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [get_groups_dns] (0x0400): Root domain uses fully-qualified names,
> objects might not be correctly added to groups with short names.
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [get_groups_dns] (0x0400): Root domain uses fully-qualified names,
> objects might not be correctly added to groups with short names.
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [ipa_s2n_save_objects] (0x2000): Updating memberships for
> a.cri.dsulli...@bsdad.uchicago.edu
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [sysdb_mod_group_member] (0x0080): ldb_modify failed: [No such
> object](32)[ldb_wait: No such object (32)]
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [sysdb_mod_group_member] (0x0400): Error: 2 (No such file or directory)
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [sysdb_update_members_ex] (0x0020): Could not add member
> [a.cri.dsulli...@bsdad.uchicago.edu] to group
> [name=cri-aaa_sms_administrat...@bsdad.uchicago.edu,cn=groups,cn=bsdad.uchicago.edu,cn=sysdb].
> Skipping.
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [sysdb_mod_group_member] (0x0080): ldb_modify failed: [No such
> object](32)[ldb_wait: No such object (32)]
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [sysdb_mod_group_member] (0x0400): Error: 2 (No such file or directory)
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [sysdb_update_members_ex] (0x0020): Could not add member
> [a.cri.dsulli...@bsdad.uchicago.edu] to group
> [name=cri-aaa_cvs_reposit...@bsdad.uchicago.edu,cn=groups,cn=bsdad.uchicago.edu,cn=sysdb].
> Skipping.
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [sysdb_mod_group_member] (0x0080): ldb_modify failed: [No such
> object](32)[ldb_wait: No such object (32)]
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [sysdb_mod_group_member] (0x0400): Error: 2 (No such file or directory)
> (Tue Jul 12 13:29:58 2016) [sssd[be[ipa.cri.uchicago.edu]]]
> [sysdb_update_members_ex] (0x0020): Could not add member
> [a.cri.dsulli...@bsdad.uchicago.edu] to group
> [name=cri-active_us...@bsdad.uchicago.edu,cn=groups,cn=bsdad.uchicago.edu,cn=sysdb].
> Skipping.
> (Tue Jul 12 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Lachlan Musicman]

This is exactly the issue I'm seeing too, various differences, but the
symptoms are the same.

Main diff would be that sometimes stopping sssd, clearing cache and
restarting sssd works, but only if individual AD domain members are added
to the external group - not AD domain groups.

Cheers
L.

--
The most dangerous phrase in the language is, "We've always done it this
way."

- Grace Hopper

On 13 July 2016 at 08:07, Sullivan, Daniel [AAA] <
dsulliv...@bsd.uchicago.edu> wrote:

> Justin,
>
> I really appreciate you taking the time to respond to me.  This problem is
> driving me crazy and I will certainly take any help I can get. My suspicion
> is that the external user group in the policy below was causing the log
> entry you specified, removing it from the policy does not remediate the
> problem, even after flushing the client cache.
>
> The way I have this setup is as follows:
>
> 1) I created a POSIX group in IPA named 'cri-cri_server_administrators_ipa<
> https://cri-ksysipadcp2.cri.uchicago.edu/ipa/ui/#cri-cri_server_administrators_ipa>’
> and allowed IPA to assign the GID.
> 2) I created an external group in IPA named
> 'cri-cri_server_administrators_external<
> https://cri-ksysipadcp2.cri.uchicago.edu/ipa/ui/#cri-cri_server_administrators_external>’
> and added the AD group in the trusted domain as an external member to this
> group (cri-cri_server_administrat...@bsdad.uchicago.edu cri-cri_server_administrat...@bsdad.uchicago.edu>).
> 3) I added the group cri-cri_server_administrators_external<
> https://cri-ksysipadcp2.cri.uchicago.edu/ipa/ui/#cri-cri_server_administrators_external>
> as a member of 'cri-cri_server_administrators_ipa<
> https://cri-ksysipadcp2.cri.uchicago.edu/ipa/ui/#cri-cri_server_administrators_ipa
> >’
>
> The HBAC rule is configured as (removing the external group does not seem
> to make a difference).
>
> [root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show
> 'cri-cri_server_administrators_allow_all'
>   Rule name: cri-cri_server_administrators_allow_all
>   Host category: all
>   Service category: all
>   Description: Allow anyone in
> cri-cri_server_administrat...@bsdad.uchicago.edu cri-cri_server_administrat...@bsdad.uchicago.edu> to login to any machine
>   Enabled: TRUE
>   User Groups: cri-cri_server_administrators_external,
> cri-cri_server_administrators_ipa
> [root@cri-ksysipadcp2 a.cri.dsullivan]#
>
> For example, the problem still persists when the policy is configured in
> this manner:
>
> [root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbacrule-show
> 'cri-cri_server_administrators_allow_all'
>   Rule name: cri-cri_server_administrators_allow_all
>   Host category: all
>   Service category: all
>   Description: Allow anyone in
> cri-cri_server_administrat...@bsdad.uchicago.edu cri-cri_server_administrat...@bsdad.uchicago.edu> to login to any machine
>   Enabled: TRUE
>   User Groups: cri-cri_server_administrators_ipa
>
> And my login validates against the host in question as follows:
>
> As I said I have this working consistently (i.e. can flush the cash) on
> another host with the same exact version of IPA and SSSD.  Here is a
> validation of hbactest (works with either of the two policy configurations
> above).
>
> [root@cri-ksysipadcp2 a.cri.dsullivan]# ipa hbactest
> User name: a.cri.dsulli...@bsdad.uchicago.edu a.cri.dsulli...@bsdad.uchicago.edu>
> Target host: cri-kcriwebgdp1.cri.uchicago.edu<
> http://cri-kcriwebgdp1.cri.uchicago.edu>
> Service: sshd
> 
> Access granted: True
> 
>   Matched rules: cri-cri_server_administrators_allow_all
>   Not matched rules: cri-hpc_server_administration
>   Not matched rules: Gardner_cluster_login_no_ssh
>   Not matched rules: s.cri.ipa-idprovisioner_domain_controllers
> [root@cri-ksysipadcp2 a.cri.dsullivan]#
>
> Thank you again for your response.
>
> Best,
>
> Dan
>
> On Jul 12, 2016, at 4:12 PM, Justin Stephenson  > wrote:
>
>
> Hello,
>
> I am assuming this is the AD trust user that is having the problem with
> HBAC, in my testing I was only allowed access when the HBAC rule is linked
> to the IDM POSIX AD trust group and not the external group used to retrieve
> AD trust users. I noticed the following in the logs which is why I mention
> this:
>
> (Tue Jul 12 13:30:12 2016) [sssd[be[ipa.cri.uchicago.edu<
> http://ipa.cri.uchicago.edu>]]] [hbac_user_attrs_to_rule] (0x2000): Added
> non-POSIX group [cri-cri_server_administrators_external] to rule
> [cri-cri_server_administrators_allow_all]
>
> If this does not help, could you share with us more about the HBAC rule
> 'cri-cri_server_administrators_allow_all' and how it is configured?
>
> # ipa hbacrule-show 'cri-cri_server_administrators_allow_all'
>
> Kind regards,
>
> Justin Stephenson
>
> On 07/12/2016 04:11 PM, Sullivan, Daniel [AAA] wrote:
>
> Hi,
>
> I am experiencing an HBAC issue that is proving to be very difficult to
> diagnose.  It appears very closely 

Re: [Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Justin Stephenson]

Hello,

I am assuming this is the AD trust user that is having the problem with 
HBAC, in my testing I was only allowed access when the HBAC rule is 
linked to the IDM POSIX AD trust group and not the external group used 
to retrieve AD trust users. I noticed the following in the logs which is 
why I mention this:


   /(Tue Jul 12 13:30:12 2016) [sssd[be[ipa.cri.uchicago.edu]]]
   [hbac_user_attrs_to_rule] (0x2000): Added non-POSIX group
   [cri-cri_server_administrators_external] to rule
   [cri-cri_server_administrators_allow_all]/

If this does not help, could you share with us more about the HBAC rule 
'cri-cri_server_administrators_allow_all' and how it is configured?


# ipa hbacrule-show 'cri-cri_server_administrators_allow_all'

Kind regards,

Justin Stephenson

On 07/12/2016 04:11 PM, Sullivan, Daniel [AAA] wrote:

Hi,

I am experiencing an HBAC issue that is proving to be very difficult to 
diagnose.  It appears very closely related to the issue described in this 
thread 
(https://lists.fedorahosted.org/archives/list/sssd-us...@lists.fedorahosted.org/thread/DTX4LP5VI2AHANMT4QFXERCN7US2TCUB/),
 except that clearing the cache does not fix the problem.  I am further stumped 
by the fact that I have an additional machine that was deployed from an 
identical VMWare template image which IPA HBAC works correctly on.  From a 
client perspective I am working with a fully updated version of RHEL 6.8 with 
ipa-client 3.0.0-50.el6.1 and sssd 1.13.3-22.el6.  We have a domain with 2 IPA 
domain controllers (RHEL 7.2 and ipa-server 4.2.0-15.el7_2.6.1); I have since 
shut down one of the two domain controllers and cleared the cache 
(/var/lib/sssd/db/*) on both clients and restarted sssd (to isolate a potential 
replication problem between DCs); the HBAC rule validates correctly on the only 
remaining DC (basically an!
   any any rule). HBAC (the ability to login via sshd) continues to work on 
only one of the two clients.

>From what I can tell, both clients have the same version of all ipa-client and 
sssd (and presumably related packages as both clients are fully updated). I have 
compared their /etc/sshd/sshd_config, /etc/sssd/sssd.conf and all configurations 
in /etc/pam.d and both systems appear consistent.

I feel that it is worthwhile to mention that I believe that one of the two 
machines in question (the one that is not working) was bound as a CentrifyDC 
client.  We are planning on replacing CentrifyDC with FreeIPA (for several 
reasons), so it is important that we are able to take an existing CentrifyDC 
client, unbind it, uninstall the CentrifyDC package(s), and install FreeIPA in 
its place.  Regardless of whether CentrifyDC was previously installed, I feel 
that my somewhat thorough examination of /etc/sshd/sshd_config and the contents 
of /etc/pam.d would negate any potential residual configuration from Centrify 
that would cause this sort of problem.  I have posted my domain log here: 
http://pastebin.com/41KeSnq4

It is also probably worthwhile to mention that I am authenticating as a user in 
a trusted domain, although I believe this should be apparent in the the 
pastebin.

I am hoping that a subject matter expert in IPA and or SSSD would be able to 
help me further diagnose the access denied by HBAC entry that is present in the 
pastebin specified above.  As I said, I have cleared /var/lib/sss/db/* and 
reinstalled IPA-client several times.  I have also rebooted the system 
completely.

Thank you for considering helping me; I appreciate your time and expertise.

Best,

Dan





This e-mail is intended only for the use of the individual or entity to which
it is addressed and may contain information that is privileged and confidential.
If the reader of this e-mail message is not the intended recipient, you are
hereby notified that any dissemination, distribution or copying of this
communication is prohibited. If you have received this e-mail in error, please
notify the sender and destroy all copies of the transmittal.

Thank you
University of Chicago Medicine and Biological Sciences




-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Freeipa-users] IPA HBAC access using SSSD for user in trusted AD domain (RHEL 6.8)

[Sullivan, Daniel [AAA]]

Hi,

I am experiencing an HBAC issue that is proving to be very difficult to 
diagnose.  It appears very closely related to the issue described in this 
thread 
(https://lists.fedorahosted.org/archives/list/sssd-us...@lists.fedorahosted.org/thread/DTX4LP5VI2AHANMT4QFXERCN7US2TCUB/),
 except that clearing the cache does not fix the problem.  I am further stumped 
by the fact that I have an additional machine that was deployed from an 
identical VMWare template image which IPA HBAC works correctly on.  From a 
client perspective I am working with a fully updated version of RHEL 6.8 with 
ipa-client 3.0.0-50.el6.1 and sssd 1.13.3-22.el6.  We have a domain with 2 IPA 
domain controllers (RHEL 7.2 and ipa-server 4.2.0-15.el7_2.6.1); I have since 
shut down one of the two domain controllers and cleared the cache 
(/var/lib/sssd/db/*) on both clients and restarted sssd (to isolate a potential 
replication problem between DCs); the HBAC rule validates correctly on the only 
remaining DC (basically an!
  any any rule). HBAC (the ability to login via sshd) continues to work on only 
one of the two clients.

>From what I can tell, both clients have the same version of all ipa-client and 
>sssd (and presumably related packages as both clients are fully updated). I 
>have compared their /etc/sshd/sshd_config, /etc/sssd/sssd.conf and all 
>configurations in /etc/pam.d and both systems appear consistent.

I feel that it is worthwhile to mention that I believe that one of the two 
machines in question (the one that is not working) was bound as a CentrifyDC 
client.  We are planning on replacing CentrifyDC with FreeIPA (for several 
reasons), so it is important that we are able to take an existing CentrifyDC 
client, unbind it, uninstall the CentrifyDC package(s), and install FreeIPA in 
its place.  Regardless of whether CentrifyDC was previously installed, I feel 
that my somewhat thorough examination of /etc/sshd/sshd_config and the contents 
of /etc/pam.d would negate any potential residual configuration from Centrify 
that would cause this sort of problem.  I have posted my domain log here: 
http://pastebin.com/41KeSnq4

It is also probably worthwhile to mention that I am authenticating as a user in 
a trusted domain, although I believe this should be apparent in the the 
pastebin.

I am hoping that a subject matter expert in IPA and or SSSD would be able to 
help me further diagnose the access denied by HBAC entry that is present in the 
pastebin specified above.  As I said, I have cleared /var/lib/sss/db/* and 
reinstalled IPA-client several times.  I have also rebooted the system 
completely.

Thank you for considering helping me; I appreciate your time and expertise.

Best,

Dan





This e-mail is intended only for the use of the individual or entity to which
it is addressed and may contain information that is privileged and confidential.
If the reader of this e-mail message is not the intended recipient, you are 
hereby notified that any dissemination, distribution or copying of this
communication is prohibited. If you have received this e-mail in error, please 
notify the sender and destroy all copies of the transmittal. 

Thank you
University of Chicago Medicine and Biological Sciences 


-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project