[ 
https://issues.apache.org/jira/browse/YARN-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993932#comment-14993932
 ] 

Jason Lowe commented on YARN-4336:
----------------------------------

bq. Seems like this could also be related...  
https://issues.apache.org/jira/browse/HADOOP-12413
Nice find!  I totally missed that when it went by.  I'll pull that fix into the 
2.6 and 2.7 lines.  I think that could eliminate the bogus lookups in practice 
when the reverse ACL isn't being used.

bq.  Do you see an issue with my workaround for now in my own env until HWX can 
provide a final solution?
It will work.  Nit: it's pricey to compile the pattern every time, could just 
compile it once.  Or as I mentioned above, I think pulling in HADOOP-12413 to 
your build could also eliminate the bogus lookups (assuming you don't use the 
reverse ACL feature).


> YARN NodeManager - Container Initialization - Excessive load on NSS/LDAP
> ------------------------------------------------------------------------
>
>                 Key: YARN-4336
>                 URL: https://issues.apache.org/jira/browse/YARN-4336
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.4.0, 2.4.1, 2.6.0, 2.7.0, 2.6.1, 2.7.1
>         Environment: NSS w/ SSSD or Dell/Quest - VASD
>            Reporter: Greg Senia
>            Assignee: Greg Senia
>         Attachments: YARN-4336-tactical.txt
>
>
> Hi folks after performing some debug for our Unix Engineering and Active 
> Directory teams it was discovered that on YARN Container Initialization a 
> call via Hadoop Common AccessControlList.java:
>   for(String group: ugi.getGroupNames()) {
>         if (groups.contains(group)) {
>           return true;
>         }
>       }
> Unfortunately with the security call to check access on 
> "appattempt_XXXXXXXXXXXXX_XXXXX_XXXXX" will always return false but will make 
> unnecessary calls to NameSwitch service on linux which will call things like 
> SSSD/Quest VASD which will then initiate LDAP calls looking for non existent 
> userid's causing excessive load on LDAP.
> For now our tactical work around is as follows:
> /**
>    * Checks if a user represented by the provided {@link UserGroupInformation}
>    * is a member of the Access Control List
>    * @param ugi UserGroupInformation to check if contained in the ACL
>    * @return true if ugi is member of the list
>    */
>   public final boolean isUserInList(UserGroupInformation ugi) {
>     if (allAllowed || users.contains(ugi.getShortUserName())) {
>       return true;
>     } else {
>         String patternString = "^appattempt_\\d+_\\d+_\\d+$";
>         Pattern pattern = Pattern.compile(patternString);
>         Matcher matcher = pattern.matcher(ugi.getShortUserName());
>         boolean matches = matcher.matches();
>         if (matches) {
>               LOG.debug("Bailing !! AppAttempt Matches DONOT call UGI FOR 
> GROUPS!!");;
>               return false;
>         }
>       
>       
>       for(String group: ugi.getGroupNames()) {
>         if (groups.contains(group)) {
>           return true;
>         }
>       }
>     }
>     return false;
>   }
>   public boolean isUserAllowed(UserGroupInformation ugi) {
>     return isUserInList(ugi);
>   }
> Example of VASD Debug log showing the lookups for one task attempt 32 of them:
> One task:
> Oct 30 22:55:43 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:55:43 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:55:43 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:55:43 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:56:15 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:56:15 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:56:15 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:56:15 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:56:45 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:56:45 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:56:45 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:56:45 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:57:18 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:57:18 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:57:18 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:57:18 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:57:49 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:57:49 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:57:49 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:57:49 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:58:22 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:58:22 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:58:22 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:58:22 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:58:52 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:58:52 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:58:52 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:58:52 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:59:30 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:59:30 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching 
> GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter 
> (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:59:30 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>
> Oct 30 22:59:30 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching 
> <GC://@EXNSD.EXA.EXAMPLE.COM> with 
> filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
>  base=<>, scope=<sub>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to