[ https://issues.apache.org/jira/browse/YARN-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993836#comment-14993836 ]
Jason Lowe commented on YARN-4336: ---------------------------------- I believe this is a duplicate of YARN-3452. We fixed it by reverting HADOOP-10650 in our internal build since we don't need the blacklisting functionality added by that feature, and that's what caused the excess lookups. IMHO the real fix is to have YARN not use bogus user names, but I don't know if that's going to be an easy change to make. > YARN NodeManager - Container Initialization - Excessive load on NSS/LDAP > ------------------------------------------------------------------------ > > Key: YARN-4336 > URL: https://issues.apache.org/jira/browse/YARN-4336 > Project: Hadoop YARN > Issue Type: Bug > Affects Versions: 2.4.0, 2.4.1, 2.6.0, 2.7.0, 2.6.1, 2.7.1 > Environment: NSS w/ SSSD or Dell/Quest - VASD > Reporter: Greg Senia > Assignee: Greg Senia > Attachments: YARN-4336-tactical.txt > > > Hi folks after performing some debug for our Unix Engineering and Active > Directory teams it was discovered that on YARN Container Initialization a > call via Hadoop Common AccessControlList.java: > for(String group: ugi.getGroupNames()) { > if (groups.contains(group)) { > return true; > } > } > Unfortunately with the security call to check access on > "appattempt_XXXXXXXXXXXXX_XXXXX_XXXXX" will always return false but will make > unnecessary calls to NameSwitch service on linux which will call things like > SSSD/Quest VASD which will then initiate LDAP calls looking for non existent > userid's causing excessive load on LDAP. > For now our tactical work around is as follows: > /** > * Checks if a user represented by the provided {@link UserGroupInformation} > * is a member of the Access Control List > * @param ugi UserGroupInformation to check if contained in the ACL > * @return true if ugi is member of the list > */ > public final boolean isUserInList(UserGroupInformation ugi) { > if (allAllowed || users.contains(ugi.getShortUserName())) { > return true; > } else { > String patternString = "^appattempt_\\d+_\\d+_\\d+$"; > Pattern pattern = Pattern.compile(patternString); > Matcher matcher = pattern.matcher(ugi.getShortUserName()); > boolean matches = matcher.matches(); > if (matches) { > LOG.debug("Bailing !! AppAttempt Matches DONOT call UGI FOR > GROUPS!!");; > return false; > } > > > for(String group: ugi.getGroupNames()) { > if (groups.contains(group)) { > return true; > } > } > } > return false; > } > public boolean isUserAllowed(UserGroupInformation ugi) { > return isUserInList(ugi); > } > Example of VASD Debug log showing the lookups for one task attempt 32 of them: > One task: > Oct 30 22:55:43 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:55:43 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:55:43 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:55:43 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:56:15 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:56:15 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:56:15 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:56:15 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:56:45 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:56:45 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:56:45 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:56:45 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:57:18 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:57:18 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:57:18 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:57:18 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:57:49 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:57:49 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:57:49 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:57:49 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:58:22 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:58:22 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:58:22 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:58:22 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:58:52 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:58:52 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:58:52 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:58:52 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:59:30 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:59:30 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching > GC for host service domain EXNSD.EXA.EXAMPLE.COM with filter > (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001)) > Oct 30 22:59:30 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> > Oct 30 22:59:30 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching > <GC://@EXNSD.EXA.EXAMPLE.COM> with > filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>, > base=<>, scope=<sub> -- This message was sent by Atlassian JIRA (v6.3.4#6332)