Greg Senia created YARN-4336:
--------------------------------
Summary: YARN NodeManager - Container Initialization - Excessive
load on NSS/LDAP
Key: YARN-4336
URL: https://issues.apache.org/jira/browse/YARN-4336
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.1, 2.6.1, 2.7.0, 2.6.0, 2.4.1, 2.4.0
Environment: NSS w/ SSSD or Dell/Quest - VASD
Reporter: Greg Senia
Hi folks after performing some debug for our Unix Engineering and Active
Directory teams it was discovered that on YARN Container Initialization a call
via Hadoop Common AccessControlList.java:
for(String group: ugi.getGroupNames()) {
if (groups.contains(group)) {
return true;
}
}
Unfortunately with the security call to check access on
"appattempt_XXXXXXXXXXXXX_XXXXX_XXXXX" will always return false but will make
unnecessary calls to NameSwitch service on linux which will call things like
SSSD/Quest VASD which will then initiate LDAP calls looking for non existent
userid's causing excessive load on LDAP.
For now our tactical work around is as follows:
/**
* Checks if a user represented by the provided {@link UserGroupInformation}
* is a member of the Access Control List
* @param ugi UserGroupInformation to check if contained in the ACL
* @return true if ugi is member of the list
*/
public final boolean isUserInList(UserGroupInformation ugi) {
if (allAllowed || users.contains(ugi.getShortUserName())) {
return true;
} else {
String patternString = "^appattempt_\\d+_\\d+_\\d+$";
Pattern pattern = Pattern.compile(patternString);
Matcher matcher = pattern.matcher(ugi.getShortUserName());
boolean matches = matcher.matches();
if (matches) {
LOG.debug("Bailing !! AppAttempt Matches DONOT call UGI FOR
GROUPS!!");;
return false;
}
for(String group: ugi.getGroupNames()) {
if (groups.contains(group)) {
return true;
}
}
}
return false;
}
public boolean isUserAllowed(UserGroupInformation ugi) {
return isUserInList(ugi);
}
Example of VASD Debug log showing the lookups for one task attempt 32 of them:
One task:
Oct 30 22:55:43 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:55:43 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:55:43 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:55:43 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:56:15 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:56:15 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:56:15 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:56:15 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:56:45 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:56:45 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:56:45 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:56:45 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:57:18 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:57:18 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:57:18 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:57:18 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:57:49 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:57:49 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:57:49 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:57:49 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:58:22 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:58:22 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:58:22 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:58:22 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:58:52 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:58:52 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:58:52 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:58:52 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:59:30 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:59:30 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC
for host service domain EXNSD.EXA.EXAMPLE.COM with filter
(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
Oct 30 22:59:30 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
Oct 30 22:59:30 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching
<GC://@EXNSD.EXA.EXAMPLE.COM> with
filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
