[
https://issues.apache.org/jira/browse/IMPALA-14699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fang-Yu Rao updated IMPALA-14699:
---------------------------------
Description:
Impala would encounter the error of "Recursive load of credential provider"
when a remote jceks path is put in the core-site.
The following is part of the stack trace we'd get if we put the following
key-value pair of ("hadoop.security.credential.provider.path",
"jceks://[email protected]:8020/secret.jceks")
in core-site.xml.
{code}
26/01/24 19:22:53 WARN security.LdapGroupsMapping: Exception while trying to
get password for alias hadoop.security.group.mapping.ldap.ssl.keystore.password:
java.io.IOException: Configuration problem with provider path.
at
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2469)
at
org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2388)
at
org.apache.hadoop.security.LdapGroupsMapping.getPassword(LdapGroupsMapping.java:879)
at
org.apache.hadoop.security.LdapGroupsMapping.loadSslConf(LdapGroupsMapping.java:837)
at
org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:733)
at
org.apache.hadoop.security.RuleBasedLdapGroupsMapping.setConf(RuleBasedLdapGroupsMapping.java:58)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
at org.apache.hadoop.security.Groups.<init>(Groups.java:105)
at org.apache.hadoop.security.Groups.<init>(Groups.java:101)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:476)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:352)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:314)
at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008)
at
org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:743)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:693)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:604)
...
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3509)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:364)
at
org.apache.hadoop.security.alias.KeyStoreProvider.initFileSystem(KeyStoreProvider.java:84)
at
org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:85)
at
org.apache.hadoop.security.alias.KeyStoreProvider.<init>(KeyStoreProvider.java:49)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:42)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:35)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68)
at
org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91)
at
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2450)
at
org.apache.hadoop.security.LdapGroupsMapping.getPasswordFromCredentialProviders(LdapGroupsMapping.java:858)
at
org.apache.hadoop.security.LdapGroupsMapping.loadSslConf(LdapGroupsMapping.java:846)
at
org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:733)
at
org.apache.hadoop.security.RuleBasedLdapGroupsMapping.setConf(RuleBasedLdapGroupsMapping.java:58)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
at org.apache.hadoop.security.Groups.<init>(Groups.java:105)
at org.apache.hadoop.security.Groups.<init>(Groups.java:101)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:476)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:352)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:314)
at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008)
at
org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:743)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:693)
Caused by: org.apache.hadoop.fs.PathIOException:
`jceks://[email protected]:8020/secret.jceks':
Recursive load of credential provider; if loading a JCEKS file, this means that
the filesystem connector is trying to load the same file
{code}
Currently we instantiate Configuration instance at
https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/service/JniFrontend.java#L719.
We need to figure out where else to instantiate Configuration so that we won't
hit this issue described above. I think it's
"Groups.getUserToGroupsMappingService(CONF)" that triggers the recursion.
{code}
// Caching this saves ~50ms per call to getHadoopConfig
private static final Configuration CONF = new Configuration();
private static final Groups GROUPS =
Groups.getUserToGroupsMappingService(CONF);
{code}
was:
Impala would encounter the error of "Recursive load of credential provider"
when a remote jceks path is put in the core-site.
The following is part of the stack trace we'd get if we put the following
key-value pair of ("hadoop.security.credential.provider.path",
"jceks://[email protected]:8020/secret.jceks")
in core-site.xml.
{code}
26/01/24 19:22:53 WARN security.LdapGroupsMapping: Exception while trying to
get password for alias hadoop.security.group.mapping.ldap.ssl.keystore.password:
java.io.IOException: Configuration problem with provider path.
at
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2469)
at
org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2388)
at
org.apache.hadoop.security.LdapGroupsMapping.getPassword(LdapGroupsMapping.java:879)
at
org.apache.hadoop.security.LdapGroupsMapping.loadSslConf(LdapGroupsMapping.java:837)
at
org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:733)
at
org.apache.hadoop.security.RuleBasedLdapGroupsMapping.setConf(RuleBasedLdapGroupsMapping.java:58)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
at org.apache.hadoop.security.Groups.<init>(Groups.java:105)
at org.apache.hadoop.security.Groups.<init>(Groups.java:101)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:476)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:352)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:314)
at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008)
at
org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:743)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:693)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:604)
...
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3509)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:364)
at
org.apache.hadoop.security.alias.KeyStoreProvider.initFileSystem(KeyStoreProvider.java:84)
at
org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:85)
at
org.apache.hadoop.security.alias.KeyStoreProvider.<init>(KeyStoreProvider.java:49)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:42)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:35)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68)
at
org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91)
at
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2450)
at
org.apache.hadoop.security.LdapGroupsMapping.getPasswordFromCredentialProviders(LdapGroupsMapping.java:858)
at
org.apache.hadoop.security.LdapGroupsMapping.loadSslConf(LdapGroupsMapping.java:846)
at
org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:733)
at
org.apache.hadoop.security.RuleBasedLdapGroupsMapping.setConf(RuleBasedLdapGroupsMapping.java:58)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
at org.apache.hadoop.security.Groups.<init>(Groups.java:105)
at org.apache.hadoop.security.Groups.<init>(Groups.java:101)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:476)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:352)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:314)
at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008)
at
org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:743)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:693)
Caused by: org.apache.hadoop.fs.PathIOException:
`jceks://[email protected]:8020/secret.jceks':
Recursive load of credential provider; if loading a JCEKS file, this means that
the filesystem connector is trying to load the same file
{code}
Currently we instantiate Configuration instance at
https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/service/JniFrontend.java#L719.
We need to figure out where else to instantiate Configuration so that we won't
hit this issue described above.
{code}
// Caching this saves ~50ms per call to getHadoopConfig
private static final Configuration CONF = new Configuration();
private static final Groups GROUPS =
Groups.getUserToGroupsMappingService(CONF);
{code}
> Impala would encounter the error of "Recursive load of credential provider"
> when a remote jceks path is put in the core-site
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-14699
> URL: https://issues.apache.org/jira/browse/IMPALA-14699
> Project: IMPALA
> Issue Type: Bug
> Reporter: Fang-Yu Rao
> Priority: Major
>
> Impala would encounter the error of "Recursive load of credential provider"
> when a remote jceks path is put in the core-site.
> The following is part of the stack trace we'd get if we put the following
> key-value pair of ("hadoop.security.credential.provider.path",
> "jceks://[email protected]:8020/secret.jceks")
> in core-site.xml.
> {code}
> 26/01/24 19:22:53 WARN security.LdapGroupsMapping: Exception while trying to
> get password for alias
> hadoop.security.group.mapping.ldap.ssl.keystore.password:
> java.io.IOException: Configuration problem with provider path.
> at
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2469)
> at
> org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2388)
> at
> org.apache.hadoop.security.LdapGroupsMapping.getPassword(LdapGroupsMapping.java:879)
> at
> org.apache.hadoop.security.LdapGroupsMapping.loadSslConf(LdapGroupsMapping.java:837)
> at
> org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:733)
> at
> org.apache.hadoop.security.RuleBasedLdapGroupsMapping.setConf(RuleBasedLdapGroupsMapping.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
> at org.apache.hadoop.security.Groups.<init>(Groups.java:105)
> at org.apache.hadoop.security.Groups.<init>(Groups.java:101)
> at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:476)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:352)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:314)
> at
> org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008)
> at
> org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:743)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:693)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:604)
> ...
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3509)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:364)
> at
> org.apache.hadoop.security.alias.KeyStoreProvider.initFileSystem(KeyStoreProvider.java:84)
> at
> org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:85)
> at
> org.apache.hadoop.security.alias.KeyStoreProvider.<init>(KeyStoreProvider.java:49)
> at
> org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:42)
> at
> org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:35)
> at
> org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68)
> at
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91)
> at
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2450)
> at
> org.apache.hadoop.security.LdapGroupsMapping.getPasswordFromCredentialProviders(LdapGroupsMapping.java:858)
> at
> org.apache.hadoop.security.LdapGroupsMapping.loadSslConf(LdapGroupsMapping.java:846)
> at
> org.apache.hadoop.security.LdapGroupsMapping.setConf(LdapGroupsMapping.java:733)
> at
> org.apache.hadoop.security.RuleBasedLdapGroupsMapping.setConf(RuleBasedLdapGroupsMapping.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
> at org.apache.hadoop.security.Groups.<init>(Groups.java:105)
> at org.apache.hadoop.security.Groups.<init>(Groups.java:101)
> at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:476)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:352)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:314)
> at
> org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008)
> at
> org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:743)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:693)
> Caused by: org.apache.hadoop.fs.PathIOException:
> `jceks://[email protected]:8020/secret.jceks':
> Recursive load of credential provider; if loading a JCEKS file, this means
> that the filesystem connector is trying to load the same file
> {code}
> Currently we instantiate Configuration instance at
> https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/service/JniFrontend.java#L719.
> We need to figure out where else to instantiate Configuration so that we
> won't hit this issue described above. I think it's
> "Groups.getUserToGroupsMappingService(CONF)" that triggers the recursion.
> {code}
> // Caching this saves ~50ms per call to getHadoopConfig
> private static final Configuration CONF = new Configuration();
> private static final Groups GROUPS =
> Groups.getUserToGroupsMappingService(CONF);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
