[
https://issues.apache.org/jira/browse/HADOOP-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446474#comment-16446474
]
Xiao Chen edited comment on HADOOP-15401 at 4/22/18 4:31 AM:
-------------------------------------------------------------
Edit:
Searched impala code but don't find any faulty usage. Will just have
HADOOP-9747 for now, to 'fix' this exception. The offending case should
theoretically come up somewhere else too in the future, which may give more
clues...
Thanks for the time here Daryn.
was (Author: xiaochen):
bq. The private credentials set returned by the subject is a synchronized set,
so a direct remove is safe.
Removing from the set can still screw up the iterator though. See the test
below, which would end up throw ConcurrentModificationException:
{code}
@Test
public void tttt() throws Exception {
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
Object ooo = new String("2");
ugi.getSubject().getPrivateCredentials().add(new String("1"));
ugi.getSubject().getPrivateCredentials().add(ooo);
ugi.getSubject().getPrivateCredentials().add(new String("3"));
ugi.getSubject().getPrivateCredentials().add(new String("4"));
Set<Object> creds = ugi.getSubject().getPrivateCredentials();
Iterator<Object> iter = creds.iterator();
// either this
creds.remove(ooo);
for (; iter.hasNext(); ) {
iter.next();
}
// or this
for (; iter.hasNext(); ) {
if (iter.next() == ooo) {
creds.add(new String ("5"));
}
}
}
{code}
And for this reason [Subject makes sure the iterators are always protected by
the set object
lock|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/javax/security/auth/Subject.java#1356].
Don't have a solid theory on what threads would run into this though. Clearly
this preexists before HADOOP-9747, and don't see offenders...
> ConcurrentModificationException on Subject.getPrivateCredentials in UGI
> constructor
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-15401
> URL: https://issues.apache.org/jira/browse/HADOOP-15401
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
> Affects Versions: 3.1.0, 3.0.3
> Reporter: Xiao Chen
> Priority: Major
>
> Seen a recent exception from KMS client provider as follows:
> {noformat}
> java.io.IOException: java.util.ConcurrentModificationException
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:488)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
> Caused by: java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070)
> at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399)
> at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1372)
> at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767)
> at
> org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(KerberosUtil.java:267)
> at
> org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:715)
> at
> org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:701)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:742)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:141)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471)
> ... 12 more
> {noformat}
> It looks like we have ran into a race modifying jdk Subject class'
> privCredentials.
> Found [https://bugs.openjdk.java.net/browse/JDK-4892913] but that jira was
> created before Hadoop....
> [~daryn], any thoughts on this?
> (We have not seen this in versions pre-3.0 yet, but it seems HADOOP-9747
> would make solve this exact exception because it removed the access in UGI
> constructor)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
