[
https://issues.apache.org/jira/browse/HADOOP-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446474#comment-16446474
]
Xiao Chen edited comment on HADOOP-15401 at 4/20/18 10:52 PM:
--------------------------------------------------------------
bq. The private credentials set returned by the subject is a synchronized set,
so a direct remove is safe.
Removing from the set can still screw up the iterator though. See the test
below, which would end up throw ConcurrentModificationException:
{code}
@Test
public void tttt() throws Exception {
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
Object ooo = new String("2");
ugi.getSubject().getPrivateCredentials().add(new String("1"));
ugi.getSubject().getPrivateCredentials().add(ooo);
ugi.getSubject().getPrivateCredentials().add(new String("3"));
ugi.getSubject().getPrivateCredentials().add(new String("4"));
Set<Object> creds = ugi.getSubject().getPrivateCredentials();
Iterator<Object> iter = creds.iterator();
// either this
creds.remove(ooo);
for (; iter.hasNext(); ) {
iter.next();
}
// or this
for (; iter.hasNext(); ) {
if (iter.next() == ooo) {
creds.add(new String ("5"));
}
}
}
{code}
And for this reason [Subject makes sure the iterators are always protected by
the set object
lock|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/javax/security/auth/Subject.java#1356].
I see we're doing the same in UGI most of the places - don't we need the same
protection here? :
{code:title=UGI#getCredentialsInternal}
private synchronized Credentials getCredentialsInternal() {
...
credentials = new Credentials();
subject.getPrivateCredentials().add(credentials); // <--
synchronized(subject.getPrivateCredentials()) { xxx.add() }
{code}
Don't have a solid theory on what threads would run into this though. Clearly
this preexists before HADOOP-9747, but looks to me this should be fixed in
trunk as well.
was (Author: xiaochen):
bq. The private credentials set returned by the subject is a synchronized set,
so a direct remove is safe.
Removing from the set can still screw up the iterator though. See the test
below:
{code}
@Test
public void tttt() throws Exception {
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
Object ooo = new String("2");
ugi.getSubject().getPrivateCredentials().add(new String("1"));
ugi.getSubject().getPrivateCredentials().add(ooo);
ugi.getSubject().getPrivateCredentials().add(new String("3"));
ugi.getSubject().getPrivateCredentials().add(new String("4"));
Set<Object> creds = ugi.getSubject().getPrivateCredentials();
Iterator<Object> iter = creds.iterator();
// either this
creds.remove(ooo);
for (; iter.hasNext(); ) {
iter.next();
}
// or this
for (; iter.hasNext(); ) {
if (iter.next() == ooo) {
creds.add(new String ("5"));
}
}
}
{code}
And for this reason [Subject makes sure the iterators are always protected by
the set object
lock|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/javax/security/auth/Subject.java#1356].
I see we're doing the same in UGI most of the places - don't we need the same
protection here? :
{code:title=UGI#getCredentialsInternal}
private synchronized Credentials getCredentialsInternal() {
...
credentials = new Credentials();
subject.getPrivateCredentials().add(credentials); // <--
synchronized(subject.getPrivateCredentials()) { xxx.add() }
{code}
Don't have a solid theory on what threads would run into this though. Clearly
this preexists before HADOOP-9747, but looks to me this should be fixed in
trunk as well.
> ConcurrentModificationException on Subject.getPrivateCredentials in UGI
> constructor
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-15401
> URL: https://issues.apache.org/jira/browse/HADOOP-15401
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
> Affects Versions: 3.1.0, 3.0.3
> Reporter: Xiao Chen
> Priority: Major
>
> Seen a recent exception from KMS client provider as follows:
> {noformat}
> java.io.IOException: java.util.ConcurrentModificationException
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:488)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
> Caused by: java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070)
> at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399)
> at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1372)
> at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767)
> at
> org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(KerberosUtil.java:267)
> at
> org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:715)
> at
> org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:701)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:742)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:141)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471)
> ... 12 more
> {noformat}
> It looks like we have ran into a race modifying jdk Subject class'
> privCredentials.
> Found [https://bugs.openjdk.java.net/browse/JDK-4892913] but that jira was
> created before Hadoop....
> [~daryn], any thoughts on this?
> (We have not seen this in versions pre-3.0 yet, but it seems HADOOP-9747
> would make solve this exact exception because it removed the access in UGI
> constructor)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
