[
https://issues.apache.org/jira/browse/HBASE-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bozhen Liu updated HBASE-24374:
-------------------------------
Description:
We run our static race detector on IntegrationTestRegionReplicaReplication, and
discovered that multiple threads (type of WorkerThread) can invoke the method
getKeyProvider() in class Encryption and read/write to keyProviderCache (type
of ConcurrentHashMap) concurrently without proper lock protection. The related
code is attached below:
{code:java}
public final class Encryption {
...
static final Map<Pair<String,String>,KeyProvider> keyProviderCache = new
ConcurrentHashMap<>();
...
public static KeyProvider getKeyProvider(Configuration conf) {
String providerClassName = conf.get(HConstants.CRYPTO_KEYPROVIDER_CONF_KEY,
KeyStoreKeyProvider.class.getName());
String providerParameters =
conf.get(HConstants.CRYPTO_KEYPROVIDER_PARAMETERS_KEY, "");
try {
Pair<String,String> providerCacheKey = new Pair<>(providerClassName,
providerParameters);
KeyProvider provider = keyProviderCache.get(providerCacheKey);
if (provider != null) {
return provider;
}
provider = (KeyProvider) ReflectionUtils.newInstance(
getClassLoaderForClass(KeyProvider.class).loadClass(providerClassName),
conf);
provider.init(providerParameters);
if (LOG.isDebugEnabled()) {
LOG.debug("Installed " + providerClassName + " into key provider
cache");
}
keyProviderCache.put(providerCacheKey, provider);
return provider;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
{code}
It is possible that multiple threads do the null check on provider, all of
which return false, and each thread instantiated an instance of KeyProvider.
However, only one provider will be added to the map and the other one is
leaking.
The possible traces of the race are shown below, where 3863 and 3864 are
threads created by a loop:
=> Race: org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map
(org/apache/hadoop/hbase/io/crypto/Encryption:553 (TID: 3863) ,
org/apache/hadoop/hbase/io/crypto/Encryption:542 (TID: 3864) )
Trace 1st node: (TID: 3863)
-> Call org/apache/hadoop/util/ToolRunner.run from
org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.main (line 224)
-> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from
org/apache/hadoop/util/ToolRunner.run (line 76)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from
org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables from
org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 589)
*> Thread (3863) created by
Application,Lorg/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables (line
880)
-> Call org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run from
java/lang/Thread.start (line -1)
-> Call org/apache/hadoop/util/ToolRunner.run from
org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run (line 931)
-> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from
org/apache/hadoop/util/ToolRunner.run (line 76)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from
org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.loadTable from
org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 591)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.initTestTable from
org/apache/hadoop/hbase/util/LoadTestTool.loadTable (line 605)
-> Call
org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions from
org/apache/hadoop/hbase/util/LoadTestTool.initTestTable (line 580)
-> Call org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey
from org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions (line
300)
-> Call
org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey from
org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey (line 107)
-> Call
org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject from
org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey (line 448)
-> Call
org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider from
org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject (line 424)
=> Write to
org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map(keyProviderCache) in
org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider (line 553)
Trace 2nd node: (TID: 3864)
-> Call org/apache/hadoop/util/ToolRunner.run from
org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.main (line 224)
-> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from
org/apache/hadoop/util/ToolRunner.run (line 76)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from
org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables from
org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 589)
*> Thread (3864) created by
Application,Lorg/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables (line
880)
-> Call org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run from
java/lang/Thread.start (line -1)
-> Call org/apache/hadoop/util/ToolRunner.run from
org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run (line 931)
-> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from
org/apache/hadoop/util/ToolRunner.run (line 76)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from
org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.loadTable from
org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 591)
-> Call org/apache/hadoop/hbase/util/LoadTestTool.initTestTable from
org/apache/hadoop/hbase/util/LoadTestTool.loadTable (line 605)
-> Call
org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions from
org/apache/hadoop/hbase/util/LoadTestTool.initTestTable (line 580)
-> Call org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey
from org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions (line
300)
-> Call
org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey from
org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey (line 107)
-> Call
org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject from
org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey (line 448)
-> Call
org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider from
org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject (line 424)
=> Write to
org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map(keyProviderCache) in
org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider (line 542)
> Potential Race in class org.apache.hadoop.hbase.io.crypto.Encryption
> --------------------------------------------------------------------
>
> Key: HBASE-24374
> URL: https://issues.apache.org/jira/browse/HBASE-24374
> Project: HBase
> Issue Type: Bug
> Reporter: Bozhen Liu
> Priority: Minor
>
> We run our static race detector on IntegrationTestRegionReplicaReplication,
> and discovered that multiple threads (type of WorkerThread) can invoke the
> method getKeyProvider() in class Encryption and read/write to
> keyProviderCache (type of ConcurrentHashMap) concurrently without proper lock
> protection. The related code is attached below:
> {code:java}
> public final class Encryption {
> ...
> static final Map<Pair<String,String>,KeyProvider> keyProviderCache = new
> ConcurrentHashMap<>();
> ...
> public static KeyProvider getKeyProvider(Configuration conf) {
> String providerClassName =
> conf.get(HConstants.CRYPTO_KEYPROVIDER_CONF_KEY,
> KeyStoreKeyProvider.class.getName());
> String providerParameters =
> conf.get(HConstants.CRYPTO_KEYPROVIDER_PARAMETERS_KEY, "");
> try {
> Pair<String,String> providerCacheKey = new Pair<>(providerClassName,
> providerParameters);
> KeyProvider provider = keyProviderCache.get(providerCacheKey);
> if (provider != null) {
> return provider;
> }
> provider = (KeyProvider) ReflectionUtils.newInstance(
>
> getClassLoaderForClass(KeyProvider.class).loadClass(providerClassName),
> conf);
> provider.init(providerParameters);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Installed " + providerClassName + " into key provider
> cache");
> }
> keyProviderCache.put(providerCacheKey, provider);
> return provider;
> } catch (Exception e) {
> throw new RuntimeException(e);
> }
> }
> }
> {code}
> It is possible that multiple threads do the null check on provider, all of
> which return false, and each thread instantiated an instance of KeyProvider.
> However, only one provider will be added to the map and the other one is
> leaking.
> The possible traces of the race are shown below, where 3863 and 3864 are
> threads created by a loop:
> => Race: org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map
> (org/apache/hadoop/hbase/io/crypto/Encryption:553 (TID: 3863) ,
> org/apache/hadoop/hbase/io/crypto/Encryption:542 (TID: 3864) )
> Trace 1st node: (TID: 3863)
> -> Call org/apache/hadoop/util/ToolRunner.run from
> org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.main (line
> 224)
> -> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from
> org/apache/hadoop/util/ToolRunner.run (line 76)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from
> org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables from
> org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 589)
> *> Thread (3863) created by
> Application,Lorg/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables
> (line 880)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run from
> java/lang/Thread.start (line -1)
> -> Call org/apache/hadoop/util/ToolRunner.run from
> org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run (line 931)
> -> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from
> org/apache/hadoop/util/ToolRunner.run (line 76)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from
> org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.loadTable from
> org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 591)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.initTestTable
> from org/apache/hadoop/hbase/util/LoadTestTool.loadTable (line 605)
> -> Call
> org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions from
> org/apache/hadoop/hbase/util/LoadTestTool.initTestTable (line 580)
> -> Call org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey
> from org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions (line
> 300)
> -> Call
> org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey from
> org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey (line 107)
> -> Call
> org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject from
> org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey (line 448)
> -> Call
> org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider from
> org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject (line 424)
> => Write to
> org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map(keyProviderCache)
> in org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider (line 553)
> Trace 2nd node: (TID: 3864)
> -> Call org/apache/hadoop/util/ToolRunner.run from
> org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.main (line
> 224)
> -> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from
> org/apache/hadoop/util/ToolRunner.run (line 76)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from
> org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables from
> org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 589)
> *> Thread (3864) created by
> Application,Lorg/apache/hadoop/hbase/util/LoadTestTool.parallelLoadTables
> (line 880)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run from
> java/lang/Thread.start (line -1)
> -> Call org/apache/hadoop/util/ToolRunner.run from
> org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.run (line 931)
> -> Call org/apache/hadoop/hbase/util/AbstractHBaseTool.run from
> org/apache/hadoop/util/ToolRunner.run (line 76)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.doWork from
> org/apache/hadoop/hbase/util/AbstractHBaseTool.run (line 154)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.loadTable from
> org/apache/hadoop/hbase/util/LoadTestTool.doWork (line 591)
> -> Call org/apache/hadoop/hbase/util/LoadTestTool.initTestTable
> from org/apache/hadoop/hbase/util/LoadTestTool.loadTable (line 605)
> -> Call
> org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions from
> org/apache/hadoop/hbase/util/LoadTestTool.initTestTable (line 580)
> -> Call org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey
> from org/apache/hadoop/hbase/util/LoadTestTool.applyColumnFamilyOptions (line
> 300)
> -> Call
> org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey from
> org/apache/hadoop/hbase/security/EncryptionUtil.wrapKey (line 107)
> -> Call
> org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject from
> org/apache/hadoop/hbase/io/crypto/Encryption.encryptWithSubjectKey (line 448)
> -> Call
> org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider from
> org/apache/hadoop/hbase/io/crypto/Encryption.getSecretKeyForSubject (line 424)
> => Write to
> org/apache/hadoop/hbase/io/crypto/Encryption.java/util/Map(keyProviderCache)
> in org/apache/hadoop/hbase/io/crypto/Encryption.getKeyProvider (line 542)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
