Sorry, something important I missed in my last email.
SoftValueSortedMap uses TreeMap. It has some synchronized methods to edit the
data from this map. If we use these methods, it’s ok. But in
HConnectionManager#getCachedLocation:
// Cut the cache so that we only get the part that could contain
// regions that match our key
SoftValueSortedMap<byte[], HRegionLocation> matchingRegions =
tableLocations.headMap(row);
// if that portion of the map is empty, then we're done. otherwise,
// we need to examine the cached location to verify that it is
// a match by end key as well.
if (!matchingRegions.isEmpty()) {
HRegionLocation possibleRegion = null;
try {
possibleRegion = matchingRegions.get(matchingRegions.lastKey());
} catch (NoSuchElementException nsee) {
LOG.warn("checkReferences() might have removed the key", nsee);
}
SoftValueSortedMap#get:
public synchronized V get(Object key) {
checkReferences();
SoftValue<K,V> value = this.internalMap.get(key);
if (value == null) {
return null;
}
if (value.get() == null) {
this.internalMap.remove(key);
return null;
}
return value.get();
}
In this method, it can remove the element from internalMap which is a heapMap
from the backed TreeMap. A concurrency issue may happen from here.
Correct me if am wrong.
Thank you.
Jieshan.
发件人: bijieshan
发送时间: 2011年12月21日 18:48
收件人: [email protected]; [email protected]
抄送: wenzaohua; Chenjian
主题: A concurrency issue on SoftValueSortedMap?
Hi all,
We use thousand of threads doing the scan operations. One thread got blocked at
the position of “java.util.TreeMap.fixAfterDeletion”. It can’t come out of the
loop of “TreeMap.fixAfterDeletion”:
So I think it maybe a concurrency issue. Has someone encountered this?
Thank you.
Jieshan.
Here’s the thread dump:
"Thread-923" prio=10 tid=0x00007f3d40553000 nid=0x3ed6 runnable
[0x00007f3d05c1b000]
java.lang.Thread.State: RUNNABLE
at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2176)
at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
at java.util.TreeMap.remove(TreeMap.java:585)
at java.util.TreeMap$NavigableSubMap.remove(TreeMap.java:1395)
at
org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:101)
- locked <0x00007f3d92937748> (a
org.apache.hadoop.hbase.util.SoftValueSortedMap)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:846)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:668)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
at
org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
at
org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1018)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
at
org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
at
com.huawei.icbc.query.SingleTabQuery.querybatch(SingleTabQuery.java:197)
at
com.huawei.icbc.benchmark.SingleTabQueryAction.query(SingleTabQueryAction.java:181)
at framework.QueryThread.run(QueryThread.java:47)
"Thread-923" prio=10 tid=0x00007f3d40553000 nid=0x3ed6 runnable
[0x00007f3d05c1a000]
java.lang.Thread.State: RUNNABLE
at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
at java.util.TreeMap.remove(TreeMap.java:585)
at java.util.TreeMap$NavigableSubMap.remove(TreeMap.java:1395)
at
org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:101)
- locked <0x00007f3d94f24f70> (a
org.apache.hadoop.hbase.util.SoftValueSortedMap)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:846)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:668)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
at
org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
at
org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1018)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
at
org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
at
com.huawei.icbc.query.SingleTabQuery.querybatch(SingleTabQuery.java:197)
at
com.huawei.icbc.benchmark.SingleTabQueryAction.query(SingleTabQueryAction.java:181)
at framework.QueryThread.run(QueryThread.java:47)
