[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433311#comment-15433311
]
Hudson commented on HBASE-16440:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1468 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1468/])
HBASE-16440 MemstoreChunkPool might cross its maxCount of chunks to
(anoopsamjohn: rev 897631f8d194911e0004b9c98d4040e9794bb2e9)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java
* (add)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Chunk.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HeapMemStoreLAB.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16440.patch, HBASE-16440_V2.patch,
> HBASE-16440_V3.patch, HBASE-16440_V4.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432694#comment-15432694
]
ramkrishna.s.vasudevan commented on HBASE-16440:
Ok +1.
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16440.patch, HBASE-16440_V2.patch,
> HBASE-16440_V3.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432483#comment-15432483
]
Anoop Sam John commented on HBASE-16440:
Ok I was just trying to clean here and there and change for the reporting
thread start went in the patch. Will remove.
Its not about multi threaded single memstore flush. It is about concurrent
MSLAB close because of diff memstore flushes. All use same MSLAB pool. So more
than one thread can call putBack API with a list of chunks concurrently. Also
before the patch, u can see that the getChunk will return a new chunk even if
the pool reached its max size and no free chunks as of now. The max count
enforcement we tried to do during putBack APIs. But as that is not thread safe
we may cross.
Now the major diff is the getChunk API enforces the max capacity. It will
return chunks which can get pooled. Else it wl return null. Means pool says to
callers that there are no pooled/poolable chunks from this pool. The atomic
int was already there. That was used for reporting alone but now that is key
in deciding whether we can create new PooleChunk or not.
bq.The reclaimedChunks blockingqueue can be initialized with max Size ==
maxcount. Currently it is with no max Size so it will go upto INTEGER.MAX_VALUE.
As per current status ya we can change. But see HBASE-16407. When we do this,
the max capacity itself is changeable. So I will keep this as is.
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16440.patch, HBASE-16440_V2.patch,
> HBASE-16440_V3.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432433#comment-15432433
]
ramkrishna.s.vasudevan commented on HBASE-16440:
Went through the patch.
In the default memstore case - I think we will clear the snapshot and call
close and it is always done by one thread only? But once we go with Compacting
memstore we will have the multi threaded case I think.
The patch now creates a Atomic Integer and that decides if we have reached the
maxCount. And putBackChunks() uses the add() method only.
Even if multi threads calls the putBackChunks - it will still be able to drain
what ever is there in the Chunkqueue right? Just asking. Your change is fine
anyway because it is making things much better so we are sure we don't cross
maxCount.
Regarding removing the thread pool creation if log is not in debug mode - I
think it is better not to change it. In UI there is a provision to change LOG
level anytime. So suppose for debugging purpose if some one changes the log
level then since the thread pool itself is not started we wont be able to log
any. I think better not to change it. May be default we can allow only one
thread to be active here?
One more suggestion - may be better to do it here -
The reclaimedChunks blockingqueue can be initialized with max Size == maxcount.
Currently it is with no max Size so it will go upto INTEGER.MAX_VALUE.
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16440.patch, HBASE-16440_V2.patch,
> HBASE-16440_V3.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427681#comment-15427681
]
Anoop Sam John commented on HBASE-16440:
Ping [~saint@gmail.com]. What do you think abt the last patch boss? I have
added test also.
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16440.patch, HBASE-16440_V2.patch,
> HBASE-16440_V3.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426844#comment-15426844
]
Anoop Sam John commented on HBASE-16440:
Failed test seems an issue with some recent commit (?)
{code}
org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush Time
elapsed: 526.451 sec <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 10 minutes
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.waitForRead(MultiVersionConcurrencyControl.java:218)
at
org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:149)
at
org.apache.hadoop.hbase.regionserver.HRegion.getNextSequenceId(HRegion.java:2604)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2321)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2216)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2188)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2178)
at
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1484)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1412)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1362)
at
org.apache.hadoop.hbase.HBaseTestingUtility.closeRegionAndWAL(HBaseTestingUtility.java:369)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testFlushCacheWhileScanning(TestHRegion.java:3820)
{code}
Saw the same with HBASE-16405 QA results aslo
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16440.patch, HBASE-16440_V2.patch,
> HBASE-16440_V3.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426501#comment-15426501
]
Anoop Sam John commented on HBASE-16440:
bq. You don't want to subclass? Seems same except for isPooled.
Ya am also thinking now ,rather than a boolean which says pooled or not, we can
extend Chunk for a pooled type. This can make more strict type enforcement in
MemstoreChunkPool APIs than boolean check. Let me explain in a patch. Coming
soon.
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16440.patch, HBASE-16440_V2.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426162#comment-15426162
]
Hadoop QA commented on HBASE-16440:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 37m 58s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
40m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 37s
{color} | {color:red} hbase-server-jdk1.8.0_101 with JDK v1.8.0_101 generated 3
new + 1 unchanged - 0 fixed = 4 total (was 1) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 46s
{color} | {color:red} hbase-server-jdk1.7.0_111 with JDK v1.7.0_111 generated 3
new + 1 unchanged - 0 fixed = 4 total (was 1) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 115m 22s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
57s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 230m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | hadoop.hbase.replication.TestMasterReplication |
| | hadoop.hbase.client.TestAdmin1 |
| Timed out junit tests | org.apache.hadoop.hbase.coprocessor.TestClassLoading |
| |
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-18 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824286/HBASE-16440_V2.patch |
| JIRA Issue | HBASE-16440 |
|
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425860#comment-15425860
]
Anoop Sam John commented on HBASE-16440:
I just moved the Chunk class outside and renamed it.. OK I will keep the name
Chunk. Just addition is the isPooled() method.
bq.In getChunk, you have three returns throughout the method. You don't want to
just have one on the end?
Ok I can change.
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-16440.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425422#comment-15425422
]
stack commented on HBASE-16440:
---
MemstoreLABChunk has a lot of Chunk in it. You don't want to subclass? Seems
same except for isPooled. You don't want to add support for this to the base
Chunk class? Chunk is a nice name (smile). The method to get one of these
things is getChunk.
In getChunk, you have three returns throughout the method. You don't want to
just have one on the end?
Otherwise patch looks good. Any chance of a test that can repro the scenario
you describe above?
Nice [~anoop.hbase]
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-16440.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
