[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Attachment: HBASE-16643_8.patch One more addition to patch. Added a closed flag in case MemstoreScanner is closed for the second time. > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch, HBASE-16643_6.patch, HBASE-16643_7.patch, > HBASE-16643_8.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Open (was: Patch Available) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch, HBASE-16643_6.patch, HBASE-16643_7.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Patch Available (was: Open) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch, HBASE-16643_6.patch, HBASE-16643_7.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Attachment: HBASE-16643_7.patch Updated patch addresses the comments. And now close() will close the scanners individually if heap is null. > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch, HBASE-16643_6.patch, HBASE-16643_7.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Open (was: Patch Available) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch, HBASE-16643_6.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Attachment: HBASE-16643_6.patch Updated patch correcting the test case failures. Every where we are now supposed to do the seek before we do next() call. The hbase core does it that way only but test cases were not doing it. So we may have to update such test cases now. > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch, HBASE-16643_6.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Patch Available (was: Open) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch, HBASE-16643_6.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
回复:[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
String msg = "Results sent from server=" + noOfResults + ". But only got " + i+ " results completely at client. Resetting the scanner to scan again."; LOG.error(msg); throw new DoNotRetryIOException(msg);} } catch (IOException ioe) { // We are getting IOE while retrieving the cells for Results. // We have to scan for the same results again. Throwing DNRIOE as a client retry on the// same scanner will result in OutOfOrderScannerNextExceptionLOG.error("Exception while reading cells from result." + "Resetting the scanner to scan again.", ioe);throw new DoNotRetryIOException("Resetting the scanner.", ioe); } cells.add(cellScanner.current()); System.out.println("ResponseConverter.getResults(1)--cells == "+ cells + "--j == " + j + "---noOfCells == " + noOfCells);}results[i] = Result.create(cells, null, response.getStale(), isPartial); System.out.println("ResponseConverter.getResults(2)--cells == "+ cells + "--results[i] == " + results[i] + "---noOfCells == " + noOfCells); } else {// Result is pure pb.results[i] = ProtobufUtil.toResult(response.getResults(i)); System.out.println("ResponseConverter.getResults(3)--results == " + results[i] + "--i == " + i); }}return results; thank you very much, anoop, do you want to see this code, the exception should occur on the red mark line. - 原始邮件 - 发件人:"ramkrishna.s.vasudevan (JIRA)"收件人:issues@hbase.apache.org 主题:[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments 日期:2016年09月23日 13点25分 [ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Open (was: Patch Available) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Open (was: Patch Available) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Attachment: HBASE-16643_5.patch Updated patch based on RB comments. I have left some replies in the RB for all the comments and questions. The SegmentScanner change is needed is what I believe. If that is not needed in this JIRA I can remove it and raise another JIRA for discussion. > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch, > HBASE-16643_5.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Open (was: Patch Available) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Attachment: HBASE-16643_4.patch Updated RB with this version. > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Patch Available (was: Open) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Attachment: HBASE-16643_3.patch Now we will initialize the heap in memstore scanner lazily depending on what type of scanner API is called on it for the first time. Removes synchronized from all the methods as it was mentioned in the earlier review comment. The patch becomes smaller because no test cases were changed and no interface is updated though I would like to update the Store#getScanners() API and unify them. Not sure how to do that if we cannot break without deprecating. So I have deferred it to another JIRA. Still I prefer changing the Store interface as it is much cleaner and passing the reverse scan info to the getScanners() API in some form is useful and cleaner. Will submit for QA. > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch, HBASE-16643_3.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Open (was: Patch Available) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Patch Available (was: Open) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Attachment: HBASE-16643_2.patch REmoves the assertion change. It was my bad. Locally no problem as assertion was not enabled so could not spot it. > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch, > HBASE-16643_2.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Open (was: Patch Available) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Patch Available (was: Open) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Attachment: HBASE-16643_1.patch Updated patch. The failed test case seems to pass. Will check once again with QA. > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch, HBASE-16643_1.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Status: Open (was: Patch Available) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Summary: Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments (was: Reverse scanner heap creation may not allow MSLAB closure due to inproper ref counting of segments) > Reverse scanner heap creation may not allow MSLAB closure due to improper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16643.patch > > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)