[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171769#comment-15171769 ] Anoop Sam John commented on HBASE-14099: [~appy] As this was fixed and long time back, we can raise ur observation as a new bug and close this one? > StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start > and stop Row > --- > > Key: HBASE-14099 > URL: https://issues.apache.org/jira/browse/HBASE-14099 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14099.patch, HBASE-14099_1.patch, > HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, > hfiles_and_testclass.zip, storefile.png > > > During profiling saw that the code here in passesKeyRangeFilter in Storefile > {code} > KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil > .createFirstOnRow(scan.getStopRow()) : > KeyValueUtil.createFirstOnRow(scan > .getStartRow()); > KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil > .createLastOnRow(scan.getStartRow()) : > KeyValueUtil.createLastOnRow(scan > .getStopRow()); > {code} > This row need not be copied now considering that we have > CellComparator.compareRows(Cell, byte[]). > We have already refactored the firstKeyKv and lastKeyKV as part of other > JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158364#comment-15158364 ] Anoop Sam John commented on HBASE-14099: Need to see all other Fake cell usages.. We have SettableTimestamp also but that is needed only in write path I believe. Ya to be on safe side make top level itself as SettableSequenceId, I would say > StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start > and stop Row > --- > > Key: HBASE-14099 > URL: https://issues.apache.org/jira/browse/HBASE-14099 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14099.patch, HBASE-14099_1.patch, > HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, > hfiles_and_testclass.zip, storefile.png > > > During profiling saw that the code here in passesKeyRangeFilter in Storefile > {code} > KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil > .createFirstOnRow(scan.getStopRow()) : > KeyValueUtil.createFirstOnRow(scan > .getStartRow()); > KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil > .createLastOnRow(scan.getStartRow()) : > KeyValueUtil.createLastOnRow(scan > .getStopRow()); > {code} > This row need not be copied now considering that we have > CellComparator.compareRows(Cell, byte[]). > We have already refactored the firstKeyKv and lastKeyKV as part of other > JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158358#comment-15158358 ] ramkrishna.s.vasudevan commented on HBASE-14099: May be make 'EmptyCell' also as SettableSequenceId type? > StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start > and stop Row > --- > > Key: HBASE-14099 > URL: https://issues.apache.org/jira/browse/HBASE-14099 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14099.patch, HBASE-14099_1.patch, > HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, > hfiles_and_testclass.zip, storefile.png > > > During profiling saw that the code here in passesKeyRangeFilter in Storefile > {code} > KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil > .createFirstOnRow(scan.getStopRow()) : > KeyValueUtil.createFirstOnRow(scan > .getStartRow()); > KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil > .createLastOnRow(scan.getStartRow()) : > KeyValueUtil.createLastOnRow(scan > .getStopRow()); > {code} > This row need not be copied now considering that we have > CellComparator.compareRows(Cell, byte[]). > We have already refactored the firstKeyKv and lastKeyKV as part of other > JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158344#comment-15158344 ] Anoop Sam John commented on HBASE-14099: Ya this will come for scanners on a bulk loaded file only.. That is why no tests found it. We need to make some of the Fake cells at least as SettableSequenceId type. (Or all to be on safe side). This went in only in 2.0 so an addendum to fix the issue is fine. (No need for new issue).. > StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start > and stop Row > --- > > Key: HBASE-14099 > URL: https://issues.apache.org/jira/browse/HBASE-14099 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14099.patch, HBASE-14099_1.patch, > HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, > hfiles_and_testclass.zip, storefile.png > > > During profiling saw that the code here in passesKeyRangeFilter in Storefile > {code} > KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil > .createFirstOnRow(scan.getStopRow()) : > KeyValueUtil.createFirstOnRow(scan > .getStartRow()); > KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil > .createLastOnRow(scan.getStartRow()) : > KeyValueUtil.createLastOnRow(scan > .getStopRow()); > {code} > This row need not be copied now considering that we have > CellComparator.compareRows(Cell, byte[]). > We have already refactored the firstKeyKv and lastKeyKV as part of other > JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158314#comment-15158314 ] Appy commented on HBASE-14099: -- I was doing some testing when I hit a weird issue, seems related to this, so re-opening it (apologies in advance if it's not). Here's the stack trace {noformat} java.io.IOException: java.lang.UnsupportedOperationException: Cell is not of type org.apache.hadoop.hbase.SettableSequenceId at org.apache.hadoop.hbase.CellUtil.setSequenceId(CellUtil.java:923) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.setCurrentCell(StoreFileScanner.java:231) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.requestSeek(StoreFileScanner.java:389) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:348) at org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:212) at org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:1873) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1863) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5487) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2577) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2563) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2544) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2534) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6659) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6624) at org.apache.hadoop.hbase.regionserver.TestWithSingleHRegion.test(TestWithSingleHRegion.java:48) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) {noformat} I think it's because of using changing from KeyValue to a different sub-class of {{Cell}}l which doesn't implement {{SettableSequenceId}} {noformat} -this.startKey = KeyValueUtil.createFirstDeleteFamilyOnRow(scan.getStartRow(), +this.startKey = CellUtil.createFirstDeleteFamilyCellOnRow(scan.getStartRow(), {noformat} To replicate it, download the attached hfiles somewhere, copy the TestWithSingleHRegion class to regionserver tests, change the ROOT_DIR appropriately and run it. > StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start > and stop Row > --- > > Key: HBASE-14099 > URL: https://issues.apache.org/jira/browse/HBASE-14099 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14099.patch, HBASE-14099_1.patch, > HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, storefile.png > > > During profiling saw that the code here in passesKeyRangeFilter in Storefile > {code} > KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil >
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634782#comment-14634782 ] Lars Hofhansl commented on HBASE-14099: --- Thanks! Yes it does. StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633446#comment-14633446 ] Anoop Sam John commented on HBASE-14099: {code} || compareLastKey(lastKeyKV, smallestScanRow) 0; return !nonOverLapping; } protected int compareLastKey(Cell lastKeyKV, byte[] scanRow) { int diff = getComparator().compareRows(lastKeyKV, scanRow, 0, scanRow.length); if (diff != 0) { return diff; } // if the rows does not match then atleast the family or qualifier will make the // lastKeyKV greater. There should not be a need to compare the ts return 1; } {code} Do you really need this extra logic for the compare? Now we are only comparing the rks. Only when the smallestScanRow is greater than the lastKeyKV we have to set the boolean. Means when both are same (return of compare is 0) we dont want to set it. And that is what the simple compare op also will do. So getComparator().compareRows(lastKeyKV, smallestScanRow, 0, smallestScanRow.length)0; whould be enough StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 2.0.0 Attachments: HBASE-14099.patch During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633465#comment-14633465 ] ramkrishna.s.vasudevan commented on HBASE-14099: Let me check if I can reduce the createFirstDeleteFamilyOnRow also. I forgot about that. StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633563#comment-14633563 ] Anoop Sam John commented on HBASE-14099: +1. StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633595#comment-14633595 ] ramkrishna.s.vasudevan commented on HBASE-14099: Will commit after QA run. StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633666#comment-14633666 ] Hadoop QA commented on HBASE-14099: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12746092/HBASE-14099_2.patch against master branch at commit 88038cf473a17a6b059902e080f1bc10d338b7c9. ATTACHMENT ID: 12746092 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests. {color:green}+1 hadoop versions{color}. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0) {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 protoc{color}. The applied patch does not increase the total number of protoc compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1 warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn post-site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: {color:red}-1 core zombie tests{color}. There are 1 zombie test(s): at org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWriteInternals(TestCacheOnWrite.java:270) at org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWrite(TestCacheOnWrite.java:473) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/14836//testReport/ Release Findbugs (version 2.0.3)warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14836//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/14836//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14836//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/14836//console This message is automatically generated. StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633668#comment-14633668 ] Hadoop QA commented on HBASE-14099: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12746086/HBASE-14099_1.patch against master branch at commit 88038cf473a17a6b059902e080f1bc10d338b7c9. ATTACHMENT ID: 12746086 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 hadoop versions{color}. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0) {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 protoc{color}. The applied patch does not increase the total number of protoc compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn post-site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/14835//testReport/ Release Findbugs (version 2.0.3)warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14835//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/14835//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/14835//console This message is automatically generated. StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633949#comment-14633949 ] Hadoop QA commented on HBASE-14099: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12746130/HBASE-14099_4.patch against master branch at commit 88038cf473a17a6b059902e080f1bc10d338b7c9. ATTACHMENT ID: 12746130 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests. {color:green}+1 hadoop versions{color}. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0) {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 protoc{color}. The applied patch does not increase the total number of protoc compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn post-site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/14838//testReport/ Release Findbugs (version 2.0.3)warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14838//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/14838//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/14838//console This message is automatically generated. StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633978#comment-14633978 ] Lars Hofhansl commented on HBASE-14099: --- Are you sure this 100% correct? First/LastKeyvalue also have the proper timestamp and code to sort before/after _any_ KeyValue. Just comparing the row key does not do the same. It might be correct in these cases... But are you _absolutely_ sure? What if the first Cell in a StoreFile is a delete marker? StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633990#comment-14633990 ] Hudson commented on HBASE-14099: FAILURE: Integrated in HBase-TRUNK #6665 (See [https://builds.apache.org/job/HBase-TRUNK/6665/]) HBASE-14099 StoreFile.passesKeyRangeFilter need not create Cells from the (ramkrishna: rev 7e4cd5982071540fdb75698a0278aac391ccc540) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTreeEncoding.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633684#comment-14633684 ] Hadoop QA commented on HBASE-14099: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12746100/HBASE-14099_3.patch against master branch at commit 88038cf473a17a6b059902e080f1bc10d338b7c9. ATTACHMENT ID: 12746100 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests. {color:green}+1 hadoop versions{color}. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0) {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 protoc{color}. The applied patch does not increase the total number of protoc compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1 warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn post-site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestFastFail Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/14837//testReport/ Release Findbugs (version 2.0.3)warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14837//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/14837//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14837//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/14837//console This message is automatically generated. StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634472#comment-14634472 ] Anoop Sam John commented on HBASE-14099: bq.What if the first Cell in a StoreFile is a delete marker? When first cell in store file is delete marker for the given rowkey (largestScanRow) , then we can NOT avoid this file and have to read. So the return of this method has to be true. Then the check getComparator().compareRows(firstKeyKV, largestScanRow, 0, largestScanRow.length) 0 will be falseand getComparator().compareRows(lastKeyKV, smallestScanRow, 0, smallestScanRow.length) 0 also will be false for sure. This wil make the boolean false and we return true. StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634547#comment-14634547 ] ramkrishna.s.vasudevan commented on HBASE-14099: Before patch it was, {code} boolean nonOverLapping = ((getComparator().compare(firstKeyKV, largestScanKeyValue)) 0 {code} Here {code} KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil 1369 .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan 1370 .getStopRow()); {code} So this is going to have the type as Minimum. So at any cost once the row part is not going to same we are going to return -1 from the comparator because {code} if (rFamLength + rQualLength == 0 right.getTypeByte() == Type.Minimum.getCode()) { return -1; } {code} The KV is not going to have any family or qualifier with it. So once the row is not going to match then we always take the first Cell as lesser only. Coming to this {code} (getComparator().compare(lastKeyKV, smallestScanKeyValue)) 0; {code} Though the smallest key value is going to have type as MAXIMUM but there is no family or qual associated with it. So at any point if the rows are not matching then we will always be returning a value 0 and thus failing the above condition. If the formed fake key is going to have a family or qual then it is different. But here we just use the row along with type and ts. Does it make sense now Lars? StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Improvement Components: Performance, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-14099.patch, HBASE-14099_1.patch, HBASE-14099_2.patch, HBASE-14099_3.patch, HBASE-14099_4.patch, storefile.png During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[ https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632599#comment-14632599 ] Hadoop QA commented on HBASE-14099: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12745965/HBASE-14099.patch against master branch at commit 338e39970ba8e4835733669b9252d073b2157b8a. ATTACHMENT ID: 12745965 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 hadoop versions{color}. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0) {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 protoc{color}. The applied patch does not increase the total number of protoc compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn post-site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/14828//testReport/ Release Findbugs (version 2.0.3)warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14828//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/14828//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/14828//console This message is automatically generated. StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row --- Key: HBASE-14099 URL: https://issues.apache.org/jira/browse/HBASE-14099 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 2.0.0 Attachments: HBASE-14099.patch During profiling saw that the code here in passesKeyRangeFilter in Storefile {code} KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil .createFirstOnRow(scan.getStopRow()) : KeyValueUtil.createFirstOnRow(scan .getStartRow()); KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil .createLastOnRow(scan.getStartRow()) : KeyValueUtil.createLastOnRow(scan .getStopRow()); {code} This row need not be copied now considering that we have CellComparator.compareRows(Cell, byte[]). We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)