[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[
https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090432#comment-15090432
]
Ted Yu commented on HBASE-15055:
>From
>https://builds.apache.org/job/PreCommit-HBASE-Build/34/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_66.txt
> :
{code}
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.04 sec <<<
FAILURE! - in
org.apache.hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleWAL
org.apache.hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleWAL
Time elapsed: 0.038 sec <<< ERROR!
java.lang.RuntimeException: Error while running command to get file permissions
: ExitCodeException exitCode=127: /bin/ls: error while loading shared
libraries: libdl.so.2: failed to map segment from shared object: Permission
denied
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
{code}
QA environment issue.
> Major compaction is not triggered when both of TTL and
> hbase.hstore.compaction.max.size are set
> ---
>
> Key: HBASE-15055
> URL: https://issues.apache.org/jira/browse/HBASE-15055
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Assignee: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch,
> HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch,
> HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch,
> HBASE-15055-v9.patch, HBASE-15055.patch
>
>
> Some large files may be skipped by hbase.hstore.compaction.max.size in
> candidate selection. It causes skipping of major compaction. So the TTL
> expired records are still remained in the disks and keep consuming disks.
> To resolve this issue, I suggest that to skip large files only if there is no
> TTL expired record.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15077) Support OffheapKV write in compaction with out copying data on heap
[
https://issues.apache.org/jira/browse/HBASE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090435#comment-15090435
]
Anoop Sam John commented on HBASE-15077:
bq.Also kept our DOS name also as DataOutputStream only. (
ByteBufferSupportedDOS is too big)
No .. have to change. Findbugs warns it.
Will keep the old name and the new interface name as
ByteBufferSupportOutputStream
> Support OffheapKV write in compaction with out copying data on heap
> ---
>
> Key: HBASE-15077
> URL: https://issues.apache.org/jira/browse/HBASE-15077
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15077.patch, HBASE-15077_V2.patch
>
>
> HBASE-14832 is not enough to handle this. Doing the remaining needed here.
> {code}
> if (cell instanceof ByteBufferedCell) {
> 890 out.writeShort(rowLen);
> 891 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getRowByteBuffer(),
> 892 ((ByteBufferedCell) cell).getRowPosition(), rowLen);
> 893 out.writeByte(fLen);
> 894 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getFamilyByteBuffer(),
> 895 ((ByteBufferedCell) cell).getFamilyPosition(), fLen);
> 896 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getQualifierByteBuffer(),
> 897 ((ByteBufferedCell) cell).getQualifierPosition(), qLen);
> {code}
> We have done this but it is not really helping us!
> In ByteBufferUtils#copyBufferToStream
> {code}
> public static void copyBufferToStream(OutputStream out, ByteBuffer in,
> int offset, int length) throws IOException {
> if (in.hasArray()) {
> out.write(in.array(), in.arrayOffset() + offset,
> length);
> } else {
> for (int i = 0; i < length; ++i) {
> out.write(toByte(in, offset + i));
> }
> }
> }
> {code}
> So for DBB it is so costly op writing byte by byte reading each to on heap.
> Even if we use writeByteBuffer(OutputStream out, ByteBuffer b, int offset,
> int length), it won't help us as the underlying stream is a
> ByteArrayOutputStream and so we will end up in copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15077) Support OffheapKV write in compaction with out copying data on heap
[
https://issues.apache.org/jira/browse/HBASE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-15077:
---
Attachment: HBASE-15077_V2.patch
> Support OffheapKV write in compaction with out copying data on heap
> ---
>
> Key: HBASE-15077
> URL: https://issues.apache.org/jira/browse/HBASE-15077
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15077.patch, HBASE-15077_V2.patch,
> HBASE-15077_V2.patch
>
>
> HBASE-14832 is not enough to handle this. Doing the remaining needed here.
> {code}
> if (cell instanceof ByteBufferedCell) {
> 890 out.writeShort(rowLen);
> 891 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getRowByteBuffer(),
> 892 ((ByteBufferedCell) cell).getRowPosition(), rowLen);
> 893 out.writeByte(fLen);
> 894 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getFamilyByteBuffer(),
> 895 ((ByteBufferedCell) cell).getFamilyPosition(), fLen);
> 896 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getQualifierByteBuffer(),
> 897 ((ByteBufferedCell) cell).getQualifierPosition(), qLen);
> {code}
> We have done this but it is not really helping us!
> In ByteBufferUtils#copyBufferToStream
> {code}
> public static void copyBufferToStream(OutputStream out, ByteBuffer in,
> int offset, int length) throws IOException {
> if (in.hasArray()) {
> out.write(in.array(), in.arrayOffset() + offset,
> length);
> } else {
> for (int i = 0; i < length; ++i) {
> out.write(toByte(in, offset + i));
> }
> }
> }
> {code}
> So for DBB it is so costly op writing byte by byte reading each to on heap.
> Even if we use writeByteBuffer(OutputStream out, ByteBuffer b, int offset,
> int length), it won't help us as the underlying stream is a
> ByteArrayOutputStream and so we will end up in copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[
https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090437#comment-15090437
]
Lars Hofhansl commented on HBASE-14213:
---
And home (JDK8, Maven 3.3.9) I still cannot build with this patch applied.
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project
hbase: failed to get report for org.apache.maven.plugins:maven-javadoc-plugin:
Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(build-legal-for-assembly) on project hbase: Unable to parse configuration of
mojo org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process for
parameter resource: Cannot find default setter in class
org.apache.maven.model.Resource -> [Help 1]
{code}
Work (JDK7, Maven 3.0.4) is fine and without patch I could build fine in both
setups.
I can't win with this one it seems :(
> Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in
> artifacts for 0.94
> --
>
> Key: HBASE-14213
> URL: https://issues.apache.org/jira/browse/HBASE-14213
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 0.94.28
>
> Attachments: 14213-LICENSE.txt, 14213-addendum.txt,
> 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh,
> 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch
>
>
> From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took
> the liberty of assigning to [~busbey].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
add thread number conf to tool upgradeTo96
when upgrade from 94 to 98, we can use class upgradeTo96 this tool support check and upgrade tool check tool is class hfileV1 Detector, and support -n to set thrednumber but, class upgradeTo96 filter this parameter
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090338#comment-15090338 ] Hudson commented on HBASE-15065: SUCCESS: Integrated in HBase-1.2 #497 (See [https://builds.apache.org/job/HBase-1.2/497/]) HBASE-15065 SimpleRegionNormalizer should return multiple normalization (tedyu: rev a5438f4a55bc6ff56a1eded712c567141f679600) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerChore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java HBASE-15065 Addendum fixes checkstyle warnings (tedyu: rev 2e08227b4b337e45dfb8282c3c657b95061073c1) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12547) Adopt server side cleanup for bulk load staging directories
[ https://issues.apache.org/jira/browse/HBASE-12547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090351#comment-15090351 ] Ted Yu commented on HBASE-12547: We need some mechanism to track the expiration of leases. Zookeeper is not favored as we're moving away from using zookeeper. > Adopt server side cleanup for bulk load staging directories > --- > > Key: HBASE-12547 > URL: https://issues.apache.org/jira/browse/HBASE-12547 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu > Labels: bulkloader > > Currently we rely on the client to send the clean up request for bulk load > staging directories. > In discussion of HBASE-12533, Enis and I raised similar comment about > cleaning up bulk load staging directories from server side. > We can keep lease on staging directory for bulk load operation. The default > lease period should be long enough. After the lease expires, the underlying > staging directory would be cleaned. > This increases the robustness of bulk load process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15057) local-master-backup.sh doesn't start HMaster correctly
[ https://issues.apache.org/jira/browse/HBASE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090368#comment-15090368 ] Hudson commented on HBASE-15057: SUCCESS: Integrated in HBase-1.3 #489 (See [https://builds.apache.org/job/HBase-1.3/489/]) HBASE-15057 local-master-backup.sh doesn't start HMaster correctly (tedyu: rev f541539da5993148bd31e978d8367e01141ffbf1) * bin/local-master-backup.sh > local-master-backup.sh doesn't start HMaster correctly > -- > > Key: HBASE-15057 > URL: https://issues.apache.org/jira/browse/HBASE-15057 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 1.1.2 > Environment: GNU/Linux x86_64. >Reporter: Cyker Way >Assignee: Samir Ahmic > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15057-v0.patch > > > ./bin/local-master-backup.sh cannot start a backup HMaster. The error in log > is a port conflict. After reading the file it seems an additional option is > required to make it work: > @@ -40,6 +40,7 @@ >DN=$2 >export HBASE_IDENT_STRING="$USER-$DN" >HBASE_MASTER_ARGS="\ > +-D hbase.master.port=`expr 16000 + $DN` \ > -D hbase.master.info.port=`expr 16010 + $DN` \ > -D hbase.regionserver.port=`expr 16020 + $DN` \ > -D hbase.regionserver.info.port=`expr 16030 + $DN` \ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090402#comment-15090402 ] Hudson commented on HBASE-15065: FAILURE: Integrated in HBase-Trunk_matrix #623 (See [https://builds.apache.org/job/HBase-Trunk_matrix/623/]) HBASE-15065 Addendum fixes checkstyle warnings (tedyu: rev 4a26644245bad6a661442f0eede6850e12d07cc9) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090367#comment-15090367 ] Hudson commented on HBASE-15065: SUCCESS: Integrated in HBase-1.3 #489 (See [https://builds.apache.org/job/HBase-1.3/489/]) HBASE-15065 Addendum fixes checkstyle warnings (tedyu: rev d277c153309c1ad5d5b08820c5dd1f6038f6ab2b) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14975) Don't color the total RIT line yellow if it's zero
[ https://issues.apache.org/jira/browse/HBASE-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090369#comment-15090369 ] Hudson commented on HBASE-14975: SUCCESS: Integrated in HBase-1.3 #489 (See [https://builds.apache.org/job/HBase-1.3/489/]) HBASE-14975 Don't color the total RIT line yellow if it's zero (Pallavi (eclark: rev 27c578d685927e814ae50d6457fb01fabdffc28f) * hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon > Don't color the total RIT line yellow if it's zero > -- > > Key: HBASE-14975 > URL: https://issues.apache.org/jira/browse/HBASE-14975 > Project: HBase > Issue Type: Bug > Components: UI >Reporter: Elliott Clark >Assignee: Pallavi Adusumilli > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-14975.patch, Screen Shot 2015-12-14 at 11.37.13 > AM.png, Screenshot 2016-01-04.png > > > Right now if there are regions in transition, sometimes the RIT over 60 > seconds line is colored yellow. It shouldn't be colored yellow if there are > no regions that have been in transition too long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090372#comment-15090372 ] Hudson commented on HBASE-15065: SUCCESS: Integrated in HBase-1.3-IT #431 (See [https://builds.apache.org/job/HBase-1.3-IT/431/]) HBASE-15065 Addendum fixes checkstyle warnings (tedyu: rev d277c153309c1ad5d5b08820c5dd1f6038f6ab2b) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt, 15065-v2.txt, 15065.addendum > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15057) local-master-backup.sh doesn't start HMaster correctly
[ https://issues.apache.org/jira/browse/HBASE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090373#comment-15090373 ] Hudson commented on HBASE-15057: SUCCESS: Integrated in HBase-1.3-IT #431 (See [https://builds.apache.org/job/HBase-1.3-IT/431/]) HBASE-15057 local-master-backup.sh doesn't start HMaster correctly (tedyu: rev f541539da5993148bd31e978d8367e01141ffbf1) * bin/local-master-backup.sh > local-master-backup.sh doesn't start HMaster correctly > -- > > Key: HBASE-15057 > URL: https://issues.apache.org/jira/browse/HBASE-15057 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 1.1.2 > Environment: GNU/Linux x86_64. >Reporter: Cyker Way >Assignee: Samir Ahmic > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15057-v0.patch > > > ./bin/local-master-backup.sh cannot start a backup HMaster. The error in log > is a port conflict. After reading the file it seems an additional option is > required to make it work: > @@ -40,6 +40,7 @@ >DN=$2 >export HBASE_IDENT_STRING="$USER-$DN" >HBASE_MASTER_ARGS="\ > +-D hbase.master.port=`expr 16000 + $DN` \ > -D hbase.master.info.port=`expr 16010 + $DN` \ > -D hbase.regionserver.port=`expr 16020 + $DN` \ > -D hbase.regionserver.info.port=`expr 16030 + $DN` \ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15057) local-master-backup.sh doesn't start HMaster correctly
[ https://issues.apache.org/jira/browse/HBASE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090399#comment-15090399 ] Hudson commented on HBASE-15057: SUCCESS: Integrated in HBase-1.2-IT #387 (See [https://builds.apache.org/job/HBase-1.2-IT/387/]) HBASE-15057 local-master-backup.sh doesn't start HMaster correctly (tedyu: rev 8a36c2bffecdab35af725c8b725be6f4477d6cf7) * bin/local-master-backup.sh > local-master-backup.sh doesn't start HMaster correctly > -- > > Key: HBASE-15057 > URL: https://issues.apache.org/jira/browse/HBASE-15057 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 1.1.2 > Environment: GNU/Linux x86_64. >Reporter: Cyker Way >Assignee: Samir Ahmic > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15057-v0.patch > > > ./bin/local-master-backup.sh cannot start a backup HMaster. The error in log > is a port conflict. After reading the file it seems an additional option is > required to make it work: > @@ -40,6 +40,7 @@ >DN=$2 >export HBASE_IDENT_STRING="$USER-$DN" >HBASE_MASTER_ARGS="\ > +-D hbase.master.port=`expr 16000 + $DN` \ > -D hbase.master.info.port=`expr 16010 + $DN` \ > -D hbase.regionserver.port=`expr 16020 + $DN` \ > -D hbase.regionserver.info.port=`expr 16030 + $DN` \ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15057) local-master-backup.sh doesn't start HMaster correctly
[ https://issues.apache.org/jira/browse/HBASE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090403#comment-15090403 ] Hudson commented on HBASE-15057: FAILURE: Integrated in HBase-Trunk_matrix #623 (See [https://builds.apache.org/job/HBase-Trunk_matrix/623/]) HBASE-15057 local-master-backup.sh doesn't start HMaster correctly (tedyu: rev c2f67a053ebe687773e5e35aa7c060de964dbb68) * bin/local-master-backup.sh > local-master-backup.sh doesn't start HMaster correctly > -- > > Key: HBASE-15057 > URL: https://issues.apache.org/jira/browse/HBASE-15057 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 1.1.2 > Environment: GNU/Linux x86_64. >Reporter: Cyker Way >Assignee: Samir Ahmic > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15057-v0.patch > > > ./bin/local-master-backup.sh cannot start a backup HMaster. The error in log > is a port conflict. After reading the file it seems an additional option is > required to make it work: > @@ -40,6 +40,7 @@ >DN=$2 >export HBASE_IDENT_STRING="$USER-$DN" >HBASE_MASTER_ARGS="\ > +-D hbase.master.port=`expr 16000 + $DN` \ > -D hbase.master.info.port=`expr 16010 + $DN` \ > -D hbase.regionserver.port=`expr 16020 + $DN` \ > -D hbase.regionserver.info.port=`expr 16030 + $DN` \ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14975) Don't color the total RIT line yellow if it's zero
[ https://issues.apache.org/jira/browse/HBASE-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090404#comment-15090404 ] Hudson commented on HBASE-14975: FAILURE: Integrated in HBase-Trunk_matrix #623 (See [https://builds.apache.org/job/HBase-Trunk_matrix/623/]) HBASE-14975 Don't color the total RIT line yellow if it's zero (Pallavi (eclark: rev 52280b64e643584b9b73924a4216442def0501a4) * hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon > Don't color the total RIT line yellow if it's zero > -- > > Key: HBASE-14975 > URL: https://issues.apache.org/jira/browse/HBASE-14975 > Project: HBase > Issue Type: Bug > Components: UI >Reporter: Elliott Clark >Assignee: Pallavi Adusumilli > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-14975.patch, Screen Shot 2015-12-14 at 11.37.13 > AM.png, Screenshot 2016-01-04.png > > > Right now if there are regions in transition, sometimes the RIT over 60 > seconds line is colored yellow. It shouldn't be colored yellow if there are > no regions that have been in transition too long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v8.patch Test cases are modified to verify "expiredRatio" variable has a proper value. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088995#comment-15088995 ] Hudson commented on HBASE-14227: SUCCESS: Integrated in HBase-1.2 #494 (See [https://builds.apache.org/job/HBase-1.2/494/]) HBASE-14227 Reduce the number of time row comparison is done in a Scan (ramkrishna: rev abcab52695e7737d78fe5285e22e2fe5caf78421) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch, HBASE-14227_v6.patch, > HBASE-14227_v7.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089015#comment-15089015 ] Hudson commented on HBASE-14227: SUCCESS: Integrated in HBase-1.3 #486 (See [https://builds.apache.org/job/HBase-1.3/486/]) HBASE-14227 Reduce the number of time row comparison is done in a Scan (ramkrishna: rev e32e4df780bf1935d8421d70af2f3d84ae88f590) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch, HBASE-14227_v6.patch, > HBASE-14227_v7.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1
[
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089033#comment-15089033
]
Hadoop QA commented on HBASE-14970:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12781154/HBASE-14970_branch-1_4.patch
against branch-1 branch at commit 28c2b18d30de4ce9564e328e5fdf42188e83fb63.
ATTACHMENT ID: 12781154
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 64 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestMasterFailover
org.apache.hadoop.hbase.client.TestMetaWithReplicas
org.apache.hadoop.hbase.snapshot.TestExportSnapshot
org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17170//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17170//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17170//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17170//console
This message is automatically generated.
> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-13082-branch-1.patch, HBASE-14970_branch-1.patch,
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch,
> HBASE-14970_branch-1_1.patch, HBASE-14970_branch-1_2.patch,
> HBASE-14970_branch-1_4.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15077) Support OffheapKV write in compaction with out copying data on heap
[
https://issues.apache.org/jira/browse/HBASE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-15077:
---
Attachment: HBASE-15077_V2.patch
Renamed new interface to ByteBufferWriter.
Also kept our DOS name also as DataOutputStream only. ( ByteBufferSupportedDOS
is too big)
Will commit once QA is clean.
Thank for the reviews.
> Support OffheapKV write in compaction with out copying data on heap
> ---
>
> Key: HBASE-15077
> URL: https://issues.apache.org/jira/browse/HBASE-15077
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15077.patch, HBASE-15077_V2.patch
>
>
> HBASE-14832 is not enough to handle this. Doing the remaining needed here.
> {code}
> if (cell instanceof ByteBufferedCell) {
> 890 out.writeShort(rowLen);
> 891 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getRowByteBuffer(),
> 892 ((ByteBufferedCell) cell).getRowPosition(), rowLen);
> 893 out.writeByte(fLen);
> 894 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getFamilyByteBuffer(),
> 895 ((ByteBufferedCell) cell).getFamilyPosition(), fLen);
> 896 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getQualifierByteBuffer(),
> 897 ((ByteBufferedCell) cell).getQualifierPosition(), qLen);
> {code}
> We have done this but it is not really helping us!
> In ByteBufferUtils#copyBufferToStream
> {code}
> public static void copyBufferToStream(OutputStream out, ByteBuffer in,
> int offset, int length) throws IOException {
> if (in.hasArray()) {
> out.write(in.array(), in.arrayOffset() + offset,
> length);
> } else {
> for (int i = 0; i < length; ++i) {
> out.write(toByte(in, offset + i));
> }
> }
> }
> {code}
> So for DBB it is so costly op writing byte by byte reading each to on heap.
> Even if we use writeByteBuffer(OutputStream out, ByteBuffer b, int offset,
> int length), it won't help us as the underlying stream is a
> ByteArrayOutputStream and so we will end up in copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089213#comment-15089213
]
Hudson commented on HBASE-14221:
FAILURE: Integrated in HBase-1.2 #495 (See
[https://builds.apache.org/job/HBase-1.2/495/])
HBASE-14221 - Reduce the number of time row comparison is done in a Scan
(ramkrishna: rev f9c69893e0a56e9eb60a65703d0b481d12c305d0)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221-branch-1.patch,
> HBASE-14221.patch, HBASE-14221_1.patch, HBASE-14221_1.patch,
> HBASE-14221_6.patch, HBASE-14221_9.patch, withmatchingRowspatch.png,
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089214#comment-15089214 ] Hudson commented on HBASE-14227: FAILURE: Integrated in HBase-1.2 #495 (See [https://builds.apache.org/job/HBase-1.2/495/]) Revert "HBASE-14227 Reduce the number of time row comparison is done in (ramkrishna: rev 44e82645838c3c5529c5c98a25dcaa60cf0e1fe4) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch, HBASE-14227_v6.patch, > HBASE-14227_v7.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15077) Support OffheapKV write in compaction with out copying data on heap
[
https://issues.apache.org/jira/browse/HBASE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089148#comment-15089148
]
ramkrishna.s.vasudevan commented on HBASE-15077:
+1 on patch.
> Support OffheapKV write in compaction with out copying data on heap
> ---
>
> Key: HBASE-15077
> URL: https://issues.apache.org/jira/browse/HBASE-15077
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15077.patch, HBASE-15077_V2.patch
>
>
> HBASE-14832 is not enough to handle this. Doing the remaining needed here.
> {code}
> if (cell instanceof ByteBufferedCell) {
> 890 out.writeShort(rowLen);
> 891 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getRowByteBuffer(),
> 892 ((ByteBufferedCell) cell).getRowPosition(), rowLen);
> 893 out.writeByte(fLen);
> 894 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getFamilyByteBuffer(),
> 895 ((ByteBufferedCell) cell).getFamilyPosition(), fLen);
> 896 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getQualifierByteBuffer(),
> 897 ((ByteBufferedCell) cell).getQualifierPosition(), qLen);
> {code}
> We have done this but it is not really helping us!
> In ByteBufferUtils#copyBufferToStream
> {code}
> public static void copyBufferToStream(OutputStream out, ByteBuffer in,
> int offset, int length) throws IOException {
> if (in.hasArray()) {
> out.write(in.array(), in.arrayOffset() + offset,
> length);
> } else {
> for (int i = 0; i < length; ++i) {
> out.write(toByte(in, offset + i));
> }
> }
> }
> {code}
> So for DBB it is so costly op writing byte by byte reading each to on heap.
> Even if we use writeByteBuffer(OutputStream out, ByteBuffer b, int offset,
> int length), it won't help us as the underlying stream is a
> ByteArrayOutputStream and so we will end up in copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089293#comment-15089293
]
Hudson commented on HBASE-14221:
SUCCESS: Integrated in HBase-1.3-IT #429 (See
[https://builds.apache.org/job/HBase-1.3-IT/429/])
HBASE-14221 - Reduce the number of time row comparison is done in a Scan
(ramkrishna: rev bc4b9eb0842bac32dd0f562e3401d5487c45b9da)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221-branch-1.patch,
> HBASE-14221.patch, HBASE-14221_1.patch, HBASE-14221_1.patch,
> HBASE-14221_6.patch, HBASE-14221_9.patch, withmatchingRowspatch.png,
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089262#comment-15089262 ] Hudson commented on HBASE-14227: SUCCESS: Integrated in HBase-1.3 #487 (See [https://builds.apache.org/job/HBase-1.3/487/]) Revert "HBASE-14227 Reduce the number of time row comparison is done in (ramkrishna: rev 323d5c97d92fe8bce3c2bf8e38e68e5fef5ceba6) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch, HBASE-14227_v6.patch, > HBASE-14227_v7.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089087#comment-15089087
]
Hudson commented on HBASE-14221:
SUCCESS: Integrated in HBase-1.2-IT #385 (See
[https://builds.apache.org/job/HBase-1.2-IT/385/])
HBASE-14221 - Reduce the number of time row comparison is done in a
(ramkrishna: rev f9c69893e0a56e9eb60a65703d0b481d12c305d0)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221-branch-1.patch,
> HBASE-14221.patch, HBASE-14221_1.patch, HBASE-14221_1.patch,
> HBASE-14221_6.patch, HBASE-14221_9.patch, withmatchingRowspatch.png,
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089088#comment-15089088 ] Hudson commented on HBASE-14227: SUCCESS: Integrated in HBase-1.2-IT #385 (See [https://builds.apache.org/job/HBase-1.2-IT/385/]) Revert "HBASE-14227 Reduce the number of time row comparison is done in (ramkrishna: rev 44e82645838c3c5529c5c98a25dcaa60cf0e1fe4) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch, HBASE-14227_v6.patch, > HBASE-14227_v7.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15016) StoreServices facility in Region
[ https://issues.apache.org/jira/browse/HBASE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089127#comment-15089127 ] Eshcar Hillel commented on HBASE-15016: --- Ok so I will work on a patch that includes a new flush policy class which also considers the data in the table/column descriptors when making decisions, and the flush size will be determined also based on this data. Nice :) > StoreServices facility in Region > > > Key: HBASE-15016 > URL: https://issues.apache.org/jira/browse/HBASE-15016 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-15016-V01.patch, HBASE-15016-V02.patch, > HBASE-15016-V03.patch, Regioncounters.pdf, suggestion.patch > > > The default implementation of a memstore ensures that between two flushes the > memstore size increases monotonically. Supporting new memstores that store > data in different formats (specifically, compressed), or that allows to > eliminate data redundancies in memory (e.g., via compaction), means that the > size of the data stored in memory can decrease even between two flushes. This > requires memstores to have access to facilities that manipulate region > counters and synchronization. > This subtasks introduces a new region interface -- StoreServices, through > which store components can access these facilities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15027) Refactor the way the CompactedHFileDischarger threads are created
[ https://issues.apache.org/jira/browse/HBASE-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089102#comment-15089102 ] Hudson commented on HBASE-15027: FAILURE: Integrated in HBase-Trunk_matrix #620 (See [https://builds.apache.org/job/HBase-Trunk_matrix/620/]) HBASE-15027 Refactor the way the CompactedHFileDischarger threads are (ramkrishna: rev 28c2b18d30de4ce9564e328e5fdf42188e83fb63) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactedHFilesDischarger.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java * hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactedHFilesDischarger.java * hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischarger.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischargeHandler.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java * hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java > Refactor the way the CompactedHFileDischarger threads are created > - > > Key: HBASE-15027 > URL: https://issues.apache.org/jira/browse/HBASE-15027 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-15027.patch, HBASE-15027_1.patch, > HBASE-15027_2.patch, HBASE-15027_3.patch, HBASE-15027_3.patch, > HBASE-15027_4.patch, HBASE-15027_5.patch, HBASE-15027_6.patch, > HBASE-15027_7.patch, HBASE-15027_8.patch > > > As per suggestion given over in HBASE-14970, if we need to create a single > thread pool service for the CompactionHFileDischarger we need to create an > exectuor service in the RegionServer level and create discharger handler > threads (Event handlers) and pass the Event to the new Exectuor service that > we create for the compaction hfiles discharger. What should be the default > number of threads here? If a HRS holds 100 of regions - will 10 threads be > enough? This issue will try to resolve this with tests and discussions and > suitable patch will be updated in HBASE-14970 for branch-1 once this is > committed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089294#comment-15089294 ] Hudson commented on HBASE-14227: SUCCESS: Integrated in HBase-1.3-IT #429 (See [https://builds.apache.org/job/HBase-1.3-IT/429/]) Revert "HBASE-14227 Reduce the number of time row comparison is done in (ramkrishna: rev 323d5c97d92fe8bce3c2bf8e38e68e5fef5ceba6) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch, HBASE-14227_v6.patch, > HBASE-14227_v7.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8481) Add cost tracking to API calls for resource management
[ https://issues.apache.org/jira/browse/HBASE-8481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089209#comment-15089209 ] Matthew Byng-Maddick commented on HBASE-8481: - In discussion with Lars, the metrics we should expose are: * for a block cache miss, the size of the block read (this is already there, I suspect, though maybe not quite as explicitly) * for a block cache miss in reading a KV, the size of the KV read Between the two, we can expose a trend of how much excess data we're reading off the disk to help with making decisions on how to size blocks in the HFile. > Add cost tracking to API calls for resource management > -- > > Key: HBASE-8481 > URL: https://issues.apache.org/jira/browse/HBASE-8481 > Project: HBase > Issue Type: New Feature >Reporter: Lars George > > All calls that require resource should be tracked internally using a > lightweight mechanism, such as a simple counter record. The idea is to be > able to tell from calls how many resources they did require, for example a > Get or Scan, how much true I/O was used. This should then be exposed to the > Coprocessor context so that a plugin can make cost based decisions going > forward with other calls. The record should include custom tags that in the > preXYZ() calls can be used to for example tag a record with a user group and > therefore be able to aggregate the statistics and accounting for groups of > users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13525) Update test-patch to leverage Apache Yetus
[ https://issues.apache.org/jira/browse/HBASE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089216#comment-15089216 ] Sean Busbey commented on HBASE-13525: - I've switched the precommit checks over. I'll leave this open for a day or two until I can see how some results go. sending a notice over to dev@ now. > Update test-patch to leverage Apache Yetus > -- > > Key: HBASE-13525 > URL: https://issues.apache.org/jira/browse/HBASE-13525 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > Labels: jenkins > Fix For: 2.0.0 > > Attachments: HBASE-13525.1.patch, HBASE-13525.2.patch > > > Once HADOOP-11746 lands over in Hadoop, incorporate its changes into our > test-patch. Most likely easiest approach is to start with the Hadoop version > and add in the features we have locally that they don't. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14876) Provide maven archetypes
[ https://issues.apache.org/jira/browse/HBASE-14876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089225#comment-15089225 ] Sean Busbey commented on HBASE-14876: - FYI, I resubmitted the latest patch to get built with the just revamped precommit tests. I don't think we have any other "add a new module" changes in patch available status and I'd like to verify that it works correctly. > Provide maven archetypes > > > Key: HBASE-14876 > URL: https://issues.apache.org/jira/browse/HBASE-14876 > Project: HBase > Issue Type: New Feature > Components: build, Usability >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Daniel Vimont > Labels: beginner, maven > Attachments: HBASE-14876-v2.patch, HBASE-14876.patch, > archetype_prototype.zip, archetype_prototype02.zip, > archetype_shaded_prototype01.zip > > > To help onboard new users, we should provide maven archetypes for hbase > client applications. Off the top of my head, we should have templates for > - hbase client application with all dependencies > - hbase client application using client-shaded jar > - mapreduce application with hbase as input and output (ie, copy table) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14128) Fix inability to run Multiple MR over the same Snapshot
[
https://issues.apache.org/jira/browse/HBASE-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089312#comment-15089312
]
Enis Soztutar commented on HBASE-14128:
---
This came up in the dev list again.
Skimmed the patch. Looks good to go. Matteo, is there anything else to do for
this. I can help if you don't have time.
> Fix inability to run Multiple MR over the same Snapshot
> ---
>
> Key: HBASE-14128
> URL: https://issues.apache.org/jira/browse/HBASE-14128
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, snapshots
>Reporter: Matteo Bertozzi
>Assignee: santosh kumar
>Priority: Minor
> Labels: beginner, noob
> Attachments: HBASE-14128-v0.patch
>
>
> from the list, running multiple MR over the same snapshot does not work
> {code}
> public static void copySnapshotForScanner(Configuration conf, FileSystem ..
> RestoreSnapshotHelper helper = new RestoreSnapshotHelper(conf, fs,
> manifest, manifest.getTableDescriptor(), restoreDir, monitor, status);
> {code}
> the problem is that manifest.getTableDescriptor() will try to clone the
> snapshot with the same target name. ending up in "file already exist"
> exceptions.
> we just need to clone that descriptor and generate a new target table name
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[
https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089196#comment-15089196
]
Hadoop QA commented on HBASE-15055:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12781170/HBASE-15055-v8.patch
against master branch at commit 28c2b18d30de4ce9564e328e5fdf42188e83fb63.
ATTACHMENT ID: 12781170
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17171//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17171//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17171//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17171//console
This message is automatically generated.
> Major compaction is not triggered when both of TTL and
> hbase.hstore.compaction.max.size are set
> ---
>
> Key: HBASE-15055
> URL: https://issues.apache.org/jira/browse/HBASE-15055
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Assignee: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch,
> HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch,
> HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch,
> HBASE-15055.patch
>
>
> Some large files may be skipped by hbase.hstore.compaction.max.size in
> candidate selection. It causes skipping of major compaction. So the TTL
> expired records are still remained in the disks and keep consuming disks.
> To resolve this issue, I suggest that to skip large files only if there is no
> TTL expired record.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089261#comment-15089261
]
Hudson commented on HBASE-14221:
SUCCESS: Integrated in HBase-1.3 #487 (See
[https://builds.apache.org/job/HBase-1.3/487/])
HBASE-14221 - Reduce the number of time row comparison is done in a Scan
(ramkrishna: rev bc4b9eb0842bac32dd0f562e3401d5487c45b9da)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221-branch-1.patch,
> HBASE-14221.patch, HBASE-14221_1.patch, HBASE-14221_1.patch,
> HBASE-14221_6.patch, HBASE-14221_9.patch, withmatchingRowspatch.png,
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089362#comment-15089362
]
Hudson commented on HBASE-14221:
FAILURE: Integrated in HBase-1.1-JDK8 #1723 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1723/])
HBASE-14221 - Reduce the number of time row comparison is done in a
(ramkrishna: rev 96da13c04649e2e4f8fbe4280a8fc00016b8d1b4)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221-branch-1.patch,
> HBASE-14221.patch, HBASE-14221_1.patch, HBASE-14221_1.patch,
> HBASE-14221_6.patch, HBASE-14221_9.patch, withmatchingRowspatch.png,
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089363#comment-15089363 ] Hudson commented on HBASE-14227: FAILURE: Integrated in HBase-1.1-JDK8 #1723 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1723/]) Revert "HBASE-14227 Reduce the number of time row comparison is done in (ramkrishna: rev fd3d4e3d6e940851f553aeab40ca3a202cdb99ff) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch, HBASE-14227_v6.patch, > HBASE-14227_v7.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15065) SimpleRegionNormalizer should return multiple normalization plans in one run
[ https://issues.apache.org/jira/browse/HBASE-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15065: --- Hadoop Flags: Reviewed Fix Version/s: 1.3.0 1.2.0 2.0.0 > SimpleRegionNormalizer should return multiple normalization plans in one run > > > Key: HBASE-15065 > URL: https://issues.apache.org/jira/browse/HBASE-15065 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15065-v1.txt > > > This is follow up to HBASE-14867 w.r.t. SimpleRegionNormalizer > Outline for enhancements: > * adjustment to the period when SimpleRegionNormalizer runs > * explore merge opportunities among all neighboring region pairs > * return multiple normalization plans -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089371#comment-15089371
]
Hudson commented on HBASE-14221:
FAILURE: Integrated in HBase-1.1-JDK7 #1636 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1636/])
HBASE-14221 - Reduce the number of time row comparison is done in a Scan
(ramkrishna: rev 96da13c04649e2e4f8fbe4280a8fc00016b8d1b4)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221-branch-1.patch,
> HBASE-14221.patch, HBASE-14221_1.patch, HBASE-14221_1.patch,
> HBASE-14221_6.patch, HBASE-14221_9.patch, withmatchingRowspatch.png,
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15077) Support OffheapKV write in compaction with out copying data on heap
[
https://issues.apache.org/jira/browse/HBASE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089567#comment-15089567
]
Hadoop QA commented on HBASE-15077:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 44s
{color} | {color:red} hbase-common in master has 14 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-server in master has 85 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 4s {color}
| {color:red} hbase-common-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new
issues (was 25, now 25). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 56s {color}
| {color:red} hbase-common-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new
issues (was 25, now 25). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 2s
{color} | {color:red} hbase-common introduced 1 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 40s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m 10s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s
{color} | {color:green} hbase-common in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} |
[jira] [Commented] (HBASE-14128) Fix inability to run Multiple MR over the same Snapshot
[
https://issues.apache.org/jira/browse/HBASE-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089316#comment-15089316
]
Hadoop QA commented on HBASE-14128:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-14128 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12756303/HBASE-14128-v0.patch |
| JIRA Issue | HBASE-14128 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/30/console |
This message was automatically generated.
> Fix inability to run Multiple MR over the same Snapshot
> ---
>
> Key: HBASE-14128
> URL: https://issues.apache.org/jira/browse/HBASE-14128
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, snapshots
>Reporter: Matteo Bertozzi
>Assignee: santosh kumar
>Priority: Minor
> Labels: beginner, noob
> Attachments: HBASE-14128-v0.patch
>
>
> from the list, running multiple MR over the same snapshot does not work
> {code}
> public static void copySnapshotForScanner(Configuration conf, FileSystem ..
> RestoreSnapshotHelper helper = new RestoreSnapshotHelper(conf, fs,
> manifest, manifest.getTableDescriptor(), restoreDir, monitor, status);
> {code}
> the problem is that manifest.getTableDescriptor() will try to clone the
> snapshot with the same target name. ending up in "file already exist"
> exceptions.
> we just need to clone that descriptor and generate a new target table name
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[
https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089327#comment-15089327
]
Ted Yu commented on HBASE-15055:
Without the floating point fix, I got:
{code}
testMajorCompactionWithTtlAndCompactionMaxSize2(org.apache.hadoop.hbase.regionserver.TestMajorCompaction)
Time elapsed: 11.038 sec <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<2>
...
testMajorCompactionWithTtlAndCompactionMaxSize3(org.apache.hadoop.hbase.regionserver.TestMajorCompaction)
Time elapsed: 11.354 sec <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<3>
{code}
Do we need two subtests shown above ?
I wonder if one of them suffices.
Thanks for the persistence, Eungsop.
> Major compaction is not triggered when both of TTL and
> hbase.hstore.compaction.max.size are set
> ---
>
> Key: HBASE-15055
> URL: https://issues.apache.org/jira/browse/HBASE-15055
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Assignee: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch,
> HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch,
> HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch,
> HBASE-15055.patch
>
>
> Some large files may be skipped by hbase.hstore.compaction.max.size in
> candidate selection. It causes skipping of major compaction. So the TTL
> expired records are still remained in the disks and keep consuming disks.
> To resolve this issue, I suggest that to skip large files only if there is no
> TTL expired record.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089547#comment-15089547
]
Hudson commented on HBASE-14221:
FAILURE: Integrated in HBase-1.0 #1132 (See
[https://builds.apache.org/job/HBase-1.0/1132/])
HBASE-14221 - Reduce the number of time row comparison is done in a Scan
(ramkrishna: rev cccf8e6a4a791aa94e738f42370b4f7e1f90353a)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221-branch-1.patch,
> HBASE-14221.patch, HBASE-14221_1.patch, HBASE-14221_1.patch,
> HBASE-14221_6.patch, HBASE-14221_9.patch, withmatchingRowspatch.png,
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089372#comment-15089372 ] Hudson commented on HBASE-14227: FAILURE: Integrated in HBase-1.1-JDK7 #1636 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1636/]) Revert "HBASE-14227 Reduce the number of time row comparison is done in (ramkrishna: rev fd3d4e3d6e940851f553aeab40ca3a202cdb99ff) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch, HBASE-14227_v6.patch, > HBASE-14227_v7.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14889) Region stuck in transition in OPEN state indefinitely in corner scenario
[
https://issues.apache.org/jira/browse/HBASE-14889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089473#comment-15089473
]
Stephen Yuan Jiang commented on HBASE-14889:
[~pankaj2461], how is your testing on the patch? Could you post the patch here
so that we can make progress?
> Region stuck in transition in OPEN state indefinitely in corner scenario
>
>
> Key: HBASE-14889
> URL: https://issues.apache.org/jira/browse/HBASE-14889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.14, 1.0.2
>Reporter: Abhishek Singh Chouhan
>Assignee: Pankaj Kumar
>
> During a failure scenario when a RS dies and the bulk assigner(BA) is
> assigning its regions to others RSs, if another RS dies(on which some regions
> are being moved) on which region is in pending open state, we end up in a
> situation where two bulk assigners try to assign the same region on the Same
> RS.
> The following happened -
> 1. While one BA was opening the region the second one sees it in pending open
> state, retries and calls unassign(...) thereby sending CLOSE RPC to the RS.
> 2. The RS meanwhile has already opened the region, hence changing the znode
> state to RS_ZK_REGION_OPENED which triggers event on master.
> 3. On master after the unassign is successful we go on to deleting the znode,
> change region state to Pending open and send open RPC to RS.
> 4. The earlier triggered event now sees the state as Pending open and happily
> changes it to OPEN, but is unable to delete the znode which by this time is
> not in RS_ZK_REGION_OPENED state but is in M_ZK_REGION_OFFLINE state. Hence
> the region remains in transition in the OPEN state.
> 5. RS goes on to changing the znode states and successfully opens the region
> (changes znode state to RS_ZK_REGION_OPENED)
> 6. This again triggers event on master but this time since the state is OPEN
> the folloing code path is taken
> {noformat}
> case RS_ZK_REGION_OPENED:
> // Should see OPENED after OPENING but possible after PENDING_OPEN.
> if (regionState == null
> || !regionState.isPendingOpenOrOpeningOnServer(sn)) {
> LOG.warn("Received OPENED for " + prettyPrintedRegionName
> + " from " + sn + " but the region isn't PENDING_OPEN/OPENING
> here: "
> + regionStates.getRegionState(encodedName));
> if (regionState != null) {
> // Close it without updating the internal region states,
> // so as not to create double assignments in unlucky scenarios
> // mentioned in OpenRegionHandler#process
> unassign(regionState.getRegion(), null, -1, null, false, sn);
> }
> return;
> }
> {noformat}
> We call unassign here with transitionInZK=false and state=null
> 7. RS closes the region but doesn't update the ZK, also state is not changed
> in master. Region remains in transition in OPEN state, when its actually
> closed. We have to restart the RS post which it opens correctly on some other
> RS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15089548#comment-15089548 ] Hudson commented on HBASE-14227: FAILURE: Integrated in HBase-1.0 #1132 (See [https://builds.apache.org/job/HBase-1.0/1132/]) HBASE-14227 Reduce the number of time row comparison is done in a Scan (ramkrishna: rev 13bbd722650affcea9e9f876c078c0da7a3aba99) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java Revert "HBASE-14227 Reduce the number of time row comparison is done in (ramkrishna: rev 5d854d3802cd1bcf50fea3c2a187d75d51567308) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch, HBASE-14227_v6.patch, > HBASE-14227_v7.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
