[jira] [Updated] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17178: --- Attachment: HBASE-17178-v6.patch > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-branch-1.patch, HBASE-17178-v1.patch, > HBASE-17178-v2.patch, HBASE-17178-v3.patch, HBASE-17178-v4.patch, > HBASE-17178-v5.patch, HBASE-17178-v6.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. > Review board: https://reviews.apache.org/r/54191/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17167) Pass mvcc to client when scan
[ https://issues.apache.org/jira/browse/HBASE-17167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707847#comment-15707847 ] Hudson commented on HBASE-17167: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2045 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2045/]) HBASE-17167 Pass mvcc to client when scan (zhangduo: rev 890fcbd0e6f916cc94b45b881b0cc060cc1e835c) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMvccConsistentScanner.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/PackagePrivateFieldAccessor.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java * (edit) hbase-protocol-shaded/src/main/protobuf/Client.proto * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos.java * (edit) hbase-protocol/src/main/protobuf/Client.proto * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java > Pass mvcc to client when scan > - > > Key: HBASE-17167 > URL: https://issues.apache.org/jira/browse/HBASE-17167 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17167-branch-1-v1.patch, > HBASE-17167-branch-1-v2.patch, HBASE-17167-branch-1.patch, > HBASE-17167-v1.patch, HBASE-17167-v2.patch, HBASE-17167-v3.patch, > HBASE-17167-v4.patch, HBASE-17167-v5.patch, HBASE-17167.patch > > > For the current implementation, if we use batch or allowPartial when scan, > then the row level atomic can not be guaranteed if we need to restart a scan > in the middle of a record due to region move or something else. > We can return the mvcc used to open scanner to client and client could use > this mvcc to restart a scan to get row level atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17178) Add region balance throttling
[
https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707838#comment-15707838
]
Hadoop QA commented on HBASE-17178:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 14s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 40s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 50s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 35s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840985/HBASE-17178-v5.patch |
| JIRA Issue | HBASE-17178 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile xml |
| uname | Linux 7b26de9777f9 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 890fcbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4703/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4703/testReport/ |
| modules | C: hbase-common
[jira] [Commented] (HBASE-17201) Edit of HFileBlock comments and javadoc
[ https://issues.apache.org/jira/browse/HBASE-17201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707835#comment-15707835 ] stack commented on HBASE-17201: --- Reuse of init was done wrong. Fix. > Edit of HFileBlock comments and javadoc > --- > > Key: HBASE-17201 > URL: https://issues.apache.org/jira/browse/HBASE-17201 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-17201.master.001.patch, > HBASE-17201.master.002.patch > > > Spent time in HFileBlock trying to do the parent issue. Failed. But did a > bunch of edits of comments/javadoc. Let me get those in at least. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17201) Edit of HFileBlock comments and javadoc
[ https://issues.apache.org/jira/browse/HBASE-17201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17201: -- Attachment: HBASE-17201.master.002.patch > Edit of HFileBlock comments and javadoc > --- > > Key: HBASE-17201 > URL: https://issues.apache.org/jira/browse/HBASE-17201 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-17201.master.001.patch, > HBASE-17201.master.002.patch > > > Spent time in HFileBlock trying to do the parent issue. Failed. But did a > bunch of edits of comments/javadoc. Let me get those in at least. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17203) unable to see the namespaces and tables when grant read-only privleges
[ https://issues.apache.org/jira/browse/HBASE-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707827#comment-15707827 ] Ashish Singhi edited comment on HBASE-17203 at 11/30/16 7:23 AM: - This is correct. To list namespaces, a user needs to have Admin privilege. Please check https://hbase.apache.org/book.html#appendix_acl_matrix was (Author: ashish singhi): This is correct. To see to namespaces a user needs to have Admin privilege. Please check https://hbase.apache.org/book.html#appendix_acl_matrix > unable to see the namespaces and tables when grant read-only privleges > -- > > Key: HBASE-17203 > URL: https://issues.apache.org/jira/browse/HBASE-17203 > Project: HBase > Issue Type: Task > Components: hbase, shell >Affects Versions: 1.0.0 >Reporter: sandeep vura > > Hi, > i have created namespaces and given grant read-only privileges. When i run > "list_namespace" on hbase shell i am unable to see the namespaces and also > the tables. > But namespaces and tables are able to see in HUE --> HBase browser. > Executed commands: > create_namespace 'q1' > grant '@hadooptest','R','@q1' > Please advise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-17203) unable to see the namespaces and tables when grant read-only privleges
[ https://issues.apache.org/jira/browse/HBASE-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi resolved HBASE-17203. --- Resolution: Invalid This is correct. To see to namespaces a user needs to have Admin privilege. Please check https://hbase.apache.org/book.html#appendix_acl_matrix > unable to see the namespaces and tables when grant read-only privleges > -- > > Key: HBASE-17203 > URL: https://issues.apache.org/jira/browse/HBASE-17203 > Project: HBase > Issue Type: Task > Components: hbase, shell >Affects Versions: 1.0.0 >Reporter: sandeep vura > > Hi, > i have created namespaces and given grant read-only privileges. When i run > "list_namespace" on hbase shell i am unable to see the namespaces and also > the tables. > But namespaces and tables are able to see in HUE --> HBase browser. > Executed commands: > create_namespace 'q1' > grant '@hadooptest','R','@q1' > Please advise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17185) Purge the seek of the next block reading HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-17185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707794#comment-15707794
]
Hadoop QA commented on HBASE-17185:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 12s
{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 18s
{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 24s
{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 35s
{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 48s
{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 2s
{color} | {color:red} The patch causes 22 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 17s
{color} | {color:red} The patch causes 22 errors with Hadoop v2.7.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 27s
{color} | {color:red} The patch causes 22 errors with Hadoop v2.7.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 41s
{color} | {color:red} The patch causes 22 errors with Hadoop v3.0.0-alpha1.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 28s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 49s {color}
| {color:black}
[jira] [Updated] (HBASE-16904) remove directory layout / fs references from snapshots
[
https://issues.apache.org/jira/browse/HBASE-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16904:
Resolution: Fixed
Fix Version/s: hbase-14439
Status: Resolved (was: Patch Available)
> remove directory layout / fs references from snapshots
> --
>
> Key: HBASE-16904
> URL: https://issues.apache.org/jira/browse/HBASE-16904
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>Assignee: Umesh Agashe
> Fix For: hbase-14439
>
> Attachments: HBASE-16904-hbase-14439.v1.patch,
> HBASE-16904-hbase-14439.v2.patch, HBASE-16904-hbase-14439.v3.patch,
> HBASE-16904-hbase-14439.v4.patch
>
>
> ensure snapshot code works through the MasterStorage / RegionStorage APIs and
> not directly on backing filesystem.
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV1.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV2.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[
https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707765#comment-15707765
]
Hadoop QA commented on HBASE-16648:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 59s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 7s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterShutdown |
| | hadoop.hbase.TestZooKeeper |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840981/HBASE-16648-v3.patch |
| JIRA Issue | HBASE-16648 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 8bee467e729e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 890fcbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4701/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit
[jira] [Commented] (HBASE-17202) Master takes a long time to initialize
[ https://issues.apache.org/jira/browse/HBASE-17202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707748#comment-15707748 ] Ashu Pachauri commented on HBASE-17202: --- I was aware of the original change, but missed the changes introduced in the addendum. Thanks [~aoxiang] and [~carp84] for pointing it out. > Master takes a long time to initialize > -- > > Key: HBASE-17202 > URL: https://issues.apache.org/jira/browse/HBASE-17202 > Project: HBase > Issue Type: Improvement > Components: Balancer, master >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > > On master failover or a full cluster restart, we create the > RegionLocationFinder cache by calculating block locations for every region; > this cache is used by the balancer for calculating region locality. This > greatly increases the time it takes for the assignment manager to start and > affects region availability in scenarios of full cluster restart. > We should initialize assignment manager with the bare minimum information and > asynchronously load the block location cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk
[
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707725#comment-15707725
]
stack commented on HBASE-17081:
---
Looking at patch:
215 public List getScanners(long readPoint, long order) {
216 List scanners = new
ArrayList(this.pipeline.size());
217 for (Segment segment : this.pipeline) {
218 scanners.add(segment.getScanner(readPoint, order));
219 order--;
220 }
221 return scanners;
222 }
How does order work in the above? Should we stop if it goes negative? Can it go
negative? We do the above pattern in another place at least in the patch.
Change
70public List getAllSegments() {
71 List res = new LinkedList(segments);
72 return res;
73}
to return new LinkedList(segments); No need to park in the local res
variable (This is done in a few places in the patch).
Why KeyValueScanner instead of SegmentScanner? CellScanner?
If we pass true for empty in below though all is zero, is it not-empty?
40public MemstoreSize(boolean isEmpty) {
41 dataSize = 0;
42 heapOverhead = 0;
43 this.isEmpty = isEmpty;
44}
Should emptyness be derivative of dataSize?
In other contstructor, what if the passed in dataSize is empty, is the size not
empty?
this.dataSize = dataSize;
Remove these rather than comment them out:
213 //assertTrue("\n<<< The memstore scanners with one snapshot are:
\n" + memstorescanners
214 //+ "\n",false);
215
Otherwise, patch looks good to me. What you fellows thing, [~ram_krish] and
[~anoop.hbase]?
> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-17081-V01.patch, HBASE-17081-V02.patch,
> HBASE-17081-V03.patch, HBASE-17081-V04.patch,
> Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another
> part is divided between immutable segments in the compacting pipeline. Upon
> flush-to-disk request we want to flush all of it to disk, in contrast to
> flushing only tail of the compacting pipeline.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17167) Pass mvcc to client when scan
[ https://issues.apache.org/jira/browse/HBASE-17167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707681#comment-15707681 ] Hudson commented on HBASE-17167: FAILURE: Integrated in Jenkins build HBase-1.4 #551 (See [https://builds.apache.org/job/HBase-1.4/551/]) HBASE-17167 Pass mvcc to client when scan (zhangduo: rev af69783128ed8bd8bf321f378adcb50d46a4f2fc) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/PackagePrivateFieldAccessor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMvccConsistentScanner.java * (edit) hbase-protocol/src/main/protobuf/Client.proto * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java * (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java > Pass mvcc to client when scan > - > > Key: HBASE-17167 > URL: https://issues.apache.org/jira/browse/HBASE-17167 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17167-branch-1-v1.patch, > HBASE-17167-branch-1-v2.patch, HBASE-17167-branch-1.patch, > HBASE-17167-v1.patch, HBASE-17167-v2.patch, HBASE-17167-v3.patch, > HBASE-17167-v4.patch, HBASE-17167-v5.patch, HBASE-17167.patch > > > For the current implementation, if we use batch or allowPartial when scan, > then the row level atomic can not be guaranteed if we need to restart a scan > in the middle of a record due to region move or something else. > We can return the mvcc used to open scanner to client and client could use > this mvcc to restart a scan to get row level atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707683#comment-15707683 ] Ashish Singhi commented on HBASE-17178: --- I left some very minor comments on RB. Thanks. > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-branch-1.patch, HBASE-17178-v1.patch, > HBASE-17178-v2.patch, HBASE-17178-v3.patch, HBASE-17178-v4.patch, > HBASE-17178-v5.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. > Review board: https://reviews.apache.org/r/54191/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-17202) Master takes a long time to initialize
[ https://issues.apache.org/jira/browse/HBASE-17202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li resolved HBASE-17202. --- Resolution: Duplicate Duplicate with HBASE-16570, please refer to the [addendum patch v7|https://issues.apache.org/jira/secure/attachment/12837729/HBASE-16570_addnum_v7.patch] and [comment|https://issues.apache.org/jira/browse/HBASE-16570?focusedCommentId=15649981=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15649981] there for more details. > Master takes a long time to initialize > -- > > Key: HBASE-17202 > URL: https://issues.apache.org/jira/browse/HBASE-17202 > Project: HBase > Issue Type: Improvement > Components: Balancer, master >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > > On master failover or a full cluster restart, we create the > RegionLocationFinder cache by calculating block locations for every region; > this cache is used by the balancer for calculating region locality. This > greatly increases the time it takes for the assignment manager to start and > affects region availability in scenarios of full cluster restart. > We should initialize assignment manager with the bare minimum information and > asynchronously load the block location cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17203) unable to see the namespaces and tables when grant read-only privleges
sandeep vura created HBASE-17203: Summary: unable to see the namespaces and tables when grant read-only privleges Key: HBASE-17203 URL: https://issues.apache.org/jira/browse/HBASE-17203 Project: HBase Issue Type: Task Components: hbase, shell Affects Versions: 1.0.0 Reporter: sandeep vura Hi, i have created namespaces and given grant read-only privileges. When i run "list_namespace" on hbase shell i am unable to see the namespaces and also the tables. But namespaces and tables are able to see in HUE --> HBase browser. Executed commands: create_namespace 'q1' grant '@hadooptest','R','@q1' Please advise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707652#comment-15707652 ] Phil Yang commented on HBASE-17178: --- Thanks [~carp84]. Will commit if QA pass. > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-branch-1.patch, HBASE-17178-v1.patch, > HBASE-17178-v2.patch, HBASE-17178-v3.patch, HBASE-17178-v4.patch, > HBASE-17178-v5.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. > Review board: https://reviews.apache.org/r/54191/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707644#comment-15707644 ] Yu Li commented on HBASE-17178: --- +1 on v5 patch > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-branch-1.patch, HBASE-17178-v1.patch, > HBASE-17178-v2.patch, HBASE-17178-v3.patch, HBASE-17178-v4.patch, > HBASE-17178-v5.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. > Review board: https://reviews.apache.org/r/54191/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17178: --- Affects Version/s: 1.4.0 2.0.0 > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-branch-1.patch, HBASE-17178-v1.patch, > HBASE-17178-v2.patch, HBASE-17178-v3.patch, HBASE-17178-v4.patch, > HBASE-17178-v5.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. > Review board: https://reviews.apache.org/r/54191/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17178: --- Attachment: HBASE-17178-branch-1.patch Attach patch for branch-1. > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-branch-1.patch, HBASE-17178-v1.patch, > HBASE-17178-v2.patch, HBASE-17178-v3.patch, HBASE-17178-v4.patch, > HBASE-17178-v5.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. > Review board: https://reviews.apache.org/r/54191/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707602#comment-15707602
]
Hadoop QA commented on HBASE-17114:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 58s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 46s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 15s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.2 Server=1.12.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840983/HBASE-17114.v4.patch |
| JIRA Issue | HBASE-17114 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile xml |
| uname | Linux 3bd628b7150d 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18
16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 890fcbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4702/testReport/ |
| modules | C: hbase-common hbase-client U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4702/console |
|
[jira] [Reopened] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reopened HBASE-17197:
---
Reopening. Having to add the -f is a PITA.
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>
> I tried to use hfile in master branch, it does not print out kv pairs or meta
> as it is supposed to be.
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/
> 53e9f9bc328f468b87831221de3a0c74 bdc6e1c4eea246a99e989e02d554cb03
> bf9275ac418d4d458904d81137e82683
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -m
> 2016-11-29 12:25:22,019 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -p
> 2016-11-29 12:25:27,333 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Scanned kv count -> 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15704) Refactoring: Move HFileArchiver from backup to tool package, remove backup.examples
[ https://issues.apache.org/jira/browse/HBASE-15704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15704: -- Description: This class is in backup package (as well as backup/examples classes) but is not backup - related. Remove examples classes from a codebase (was: This class is in backup package (as well as backup/examples classes) but is not backup - related. Move examples classes to hbase-examples package.) > Refactoring: Move HFileArchiver from backup to tool package, remove > backup.examples > --- > > Key: HBASE-15704 > URL: https://issues.apache.org/jira/browse/HBASE-15704 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15704-v2.patch, HBASE-15704-v3.patch > > > This class is in backup package (as well as backup/examples classes) but is > not backup - related. Remove examples classes from a codebase -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15704) Refactoring: Move HFileArchiver from backup to tool package, remove backup.examples
[ https://issues.apache.org/jira/browse/HBASE-15704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15704: -- Summary: Refactoring: Move HFileArchiver from backup to tool package, remove backup.examples (was: Refactoring: Move HFileArchiver from backup to its own package) > Refactoring: Move HFileArchiver from backup to tool package, remove > backup.examples > --- > > Key: HBASE-15704 > URL: https://issues.apache.org/jira/browse/HBASE-15704 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15704-v2.patch, HBASE-15704-v3.patch > > > This class is in backup package (as well as backup/examples classes) but is > not backup - related. Move examples classes to hbase-examples package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17178: --- Attachment: HBASE-17178-v5.patch Attach a v5 patch addressed review comments. > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-v1.patch, HBASE-17178-v2.patch, > HBASE-17178-v3.patch, HBASE-17178-v4.patch, HBASE-17178-v5.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. > Review board: https://reviews.apache.org/r/54191/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17114:
--
Attachment: HBASE-17114.v4.patch
Fix white space.
> Add an option to set special retry pause when encountering
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch, HBASE-17114.v2.patch,
> HBASE-17114.v3.patch, HBASE-17114.v3.patch, HBASE-17114.v4.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}}
> instead of dead-wait. This is good for performance for most cases but might
> cause a side-effect that if too many clients connect to the busy RS, that the
> retry requests may come over and over again and RS never got the chance for
> recovering, and the issue will become especially critical when the target
> region is META.
> So here in this JIRA we propose to add a new property in name of
> {{hbase.client.pause.cqtbe}} to make it possible to set a special-longer
> pause for CallQueueTooBigException, and by default it will use the setting of
> {{hbase.client.pause}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17140) Reduce meta request number by skipping table state check
[
https://issues.apache.org/jira/browse/HBASE-17140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707486#comment-15707486
]
Guanghao Zhang commented on HBASE-17140:
Thanks for your reply.
bq. In the case of (2), the HRI for parent region is saved with split=true,
offline=true (similar for merge).
If I am not wrong, when merge A and B to a new region, the region info of A and
B are deleted directly? So split=true, offline=true means a split parent
region. And offline=true means a region of disabled table.
bq. When the table is re-enabled again, we do not want to bring back the old
parents.
When enable a table, it need to get the table regions first and the split
parent region will be filtered in this step. So I thought it can't bring back?
> Reduce meta request number by skipping table state check
>
>
> Key: HBASE-17140
> URL: https://issues.apache.org/jira/browse/HBASE-17140
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17140-v1.patch, HBASE-17140-v2.patch,
> HBASE-17140-v3.patch, HBASE-17140-v4.patch, HBASE-17140-v5.patch
>
>
> Now when request for a disabled table, it need 3 rpc calls before fail.
> 1. get region location
> 2. send call to rs and get NotServeRegionException
> 3. retry and check the table state, then throw TableNotEnabledException
> The table state check is added for disabled table. But now the prepare method
> in RegionServerCallable shows that all retry request will get table state
> first.
> {code}
> public void prepare(final boolean reload) throws IOException {
> // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
> throw new TableNotEnabledException(tableName.getNameAsString() + " is
> disabled.");
> }
> try (RegionLocator regionLocator =
> connection.getRegionLocator(tableName)) {
> this.location = regionLocator.getRegionLocation(row);
> }
> if (this.location == null) {
> throw new IOException("Failed to find location, tableName=" + tableName
> +
> ", row=" + Bytes.toString(row) + ", reload=" + reload);
> }
> setStubByServiceName(this.location.getServerName());
> }
> {code}
> An improvement is set the region offline in HRegionInfo and throw the
> RegionOfflineException when get region location. Then we don't need check
> table state for any retry request.
> Review board: https://reviews.apache.org/r/54071/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16648: -- Attachment: HBASE-16648-v3.patch Let's finish this issue. I use the CollectionUtils.computeIfAbsent instead of CM.computeIfAbsent As the CM.get is much faster than CM.computeIfAbsent. The implementation of CollectionUtils.computeIfAbsent uses get and putIfAbsent. > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, > HBASE-16648-v3.patch, HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16648: -- Status: Patch Available (was: Open) > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, > HBASE-16648-v3.patch, HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17178) Add region balance throttling
[
https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707454#comment-15707454
]
Hadoop QA commented on HBASE-17178:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 48s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 14s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m 16s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 139m 6s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840972/HBASE-17178-v4.patch |
| JIRA Issue | HBASE-17178 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile xml |
| uname | Linux 6d5f4d5ed08f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c43a23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4699/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4699/console |
| Powered
[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707427#comment-15707427
]
Hadoop QA commented on HBASE-17114:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 54s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840977/HBASE-17114.v3.patch |
| JIRA Issue | HBASE-17114 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile xml |
| uname | Linux 0fe37afb922e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 890fcbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/4700/artifact/patchprocess/whitespace-tabs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4700/testReport/ |
| modules | C: hbase-common hbase-client U:
[jira] [Commented] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707404#comment-15707404
]
huaxiang sun commented on HBASE-17197:
--
Thanks [~carp84], let me post a minor patch to remind user if there is no file
to process.
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>
> I tried to use hfile in master branch, it does not print out kv pairs or meta
> as it is supposed to be.
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/
> 53e9f9bc328f468b87831221de3a0c74 bdc6e1c4eea246a99e989e02d554cb03
> bf9275ac418d4d458904d81137e82683
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -m
> 2016-11-29 12:25:22,019 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -p
> 2016-11-29 12:25:27,333 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Scanned kv count -> 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707388#comment-15707388
]
Yu Li commented on HBASE-17197:
---
IMHO we should print some error message and exit rather than just moving on w/o
-f option? Seems something to fix here instead of marking JIRA as invalid
(smile).
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>
> I tried to use hfile in master branch, it does not print out kv pairs or meta
> as it is supposed to be.
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/
> 53e9f9bc328f468b87831221de3a0c74 bdc6e1c4eea246a99e989e02d554cb03
> bf9275ac418d4d458904d81137e82683
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -m
> 2016-11-29 12:25:22,019 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -p
> 2016-11-29 12:25:27,333 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Scanned kv count -> 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17151) New API to create HFile.Reader without instantiating block cache
[
https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707364#comment-15707364
]
Hadoop QA commented on HBASE-17151:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 47s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m 9s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 27s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840963/HBASE-17151-v3.patch |
| JIRA Issue | HBASE-17151 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7394de3e683c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c43a23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4697/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4697/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> New API to create HFile.Reader without instantiating block cache
> -
>
> Key: HBASE-17151
> URL: https://issues.apache.org/jira/browse/HBASE-17151
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch,
>
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707354#comment-15707354
]
Yu Li commented on HBASE-17110:
---
Ok, planing to commit it soon if no objections. Will update the JIRA title and
release note accordingly to reflect changes in the final patch.
[~xharlie] please prepare a patch for branch-1 if necessary, thanks.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch,
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110-V6.patch,
> HBASE-17110-V7.patch, HBASE-17110-V8.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[
https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707351#comment-15707351
]
Hadoop QA commented on HBASE-16209:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m 6s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 17s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 8s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 165m 21s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.2 Server=1.12.2 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840959/HBASE-16209-addendum.v6.branch-1.patch
|
| JIRA Issue | HBASE-16209 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c2f2b803593f 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18
16:48:51 UTC 2016 x86_64
[jira] [Commented] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707345#comment-15707345 ] Yu Li commented on HBASE-17178: --- Overall LGTM, +1. Left some comment in RB, please check it there. Thanks. > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-v1.patch, HBASE-17178-v2.patch, > HBASE-17178-v3.patch, HBASE-17178-v4.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. > Review board: https://reviews.apache.org/r/54191/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17140) Reduce meta request number by skipping table state check
[
https://issues.apache.org/jira/browse/HBASE-17140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707341#comment-15707341
]
Enis Soztutar commented on HBASE-17140:
---
We have to be careful here table disabled and region offline are different
concepts. In the current code, region offline can mean two things (depending on
meta or AM context):
(1) Region is not assigned yet.
(2) Region is no more "assignable", meaning the region is split or merged.
In the case of (2), the HRI for parent region is saved with split=true,
offline=true (similar for merge). If we are marking every region to be offline
in a disable table, we have to be careful of this double meaning. When the
table is re-enabled again, we do not want to bring back the old parents.
Can we instead do a multi-get to meta for getting the table state and the
region location everytime? This way there will be a single RPC to the meta
table returning both the result of table state and the region location.
> Reduce meta request number by skipping table state check
>
>
> Key: HBASE-17140
> URL: https://issues.apache.org/jira/browse/HBASE-17140
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17140-v1.patch, HBASE-17140-v2.patch,
> HBASE-17140-v3.patch, HBASE-17140-v4.patch, HBASE-17140-v5.patch
>
>
> Now when request for a disabled table, it need 3 rpc calls before fail.
> 1. get region location
> 2. send call to rs and get NotServeRegionException
> 3. retry and check the table state, then throw TableNotEnabledException
> The table state check is added for disabled table. But now the prepare method
> in RegionServerCallable shows that all retry request will get table state
> first.
> {code}
> public void prepare(final boolean reload) throws IOException {
> // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
> throw new TableNotEnabledException(tableName.getNameAsString() + " is
> disabled.");
> }
> try (RegionLocator regionLocator =
> connection.getRegionLocator(tableName)) {
> this.location = regionLocator.getRegionLocation(row);
> }
> if (this.location == null) {
> throw new IOException("Failed to find location, tableName=" + tableName
> +
> ", row=" + Bytes.toString(row) + ", reload=" + reload);
> }
> setStubByServiceName(this.location.getServerName());
> }
> {code}
> An improvement is set the region offline in HRegionInfo and throw the
> RegionOfflineException when get region location. Then we don't need check
> table state for any retry request.
> Review board: https://reviews.apache.org/r/54071/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17114:
--
Attachment: HBASE-17114.v3.patch
> Add an option to set special retry pause when encountering
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch, HBASE-17114.v2.patch,
> HBASE-17114.v3.patch, HBASE-17114.v3.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}}
> instead of dead-wait. This is good for performance for most cases but might
> cause a side-effect that if too many clients connect to the busy RS, that the
> retry requests may come over and over again and RS never got the chance for
> recovering, and the issue will become especially critical when the target
> region is META.
> So here in this JIRA we propose to add a new property in name of
> {{hbase.client.pause.cqtbe}} to make it possible to set a special-longer
> pause for CallQueueTooBigException, and by default it will use the setting of
> {{hbase.client.pause}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17194) Assign the new region to the idle server after splitting
[ https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707325#comment-15707325 ] ChiaPing Tsai commented on HBASE-17194: --- bq. Can you rename 'predicater' to something like 'idleServerPredicator' to be specific? bq. Also, could you rename the new getOnlineServersList() to something like 'getOnlineServersListWithIdlePredicator' so that it won't cause confusion like stack mentioned? copy that. bq. Should it be 'keys.forEach'? You are right. I will fix it. Thanks for your review. > Assign the new region to the idle server after splitting > > > Key: HBASE-17194 > URL: https://issues.apache.org/jira/browse/HBASE-17194 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17194.v0.patch, evaluation-v0.png > > > The new regions are assigned to the random servers after splitting, but there > always are some idle servers which don’t be assigned any regions on the new > cluster. It is a bad start of load balance, hence we should give priority to > the idle servers for assignment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17194) Assign the new region to the idle server after splitting
[ https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17194: -- Status: Open (was: Patch Available) > Assign the new region to the idle server after splitting > > > Key: HBASE-17194 > URL: https://issues.apache.org/jira/browse/HBASE-17194 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17194.v0.patch, evaluation-v0.png > > > The new regions are assigned to the random servers after splitting, but there > always are some idle servers which don’t be assigned any regions on the new > cluster. It is a bad start of load balance, hence we should give priority to > the idle servers for assignment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17194) Assign the new region to the idle server after splitting
[ https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707322#comment-15707322 ] ChiaPing Tsai commented on HBASE-17194: --- bq. There seems to be no relation between the current getOnlineServersList and the new one below I will rename the new method. bq. Does this happen alot? Yes, it often happen when we reset and reload data into the testing cluster. > Assign the new region to the idle server after splitting > > > Key: HBASE-17194 > URL: https://issues.apache.org/jira/browse/HBASE-17194 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17194.v0.patch, evaluation-v0.png > > > The new regions are assigned to the random servers after splitting, but there > always are some idle servers which don’t be assigned any regions on the new > cluster. It is a bad start of load balance, hence we should give priority to > the idle servers for assignment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17151) New API to create HFile.Reader without instantiating block cache
[ https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707316#comment-15707316 ] Enis Soztutar commented on HBASE-17151: --- Thanks Vladimir. v3 looks good. There are no assertions in the test, so even if we end up using the block cache, the test will not fail. > New API to create HFile.Reader without instantiating block cache > - > > Key: HBASE-17151 > URL: https://issues.apache.org/jira/browse/HBASE-17151 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch, > HBASE-17151-v3.patch > > > Currently, to create HFile.Reader instance, the CacheConfig instance is > required (which instantiates block cache). We need API for reading HFile w/o > block cache being involved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15704) Refactoring: Move HFileArchiver from backup to its own package
[
https://issues.apache.org/jira/browse/HBASE-15704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707312#comment-15707312
]
Hadoop QA commented on HBASE-15704:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 19s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 89m 4s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 29s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840960/HBASE-15704-v3.patch |
| JIRA Issue | HBASE-15704 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6383ad87e1b8 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c43a23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4696/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4696/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Refactoring: Move HFileArchiver from backup to its own package
> --
>
> Key: HBASE-15704
> URL: https://issues.apache.org/jira/browse/HBASE-15704
> Project: HBase
> Issue Type: Task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15704-v2.patch, HBASE-15704-v3.patch
>
>
> This class
[jira] [Commented] (HBASE-17202) Master takes a long time to initialize
[ https://issues.apache.org/jira/browse/HBASE-17202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707306#comment-15707306 ] binlijin commented on HBASE-17202: -- Do you have HBASE-16570? It is useful. > Master takes a long time to initialize > -- > > Key: HBASE-17202 > URL: https://issues.apache.org/jira/browse/HBASE-17202 > Project: HBase > Issue Type: Improvement > Components: Balancer, master >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > > On master failover or a full cluster restart, we create the > RegionLocationFinder cache by calculating block locations for every region; > this cache is used by the balancer for calculating region locality. This > greatly increases the time it takes for the assignment manager to start and > affects region availability in scenarios of full cluster restart. > We should initialize assignment manager with the bare minimum information and > asynchronously load the block location cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17167) Pass mvcc to client when scan
[ https://issues.apache.org/jira/browse/HBASE-17167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17167: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-1. Thanks all for reviewing. > Pass mvcc to client when scan > - > > Key: HBASE-17167 > URL: https://issues.apache.org/jira/browse/HBASE-17167 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17167-branch-1-v1.patch, > HBASE-17167-branch-1-v2.patch, HBASE-17167-branch-1.patch, > HBASE-17167-v1.patch, HBASE-17167-v2.patch, HBASE-17167-v3.patch, > HBASE-17167-v4.patch, HBASE-17167-v5.patch, HBASE-17167.patch > > > For the current implementation, if we use batch or allowPartial when scan, > then the row level atomic can not be guaranteed if we need to restart a scan > in the middle of a record due to region move or something else. > We can return the mvcc used to open scanner to client and client could use > this mvcc to restart a scan to get row level atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17202) Master takes a long time to initialize
[ https://issues.apache.org/jira/browse/HBASE-17202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-17202: -- Description: On master failover or a full cluster restart, we create the RegionLocationFinder cache by calculating block locations for every region; this cache is used by the balancer for calculating region locality. This greatly increases the time it takes for the assignment manager to start and affects region availability in scenarios of full cluster restart. We should initialize assignment manager with the bare minimum information and asynchronously load the block location cache. was: On master failover or a full cluster restart, we create the RegionLocationFinder cache by calculating block locations for every region used by the balancer. This greatly increases the time it takes for the assignment manager to start and affects region availability in scenarios of full cluster restart. We should initialize assignment manager with the bare minimum information and asynchronously load the block location cache. > Master takes a long time to initialize > -- > > Key: HBASE-17202 > URL: https://issues.apache.org/jira/browse/HBASE-17202 > Project: HBase > Issue Type: Improvement > Components: Balancer, master >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > > On master failover or a full cluster restart, we create the > RegionLocationFinder cache by calculating block locations for every region; > this cache is used by the balancer for calculating region locality. This > greatly increases the time it takes for the assignment manager to start and > affects region availability in scenarios of full cluster restart. > We should initialize assignment manager with the bare minimum information and > asynchronously load the block location cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17202) Master takes a long time to initialize
Ashu Pachauri created HBASE-17202: - Summary: Master takes a long time to initialize Key: HBASE-17202 URL: https://issues.apache.org/jira/browse/HBASE-17202 Project: HBase Issue Type: Improvement Components: Balancer, master Reporter: Ashu Pachauri Assignee: Ashu Pachauri On master failover or a full cluster restart, we create the RegionLocationFinder cache by calculating block locations for every region used by the balancer. This greatly increases the time it takes for the assignment manager to start and affects region availability in scenarios of full cluster restart. We should initialize assignment manager with the bare minimum information and asynchronously load the block location cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17167) Pass mvcc to client when scan
[ https://issues.apache.org/jira/browse/HBASE-17167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707249#comment-15707249 ] Duo Zhang commented on HBASE-17167: --- The failed UT of branch-1 is not related. But seems it fails consistently. Will open another issue to fix it. Will commit shortly. > Pass mvcc to client when scan > - > > Key: HBASE-17167 > URL: https://issues.apache.org/jira/browse/HBASE-17167 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17167-branch-1-v1.patch, > HBASE-17167-branch-1-v2.patch, HBASE-17167-branch-1.patch, > HBASE-17167-v1.patch, HBASE-17167-v2.patch, HBASE-17167-v3.patch, > HBASE-17167-v4.patch, HBASE-17167-v5.patch, HBASE-17167.patch > > > For the current implementation, if we use batch or allowPartial when scan, > then the row level atomic can not be guaranteed if we need to restart a scan > in the middle of a record due to region move or something else. > We can return the mvcc used to open scanner to client and client could use > this mvcc to restart a scan to get row level atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17181) Let HBase thrift2 support TThreadedSelectorServer
[ https://issues.apache.org/jira/browse/HBASE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707241#comment-15707241 ] Jian Yi commented on HBASE-17181: - how to solve the following error: The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. Start command: bin/hbase-daemon.sh start thrift2 --framed --selector --selectors 5 --workers 30 > Let HBase thrift2 support TThreadedSelectorServer > - > > Key: HBASE-17181 > URL: https://issues.apache.org/jira/browse/HBASE-17181 > Project: HBase > Issue Type: New Feature > Components: Thrift >Affects Versions: 2.0.0, 1.4.0, 1.2.4 >Reporter: Jian Yi >Assignee: Jian Yi >Priority: Minor > Labels: features > Fix For: 2.0.0, 1.4.0, 1.2.5 > > Attachments: HBASE-17181-V1.patch, HBASE-17181-V2.patch, > HBASE-17181-V3.patch, HBASE-17181-V4.patch, HBASE-17181-V5.patch, > HBASE-17181-V6.patch, ThriftServer.java > > Original Estimate: 2h > Remaining Estimate: 2h > > Add TThreadedSelectorServer for HBase Thrift2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17181) Let HBase thrift2 support TThreadedSelectorServer
[
https://issues.apache.org/jira/browse/HBASE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707237#comment-15707237
]
Hadoop QA commented on HBASE-17181:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
6s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 31s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840971/HBASE-17181-V6.patch |
| JIRA Issue | HBASE-17181 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 82e5ba8ed091 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c43a23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4698/testReport/ |
| modules | C: hbase-thrift U: hbase-thrift |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4698/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Let HBase thrift2 support TThreadedSelectorServer
> -
>
> Key: HBASE-17181
> URL: https://issues.apache.org/jira/browse/HBASE-17181
> Project: HBase
> Issue Type: New Feature
> Components: Thrift
>Affects Versions: 2.0.0, 1.4.0, 1.2.4
>
[jira] [Comment Edited] (HBASE-17151) New API to create HFile.Reader without instantiating block cache
[ https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707086#comment-15707086 ] Vladimir Rodionov edited comment on HBASE-17151 at 11/30/16 1:57 AM: - v3. cc:[~enis] was (Author: vrodionov): v3. > New API to create HFile.Reader without instantiating block cache > - > > Key: HBASE-17151 > URL: https://issues.apache.org/jira/browse/HBASE-17151 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch, > HBASE-17151-v3.patch > > > Currently, to create HFile.Reader instance, the CacheConfig instance is > required (which instantiates block cache). We need API for reading HFile w/o > block cache being involved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17140) Reduce meta request number by skipping table state check
[
https://issues.apache.org/jira/browse/HBASE-17140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17140:
---
Affects Version/s: 2.0.0
Description:
Now when request for a disabled table, it need 3 rpc calls before fail.
1. get region location
2. send call to rs and get NotServeRegionException
3. retry and check the table state, then throw TableNotEnabledException
The table state check is added for disabled table. But now the prepare method
in RegionServerCallable shows that all retry request will get table state first.
{code}
public void prepare(final boolean reload) throws IOException {
// check table state if this is a retry
if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
getConnection().isTableDisabled(tableName)) {
throw new TableNotEnabledException(tableName.getNameAsString() + " is
disabled.");
}
try (RegionLocator regionLocator = connection.getRegionLocator(tableName)) {
this.location = regionLocator.getRegionLocation(row);
}
if (this.location == null) {
throw new IOException("Failed to find location, tableName=" + tableName +
", row=" + Bytes.toString(row) + ", reload=" + reload);
}
setStubByServiceName(this.location.getServerName());
}
{code}
An improvement is set the region offline in HRegionInfo and throw the
RegionOfflineException when get region location. Then we don't need check table
state for any retry request.
Review board: https://reviews.apache.org/r/54071/
was:
Now when request for a disabled table, it need 3 rpc calls before fail.
1. get region location
2. send call to rs and get NotServeRegionException
3. retry and check the table state, then throw TableNotEnabledException
The table state check is added for disabled table. But now the prepare method
in RegionServerCallable shows that all retry request will get table state first.
{code}
public void prepare(final boolean reload) throws IOException {
// check table state if this is a retry
if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
getConnection().isTableDisabled(tableName)) {
throw new TableNotEnabledException(tableName.getNameAsString() + " is
disabled.");
}
try (RegionLocator regionLocator = connection.getRegionLocator(tableName)) {
this.location = regionLocator.getRegionLocation(row);
}
if (this.location == null) {
throw new IOException("Failed to find location, tableName=" + tableName +
", row=" + Bytes.toString(row) + ", reload=" + reload);
}
setStubByServiceName(this.location.getServerName());
}
{code}
An improvement is set the region offline in HRegionInfo. Then throw the
RegionOfflineException when get region location.
Review board: https://reviews.apache.org/r/54071/
> Reduce meta request number by skipping table state check
>
>
> Key: HBASE-17140
> URL: https://issues.apache.org/jira/browse/HBASE-17140
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17140-v1.patch, HBASE-17140-v2.patch,
> HBASE-17140-v3.patch, HBASE-17140-v4.patch, HBASE-17140-v5.patch
>
>
> Now when request for a disabled table, it need 3 rpc calls before fail.
> 1. get region location
> 2. send call to rs and get NotServeRegionException
> 3. retry and check the table state, then throw TableNotEnabledException
> The table state check is added for disabled table. But now the prepare method
> in RegionServerCallable shows that all retry request will get table state
> first.
> {code}
> public void prepare(final boolean reload) throws IOException {
> // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
> throw new TableNotEnabledException(tableName.getNameAsString() + " is
> disabled.");
> }
> try (RegionLocator regionLocator =
> connection.getRegionLocator(tableName)) {
> this.location = regionLocator.getRegionLocation(row);
> }
> if (this.location == null) {
> throw new IOException("Failed to find location, tableName=" + tableName
> +
> ", row=" + Bytes.toString(row) + ", reload=" + reload);
> }
> setStubByServiceName(this.location.getServerName());
> }
> {code}
> An improvement is set the region offline in HRegionInfo and throw the
> RegionOfflineException when get region location. Then we don't need check
> table state for any retry request.
> Review board: https://reviews.apache.org/r/54071/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17140) Reduce meta request number by skipping table state check
[
https://issues.apache.org/jira/browse/HBASE-17140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17140:
---
Summary: Reduce meta request number by skipping table state check (was:
Throw RegionOfflineException directly when request for a disabled table)
> Reduce meta request number by skipping table state check
>
>
> Key: HBASE-17140
> URL: https://issues.apache.org/jira/browse/HBASE-17140
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17140-v1.patch, HBASE-17140-v2.patch,
> HBASE-17140-v3.patch, HBASE-17140-v4.patch, HBASE-17140-v5.patch
>
>
> Now when request for a disabled table, it need 3 rpc calls before fail.
> 1. get region location
> 2. send call to rs and get NotServeRegionException
> 3. retry and check the table state, then throw TableNotEnabledException
> The table state check is added for disabled table. But now the prepare method
> in RegionServerCallable shows that all retry request will get table state
> first.
> {code}
> public void prepare(final boolean reload) throws IOException {
> // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
> throw new TableNotEnabledException(tableName.getNameAsString() + " is
> disabled.");
> }
> try (RegionLocator regionLocator =
> connection.getRegionLocator(tableName)) {
> this.location = regionLocator.getRegionLocation(row);
> }
> if (this.location == null) {
> throw new IOException("Failed to find location, tableName=" + tableName
> +
> ", row=" + Bytes.toString(row) + ", reload=" + reload);
> }
> setStubByServiceName(this.location.getServerName());
> }
> {code}
> An improvement is set the region offline in HRegionInfo. Then throw the
> RegionOfflineException when get region location.
> Review board: https://reviews.apache.org/r/54071/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707192#comment-15707192
]
huaxiang sun commented on HBASE-17197:
--
print something if there is no files to process?
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>
> I tried to use hfile in master branch, it does not print out kv pairs or meta
> as it is supposed to be.
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/
> 53e9f9bc328f468b87831221de3a0c74 bdc6e1c4eea246a99e989e02d554cb03
> bf9275ac418d4d458904d81137e82683
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -m
> 2016-11-29 12:25:22,019 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -p
> 2016-11-29 12:25:27,333 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Scanned kv count -> 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707188#comment-15707188 ] Guanghao Zhang edited comment on HBASE-17178 at 11/30/16 1:38 AM: -- Review board: https://reviews.apache.org/r/54191/ bq. Move this line out of synchronized Fixed in v4 patch. bq. Shall the balancing be affected by other RIT? Assuming RS crash happened in middle of balancing, shall we wait? Yes, balancing will be affected by other RIT. This is for availability. If RS crash happend in middle of balancing, there will be more regions in transition. Then the balancer can't finish all region plans. The cluster need a next round balance to reach a balance state. bq. the code flow of balancer might block here and not controlled by the cutoffTime? Fixed in v4 patch. It need break the sleep when exceeds cutoff time. was (Author: zghaobac): Review board: https://reviews.apache.org/r/54191/ bq. Move this line out of synchronized Fixed in v4 patch. bq. Shall the balancing be affected by other RIT? Assuming RS crash happened in middle of balancing, shall we wait? Yes, balancing will be affected by other RIT. This is for availability. If RS crash happend in middle of balancing, there will be more regions in transition. Then the balancer can't finish all region plans. The cluster need a next round balance to reach a balance state. bq. the code flow of balancer might block here and not controlled by the cutoffTime? Fixed in v4 patch. It need break the sleep when exceeds cutoff time. Review board: https://reviews.apache.org/r/54191/ > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-v1.patch, HBASE-17178-v2.patch, > HBASE-17178-v3.patch, HBASE-17178-v4.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. > Review board: https://reviews.apache.org/r/54191/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707188#comment-15707188 ] Guanghao Zhang commented on HBASE-17178: Review board: https://reviews.apache.org/r/54191/ bq. Move this line out of synchronized Fixed in v4 patch. bq. Shall the balancing be affected by other RIT? Assuming RS crash happened in middle of balancing, shall we wait? Yes, balancing will be affected by other RIT. This is for availability. If RS crash happend in middle of balancing, there will be more regions in transition. Then the balancer can't finish all region plans. The cluster need a next round balance to reach a balance state. bq. the code flow of balancer might block here and not controlled by the cutoffTime? Fixed in v4 patch. It need break the sleep when exceeds cutoff time. Review board: https://reviews.apache.org/r/54191/ > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-v1.patch, HBASE-17178-v2.patch, > HBASE-17178-v3.patch, HBASE-17178-v4.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. > Review board: https://reviews.apache.org/r/54191/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17201) Edit of HFileBlock comments and javadoc
[
https://issues.apache.org/jira/browse/HBASE-17201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707186#comment-15707186
]
Hadoop QA commented on HBASE-17201:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 27s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 20s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.io.hfile.TestChecksum |
| | hadoop.hbase.io.hfile.TestLazyDataBlockDecompression |
| | hadoop.hbase.io.hfile.TestHFileWriterV3 |
| | hadoop.hbase.io.hfile.TestHFileEncryption |
| | hadoop.hbase.regionserver.TestScanWithBloomError |
| | hadoop.hbase.io.hfile.TestHFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.2 Server=1.12.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840957/HBASE-17201.master.001.patch
|
| JIRA Issue | HBASE-17201 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f87354decf75 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18
16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c43a23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4694/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4694/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4694/testReport/ |
| modules
[jira] [Updated] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17178: --- Attachment: HBASE-17178-v4.patch > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-v1.patch, HBASE-17178-v2.patch, > HBASE-17178-v3.patch, HBASE-17178-v4.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any time. It helps a lot for our use case and it has > been running a long time > our production cluster. > But for some use case, we need the balancer run faster. If a cluster has 100 > regionservers, then it add 50 new regionservers for peak requests. Then it > need balancer run as soon as > possible and let the cluster reach a balance state soon. Our idea is compute > max number of regions in transition by the max balancing time and the average > time of region in transition. > Then the balancer use the computed value to throttling. > Examples for understanding. > A cluster has 100 regionservers, each regionserver has 200 regions and the > average time of region in transition is 1 seconds, we config the max > balancing time is 10 * 60 seconds. > Case 1. One regionserver crash, the cluster at most need balance 200 regions. > Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in > transition is 1 when balancing. Then the balancer can move region one by one > and the cluster will have high availability when balancing. > Case 2. Add other 100 regionservers, the cluster at most need balance 1 > regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of > regions in transition is 17 when balancing. Then the cluster can reach a > balance state within the max balancing time. > Any suggestions are welcomed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17178) Add region balance throttling
[ https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17178: --- Description: Our online cluster serves dozens of tables and different tables serve for different services. If the balancer moves too many regions in the same time, it will decrease the availability for some table or some services. So we add region balance throttling on our online serve cluster. We introduce a new config hbase.balancer.max.balancing.regions, which means the max number of regions in transition when balancing. If we config this to 1 and a table have 100 regions, then the table will have 99 regions available at any time. It helps a lot for our use case and it has been running a long time our production cluster. But for some use case, we need the balancer run faster. If a cluster has 100 regionservers, then it add 50 new regionservers for peak requests. Then it need balancer run as soon as possible and let the cluster reach a balance state soon. Our idea is compute max number of regions in transition by the max balancing time and the average time of region in transition. Then the balancer use the computed value to throttling. Examples for understanding. A cluster has 100 regionservers, each regionserver has 200 regions and the average time of region in transition is 1 seconds, we config the max balancing time is 10 * 60 seconds. Case 1. One regionserver crash, the cluster at most need balance 200 regions. Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in transition is 1 when balancing. Then the balancer can move region one by one and the cluster will have high availability when balancing. Case 2. Add other 100 regionservers, the cluster at most need balance 1 regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of regions in transition is 17 when balancing. Then the cluster can reach a balance state within the max balancing time. Any suggestions are welcomed. Review board: https://reviews.apache.org/r/54191/ was: Our online cluster serves dozens of tables and different tables serve for different services. If the balancer moves too many regions in the same time, it will decrease the availability for some table or some services. So we add region balance throttling on our online serve cluster. We introduce a new config hbase.balancer.max.balancing.regions, which means the max number of regions in transition when balancing. If we config this to 1 and a table have 100 regions, then the table will have 99 regions available at any time. It helps a lot for our use case and it has been running a long time our production cluster. But for some use case, we need the balancer run faster. If a cluster has 100 regionservers, then it add 50 new regionservers for peak requests. Then it need balancer run as soon as possible and let the cluster reach a balance state soon. Our idea is compute max number of regions in transition by the max balancing time and the average time of region in transition. Then the balancer use the computed value to throttling. Examples for understanding. A cluster has 100 regionservers, each regionserver has 200 regions and the average time of region in transition is 1 seconds, we config the max balancing time is 10 * 60 seconds. Case 1. One regionserver crash, the cluster at most need balance 200 regions. Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in transition is 1 when balancing. Then the balancer can move region one by one and the cluster will have high availability when balancing. Case 2. Add other 100 regionservers, the cluster at most need balance 1 regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of regions in transition is 17 when balancing. Then the cluster can reach a balance state within the max balancing time. Any suggestions are welcomed. > Add region balance throttling > - > > Key: HBASE-17178 > URL: https://issues.apache.org/jira/browse/HBASE-17178 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17178-v1.patch, HBASE-17178-v2.patch, > HBASE-17178-v3.patch, HBASE-17178-v4.patch > > > Our online cluster serves dozens of tables and different tables serve for > different services. If the balancer moves too many regions in the same time, > it will decrease the availability for some table or some services. So we add > region balance throttling on our online serve cluster. > We introduce a new config hbase.balancer.max.balancing.regions, which means > the max number of regions in transition when balancing. > If we config this to 1 and a table have 100 regions, then the table will have > 99 regions available at any
[jira] [Updated] (HBASE-17181) Let HBase thrift2 support TThreadedSelectorServer
[ https://issues.apache.org/jira/browse/HBASE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Yi updated HBASE-17181: Attachment: HBASE-17181-V6.patch HBASE-17181 Let HBase thrift2 support TThreadedSelectorServer > Let HBase thrift2 support TThreadedSelectorServer > - > > Key: HBASE-17181 > URL: https://issues.apache.org/jira/browse/HBASE-17181 > Project: HBase > Issue Type: New Feature > Components: Thrift >Affects Versions: 2.0.0, 1.4.0, 1.2.4 >Reporter: Jian Yi >Assignee: Jian Yi >Priority: Minor > Labels: features > Fix For: 2.0.0, 1.4.0, 1.2.5 > > Attachments: HBASE-17181-V1.patch, HBASE-17181-V2.patch, > HBASE-17181-V3.patch, HBASE-17181-V4.patch, HBASE-17181-V5.patch, > HBASE-17181-V6.patch, ThriftServer.java > > Original Estimate: 2h > Remaining Estimate: 2h > > Add TThreadedSelectorServer for HBase Thrift2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17181) Let HBase thrift2 support TThreadedSelectorServer
[ https://issues.apache.org/jira/browse/HBASE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707146#comment-15707146 ] Jian Yi commented on HBASE-17181: - empty path file, using: git format-patch HEAD^ --stdout > HBASE-17181-vx.patch use dev-support\submit-patch.py is ok: git add hbase-thrift\src\main\java\org\apache\hadoop\hbase\thrift2\ThriftServer.java git commit -m "HBASE-17181 Let HBase thrift2 support TThreadedSelectorServer" dev-support\submit-patch.py > Let HBase thrift2 support TThreadedSelectorServer > - > > Key: HBASE-17181 > URL: https://issues.apache.org/jira/browse/HBASE-17181 > Project: HBase > Issue Type: New Feature > Components: Thrift >Affects Versions: 2.0.0, 1.4.0, 1.2.4 >Reporter: Jian Yi >Assignee: Jian Yi >Priority: Minor > Labels: features > Fix For: 2.0.0, 1.4.0, 1.2.5 > > Attachments: HBASE-17181-V1.patch, HBASE-17181-V2.patch, > HBASE-17181-V3.patch, HBASE-17181-V4.patch, HBASE-17181-V5.patch, > ThriftServer.java > > Original Estimate: 2h > Remaining Estimate: 2h > > Add TThreadedSelectorServer for HBase Thrift2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16941) FavoredNodes - Split/Merge code paths
[
https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707121#comment-15707121
]
Hadoop QA commented on HBASE-16941:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 2s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m 13s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 9s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840940/HBASE-16941.master.009.patch
|
| JIRA Issue | HBASE-16941 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux bad7c512c3af 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c43a23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4692/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4692/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> FavoredNodes - Split/Merge code paths
>
[jira] [Commented] (HBASE-17200) Document an interesting implication of HBASE-15212
[ https://issues.apache.org/jira/browse/HBASE-17200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707108#comment-15707108 ] Gary Helmling commented on HBASE-17200: --- Yeah, we saw the same thing in HBASE-16752. There [~ashu210890] fixed the error handling so that the problem would be clearer to the client (replication source), but agree that explicitly documenting this would be good. > Document an interesting implication of HBASE-15212 > -- > > Key: HBASE-17200 > URL: https://issues.apache.org/jira/browse/HBASE-17200 > Project: HBase > Issue Type: Bug > Components: documentation, Operability, Replication >Reporter: Andrew Purtell >Priority: Minor > > We had a Phoenix client application unfortunately batching up 1000 rows at a > time. Phoenix bundles mutations up only considering the row count not byte > count (see PHOENIX-541) so this lead to some *single WALEdits* in excess of > 600 MB. A cluster without max RPC size enforcement accepted them. (That may > be something we should fix - WALEdits that large are crazy.) Then replication > workers attempting to ship the monster edits from this cluster to a remote > cluster recently upgraded with RPC size enforcement active would see all > their RPC attempts rejected, because the default limit is 256 MB. > This is an edge case but I can see it happening in practice and taking users > by surprise, most likely when replicating between mixed versions. We should > document this in the troubleshooting section. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16668) Admin class should have a synchronous Admin#mergeRegions* method
[
https://issues.apache.org/jira/browse/HBASE-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707106#comment-15707106
]
Stephen Yuan Jiang commented on HBASE-16668:
As a side effect :-) of HBASE-14552, I created a synchronous version of merge
region - {{HBaseAdmin#mergeRegionsSync() }} - for testing-only in HBaseAdmin
class
{code}
/**
* Merge two regions. Synchronous operation.
* Note: It is not feasible to predict the length of merge.
* Therefore, this is for internal testing only.
* @param nameOfRegionA encoded or full name of region a
* @param nameOfRegionB encoded or full name of region b
* @param forcible true if do a compulsory merge, otherwise we will only merge
* two adjacent regions
* @throws IOException
*/
@VisibleForTesting
public void mergeRegionsSync(
final byte[] nameOfRegionA,
final byte[] nameOfRegionB,
final boolean forcible) throws IOException {
get(
mergeRegionsAsync(nameOfRegionA, nameOfRegionB, forcible),
syncWaitTimeout,
TimeUnit.MILLISECONDS);
}
{code}
> Admin class should have a synchronous Admin#mergeRegions* method
> -
>
> Key: HBASE-16668
> URL: https://issues.apache.org/jira/browse/HBASE-16668
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
> Fix For: 2.0.0
>
>
> In trunk from HBASE-14552, we have deprecated {{void Admin#mergeRegions}} (in
> 1.x this was an asynchronous call) and replaced it with {{Future
> Admin#mergeRegionsAsync}} which is clearly async.
> This leaves us only with the async version.
> We should have an easy way to make {{mergeRegions}} or an equivalant behave
> synchronously.
> For normal java Futures, we could just call the future's {{get()}} method.
> Unforutnately, the future this method returns doesn't follow java Future
> convention and throws Unimplemented operation when a plain {{get()}} is
> called and makes the api harder to use and read. We could make this future
> act more normally, and have the timeout throw an InterruptedException.
> Alternately, we could expose a new method in {{Admin}} that behaves
> synchronously such as {{HBaseAdmin#mergeRegionsSync}}. The caveat here is
> that we shouldn't use the name {{#mergeRegions}} since it exists in 1.x with
> async semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17151) New API to create HFile.Reader without instantiating block cache
[ https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-17151: -- Attachment: HBASE-17151-v3.patch v3. > New API to create HFile.Reader without instantiating block cache > - > > Key: HBASE-17151 > URL: https://issues.apache.org/jira/browse/HBASE-17151 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch, > HBASE-17151-v3.patch > > > Currently, to create HFile.Reader instance, the CacheConfig instance is > required (which instantiates block cache). We need API for reading HFile w/o > block cache being involved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17172) Optimize major mob compaction with _del files
[
https://issues.apache.org/jira/browse/HBASE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707080#comment-15707080
]
huaxiang sun commented on HBASE-17172:
--
Hi [~jingcheng.du] and [~anoop.hbase], just did more code reading and found
that _del files can be included in minor mob compaction when the file size is
less than the threshold. Assume that user sets a high threshold value, even for
already compacted-files, it can be included in the compact list again and be
compacted with the del files. If we want to deal with _del files mainly in
major mob compaction. Can we skip these already-compacted files in the minor
compaction? something like in the select() after files are added to
filesToCompact map. This is to speed up minor compaction with del files.
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java
index 33aecc0..dab05d2 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java
@@ -25,6 +25,7 @@ import java.util.Collection;
import java.util.Collections;
import java.util.Date;
import java.util.HashMap;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
@@ -179,6 +180,23 @@ public class PartitionedMobCompactor extends MobCompactor {
selectedFileCount++;
}
}
+
+/*
+ * If it is not a major mob compaction with del files, and the file number
in Partition is 1,
+ * remove the partition from filesToCompact list to avoid re-compacting
files which has been
+ * compacted with del files.
+ */
+if (!allFiles && (allDelFiles.size() > 0)) {
+ for(Iterator> it =
+ filesToCompact.entrySet().iterator(); it.hasNext(); ) {
+Map.Entry entry =
it.next();
+if (entry.getValue().getFileNumbers() <= 1) {
+ it.remove();
+ --selectedFileCount;
+}
+ }
+}
+
PartitionedMobCompactionRequest request = new
PartitionedMobCompactionRequest(
filesToCompact.values(), allDelFiles);
if (candidates.size() == (allDelFiles.size() + selectedFileCount +
irrelevantFileCount)) {
{code}
> Optimize major mob compaction with _del files
> -
>
> Key: HBASE-17172
> URL: https://issues.apache.org/jira/browse/HBASE-17172
> Project: HBase
> Issue Type: Improvement
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>
> Today, when there is a _del file in mobdir, with major mob compaction, every
> mob file will be recompacted, this causes lots of IO and slow down major mob
> compaction (may take months to finish). This needs to be improved. A few
> ideas are:
> 1) Do not compact all _del files into one, instead, compact them based on
> groups with startKey as the key. Then use firstKey/startKey to make each mob
> file to see if the _del file needs to be included for this partition.
> 2). Based on the timerange of the _del file, compaction for files after that
> timerange does not need to include the _del file as these are newer files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17151) New API to create HFile.Reader without instantiating block cache
[
https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707071#comment-15707071
]
Hadoop QA commented on HBASE-17151:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 4s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 4s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 144m 52s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840935/HBASE-17151-v2.patch |
| JIRA Issue | HBASE-17151 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux adc3ba03defc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c43a23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4691/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4691/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> New API to create HFile.Reader without instantiating block cache
> -
>
> Key: HBASE-17151
> URL: https://issues.apache.org/jira/browse/HBASE-17151
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch
>
[jira] [Commented] (HBASE-17151) New API to create HFile.Reader without instantiating block cache
[
https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707051#comment-15707051
]
Vladimir Rodionov commented on HBASE-17151:
---
{quote}
How about doing a static CacheConfig.DISABLED, and use it when creating a
reader. Doing null checks everywhere unnecessarily complicates the code.
{quote}
OK.
> New API to create HFile.Reader without instantiating block cache
> -
>
> Key: HBASE-17151
> URL: https://issues.apache.org/jira/browse/HBASE-17151
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch
>
>
> Currently, to create HFile.Reader instance, the CacheConfig instance is
> required (which instantiates block cache). We need API for reading HFile w/o
> block cache being involved.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15704) Refactoring: Move HFileArchiver from backup to its own package
[ https://issues.apache.org/jira/browse/HBASE-15704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15704: -- Attachment: HBASE-15704-v3.patch v3. Removes example package and renames (moves) HFileArchiver. cc: [~enis] > Refactoring: Move HFileArchiver from backup to its own package > -- > > Key: HBASE-15704 > URL: https://issues.apache.org/jira/browse/HBASE-15704 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15704-v2.patch, HBASE-15704-v3.patch > > > This class is in backup package (as well as backup/examples classes) but is > not backup - related. Move examples classes to hbase-examples package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17151) New API to create HFile.Reader without instantiating block cache
[
https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707038#comment-15707038
]
Hadoop QA commented on HBASE-17151:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 50s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 49s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 27s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840935/HBASE-17151-v2.patch |
| JIRA Issue | HBASE-17151 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0e1dee782360 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7c43a23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4690/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4690/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> New API to create HFile.Reader without instantiating block cache
> -
>
> Key: HBASE-17151
> URL: https://issues.apache.org/jira/browse/HBASE-17151
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch
>
[jira] [Commented] (HBASE-17197) hfile does not work in 2.0
[
https://issues.apache.org/jira/browse/HBASE-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707033#comment-15707033
]
stack commented on HBASE-17197:
---
Seems silly having to pass file w/ a -f
> hfile does not work in 2.0
> --
>
> Key: HBASE-17197
> URL: https://issues.apache.org/jira/browse/HBASE-17197
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>
> I tried to use hfile in master branch, it does not print out kv pairs or meta
> as it is supposed to be.
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/
> 53e9f9bc328f468b87831221de3a0c74 bdc6e1c4eea246a99e989e02d554cb03
> bf9275ac418d4d458904d81137e82683
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -m
> 2016-11-29 12:25:22,019 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase hfile
> file:///Users/hsun/work/local-hbase-cluster/data/data/default/t1/755b5d7a44148492b7138c79c5e4f39f/f1/bf9275ac418d4d458904d81137e82683
> -p
> 2016-11-29 12:25:27,333 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> Scanned kv count -> 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17151) New API to create HFile.Reader without instantiating block cache
[ https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707026#comment-15707026 ] Enis Soztutar commented on HBASE-17151: --- How about doing a static CacheConfig.DISABLED, and use it when creating a reader. Doing null checks everywhere unnecessarily complicates the code. > New API to create HFile.Reader without instantiating block cache > - > > Key: HBASE-17151 > URL: https://issues.apache.org/jira/browse/HBASE-17151 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch > > > Currently, to create HFile.Reader instance, the CacheConfig instance is > required (which instantiates block cache). We need API for reading HFile w/o > block cache being involved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16295) InvalidFamilyOperationException while deleting a column family in shell
[
https://issues.apache.org/jira/browse/HBASE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashu Pachauri resolved HBASE-16295.
---
Resolution: Cannot Reproduce
> InvalidFamilyOperationException while deleting a column family in shell
> ---
>
> Key: HBASE-16295
> URL: https://issues.apache.org/jira/browse/HBASE-16295
> Project: HBase
> Issue Type: Bug
> Components: master, shell
>Affects Versions: 1.2.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>Priority: Minor
>
> The column family exists and is actually deleted, the regions are also
> reopened. But, the following exception is thrown in the shell:
> {code}
> alter 't1', 'delete' => 'cf'
> ERROR: org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'cf'
> does not exist, so it cannot be deleted
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
> at
> org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1916)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteColumn(MasterRpcServices.java:474)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55658)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:137)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:112)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.InvalidFamilyOperationException):
> Family 'cf' does not exist, so it cannot be deleted
> at
> org.apache.hadoop.hbase.master.procedure.DeleteColumnFamilyProcedure.prepareDelete(DeleteColumnFamilyProcedure.java:281)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteColumnFamilyProcedure.executeFromState(DeleteColumnFamilyProcedure.java:93)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteColumnFamilyProcedure.executeFromState(DeleteColumnFamilyProcedure.java:48)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:465)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1061)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:856)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:809)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:495)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16295) InvalidFamilyOperationException while deleting a column family in shell
[
https://issues.apache.org/jira/browse/HBASE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707012#comment-15707012
]
Ashu Pachauri commented on HBASE-16295:
---
I also tried reproducing this but without any luck. Since it has happened only
once and we don't have a repro, it seems super rare. I am going to close it.
> InvalidFamilyOperationException while deleting a column family in shell
> ---
>
> Key: HBASE-16295
> URL: https://issues.apache.org/jira/browse/HBASE-16295
> Project: HBase
> Issue Type: Bug
> Components: master, shell
>Affects Versions: 1.2.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>Priority: Minor
>
> The column family exists and is actually deleted, the regions are also
> reopened. But, the following exception is thrown in the shell:
> {code}
> alter 't1', 'delete' => 'cf'
> ERROR: org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'cf'
> does not exist, so it cannot be deleted
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
> at
> org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1916)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteColumn(MasterRpcServices.java:474)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55658)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:137)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:112)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.InvalidFamilyOperationException):
> Family 'cf' does not exist, so it cannot be deleted
> at
> org.apache.hadoop.hbase.master.procedure.DeleteColumnFamilyProcedure.prepareDelete(DeleteColumnFamilyProcedure.java:281)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteColumnFamilyProcedure.executeFromState(DeleteColumnFamilyProcedure.java:93)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteColumnFamilyProcedure.executeFromState(DeleteColumnFamilyProcedure.java:48)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:465)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1061)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:856)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:809)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:495)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-16295) InvalidFamilyOperationException while deleting a column family in shell
[
https://issues.apache.org/jira/browse/HBASE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashu Pachauri reassigned HBASE-16295:
-
Assignee: Ashu Pachauri
> InvalidFamilyOperationException while deleting a column family in shell
> ---
>
> Key: HBASE-16295
> URL: https://issues.apache.org/jira/browse/HBASE-16295
> Project: HBase
> Issue Type: Bug
> Components: master, shell
>Affects Versions: 1.2.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>Priority: Minor
>
> The column family exists and is actually deleted, the regions are also
> reopened. But, the following exception is thrown in the shell:
> {code}
> alter 't1', 'delete' => 'cf'
> ERROR: org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'cf'
> does not exist, so it cannot be deleted
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:114)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:85)
> at
> org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1916)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteColumn(MasterRpcServices.java:474)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55658)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:137)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:112)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.InvalidFamilyOperationException):
> Family 'cf' does not exist, so it cannot be deleted
> at
> org.apache.hadoop.hbase.master.procedure.DeleteColumnFamilyProcedure.prepareDelete(DeleteColumnFamilyProcedure.java:281)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteColumnFamilyProcedure.executeFromState(DeleteColumnFamilyProcedure.java:93)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteColumnFamilyProcedure.executeFromState(DeleteColumnFamilyProcedure.java:48)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:465)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1061)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:856)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:809)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:495)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17185) Purge the seek of the next block reading HFileBlocks
[ https://issues.apache.org/jira/browse/HBASE-17185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706992#comment-15706992 ] stack commented on HBASE-17185: --- Did some study. The read-of-the-next-blocks header is used only in the rare case where we are loading metadata on file open. Metadata includes hfile indices themselves stored as blocks. In this opening case, we do not have an hfile index to get block lengths from (we do not have an index for the indices -- TODO). There are three or so metadata blocks in the normal case. We could double the seeks done for the file open case doing a seek for the header to get lengths and then body for each metablock and I think it'd be fine given these are not real 'seeks' but just read-forwards in an already loaded hdfs stream but I did not do the work to prove this assertion. Needs a bit of work comparing before and after. I looked at undoing the read-ahead into the next block for all but the startup case and it'd involve code duplication and would undo much of the simplification/benefit the attached patch brings. Putting aside for now until time to do the perf/resource compare (though in a subtask, have updated HFileBlock doc w/o changing functionality to inculcate findings of my study). > Purge the seek of the next block reading HFileBlocks > > > Key: HBASE-17185 > URL: https://issues.apache.org/jira/browse/HBASE-17185 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-17185.master.001.patch, HBASE-17185.patch > > > When we read HFileBlocks, we read the asked-for block AND the next block's > header which we add to a cache (see HBASE-17072). We do this extra read to > get the next block's length purportedly. This seek of the next block's header > complicates the HFileBlock construction (not to mind other consequences -- > again see HBASE-17072). > Study done in HBASE-17072 shows that we normally do not need this extra read > of the next block's header. In the usual case, the length of the block is > gotten from the hfile index. > A simplification of block reading can be done purging this extra header read. > We can also save some space in cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-16209: -- Attachment: HBASE-16209-addendum.v6.branch-1.patch This has been open for a while now. I can't repro the findbugs warning locally, and can't really see why it's complaining! HBASE-16209-addendum.v6.branch-1: Rebasing the patch to current branch-1. > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-addendum.patch, > HBASE-16209-addendum.v6.branch-1.patch, > HBASE-16209-branch-1-addendum-v2.patch, HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1-v3.patch, HBASE-16209-branch-1-v4.patch, > HBASE-16209-branch-1-v5.patch, HBASE-16209-branch-1.patch, > HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17185) Purge the seek of the next block reading HFileBlocks
[ https://issues.apache.org/jira/browse/HBASE-17185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17185: -- Labels: beginner (was: ) > Purge the seek of the next block reading HFileBlocks > > > Key: HBASE-17185 > URL: https://issues.apache.org/jira/browse/HBASE-17185 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-17185.master.001.patch, HBASE-17185.patch > > > When we read HFileBlocks, we read the asked-for block AND the next block's > header which we add to a cache (see HBASE-17072). We do this extra read to > get the next block's length purportedly. This seek of the next block's header > complicates the HFileBlock construction (not to mind other consequences -- > again see HBASE-17072). > Study done in HBASE-17072 shows that we normally do not need this extra read > of the next block's header. In the usual case, the length of the block is > gotten from the hfile index. > A simplification of block reading can be done purging this extra header read. > We can also save some space in cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17185) Purge the seek of the next block reading HFileBlocks
[ https://issues.apache.org/jira/browse/HBASE-17185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17185: -- Priority: Minor (was: Major) > Purge the seek of the next block reading HFileBlocks > > > Key: HBASE-17185 > URL: https://issues.apache.org/jira/browse/HBASE-17185 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Minor > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-17185.master.001.patch, HBASE-17185.patch > > > When we read HFileBlocks, we read the asked-for block AND the next block's > header which we add to a cache (see HBASE-17072). We do this extra read to > get the next block's length purportedly. This seek of the next block's header > complicates the HFileBlock construction (not to mind other consequences -- > again see HBASE-17072). > Study done in HBASE-17072 shows that we normally do not need this extra read > of the next block's header. In the usual case, the length of the block is > gotten from the hfile index. > A simplification of block reading can be done purging this extra header read. > We can also save some space in cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17201) Edit of HFileBlock comments and javadoc
[ https://issues.apache.org/jira/browse/HBASE-17201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17201: -- Affects Version/s: 2.0.0 Status: Patch Available (was: Open) > Edit of HFileBlock comments and javadoc > --- > > Key: HBASE-17201 > URL: https://issues.apache.org/jira/browse/HBASE-17201 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-17201.master.001.patch > > > Spent time in HFileBlock trying to do the parent issue. Failed. But did a > bunch of edits of comments/javadoc. Let me get those in at least. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17201) Edit of HFileBlock comments and javadoc
[ https://issues.apache.org/jira/browse/HBASE-17201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17201: -- Attachment: HBASE-17201.master.001.patch > Edit of HFileBlock comments and javadoc > --- > > Key: HBASE-17201 > URL: https://issues.apache.org/jira/browse/HBASE-17201 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-17201.master.001.patch > > > Spent time in HFileBlock trying to do the parent issue. Failed. But did a > bunch of edits of comments/javadoc. Let me get those in at least. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17201) Edit of HFileBlock comments and javadoc
stack created HBASE-17201: - Summary: Edit of HFileBlock comments and javadoc Key: HBASE-17201 URL: https://issues.apache.org/jira/browse/HBASE-17201 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: stack Spent time in HFileBlock trying to do the parent issue. Failed. But did a bunch of edits of comments/javadoc. Let me get those in at least. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16904) remove directory layout / fs references from snapshots
[
https://issues.apache.org/jira/browse/HBASE-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706926#comment-15706926
]
Umesh Agashe commented on HBASE-16904:
--
+1
> remove directory layout / fs references from snapshots
> --
>
> Key: HBASE-16904
> URL: https://issues.apache.org/jira/browse/HBASE-16904
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>Assignee: Umesh Agashe
> Attachments: HBASE-16904-hbase-14439.v1.patch,
> HBASE-16904-hbase-14439.v2.patch, HBASE-16904-hbase-14439.v3.patch,
> HBASE-16904-hbase-14439.v4.patch
>
>
> ensure snapshot code works through the MasterStorage / RegionStorage APIs and
> not directly on backing filesystem.
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV1.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV2.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17192) remove use of scala-tools.org from pom
[
https://issues.apache.org/jira/browse/HBASE-17192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706896#comment-15706896
]
Andrew Purtell commented on HBASE-17192:
+1
Yuck
> remove use of scala-tools.org from pom
> --
>
> Key: HBASE-17192
> URL: https://issues.apache.org/jira/browse/HBASE-17192
> Project: HBase
> Issue Type: Bug
> Components: spark, website
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-17192.1.patch, HBASE-17192.1.test.patch
>
>
> our pom makes use of scala-tools.org for a repository. That domain currently
> issues redirects for all URLs; for maven coordinates those redirects lead to
> 'not found' and the 'permantenly moved' HTML gets saved. this corrupts the
> local maven repository in a way that cause the mvn:site goal to give an
> opaque error:
> {code}
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 01:46 min
> [INFO] Finished at: 2016-11-28T14:17:10+00:00
> [INFO] Final Memory: 292M/6583M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.4:site (default-site) on project
> hbase: Execution default-site of goal
> org.apache.maven.plugins:maven-site-plugin:3.4:site failed: For artifact
> {null:null:null:jar}: The groupId cannot be empty. -> [Help 1]
> [ERROR]
> {code}
> Rerunning in debug mode with {{mvn -X}} gives no additional useful
> information.
> All artifacts from scala-tools.org are now found in maven central.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16904) remove directory layout / fs references from snapshots
[
https://issues.apache.org/jira/browse/HBASE-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706887#comment-15706887
]
Umesh Agashe commented on HBASE-16904:
--
Thanks for reviewing the changes [~busbey]! The addendum changes looks good to
me.
> remove directory layout / fs references from snapshots
> --
>
> Key: HBASE-16904
> URL: https://issues.apache.org/jira/browse/HBASE-16904
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>Assignee: Umesh Agashe
> Attachments: HBASE-16904-hbase-14439.v1.patch,
> HBASE-16904-hbase-14439.v2.patch, HBASE-16904-hbase-14439.v3.patch,
> HBASE-16904-hbase-14439.v4.patch
>
>
> ensure snapshot code works through the MasterStorage / RegionStorage APIs and
> not directly on backing filesystem.
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV1.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV2.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17200) Document an interesting implication of HBASE-15212
Andrew Purtell created HBASE-17200: -- Summary: Document an interesting implication of HBASE-15212 Key: HBASE-17200 URL: https://issues.apache.org/jira/browse/HBASE-17200 Project: HBase Issue Type: Bug Components: documentation, Operability, Replication Reporter: Andrew Purtell Priority: Minor We had a Phoenix client application unfortunately batching up 1000 rows at a time. Phoenix bundles mutations up only considering the row count not byte count (see PHOENIX-541) so this lead to some *single WALEdits* in excess of 600 MB. A cluster without max RPC size enforcement accepted them. (That may be something we should fix - WALEdits that large are crazy.) Then replication workers attempting to ship the monster edits from this cluster to a remote cluster recently upgraded with RPC size enforcement active would see all their RPC attempts rejected, because the default limit is 256 MB. This is an edge case but I can see it happening in practice and taking users by surprise, most likely when replicating between mixed versions. We should document this in the troubleshooting section. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17199) Back-port HBASE-17151 to HBASE-7912 branch
[ https://issues.apache.org/jira/browse/HBASE-17199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-17199: -- Description: HBASE-17151 introduces new API to read HFile w/o instantiating block cache. > Back-port HBASE-17151 to HBASE-7912 branch > -- > > Key: HBASE-17199 > URL: https://issues.apache.org/jira/browse/HBASE-17199 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-17199-v1.patch > > > HBASE-17151 introduces new API to read HFile w/o instantiating block cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17199) Back-port HBASE-17151 to HBASE-7912 branch
[
https://issues.apache.org/jira/browse/HBASE-17199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706840#comment-15706840
]
Hadoop QA commented on HBASE-17199:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s {color}
| {color:red} HBASE-17199 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840945/HBASE-17199-v1.patch |
| JIRA Issue | HBASE-17199 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4693/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Back-port HBASE-17151 to HBASE-7912 branch
> --
>
> Key: HBASE-17199
> URL: https://issues.apache.org/jira/browse/HBASE-17199
> Project: HBase
> Issue Type: Task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-17199-v1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17199) Back-port HBASE-17151 to HBASE-7912 branch
[ https://issues.apache.org/jira/browse/HBASE-17199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-17199: -- Status: Patch Available (was: Open) > Back-port HBASE-17151 to HBASE-7912 branch > -- > > Key: HBASE-17199 > URL: https://issues.apache.org/jira/browse/HBASE-17199 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-17199-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17199) Back-port HBASE-17151 to HBASE-7912 branch
[ https://issues.apache.org/jira/browse/HBASE-17199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-17199: -- Attachment: HBASE-17199-v1.patch Patch v1. cc: [~enis], [~tedyu] > Back-port HBASE-17151 to HBASE-7912 branch > -- > > Key: HBASE-17199 > URL: https://issues.apache.org/jira/browse/HBASE-17199 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-17199-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17199) Back-port HBASE-17151 to HBASE-7912 branch
Vladimir Rodionov created HBASE-17199: - Summary: Back-port HBASE-17151 to HBASE-7912 branch Key: HBASE-17199 URL: https://issues.apache.org/jira/browse/HBASE-17199 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17196) deleted mob cell can come back after major compaction and minor mob compaction
[
https://issues.apache.org/jira/browse/HBASE-17196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706800#comment-15706800
]
huaxiang sun commented on HBASE-17196:
--
I rechecked the code again. I think my testing is wrong as I removed these _del
files before doing minor mob compaction. The logic in code will always include
_del files if a file needs to be compacted so those deleted cells will never
come back. I am resolving the issue as invalid.
> deleted mob cell can come back after major compaction and minor mob compaction
> --
>
> Key: HBASE-17196
> URL: https://issues.apache.org/jira/browse/HBASE-17196
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>
> In the following case, the deleted mob cell can come back.
> {code}
> 1) hbase(main):001:0> create 't1', {NAME => 'f1', IS_MOB => true,
> MOB_THRESHOLD => 10}
> 2) hbase(main):002:0> put 't1', 'r1', 'f1:q1', ''
> 3) hbase(main):003:0> flush 't1'
> 4) hbase(main):004:0> deleteall 't1', 'r1'
> 5) hbase(main):005:0> scan 't1'
> ROW COLUMN+CELL
>
>
> 0 row(s)
> 6) hbase(main):006:0> flush 't1'
> 7) hbase(main):007:0> major_compact 't1'
> After that, go to mobdir, remove the _del file, this is to simulate the case
> that mob minor compaction does not the _del file. Right now, the cell in
> normal region is gone after the major compaction.
> 8) hbase(main):008:0> put 't1', 'r2', 'f1:q1', ''
>
>
> 9) hbase(main):009:0> flush 't1'
> 10) hbase(main):010:0> scan 't1'
> ROW COLUMN+CELL
>
>
> r2 column=f1:q1,
> timestamp=1480451201393, value=
>
> 1 row(s)
> 11) hbase(main):011:0> compact 't1', 'f1', 'MOB'
> 12) hbase(main):012:0> scan 't1'
> ROW COLUMN+CELL
>
>
> r1 column=f1:q1,
> timestamp=1480450987725, value=
>
> r2 column=f1:q1,
> timestamp=1480451201393, value=
>
> 2 row(s)
> The deleted "r1" comes back. The reason is that mob minor compaction does not
> include _del files so it generates references for the deleted cell.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-17196) deleted mob cell can come back after major compaction and minor mob compaction
[
https://issues.apache.org/jira/browse/HBASE-17196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huaxiang sun resolved HBASE-17196.
--
Resolution: Invalid
As explained in the comments.
> deleted mob cell can come back after major compaction and minor mob compaction
> --
>
> Key: HBASE-17196
> URL: https://issues.apache.org/jira/browse/HBASE-17196
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>
> In the following case, the deleted mob cell can come back.
> {code}
> 1) hbase(main):001:0> create 't1', {NAME => 'f1', IS_MOB => true,
> MOB_THRESHOLD => 10}
> 2) hbase(main):002:0> put 't1', 'r1', 'f1:q1', ''
> 3) hbase(main):003:0> flush 't1'
> 4) hbase(main):004:0> deleteall 't1', 'r1'
> 5) hbase(main):005:0> scan 't1'
> ROW COLUMN+CELL
>
>
> 0 row(s)
> 6) hbase(main):006:0> flush 't1'
> 7) hbase(main):007:0> major_compact 't1'
> After that, go to mobdir, remove the _del file, this is to simulate the case
> that mob minor compaction does not the _del file. Right now, the cell in
> normal region is gone after the major compaction.
> 8) hbase(main):008:0> put 't1', 'r2', 'f1:q1', ''
>
>
> 9) hbase(main):009:0> flush 't1'
> 10) hbase(main):010:0> scan 't1'
> ROW COLUMN+CELL
>
>
> r2 column=f1:q1,
> timestamp=1480451201393, value=
>
> 1 row(s)
> 11) hbase(main):011:0> compact 't1', 'f1', 'MOB'
> 12) hbase(main):012:0> scan 't1'
> ROW COLUMN+CELL
>
>
> r1 column=f1:q1,
> timestamp=1480450987725, value=
>
> r2 column=f1:q1,
> timestamp=1480451201393, value=
>
> 2 row(s)
> The deleted "r1" comes back. The reason is that mob minor compaction does not
> include _del files so it generates references for the deleted cell.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16941: - Status: Patch Available (was: Open) > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch, HBASE-16941.master.003.patch, > HBASE-16941.master.004.patch, HBASE-16941.master.005.patch, > HBASE-16941.master.006.patch, HBASE-16941.master.007.patch, > HBASE-16941.master.008.patch, HBASE-16941.master.009.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. The design document can be seen at HBASE-15531. The specific > changes are: > Split and merged regions should inherit favored node information from parent > regions. For splits also include some randomness so even if there are > subsequent splits, the regions will be more or less distributed. For split, > we include 2 FN from the parent and generate one random node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16941: - Attachment: HBASE-16941.master.009.patch > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch, HBASE-16941.master.003.patch, > HBASE-16941.master.004.patch, HBASE-16941.master.005.patch, > HBASE-16941.master.006.patch, HBASE-16941.master.007.patch, > HBASE-16941.master.008.patch, HBASE-16941.master.009.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. The design document can be seen at HBASE-15531. The specific > changes are: > Split and merged regions should inherit favored node information from parent > regions. For splits also include some randomness so even if there are > subsequent splits, the regions will be more or less distributed. For split, > we include 2 FN from the parent and generate one random node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16941: - Status: Open (was: Patch Available) Looks like https://builds.apache.org/view/All/job/PreCommit-HBASE-Build/4685/console failed, unrelated to this patch. Couple of other builds also failed around this time. Will reupload. > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch, HBASE-16941.master.003.patch, > HBASE-16941.master.004.patch, HBASE-16941.master.005.patch, > HBASE-16941.master.006.patch, HBASE-16941.master.007.patch, > HBASE-16941.master.008.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. The design document can be seen at HBASE-15531. The specific > changes are: > Split and merged regions should inherit favored node information from parent > regions. For splits also include some randomness so even if there are > subsequent splits, the regions will be more or less distributed. For split, > we include 2 FN from the parent and generate one random node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17128) Find Cause of a Write Perf Regression in branch-1.2
[
https://issues.apache.org/jira/browse/HBASE-17128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706747#comment-15706747
]
Appy commented on HBASE-17128:
--
Nice, that looks good [~gbaecher], keep us posted.
Btw, here's a copy of doc mentioned above from my personal account so that it's
publicly visible
(https://docs.google.com/document/d/1DDWiA0ZVYRpvLk-uImDajXbG1d_XUN-xhC1CKBOcd8g/edit?usp=sharing).
> Find Cause of a Write Perf Regression in branch-1.2
> ---
>
> Key: HBASE-17128
> URL: https://issues.apache.org/jira/browse/HBASE-17128
> Project: HBase
> Issue Type: Task
>Reporter: stack
>
> As reported by [~gbaecher] up on the mailing list, there is a regression in
> 1.2. The regression is in a CDH version of 1.2 actually but the CDH hbase is
> a near pure 1.2. This is a working issue to figure which of the below changes
> brought on slower writes (The list comes from doing the following...git log
> --oneline
> remotes/origin/cdh5-1.2.0_5.8.0_dev..remotes/origin/cdh5-1.2.0_5.9.0_dev ...
> I stripped the few CDH specific changes, packaging and tagging only, and then
> made two groupings; candidates and the unlikelies):
> {code}
> 1 bbc6762 HBASE-16023 Fastpath for the FIFO rpcscheduler Adds an executor
> that does balanced queue and fast path handing off requests directly to
> waiting handlers if any present. Idea taken from Apace Kudu (incubating). See
> https://gerr#
> 2 a260917 HBASE-16288 HFile intermediate block level indexes might recurse
> forever creating multi TB files
> 3 5633281 HBASE-15811 Batch Get after batch Put does not fetch all Cells We
> were not waiting on all executors in a batch to complete. The test for
> no-more-executors was damaged by the 0.99/0.98.4 fix "HBASE-11403 Fix race
> conditions aro#
> 4 780f720 HBASE-11625 - Verifies data before building HFileBlock. - Adds
> HFileBlock.Header class which contains information about location of fields.
> Testing: Adds CorruptedFSReaderImpl to TestChecksum. (Apekshit)
> 5 d735680 HBASE-12133 Add FastLongHistogram for metric computation (Yi Deng)
> 6 c4ee832 HBASE-15222 Use less contended classes for metrics
> 7
> 8 17320a4 HBASE-15683 Min latency in latency histograms are emitted as
> Long.MAX_VALUE
> 9 283b39f HBASE-15396 Enhance mapreduce.TableSplit to add encoded region
> name
> 10 39db592 HBASE-16195 Should not add chunk into chunkQueue if not using
> chunk pool in HeapMemStoreLAB
> 11 5ff28b7 HBASE-16194 Should count in MSLAB chunk allocation into heap size
> change when adding duplicate cells
> 12 5e3e0d2 HBASE-16318 fail build while rendering velocity template if
> dependency license isn't in whitelist.
> 13 3ed66e3 HBASE-16318 consistently use the correct name for 'Apache
> License, Version 2.0'
> 14 351832d HBASE-16340 exclude Xerces iplementation jars from coming in
> transitively.
> 15 b6aa4be HBASE-16321 ensure no findbugs-jsr305
> 16 4f9dde7 HBASE-16317 revert all ESAPI changes
> 17 71b6a8a HBASE-16284 Unauthorized client can shutdown the cluster (Deokwoo
> Han)
> 18 523753f HBASE-16450 Shell tool to dump replication queues
> 19 ca5f2ee HBASE-16379 [replication] Minor improvement to
> replication/copy_tables_desc.rb
> 20 effd105 HBASE-16135 PeerClusterZnode under rs of removed peer may never
> be deleted
> 21 a5c6610 HBASE-16319 Fix TestCacheOnWrite after HBASE-16288
> 22 1956bb0 HBASE-15808 Reduce potential bulk load intermediate space usage
> and waste
> 23 031c54e HBASE-16096 Backport. Cleanly remove replication peers from
> ZooKeeper.
> 24 60a3b12 HBASE-14963 Remove use of Guava Stopwatch from HBase client code
> (Devaraj Das)
> 25 c7724fc HBASE-16207 can't restore snapshot without "Admin" permission
> 26 8322a0b HBASE-16227 [Shell] Column value formatter not working in scans.
> Tested : manually using shell.
> 27 8f86658 HBASE-14818 user_permission does not list namespace permissions
> (li xiang)
> 28 775cd21 HBASE-15465 userPermission returned by getUserPermission() for
> the selected namespace does not have namespace set (li xiang)
> 29 8d85aff HBASE-16093 Fix splits failed before creating daughter regions
> leave meta inconsistent
> 30 bc41317 HBASE-16140 bump owasp.esapi from 2.1.0 to 2.1.0.1
> 31 6fc70cd HBASE-16035 Nested AutoCloseables might not all get closed (Sean
> Mackrory)
> 32 fe28fe84 HBASE-15891. Closeable resources potentially not getting closed
> if exception is thrown.
> 33 1d2bf3c HBASE-14644 Region in transition metric is broken -- addendum
> (Huaxiang Sun)
> 34 fd5f56c HBASE-16056 Procedure v2 - fix master crash for FileNotFound
> 35 10cd038 HBASE-16034 Fix ProcedureTestingUtility#LoadCounter.setMaxProcId()
> 36 dae4db4 HBASE-15872 Split TestWALProcedureStore
> 37 e638d86 HBASE-14644 Region in transition metric is broken (Huaxiang Sun)
>
[jira] [Commented] (HBASE-16904) remove directory layout / fs references from snapshots
[
https://issues.apache.org/jira/browse/HBASE-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706705#comment-15706705
]
Sean Busbey commented on HBASE-16904:
-
I'm a bit stumped on the javadoc warnings. they don't happen locally with
javadoc:javadoc.
> remove directory layout / fs references from snapshots
> --
>
> Key: HBASE-16904
> URL: https://issues.apache.org/jira/browse/HBASE-16904
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>Assignee: Umesh Agashe
> Attachments: HBASE-16904-hbase-14439.v1.patch,
> HBASE-16904-hbase-14439.v2.patch, HBASE-16904-hbase-14439.v3.patch,
> HBASE-16904-hbase-14439.v4.patch
>
>
> ensure snapshot code works through the MasterStorage / RegionStorage APIs and
> not directly on backing filesystem.
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV1.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV2.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17151) New API to create HFile.Reader without instantiating block cache
[ https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-17151: -- Fix Version/s: 2.0.0 > New API to create HFile.Reader without instantiating block cache > - > > Key: HBASE-17151 > URL: https://issues.apache.org/jira/browse/HBASE-17151 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch > > > Currently, to create HFile.Reader instance, the CacheConfig instance is > required (which instantiates block cache). We need API for reading HFile w/o > block cache being involved. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
