[jira] [Updated] (HBASE-20885) Remove entry for RPC quota from hbase:quota when RPC quota is removed.
[
https://issues.apache.org/jira/browse/HBASE-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi updated HBASE-20885:
---
Attachment: hbase-20885.master.002.patch
> Remove entry for RPC quota from hbase:quota when RPC quota is removed.
> --
>
> Key: HBASE-20885
> URL: https://issues.apache.org/jira/browse/HBASE-20885
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20885.master.001.patch,
> hbase-20885.master.002.patch
>
>
> When a RPC quota is removed (using LIMIT => 'NONE'), the entry from
> hbase:quota table is not completely removed. For e.g. see below:
> {noformat}
> hbase(main):005:0> create 't2','cf1'
> Created table t2
> Took 0.8000 seconds
> => Hbase::Table - t2
> hbase(main):006:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.1024 seconds
> hbase(main):007:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE =>
> REQUEST_SIZE, LIMIT => 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0622 seconds
> hbase(main):008:0> scan 'hbase:quota'
> ROWCOLUMN+CELL
> t.t2 column=q:s, timestamp=1531513014463,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80
>\x05 \x02
> 1 row(s)
> Took 0.0453 seconds
> hbase(main):009:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT => 'NONE'
> Took 0.0097 seconds
> hbase(main):010:0> list_quotas
> OWNER QUOTAS
> 0 row(s)
> Took 0.0338 seconds
> hbase(main):011:0> scan 'hbase:quota'
> ROWCOLUMN+CELL
> t.t2 column=q:s, timestamp=1531513039505,
> value=PBUF\x12\x00
> 1 row(s)
> Took 0.0066 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20885) Remove entry for RPC quota from hbase:quota when RPC quota is removed.
[
https://issues.apache.org/jira/browse/HBASE-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi updated HBASE-20885:
---
Attachment: hbase-20885.master.001.patch
> Remove entry for RPC quota from hbase:quota when RPC quota is removed.
> --
>
> Key: HBASE-20885
> URL: https://issues.apache.org/jira/browse/HBASE-20885
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20885.master.001.patch
>
>
> When a RPC quota is removed (using LIMIT => 'NONE'), the entry from
> hbase:quota table is not completely removed. For e.g. see below:
> {noformat}
> hbase(main):005:0> create 't2','cf1'
> Created table t2
> Took 0.8000 seconds
> => Hbase::Table - t2
> hbase(main):006:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.1024 seconds
> hbase(main):007:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE =>
> REQUEST_SIZE, LIMIT => 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0622 seconds
> hbase(main):008:0> scan 'hbase:quota'
> ROWCOLUMN+CELL
> t.t2 column=q:s, timestamp=1531513014463,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80
>\x05 \x02
> 1 row(s)
> Took 0.0453 seconds
> hbase(main):009:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT => 'NONE'
> Took 0.0097 seconds
> hbase(main):010:0> list_quotas
> OWNER QUOTAS
> 0 row(s)
> Took 0.0338 seconds
> hbase(main):011:0> scan 'hbase:quota'
> ROWCOLUMN+CELL
> t.t2 column=q:s, timestamp=1531513039505,
> value=PBUF\x12\x00
> 1 row(s)
> Took 0.0066 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20885) Remove entry for RPC quota from hbase:quota when RPC quota is removed.
[
https://issues.apache.org/jira/browse/HBASE-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi updated HBASE-20885:
---
Status: Patch Available (was: In Progress)
> Remove entry for RPC quota from hbase:quota when RPC quota is removed.
> --
>
> Key: HBASE-20885
> URL: https://issues.apache.org/jira/browse/HBASE-20885
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20885.master.001.patch
>
>
> When a RPC quota is removed (using LIMIT => 'NONE'), the entry from
> hbase:quota table is not completely removed. For e.g. see below:
> {noformat}
> hbase(main):005:0> create 't2','cf1'
> Created table t2
> Took 0.8000 seconds
> => Hbase::Table - t2
> hbase(main):006:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.1024 seconds
> hbase(main):007:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE =>
> REQUEST_SIZE, LIMIT => 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0622 seconds
> hbase(main):008:0> scan 'hbase:quota'
> ROWCOLUMN+CELL
> t.t2 column=q:s, timestamp=1531513014463,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80
>\x05 \x02
> 1 row(s)
> Took 0.0453 seconds
> hbase(main):009:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT => 'NONE'
> Took 0.0097 seconds
> hbase(main):010:0> list_quotas
> OWNER QUOTAS
> 0 row(s)
> Took 0.0338 seconds
> hbase(main):011:0> scan 'hbase:quota'
> ROWCOLUMN+CELL
> t.t2 column=q:s, timestamp=1531513039505,
> value=PBUF\x12\x00
> 1 row(s)
> Took 0.0066 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20885) Remove entry for RPC quota from hbase:quota when RPC quota is removed.
[
https://issues.apache.org/jira/browse/HBASE-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555343#comment-16555343
]
Sakthi edited comment on HBASE-20885 at 7/26/18 5:49 AM:
-
Testing done:
# The way of reproducing the issue, which is mentioned in the description, now
no longer produces the issue.
# Test case added: TestQuotaAdmin#testSetGetRemoveRPCQuota,
TestQuotaAdmin#testSetModifyRemoveRPCQuota
was (Author: jatsakthi):
Testing done:
# The way of reproducing the issue, which is mentioned in the description, now
no longer produces the issue.
# Test case added: TestQuotaAdmin#testSetGetRemoveRPCQuota
> Remove entry for RPC quota from hbase:quota when RPC quota is removed.
> --
>
> Key: HBASE-20885
> URL: https://issues.apache.org/jira/browse/HBASE-20885
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
>
> When a RPC quota is removed (using LIMIT => 'NONE'), the entry from
> hbase:quota table is not completely removed. For e.g. see below:
> {noformat}
> hbase(main):005:0> create 't2','cf1'
> Created table t2
> Took 0.8000 seconds
> => Hbase::Table - t2
> hbase(main):006:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.1024 seconds
> hbase(main):007:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE =>
> REQUEST_SIZE, LIMIT => 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0622 seconds
> hbase(main):008:0> scan 'hbase:quota'
> ROWCOLUMN+CELL
> t.t2 column=q:s, timestamp=1531513014463,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80
>\x05 \x02
> 1 row(s)
> Took 0.0453 seconds
> hbase(main):009:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT => 'NONE'
> Took 0.0097 seconds
> hbase(main):010:0> list_quotas
> OWNER QUOTAS
> 0 row(s)
> Took 0.0338 seconds
> hbase(main):011:0> scan 'hbase:quota'
> ROWCOLUMN+CELL
> t.t2 column=q:s, timestamp=1531513039505,
> value=PBUF\x12\x00
> 1 row(s)
> Took 0.0066 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20885) Remove entry for RPC quota from hbase:quota when RPC quota is removed.
[
https://issues.apache.org/jira/browse/HBASE-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555343#comment-16555343
]
Sakthi edited comment on HBASE-20885 at 7/26/18 5:49 AM:
-
Testing done:
# The way of reproducing the issue, which is mentioned in the description, now
no longer produces the issue.
# Test cases added: TestQuotaAdmin#testSetGetRemoveRPCQuota,
TestQuotaAdmin#testSetModifyRemoveRPCQuota
was (Author: jatsakthi):
Testing done:
# The way of reproducing the issue, which is mentioned in the description, now
no longer produces the issue.
# Test case added: TestQuotaAdmin#testSetGetRemoveRPCQuota,
TestQuotaAdmin#testSetModifyRemoveRPCQuota
> Remove entry for RPC quota from hbase:quota when RPC quota is removed.
> --
>
> Key: HBASE-20885
> URL: https://issues.apache.org/jira/browse/HBASE-20885
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
>
> When a RPC quota is removed (using LIMIT => 'NONE'), the entry from
> hbase:quota table is not completely removed. For e.g. see below:
> {noformat}
> hbase(main):005:0> create 't2','cf1'
> Created table t2
> Took 0.8000 seconds
> => Hbase::Table - t2
> hbase(main):006:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.1024 seconds
> hbase(main):007:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE =>
> REQUEST_SIZE, LIMIT => 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0622 seconds
> hbase(main):008:0> scan 'hbase:quota'
> ROWCOLUMN+CELL
> t.t2 column=q:s, timestamp=1531513014463,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80
>\x05 \x02
> 1 row(s)
> Took 0.0453 seconds
> hbase(main):009:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT => 'NONE'
> Took 0.0097 seconds
> hbase(main):010:0> list_quotas
> OWNER QUOTAS
> 0 row(s)
> Took 0.0338 seconds
> hbase(main):011:0> scan 'hbase:quota'
> ROWCOLUMN+CELL
> t.t2 column=q:s, timestamp=1531513039505,
> value=PBUF\x12\x00
> 1 row(s)
> Took 0.0066 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556910#comment-16556910
]
stack commented on HBASE-20936:
---
[~xucang] I can't repro this. I just reset my cluster w/ a patch to dump more
info but now the Exception is missing. I like your theory though, that the
array may edited elsewhere.
> BaseLoadBalancer throws ArrayIndexOutOfBoundsException
> --
>
> Key: HBASE-20936
> URL: https://issues.apache.org/jira/browse/HBASE-20936
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.1
>Reporter: stack
>Priority: Major
>
> I see the below in a test cluster. The balancer code has been in place a long
> time. Not sure why its doing the below. Will look later. Meantime, making a
> record of it.
> {code}
> 2018-07-24 22:03:43,261 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.47655219381977, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288571056819432);
> 2018-07-24 22:03:43,322 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 64
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:723)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:399)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1514)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1445)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-24 22:08:20,101 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=4.61 MB, freeSize=6.14 GB, max=6.14 GB,
> blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0,
> cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2018-07-24 22:08:38,101 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.4765419610035, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288550591186913);
> 2018-07-24 22:08:38,103 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 100
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
>
[jira] [Commented] (HBASE-20899) Add Hadoop KMS dependency and basic HDFS at-rest encryption tests
[
https://issues.apache.org/jira/browse/HBASE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556905#comment-16556905
]
Hadoop QA commented on HBASE-20899:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
31s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
13s{color} | {color:red} root: The patch generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 9s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}192m
54s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}253m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20899 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933119/HBASE-20899.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile findbugs hbaseanti checkstyle |
| uname | Linux 97e86805a83a 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Attachment: HBASE-20886.master.006.patch
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch,
> HBASE-20886.master.006.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20921) Possible NPE in ReopenTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-20921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556899#comment-16556899
]
Hadoop QA commented on HBASE-20921:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
52s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
52s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 3s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}153m
5s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:6f01af0 |
| JIRA Issue | HBASE-20921 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933126/HBASE-20921.branch-2.0.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 414a8905dfa7 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.0 / c11f0e47a6 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13800/testReport/ |
| Max. process+thread count | 3989 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13800/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically
[jira] [Commented] (HBASE-20932) Effective MemStoreSize::hashCode()
[
https://issues.apache.org/jira/browse/HBASE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556894#comment-16556894
]
Hudson commented on HBASE-20932:
Results for branch branch-2
[build #1027 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1027/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1027//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1027//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1027//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Effective MemStoreSize::hashCode()
> ---
>
> Key: HBASE-20932
> URL: https://issues.apache.org/jira/browse/HBASE-20932
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.2
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: HBASE-20932.001.patch, HBASE-20932.002.patch
>
>
> After HBASE-20411 we have
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = 31 * this.dataSize;
> h = h + 31 * this.heapSize;
> h = h + 31 * this.offHeapSize;
> return (int) h;
> }
> {code}
> This is not effective {{hashCode()}} implementation. Instead we can use:
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = this.dataSize;
> h = h * 31 + this.heapSize;
> h = h * 31 + this.offHeapSize;
> return (int) h;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20935) HStore.removeCompactedfiles should log incase it unable to delete a file
[ https://issues.apache.org/jira/browse/HBASE-20935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556886#comment-16556886 ] Ted Yu commented on HBASE-20935: +1 > HStore.removeCompactedfiles should log incase it unable to delete a file > > > Key: HBASE-20935 > URL: https://issues.apache.org/jira/browse/HBASE-20935 > Project: HBase > Issue Type: Improvement >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Minor > Fix For: 1.3.3 > > Attachments: HBASE-20935.branch-1.3.patch, > HBASE-20935.branch-1.3.v2.patch, HBASE-20935.patch, HBASE-20935.v2.patch > > > if (r != null && r.isCompactedAway() && !r.isReferencedInReads()) > If above check fails then there will be some files which are compacted but > not getting cleaned up. It is good to log which helps in debugging the issue. > This would let us know why is getting cleaned. either with reference pending > or compatedaway is not set. > This will help debug issues like : > # HBASE-20933 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20935) HStore.removeCompactedfiles should log incase it unable to delete a file
[ https://issues.apache.org/jira/browse/HBASE-20935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556884#comment-16556884 ] Vishal Khandelwal commented on HBASE-20935: --- Thanks [~yuzhih...@gmail.com] for review. I have updated the patch. > HStore.removeCompactedfiles should log incase it unable to delete a file > > > Key: HBASE-20935 > URL: https://issues.apache.org/jira/browse/HBASE-20935 > Project: HBase > Issue Type: Improvement >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Minor > Fix For: 1.3.3 > > Attachments: HBASE-20935.branch-1.3.patch, > HBASE-20935.branch-1.3.v2.patch, HBASE-20935.patch, HBASE-20935.v2.patch > > > if (r != null && r.isCompactedAway() && !r.isReferencedInReads()) > If above check fails then there will be some files which are compacted but > not getting cleaned up. It is good to log which helps in debugging the issue. > This would let us know why is getting cleaned. either with reference pending > or compatedaway is not set. > This will help debug issues like : > # HBASE-20933 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20935) HStore.removeCompactedfiles should log incase it unable to delete a file
[ https://issues.apache.org/jira/browse/HBASE-20935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal Khandelwal updated HBASE-20935: -- Attachment: HBASE-20935.v2.patch HBASE-20935.branch-1.3.v2.patch > HStore.removeCompactedfiles should log incase it unable to delete a file > > > Key: HBASE-20935 > URL: https://issues.apache.org/jira/browse/HBASE-20935 > Project: HBase > Issue Type: Improvement >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Minor > Fix For: 1.3.3 > > Attachments: HBASE-20935.branch-1.3.patch, > HBASE-20935.branch-1.3.v2.patch, HBASE-20935.patch, HBASE-20935.v2.patch > > > if (r != null && r.isCompactedAway() && !r.isReferencedInReads()) > If above check fails then there will be some files which are compacted but > not getting cleaned up. It is good to log which helps in debugging the issue. > This would let us know why is getting cleaned. either with reference pending > or compatedaway is not set. > This will help debug issues like : > # HBASE-20933 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20930) MetaScanner.metaScan should use passed variable for meta table name rather than TableName.META_TABLE_NAME
[
https://issues.apache.org/jira/browse/HBASE-20930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556878#comment-16556878
]
Vishal Khandelwal commented on HBASE-20930:
---
Thanks [~elserj]. Yeah by mistake I attached patch of different JIRA
> MetaScanner.metaScan should use passed variable for meta table name rather
> than TableName.META_TABLE_NAME
> -
>
> Key: HBASE-20930
> URL: https://issues.apache.org/jira/browse/HBASE-20930
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.3.3
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Minor
> Fix For: 1.3.3
>
> Attachments: HBASE-20930.branch-1.3.patch
>
>
> MetaScanner.metaScan
> try (Table metaTable = new HTable(TableName.META_TABLE_NAME, connection,
> null)) {
> should be changed to
> metaScan(connection, visitor, userTableName, null, Integer.MAX_VALUE,
> metaTableName)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20930) MetaScanner.metaScan should use passed variable for meta table name rather than TableName.META_TABLE_NAME
[
https://issues.apache.org/jira/browse/HBASE-20930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-20930:
--
Attachment: (was: HBASE-20935.branch-1.3.patch)
> MetaScanner.metaScan should use passed variable for meta table name rather
> than TableName.META_TABLE_NAME
> -
>
> Key: HBASE-20930
> URL: https://issues.apache.org/jira/browse/HBASE-20930
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.3.3
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Minor
> Fix For: 1.3.3
>
> Attachments: HBASE-20930.branch-1.3.patch
>
>
> MetaScanner.metaScan
> try (Table metaTable = new HTable(TableName.META_TABLE_NAME, connection,
> null)) {
> should be changed to
> metaScan(connection, visitor, userTableName, null, Integer.MAX_VALUE,
> metaTableName)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20930) MetaScanner.metaScan should use passed variable for meta table name rather than TableName.META_TABLE_NAME
[
https://issues.apache.org/jira/browse/HBASE-20930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-20930:
--
Attachment: HBASE-20930.branch-1.3.patch
> MetaScanner.metaScan should use passed variable for meta table name rather
> than TableName.META_TABLE_NAME
> -
>
> Key: HBASE-20930
> URL: https://issues.apache.org/jira/browse/HBASE-20930
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.3.3
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Minor
> Fix For: 1.3.3
>
> Attachments: HBASE-20930.branch-1.3.patch
>
>
> MetaScanner.metaScan
> try (Table metaTable = new HTable(TableName.META_TABLE_NAME, connection,
> null)) {
> should be changed to
> metaScan(connection, visitor, userTableName, null, Integer.MAX_VALUE,
> metaTableName)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556869#comment-16556869
]
Hadoop QA commented on HBASE-20886:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
2s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 2m
2s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
31s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 31s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
14s{color} | {color:red} root: The patch generated 5 new + 25 unchanged - 1
fixed = 30 total (was 26) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
10s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 3m
32s{color} | {color:red} patch has 20 errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m
52s{color} | {color:red} The patch causes 20 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
51s{color} | {color:red} The patch causes 20 errors with Hadoop v3.0.0. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 10s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 3s{color} |
{color:black} {color} |
\\
\\
[jira] [Commented] (HBASE-20896) Port HBASE-20866 to branch-1 and branch-1.4
[
https://issues.apache.org/jira/browse/HBASE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556860#comment-16556860
]
Hadoop QA commented on HBASE-20896:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-1.4 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
56s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 5s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.5 2.7.4. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
17s{color} | {color:red} hbase-client-jdk1.8.0_172 with JDK v1.8.0_172
generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} hbase-client-jdk1.7.0_181 with JDK v1.7.0_181
generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
14s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:2cf636a |
| JIRA Issue | HBASE-20896 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933132/HBASE-20896.branch-1.4.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux
[jira] [Commented] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556853#comment-16556853
]
Xu Cang commented on HBASE-20936:
-
I did but I think it's not correct so I canceled it. Sorry about that. [~stack]
A small finding is: array "regionsPerServer" must get modified when
stochasticLocalbalancer walking through many candidates and comparing costs. So
in #removeRegion, it cannot find the region in that particular array. I am
still trying to find out where and why regionsPerServer got changed. Just
curious, do you have a way to reproduce this or do you have some special
configuration might be related?
> BaseLoadBalancer throws ArrayIndexOutOfBoundsException
> --
>
> Key: HBASE-20936
> URL: https://issues.apache.org/jira/browse/HBASE-20936
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.1
>Reporter: stack
>Priority: Major
>
> I see the below in a test cluster. The balancer code has been in place a long
> time. Not sure why its doing the below. Will look later. Meantime, making a
> record of it.
> {code}
> 2018-07-24 22:03:43,261 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.47655219381977, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288571056819432);
> 2018-07-24 22:03:43,322 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 64
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:723)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:399)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1514)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1445)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-24 22:08:20,101 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=4.61 MB, freeSize=6.14 GB, max=6.14 GB,
> blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0,
> cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2018-07-24 22:08:38,101 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.4765419610035, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0);
[jira] [Commented] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556838#comment-16556838
]
stack commented on HBASE-20936:
---
Thank you for working on this [~xucang]. Did you attach a patch? It seems to be
missing? Thanks.
> BaseLoadBalancer throws ArrayIndexOutOfBoundsException
> --
>
> Key: HBASE-20936
> URL: https://issues.apache.org/jira/browse/HBASE-20936
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.1
>Reporter: stack
>Priority: Major
>
> I see the below in a test cluster. The balancer code has been in place a long
> time. Not sure why its doing the below. Will look later. Meantime, making a
> record of it.
> {code}
> 2018-07-24 22:03:43,261 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.47655219381977, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288571056819432);
> 2018-07-24 22:03:43,322 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 64
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:723)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:399)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1514)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1445)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-24 22:08:20,101 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=4.61 MB, freeSize=6.14 GB, max=6.14 GB,
> blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0,
> cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2018-07-24 22:08:38,101 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.4765419610035, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288550591186913);
> 2018-07-24 22:08:38,103 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 100
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
>
[jira] [Updated] (HBASE-20896) Port HBASE-20866 to branch-1 and branch-1.4
[ https://issues.apache.org/jira/browse/HBASE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Vishwakarma updated HBASE-20896: -- Attachment: HBASE-20896.branch-1.4.002.patch > Port HBASE-20866 to branch-1 and branch-1.4 > > > Key: HBASE-20896 > URL: https://issues.apache.org/jira/browse/HBASE-20896 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Vikas Vishwakarma >Priority: Major > Fix For: 1.5.0, 1.4.7 > > Attachments: HBASE-20896.branch-1.4.001.patch, > HBASE-20896.branch-1.4.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20896) Port HBASE-20866 to branch-1 and branch-1.4
[ https://issues.apache.org/jira/browse/HBASE-20896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Vishwakarma updated HBASE-20896: -- Attachment: (was: HBASE-20896.branch-1.4.002.patch) > Port HBASE-20866 to branch-1 and branch-1.4 > > > Key: HBASE-20896 > URL: https://issues.apache.org/jira/browse/HBASE-20896 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Vikas Vishwakarma >Priority: Major > Fix For: 1.5.0, 1.4.7 > > Attachments: HBASE-20896.branch-1.4.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Status: Patch Available (was: Open)
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Status: Open (was: Patch Available)
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Attachment: HBASE-20886.master.005.patch
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Attachment: (was: HBASE-20886.master.005.patch)
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556812#comment-16556812
]
Reid Chan commented on HBASE-20886:
---
Trigger QA again.
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556771#comment-16556771
]
Hadoop QA commented on HBASE-20936:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
38s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
10s{color} | {color:red} hbase-server: The patch generated 3 new + 59 unchanged
- 1 fixed = 62 total (was 60) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 8s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}171m 19s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}217m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20936 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933103/HBASE-20936.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7784dea5a22b 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a392c017ed |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13796/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
[jira] [Updated] (HBASE-20921) Possible NPE in ReopenTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-20921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-20921:
---
Attachment: HBASE-20921.branch-2.0.002.patch
> Possible NPE in ReopenTableRegionsProcedure
> ---
>
> Key: HBASE-20921
> URL: https://issues.apache.org/jira/browse/HBASE-20921
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 3.0.0, 2.1.0, 2.0.2
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Attachments: HBASE-20921.branch-2.0.001.patch,
> HBASE-20921.branch-2.0.002.patch
>
>
> After HBASE-20752， we issue a ReopenTableRegionsProcedure in
> ModifyTableProcedure to ensure all regions are reopened.
> But, ModifyTableProcedure and ReopenTableRegionsProcedure do not hold the
> lock (why?), so there is a chance that while ModifyTableProcedure executing,
> a merge/split procedure can be executed at the same time.
> So, when ReopenTableRegionsProcedure reaches the state of
> "REOPEN_TABLE_REGIONS_CONFIRM_REOPENED", some of the persisted regions to
> check is actually not exists, thus a NPE will throw.
> {code}
> 2018-07-18 01:38:57,528 INFO [PEWorker-9]
> procedure2.ProcedureExecutor(1246): Finished pid=6110, state=SUCCESS;
> MergeTableRegionsProcedure table=IntegrationTestBigLinkedList,
> regions=[845d286231eb01b7
> 1aeaa17b0e30058d, 4a46ab0918c99cada72d5336ad83a828], forcibly=false in
> 10.8610sec
> 2018-07-18 01:38:57,530 ERROR [PEWorker-8]
> procedure2.ProcedureExecutor(1478): CODE-BUG: Uncaught runtime exception:
> pid=5974, ppid=5973, state=RUNNABLE:REOPEN_TABLE_REGIONS_CONFIRM_REOPENED;
> ReopenTab
> leRegionsProcedure table=IntegrationTestBigLinkedList
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.checkReopened(RegionStates.java:651)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.hadoop.hbase.master.procedure.ReopenTableRegionsProcedure.executeFromState(ReopenTableRegionsProcedure.java:102)
> at
> org.apache.hadoop.hbase.master.procedure.ReopenTableRegionsProcedure.executeFromState(ReopenTableRegionsProcedure.java:45)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1453)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1741)
> {code}
> I think we need to renew the region list of the table at the
> "REOPEN_TABLE_REGIONS_CONFIRM_REOPENED" state. For the regions which are
> merged or split, we do not need to check it. Since we can be sure that they
> are opened after we made change to table descriptor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20921) Possible NPE in ReopenTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-20921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-20921:
---
Attachment: (was: HBASE-20921.branch-2.0.002.patch)
> Possible NPE in ReopenTableRegionsProcedure
> ---
>
> Key: HBASE-20921
> URL: https://issues.apache.org/jira/browse/HBASE-20921
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 3.0.0, 2.1.0, 2.0.2
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Attachments: HBASE-20921.branch-2.0.001.patch,
> HBASE-20921.branch-2.0.002.patch
>
>
> After HBASE-20752， we issue a ReopenTableRegionsProcedure in
> ModifyTableProcedure to ensure all regions are reopened.
> But, ModifyTableProcedure and ReopenTableRegionsProcedure do not hold the
> lock (why?), so there is a chance that while ModifyTableProcedure executing,
> a merge/split procedure can be executed at the same time.
> So, when ReopenTableRegionsProcedure reaches the state of
> "REOPEN_TABLE_REGIONS_CONFIRM_REOPENED", some of the persisted regions to
> check is actually not exists, thus a NPE will throw.
> {code}
> 2018-07-18 01:38:57,528 INFO [PEWorker-9]
> procedure2.ProcedureExecutor(1246): Finished pid=6110, state=SUCCESS;
> MergeTableRegionsProcedure table=IntegrationTestBigLinkedList,
> regions=[845d286231eb01b7
> 1aeaa17b0e30058d, 4a46ab0918c99cada72d5336ad83a828], forcibly=false in
> 10.8610sec
> 2018-07-18 01:38:57,530 ERROR [PEWorker-8]
> procedure2.ProcedureExecutor(1478): CODE-BUG: Uncaught runtime exception:
> pid=5974, ppid=5973, state=RUNNABLE:REOPEN_TABLE_REGIONS_CONFIRM_REOPENED;
> ReopenTab
> leRegionsProcedure table=IntegrationTestBigLinkedList
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.checkReopened(RegionStates.java:651)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.hadoop.hbase.master.procedure.ReopenTableRegionsProcedure.executeFromState(ReopenTableRegionsProcedure.java:102)
> at
> org.apache.hadoop.hbase.master.procedure.ReopenTableRegionsProcedure.executeFromState(ReopenTableRegionsProcedure.java:45)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1453)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1741)
> {code}
> I think we need to renew the region list of the table at the
> "REOPEN_TABLE_REGIONS_CONFIRM_REOPENED" state. For the regions which are
> merged or split, we do not need to check it. Since we can be sure that they
> are opened after we made change to table descriptor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19751) Use RegionInfo directly instead of an identifier and a namespace when getting WAL
[ https://issues.apache.org/jira/browse/HBASE-19751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556573#comment-16556573 ] Hudson commented on HBASE-19751: FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1930 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1930/]) PHOENIX-4536 Change getWAL usage due HBASE-19751 (ssa: rev 8ac5ec08d6dac484a80d453b44ee2f69609ca6a5) * (edit) phoenix-core/src/test/java/org/apache/phoenix/hbase/index/write/recovery/TestPerRegionIndexWriteCache.java * (edit) phoenix-core/src/it/java/org/apache/hadoop/hbase/regionserver/wal/WALReplayWithIndexWritesAndCompressedWALIT.java > Use RegionInfo directly instead of an identifier and a namespace when getting > WAL > - > > Key: HBASE-19751 > URL: https://issues.apache.org/jira/browse/HBASE-19751 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2, 2.0.0 > > Attachments: HBASE-19751-v1.patch, HBASE-19751-v2.patch, > HBASE-19751.patch > > > This is needed for synchronous replication as we need to determine whether a > region should use the special WAL which writes to two FileSystems. > But I think it is a general simplification so we can apply it to master and > branch-2 first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19122) preCompact and preFlush can bypass by returning null scanner; shut it down
[
https://issues.apache.org/jira/browse/HBASE-19122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556546#comment-16556546
]
Hudson commented on HBASE-19122:
FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1930 (See
[https://builds.apache.org/job/PreCommit-PHOENIX-Build/1930/])
PHOENIX-4477 Reflect changes in Cell/CellBuilder.DataType (HBASE-19122) (ssa:
rev d29da25244389f7ae59ab590622cc78ca780624d)
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
* (edit)
phoenix-core/src/test/java/org/apache/phoenix/execute/UnnestArrayPlanTest.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
* (edit)
phoenix-core/src/test/java/org/apache/phoenix/query/EncodedColumnQualifierCellsListTest.java
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/util/PhoenixKeyValueUtil.java
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionCoprocessorEnvironment.java
> preCompact and preFlush can bypass by returning null scanner; shut it down
> --
>
> Key: HBASE-19122
> URL: https://issues.apache.org/jira/browse/HBASE-19122
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, Scanners
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0-beta-1, 2.0.0
>
> Attachments: HBASE-19122.master.001.patch,
> HBASE-19122.master.002.patch, HBASE-19122.master.003.patch,
> HBASE-19122.master.004.patch
>
>
> Noticed by [~anoop.hbase] during review of HBASE-18770, preCompact and
> preFlush can bypass normal processing by returning null. They are not
> bypasable by ordained route. We should shut down this avenue.
> The preCompact at least may be new coming in with:
> {code}
> tree dbf13093842f85a713f023d7219caccf8f4eb05f
> parent a4dcf51415616772e462091ce93622f070ea8810
> author zhangduo Sat Apr 9 16:18:08 2016 +0800
> committer zhangduo Sun Apr 10 09:26:28 2016 +0800
> HBASE-15527 Refactor Compactor related classes
> {code}
> Would have to dig in more to figure for sure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19134) Make WALKey an Interface; expose Read-Only version to CPs
[ https://issues.apache.org/jira/browse/HBASE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556541#comment-16556541 ] Hudson commented on HBASE-19134: FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1930 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1930/]) PHOENIX-4452 change usage of WALKey to WALKeyImpl due HBASE-19134 (ssa: rev b3edf390353c19b8b4c5de2cca35a97457743768) * (edit) phoenix-core/src/it/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilterIT.java > Make WALKey an Interface; expose Read-Only version to CPs > - > > Key: HBASE-19134 > URL: https://issues.apache.org/jira/browse/HBASE-19134 > Project: HBase > Issue Type: Bug > Components: Coprocessors, wal >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-1, 2.0.0 > > Attachments: HBASE-19134.master.001.patch, > HBASE-19134.master.002.patch, HBASE-19134.master.003.patch, > HBASE-19134.master.004.patch, HBASE-19134.master.005.patch > > > WALKey has been made IA.Private in hbase2. Even so, given we don't have an > alternative to expose at this time, it is exposed to coprocessors still at a > few (now deprecated) locations. > In review of HBASE-18770, [~chia7712] makes reasonable suggestion that what > we expose to CPs be a read-only WALKey. He gets pushback on doing this for > hbase2 (Do we even want to expose WALKey to CPs, is WALKey right going > forward, etc.). Chia-Ping comes back w/ the below (copied from HBASE-18770): > What we want to fix for WALKey are shown below. > * expose some methods to CP user safety > * refactor/redo > As I see it, adding an interface exposed to CP user for WALKey is a right > choice because it can bring some benefit. > * We can expose part of WALKey's methods to CP users - a read-only interface > or an interface with some modifiable setting. > * The related CP hooks won't be deprecated > * Doing the refactor for WALKey doesn't essentially impact the CP hook after > 2.0 release. > Although, it will be better to redo WALKey before 2.0 release. In short, I > think it warrants such an interface. > (We both agree this would be a load of work given WALKey is written to > HFiles. Warrants a look though). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20899) Add Hadoop KMS dependency and basic HDFS at-rest encryption tests
[ https://issues.apache.org/jira/browse/HBASE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HBASE-20899: Attachment: HBASE-20899.master.002.patch > Add Hadoop KMS dependency and basic HDFS at-rest encryption tests > - > > Key: HBASE-20899 > URL: https://issues.apache.org/jira/browse/HBASE-20899 > Project: HBase > Issue Type: Sub-task > Components: encryption >Affects Versions: 2.0.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HBASE-20899.master.001.patch, > HBASE-20899.master.002.patch > > > We should start by adding hadoop-kms dependency in HBase test scope, and add > basic HDFS at-rest encryption tests using the hadoop-kms dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20919) meta region can't be re-onlined when restarting cluster if opening rsgroup
[
https://issues.apache.org/jira/browse/HBASE-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556448#comment-16556448
]
Ted Yu commented on HBASE-20919:
Triggered QA again:
https://builds.apache.org/job/PreCommit-HBASE-Build/13798/
> meta region can't be re-onlined when restarting cluster if opening rsgroup
> --
>
> Key: HBASE-20919
> URL: https://issues.apache.org/jira/browse/HBASE-20919
> Project: HBase
> Issue Type: Bug
> Components: Balancer, master, rsgroup
>Affects Versions: 2.0.1
>Reporter: chenyang
>Assignee: ChenYang
>Priority: Major
> Attachments: HBASE-20919-branch-2.0-01.patch,
> HBASE-20919-branch-2.0-02.patch, bug2.png,
> hbase-hbase-master-bjpg-rs4729.yz02.no_02patch.log,
> hbase-hbase-master-bjpg-rs4729.yz02.with_02patch.log,
> hbase-hbase-master-bjpg-rs4730.yz02.log.test
>
>
> if you open rsgroup, hbase-site.xml contains below configuration.
> {code:java}
>
> hbase.coprocessor.master.classes
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint
>
>
> hbase.master.loadbalancer.class/name>
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer
>
> {code}
> And you shut down the whole HBase cluster in the way:
> # first shut down region server one by one
> # shut down master
> Then you restart whole cluster in the way:
> # start master
> # start regionserver
> The hbase:meta region can not be re-online and the rsgroup can not be
> initialized successfully.
> master logs:
> {code:java}
> 2018-07-12 18:27:08,775 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> rsgroup.RSGro
> upInfoManagerImpl$RSGroupStartupWorker: Waiting for catalog tables to come
> online
> 2018-07-12 18:27:08,876 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> zookeeper.Met
> aTableLocator: Failed verification of hbase:meta,,1 at
> address=bjpg-rs4732.yz02,60020,1531388712053,
> exception=org.apache.hadoop.hbase.NotServingRegionExcepti
> on: hbase:meta,,1 is not online on bjpg-rs4732.yz02,60020,1531389727928
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3249)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3226)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1729)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28286)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> The logs show that hbase:meta region is not online and rsgroup keeps retrying
> to initialize.
>
> but why the hbase:meta region is not online?
> The info-level logs and jstack had not enough infomation, so I added some
> debug logs in test-source-code. Then i checked the master`s logs and region
> server`s logs, and found the meta region assign procedure which hold the meta
> region lock not completed and not released the lock forever, so the
> recoverMetaProcedure could not be executed.
>
> Why the first procedure not completed and not released meta region lock?
> In the test logs, i found when assignmentManager assigned the region, it
> need to call the rsgroup balancer which have not been initialized
> completely, so throw NPE. As a result, the procedure not completed and not
> released the lock forever.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:262)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignmentPlans(AssignmentManager.java:1864)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1809)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$400(AssignmentManager.java:113)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager$2.run(AssignmentManager.java:1693)
> {code}
> !bug2.png!
> As shown in the figure named bug2.png listed in attachments, when we shutdown
> the last region server, the master submit a ServerCrashProcedure. In the
> procedure, it will reassign hbase:meta region, but
[jira] [Assigned] (HBASE-20919) meta region can't be re-onlined when restarting cluster if opening rsgroup
[
https://issues.apache.org/jira/browse/HBASE-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-20919:
--
Assignee: ChenYang
> meta region can't be re-onlined when restarting cluster if opening rsgroup
> --
>
> Key: HBASE-20919
> URL: https://issues.apache.org/jira/browse/HBASE-20919
> Project: HBase
> Issue Type: Bug
> Components: Balancer, master, rsgroup
>Affects Versions: 2.0.1
>Reporter: chenyang
>Assignee: ChenYang
>Priority: Major
> Attachments: HBASE-20919-branch-2.0-01.patch,
> HBASE-20919-branch-2.0-02.patch, bug2.png,
> hbase-hbase-master-bjpg-rs4729.yz02.no_02patch.log,
> hbase-hbase-master-bjpg-rs4729.yz02.with_02patch.log,
> hbase-hbase-master-bjpg-rs4730.yz02.log.test
>
>
> if you open rsgroup, hbase-site.xml contains below configuration.
> {code:java}
>
> hbase.coprocessor.master.classes
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint
>
>
> hbase.master.loadbalancer.class/name>
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer
>
> {code}
> And you shut down the whole HBase cluster in the way:
> # first shut down region server one by one
> # shut down master
> Then you restart whole cluster in the way:
> # start master
> # start regionserver
> The hbase:meta region can not be re-online and the rsgroup can not be
> initialized successfully.
> master logs:
> {code:java}
> 2018-07-12 18:27:08,775 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> rsgroup.RSGro
> upInfoManagerImpl$RSGroupStartupWorker: Waiting for catalog tables to come
> online
> 2018-07-12 18:27:08,876 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> zookeeper.Met
> aTableLocator: Failed verification of hbase:meta,,1 at
> address=bjpg-rs4732.yz02,60020,1531388712053,
> exception=org.apache.hadoop.hbase.NotServingRegionExcepti
> on: hbase:meta,,1 is not online on bjpg-rs4732.yz02,60020,1531389727928
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3249)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3226)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1729)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28286)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> The logs show that hbase:meta region is not online and rsgroup keeps retrying
> to initialize.
>
> but why the hbase:meta region is not online?
> The info-level logs and jstack had not enough infomation, so I added some
> debug logs in test-source-code. Then i checked the master`s logs and region
> server`s logs, and found the meta region assign procedure which hold the meta
> region lock not completed and not released the lock forever, so the
> recoverMetaProcedure could not be executed.
>
> Why the first procedure not completed and not released meta region lock?
> In the test logs, i found when assignmentManager assigned the region, it
> need to call the rsgroup balancer which have not been initialized
> completely, so throw NPE. As a result, the procedure not completed and not
> released the lock forever.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:262)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignmentPlans(AssignmentManager.java:1864)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1809)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$400(AssignmentManager.java:113)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager$2.run(AssignmentManager.java:1693)
> {code}
> !bug2.png!
> As shown in the figure named bug2.png listed in attachments, when we shutdown
> the last region server, the master submit a ServerCrashProcedure. In the
> procedure, it will reassign hbase:meta region, but at that moment, there is
> no online region server, so the procedure can not be executed
[jira] [Commented] (HBASE-20943) Add offline/online region count into metrics
[ https://issues.apache.org/jira/browse/HBASE-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556437#comment-16556437 ] Ted Yu commented on HBASE-20943: Looking forward to the patch. > Add offline/online region count into metrics > > > Key: HBASE-20943 > URL: https://issues.apache.org/jira/browse/HBASE-20943 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0, 1.2.6.1 >Reporter: Tianying Chang >Priority: Minor > Attachments: Screen Shot 2018-07-25 at 2.51.19 PM.png > > > We intensively use metrics to monitor the health of our HBase production > cluster. We have seen some regions of a table stuck and cannot be brought > online due to AWS issue which cause some log file corrupted. It will be good > if we can catch this early. Although WebUI has this information, it is not > useful for automated monitoring. By adding this metric, we can easily monitor > them with our monitoring system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20943) Add offline/online region count into metrics
[ https://issues.apache.org/jira/browse/HBASE-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-20943: --- Attachment: Screen Shot 2018-07-25 at 2.51.19 PM.png > Add offline/online region count into metrics > > > Key: HBASE-20943 > URL: https://issues.apache.org/jira/browse/HBASE-20943 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0, 1.2.6.1 >Reporter: Tianying Chang >Priority: Minor > Attachments: Screen Shot 2018-07-25 at 2.51.19 PM.png > > > We intensively use metrics to monitor the health of our HBase production > cluster. We have seen some regions of a table stuck and cannot be brought > online due to AWS issue which cause some log file corrupted. It will be good > if we can catch this early. Although WebUI has this information, it is not > useful for automated monitoring. By adding this metric, we can easily monitor > them with our monitoring system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20943) Add offline/online region count into metrics
Tianying Chang created HBASE-20943: -- Summary: Add offline/online region count into metrics Key: HBASE-20943 URL: https://issues.apache.org/jira/browse/HBASE-20943 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 1.2.6.1, 2.0.0 Reporter: Tianying Chang We intensively use metrics to monitor the health of our HBase production cluster. We have seen some regions of a table stuck and cannot be brought online due to AWS issue which cause some log file corrupted. It will be good if we can catch this early. Although WebUI has this information, it is not useful for automated monitoring. By adding this metric, we can easily monitor them with our monitoring system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20938) Set version to 2.1.1-SNAPSHOT for branch-2.1
[
https://issues.apache.org/jira/browse/HBASE-20938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556421#comment-16556421
]
Hudson commented on HBASE-20938:
Results for branch branch-2.1
[build #104 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/104/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/104//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/104//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/104//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Set version to 2.1.1-SNAPSHOT for branch-2.1
>
>
> Key: HBASE-20938
> URL: https://issues.apache.org/jira/browse/HBASE-20938
> Project: HBase
> Issue Type: Sub-task
> Components: build
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-20938-branch-2.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20928) Rewrite calculation of midpoint in binarySearch functions to prevent overflow
[
https://issues.apache.org/jira/browse/HBASE-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556410#comment-16556410
]
Hudson commented on HBASE-20928:
Results for branch branch-2
[build #1026 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1026/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1026//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1026//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1026//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Rewrite calculation of midpoint in binarySearch functions to prevent overflow
> -
>
> Key: HBASE-20928
> URL: https://issues.apache.org/jira/browse/HBASE-20928
> Project: HBase
> Issue Type: Bug
> Components: io
>Reporter: saurabh singh
>Assignee: saurabh singh
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HBASE-20928-addendum.patch,
> HBASE-20928-fix-binarySearch-v5.patch, HBASE-20928-fix-binarySearch-v5.patch
>
>
> There are couple of issues in the function:
> * {{>>>}} operator would mess the values if {{low}} + {{high}} end up being
> negative. This shouldn't happen but I don't see anything to prevent this from
> happening.
> * The code fails around boundary values of {{low}} and {{high}}. This is a
> well known binary search catch.
> [https://ai.googleblog.com/2006/06/extra-extra-read-all-about-it-nearly.html]
>
> Most of the code should already be covered by tests. I would have liked to
> add a test that actually fails without the fix but given these are private
> methods I am not sure on the best place to add the test. Suggestions?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20649) Validate HFiles do not have PREFIX_TREE DataBlockEncoding
[
https://issues.apache.org/jira/browse/HBASE-20649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556409#comment-16556409
]
Hudson commented on HBASE-20649:
Results for branch branch-2
[build #1026 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1026/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1026//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1026//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1026//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Validate HFiles do not have PREFIX_TREE DataBlockEncoding
> -
>
> Key: HBASE-20649
> URL: https://issues.apache.org/jira/browse/HBASE-20649
> Project: HBase
> Issue Type: New Feature
> Components: Operability, tooling
>Reporter: Peter Somogyi
>Assignee: Balazs Meszaros
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20649.master.001.patch,
> HBASE-20649.master.002.patch, HBASE-20649.master.003.patch,
> HBASE-20649.master.004.patch, HBASE-20649.master.005.patch,
> HBASE-20649.master.006.patch
>
>
> HBASE-20592 adds a tool to check column families on the cluster do not have
> PREFIX_TREE encoding.
> Since it is possible that DataBlockEncoding was already changed but HFiles
> are not rewritten yet we would need a tool that can verify the content of
> hfiles in the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated HBASE-20936:
Attachment: (was: HBASE-20936.master.001.patch)
> BaseLoadBalancer throws ArrayIndexOutOfBoundsException
> --
>
> Key: HBASE-20936
> URL: https://issues.apache.org/jira/browse/HBASE-20936
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.1
>Reporter: stack
>Priority: Major
>
> I see the below in a test cluster. The balancer code has been in place a long
> time. Not sure why its doing the below. Will look later. Meantime, making a
> record of it.
> {code}
> 2018-07-24 22:03:43,261 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.47655219381977, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288571056819432);
> 2018-07-24 22:03:43,322 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 64
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:723)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:399)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1514)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1445)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-24 22:08:20,101 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=4.61 MB, freeSize=6.14 GB, max=6.14 GB,
> blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0,
> cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2018-07-24 22:08:38,101 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.4765419610035, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288550591186913);
> 2018-07-24 22:08:38,103 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 100
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
>
[jira] [Updated] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated HBASE-20936:
Status: Open (was: Patch Available)
> BaseLoadBalancer throws ArrayIndexOutOfBoundsException
> --
>
> Key: HBASE-20936
> URL: https://issues.apache.org/jira/browse/HBASE-20936
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.1
>Reporter: stack
>Priority: Major
>
> I see the below in a test cluster. The balancer code has been in place a long
> time. Not sure why its doing the below. Will look later. Meantime, making a
> record of it.
> {code}
> 2018-07-24 22:03:43,261 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.47655219381977, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288571056819432);
> 2018-07-24 22:03:43,322 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 64
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:723)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:399)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1514)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1445)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-24 22:08:20,101 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=4.61 MB, freeSize=6.14 GB, max=6.14 GB,
> blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0,
> cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2018-07-24 22:08:38,101 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.4765419610035, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288550591186913);
> 2018-07-24 22:08:38,103 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 100
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
>
[jira] [Issue Comment Deleted] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated HBASE-20936:
Comment: was deleted
(was:
[https://github.com/apache/hbase/blob/067388bfd9dccdb42fcbeedc45a3862555603892/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L417]
The undoAction is not implemented.
[https://github.com/apache/hbase/blob/067388bfd9dccdb42fcbeedc45a3862555603892/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java#L643]
So "regionsPerServer" can be changed if StochasticLoadBalancer tried some
actions before which modified the regionsPerServer.
!/jira/secure/viewavatar?size=xsmall=21133=issuetype|width=16,height=16!
HBASE-19263 is also caused by StochasticLoadBalancer.
So my fix is to check if the region is still on the server or not to avoid NPE.
)
> BaseLoadBalancer throws ArrayIndexOutOfBoundsException
> --
>
> Key: HBASE-20936
> URL: https://issues.apache.org/jira/browse/HBASE-20936
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.1
>Reporter: stack
>Priority: Major
> Attachments: HBASE-20936.master.001.patch
>
>
> I see the below in a test cluster. The balancer code has been in place a long
> time. Not sure why its doing the below. Will look later. Meantime, making a
> record of it.
> {code}
> 2018-07-24 22:03:43,261 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.47655219381977, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288571056819432);
> 2018-07-24 22:03:43,322 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 64
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:723)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:399)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1514)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1445)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-24 22:08:20,101 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=4.61 MB, freeSize=6.14 GB, max=6.14 GB,
> blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0,
> cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2018-07-24 22:08:38,101 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.4765419610035, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0,
[jira] [Commented] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556370#comment-16556370
]
Xu Cang commented on HBASE-20936:
-
[https://github.com/apache/hbase/blob/067388bfd9dccdb42fcbeedc45a3862555603892/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L417]
The undoAction is not implemented.
[https://github.com/apache/hbase/blob/067388bfd9dccdb42fcbeedc45a3862555603892/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java#L643]
So "regionsPerServer" can be changed if StochasticLoadBalancer tried some
actions before which modified the regionsPerServer.
!/jira/secure/viewavatar?size=xsmall=21133=issuetype|width=16,height=16!
HBASE-19263 is also caused by StochasticLoadBalancer.
So my fix is to check if the region is still on the server or not to avoid NPE.
> BaseLoadBalancer throws ArrayIndexOutOfBoundsException
> --
>
> Key: HBASE-20936
> URL: https://issues.apache.org/jira/browse/HBASE-20936
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.1
>Reporter: stack
>Priority: Major
> Attachments: HBASE-20936.master.001.patch
>
>
> I see the below in a test cluster. The balancer code has been in place a long
> time. Not sure why its doing the below. Will look later. Meantime, making a
> record of it.
> {code}
> 2018-07-24 22:03:43,261 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.47655219381977, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288571056819432);
> 2018-07-24 22:03:43,322 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 64
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:723)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:399)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1514)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1445)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-24 22:08:20,101 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=4.61 MB, freeSize=6.14 GB, max=6.14 GB,
> blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0,
> cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2018-07-24 22:08:38,101 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.4765419610035, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction :
[jira] [Commented] (HBASE-20815) In TestServerCrashProcedure collect and assert on submitted and failed counts for ServerCrashProcedure
[ https://issues.apache.org/jira/browse/HBASE-20815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556368#comment-16556368 ] Umesh Agashe commented on HBASE-20815: -- +1, lgtm. Thanks for adding testRecoveryOnRsWithMeta() , [~xucang]! > In TestServerCrashProcedure collect and assert on submitted and failed counts > for ServerCrashProcedure > -- > > Key: HBASE-20815 > URL: https://issues.apache.org/jira/browse/HBASE-20815 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: Umesh Agashe >Assignee: Xu Cang >Priority: Minor > Attachments: HBASE-20815.master.001.patch, > HBASE-20815.master.002.patch, HBASE-20815.master.002.patch > > > We need to collect and possibly assert on number of procedures submitted and > failed for ServerCrashProcedures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20931) [branch-1] Add -Dhttps.protocols=TLSv1.2 to Maven command line in make_rc.sh
[
https://issues.apache.org/jira/browse/HBASE-20931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556359#comment-16556359
]
Hudson commented on HBASE-20931:
Results for branch branch-1.3
[build #403 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/403/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/403//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/403//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/403//console].
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> [branch-1] Add -Dhttps.protocols=TLSv1.2 to Maven command line in make_rc.sh
>
>
> Key: HBASE-20931
> URL: https://issues.apache.org/jira/browse/HBASE-20931
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Trivial
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20931-branch-1.patch
>
>
> As of June 2018 the insecure TLS 1.0 and 1.1 protocols are no longer
> supported for SSL connections to Maven Central and perhaps other public Maven
> repositories. The branch-1 builds which require Java 7, of which the latest
> public release was 7u80, need to add {{-Dhttps.protocols=TLSv1.2}} to the
> Maven command line in order to avoid artifact retrieval problems during
> builds.
> We especially need this in make_rc.sh which starts up with an empty local
> Maven cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated HBASE-20936:
Attachment: HBASE-20936.master.001.patch
> BaseLoadBalancer throws ArrayIndexOutOfBoundsException
> --
>
> Key: HBASE-20936
> URL: https://issues.apache.org/jira/browse/HBASE-20936
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.1
>Reporter: stack
>Priority: Major
> Attachments: HBASE-20936.master.001.patch
>
>
> I see the below in a test cluster. The balancer code has been in place a long
> time. Not sure why its doing the below. Will look later. Meantime, making a
> record of it.
> {code}
> 2018-07-24 22:03:43,261 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.47655219381977, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288571056819432);
> 2018-07-24 22:03:43,322 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 64
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:723)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:399)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1514)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1445)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-24 22:08:20,101 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=4.61 MB, freeSize=6.14 GB, max=6.14 GB,
> blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0,
> cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2018-07-24 22:08:38,101 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.4765419610035, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288550591186913);
> 2018-07-24 22:08:38,103 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 100
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
>
[jira] [Updated] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated HBASE-20936:
Attachment: HBASE-20936.master.001.patch
Status: Patch Available (was: Open)
> BaseLoadBalancer throws ArrayIndexOutOfBoundsException
> --
>
> Key: HBASE-20936
> URL: https://issues.apache.org/jira/browse/HBASE-20936
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.1
>Reporter: stack
>Priority: Major
> Attachments: HBASE-20936.master.001.patch
>
>
> I see the below in a test cluster. The balancer code has been in place a long
> time. Not sure why its doing the below. Will look later. Meantime, making a
> record of it.
> {code}
> 2018-07-24 22:03:43,261 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.47655219381977, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288571056819432);
> 2018-07-24 22:03:43,322 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 64
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:723)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:399)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1514)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1445)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-24 22:08:20,101 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=4.61 MB, freeSize=6.14 GB, max=6.14 GB,
> blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0,
> cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2018-07-24 22:08:38,101 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.4765419610035, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288550591186913);
> 2018-07-24 22:08:38,103 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 100
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
>
[jira] [Updated] (HBASE-20936) BaseLoadBalancer throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated HBASE-20936:
Attachment: (was: HBASE-20936.master.001.patch)
> BaseLoadBalancer throws ArrayIndexOutOfBoundsException
> --
>
> Key: HBASE-20936
> URL: https://issues.apache.org/jira/browse/HBASE-20936
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.1
>Reporter: stack
>Priority: Major
> Attachments: HBASE-20936.master.001.patch
>
>
> I see the below in a test cluster. The balancer code has been in place a long
> time. Not sure why its doing the below. Will look later. Meantime, making a
> record of it.
> {code}
> 2018-07-24 22:03:43,261 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.47655219381977, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288571056819432);
> 2018-07-24 22:03:43,322 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 64
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.regionMoved(BaseLoadBalancer.java:827)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.doAction(BaseLoadBalancer.java:723)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:399)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:318)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1514)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1445)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-24 22:08:20,101 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=4.61 MB, freeSize=6.14 GB, max=6.14 GB,
> blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0,
> cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2018-07-24 22:08:38,101 INFO [master/ve0524:16000.Chore.1]
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=280.4765419610035, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.4992354740061162); PrimaryRegionCountSkewCostFunction : (500.0,
> 0.0); MoveCostFunction : (7.0, 0.0); ServerLocalityCostFunction : (25.0,
> 0.5961002123692674); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (35.0, 0.3660578386605784);
> RegionReplicaHostCostFunction : (10.0, 0.0);
> RegionReplicaRackCostFunction : (1.0, 0.0); ReadRequestCostFunction :
> (5.0, 0.0); WriteRequestCostFunction : (5.0, 0.0); MemStoreSizeCostFunction :
> (5.0, 0.0); StoreFileCostFunction : (5.0, 0.6288550591186913);
> 2018-07-24 22:08:38,103 ERROR [master/ve0524:16000.Chore.1]
> hbase.ScheduledChore: Caught error
> java.lang.ArrayIndexOutOfBoundsException: 100
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.removeRegion(BaseLoadBalancer.java:872)
> at
>
[jira] [Updated] (HBASE-20894) Move BucketCache from java serialization to protobuf
[ https://issues.apache.org/jira/browse/HBASE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20894: -- Assignee: Mike Drob Status: Patch Available (was: Open) Let's see what QA thinks of this anyway > Move BucketCache from java serialization to protobuf > > > Key: HBASE-20894 > URL: https://issues.apache.org/jira/browse/HBASE-20894 > Project: HBase > Issue Type: Task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20894.WIP-2.patch, HBASE-20894.WIP.patch, > HBASE-20894.master.001.patch, HBASE-20894.master.002.patch > > > We should use a better serialization format instead of Java Serialization for > the BucketCache entry persistence. > Suggested by Chris McCown, who does not appear to have a JIRA account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20894) Move BucketCache from java serialization to protobuf
[ https://issues.apache.org/jira/browse/HBASE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556331#comment-16556331 ] Mike Drob commented on HBASE-20894: --- I added the access counter to the list of things getting serialized and the unit test passes for me now. I don't feel confident in the test's thoroughness though, because I still don't have anything for the serializers. > Move BucketCache from java serialization to protobuf > > > Key: HBASE-20894 > URL: https://issues.apache.org/jira/browse/HBASE-20894 > Project: HBase > Issue Type: Task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: Mike Drob >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20894.WIP-2.patch, HBASE-20894.WIP.patch, > HBASE-20894.master.001.patch, HBASE-20894.master.002.patch > > > We should use a better serialization format instead of Java Serialization for > the BucketCache entry persistence. > Suggested by Chris McCown, who does not appear to have a JIRA account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20894) Move BucketCache from java serialization to protobuf
[ https://issues.apache.org/jira/browse/HBASE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20894: -- Attachment: HBASE-20894.master.002.patch > Move BucketCache from java serialization to protobuf > > > Key: HBASE-20894 > URL: https://issues.apache.org/jira/browse/HBASE-20894 > Project: HBase > Issue Type: Task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: Mike Drob >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20894.WIP-2.patch, HBASE-20894.WIP.patch, > HBASE-20894.master.001.patch, HBASE-20894.master.002.patch > > > We should use a better serialization format instead of Java Serialization for > the BucketCache entry persistence. > Suggested by Chris McCown, who does not appear to have a JIRA account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20932) Effective MemStoreSize::hashCode()
[
https://issues.apache.org/jira/browse/HBASE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556324#comment-16556324
]
Mingliang Liu commented on HBASE-20932:
---
Thanks [~yuzhih...@gmail.com] for prompt review and commit!
> Effective MemStoreSize::hashCode()
> ---
>
> Key: HBASE-20932
> URL: https://issues.apache.org/jira/browse/HBASE-20932
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.2
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: HBASE-20932.001.patch, HBASE-20932.002.patch
>
>
> After HBASE-20411 we have
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = 31 * this.dataSize;
> h = h + 31 * this.heapSize;
> h = h + 31 * this.offHeapSize;
> return (int) h;
> }
> {code}
> This is not effective {{hashCode()}} implementation. Instead we can use:
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = this.dataSize;
> h = h * 31 + this.heapSize;
> h = h * 31 + this.offHeapSize;
> return (int) h;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20815) In TestServerCrashProcedure collect and assert on submitted and failed counts for ServerCrashProcedure
[
https://issues.apache.org/jira/browse/HBASE-20815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556322#comment-16556322
]
Hadoop QA commented on HBASE-20815:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 6s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}122m
22s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20815 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933080/HBASE-20815.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 134628383ae4 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1913164970 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13794/testReport/ |
| Max. process+thread count | 4735 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13794/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> In TestServerCrashProcedure
[jira] [Commented] (HBASE-20846) Restore procedure locks when master restarts
[
https://issues.apache.org/jira/browse/HBASE-20846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556315#comment-16556315
]
Hudson commented on HBASE-20846:
Results for branch branch-2.1
[build #103 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/103/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/103//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/103//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/103//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Restore procedure locks when master restarts
>
>
> Key: HBASE-20846
> URL: https://issues.apache.org/jira/browse/HBASE-20846
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-20846-v1.patch, HBASE-20846-v2.patch,
> HBASE-20846-v3.patch, HBASE-20846-v4.patch, HBASE-20846-v4.patch,
> HBASE-20846-v4.patch, HBASE-20846-v5.patch, HBASE-20846-v6.patch,
> HBASE-20846.branch-2.0.002.patch, HBASE-20846.branch-2.0.patch,
> HBASE-20846.patch
>
>
> Found this one when investigating ModifyTableProcedure got stuck while there
> was a MoveRegionProcedure going on after master restart.
> Though this issue can be solved by HBASE-20752. But I discovered something
> else.
> Before a MoveRegionProcedure can execute, it will hold the table's shared
> lock. so,, when a UnassignProcedure was spwaned, it will not check the
> table's shared lock since it is sure that its parent(MoveRegionProcedure) has
> aquired the table's lock.
> {code:java}
> // If there is parent procedure, it would have already taken xlock, so no
> need to take
> // shared lock here. Otherwise, take shared lock.
> if (!procedure.hasParent()
> && waitTableQueueSharedLock(procedure, table) == null) {
> return true;
> }
> {code}
> But, it is not the case when Master was restarted. The child
> procedure(UnassignProcedure) will be executed first after restart. Though it
> has a parent(MoveRegionProcedure), but apprently the parent didn't hold the
> table's lock.
> So, since it began to execute without hold the table's shared lock. A
> ModifyTableProcedure can aquire the table's exclusive lock and execute at the
> same time. Which is not possible if the master was not restarted.
> This will cause a stuck before HBASE-20752. But since HBASE-20752 has fixed,
> I wrote a simple UT to repo this case.
> I think we don't have to check the parent for table's shared lock. It is a
> shared lock, right? I think we can acquire it every time we need it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20867) RS may get killed while master restarts
[
https://issues.apache.org/jira/browse/HBASE-20867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556316#comment-16556316
]
Hudson commented on HBASE-20867:
Results for branch branch-2.1
[build #103 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/103/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/103//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/103//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/103//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> RS may get killed while master restarts
> ---
>
> Key: HBASE-20867
> URL: https://issues.apache.org/jira/browse/HBASE-20867
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20867.branch-2.0.001.patch,
> HBASE-20867.branch-2.0.002.patch, HBASE-20867.branch-2.0.003.patch,
> HBASE-20867.branch-2.0.004.patch, HBASE-20867.branch-2.0.005.patch,
> HBASE-20867.branch-2.0.006.patch
>
>
> If the master is dispatching a RPC call to RS when aborting. A connection
> exception may be thrown by the RPC layer(A IOException with "Connection
> closed" message in this case). The RSProcedureDispatcher will regard is as an
> un-retryable exception and pass it to UnassignProcedue.remoteCallFailed,
> which will expire the RS.
> Actually, the RS is very healthy, only the master is restarting.
> I think we should deal with those kinds of connection exceptions in
> RSProcedureDispatcher and retry the rpc call
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20932) Effective MemStoreSize::hashCode()
[
https://issues.apache.org/jira/browse/HBASE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20932:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Thanks for the patch, Mingliang
> Effective MemStoreSize::hashCode()
> ---
>
> Key: HBASE-20932
> URL: https://issues.apache.org/jira/browse/HBASE-20932
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.2
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: HBASE-20932.001.patch, HBASE-20932.002.patch
>
>
> After HBASE-20411 we have
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = 31 * this.dataSize;
> h = h + 31 * this.heapSize;
> h = h + 31 * this.offHeapSize;
> return (int) h;
> }
> {code}
> This is not effective {{hashCode()}} implementation. Instead we can use:
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = this.dataSize;
> h = h * 31 + this.heapSize;
> h = h * 31 + this.offHeapSize;
> return (int) h;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20932) Effective MemStoreSize::hashCode()
[
https://issues.apache.org/jira/browse/HBASE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556303#comment-16556303
]
Mingliang Liu commented on HBASE-20932:
---
Thanks [~yuzhih...@gmail.com]. I generate the same patch with meta info as
[^HBASE-20932.002.patch] .
> Effective MemStoreSize::hashCode()
> ---
>
> Key: HBASE-20932
> URL: https://issues.apache.org/jira/browse/HBASE-20932
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.2
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Major
> Attachments: HBASE-20932.001.patch, HBASE-20932.002.patch
>
>
> After HBASE-20411 we have
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = 31 * this.dataSize;
> h = h + 31 * this.heapSize;
> h = h + 31 * this.offHeapSize;
> return (int) h;
> }
> {code}
> This is not effective {{hashCode()}} implementation. Instead we can use:
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = this.dataSize;
> h = h * 31 + this.heapSize;
> h = h * 31 + this.offHeapSize;
> return (int) h;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20932) Effective MemStoreSize::hashCode()
[
https://issues.apache.org/jira/browse/HBASE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HBASE-20932:
--
Attachment: HBASE-20932.002.patch
> Effective MemStoreSize::hashCode()
> ---
>
> Key: HBASE-20932
> URL: https://issues.apache.org/jira/browse/HBASE-20932
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.2
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Major
> Attachments: HBASE-20932.001.patch, HBASE-20932.002.patch
>
>
> After HBASE-20411 we have
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = 31 * this.dataSize;
> h = h + 31 * this.heapSize;
> h = h + 31 * this.offHeapSize;
> return (int) h;
> }
> {code}
> This is not effective {{hashCode()}} implementation. Instead we can use:
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = this.dataSize;
> h = h * 31 + this.heapSize;
> h = h * 31 + this.offHeapSize;
> return (int) h;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20899) Add Hadoop KMS dependency and basic HDFS at-rest encryption tests
[ https://issues.apache.org/jira/browse/HBASE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556301#comment-16556301 ] Wei-Chiu Chuang commented on HBASE-20899: - I forgot that in Hadoop 3, KMS switched from Tomcat to Jetty and therefore the package changed. Will update with a new patch. > Add Hadoop KMS dependency and basic HDFS at-rest encryption tests > - > > Key: HBASE-20899 > URL: https://issues.apache.org/jira/browse/HBASE-20899 > Project: HBase > Issue Type: Sub-task > Components: encryption >Affects Versions: 2.0.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HBASE-20899.master.001.patch > > > We should start by adding hadoop-kms dependency in HBase test scope, and add > basic HDFS at-rest encryption tests using the hadoop-kms dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20932) Effective MemStoreSize::hashCode()
[
https://issues.apache.org/jira/browse/HBASE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556298#comment-16556298
]
Ted Yu commented on HBASE-20932:
Mingliang:
Can you generate patch with proper header ?
You can find example on HBASE-20928.
This way, your identity would appear as author in commit log.
> Effective MemStoreSize::hashCode()
> ---
>
> Key: HBASE-20932
> URL: https://issues.apache.org/jira/browse/HBASE-20932
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.2
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Major
> Attachments: HBASE-20932.001.patch
>
>
> After HBASE-20411 we have
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = 31 * this.dataSize;
> h = h + 31 * this.heapSize;
> h = h + 31 * this.offHeapSize;
> return (int) h;
> }
> {code}
> This is not effective {{hashCode()}} implementation. Instead we can use:
> {code:java|title=MemStoreSize::hashCode()}
> @Override
> public int hashCode() {
> long h = this.dataSize;
> h = h * 31 + this.heapSize;
> h = h * 31 + this.offHeapSize;
> return (int) h;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20899) Add Hadoop KMS dependency and basic HDFS at-rest encryption tests
[
https://issues.apache.org/jira/browse/HBASE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556276#comment-16556276
]
Hadoop QA commented on HBASE-20899:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
5s{color} | {color:red} root: The patch generated 2 new + 0 unchanged - 0 fixed
= 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
13s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
11s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}216m 14s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}273m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20899 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933065/HBASE-20899.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile findbugs hbaseanti checkstyle |
| uname | Linux 639b94b627b7 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 GNU/Linux
[jira] [Commented] (HBASE-20932) Effective MemStoreSize::hashCode()
[
https://issues.apache.org/jira/browse/HBASE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556243#comment-16556243
]
Hadoop QA commented on HBASE-20932:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
31s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 14s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}122m
15s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20932 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932970/HBASE-20932.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c92b0bc8a255 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1913164970 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13792/testReport/ |
| Max. process+thread count | 4955 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13792/console |
|
[jira] [Created] (HBASE-20942) Make RpcServer trace log length configurable
Mike Drob created HBASE-20942: - Summary: Make RpcServer trace log length configurable Key: HBASE-20942 URL: https://issues.apache.org/jira/browse/HBASE-20942 Project: HBase Issue Type: Task Reporter: Esteban Gutierrez We truncate RpcServer output to 1000 characters for trace logging. Would be better if that value was configurable. Esteban mentioned this to me earlier, so I'm crediting him as the reporter. cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20930) MetaScanner.metaScan should use passed variable for meta table name rather than TableName.META_TABLE_NAME
[
https://issues.apache.org/jira/browse/HBASE-20930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556236#comment-16556236
]
Josh Elser commented on HBASE-20930:
[~vishk], looks like you attached the wrong file.
> MetaScanner.metaScan should use passed variable for meta table name rather
> than TableName.META_TABLE_NAME
> -
>
> Key: HBASE-20930
> URL: https://issues.apache.org/jira/browse/HBASE-20930
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.3.3
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Minor
> Fix For: 1.3.3
>
> Attachments: HBASE-20935.branch-1.3.patch
>
>
> MetaScanner.metaScan
> try (Table metaTable = new HTable(TableName.META_TABLE_NAME, connection,
> null)) {
> should be changed to
> metaScan(connection, visitor, userTableName, null, Integer.MAX_VALUE,
> metaTableName)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters
[
https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556202#comment-16556202
]
Hudson commented on HBASE-18477:
Results for branch HBASE-18477
[build #275 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/275/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/275//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/275//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/275//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/275//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Umbrella JIRA for HBase Read Replica clusters
> -
>
> Key: HBASE-18477
> URL: https://issues.apache.org/jira/browse/HBASE-18477
> Project: HBase
> Issue Type: New Feature
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase
> Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope
> doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf
>
>
> Recently, changes (such as HBASE-17437) have unblocked HBase to run with a
> root directory external to the cluster (such as in Amazon S3). This means
> that the data is stored outside of the cluster and can be accessible after
> the cluster has been terminated. One use case that is often asked about is
> pointing multiple clusters to one root directory (sharing the data) to have
> read resiliency in the case of a cluster failure.
>
> This JIRA is an umbrella JIRA to contain all the tasks necessary to create a
> read-replica HBase cluster that is pointed at the same root directory.
>
> This requires making the Read-Replica cluster Read-Only (no metadata
> operation or data operations).
> Separating the hbase:meta table for each cluster (Otherwise HBase gets
> confused with multiple clusters trying to update the meta table with their ip
> addresses)
> Adding refresh functionality for the meta table to ensure new metadata is
> picked up on the read replica cluster.
> Adding refresh functionality for HFiles for a given table to ensure new data
> is picked up on the read replica cluster.
>
> This can be used with any existing cluster that is backed by an external
> filesystem.
>
> Please note that this feature is still quite manual (with the potential for
> automation later).
>
> More information on this particular feature can be found here:
> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20941) Create and implement HbckService in master
[ https://issues.apache.org/jira/browse/HBASE-20941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-20941: - Summary: Create and implement HbckService in master (was: Cre) > Create and implement HbckService in master > -- > > Key: HBASE-20941 > URL: https://issues.apache.org/jira/browse/HBASE-20941 > Project: HBase > Issue Type: Sub-task >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > > Create HbckService in master and implement following methods: > # purgeProcedure/s(): some procedures do not support abort at every step. > When these procedures get stuck then they can not be aborted or make further > progress. Corrective action is to purge these procedures from ProcWAL. > Provide option to purge sub-procedures as well. > # setTable/RegionState(): If table/ region state are inconsistent with > action/ procedures working on them, sometimes manipulating their states in > meta fix things. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20941) Cre
Umesh Agashe created HBASE-20941: Summary: Cre Key: HBASE-20941 URL: https://issues.apache.org/jira/browse/HBASE-20941 Project: HBase Issue Type: Sub-task Reporter: Umesh Agashe Assignee: Umesh Agashe Create HbckService in master and implement following methods: # purgeProcedure/s(): some procedures do not support abort at every step. When these procedures get stuck then they can not be aborted or make further progress. Corrective action is to purge these procedures from ProcWAL. Provide option to purge sub-procedures as well. # setTable/RegionState(): If table/ region state are inconsistent with action/ procedures working on them, sometimes manipulating their states in meta fix things. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.
[
https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556097#comment-16556097
]
Hadoop QA commented on HBASE-18822:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
38s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
9s{color} | {color:red} hbase-server: The patch generated 1 new + 12 unchanged
- 0 fixed = 13 total (was 12) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
22s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}123m
41s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-18822 |
| JIRA Patch URL |
[jira] [Reopened] (HBASE-20893) Data loss if splitting region while ServerCrashProcedure executing
[
https://issues.apache.org/jira/browse/HBASE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reopened HBASE-20893:
---
Reopening to look at these logs I see running this patch on cluster (Its great
it detected recovered.edits... but it looks like the patch causes us to hit
CODE-BUG... though we seem to be ok...Minimally it will freak-out an operator):
{code}
2018-07-25 06:46:56,692 ERROR [PEWorker-3]
assignment.SplitTableRegionProcedure: Error trying to split region
2cb977a87bc6bdf90ef7fc71320d7b50 in the table IntegrationTestBigLinkedList (in
state=SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS)
java.io.IOException: Recovered.edits are found in Region: {ENCODED =>
2cb977a87bc6bdf90ef7fc71320d7b50, NAME =>
'IntegrationTestBigLinkedList,z\xAA;\xC7M\x1Bf8\x85\xB5\x07\xD5\x9B#\xCD\xCC,1531911202047.2cb977a87bc6bdf90ef7fc71320d7b50.',
STARTKEY => 'z\xAA;\xC7M\x1Bf8\x85\xB5\x07\xD5\x9B#\xCD\xCC', ENDKEY =>
'{\x8D\xF2?'}, abort split to prevent data loss
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.checkClosedRegion(SplitTableRegionProcedure.java:151)
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:259)
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:92)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1472)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
2018-07-25 06:46:56,934 INFO [PEWorker-3]
procedure.MasterProcedureScheduler: pid=4106, ppid=4105, state=SUCCESS;
UnassignProcedure table=IntegrationTestBigLinkedList,
region=2cb977a87bc6bdf90ef7fc71320d7b50,
server=ve0540.halxg.cloudera.com,16020,1532501580658 checking lock on
2cb977a87bc6bdf90ef7fc71320d7b50
2018-07-25 06:46:56,934 ERROR [PEWorker-3] procedure2.ProcedureExecutor:
CODE-BUG: Uncaught runtime exception for pid=4106, ppid=4105, state=SUCCESS;
UnassignProcedure table=IntegrationTestBigLinkedList,
region=2cb977a87bc6bdf90ef7fc71320d7b50,
server=ve0540.halxg.cloudera.com,16020,1532501580658
java.lang.UnsupportedOperationException:
Unhandled state REGION_TRANSITION_FINISH; there is no rollback for assignment
unless we cancel the operation by dropping/disabling the table
at
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.rollback(RegionTransitionProcedure.java:412)
at
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.rollback(RegionTransitionProcedure.java:95)
at
org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:864)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1372)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1328)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1197)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
2018-07-25 06:46:57,088 ERROR [PEWorker-3] procedure2.ProcedureExecutor:
CODE-BUG: Uncaught runtime exception for pid=4106, ppid=4105, state=SUCCESS;
UnassignProcedure table=IntegrationTestBigLinkedList,
region=2cb977a87bc6bdf90ef7fc71320d7b50,
server=ve0540.halxg.cloudera.com,16020,1532501580658
java.lang.UnsupportedOperationException:
[jira] [Commented] (HBASE-20815) In TestServerCrashProcedure collect and assert on submitted and failed counts for ServerCrashProcedure
[ https://issues.apache.org/jira/browse/HBASE-20815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556075#comment-16556075 ] Umesh Agashe commented on HBASE-20815: -- Unit test for patch 002 failed. Failure doesn't look related to the changes. Retrying. > In TestServerCrashProcedure collect and assert on submitted and failed counts > for ServerCrashProcedure > -- > > Key: HBASE-20815 > URL: https://issues.apache.org/jira/browse/HBASE-20815 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: Umesh Agashe >Assignee: Xu Cang >Priority: Minor > Attachments: HBASE-20815.master.001.patch, > HBASE-20815.master.002.patch, HBASE-20815.master.002.patch > > > We need to collect and possibly assert on number of procedures submitted and > failed for ServerCrashProcedures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20815) In TestServerCrashProcedure collect and assert on submitted and failed counts for ServerCrashProcedure
[ https://issues.apache.org/jira/browse/HBASE-20815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-20815: - Attachment: HBASE-20815.master.002.patch > In TestServerCrashProcedure collect and assert on submitted and failed counts > for ServerCrashProcedure > -- > > Key: HBASE-20815 > URL: https://issues.apache.org/jira/browse/HBASE-20815 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: Umesh Agashe >Assignee: Xu Cang >Priority: Minor > Attachments: HBASE-20815.master.001.patch, > HBASE-20815.master.002.patch, HBASE-20815.master.002.patch > > > We need to collect and possibly assert on number of procedures submitted and > failed for ServerCrashProcedures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20895) NPE in RpcServer#readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-20895:
---
Attachment: HBASE-20895-branch-1.patch
> NPE in RpcServer#readAndProcess
> ---
>
> Key: HBASE-20895
> URL: https://issues.apache.org/jira/browse/HBASE-20895
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 1.3.2
>Reporter: Andrew Purtell
>Assignee: Monani Mihir
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.7
>
> Attachments: HBASE-20895-branch-1.patch, HBASE-20895-branch-1.patch
>
>
> {noformat}
> 2018-07-10 16:25:55,005 DEBUG [.sfdc.net,port=60020] ipc.RpcServer -
> RpcServer.listener,port=60020: Caught exception while reading:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1761)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:949)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:730)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:706)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This looks like it could be a use after close problem if there is concurrent
> access to a Connection.
> In process() we might store a null back to the 'data' field.
> Meanwhile in readAndProcess() we have a case where we might be blocked on a
> channel read and then after coming back from the read we go to use 'data'
> after a null has been written back, leading to a NPE.
> {quote}count = channelRead(channel, data);
> 1761 ---> if (count >= 0 && *data.remaining()* == 0)
> \{ process(); }{quote}
> Whether a NPE happens or not is going to depend on the timing of the store
> back to 'data' in another thread and use of 'data' in this thread and whether
> or not the JVM has optimized away a reload of 'data' (it's not declared
> volatile)
> We should do a null check here just to be defensive. We should also look at
> whether concurrent access to the Connection is happening and intended.The
> above is just a theory. We should also look at other execution sequences that
> could lead to 'data' being null in this location. At a glance I didn't find
> one but the store to 'data' happens behind conditionals so it is possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20925) Canary test to expose results per table/ per region to result file
[ https://issues.apache.org/jira/browse/HBASE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-20925: Description: This change will provide a new cmd line argument for users to specify as "-verboseTestResultFilePath" and Canary will output structured/ easy to be parsed/ easy to be read strings to the file. In the result file, it should provide a summary of the canary test and shows which region has read or write failures. Also to include some stats regarding the test. Such as, "region read count: 500, region read success count: 499" (Before, Canary test sends some of above information into log file, mixed with other debugging information. Though, the format for different tests various, which is no easy to be parsed. ) was: Canary test to expose results per table/ per region to a result file. In the result file, it should provide a summary of the canary test and shows which region has read or write failures. Also, include some stats regarding the test. Such as, "region read count: 500, region read success count: 499" > Canary test to expose results per table/ per region to result file > -- > > Key: HBASE-20925 > URL: https://issues.apache.org/jira/browse/HBASE-20925 > Project: HBase > Issue Type: Improvement > Components: canary >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Major > > This change will provide a new cmd line argument for users to specify as > "-verboseTestResultFilePath" and Canary will output structured/ easy to be > parsed/ easy to be read strings to the file. > In the result file, it should provide a summary of the canary test and shows > which region has read or write failures. > Also to include some stats regarding the test. Such as, "region read count: > 500, region read success count: 499" > > (Before, Canary test sends some of above information into log file, mixed > with other debugging information. Though, the format for different tests > various, > which is no easy to be parsed. ) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20925) Canary test to expose results per table/ per region to result file
[ https://issues.apache.org/jira/browse/HBASE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-20925: Description: Canary test to expose results per table/ per region to a result file. In the result file, it should provide a summary of the canary test and shows which region has read or write failures. Also, include some stats regarding the test. Such as, "region read count: 500, region read success count: 499" was:Canary test to expose results per table. Summary: Canary test to expose results per table/ per region to result file (was: Canary test to expose results per table) > Canary test to expose results per table/ per region to result file > -- > > Key: HBASE-20925 > URL: https://issues.apache.org/jira/browse/HBASE-20925 > Project: HBase > Issue Type: Improvement > Components: canary >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Major > > Canary test to expose results per table/ per region to a result file. > In the result file, it should provide a summary of the canary test and shows > which region has read or write failures. > Also, include some stats regarding the test. Such as, "region read count: > 500, region read success count: 499" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555997#comment-16555997
]
Reid Chan commented on HBASE-20886:
---
bq. One final thought: ...
Multiple credentials-like issues always happens on server-side, e.g, hbase
thrift server. But i doubt client-side should handle this, or leave it to
client's application i think...
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555985#comment-16555985
]
Reid Chan commented on HBASE-20886:
---
v5 addresses the following:
* bq.Could you expand this check to include the principal of the current user
with krb credentials against the specified principal in the configuration?
* bq. update the javadoc for AuthUtil
* bq. make AuthUtil IA.Private in 3.0
* bq. mark AuthUtil as deprecated in any earlier release lines
* update both ConnectionFactory class javadocs and the ["Client-side
Configuration for Secure Operation
" section of the ref
guide|http://hbase.apache.org/book.html#_client_side_configuration_for_secure_operation]
i'll also update release note later.
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20934) Create an hbase-connectors repository; commit new kafka connect here
[ https://issues.apache.org/jira/browse/HBASE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20934: -- Component/s: mapreduce > Create an hbase-connectors repository; commit new kafka connect here > > > Key: HBASE-20934 > URL: https://issues.apache.org/jira/browse/HBASE-20934 > Project: HBase > Issue Type: Bug > Components: kafka, mapreduce, REST, spark, Thrift >Affects Versions: 2.1.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.2.0 > > > Create a new repository at hbase.apache.org and commit the new kafka proxy > here (HBASE-15320). Make sure it plays nicely with hbase core making use of > public-apis only (It does as best as I can see... but saying this anyways). > Once the kafka proxy is working, as subissue, move REST and thrift over... > SPARK too. > This might be better done for an hbase3 target. I filed it against hbase2.2 > for now. > See discussion up on dev list, "[DISCUSS] Kafka Connection, HBASE-15320", > https://s.apache.org/RQcC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20867) RS may get killed while master restarts
[
https://issues.apache.org/jira/browse/HBASE-20867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555964#comment-16555964
]
Hudson commented on HBASE-20867:
Results for branch branch-2
[build #1025 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1025/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1025//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1025//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1025//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> RS may get killed while master restarts
> ---
>
> Key: HBASE-20867
> URL: https://issues.apache.org/jira/browse/HBASE-20867
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20867.branch-2.0.001.patch,
> HBASE-20867.branch-2.0.002.patch, HBASE-20867.branch-2.0.003.patch,
> HBASE-20867.branch-2.0.004.patch, HBASE-20867.branch-2.0.005.patch,
> HBASE-20867.branch-2.0.006.patch
>
>
> If the master is dispatching a RPC call to RS when aborting. A connection
> exception may be thrown by the RPC layer(A IOException with "Connection
> closed" message in this case). The RSProcedureDispatcher will regard is as an
> un-retryable exception and pass it to UnassignProcedue.remoteCallFailed,
> which will expire the RS.
> Actually, the RS is very healthy, only the master is restarting.
> I think we should deal with those kinds of connection exceptions in
> RSProcedureDispatcher and retry the rpc call
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20846) Restore procedure locks when master restarts
[
https://issues.apache.org/jira/browse/HBASE-20846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555963#comment-16555963
]
Hudson commented on HBASE-20846:
Results for branch branch-2
[build #1025 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1025/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1025//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1025//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1025//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Restore procedure locks when master restarts
>
>
> Key: HBASE-20846
> URL: https://issues.apache.org/jira/browse/HBASE-20846
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-20846-v1.patch, HBASE-20846-v2.patch,
> HBASE-20846-v3.patch, HBASE-20846-v4.patch, HBASE-20846-v4.patch,
> HBASE-20846-v4.patch, HBASE-20846-v5.patch, HBASE-20846-v6.patch,
> HBASE-20846.branch-2.0.002.patch, HBASE-20846.branch-2.0.patch,
> HBASE-20846.patch
>
>
> Found this one when investigating ModifyTableProcedure got stuck while there
> was a MoveRegionProcedure going on after master restart.
> Though this issue can be solved by HBASE-20752. But I discovered something
> else.
> Before a MoveRegionProcedure can execute, it will hold the table's shared
> lock. so,, when a UnassignProcedure was spwaned, it will not check the
> table's shared lock since it is sure that its parent(MoveRegionProcedure) has
> aquired the table's lock.
> {code:java}
> // If there is parent procedure, it would have already taken xlock, so no
> need to take
> // shared lock here. Otherwise, take shared lock.
> if (!procedure.hasParent()
> && waitTableQueueSharedLock(procedure, table) == null) {
> return true;
> }
> {code}
> But, it is not the case when Master was restarted. The child
> procedure(UnassignProcedure) will be executed first after restart. Though it
> has a parent(MoveRegionProcedure), but apprently the parent didn't hold the
> table's lock.
> So, since it began to execute without hold the table's shared lock. A
> ModifyTableProcedure can aquire the table's exclusive lock and execute at the
> same time. Which is not possible if the master was not restarted.
> This will cause a stuck before HBASE-20752. But since HBASE-20752 has fixed,
> I wrote a simple UT to repo this case.
> I think we don't have to check the parent for table's shared lock. It is a
> shared lock, right? I think we can acquire it every time we need it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20899) Add Hadoop KMS dependency and basic HDFS at-rest encryption tests
[ https://issues.apache.org/jira/browse/HBASE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555935#comment-16555935 ] Wei-Chiu Chuang commented on HBASE-20899: - I thought this is affected by HBASE-20538, but running against the latest JDK8 (8u181) on Mac and test passed. Submit for precommit check > Add Hadoop KMS dependency and basic HDFS at-rest encryption tests > - > > Key: HBASE-20899 > URL: https://issues.apache.org/jira/browse/HBASE-20899 > Project: HBase > Issue Type: Sub-task > Components: encryption >Affects Versions: 2.0.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HBASE-20899.master.001.patch > > > We should start by adding hadoop-kms dependency in HBase test scope, and add > basic HDFS at-rest encryption tests using the hadoop-kms dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20899) Add Hadoop KMS dependency and basic HDFS at-rest encryption tests
[ https://issues.apache.org/jira/browse/HBASE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HBASE-20899: Status: Patch Available (was: Open) > Add Hadoop KMS dependency and basic HDFS at-rest encryption tests > - > > Key: HBASE-20899 > URL: https://issues.apache.org/jira/browse/HBASE-20899 > Project: HBase > Issue Type: Sub-task > Components: encryption >Affects Versions: 2.0.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HBASE-20899.master.001.patch > > > We should start by adding hadoop-kms dependency in HBase test scope, and add > basic HDFS at-rest encryption tests using the hadoop-kms dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19263) Hbase BaseLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-19263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-19263: Description: hbase master as a regionserver,BaseLoadBalancer throw Exception ArrayIndexOutOfBoundsException because of count of region is contain hbase master.However,hbase master can not resign any region except hbase-site.xml config. java.lang.ArrayIndexOutOfBoundsException: 2 at org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.getLocalityOfRegion(BaseLoadBalancer.java:843) at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$LocalityCostFunction.cost(StochasticLoadBalancer.java:1076) at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.computeCost(StochasticLoadBalancer.java:413) at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:280) at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1341) at org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:48) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) was: hbase master as a regionserver,BaseLoadBalancer throw Exception ArrayIndexOutOfBoundsException because of count of region is contain hbase master.However,hbase master can not resign any region except hbase-site.xml config. java.lang.ArrayIndexOutOfBoundsException: 2 at org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.getLocalityOfRegion(BaseLoadBalancer.java:843) at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$LocalityCostFunction.cost(StochasticLoadBalancer.java:1076) at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.computeCost(StochasticLoadBalancer.java:413) at org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:280) at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1341) at org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:48) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) > Hbase BaseLoadBalancer > -- > > Key: HBASE-19263 > URL: https://issues.apache.org/jira/browse/HBASE-19263 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 > Environment: hbase 1.2.6 >Reporter: SuperbDong >Priority: Major > > hbase master as a regionserver,BaseLoadBalancer throw Exception > ArrayIndexOutOfBoundsException because of count of region is contain hbase > master.However,hbase master can not resign any region except hbase-site.xml > config. > java.lang.ArrayIndexOutOfBoundsException: 2 > at > org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.getLocalityOfRegion(BaseLoadBalancer.java:843) > at > org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$LocalityCostFunction.cost(StochasticLoadBalancer.java:1076) > at > org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.computeCost(StochasticLoadBalancer.java:413) > at > org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:280) > at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1341) > at >
[jira] [Commented] (HBASE-20919) meta region can't be re-onlined when restarting cluster if opening rsgroup
[
https://issues.apache.org/jira/browse/HBASE-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555922#comment-16555922
]
Ted Yu commented on HBASE-20919:
{code}
2018-07-25 14:27:12,356 WARN [master/bjpg-rs4729:16000]
assignment.AssignmentManager: unable to round-robin assignment
org.apache.hadoop.hbase.HBaseIOException: RSGroupBasedLoadBalancer has not been
initialized
at
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.checkInitializedState(RSGroupBasedLoadBalancer.java:480)
{code}
Ultimately RSGroupBasedLoadBalancer would be initialized.
Shouldn't the above log be at DEBUG level since there is nothing required from
operator ?
> meta region can't be re-onlined when restarting cluster if opening rsgroup
> --
>
> Key: HBASE-20919
> URL: https://issues.apache.org/jira/browse/HBASE-20919
> Project: HBase
> Issue Type: Bug
> Components: Balancer, master, rsgroup
>Affects Versions: 2.0.1
>Reporter: chenyang
>Priority: Major
> Attachments: HBASE-20919-branch-2.0-01.patch,
> HBASE-20919-branch-2.0-02.patch, bug2.png,
> hbase-hbase-master-bjpg-rs4729.yz02.no_02patch.log,
> hbase-hbase-master-bjpg-rs4729.yz02.with_02patch.log,
> hbase-hbase-master-bjpg-rs4730.yz02.log.test
>
>
> if you open rsgroup, hbase-site.xml contains below configuration.
> {code:java}
>
> hbase.coprocessor.master.classes
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint
>
>
> hbase.master.loadbalancer.class/name>
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer
>
> {code}
> And you shut down the whole HBase cluster in the way:
> # first shut down region server one by one
> # shut down master
> Then you restart whole cluster in the way:
> # start master
> # start regionserver
> The hbase:meta region can not be re-online and the rsgroup can not be
> initialized successfully.
> master logs:
> {code:java}
> 2018-07-12 18:27:08,775 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> rsgroup.RSGro
> upInfoManagerImpl$RSGroupStartupWorker: Waiting for catalog tables to come
> online
> 2018-07-12 18:27:08,876 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> zookeeper.Met
> aTableLocator: Failed verification of hbase:meta,,1 at
> address=bjpg-rs4732.yz02,60020,1531388712053,
> exception=org.apache.hadoop.hbase.NotServingRegionExcepti
> on: hbase:meta,,1 is not online on bjpg-rs4732.yz02,60020,1531389727928
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3249)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3226)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1729)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28286)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> The logs show that hbase:meta region is not online and rsgroup keeps retrying
> to initialize.
>
> but why the hbase:meta region is not online?
> The info-level logs and jstack had not enough infomation, so I added some
> debug logs in test-source-code. Then i checked the master`s logs and region
> server`s logs, and found the meta region assign procedure which hold the meta
> region lock not completed and not released the lock forever, so the
> recoverMetaProcedure could not be executed.
>
> Why the first procedure not completed and not released meta region lock?
> In the test logs, i found when assignmentManager assigned the region, it
> need to call the rsgroup balancer which have not been initialized
> completely, so throw NPE. As a result, the procedure not completed and not
> released the lock forever.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:262)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignmentPlans(AssignmentManager.java:1864)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1809)
> at
>
[jira] [Commented] (HBASE-20919) meta region can't be re-onlined when restarting cluster if opening rsgroup
[
https://issues.apache.org/jira/browse/HBASE-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555904#comment-16555904
]
Ted Yu commented on HBASE-20919:
Checked HBASE-20919-branch-2.0-02.patch which seems fine.
Triggered QA run:
https://builds.apache.org/job/PreCommit-HBASE-Build/13789/
> meta region can't be re-onlined when restarting cluster if opening rsgroup
> --
>
> Key: HBASE-20919
> URL: https://issues.apache.org/jira/browse/HBASE-20919
> Project: HBase
> Issue Type: Bug
> Components: Balancer, master, rsgroup
>Affects Versions: 2.0.1
>Reporter: chenyang
>Priority: Major
> Attachments: HBASE-20919-branch-2.0-01.patch,
> HBASE-20919-branch-2.0-02.patch, bug2.png,
> hbase-hbase-master-bjpg-rs4729.yz02.no_02patch.log,
> hbase-hbase-master-bjpg-rs4729.yz02.with_02patch.log,
> hbase-hbase-master-bjpg-rs4730.yz02.log.test
>
>
> if you open rsgroup, hbase-site.xml contains below configuration.
> {code:java}
>
> hbase.coprocessor.master.classes
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint
>
>
> hbase.master.loadbalancer.class/name>
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer
>
> {code}
> And you shut down the whole HBase cluster in the way:
> # first shut down region server one by one
> # shut down master
> Then you restart whole cluster in the way:
> # start master
> # start regionserver
> The hbase:meta region can not be re-online and the rsgroup can not be
> initialized successfully.
> master logs:
> {code:java}
> 2018-07-12 18:27:08,775 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> rsgroup.RSGro
> upInfoManagerImpl$RSGroupStartupWorker: Waiting for catalog tables to come
> online
> 2018-07-12 18:27:08,876 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> zookeeper.Met
> aTableLocator: Failed verification of hbase:meta,,1 at
> address=bjpg-rs4732.yz02,60020,1531388712053,
> exception=org.apache.hadoop.hbase.NotServingRegionExcepti
> on: hbase:meta,,1 is not online on bjpg-rs4732.yz02,60020,1531389727928
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3249)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3226)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1729)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28286)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> The logs show that hbase:meta region is not online and rsgroup keeps retrying
> to initialize.
>
> but why the hbase:meta region is not online?
> The info-level logs and jstack had not enough infomation, so I added some
> debug logs in test-source-code. Then i checked the master`s logs and region
> server`s logs, and found the meta region assign procedure which hold the meta
> region lock not completed and not released the lock forever, so the
> recoverMetaProcedure could not be executed.
>
> Why the first procedure not completed and not released meta region lock?
> In the test logs, i found when assignmentManager assigned the region, it
> need to call the rsgroup balancer which have not been initialized
> completely, so throw NPE. As a result, the procedure not completed and not
> released the lock forever.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:262)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignmentPlans(AssignmentManager.java:1864)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1809)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$400(AssignmentManager.java:113)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager$2.run(AssignmentManager.java:1693)
> {code}
> !bug2.png!
> As shown in the figure named bug2.png listed in attachments, when we shutdown
> the last region server, the master submit a ServerCrashProcedure. In the
> procedure, it will
[jira] [Commented] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.
[ https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555870#comment-16555870 ] Xu Cang commented on HBASE-18822: - [~openinx] thanks for showing this patch. One question, seems like this patch makes the best effort to sync tables but does not guarantee all synced. Especially when I read #syncSchemaModificationToPeer, I realize it only modifies table when cf count is different. So there are still some work users need to do to verify/make sure table schema is synced. > Create table for peer cluster automatically when creating table in source > cluster of using namespace replication. > - > > Key: HBASE-18822 > URL: https://issues.apache.org/jira/browse/HBASE-18822 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0-alpha-2 >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-18822.v1.patch, HBASE-18822.v1.patch > > > In our cluster of using namespace replication, we always forget to create > table in peer cluster, which lead to replication get stuck. > We have implemented the feature in our cluster: create table for peer > cluster automatically when creating table in source cluster of using > namespace replication. > > I'm not sure if someone else needs this feature, so create an issue here for > discussing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20928) Rewrite calculation of midpoint in binarySearch functions to prevent overflow
[
https://issues.apache.org/jira/browse/HBASE-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20928:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Rewrite calculation of midpoint in binarySearch functions to prevent overflow
> -
>
> Key: HBASE-20928
> URL: https://issues.apache.org/jira/browse/HBASE-20928
> Project: HBase
> Issue Type: Bug
> Components: io
>Reporter: saurabh singh
>Assignee: saurabh singh
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HBASE-20928-addendum.patch,
> HBASE-20928-fix-binarySearch-v5.patch, HBASE-20928-fix-binarySearch-v5.patch
>
>
> There are couple of issues in the function:
> * {{>>>}} operator would mess the values if {{low}} + {{high}} end up being
> negative. This shouldn't happen but I don't see anything to prevent this from
> happening.
> * The code fails around boundary values of {{low}} and {{high}}. This is a
> well known binary search catch.
> [https://ai.googleblog.com/2006/06/extra-extra-read-all-about-it-nearly.html]
>
> Most of the code should already be covered by tests. I would have liked to
> add a test that actually fails without the fix but given these are private
> methods I am not sure on the best place to add the test. Suggestions?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20782) Fix duplication of TestServletFilter.access
[
https://issues.apache.org/jira/browse/HBASE-20782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated HBASE-20782:
Attachment: HBASE-20782.master.003.patch
> Fix duplication of TestServletFilter.access
> ---
>
> Key: HBASE-20782
> URL: https://issues.apache.org/jira/browse/HBASE-20782
> Project: HBase
> Issue Type: Improvement
>Reporter: Jan Hentschel
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-20782.master.001.patch,
> HBASE-20782.master.002.patch, HBASE-20782.master.003.patch,
> HBASE-20782.master.003.patch
>
>
> The {{access}} method in {{TestServletFilter}} is duplicated in
> {{TestPathFilter}}. The method should be moved into a common place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.
[ https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-18822: - Attachment: HBASE-18822.v1.patch > Create table for peer cluster automatically when creating table in source > cluster of using namespace replication. > - > > Key: HBASE-18822 > URL: https://issues.apache.org/jira/browse/HBASE-18822 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0-alpha-2 >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-18822.v1.patch, HBASE-18822.v1.patch > > > In our cluster of using namespace replication, we always forget to create > table in peer cluster, which lead to replication get stuck. > We have implemented the feature in our cluster: create table for peer > cluster automatically when creating table in source cluster of using > namespace replication. > > I'm not sure if someone else needs this feature, so create an issue here for > discussing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.
[ https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555816#comment-16555816 ] Ashish Singhi commented on HBASE-18822: --- I get that point [~openinx]. My only concern is, unless explicitly a user specify any table for replication all the tables will be synced to peer cluster, which IMO is not correct. If others here think otherwise also I'm fine with it. I don't want to be a blocker. > Create table for peer cluster automatically when creating table in source > cluster of using namespace replication. > - > > Key: HBASE-18822 > URL: https://issues.apache.org/jira/browse/HBASE-18822 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0-alpha-2 >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-18822.v1.patch > > > In our cluster of using namespace replication, we always forget to create > table in peer cluster, which lead to replication get stuck. > We have implemented the feature in our cluster: create table for peer > cluster automatically when creating table in source cluster of using > namespace replication. > > I'm not sure if someone else needs this feature, so create an issue here for > discussing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20919) meta region can't be re-onlined when restarting cluster if opening rsgroup
[
https://issues.apache.org/jira/browse/HBASE-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555778#comment-16555778
]
chenyang commented on HBASE-20919:
--
hi, Ted Yu
HBASE-20919-branch-2.0-01.patch is deprecated.
HBASE-20919-branch-2.0-02.patch offers a better solution which not block
current thread.
Please review HBASE-20919-branch-2.0-02.patch, thank you.
> meta region can't be re-onlined when restarting cluster if opening rsgroup
> --
>
> Key: HBASE-20919
> URL: https://issues.apache.org/jira/browse/HBASE-20919
> Project: HBase
> Issue Type: Bug
> Components: Balancer, master, rsgroup
>Affects Versions: 2.0.1
>Reporter: chenyang
>Priority: Major
> Attachments: HBASE-20919-branch-2.0-01.patch,
> HBASE-20919-branch-2.0-02.patch, bug2.png,
> hbase-hbase-master-bjpg-rs4729.yz02.no_02patch.log,
> hbase-hbase-master-bjpg-rs4729.yz02.with_02patch.log,
> hbase-hbase-master-bjpg-rs4730.yz02.log.test
>
>
> if you open rsgroup, hbase-site.xml contains below configuration.
> {code:java}
>
> hbase.coprocessor.master.classes
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint
>
>
> hbase.master.loadbalancer.class/name>
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer
>
> {code}
> And you shut down the whole HBase cluster in the way:
> # first shut down region server one by one
> # shut down master
> Then you restart whole cluster in the way:
> # start master
> # start regionserver
> The hbase:meta region can not be re-online and the rsgroup can not be
> initialized successfully.
> master logs:
> {code:java}
> 2018-07-12 18:27:08,775 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> rsgroup.RSGro
> upInfoManagerImpl$RSGroupStartupWorker: Waiting for catalog tables to come
> online
> 2018-07-12 18:27:08,876 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> zookeeper.Met
> aTableLocator: Failed verification of hbase:meta,,1 at
> address=bjpg-rs4732.yz02,60020,1531388712053,
> exception=org.apache.hadoop.hbase.NotServingRegionExcepti
> on: hbase:meta,,1 is not online on bjpg-rs4732.yz02,60020,1531389727928
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3249)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3226)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1729)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28286)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> The logs show that hbase:meta region is not online and rsgroup keeps retrying
> to initialize.
>
> but why the hbase:meta region is not online?
> The info-level logs and jstack had not enough infomation, so I added some
> debug logs in test-source-code. Then i checked the master`s logs and region
> server`s logs, and found the meta region assign procedure which hold the meta
> region lock not completed and not released the lock forever, so the
> recoverMetaProcedure could not be executed.
>
> Why the first procedure not completed and not released meta region lock?
> In the test logs, i found when assignmentManager assigned the region, it
> need to call the rsgroup balancer which have not been initialized
> completely, so throw NPE. As a result, the procedure not completed and not
> released the lock forever.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:262)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignmentPlans(AssignmentManager.java:1864)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1809)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$400(AssignmentManager.java:113)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager$2.run(AssignmentManager.java:1693)
> {code}
> !bug2.png!
> As shown in the figure named bug2.png listed in attachments, when we shutdown
> the last region server,
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Attachment: HBASE-20886.master.005.patch
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20649) Validate HFiles do not have PREFIX_TREE DataBlockEncoding
[ https://issues.apache.org/jira/browse/HBASE-20649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20649: Component/s: tooling Operability > Validate HFiles do not have PREFIX_TREE DataBlockEncoding > - > > Key: HBASE-20649 > URL: https://issues.apache.org/jira/browse/HBASE-20649 > Project: HBase > Issue Type: New Feature > Components: Operability, tooling >Reporter: Peter Somogyi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20649.master.001.patch, > HBASE-20649.master.002.patch, HBASE-20649.master.003.patch, > HBASE-20649.master.004.patch, HBASE-20649.master.005.patch, > HBASE-20649.master.006.patch > > > HBASE-20592 adds a tool to check column families on the cluster do not have > PREFIX_TREE encoding. > Since it is possible that DataBlockEncoding was already changed but HFiles > are not rewritten yet we would need a tool that can verify the content of > hfiles in the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.
[
https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555716#comment-16555716
]
Hadoop QA commented on HBASE-18822:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
20s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
1s{color} | {color:red} hbase-server: The patch generated 1 new + 12 unchanged
- 0 fixed = 13 total (was 12) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
19s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 42s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
42s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}225m 8s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}286m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
| | hadoop.hbase.client.TestMobSnapshotCloneIndependence |
| |
[jira] [Commented] (HBASE-20934) Create an hbase-connectors repository; commit new kafka connect here
[ https://issues.apache.org/jira/browse/HBASE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555680#comment-16555680 ] Sean Busbey commented on HBASE-20934: - mapreduce stuff too? > Create an hbase-connectors repository; commit new kafka connect here > > > Key: HBASE-20934 > URL: https://issues.apache.org/jira/browse/HBASE-20934 > Project: HBase > Issue Type: Bug > Components: kafka, REST, spark, Thrift >Affects Versions: 2.1.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.2.0 > > > Create a new repository at hbase.apache.org and commit the new kafka proxy > here (HBASE-15320). Make sure it plays nicely with hbase core making use of > public-apis only (It does as best as I can see... but saying this anyways). > Once the kafka proxy is working, as subissue, move REST and thrift over... > SPARK too. > This might be better done for an hbase3 target. I filed it against hbase2.2 > for now. > See discussion up on dev list, "[DISCUSS] Kafka Connection, HBASE-15320", > https://s.apache.org/RQcC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20938) Set version to 2.1.1-SNAPSHOT for branch-2.1
[ https://issues.apache.org/jira/browse/HBASE-20938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20938: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-2.1. > Set version to 2.1.1-SNAPSHOT for branch-2.1 > > > Key: HBASE-20938 > URL: https://issues.apache.org/jira/browse/HBASE-20938 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-20938-branch-2.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20934) Create an hbase-connectors repository; commit new kafka connect here
[ https://issues.apache.org/jira/browse/HBASE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555680#comment-16555680 ] Sean Busbey edited comment on HBASE-20934 at 7/25/18 1:41 PM: -- mapreduce stuff too? would require us clean up how our tools load implementations that are specific to an execution framework. was (Author: busbey): mapreduce stuff too? > Create an hbase-connectors repository; commit new kafka connect here > > > Key: HBASE-20934 > URL: https://issues.apache.org/jira/browse/HBASE-20934 > Project: HBase > Issue Type: Bug > Components: kafka, REST, spark, Thrift >Affects Versions: 2.1.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.2.0 > > > Create a new repository at hbase.apache.org and commit the new kafka proxy > here (HBASE-15320). Make sure it plays nicely with hbase core making use of > public-apis only (It does as best as I can see... but saying this anyways). > Once the kafka proxy is working, as subissue, move REST and thrift over... > SPARK too. > This might be better done for an hbase3 target. I filed it against hbase2.2 > for now. > See discussion up on dev list, "[DISCUSS] Kafka Connection, HBASE-15320", > https://s.apache.org/RQcC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-11715) HBase should provide a tool to compare 2 remote tables.
[ https://issues.apache.org/jira/browse/HBASE-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555943#comment-16555943 ] Divesh Jain edited comment on HBASE-11715 at 7/25/18 4:46 PM: -- How about introducing and using a new tool that uses the following approach: 1) Takes an HBase snapshot of the table on both the clusters. 2) Launches a map reduce job to calculate the total number of row keys, min key, max key and hash sum on both cluster. 3) Compare the above values with the remote cluster to figure out if the data is same or not. 4) Delete HBase snapshot taken in step 1. Since only a few bytes worth of data is transferred, the tool will perform faster than VerifyReplicatedData. We can also look at extending the functionality of VerifyReplicatedData to operate on fast mode using the above logic. * The tool can also be configurable enough to accept start and end row keys, enabling/disabling raw scan, accept start time and end time range, etc. > HBase should provide a tool to compare 2 remote tables. > --- > > Key: HBASE-11715 > URL: https://issues.apache.org/jira/browse/HBASE-11715 > Project: HBase > Issue Type: New Feature > Components: util >Reporter: Jean-Marc Spaggiari >Priority: Major > > As discussed in the mailing list, when a table is copied to another cluster > and need to be validated against the first one, only VerifyReplication can be > used. However, this can be very long since data need to be copied again. > We should provide an easier and faster way to compare the tables. > One option is to calculate hashs per ranges. User can define number of > buckets, then we split the table into this number of buckets and calculate an > hash for each (Like partitioner is already doing). We can also optionally > calculate an overall CRC to reduce even more hash collision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
