[jira] [Commented] (HBASE-17802) Add note that minor versions can add methods to Interfaces
[ https://issues.apache.org/jira/browse/HBASE-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931053#comment-15931053 ] Yu Li commented on HBASE-17802: --- +1 > Add note that minor versions can add methods to Interfaces > -- > > Key: HBASE-17802 > URL: https://issues.apache.org/jira/browse/HBASE-17802 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Attachments: HBASE-17802.master.001.patch > > > Clarify that adding methods to Interfaces in minor releases is allowed and > that we'll always try to do it in a backward compatible way. > Here is discussion from the list: > http://search-hadoop.com/m/HBase/YGbbQfpjp1kozD7?subj=About+adding+methods+to+an+interface+which+is+part+of+our+public+API+in+minor+release -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17803) PE always re-creates table when we specify the split policy
[
https://issues.apache.org/jira/browse/HBASE-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931051#comment-15931051
]
Jerry He commented on HBASE-17803:
--
+1
> PE always re-creates table when we specify the split policy
> ---
>
> Key: HBASE-17803
> URL: https://issues.apache.org/jira/browse/HBASE-17803
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17803.v0.patch
>
>
> I find this bug when i run the tests for HBASE-17623
> The critical code is shown below.
> {noformat}
> if ((exists && opts.presplitRegions != DEFAULT_OPTS.presplitRegions)
> || (!isReadCmd && desc != null && desc.getRegionSplitPolicyClassName()
> != opts.splitPolicy)
> || (!isReadCmd && desc != null && desc.getRegionReplication() !=
> opts.replicas)) {
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17803) PE always re-creates table when we specify the split policy
[
https://issues.apache.org/jira/browse/HBASE-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17803:
---
Status: Patch Available (was: Open)
> PE always re-creates table when we specify the split policy
> ---
>
> Key: HBASE-17803
> URL: https://issues.apache.org/jira/browse/HBASE-17803
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17803.v0.patch
>
>
> I find this bug when i run the tests for HBASE-17623
> The critical code is shown below.
> {noformat}
> if ((exists && opts.presplitRegions != DEFAULT_OPTS.presplitRegions)
> || (!isReadCmd && desc != null && desc.getRegionSplitPolicyClassName()
> != opts.splitPolicy)
> || (!isReadCmd && desc != null && desc.getRegionReplication() !=
> opts.replicas)) {
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17803) PE always re-creates table when we specify the split policy
[
https://issues.apache.org/jira/browse/HBASE-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17803:
---
Attachment: HBASE-17803.v0.patch
> PE always re-creates table when we specify the split policy
> ---
>
> Key: HBASE-17803
> URL: https://issues.apache.org/jira/browse/HBASE-17803
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17803.v0.patch
>
>
> I find this bug when i run the tests for HBASE-17623
> The critical code is shown below.
> {noformat}
> if ((exists && opts.presplitRegions != DEFAULT_OPTS.presplitRegions)
> || (!isReadCmd && desc != null && desc.getRegionSplitPolicyClassName()
> != opts.splitPolicy)
> || (!isReadCmd && desc != null && desc.getRegionReplication() !=
> opts.replicas)) {
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17803) PE always re-creates table when we specify the split policy
Chia-Ping Tsai created HBASE-17803:
--
Summary: PE always re-creates table when we specify the split
policy
Key: HBASE-17803
URL: https://issues.apache.org/jira/browse/HBASE-17803
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0, 1.4.0
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
Priority: Minor
Fix For: 2.0.0, 1.4.0
I find this bug when i run the tests for HBASE-17623
The critical code is shown below.
{noformat}
if ((exists && opts.presplitRegions != DEFAULT_OPTS.presplitRegions)
|| (!isReadCmd && desc != null && desc.getRegionSplitPolicyClassName() !=
opts.splitPolicy)
|| (!isReadCmd && desc != null && desc.getRegionReplication() !=
opts.replicas)) {
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17392) Load DefaultStoreEngine when user misconfigures 'hbase.hstore.engine.class'
[
https://issues.apache.org/jira/browse/HBASE-17392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931034#comment-15931034
]
Hadoop QA commented on HBASE-17392:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 110m 45s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12859386/HBASE-17392-master-001.patch
|
| JIRA Issue | HBASE-17392 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3509cb88e66f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 8e5eeb4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6133/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6133/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Load DefaultStoreEngine when user misconfigures 'hbase.hstore.engine.class'
> ---
>
> Key: HBASE-17392
> URL: https://issues.apache.org/jira/browse/HBASE-17392
> Project: HBase
> Issue Type: Improvement
>Reporter:
[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager
[
https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931035#comment-15931035
]
Hadoop QA commented on HBASE-14614:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 71 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 58s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
0s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 68 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 59s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 48s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 17s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 127m 5s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 58s {color}
| {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 214m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAdmin1 |
| | hadoop.hbase.regionserver.wal.TestAsyncLogRolling |
| |
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931023#comment-15931023 ] Ted Yu commented on HBASE-17707: Which patch were you referring to ? The latest patch was not integrated. Earlier patches were already reverted. > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, HBASE-17707-11.patch, > HBASE-17707-11.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931018#comment-15931018 ] Enis Soztutar commented on HBASE-17707: --- [~ted_yu] please revert the patch. I think the problem might be that new cost function is not generated in the [0-1] range. If you notice that all of the cost functions are in that range so that they will play well with each other. The problem seems to be that the new cost function dominates over the other cost functions including region replicas. The reason region replicas have a very high cost associated is that we are implementing a "soft" requirement on replica colocating. In a plan, where there are N replicas and N servers, the super high cost ensures that all replicas are immediately moved to separate servers. However, in cases where N replicas, but New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, HBASE-17707-11.patch, > HBASE-17707-11.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17754) [C++] RawAsyncTable
[
https://issues.apache.org/jira/browse/HBASE-17754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar resolved HBASE-17754.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HBASE-14850
Pushed to branch.
> [C++] RawAsyncTable
> ---
>
> Key: HBASE-17754
> URL: https://issues.apache.org/jira/browse/HBASE-17754
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: hbase-17754_v0.patch, hbase-17754_v1.patch,
> hbase-17754-v2.patch, hbase-17754-v3.patch, hbase-17754-v4.patch
>
>
> We need RawAsyncTable to connect {{table.h}} to the async retrying callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17754) [C++] RawAsyncTable
[
https://issues.apache.org/jira/browse/HBASE-17754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-17754:
--
Attachment: hbase-17754-v4.patch
Thanks Xiaobing for the review. v4 fixes a small segfault in the retry-test.
> [C++] RawAsyncTable
> ---
>
> Key: HBASE-17754
> URL: https://issues.apache.org/jira/browse/HBASE-17754
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-17754_v0.patch, hbase-17754_v1.patch,
> hbase-17754-v2.patch, hbase-17754-v3.patch, hbase-17754-v4.patch
>
>
> We need RawAsyncTable to connect {{table.h}} to the async retrying callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17798) RpcServer.Listener.Reader can abort due to CancelledKeyException
[
https://issues.apache.org/jira/browse/HBASE-17798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930969#comment-15930969
]
Andrew Purtell commented on HBASE-17798:
If I can make a suggestion, though, change:
{code}
LOG.error(getName() + ": unexpectedly exception", e);
{code}
to
{code}
LOG.error(getName() + ": error in Reader", ex);
{code}
which is the same error message we use for IOException.
> RpcServer.Listener.Reader can abort due to CancelledKeyException
>
>
> Key: HBASE-17798
> URL: https://issues.apache.org/jira/browse/HBASE-17798
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24
>Reporter: Guangxu Cheng
> Attachments: HBASE-17798-0.98-v1.patch,
> HBASE-17798-branch-1-v1.patch, HBASE-17798-master-v1.patch
>
>
> In our production cluster(0.98), some of the requests were unacceptable
> because RpcServer.Listener.Reader were aborted.
> getReader() will return the next reader to deal with request.
> The implementation of getReader() as below：
> {code:title=RpcServer.java|borderStyle=solid}
> // The method that will return the next reader to work with
> // Simplistic implementation of round robin for now
> Reader getReader() {
> currentReader = (currentReader + 1) % readers.length;
> return readers[currentReader];
> }
> {code}
> If one of the readers abort, then it will lead to fall on the reader's
> request will never be dealt with.
> Why does RpcServer.Listener.Reader abort?We add the debug log to get it.
> After a while, we got the following exception:
> {code}
> 2017-03-10 08:05:13,247 ERROR [RpcServer.reader=3,port=60020] ipc.RpcServer:
> RpcServer.listener,port=60020: unexpectedly error in Reader(Throwable)
> java.nio.channels.CancelledKeyException
> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
> at java.nio.channels.SelectionKey.isReadable(SelectionKey.java:289)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:592)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:566)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> So, when deal with the request in reader, we should handle
> CanceledKeyException.
> --
> versions 1.x and 2.0 will log and retrun when dealing with the
> InterruptedException in Reader#doRunLoop after HBASE-10521. It will lead to
> the same problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17798) RpcServer.Listener.Reader can abort due to CancelledKeyException
[
https://issues.apache.org/jira/browse/HBASE-17798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930968#comment-15930968
]
Andrew Purtell commented on HBASE-17798:
+1
> RpcServer.Listener.Reader can abort due to CancelledKeyException
>
>
> Key: HBASE-17798
> URL: https://issues.apache.org/jira/browse/HBASE-17798
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24
>Reporter: Guangxu Cheng
> Attachments: HBASE-17798-0.98-v1.patch,
> HBASE-17798-branch-1-v1.patch, HBASE-17798-master-v1.patch
>
>
> In our production cluster(0.98), some of the requests were unacceptable
> because RpcServer.Listener.Reader were aborted.
> getReader() will return the next reader to deal with request.
> The implementation of getReader() as below：
> {code:title=RpcServer.java|borderStyle=solid}
> // The method that will return the next reader to work with
> // Simplistic implementation of round robin for now
> Reader getReader() {
> currentReader = (currentReader + 1) % readers.length;
> return readers[currentReader];
> }
> {code}
> If one of the readers abort, then it will lead to fall on the reader's
> request will never be dealt with.
> Why does RpcServer.Listener.Reader abort?We add the debug log to get it.
> After a while, we got the following exception:
> {code}
> 2017-03-10 08:05:13,247 ERROR [RpcServer.reader=3,port=60020] ipc.RpcServer:
> RpcServer.listener,port=60020: unexpectedly error in Reader(Throwable)
> java.nio.channels.CancelledKeyException
> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
> at java.nio.channels.SelectionKey.isReadable(SelectionKey.java:289)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:592)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:566)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> So, when deal with the request in reader, we should handle
> CanceledKeyException.
> --
> versions 1.x and 2.0 will log and retrun when dealing with the
> InterruptedException in Reader#doRunLoop after HBASE-10521. It will lead to
> the same problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17771) [C++] Classes required for implementation of BatchCallerBuilder
[
https://issues.apache.org/jira/browse/HBASE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930967#comment-15930967
]
Enis Soztutar commented on HBASE-17771:
---
- Either return {{const Row &}} or {{std::shared_ptr}} here:
{code}
+ Row *action() const { return action_.get(); }
{code}
- {{RegionRequest:: set_action()}} should be named {{AddAction}} or
{{add_action()}}, because it is not setting an action, but adding to a list. In
terms of naming, we are trying to follow:
https://google.github.io/styleguide/cppguide.html#Function_Names. Specifically,
ordinary methods are named like {{Foo::BarBaz()}}, but "accessor" methods for
get / set are named {{Foo::bar()}}. That is why {{RegionRequest::actions()}} is
fine. Let's follow these conventions going forward.
- We should not be doing dynamic cast to Get like this here:
{code}
+if (auto pget = dynamic_cast(action)) {
+ auto pb_get = RequestConverter::ToGet(*pget);
{code}
The reason is that when we add Put objects as actions, then we have to use
{{typeid}} and will then have to depend on RTTI. Maybe we should templetize the
{{Action}} class instead of using Row? Anyway, let's leave this for a follow up
issue, because I do not want to hold on this patch for longer.
- Why is this returning a Future? You should just return
{{std::unique_ptr}}.
{code}
+folly::Future RequestConverter::ToMultiRequest(
{code}
- remove this:
{code}
+ // return
folly::makeFuture(std::move(multi_results));
{code}
- We are not gonna use region load statistics for the foreseeable future, so
you can ignore that for future patches. Feel free to keep these here since we
already have them.
{code}
+hbase::pb::MultiRegionLoadStats stats = multi_resp->regionstatistics();
{code}
- Here you are converting a unique_ptr from the factory to a shared_ptr:
{code}
+auto unique_result = ToResult(roe.result(), resp.cell_scanner());
+result = std::make_shared(*unique_result);
{code}
The efficient way to do this is something like:
{code}
+auto unique_result = ToResult(roe.result(), resp.cell_scanner());
+result = std::move(unique_result);
{code}
- {{ResponseType}} is not used, remove.
- {{+ explicit MultiResponse(const std::string& region_name);}} does not look
correct (multi response is for multiple regions). And the constructor does not
seem to be used, remove if so.
- We had talked (offline) about the maps in {{RegionRequest}} and
{{ServerRequest}} being Concurrent maps. It is pretty important to have these
data structures to be thread-safe. Remember that in the multi-request scatter /
gather logic, the Batch caller will be scheduling RPCs to multiple servers, and
each action maybe retried by the retry timer. That is why we have to have some
kind of thread-safety in these. You can either use Concurent maps from folly,
or, use mutexes for accessing these maps.
> [C++] Classes required for implementation of BatchCallerBuilder
> ---
>
> Key: HBASE-17771
> URL: https://issues.apache.org/jira/browse/HBASE-17771
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Fix For: HBASE-14850
>
> Attachments: HBASE-17771.HBASE-14850.v1.patch,
> HBASE-17771.HBASE-14850.v2.patch
>
>
> Separating depedencies of BatchCallerBuilder.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17392) Load DefaultStoreEngine when user misconfigures 'hbase.hstore.engine.class'
[ https://issues.apache.org/jira/browse/HBASE-17392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-17392: - Status: Patch Available (was: Open) > Load DefaultStoreEngine when user misconfigures 'hbase.hstore.engine.class' > --- > > Key: HBASE-17392 > URL: https://issues.apache.org/jira/browse/HBASE-17392 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17392-master-001.patch > > > When user misconfigures 'hbase.hstore.engine.class', region server complains > "Class not found" and gives up. In this case, we need to load the > DefaultStoreEngine to avoid that. Sanity check needs to be done to prevent > user from misconfiguration as well. > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java#L121 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17392) Load DefaultStoreEngine when user misconfigures 'hbase.hstore.engine.class'
[ https://issues.apache.org/jira/browse/HBASE-17392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-17392: - Attachment: HBASE-17392-master-001.patch If user configures a wrong storeEngine, log an log with more details. Only ClassNotFoundException is handled since misconfiguration will easily trigger it. > Load DefaultStoreEngine when user misconfigures 'hbase.hstore.engine.class' > --- > > Key: HBASE-17392 > URL: https://issues.apache.org/jira/browse/HBASE-17392 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17392-master-001.patch > > > When user misconfigures 'hbase.hstore.engine.class', region server complains > "Class not found" and gives up. In this case, we need to load the > DefaultStoreEngine to avoid that. Sanity check needs to be done to prevent > user from misconfiguration as well. > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java#L121 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17802) Add note that minor versions can add methods to Interfaces
[ https://issues.apache.org/jira/browse/HBASE-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930923#comment-15930923 ] Andrew Purtell commented on HBASE-17802: +1 A suggestion: s/should never/should not/ Never say never > Add note that minor versions can add methods to Interfaces > -- > > Key: HBASE-17802 > URL: https://issues.apache.org/jira/browse/HBASE-17802 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Attachments: HBASE-17802.master.001.patch > > > Clarify that adding methods to Interfaces in minor releases is allowed and > that we'll always try to do it in a backward compatible way. > Here is discussion from the list: > http://search-hadoop.com/m/HBase/YGbbQfpjp1kozD7?subj=About+adding+methods+to+an+interface+which+is+part+of+our+public+API+in+minor+release -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930917#comment-15930917 ] stack commented on HBASE-14614: --- .015 addreses [~syuanjiang] 's comments up on rb and fixes a few tests. > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-14614.master.001.patch, > HBASE-14614.master.002.patch, HBASE-14614.master.003.patch, > HBASE-14614.master.004.patch, HBASE-14614.master.005.patch, > HBASE-14614.master.006.patch, HBASE-14614.master.007.patch, > HBASE-14614.master.008.patch, HBASE-14614.master.009.patch, > HBASE-14614.master.010.patch, HBASE-14614.master.011.patch, > HBASE-14614.master.012.patch, HBASE-14614.master.012.patch, > HBASE-14614.master.013.patch, HBASE-14614.master.014.patch, > HBASE-14614.master.015.patch > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of HBASE-14616 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14614: -- Attachment: HBASE-14614.master.015.patch > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-14614.master.001.patch, > HBASE-14614.master.002.patch, HBASE-14614.master.003.patch, > HBASE-14614.master.004.patch, HBASE-14614.master.005.patch, > HBASE-14614.master.006.patch, HBASE-14614.master.007.patch, > HBASE-14614.master.008.patch, HBASE-14614.master.009.patch, > HBASE-14614.master.010.patch, HBASE-14614.master.011.patch, > HBASE-14614.master.012.patch, HBASE-14614.master.012.patch, > HBASE-14614.master.013.patch, HBASE-14614.master.014.patch, > HBASE-14614.master.015.patch > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of HBASE-14616 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17802) Add note that minor versions can add methods to Interfaces
[ https://issues.apache.org/jira/browse/HBASE-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17802: -- Attachment: HBASE-17802.master.001.patch > Add note that minor versions can add methods to Interfaces > -- > > Key: HBASE-17802 > URL: https://issues.apache.org/jira/browse/HBASE-17802 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Attachments: HBASE-17802.master.001.patch > > > Clarify that adding methods to Interfaces in minor releases is allowed and > that we'll always try to do it in a backward compatible way. > Here is discussion from the list: > http://search-hadoop.com/m/HBase/YGbbQfpjp1kozD7?subj=About+adding+methods+to+an+interface+which+is+part+of+our+public+API+in+minor+release -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17802) Add note that minor versions can add methods to Interfaces
stack created HBASE-17802: - Summary: Add note that minor versions can add methods to Interfaces Key: HBASE-17802 URL: https://issues.apache.org/jira/browse/HBASE-17802 Project: HBase Issue Type: Task Components: documentation Reporter: stack Assignee: stack Clarify that adding methods to Interfaces in minor releases is allowed and that we'll always try to do it in a backward compatible way. Here is discussion from the list: http://search-hadoop.com/m/HBase/YGbbQfpjp1kozD7?subj=About+adding+methods+to+an+interface+which+is+part+of+our+public+API+in+minor+release -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930898#comment-15930898
]
Hudson commented on HBASE-17758:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2691 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2691/])
HBASE-17758 [RSGROUP] Add shell command to move servers and tables at (tedyu:
rev 8e5eeb4db3be14371a38d0d54169a6e5c7b5f983)
* (add) hbase-shell/src/main/ruby/shell/commands/move_servers_tables_rsgroup.rb
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Fix For: 2.0.0
>
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch,
> HBASE-17758-v3.patch, HBASE-17758-v4.patch, HBASE-17758-v5.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-17758.
Resolution: Fixed
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Fix For: 2.0.0
>
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch,
> HBASE-17758-v3.patch, HBASE-17758-v4.patch, HBASE-17758-v5.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17801) Assigning dead region causing FAILED_OPEN permanent RIT that needs manual resolve
[
https://issues.apache.org/jira/browse/HBASE-17801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930742#comment-15930742
]
Ted Yu commented on HBASE-17801:
Is there a way for DeleteTableProcedure to notify ServerShutdownHandler that
these regions are being offlined ?
> Assigning dead region causing FAILED_OPEN permanent RIT that needs manual
> resolve
> --
>
> Key: HBASE-17801
> URL: https://issues.apache.org/jira/browse/HBASE-17801
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 1.1.2
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Critical
>
> In Apache 1.x, there is a Assignment Manager bug when SSH and drop table
> happens at the same time. Here is the sequence:
> (1). The Region Server hosting the target region is dead, SSH (server
> shutdown handler) offlined all regions hosted by the RS:
> {noformat}
> 2017-02-20 20:39:25,022 ERROR
> org.apache.hadoop.hbase.master.MasterRpcServices: Region server
> rs01.foo.com,60020,1486760911253 reported a fatal error:
> ABORTING region server rs01.foo.com,60020,1486760911253:
> regionserver:60020-0x55a076071923f5f,
> quorum=zk01.foo.com:2181,zk02.foo.com:2181,zk3.foo.com:2181, baseZNode=/hbase
> regionserver:60020-0x1234567890abcdf received expired from ZooKeeper, aborting
> Cause:
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode
> = Session expired
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:613)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:524)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> 2017-02-20 20:42:43,775 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs
> for rs01.foo.com,60020,1486760911253 before assignment; region count=999
> 2017-02-20 20:43:31,784 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {783a4814b862a6e23a3265a874c3048b state=OPEN, ts=1487568368296,
> server=rs01.foo.com,60020,1486760911253} to {783a4814b862a6e23a3265a874c3048b
> state=OFFLINE, ts=1487648611784, server=rs01.foo.com,60020,1486760911253}
> {noformat}
> (2). Now SSH goes through each region and check whether it should be
> re-assigned (at this time, SSH do check whether a table is disabled/deleted).
> If a region needs to be re-assigned, it would put into a list. Since at
> this time, the troubled region is still on the table that is enabled, it will
> be in the list.
> {noformat}
> 2017-02-20 20:43:31,795 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Reassigning 999
> region(s) that rs01.foo.com,60020,1486760911253 was carrying (and 0
> regions(s) that were opening on this server)
> {noformat}
> (3). Now, disable and delete table come in and also try to offline the
> region; since the region is already offlined, the deleted table just removes
> the region from meta and in-memory.
> {noformat}
> 2017-02-20 20:43:32,429 INFO org.apache.hadoop.hbase.master.HMaster:
> Client=b_kylin/null disable t1
> 2017-02-20 20:43:34,275 INFO
> org.apache.hadoop.hbase.zookeeper.ZKTableStateManager: Moving table t1 state
> from DISABLING to DISABLED
> 2017-02-20 20:43:34,276 INFO
> org.apache.hadoop.hbase.master.procedure.DisableTableProcedure: Disabled
> table, t1, is completed.
> 2017-02-20 20:43:35,624 INFO org.apache.hadoop.hbase.master.HMaster:
> Client=b_kylin/null delete t1
> 2017-02-20 20:43:36,011 INFO org.apache.hadoop.hbase.MetaTableAccessor:
> Deleted [{ENCODED => fbf9fda1381636aa5b3cd6e3fe0f6c1e, NAME =>
> 't1,,1487568367030.fbf9fda1381636aa5b3cd6e3fe0f6c1e.', STARTKEY => '', ENDKEY
> => '\x00\x01'}, {ENCODED => 783a4814b862a6e23a3265a874c3048b, NAME =>
> 't1,\x00\x01,1487568367030.783a4814b862a6e23a3265a874c3048b.', STARTKEY =>
> '\x00\x01', ENDKEY => ''}]
> {noformat}
> (4). However, SSH calls Assignment Manager to reassign the dead region (note
> that the dead region is in the re-assign list SSH collected and we don't
> re-check again)
> {noformat}
> 2017-02-20 20:43:52,725 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning but not in region
> states: {ENCODED => 783a4814b862a6e23a3265a874c3048b, NAME =>
> 't1,\x00\x01,1487568367030.783a4814b862a6e23a3265a874c3048b.', STARTKEY =>
> '\x00\x01', ENDKEY => ''}
> {noformat}
> (5). In the region server that the dead region tries to land, because the
> table is dropped, we could not open region and now the dead region is in
> FAILED_OPEN, which is in permanent RIT state.
> {noformat}
> 2017-02-20 20:43:52,861 INFO
>
[jira] [Commented] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930714#comment-15930714
]
Andrew Purtell commented on HBASE-17758:
Thanks. Missed a 'git add' I guess.
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Fix For: 2.0.0
>
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch,
> HBASE-17758-v3.patch, HBASE-17758-v4.patch, HBASE-17758-v5.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930710#comment-15930710
]
Ted Yu commented on HBASE-17758:
TestShell fails without move_servers_tables_rsgroup.rb
I have checked in move_servers_tables_rsgroup.rb
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Fix For: 2.0.0
>
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch,
> HBASE-17758-v3.patch, HBASE-17758-v4.patch, HBASE-17758-v5.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17801) Assigning dead region causing FAILED_OPEN permanent RIT that needs manual resolve
Stephen Yuan Jiang created HBASE-17801:
--
Summary: Assigning dead region causing FAILED_OPEN permanent RIT
that needs manual resolve
Key: HBASE-17801
URL: https://issues.apache.org/jira/browse/HBASE-17801
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 1.1.2
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Priority: Critical
In Apache 1.x, there is a Assignment Manager bug when SSH and drop table
happens at the same time. Here is the sequence:
(1). The Region Server hosting the target region is dead, SSH (server shutdown
handler) offlined all regions hosted by the RS:
{noformat}
2017-02-20 20:39:25,022 ERROR org.apache.hadoop.hbase.master.MasterRpcServices:
Region server rs01.foo.com,60020,1486760911253 reported a fatal error:
ABORTING region server rs01.foo.com,60020,1486760911253:
regionserver:60020-0x55a076071923f5f,
quorum=zk01.foo.com:2181,zk02.foo.com:2181,zk3.foo.com:2181, baseZNode=/hbase
regionserver:60020-0x1234567890abcdf received expired from ZooKeeper, aborting
Cause:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:613)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:524)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2017-02-20 20:42:43,775 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs
for rs01.foo.com,60020,1486760911253 before assignment; region count=999
2017-02-20 20:43:31,784 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {783a4814b862a6e23a3265a874c3048b state=OPEN, ts=1487568368296,
server=rs01.foo.com,60020,1486760911253} to {783a4814b862a6e23a3265a874c3048b
state=OFFLINE, ts=1487648611784, server=rs01.foo.com,60020,1486760911253}
{noformat}
(2). Now SSH goes through each region and check whether it should be
re-assigned (at this time, SSH do check whether a table is disabled/deleted).
If a region needs to be re-assigned, it would put into a list. Since at this
time, the troubled region is still on the table that is enabled, it will be in
the list.
{noformat}
2017-02-20 20:43:31,795 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Reassigning 999
region(s) that rs01.foo.com,60020,1486760911253 was carrying (and 0 regions(s)
that were opening on this server)
{noformat}
(3). Now, disable and delete table come in and also try to offline the region;
since the region is already offlined, the deleted table just removes the region
from meta and in-memory.
{noformat}
2017-02-20 20:43:32,429 INFO org.apache.hadoop.hbase.master.HMaster:
Client=b_kylin/null disable t1
2017-02-20 20:43:34,275 INFO
org.apache.hadoop.hbase.zookeeper.ZKTableStateManager: Moving table t1 state
from DISABLING to DISABLED
2017-02-20 20:43:34,276 INFO
org.apache.hadoop.hbase.master.procedure.DisableTableProcedure: Disabled table,
t1, is completed.
2017-02-20 20:43:35,624 INFO org.apache.hadoop.hbase.master.HMaster:
Client=b_kylin/null delete t1
2017-02-20 20:43:36,011 INFO org.apache.hadoop.hbase.MetaTableAccessor: Deleted
[{ENCODED => fbf9fda1381636aa5b3cd6e3fe0f6c1e, NAME =>
't1,,1487568367030.fbf9fda1381636aa5b3cd6e3fe0f6c1e.', STARTKEY => '', ENDKEY
=> '\x00\x01'}, {ENCODED => 783a4814b862a6e23a3265a874c3048b, NAME =>
't1,\x00\x01,1487568367030.783a4814b862a6e23a3265a874c3048b.', STARTKEY =>
'\x00\x01', ENDKEY => ''}]
{noformat}
(4). However, SSH calls Assignment Manager to reassign the dead region (note
that the dead region is in the re-assign list SSH collected and we don't
re-check again)
{noformat}
2017-02-20 20:43:52,725 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Assigning but not in region states: {ENCODED =>
783a4814b862a6e23a3265a874c3048b, NAME =>
't1,\x00\x01,1487568367030.783a4814b862a6e23a3265a874c3048b.', STARTKEY =>
'\x00\x01', ENDKEY => ''}
{noformat}
(5). In the region server that the dead region tries to land, because the
table is dropped, we could not open region and now the dead region is in
FAILED_OPEN, which is in permanent RIT state.
{noformat}
2017-02-20 20:43:52,861 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Open
t1,\x00\x01,1487568367030.783a4814b862a6e23a3265a874c3048b.
2017-02-20 20:43:52,865 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of
region=t1,\x00\x01,1487568367030.783a4814b862a6e23a3265a874c3048b., starting to
roll back the global memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
at
[jira] [Commented] (HBASE-17691) Add ScanMetrics support for async scan
[ https://issues.apache.org/jira/browse/HBASE-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930536#comment-15930536 ] stack commented on HBASE-17691: --- We should deprecate the useless metric and then remove it. Nice digging [~Apache9]. Make an issue and I'll do it. We need to develop a process for undoing no-good metrics. Thanks. > Add ScanMetrics support for async scan > -- > > Key: HBASE-17691 > URL: https://issues.apache.org/jira/browse/HBASE-17691 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17691.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930496#comment-15930496 ] stack commented on HBASE-15143: --- You want to say what of the review you addressed in the .005 patch mighty [~balazs.meszaros]? An overview helps the reviewer. Thank you sir. > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, HBASE-15143-BM-0004.patch, > HBASE-15143-BM-0005.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16247) SparkSQL Avro serialization doesn't handle enums correctly
[ https://issues.apache.org/jira/browse/HBASE-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930490#comment-15930490 ] stack commented on HBASE-16247: --- ([~jerryhe] FYI, Sean is out this week...) > SparkSQL Avro serialization doesn't handle enums correctly > -- > > Key: HBASE-16247 > URL: https://issues.apache.org/jira/browse/HBASE-16247 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > Avro's generic api expects GenericEnumSymbol as the runtime type for > instances of fields that are of Avro type ENUM. The Avro 1.7 libraries are > lax in some cases for handling this, but the 1.8 libraries are strict. We > should proactively fix our serialization. > (the lax serialization in 1.7 fails for some nested use in unions, see > AVRO-997 for details) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17754) [C++] RawAsyncTable
[
https://issues.apache.org/jira/browse/HBASE-17754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930467#comment-15930467
]
Xiaobing Zhou commented on HBASE-17754:
---
Thanks for v3. It looks good to me.
> [C++] RawAsyncTable
> ---
>
> Key: HBASE-17754
> URL: https://issues.apache.org/jira/browse/HBASE-17754
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-17754_v0.patch, hbase-17754_v1.patch,
> hbase-17754-v2.patch, hbase-17754-v3.patch
>
>
> We need RawAsyncTable to connect {{table.h}} to the async retrying callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17800) [C++] handle exceptions in client RPC
Xiaobing Zhou created HBASE-17800: - Summary: [C++] handle exceptions in client RPC Key: HBASE-17800 URL: https://issues.apache.org/jira/browse/HBASE-17800 Project: HBase Issue Type: Sub-task Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou Exceptions are ignored in current client RPC. They should be handled properly to be consumed by RPC retry or propagated up to APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe reassigned HBASE-17786:
Assignee: Umesh Agashe
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HBASE-16549) Procedure v2 - Add new AM metrics
[ https://issues.apache.org/jira/browse/HBASE-16549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe reassigned HBASE-16549: Assignee: Umesh Agashe (was: Matteo Bertozzi) > Procedure v2 - Add new AM metrics > - > > Key: HBASE-16549 > URL: https://issues.apache.org/jira/browse/HBASE-16549 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Umesh Agashe > Fix For: 2.0.0 > > > With the new AM we can add a bunch of metrics > - assign/unassign time > - server crash time > - grouping related metrics? (how many batch we do, and similar?) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930390#comment-15930390
]
Anoop Sam John commented on HBASE-17623:
Thanks for the detailed tests and report man. This looks great. Will have a
final go at the patch once more.
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, GC measurement.xlsx,
> HBASE-17623.branch-1.v0.patch, HBASE-17623.branch-1.v1.patch,
> HBASE-17623.branch-1.v2.patch, HBASE-17623.branch-1.v2.patch,
> HBASE-17623.v0.patch, HBASE-17623.v1.patch, HBASE-17623.v1.patch,
> HBASE-17623.v2.patch, memory allocation measurement.xlsx
>
>
> There are two improvements.
> # The onDiskBlockBytesWithHeader should maintain a bytes array which can be
> reused when building the hfile.
> # The onDiskBlockBytesWithHeader is copied to an new bytes array only when we
> need to cache the block.
> # If no block need to be cached, the uncompressedBlockBytesWithHeader will
> never be created.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-17623:
---
Priority: Major (was: Minor)
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, GC measurement.xlsx,
> HBASE-17623.branch-1.v0.patch, HBASE-17623.branch-1.v1.patch,
> HBASE-17623.branch-1.v2.patch, HBASE-17623.branch-1.v2.patch,
> HBASE-17623.v0.patch, HBASE-17623.v1.patch, HBASE-17623.v1.patch,
> HBASE-17623.v2.patch, memory allocation measurement.xlsx
>
>
> There are two improvements.
> # The onDiskBlockBytesWithHeader should maintain a bytes array which can be
> reused when building the hfile.
> # The onDiskBlockBytesWithHeader is copied to an new bytes array only when we
> need to cache the block.
> # If no block need to be cached, the uncompressedBlockBytesWithHeader will
> never be created.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17799) HBCK region boundaries check can return false negatives when IOExceptions are thrown
Esteban Gutierrez created HBASE-17799:
-
Summary: HBCK region boundaries check can return false negatives
when IOExceptions are thrown
Key: HBASE-17799
URL: https://issues.apache.org/jira/browse/HBASE-17799
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.5
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
When enabled, HBaseFsck#checkRegionBoundaries will crawl all HFiles across all
namespaces and tables when {{-boundaries}} is specified. However if an
IOException is thrown by accessing a corrupt HFile, an un-handled HLink or by
any other reason, we will only log the exception and stop crawling the HFiles
and potentially reporting the wrong result.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16247) SparkSQL Avro serialization doesn't handle enums correctly
[ https://issues.apache.org/jira/browse/HBASE-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930336#comment-15930336 ] Jerry He commented on HBASE-16247: -- Just want to clarify a little more. Hope I am right. As soon as the hbase spark data source implementation sees the Avro schema as a column, it converts the Avro types to Spark SQL types. ENUM is mapped to StringType. After that, ENUM will just disappear throughout the catalyst data flow, including serialization or deserialization in the hbase spark data source. > SparkSQL Avro serialization doesn't handle enums correctly > -- > > Key: HBASE-16247 > URL: https://issues.apache.org/jira/browse/HBASE-16247 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > Avro's generic api expects GenericEnumSymbol as the runtime type for > instances of fields that are of Avro type ENUM. The Avro 1.7 libraries are > lax in some cases for handling this, but the 1.8 libraries are strict. We > should proactively fix our serialization. > (the lax serialization in 1.7 fails for some nested use in unions, see > AVRO-997 for details) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930329#comment-15930329 ] Ted Yu commented on HBASE-17707: [~enis]: What do you think about the change in RegionReplicaHostCostFunction ? > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, HBASE-17707-11.patch, > HBASE-17707-11.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16356) REST API scanner: row prefix filter and custom filter parameter are mutually exclusive
[
https://issues.apache.org/jira/browse/HBASE-16356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930299#comment-15930299
]
Ben Watson commented on HBASE-16356:
I have a patch for this ticket. I presume I need to be added as the Assignee to
be able to submit it, so please could someone add me?
> REST API scanner: row prefix filter and custom filter parameter are mutually
> exclusive
> --
>
> Key: HBASE-16356
> URL: https://issues.apache.org/jira/browse/HBASE-16356
> Project: HBase
> Issue Type: Bug
> Components: REST
>Affects Versions: 1.1.2
> Environment: Not environment specific (tested on HDP 2.4.2)
>Reporter: Bjorn Olsen
>Priority: Minor
>
> A user can optionally specify a row PrefixFilter, or a list of custom
> filters, to the REST API scanner.
> Prefix filter example:
> /123*?startrow=0=9
> Custom filters example:
> /*?startrow=0=9=RowFilter(=,'substring:456)
> This works when specified separately, like above.
> However, specifying both a prefix filter and a list of custom filters causes
> the API to ignore the prefix filter.
> Example using both parameters:
> /123*?startrow=0=9=RowFilter(=,'substring:456)
> It appears that code in the TableResource.getScanResource function is causing
> this issue as follows:
> (see
> https://hbase.apache.org/devapidocs/src-html/org/apache/hadoop/hbase/rest/TableResource.html#line.196
> )
> if (filterList != null) {
> tableScan.setFilter(filterList); /*comes from custom filters parameter*/
> } else if (filter != null) {
> tableScan.setFilter(filter);
> /*comes from row prefix parameter*/
> }
> This should probably be changed to use a single filterList for both
> parameters. The prefix filter can be "Popped" onto the filter list and then
> these parameters will work even when both are specified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930268#comment-15930268 ] Kahlil Oppenheimer commented on HBASE-17707: I just added a patch that will make the RegionReplicaHostCostFunction return higher values even when the cluster is very large (a minimum value of .1 for any co-hosted replicas). I believe this solution will scale better because it will more logically preserve the constraint that the test is checking for (that absolutely no region replicas end up on the same host), even as people add new cost functions and such. > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, HBASE-17707-11.patch, > HBASE-17707-11.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kahlil Oppenheimer updated HBASE-17707: --- Attachment: HBASE-17707-11.patch > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, HBASE-17707-11.patch, > test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kahlil Oppenheimer updated HBASE-17707: --- Status: Patch Available (was: Reopened) > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, HBASE-17707-11.patch, > HBASE-17707-11.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kahlil Oppenheimer updated HBASE-17707: --- Attachment: HBASE-17707-11.patch > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, HBASE-17707-11.patch, > HBASE-17707-11.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930151#comment-15930151 ] Ted Yu commented on HBASE-17707: I mentioned something similar to option #1. >From my previous comment: bq. using the new table skew cost function when there is no region replica in the cluster and using the existing one when there is region replica > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930137#comment-15930137 ] Kahlil Oppenheimer commented on HBASE-17707: The issue here is that the tests are checking to make sure no two replicas of the same region end up on the same host, but that constraint is only enforced by having a very high weight associated with the RegionReplicaHost cost function. The issue with this is that even with a very high weight (like 5000), the cost value for this function can get really small (like .26) as the number of regions grows large. Thus, the balancer might decide to move a replica of a region to the same host as another because it benefits other cost functions (like table skew) because the RegionReplicaHost cost is so small. I have two solutions that would fix this: 1) Disable table skew generator/cost function when there are region replicas. 2) Change the RegionReplicaHost cost function to make the cost super high for any amount of replicas on the same host, regardless of how many regions are in the cluster. What are your thoughts? > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17798) RpcServer.Listener.Reader can abort due to CancelledKeyException
[
https://issues.apache.org/jira/browse/HBASE-17798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930081#comment-15930081
]
Ted Yu commented on HBASE-17798:
Did you put the proposed change in your cluster ?
If so, what effect do you observe ?
> RpcServer.Listener.Reader can abort due to CancelledKeyException
>
>
> Key: HBASE-17798
> URL: https://issues.apache.org/jira/browse/HBASE-17798
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24
>Reporter: Guangxu Cheng
> Attachments: HBASE-17798-0.98-v1.patch,
> HBASE-17798-branch-1-v1.patch, HBASE-17798-master-v1.patch
>
>
> In our production cluster(0.98), some of the requests were unacceptable
> because RpcServer.Listener.Reader were aborted.
> getReader() will return the next reader to deal with request.
> The implementation of getReader() as below：
> {code:title=RpcServer.java|borderStyle=solid}
> // The method that will return the next reader to work with
> // Simplistic implementation of round robin for now
> Reader getReader() {
> currentReader = (currentReader + 1) % readers.length;
> return readers[currentReader];
> }
> {code}
> If one of the readers abort, then it will lead to fall on the reader's
> request will never be dealt with.
> Why does RpcServer.Listener.Reader abort?We add the debug log to get it.
> After a while, we got the following exception:
> {code}
> 2017-03-10 08:05:13,247 ERROR [RpcServer.reader=3,port=60020] ipc.RpcServer:
> RpcServer.listener,port=60020: unexpectedly error in Reader(Throwable)
> java.nio.channels.CancelledKeyException
> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
> at java.nio.channels.SelectionKey.isReadable(SelectionKey.java:289)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:592)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:566)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> So, when deal with the request in reader, we should handle
> CanceledKeyException.
> --
> versions 1.x and 2.0 will log and retrun when dealing with the
> InterruptedException in Reader#doRunLoop after HBASE-10521. It will lead to
> the same problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929913#comment-15929913
]
Hadoop QA commented on HBASE-17125:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 59s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 14s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 99m 33s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 144m 59s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12859270/HBASE-17125.master.006.patch
|
| JIRA Issue | HBASE-17125 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7392b7ea497a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0ed50f4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6130/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6130/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Inconsistent result when use filter to
[jira] [Updated] (HBASE-17798) RpcServer.Listener.Reader can abort due to CancelledKeyException
[
https://issues.apache.org/jira/browse/HBASE-17798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guangxu Cheng updated HBASE-17798:
--
Attachment: HBASE-17798-master-v1.patch
HBASE-17798-branch-1-v1.patch
HBASE-17798-0.98-v1.patch
> RpcServer.Listener.Reader can abort due to CancelledKeyException
>
>
> Key: HBASE-17798
> URL: https://issues.apache.org/jira/browse/HBASE-17798
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24
>Reporter: Guangxu Cheng
> Attachments: HBASE-17798-0.98-v1.patch,
> HBASE-17798-branch-1-v1.patch, HBASE-17798-master-v1.patch
>
>
> In our production cluster(0.98), some of the requests were unacceptable
> because RpcServer.Listener.Reader were aborted.
> getReader() will return the next reader to deal with request.
> The implementation of getReader() as below：
> {code:title=RpcServer.java|borderStyle=solid}
> // The method that will return the next reader to work with
> // Simplistic implementation of round robin for now
> Reader getReader() {
> currentReader = (currentReader + 1) % readers.length;
> return readers[currentReader];
> }
> {code}
> If one of the readers abort, then it will lead to fall on the reader's
> request will never be dealt with.
> Why does RpcServer.Listener.Reader abort?We add the debug log to get it.
> After a while, we got the following exception:
> {code}
> 2017-03-10 08:05:13,247 ERROR [RpcServer.reader=3,port=60020] ipc.RpcServer:
> RpcServer.listener,port=60020: unexpectedly error in Reader(Throwable)
> java.nio.channels.CancelledKeyException
> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
> at java.nio.channels.SelectionKey.isReadable(SelectionKey.java:289)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:592)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:566)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> So, when deal with the request in reader, we should handle
> CanceledKeyException.
> --
> versions 1.x and 2.0 will log and retrun when dealing with the
> InterruptedException in Reader#doRunLoop after HBASE-10521. It will lead to
> the same problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17798) RpcServer.Listener.Reader can abort due to CancelledKeyException
Guangxu Cheng created HBASE-17798:
-
Summary: RpcServer.Listener.Reader can abort due to
CancelledKeyException
Key: HBASE-17798
URL: https://issues.apache.org/jira/browse/HBASE-17798
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.24, 1.2.4, 1.3.0, 2.0.0
Reporter: Guangxu Cheng
In our production cluster(0.98), some of the requests were unacceptable because
RpcServer.Listener.Reader were aborted.
getReader() will return the next reader to deal with request.
The implementation of getReader() as below：
{code:title=RpcServer.java|borderStyle=solid}
// The method that will return the next reader to work with
// Simplistic implementation of round robin for now
Reader getReader() {
currentReader = (currentReader + 1) % readers.length;
return readers[currentReader];
}
{code}
If one of the readers abort, then it will lead to fall on the reader's request
will never be dealt with.
Why does RpcServer.Listener.Reader abort?We add the debug log to get it.
After a while, we got the following exception:
{code}
2017-03-10 08:05:13,247 ERROR [RpcServer.reader=3,port=60020] ipc.RpcServer:
RpcServer.listener,port=60020: unexpectedly error in Reader(Throwable)
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
at java.nio.channels.SelectionKey.isReadable(SelectionKey.java:289)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:592)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:566)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
So, when deal with the request in reader, we should handle CanceledKeyException.
--
versions 1.x and 2.0 will log and retrun when dealing with the
InterruptedException in Reader#doRunLoop after HBASE-10521. It will lead to the
same problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Reopened] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai reopened HBASE-17758:
move_servers_tables_rsgroup.rb disappeared
[~apurtell] Would you please check it? Thanks.
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Fix For: 2.0.0
>
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch,
> HBASE-17758-v3.patch, HBASE-17758-v4.patch, HBASE-17758-v5.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929723#comment-15929723
]
Guanghao Zhang commented on HBASE-17125:
Open a issue HBASE-17797 to introduce a filter to implement read the special
number of versions of column.
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch,
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HBASE-17797) Add a filter to implement the function which return the special number of versions of each column
[ https://issues.apache.org/jira/browse/HBASE-17797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-17797: -- Assignee: Guanghao Zhang > Add a filter to implement the function which return the special number of > versions of each column > - > > Key: HBASE-17797 > URL: https://issues.apache.org/jira/browse/HBASE-17797 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > > After HBASE-17125, ScanQueryMatch will first check column then check by > filter. The scan/get will get consistent result when use filter to read data. > But scan/get setMaxVersions() can not return the special number of versions > of each column. So this issue will introduce a new filter to implement this > function which return the special number of versions of each column. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17797) Add a filter to implement the function which return the special number of versions of each column
[ https://issues.apache.org/jira/browse/HBASE-17797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17797: --- Affects Version/s: 2.0.0 > Add a filter to implement the function which return the special number of > versions of each column > - > > Key: HBASE-17797 > URL: https://issues.apache.org/jira/browse/HBASE-17797 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang > > After HBASE-17125, ScanQueryMatch will first check column then check by > filter. The scan/get will get consistent result when use filter to read data. > But scan/get setMaxVersions() can not return the special number of versions > of each column. So this issue will introduce a new filter to implement this > function which return the special number of versions of each column. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17797) Add a filter to implement the function which return the special number of versions of each column
Guanghao Zhang created HBASE-17797: -- Summary: Add a filter to implement the function which return the special number of versions of each column Key: HBASE-17797 URL: https://issues.apache.org/jira/browse/HBASE-17797 Project: HBase Issue Type: Bug Reporter: Guanghao Zhang After HBASE-17125, ScanQueryMatch will first check column then check by filter. The scan/get will get consistent result when use filter to read data. But scan/get setMaxVersions() can not return the special number of versions of each column. So this issue will introduce a new filter to implement this function which return the special number of versions of each column. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17125:
---
Attachment: HBASE-17125.master.006.patch
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch,
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17691) Add ScanMetrics support for async scan
[
https://issues.apache.org/jira/browse/HBASE-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929685#comment-15929685
]
Hadoop QA commented on HBASE-17691:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 18s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12859241/HBASE-17691.patch |
| JIRA Issue | HBASE-17691 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux b6eaa59c253e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0ed50f4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6128/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6128/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-17691) Add ScanMetrics support for async scan
[
https://issues.apache.org/jira/browse/HBASE-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929597#comment-15929597
]
Duo Zhang commented on HBASE-17691:
---
Add an onScanMetricsCreated method to RawScanResultConsumer and
ScanResultConsumer to pass the ScanMetrics instance to user.
And I haven't implemented the {{sumOfMillisSecBetweenNexts}} as the current
implementation seems uesless. It will record lastNext after calling next on
scanner, and use the current lastNext minus the previous lastNext and calculate
the sum. The final result just equals the whole time of the scan operation, and
in the old implementation we can only get ScanMetrics after we finish the scan,
so it does not make sense to keep a metric like this...
It was added far away from now so git blame can not tell anything. I guess the
metric is used to collect the time used by user for processing the returned
results? I.e, we should use timeBeforeNext to minus lastTimeAfterNext to get
the value which we want to add to the {{sumOfMillisSecBetweenNexts}}?
Any thoughts sir [~stack]? Thanks.
> Add ScanMetrics support for async scan
> --
>
> Key: HBASE-17691
> URL: https://issues.apache.org/jira/browse/HBASE-17691
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17691.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17757) Unify blocksize after encoding to decrease memory fragment
[ https://issues.apache.org/jira/browse/HBASE-17757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929593#comment-15929593 ] Allan Yang commented on HBASE-17757: ping [~stack] , [~anoopsamjohn], any new idea about the patch? I think it is OK to set the default ratio to 0.5. Like the default value of datablocksize(64KB), it also is a empirical value. Anyone don't like it can change it. But for most people, it works well. > Unify blocksize after encoding to decrease memory fragment > --- > > Key: HBASE-17757 > URL: https://issues.apache.org/jira/browse/HBASE-17757 > Project: HBase > Issue Type: New Feature >Reporter: Allan Yang >Assignee: Allan Yang > Attachments: HBASE-17757.patch, HBASE-17757v2.patch > > > Usually, we store encoded block(uncompressed) in blockcache/bucketCache. > Though we have set the blocksize, after encoding, blocksize is varied. Varied > blocksize will cause memory fragment problem, which will result in more FGC > finally.In order to relief the memory fragment, This issue adjusts the > encoded block to a unified size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17229) Backport of purge ThreadLocals
[
https://issues.apache.org/jira/browse/HBASE-17229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929582#comment-15929582
]
Hoonmin Kim edited comment on HBASE-17229 at 3/17/17 8:38 AM:
--
What about 1.1 branch? Do you have any plan to backport to 1.1?
I'm experiencing the same(I think) problems with HBASE-16146, HBASE-17072 in
HBase-1.1.5.
* chart for a region server
!https://gist.githubusercontent.com/hoonmin/56c6184009677ce4da480cf3d1d46324/raw/5fb0e7a1b19db9e0e0bdd7fedb2bda6261107ce2/hbase-cpu-high.png|width=100%!
* piece of the thread dump at cpu-high
{code}
"RW.default.scanRpcServer.handler=xxx,queue=xxx,port=xxx" #214 daemon prio=5
os_prio=0 tid=0x7fe79bf46000 nid=0x28f3 runnable [0x7fd57bb41000]
java.lang.Thread.State: RUNNABLE
at java.lang.ThreadLocal$ThreadLocalMap.remove(ThreadLocal.java:496)
at java.lang.ThreadLocal$ThreadLocalMap.access$200(ThreadLocal.java:298)
at java.lang.ThreadLocal.remove(ThreadLocal.java:222)
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:426)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
at
com.yammer.metrics.stats.ExponentiallyDecayingSample.unlockForRegularUsage(ExponentiallyDecayingSample.java:196)
at
com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:113)
at
com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:81)
at
org.apache.hadoop.metrics2.lib.MutableHistogram.add(MutableHistogram.java:81)
at
org.apache.hadoop.metrics2.lib.MutableRangeHistogram.add(MutableRangeHistogram.java:59)
at
org.apache.hadoop.hbase.ipc.MetricsHBaseServerSourceImpl.dequeuedCall(MetricsHBaseServerSourceImpl.java:165)
at
org.apache.hadoop.hbase.ipc.MetricsHBaseServer.dequeuedCall(MetricsHBaseServer.java:65)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2139)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
{code}
Thanks!
was (Author: hoonmin):
What about 1.1 branch? Do you have any plan to backport to 1.1?
I'm experiencing the same(I think) problems with HBASE-16146, HBASE-17072 in
HBase-1.1.5.
!https://gist.githubusercontent.com/hoonmin/56c6184009677ce4da480cf3d1d46324/raw/5fb0e7a1b19db9e0e0bdd7fedb2bda6261107ce2/hbase-cpu-high.png|width=100%!
Thanks!
> Backport of purge ThreadLocals
> --
>
> Key: HBASE-17229
> URL: https://issues.apache.org/jira/browse/HBASE-17229
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Fix For: 1.3.1, 1.2.6
>
>
> Backport HBASE-17072 and HBASE-16146. The former needs to be backported to
> 1.3 ([~mantonov]) and 1.2 ([~busbey]). The latter is already in 1.3. Needs
> to be backported to 1.2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16014) Get and Put constructor argument lists are divergent
[
https://issues.apache.org/jira/browse/HBASE-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929586#comment-15929586
]
Hadoop QA commented on HBASE-16014:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 33s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 20s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 52s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12819895/HBASE-16014_v1.patch |
| JIRA Issue | HBASE-16014 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d1168089474b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 0ed50f4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6129/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6129/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Get and Put constructor argument lists are divergent
>
>
> Key: HBASE-16014
> URL: https://issues.apache.org/jira/browse/HBASE-16014
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: brandboat
> Fix For: 2.0.0
>
> Attachments: HBASE-16014_v0.patch, HBASE-16014_v1.patch
>
>
> API for
[jira] [Commented] (HBASE-17229) Backport of purge ThreadLocals
[ https://issues.apache.org/jira/browse/HBASE-17229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929582#comment-15929582 ] Hoonmin Kim commented on HBASE-17229: - What about 1.1 branch? Do you have any plan to backport to 1.1? I'm experiencing the same(I think) problems with HBASE-16146, HBASE-17072 in HBase-1.1.5. !https://gist.githubusercontent.com/hoonmin/56c6184009677ce4da480cf3d1d46324/raw/5fb0e7a1b19db9e0e0bdd7fedb2bda6261107ce2/hbase-cpu-high.png|width=100%! Thanks! > Backport of purge ThreadLocals > -- > > Key: HBASE-17229 > URL: https://issues.apache.org/jira/browse/HBASE-17229 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Critical > Fix For: 1.3.1, 1.2.6 > > > Backport HBASE-17072 and HBASE-16146. The former needs to be backported to > 1.3 ([~mantonov]) and 1.2 ([~busbey]). The latter is already in 1.3. Needs > to be backported to 1.2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929565#comment-15929565 ] Eshcar Hillel commented on HBASE-16417: --- Updated benchmarks report. You can see in Figure 8 that merging the indices of pipeline segments into a single index (depicted by orange bars) eliminates the overhead of reading multiple segments. In addition, figures 9 and 10 show that the performance of basic and basic with merge are equivalent in write-only workload both for the case of uniform distribution and for zipfian distribution. I also ran write-only benchmarks with big values. As we anticipated the affect of reducing the meta data decreases as the size of data itself increases. We still see the same trend with respect to write amplification with eager policy in zipfian distribution. It means that the main benefit we see in performance in the experiment is from reducing the gc. We almost don’t see any gain in throughput due to reduction in compaction. This might be due to the hardware - SSD which handles disk compaction well, or since we were not able to saturate the server enough so that compaction becomes a problem. But in any case running with basic+merge or eager is as good as running with no compaction, and we are not seeing any degradation in performance. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-16417-benchmarkresults-20161101.pdf, > HBASE-16417-benchmarkresults-20161110.pdf, > HBASE-16417-benchmarkresults-20161123.pdf, > HBASE-16417-benchmarkresults-20161205.pdf, > HBASE-16417-benchmarkresults-20170309.pdf, > HBASE-16417-benchmarkresults-20170317.pdf > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-16417: -- Attachment: HBASE-16417-benchmarkresults-20170317.pdf > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-16417-benchmarkresults-20161101.pdf, > HBASE-16417-benchmarkresults-20161110.pdf, > HBASE-16417-benchmarkresults-20161123.pdf, > HBASE-16417-benchmarkresults-20161205.pdf, > HBASE-16417-benchmarkresults-20170309.pdf, > HBASE-16417-benchmarkresults-20170317.pdf > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17796) Minor improvements on the javadoc and naming for rpc client implementation
Duo Zhang created HBASE-17796: - Summary: Minor improvements on the javadoc and naming for rpc client implementation Key: HBASE-17796 URL: https://issues.apache.org/jira/browse/HBASE-17796 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0, 1.4.0 Reporter: Duo Zhang Fix For: 2.0.0, 1.4.0 Address the comments here https://reviews.apache.org/r/57459/ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16014) Get and Put constructor argument lists are divergent
[
https://issues.apache.org/jira/browse/HBASE-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929556#comment-15929556
]
Chia-Ping Tsai commented on HBASE-16014:
LGTM. +1
[~brandboat]
Would you please rebase the patch? Consider using the "git format-patch" to
create the patch. Thanks.
> Get and Put constructor argument lists are divergent
>
>
> Key: HBASE-16014
> URL: https://issues.apache.org/jira/browse/HBASE-16014
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: brandboat
> Fix For: 2.0.0
>
> Attachments: HBASE-16014_v0.patch, HBASE-16014_v1.patch
>
>
> API for construing Get and Put objects for a specific rowkey is quite
> different.
> [Put|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html#constructor_summary]
> supports many more variations for specifying the target rowkey and timestamp
> compared to
> [Get|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html#constructor_summary].
> Notably lacking are {{Get(byte[], int, int)}} and {{Get(ByteBuffer)}}
> variations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929551#comment-15929551
]
Hudson commented on HBASE-17758:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2687 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2687/])
HBASE-17758 [RSGROUP] Add shell command to move servers and tables at
(apurtell: rev 7f0e6f1c9ec6423aee96dcbdd4240fb76f45d5a9)
* (edit) hbase-shell/src/main/ruby/shell.rb
* (edit)
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* (edit)
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
* (edit)
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* (edit) hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb
* (edit)
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
* (edit)
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
* (edit)
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
* (edit)
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupAdminProtos.java
* (edit)
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
* (edit)
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
* (edit) hbase-rsgroup/src/main/protobuf/RSGroupAdmin.proto
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Fix For: 2.0.0
>
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch,
> HBASE-17758-v3.patch, HBASE-17758-v4.patch, HBASE-17758-v5.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929553#comment-15929553 ] Hudson commented on HBASE-17707: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2687 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2687/]) HBASE-17707 New More Accurate Table Skew cost function/generator - (tedyu: rev 0ed50f42e71599a86738643822e1ec4f2e432026) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer2.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17706) TableSkewCostFunction improperly computes max skew
[ https://issues.apache.org/jira/browse/HBASE-17706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929552#comment-15929552 ] Hudson commented on HBASE-17706: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2687 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2687/]) HBASE-17706 TableSkewCostFunction improperly computes max skew - revert (tedyu: rev a69c23abfeed9246f308bf470b72a9a8afa46f5d) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java > TableSkewCostFunction improperly computes max skew > -- > > Key: HBASE-17706 > URL: https://issues.apache.org/jira/browse/HBASE-17706 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Labels: patch > Fix For: 2.0 > > Attachments: HBASE-17706-01.patch, HBASE-17706-02.patch, > HBASE-17706-03.patch, HBASE-17706-04.patch, HBASE-17706-05.patch, > HBASE-17706-06.patch, HBASE-17706-07.patch, HBASE-17706.patch > > > We noticed while running unit tests that the TableSkewCostFunction computed > cost did not change as the balancer ran and simulated moves across the > cluster. After investigating, we found that this happened in particular when > the cluster started out with at least one table very strongly skewed. > We noticed that the TableSkewCostFunction depends on a field of the > BaseLoadBalancer.Cluster class called numMaxRegionsPerTable, but this field > is not properly maintained as regionMoves are simulated for the cluster. The > field only ever increases as the maximum number of regions per table > increases, but it does not decrease as the maximum number per table goes down. > This patch corrects that behavior so that the field is accurately maintained, > and thus the TableSkewCostFunction produces a more correct value as the > balancer runs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17691) Add ScanMetrics support for async scan
[ https://issues.apache.org/jira/browse/HBASE-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17691: -- Attachment: HBASE-17691.patch > Add ScanMetrics support for async scan > -- > > Key: HBASE-17691 > URL: https://issues.apache.org/jira/browse/HBASE-17691 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17691.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17691) Add ScanMetrics support for async scan
[ https://issues.apache.org/jira/browse/HBASE-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17691: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > Add ScanMetrics support for async scan > -- > > Key: HBASE-17691 > URL: https://issues.apache.org/jira/browse/HBASE-17691 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17691.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17795) Async ReplicationSource
[ https://issues.apache.org/jira/browse/HBASE-17795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17795: -- Description: We need change ReplicationSource to async so we can push same entries to different peers at a same time. (was: We need change ReplicationEndpoint to async so we can push same entries to different peers at a same time.) > Async ReplicationSource > --- > > Key: HBASE-17795 > URL: https://issues.apache.org/jira/browse/HBASE-17795 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0 > > > We need change ReplicationSource to async so we can push same entries to > different peers at a same time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17795) Async ReplicationSource
[ https://issues.apache.org/jira/browse/HBASE-17795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17795: -- Summary: Async ReplicationSource (was: Async ReplicationEndpoint) > Async ReplicationSource > --- > > Key: HBASE-17795 > URL: https://issues.apache.org/jira/browse/HBASE-17795 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0 > > > We need change ReplicationEndpoint to async so we can push same entries to > different peers at a same time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17795) Async ReplicationEndpoint
Phil Yang created HBASE-17795: - Summary: Async ReplicationEndpoint Key: HBASE-17795 URL: https://issues.apache.org/jira/browse/HBASE-17795 Project: HBase Issue Type: Sub-task Components: Replication Reporter: Phil Yang Assignee: Phil Yang Fix For: 2.0.0 We need change ReplicationEndpoint to async so we can push same entries to different peers at a same time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17426) Inconsistent environment variable names for enabling JMX
[
https://issues.apache.org/jira/browse/HBASE-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929480#comment-15929480
]
Hudson commented on HBASE-17426:
FAILURE: Integrated in Jenkins build HBase-1.4 #674 (See
[https://builds.apache.org/job/HBase-1.4/674/])
HBASE-17426 Inconsistent environment variable names for enabling JMX (chia7712:
rev e1eaf612f6992f3339065c27cd7e3aba5e3e15a1)
* (edit) bin/hbase-config.sh
> Inconsistent environment variable names for enabling JMX
>
>
> Key: HBASE-17426
> URL: https://issues.apache.org/jira/browse/HBASE-17426
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17426.branch-1.v0.patch,
> HBASE-17426.branch-1.v0.patch, HBASE-17426.v0.patch
>
>
> In conf/hbase-env.sh :
> {code}
> # export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false"
> # If you want to configure BucketCache, specify '-XX: MaxDirectMemorySize='
> with proper direct memory size
> # export HBASE_THRIFT_OPTS="$HBASE_JMX_BASE
> -Dcom.sun.management.jmxremote.port=10103"
> {code}
> But in bin/hbase-config.sh , a different variable is used:
> {code}
> # Thrift JMX opts
> if [ -n "$HBASE_JMX_OPTS" ] && [ -z "$HBASE_THRIFT_JMX_OPTS" ]; then
> HBASE_THRIFT_JMX_OPTS="$HBASE_JMX_OPTS
> -Dcom.sun.management.jmxremote.port=10103"
> fi
> {code}
> The variable names should be aligned for better user experience.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17793) Backport ScanResultCache related code to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929479#comment-15929479 ] Hudson commented on HBASE-17793: FAILURE: Integrated in Jenkins build HBase-1.4 #674 (See [https://builds.apache.org/job/HBase-1.4/674/]) HBASE-17793 Backport ScanResultCache related code to branch-1 (zhangduo: rev 6be8d2041b9dfb99c77d824776fb39da8984bea1) * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAllowPartialScanResultCache.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BatchScanResultCache.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AllowPartialScanResultCache.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/CompleteScanResultCache.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestCompleteResultScanResultCache.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScanResultCache.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestBatchScanResultCache.java > Backport ScanResultCache related code to branch-1 > - > > Key: HBASE-17793 > URL: https://issues.apache.org/jira/browse/HBASE-17793 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-17793-branch-1.patch, HBASE-17793-branch-1-v1.patch > > > To simplity our code, and also keep the same code base for master and > branch-1. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
