date:20180609

[jira] [Commented] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507243#comment-16507243
 ] 

Hudson commented on HBASE-20698:


Results for branch branch-2
[build #844 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/844/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/844//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/844//JDK8_Nightly_Build_Report_(Hadoop2)/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/844//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch, 
> HBASE-20698.master.addendum.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19064) Synchronous replication for HBase

2018-06-09 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507238#comment-16507238
 ] 

Hudson commented on HBASE-19064:


Results for branch HBASE-19064
[build #157 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/157/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/157//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/157//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/157//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Synchronous replication for HBase
> -
>
> Key: HBASE-19064
> URL: https://issues.apache.org/jira/browse/HBASE-19064
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
>
> The guys from Alibaba made a presentation on HBaseCon Asia about the 
> synchronous replication for HBase. We(Xiaomi) think this is a very useful 
> feature for HBase so we want to bring it into the community version.
> This is a big feature so we plan to do it in a feature branch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20662) Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement

2018-06-09 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507229#comment-16507229
 ] 

Hadoop QA commented on HBASE-20662:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
45s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  1m 
20s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  1m 20s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} hbase-client: The patch generated 0 new + 5 
unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
11s{color} | {color:red} hbase-server: The patch generated 1 new + 177 
unchanged - 7 fixed = 178 total (was 184) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
48s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
9m 45s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
58s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}113m 
33s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
39s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20662 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 821aa7275d23 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |

[jira] [Updated] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Guanghao Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20698:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Pushed addendum patch to master, branch-2 and branch-2.0. Thanks [~Apache9] for 
reviewing.

> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch, 
> HBASE-20698.master.addendum.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20711) Save on a Cell iteration when writing

2018-06-09 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507211#comment-16507211
 ] 

Hadoop QA commented on HBASE-20711:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
14s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
9s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
31s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
40s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
37s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
30s{color} | {color:red} hbase-client: The patch generated 1 new + 182 
unchanged - 0 fixed = 183 total (was 182) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
38s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
10m  8s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 46s{color} 
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}143m 40s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
40s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}188m 28s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.shaded.protobuf.TestProtobufUtil |
|   | hadoop.hbase.client.TestFromClientSide |
|   | hadoop.hbase.client.TestFromClientSideWithCoprocessor |
|   | hadoop.hbase.client.TestMalformedCellFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:6f01af0 |
| JIRA Issue | HBASE-20711 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12927175/HBASE-20711.branch-2.0.001.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck

[jira] [Commented] (HBASE-20712) HBase eclipse formatter should not format the ASF license header

2018-06-09 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507203#comment-16507203
 ] 

Hadoop QA commented on HBASE-20712:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
10s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  0m 37s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20712 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12927188/HBASE-20712.master.001.patch
 |
| Optional Tests |  asflicense  xml  |
| uname | Linux 815c8fa780fb 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / 5fd16f3853 |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Max. process+thread count | 43 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13178/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> HBase eclipse formatter should not format the ASF license header
> 
>
> Key: HBASE-20712
> URL: https://issues.apache.org/jira/browse/HBASE-20712
> Project: HBase
>  Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20712.master.001.patch
>
>
> Whenever we add a new class along with the ASF license header, we cannot 
> press {{ctrl + A}} to format the whole file as it also formats the header 
> text.
> IMO we should disable formatting of headers in our code formatter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20712) HBase eclipse formatter should not format the ASF license header

2018-06-09 Thread Nihal Jain (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nihal Jain updated HBASE-20712:
---
Fix Version/s: 3.0.0
   Status: Patch Available  (was: Open)

> HBase eclipse formatter should not format the ASF license header
> 
>
> Key: HBASE-20712
> URL: https://issues.apache.org/jira/browse/HBASE-20712
> Project: HBase
>  Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20712.master.001.patch
>
>
> Whenever we add a new class along with the ASF license header, we cannot 
> press {{ctrl + A}} to format the whole file as it also formats the header 
> text.
> IMO we should disable formatting of headers in our code formatter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20712) HBase eclipse formatter should not format the ASF license header

2018-06-09 Thread Nihal Jain (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nihal Jain updated HBASE-20712:
---
Attachment: HBASE-20712.master.001.patch

> HBase eclipse formatter should not format the ASF license header
> 
>
> Key: HBASE-20712
> URL: https://issues.apache.org/jira/browse/HBASE-20712
> Project: HBase
>  Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Attachments: HBASE-20712.master.001.patch
>
>
> Whenever we add a new class along with the ASF license header, we cannot 
> press {{ctrl + A}} to format the whole file as it also formats the header 
> text.
> IMO we should disable formatting of headers in our code formatter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20712) HBase eclipse formatter should not format the ASF license header

2018-06-09 Thread Nihal Jain (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507199#comment-16507199
 ] 

Nihal Jain commented on HBASE-20712:


[~busbey] what do you think about this?

> HBase eclipse formatter should not format the ASF license header
> 
>
> Key: HBASE-20712
> URL: https://issues.apache.org/jira/browse/HBASE-20712
> Project: HBase
>  Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Attachments: HBASE-20712.master.001.patch
>
>
> Whenever we add a new class along with the ASF license header, we cannot 
> press {{ctrl + A}} to format the whole file as it also formats the header 
> text.
> IMO we should disable formatting of headers in our code formatter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20712) HBase eclipse formatter should not format the ASF license header

2018-06-09 Thread Nihal Jain (JIRA)

Nihal Jain created HBASE-20712:
--

 Summary: HBase eclipse formatter should not format the ASF license 
header
 Key: HBASE-20712
 URL: https://issues.apache.org/jira/browse/HBASE-20712
 Project: HBase
  Issue Type: Bug
Reporter: Nihal Jain
Assignee: Nihal Jain


Whenever we add a new class along with the ASF license header, we cannot press 
{{ctrl + A}} to format the whole file as it also formats the header text.

IMO we should disable formatting of headers in our code formatter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20662) Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement

2018-06-09 Thread Nihal Jain (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507189#comment-16507189
 ] 

Nihal Jain commented on HBASE-20662:


[^HBASE-20662.master.002.patch] does the following:
 * Now increasing space quota on a violated table will remove 
SpaceViolationPolicy.DISABLE enforcement
 * The table disable/enable logic has been shifted at master side, saving us on 
multiple disable RPCs from multiple RSs
 * Test class has been refactored:
 ** Created a base class for space quota tests {{SpaceQuotasTestBase}}
 ** Moved table space quota tests to {{TestSpaceQuotasOnTables}} and made it a 
parameterized class, thus removing duplicate methods
 ** Created and added namespace space quota tests in TestSpaceQuotasOnNamespaces

Also few new issues were discovered while I was writing space quota for 
namespace test:
 * {{TestSpaceQuotasOnNamespaces.testSetQuotaAndThenRemove()}}
 * {{TestSpaceQuotasOnNamespaces.testSetQuotaAndThenDropNamespace()}}
 * {{TestSpaceQuotasOnNamespaces.testSetQuotaAndThenRemoveInOne()}}

Currently I have marked these tests with {{@Ignore}} annotation, may be we can 
fix these in another JIRA and mark the with {{@Test}} there or else I can 
remove them now and add them in the JIRA where we fix them

> Increasing space quota on a violated table does not remove 
> SpaceViolationPolicy.DISABLE enforcement
> ---
>
> Key: HBASE-20662
> URL: https://issues.apache.org/jira/browse/HBASE-20662
> Project: HBase
>  Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20662.master.001.patch, 
> HBASE-20662.master.002.patch
>
>
> *Steps to reproduce*
>  * Create a table and set quota with {{SpaceViolationPolicy.DISABLE}} having 
> limit say 2MB
>  * Now put rows until space quota is violated and table gets disabled
>  * Next, increase space quota with limit say 4MB on the table
>  * Now try putting a row into the table
> {code:java}
>  private void testSetQuotaThenViolateAndFinallyIncreaseQuota() throws 
> Exception {
> SpaceViolationPolicy policy = SpaceViolationPolicy.DISABLE;
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1), 
> Bytes.toBytes("to"),
>   Bytes.toBytes("reject"));
> // Do puts until we violate space policy
> final TableName tn = writeUntilViolationAndVerifyViolation(policy, put);
> // Now, increase limit
> setQuotaLimit(tn, policy, 4L);
> // Put some row now: should not violate as quota limit increased
> verifyNoViolation(policy, tn, put);
>   }
> {code}
> *Expected*
> We should be able to put data as long as newly set quota limit is not reached
> *Actual*
> We fail to put any new row even after increasing limit
> *Root cause*
> Increasing quota on a violated table triggers the table to be enabled, but 
> since the table is already in violation, the system does not allow it to be 
> enabled (may be thinking that a user is trying to enable it)
> *Relevant exception trace*
> {noformat}
> 2018-05-31 00:34:27,563 INFO  [regionserver/root1-ThinkPad-T440p:0.Chore.1] 
> client.HBaseAdmin$14(844): Started enable of 
> testSetQuotaAndThenIncreaseQuotaWithDisable0
> 2018-05-31 00:34:27,571 DEBUG 
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42525] 
> ipc.CallRunner(142): callId: 11 service: MasterService methodName: 
> EnableTable size: 104 connection: 127.0.0.1:38030 deadline: 1527707127568, 
> exception=org.apache.hadoop.hbase.security.AccessDeniedException: Enabling 
> the table 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to 
> a violated space quota.
> 2018-05-31 00:34:27,571 ERROR [regionserver/root1-ThinkPad-T440p:0.Chore.1] 
> quotas.RegionServerSpaceQuotaManager(210): Failed to disable space violation 
> policy for testSetQuotaAndThenIncreaseQuotaWithDisable0. This table will 
> remain in violation.
> org.apache.hadoop.hbase.security.AccessDeniedException: 
> org.apache.hadoop.hbase.security.AccessDeniedException: Enabling the table 
> 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to a 
> violated space quota.
>   at org.apache.hadoop.hbase.master.HMaster$6.run(HMaster.java:2275)
>   at 
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:131)
>   at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:2258)
>   at 
> org.apache.hadoop.hbase.master.MasterRpcServices.enableTable(MasterRpcServices.java:725)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
>   at

[jira] [Updated] (HBASE-20662) Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement

2018-06-09 Thread Nihal Jain (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nihal Jain updated HBASE-20662:
---
Attachment: HBASE-20662.master.002.patch

> Increasing space quota on a violated table does not remove 
> SpaceViolationPolicy.DISABLE enforcement
> ---
>
> Key: HBASE-20662
> URL: https://issues.apache.org/jira/browse/HBASE-20662
> Project: HBase
>  Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20662.master.001.patch, 
> HBASE-20662.master.002.patch
>
>
> *Steps to reproduce*
>  * Create a table and set quota with {{SpaceViolationPolicy.DISABLE}} having 
> limit say 2MB
>  * Now put rows until space quota is violated and table gets disabled
>  * Next, increase space quota with limit say 4MB on the table
>  * Now try putting a row into the table
> {code:java}
>  private void testSetQuotaThenViolateAndFinallyIncreaseQuota() throws 
> Exception {
> SpaceViolationPolicy policy = SpaceViolationPolicy.DISABLE;
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1), 
> Bytes.toBytes("to"),
>   Bytes.toBytes("reject"));
> // Do puts until we violate space policy
> final TableName tn = writeUntilViolationAndVerifyViolation(policy, put);
> // Now, increase limit
> setQuotaLimit(tn, policy, 4L);
> // Put some row now: should not violate as quota limit increased
> verifyNoViolation(policy, tn, put);
>   }
> {code}
> *Expected*
> We should be able to put data as long as newly set quota limit is not reached
> *Actual*
> We fail to put any new row even after increasing limit
> *Root cause*
> Increasing quota on a violated table triggers the table to be enabled, but 
> since the table is already in violation, the system does not allow it to be 
> enabled (may be thinking that a user is trying to enable it)
> *Relevant exception trace*
> {noformat}
> 2018-05-31 00:34:27,563 INFO  [regionserver/root1-ThinkPad-T440p:0.Chore.1] 
> client.HBaseAdmin$14(844): Started enable of 
> testSetQuotaAndThenIncreaseQuotaWithDisable0
> 2018-05-31 00:34:27,571 DEBUG 
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42525] 
> ipc.CallRunner(142): callId: 11 service: MasterService methodName: 
> EnableTable size: 104 connection: 127.0.0.1:38030 deadline: 1527707127568, 
> exception=org.apache.hadoop.hbase.security.AccessDeniedException: Enabling 
> the table 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to 
> a violated space quota.
> 2018-05-31 00:34:27,571 ERROR [regionserver/root1-ThinkPad-T440p:0.Chore.1] 
> quotas.RegionServerSpaceQuotaManager(210): Failed to disable space violation 
> policy for testSetQuotaAndThenIncreaseQuotaWithDisable0. This table will 
> remain in violation.
> org.apache.hadoop.hbase.security.AccessDeniedException: 
> org.apache.hadoop.hbase.security.AccessDeniedException: Enabling the table 
> 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to a 
> violated space quota.
>   at org.apache.hadoop.hbase.master.HMaster$6.run(HMaster.java:2275)
>   at 
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:131)
>   at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:2258)
>   at 
> org.apache.hadoop.hbase.master.MasterRpcServices.enableTable(MasterRpcServices.java:725)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
>   at 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:360)
>   at 
>

[jira] [Commented] (HBASE-20657) Retrying RPC call for ModifyTableProcedure may get stuck

2018-06-09 Thread Josh Elser (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507174#comment-16507174
 ] 

Josh Elser commented on HBASE-20657:


[~stack], just for your context, we've been running with this one internally 
and having good success with it (has been uncovering a new round of bugs).

Not sure if you want to get this in while you (we? :P) pursue the proper 
MTP/MRP fix. That would be my suggestion, anyways. Will save you headache.

> Retrying RPC call for ModifyTableProcedure may get stuck
> 
>
> Key: HBASE-20657
> URL: https://issues.apache.org/jira/browse/HBASE-20657
> Project: HBase
>  Issue Type: Bug
>  Components: Client, proc-v2
>Affects Versions: 2.0.0
>Reporter: Sergey Soldatov
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20657-1-branch-2.patch, 
> HBASE-20657-2-branch-2.patch, HBASE-20657-3-branch-2.patch, 
> HBASE-20657-testcase-branch2.patch
>
>
> Env: 2 masters, 1 RS. 
> Steps to reproduce: Active master is killed while ModifyTableProcedure is 
> executed. 
> If the table has enough regions it may come that when the secondary master 
> get active some of the regions may be closed, so once client retries the call 
> to the new active master, a new ModifyTableProcedure is created and get stuck 
> during MODIFY_TABLE_REOPEN_ALL_REGIONS state handling. That happens because:
> 1. When we are retrying from client side, we call modifyTableAsync which 
> create a procedure with a new nonce key:
> {noformat}
>  ModifyTableRequest request = 
> RequestConverter.buildModifyTableRequest(
> td.getTableName(), td, ng.getNonceGroup(), ng.newNonce());
> {noformat}
>  So on the server side, it's considered as a new procedure and starts 
> executing immediately.
> 2. When we are processing  MODIFY_TABLE_REOPEN_ALL_REGIONS we create 
> MoveRegionProcedure for each region, but it checks whether the region is 
> online (and it's not), so it fails immediately, forcing the procedure to 
> restart.
> [~an...@apache.org] saw a similar case when two concurrent ModifyTable 
> procedures were running and got stuck in the similar way. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20707) Move MissingSwitchDefault check from checkstyle to error-prone

2018-06-09 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507163#comment-16507163
 ] 

Hadoop QA commented on HBASE-20707:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m  
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
17s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hbase-checkstyle hbase-build-configuration {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
25s{color} | {color:red} hbase-common: The patch generated 2 new + 47 unchanged 
- 3 fixed = 49 total (was 50) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
16s{color} | {color:red} hbase-server: The patch generated 2 new + 13 unchanged 
- 4 fixed = 15 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
10s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
11m  6s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.7.4 or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hbase-checkstyle hbase-build-configuration {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
10s{color} | {color:green} hbase-checkstyle in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
9s{color} | {color:green} hbase-build-configuration in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
58s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 16s{color} 
| {color:red} hbase-server in the patch failed.

[jira] [Updated] (HBASE-20711) Save on a Cell iteration when writing

2018-06-09 Thread stack (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20711:
--
Status: Patch Available  (was: Open)

> Save on a Cell iteration when writing
> -
>
> Key: HBASE-20711
> URL: https://issues.apache.org/jira/browse/HBASE-20711
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Attachments: HBASE-20711.branch-2.0.001.patch
>
>
> This is a minor savings. We were doing a spin through all Cells on receipt 
> just to check their size when subsequently, we were doing an iteration of all 
> Cells to insert. It manifest as a little spike in perf output. This change 
> removes the purposed spin through Cells and just does size check as part of 
> the general Cell insert (perf spike no longer shows but the cost of the size 
> check still remains).
> There is also a minor bug fix where on receipt we were using the Puts row 
> rather than the Cells row; client may have succeeded in submitting a Cell 
> that disagreed with the hosting Mutation and it would have been written as 
> something else altogether -- with the Puts row -- rather than being rejected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-13583) AOT compile our JRuby

2018-06-09 Thread Mike Drob (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated HBASE-13583:
--
Status: Open  (was: Patch Available)

Current patch compiles straight to java bytecode/class files, does not do 
intermediate source files, and does not do linking or resolution, so we don't 
get to validate if we have illegal java calls in our ruby code.

> AOT compile our JRuby
> -
>
> Key: HBASE-13583
> URL: https://issues.apache.org/jira/browse/HBASE-13583
> Project: HBase
>  Issue Type: Improvement
>  Components: build, scripts, shell
>Reporter: Nick Dimiduk
>Assignee: Mike Drob
>Priority: Major
> Attachments: HBASE-13583.patch, HBASE-13583.v2.patch, 
> HBASE-13583.v3.patch
>
>
> Our Jruby code seems to not keep up well with Java changes. We should 
> investigate adding a compilation step for our shell and the rb scripts in bin 
> to ensure they're calling methods that exist on classes that exist. This 
> looks like as good a starting point as any: 
> https://github.com/jruby/jruby/wiki/GeneratingJavaClasses



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20707) Move MissingSwitchDefault check from checkstyle to error-prone

2018-06-09 Thread Mike Drob (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507137#comment-16507137
 ] 

Mike Drob commented on HBASE-20707:
---

[~apurtell], [~Jan Hentschel] - do you see any issues with this approach?

> Move MissingSwitchDefault check from checkstyle to error-prone
> --
>
> Key: HBASE-20707
> URL: https://issues.apache.org/jira/browse/HBASE-20707
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Attachments: HBASE-20707.patch
>
>
> Both checkstyle and error-prone can alert when a switch statement is missing 
> a default. However, because checkstyle does it via static analysis and 
> error-prone does it during compilation, e-p can detect when all cases of an 
> enum have been covered, and will _not_ warn about the needed default case.
> In fact, checkstyle explicitly mentions in their docs that even if you cover 
> all enum cases now, you should still have a default label because the enum 
> could change in the future. Which seems silly to me, because your analysis 
> tools should still be running in the future and would catch it then.
> Se we should migrate the check from checkstyle to a slightly smarter 
> error-prone check.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20707) Move MissingSwitchDefault check from checkstyle to error-prone

2018-06-09 Thread Mike Drob (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated HBASE-20707:
--
Assignee: Mike Drob
  Status: Patch Available  (was: Open)

> Move MissingSwitchDefault check from checkstyle to error-prone
> --
>
> Key: HBASE-20707
> URL: https://issues.apache.org/jira/browse/HBASE-20707
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Attachments: HBASE-20707.patch
>
>
> Both checkstyle and error-prone can alert when a switch statement is missing 
> a default. However, because checkstyle does it via static analysis and 
> error-prone does it during compilation, e-p can detect when all cases of an 
> enum have been covered, and will _not_ warn about the needed default case.
> In fact, checkstyle explicitly mentions in their docs that even if you cover 
> all enum cases now, you should still have a default label because the enum 
> could change in the future. Which seems silly to me, because your analysis 
> tools should still be running in the future and would catch it then.
> Se we should migrate the check from checkstyle to a slightly smarter 
> error-prone check.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20701) too much logging when balancer runs from BaseLoadBalancer

2018-06-09 Thread Monani Mihir (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Monani Mihir updated HBASE-20701:
-
Status: Open  (was: Patch Available)

> too much logging when balancer runs from BaseLoadBalancer
> -
>
> Key: HBASE-20701
> URL: https://issues.apache.org/jira/browse/HBASE-20701
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer
>Reporter: Monani Mihir
>Assignee: Monani Mihir
>Priority: Trivial
> Attachments: HBASE-20701-branch-1.3.patch, 
> HBASE-20701-branch-1.4.patch
>
>
> When balancer runs, it tries to find least loaded server with better locality 
> for current region. During this, we make debug level logging for each of 
> those regions. It creates too much amount of logging at debug level , we 
> should move this to trace level logging.
> {code:java}
> int getLeastLoadedTopServerForRegion (int region, int currentServer) {
> ...
> if (leastLoadedServerIndex != -1) {
> LOG.debug("Pick the least loaded server " + 
> servers[leastLoadedServerIndex].getHostname()
> + " with better locality for region " + regions[region]);
> }
> ...
> }{code}
> This was fixed in branch-2.0 as part of -HBASE-14614-  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20331) clean up shaded packaging for 2.1

2018-06-09 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507100#comment-16507100
 ] 

Hudson commented on HBASE-20331:


Results for branch HBASE-20331
[build #35 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/35/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/35//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/35//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/35//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for 
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/35//artifacts/output-integration/hadoop-2.log].
 (note that this means we didn't run on Hadoop 3)


> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
>  Issue Type: Umbrella
>  Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more 
> contexts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507068#comment-16507068
 ] 

Hudson commented on HBASE-20698:


Results for branch branch-2
[build #842 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/842/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/842//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/842//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/842//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch, 
> HBASE-20698.master.addendum.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507055#comment-16507055
 ] 

Hudson commented on HBASE-20698:


Results for branch branch-2.0
[build #407 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/407/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/407//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/407//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/407//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch, 
> HBASE-20698.master.addendum.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)

2018-06-09 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507053#comment-16507053
 ] 

stack commented on HBASE-20710:
---

@hsun do as you see best. Above was just confirmation that another sees similar 
to what you were seeing. Sounds good

> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.1
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent 
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB 
> backed ByteBuffKeyValueCell (which is default with Netty Rpc), 
> cell.getFamilyArray() will call cloneFamily() and there is again a 
> cloneFamily() in the following line of the code. since this is the critical 
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507054#comment-16507054
 ] 

Hadoop QA commented on HBASE-20698:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
15s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  1m 30s{color} 
| {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188 
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
16s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
8m 50s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}149m 
13s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}186m  0s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20698 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12927168/HBASE-20698.master.addendum.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 20f1decdd7d7 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 5fd16f3853 |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| javac | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13173/artifact/patchprocess/diff-compile-javac-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13173/testReport/ |
| Max.

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-06-09 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507051#comment-16507051
 ] 

stack commented on HBASE-20483:
---

Link to report comparing flushing rates before and after

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: 
> 0001-HBASE-20483-perpertual-flush-and-memstore-fill-branch-2.0.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-Perpetual-flush-memstore-fill-etc.-tools.patch, 
> PerpetualFlushingProgram.patch, PerpetualFlushingProgram1.2.patch, 
> SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507050#comment-16507050
 ] 

Hadoop QA commented on HBASE-20709:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
50s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
45s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
9m 52s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}110m 
22s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 46s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20709 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12927171/HBASE-20709.master.001.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 4fb5799079ab 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 5fd16f3853 |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13174/testReport/ |
| Max. process+thread count | 4465 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13174/console |
| Powered by |

[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)

2018-06-09 Thread huaxiang sun (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507046#comment-16507046
 ] 

huaxiang sun commented on HBASE-20710:
--

Thanks Stack. For the case of  MutationProto (not cell block), it is looping 
cells for each family,  the family should be checked once for all cells under 
it (no need to clone for each cell), will see if similar approach can be taken 
for cell bock.

> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.1
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent 
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB 
> backed ByteBuffKeyValueCell (which is default with Netty Rpc), 
> cell.getFamilyArray() will call cloneFamily() and there is again a 
> cloneFamily() in the following line of the code. since this is the critical 
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20188) [TESTING] Performance

2018-06-09 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507045#comment-16507045
 ] 

stack commented on HBASE-20188:
---

Update. Have been working on perf in background focused on writes. Writes were 
bottlenecking on flush; our flush in hbase2 was 2x slower than hbase1s. It was 
also erratic in that it was flushing sometimes at the limit, other times at 
well in excess of the limits. With input from the likes of [~ram_krish] and 
[~anoop.hbase], flushes are 'regular' now with same profile as hbase1. See 
HBASE-20483 for detail.

Our writes are still slower. The bottleneck now seems to be our WAL writing. 
While the dfsclient is a ball of synchronization knots, it is able to take in 
bigger blobs than our async WAL writer and so in a simple benchmark where 
region count is small, the old FSHLog does better (hbase2 is up to 30% slower 
than hbase1 in certain setups). But if you up the contention and up the region 
count so it resembles a real deploy, the async WAL starts to shine. At hundreds 
of regions, it can write faster, and almost as importantly, requires way less 
resources (To learn more see messy experiments here abouts: 
https://docs.google.com/document/d/1vZ_k6_pNR1eQxID5u1xFihuPC7FkPaJQW8c4M5eA2AQ/edit#heading=h.niiqwjd247t4).
 A few of us are working on it.

> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
>  Issue Type: Umbrella
>  Components: Performance
>Reporter: stack
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0
>
> Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, 
> HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 
> performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs 
> None_ system settings.pdf, HBase 2.0 performance evaluation - throughput 
> SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, 
> ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, 
> ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, 
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, 
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, 
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, 
> hbase-site.xml, hits.png, hits_with_fp_scheduler.png, 
> lock.127.workloadc.20180402T200918Z.svg, 
> lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, 
> total.png, tree.txt, workloadx, workloadx
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor 
> that it is much slower, that the problem is the asyncwal writing. Does 
> in-memory compaction slow us down or speed us up? What happens when you 
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something 
> about perf when 2.0.0 ships.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20711) Save on a Cell iteration when writing

2018-06-09 Thread stack (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20711:
--
Attachment: HBASE-20711.branch-2.0.001.patch

> Save on a Cell iteration when writing
> -
>
> Key: HBASE-20711
> URL: https://issues.apache.org/jira/browse/HBASE-20711
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Attachments: HBASE-20711.branch-2.0.001.patch
>
>
> This is a minor savings. We were doing a spin through all Cells on receipt 
> just to check their size when subsequently, we were doing an iteration of all 
> Cells to insert. It manifest as a little spike in perf output. This change 
> removes the purposed spin through Cells and just does size check as part of 
> the general Cell insert (perf spike no longer shows but the cost of the size 
> check still remains).
> There is also a minor bug fix where on receipt we were using the Puts row 
> rather than the Cells row; client may have succeeded in submitting a Cell 
> that disagreed with the hosting Mutation and it would have been written as 
> something else altogether -- with the Puts row -- rather than being rejected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20711) Save on a Cell iteration when writing

2018-06-09 Thread stack (JIRA)

stack created HBASE-20711:
-

 Summary: Save on a Cell iteration when writing
 Key: HBASE-20711
 URL: https://issues.apache.org/jira/browse/HBASE-20711
 Project: HBase
  Issue Type: Sub-task
  Components: Performance
Reporter: stack
Assignee: stack


This is a minor savings. We were doing a spin through all Cells on receipt just 
to check their size when subsequently, we were doing an iteration of all Cells 
to insert. It manifest as a little spike in perf output. This change removes 
the purposed spin through Cells and just does size check as part of the general 
Cell insert (perf spike no longer shows but the cost of the size check still 
remains).

There is also a minor bug fix where on receipt we were using the Puts row 
rather than the Cells row; client may have succeeded in submitting a Cell that 
disagreed with the hosting Mutation and it would have been written as something 
else altogether -- with the Puts row -- rather than being rejected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)

2018-06-09 Thread stack (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20710:
--
Fix Version/s: 2.0.1

> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.1
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent 
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB 
> backed ByteBuffKeyValueCell (which is default with Netty Rpc), 
> cell.getFamilyArray() will call cloneFamily() and there is again a 
> cloneFamily() in the following line of the code. since this is the critical 
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)

2018-06-09 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507030#comment-16507030
 ] 

stack edited comment on HBASE-20710 at 6/9/18 3:12 PM:
---

+1

I had this as a 'fix' in branch:

diff --git 
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java 
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
index a6ddc14cca..1b750b950e 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
@@ -788,11 +788,10 @@ public abstract class Mutation extends 
OperationWithAttributes implements Row, C
 " doesn't match the original one " +  Bytes.toStringBinary(this.row));
 }

-if (cell.getFamilyArray() == null || cell.getFamilyLength() == 0) {
+byte [] family = CellUtil.cloneFamily(cell);
+if (family == null || family.length == 0) {
   throw new IllegalArgumentException("Family cannot be null");
 }
-
-byte[] family = CellUtil.cloneFamily(cell);
 if (cell instanceof ExtendedCell) {
   getCellList(family).add(cell);
 } else {


The offence is the extra copy and the deserializations to find lengths to use 
locating the family bytes to copy.

Good one [~huaxiang]. Go for it.




was (Author: stack):
+1

I had this as a 'fix' in branch:

diff --git 
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java 
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
index a6ddc14cca..1b750b950e 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
@@ -788,11 +788,10 @@ public abstract class Mutation extends 
OperationWithAttributes implements Row, C
 " doesn't match the original one " +  Bytes.toStringBinary(this.row));
 }

-if (cell.getFamilyArray() == null || cell.getFamilyLength() == 0) {
+byte [] family = CellUtil.cloneFamily(cell);
+if (family == null || family.length == 0) {
   throw new IllegalArgumentException("Family cannot be null");
 }
-
-byte[] family = CellUtil.cloneFamily(cell);
 if (cell instanceof ExtendedCell) {
   getCellList(family).add(cell);
 } else {


The offence is the extra copy and the deserializations to find lengths to use 
locating the family bytes to copy.

Good one [~huaxiang].



> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.1
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent 
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB 
> backed ByteBuffKeyValueCell (which is default with Netty Rpc), 
> cell.getFamilyArray() will call cloneFamily() and there is again a 
> cloneFamily() in the following line of the code. since this is the critical 
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)

2018-06-09 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507030#comment-16507030
 ] 

stack commented on HBASE-20710:
---

+1

I had this as a 'fix' in branch:

diff --git 
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java 
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
index a6ddc14cca..1b750b950e 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
@@ -788,11 +788,10 @@ public abstract class Mutation extends 
OperationWithAttributes implements Row, C
 " doesn't match the original one " +  Bytes.toStringBinary(this.row));
 }

-if (cell.getFamilyArray() == null || cell.getFamilyLength() == 0) {
+byte [] family = CellUtil.cloneFamily(cell);
+if (family == null || family.length == 0) {
   throw new IllegalArgumentException("Family cannot be null");
 }
-
-byte[] family = CellUtil.cloneFamily(cell);
 if (cell instanceof ExtendedCell) {
   getCellList(family).add(cell);
 } else {


The offence is the extra copy and the deserializations to find lengths to use 
locating the family bytes to copy.

Good one [~huaxiang].



> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent 
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB 
> backed ByteBuffKeyValueCell (which is default with Netty Rpc), 
> cell.getFamilyArray() will call cloneFamily() and there is again a 
> cloneFamily() in the following line of the code. since this is the critical 
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)

2018-06-09 Thread stack (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-20710:
--
Issue Type: Sub-task  (was: Improvement)
Parent: HBASE-20188

> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent 
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB 
> backed ByteBuffKeyValueCell (which is default with Netty Rpc), 
> cell.getFamilyArray() will call cloneFamily() and there is again a 
> cloneFamily() in the following line of the code. since this is the critical 
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()

2018-06-09 Thread zhaoyuan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-20697 started by zhaoyuan.

> Can't cache All region locations of the specify table by calling 
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Minor
>
> When we upgrade and restart  a new version application which will read and 
> write to HBase, we will get some operation timeout. The time out is expected 
> because when the application restarts，It will not hold any region locations 
> cache and do communication with zk and meta regionserver to get region 
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am 
> concerned,the method table.getRegionLocator().getAllRegionLocations() will 
> fetch all region locations and cache them. However, it didn't work good. 
> There are still a lot of time outs,so it confused me. 
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
>   TableName tableName = getName();
>   NavigableMap locations =
>   MetaScanner.allTableRegions(this.connection, tableName);
>   ArrayList regions = new ArrayList<>(locations.size());
>   for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
>   }
>   if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
>   }
>   return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations 
> locations) {
>   byte [] startKey = 
> locations.getRegionLocation().getRegionInfo().getStartKey();
>   ConcurrentMap tableLocations = 
> getTableLocations(tableName);
>   RegionLocations oldLocation = tableLocations.putIfAbsent(startKey, 
> locations);
>   boolean isNewCacheEntry = (oldLocation == null);
>   if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
>   LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
>   }
> {code}
> It will collect all regions into one RegionLocations object and only cache 
> the first not null region location and then when we put or get to hbase, we 
> do getCacheLocation() 
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final 
> byte [] row) {
>   ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
>   Entry e = tableLocations.floorEntry(row);
>   if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
>   }
>   RegionLocations possibleRegion = e.getValue();
>   // make sure that the end key is greater than the row we're looking
>   // for, otherwise the row actually belongs in the next region, not
>   // this one. the exception case is when the endkey is
>   // HConstants.EMPTY_END_ROW, signifying that the region we're
>   // checking is actually the last region in the table.
>   byte[] endKey = 
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
>   if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
>   getRowComparator(tableName).compareRows(
>   endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
>   }
>   // Passed all the way through, so we got nothing - complete cache miss
>   if (metrics != null) metrics.incrMetaCacheMiss();
>   return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will 
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug 
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Guanghao Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507018#comment-16507018
 ] 

Guanghao Zhang commented on HBASE-20709:


A failed procedure may be better than a stuck procedure..

> CompatRemoteProcedureResolver should call remoteCallFailed method instead of 
> throw UnsupportedOperationException
> 
>
> Key: HBASE-20709
> URL: https://issues.apache.org/jira/browse/HBASE-20709
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 2.0.0
> Environment: # 
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-20709.master.001.patch
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> {code:java}
> @Override
> public void dispatchServerOperations(MasterProcedureEnv env, 
> List operations) {
>   throw new UnsupportedOperationException();
> }
> {code}
> As the procedure request not send to remote server, remoteOperationFailed and 
> remoteOperationCompleted can't be called. So the procedure will stuck and no 
> log to show this. The new patach will call remoteCallFailed method and make 
> the procedure failed. It is easy to find problem by the new exception message.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20188) [TESTING] Performance

2018-06-09 Thread huaxiang sun (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507016#comment-16507016
 ] 

huaxiang sun commented on HBASE-20188:
--

A minor issue found by cpu profiling during PE randomWrite testing.

> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
>  Issue Type: Umbrella
>  Components: Performance
>Reporter: stack
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0
>
> Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, 
> HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 
> performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs 
> None_ system settings.pdf, HBase 2.0 performance evaluation - throughput 
> SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, 
> ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, 
> ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, 
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, 
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, 
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, 
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, 
> hbase-site.xml, hits.png, hits_with_fp_scheduler.png, 
> lock.127.workloadc.20180402T200918Z.svg, 
> lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, 
> total.png, tree.txt, workloadx, workloadx
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor 
> that it is much slower, that the problem is the asyncwal writing. Does 
> in-memory compaction slow us down or speed us up? What happens when you 
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something 
> about perf when 2.0.0 ships.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)

2018-06-09 Thread huaxiang sun (JIRA)

huaxiang sun created HBASE-20710:


 Summary: extra cloneFamily() in Mutation.add(Cell)
 Key: HBASE-20710
 URL: https://issues.apache.org/jira/browse/HBASE-20710
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Affects Versions: 2.0.1
Reporter: huaxiang sun
Assignee: huaxiang sun


The cpu profiling shows that during PE randomWrite testing, about 1 percent of 
time is spent in cloneFamily. Reviewing code found that when a cell is DBB 
backed ByteBuffKeyValueCell (which is default with Netty Rpc), 
cell.getFamilyArray() will call cloneFamily() and there is again a 
cloneFamily() in the following line of the code. since this is the critical 
write path processing, this needs to be optimized.

https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791


https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Guanghao Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507011#comment-16507011
 ] 

Guanghao Zhang commented on HBASE-20709:


Yes. But if there are some other bugs like HBASE-20698 to make this happen, a 
failed procedure and detailed exception message will help us to find the 
problem.

> CompatRemoteProcedureResolver should call remoteCallFailed method instead of 
> throw UnsupportedOperationException
> 
>
> Key: HBASE-20709
> URL: https://issues.apache.org/jira/browse/HBASE-20709
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 2.0.0
> Environment: # 
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-20709.master.001.patch
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> {code:java}
> @Override
> public void dispatchServerOperations(MasterProcedureEnv env, 
> List operations) {
>   throw new UnsupportedOperationException();
> }
> {code}
> As the procedure request not send to remote server, remoteOperationFailed and 
> remoteOperationCompleted can't be called. So the procedure will stuck and no 
> log to show this. The new patach will call remoteCallFailed method and make 
> the procedure failed. It is easy to find problem by the new exception message.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Duo Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506988#comment-16506988
 ] 

Duo Zhang commented on HBASE-20709:
---

I think the design philosophy here is that these method should never be called 
as it is only used for open/close a region?

> CompatRemoteProcedureResolver should call remoteCallFailed method instead of 
> throw UnsupportedOperationException
> 
>
> Key: HBASE-20709
> URL: https://issues.apache.org/jira/browse/HBASE-20709
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 2.0.0
> Environment: # 
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-20709.master.001.patch
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> {code:java}
> @Override
> public void dispatchServerOperations(MasterProcedureEnv env, 
> List operations) {
>   throw new UnsupportedOperationException();
> }
> {code}
> As the procedure request not send to remote server, remoteOperationFailed and 
> remoteOperationCompleted can't be called. So the procedure will stuck and no 
> log to show this. The new patach will call remoteCallFailed method and make 
> the procedure failed. It is easy to find problem by the new exception message.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Guanghao Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506983#comment-16506983
 ] 

Guanghao Zhang commented on HBASE-20709:


Updated issue description.

> CompatRemoteProcedureResolver should call remoteCallFailed method instead of 
> throw UnsupportedOperationException
> 
>
> Key: HBASE-20709
> URL: https://issues.apache.org/jira/browse/HBASE-20709
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 2.0.0
> Environment: # 
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-20709.master.001.patch
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> {code:java}
> @Override
> public void dispatchServerOperations(MasterProcedureEnv env, 
> List operations) {
>   throw new UnsupportedOperationException();
> }
> {code}
> As the procedure request not send to remote server, remoteOperationFailed and 
> remoteOperationCompleted can't be called. So the procedure will stuck and no 
> log to show this. The new patach will call remoteCallFailed method and make 
> the procedure failed. It is easy to find problem by the new exception message.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Guanghao Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20709:
---
Environment: # 
Description: 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
{code:java}
@Override
public void dispatchServerOperations(MasterProcedureEnv env, 
List operations) {
  throw new UnsupportedOperationException();
}
{code}

As the procedure request not send to remote server, remoteOperationFailed and 
remoteOperationCompleted can't be called. So the procedure will stuck and no 
log to show this. The new patach will call remoteCallFailed method and make the 
procedure failed. It is easy to find problem by the new exception message.


  was:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
{code:java}
@Override
public void dispatchServerOperations(MasterProcedureEnv env, 
List operations) {
  throw new UnsupportedOperationException();
}
{code}



> CompatRemoteProcedureResolver should call remoteCallFailed method instead of 
> throw UnsupportedOperationException
> 
>
> Key: HBASE-20709
> URL: https://issues.apache.org/jira/browse/HBASE-20709
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 2.0.0
> Environment: # 
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-20709.master.001.patch
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> {code:java}
> @Override
> public void dispatchServerOperations(MasterProcedureEnv env, 
> List operations) {
>   throw new UnsupportedOperationException();
> }
> {code}
> As the procedure request not send to remote server, remoteOperationFailed and 
> remoteOperationCompleted can't be called. So the procedure will stuck and no 
> log to show this. The new patach will call remoteCallFailed method and make 
> the procedure failed. It is easy to find problem by the new exception message.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20699) QuotaCache should cancel the QuotaRefresherChore service inside its stop()

2018-06-09 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506978#comment-16506978
 ] 

Hudson commented on HBASE-20699:


Results for branch master
[build #360 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/360/]: (x) 
*{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/360//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/360//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/360//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> QuotaCache should cancel the QuotaRefresherChore service inside its stop()
> --
>
> Key: HBASE-20699
> URL: https://issues.apache.org/jira/browse/HBASE-20699
> Project: HBase
>  Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20699.master.001.patch, 
> HBASE-20699.master.002.patch
>
>
> *ANALYSIS*
>  * Called from {{HRegionServer.run()}} in case a rs is aborted:
> {code:java}
> // Stop the quota manager
> if (rsQuotaManager != null) {
>   rsQuotaManager.stop();
> }
> {code}
>  * Inside {{RegionServerRpcQuotaManager.stop()}}:
> {code:java}
>   public void stop() {
> if (isQuotaEnabled()) {
>   quotaCache.stop("shutdown");
> }
>   }
> {code}
>  * {{QuotaCache starts QuotaRefresherChore in}}{{ QuotaCache.start()}}:
> {code:java}
>   public void start() throws IOException {
> stopped = false;
> // TODO: This will be replaced once we have the notification bus ready.
> Configuration conf = rsServices.getConfiguration();
> int period = conf.getInt(REFRESH_CONF_KEY, REFRESH_DEFAULT_PERIOD);
> refreshChore = new QuotaRefresherChore(period, this);
> rsServices.getChoreService().scheduleChore(refreshChore);
>   }
> {code}
>  * {{QuotaCache does not cancel refreshChore inside }}{{QuotaCache.stop()}}:
> {code:java}
>   @Override
>   public void stop(final String why) {
> stopped = true;
>   }
> {code}
> *IMPACT:*
>  QuotaRefresherChore may cause some retrying operation to delay rs abort



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20672) Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at every monitoring interval

2018-06-09 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506979#comment-16506979
 ] 

Hudson commented on HBASE-20672:


Results for branch master
[build #360 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/360/]: (x) 
*{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/360//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/360//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/360//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at 
> every monitoring interval
> -
>
> Key: HBASE-20672
> URL: https://issues.apache.org/jira/browse/HBASE-20672
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Ankit Jain
>Assignee: Ankit Jain
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20672.branch-1.001.patch, 
> HBASE-20672.master.001.patch, HBASE-20672.master.002.patch, 
> HBASE-20672.master.003.patch
>
>
> Hbase currently provides counter read/write requests (ReadRequestCount, 
> WriteRequestCount). That said it is not easy to use counter that reset only 
> after a restart of the service, we would like to expose 2 new metrics in 
> HBase to provide ReadRequestRate and WriteRequestRate at region server level.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506980#comment-16506980
 ] 

Hudson commented on HBASE-20698:


Results for branch master
[build #360 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/360/]: (x) 
*{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/360//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/360//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/360//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch, 
> HBASE-20698.master.addendum.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Duo Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506976#comment-16506976
 ] 

Duo Zhang commented on HBASE-20709:
---

Could you please explain the reason?

> CompatRemoteProcedureResolver should call remoteCallFailed method instead of 
> throw UnsupportedOperationException
> 
>
> Key: HBASE-20709
> URL: https://issues.apache.org/jira/browse/HBASE-20709
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-20709.master.001.patch
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> {code:java}
> @Override
> public void dispatchServerOperations(MasterProcedureEnv env, 
> List operations) {
>   throw new UnsupportedOperationException();
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Guanghao Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20709:
---
Affects Version/s: 2.1.0
   3.0.0

> CompatRemoteProcedureResolver should call remoteCallFailed method instead of 
> throw UnsupportedOperationException
> 
>
> Key: HBASE-20709
> URL: https://issues.apache.org/jira/browse/HBASE-20709
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-20709.master.001.patch
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> {code:java}
> @Override
> public void dispatchServerOperations(MasterProcedureEnv env, 
> List operations) {
>   throw new UnsupportedOperationException();
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Guanghao Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20709:
---
Status: Patch Available  (was: Open)

> CompatRemoteProcedureResolver should call remoteCallFailed method instead of 
> throw UnsupportedOperationException
> 
>
> Key: HBASE-20709
> URL: https://issues.apache.org/jira/browse/HBASE-20709
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-20709.master.001.patch
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> {code:java}
> @Override
> public void dispatchServerOperations(MasterProcedureEnv env, 
> List operations) {
>   throw new UnsupportedOperationException();
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Guanghao Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20709:
---
Attachment: HBASE-20709.master.001.patch

> CompatRemoteProcedureResolver should call remoteCallFailed method instead of 
> throw UnsupportedOperationException
> 
>
> Key: HBASE-20709
> URL: https://issues.apache.org/jira/browse/HBASE-20709
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-20709.master.001.patch
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> {code:java}
> @Override
> public void dispatchServerOperations(MasterProcedureEnv env, 
> List operations) {
>   throw new UnsupportedOperationException();
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20709) CompatRemoteProcedureResolver should call remoteCallFailed method instead of throw UnsupportedOperationException

2018-06-09 Thread Guanghao Zhang (JIRA)

Guanghao Zhang created HBASE-20709:
--

 Summary: CompatRemoteProcedureResolver should call 
remoteCallFailed method instead of throw UnsupportedOperationException
 Key: HBASE-20709
 URL: https://issues.apache.org/jira/browse/HBASE-20709
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Guanghao Zhang
Assignee: Guanghao Zhang


hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
{code:java}
@Override
public void dispatchServerOperations(MasterProcedureEnv env, 
List operations) {
  throw new UnsupportedOperationException();
}
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Duo Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506963#comment-16506963
 ] 

Duo Zhang commented on HBASE-20698:
---

The addendum is OK for me. +1.

Thanks.

> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch, 
> HBASE-20698.master.addendum.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Guanghao Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506961#comment-16506961
 ] 

Guanghao Zhang commented on HBASE-20698:


Add a addendum patch. The plan is tell the upper layer that the version may be 
0 when server is not online. So the upper layer need to handle this.

> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch, 
> HBASE-20698.master.addendum.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Guanghao Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20698:
---
Status: Patch Available  (was: Reopened)

> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch, 
> HBASE-20698.master.addendum.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Guanghao Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20698:
---
Attachment: HBASE-20698.master.addendum.patch

> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch, 
> HBASE-20698.master.addendum.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20689) Docker fails to install rubocop for precommit

2018-06-09 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506956#comment-16506956
 ] 

Hadoop QA commented on HBASE-20689:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue}  0m  
7s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
45s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 12s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:36a7029 |
| JIRA Issue | HBASE-20689 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12927167/HBASE-20689.branch-1.001.patch
 |
| Optional Tests |  asflicense  shellcheck  shelldocs  |
| uname | Linux de900ea207e8 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 
12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | branch-1 / c596fb6 |
| maven | version: Apache Maven 3.0.5 |
| shellcheck | v0.4.7 |
| Max. process+thread count | 42 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/13172/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Docker fails to install rubocop for precommit
> -
>
> Key: HBASE-20689
> URL: https://issues.apache.org/jira/browse/HBASE-20689
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20689.branch-1.001.patch, 
> HBASE-20689.master.001.patch
>
>
> Precommit started to fail with rubocop install in 'Building base image' phase.
>  The used maven:3.5-jdk8 image was updated ~10h ago, probably related. 
> [https://hub.docker.com/r/library/maven/]
> One of the failed jobs: 
> [https://builds.apache.org/job/PreCommit-HBASE-Build/13105/console] 
> {noformat}
> 14:42:23 Setting up ruby (1:2.3.3) ...
> 14:42:23 Processing triggers for libc-bin (2.24-11+deb9u3) ...
> 14:42:23 Successfully installed rake-12.3.1
> 14:42:23 Parsing documentation for rake-12.3.1
> 14:42:23 Installing ri documentation for rake-12.3.1
> 14:42:23 Done installing documentation for rake after 1 seconds
> 14:42:23 Building native extensions.  This could take a while...
> 14:42:23 [91mERROR:  Error installing rubocop:
> 14:42:23  ERROR: Failed to build gem native extension.
> 14:42:23 
> 14:42:23 current directory: 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0/ext/jaro_winkler
> 14:42:23 /usr/bin/ruby2.3 -r ./siteconf20180606-949-99wdpz.rb extconf.rb
> 14:42:23 mkmf.rb can't find header files for ruby at 
> /usr/lib/ruby/include/ruby.h
> 14:42:23 
> 14:42:23 extconf failed, exit code 1
> 14:42:23 
> 14:42:23 Gem files will remain installed in 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0 for inspection.
> 14:42:23 Results logged to 
> /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/jaro_winkler-1.4.0/gem_make.out
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20708) Make sure there is no race between the RMP scheduled when start up and the SCP

2018-06-09 Thread Duo Zhang (JIRA)

Duo Zhang created HBASE-20708:
-

 Summary: Make sure there is no race between the RMP scheduled when 
start up and the SCP
 Key: HBASE-20708
 URL: https://issues.apache.org/jira/browse/HBASE-20708
 Project: HBase
  Issue Type: Sub-task
  Components: proc-v2, Region Assignment
Reporter: Duo Zhang
 Fix For: 3.0.0, 2.1.0, 2.0.1






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Duo Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506953#comment-16506953
 ] 

Duo Zhang commented on HBASE-20698:
---

So what's your plan?

> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20689) Docker fails to install rubocop for precommit

2018-06-09 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506948#comment-16506948
 ] 

Hadoop QA commented on HBASE-20689:
---

(!) A patch to the testing environment has been detected. 
Re-executing against the patched versions to perform further tests. 
The console is at 
https://builds.apache.org/job/PreCommit-HBASE-Build/13172/console in case of 
problems.


> Docker fails to install rubocop for precommit
> -
>
> Key: HBASE-20689
> URL: https://issues.apache.org/jira/browse/HBASE-20689
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20689.branch-1.001.patch, 
> HBASE-20689.master.001.patch
>
>
> Precommit started to fail with rubocop install in 'Building base image' phase.
>  The used maven:3.5-jdk8 image was updated ~10h ago, probably related. 
> [https://hub.docker.com/r/library/maven/]
> One of the failed jobs: 
> [https://builds.apache.org/job/PreCommit-HBASE-Build/13105/console] 
> {noformat}
> 14:42:23 Setting up ruby (1:2.3.3) ...
> 14:42:23 Processing triggers for libc-bin (2.24-11+deb9u3) ...
> 14:42:23 Successfully installed rake-12.3.1
> 14:42:23 Parsing documentation for rake-12.3.1
> 14:42:23 Installing ri documentation for rake-12.3.1
> 14:42:23 Done installing documentation for rake after 1 seconds
> 14:42:23 Building native extensions.  This could take a while...
> 14:42:23 [91mERROR:  Error installing rubocop:
> 14:42:23  ERROR: Failed to build gem native extension.
> 14:42:23 
> 14:42:23 current directory: 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0/ext/jaro_winkler
> 14:42:23 /usr/bin/ruby2.3 -r ./siteconf20180606-949-99wdpz.rb extconf.rb
> 14:42:23 mkmf.rb can't find header files for ruby at 
> /usr/lib/ruby/include/ruby.h
> 14:42:23 
> 14:42:23 extconf failed, exit code 1
> 14:42:23 
> 14:42:23 Gem files will remain installed in 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0 for inspection.
> 14:42:23 Results logged to 
> /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/jaro_winkler-1.4.0/gem_make.out
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20689) Docker fails to install rubocop for precommit

2018-06-09 Thread Peter Somogyi (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506945#comment-16506945
 ] 

Peter Somogyi commented on HBASE-20689:
---

Attached a patch for branch-1.

> Docker fails to install rubocop for precommit
> -
>
> Key: HBASE-20689
> URL: https://issues.apache.org/jira/browse/HBASE-20689
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20689.branch-1.001.patch, 
> HBASE-20689.master.001.patch
>
>
> Precommit started to fail with rubocop install in 'Building base image' phase.
>  The used maven:3.5-jdk8 image was updated ~10h ago, probably related. 
> [https://hub.docker.com/r/library/maven/]
> One of the failed jobs: 
> [https://builds.apache.org/job/PreCommit-HBASE-Build/13105/console] 
> {noformat}
> 14:42:23 Setting up ruby (1:2.3.3) ...
> 14:42:23 Processing triggers for libc-bin (2.24-11+deb9u3) ...
> 14:42:23 Successfully installed rake-12.3.1
> 14:42:23 Parsing documentation for rake-12.3.1
> 14:42:23 Installing ri documentation for rake-12.3.1
> 14:42:23 Done installing documentation for rake after 1 seconds
> 14:42:23 Building native extensions.  This could take a while...
> 14:42:23 [91mERROR:  Error installing rubocop:
> 14:42:23  ERROR: Failed to build gem native extension.
> 14:42:23 
> 14:42:23 current directory: 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0/ext/jaro_winkler
> 14:42:23 /usr/bin/ruby2.3 -r ./siteconf20180606-949-99wdpz.rb extconf.rb
> 14:42:23 mkmf.rb can't find header files for ruby at 
> /usr/lib/ruby/include/ruby.h
> 14:42:23 
> 14:42:23 extconf failed, exit code 1
> 14:42:23 
> 14:42:23 Gem files will remain installed in 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0 for inspection.
> 14:42:23 Results logged to 
> /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/jaro_winkler-1.4.0/gem_make.out
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20689) Docker fails to install rubocop for precommit

2018-06-09 Thread Peter Somogyi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi updated HBASE-20689:
--
Status: Patch Available  (was: Reopened)

> Docker fails to install rubocop for precommit
> -
>
> Key: HBASE-20689
> URL: https://issues.apache.org/jira/browse/HBASE-20689
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20689.branch-1.001.patch, 
> HBASE-20689.master.001.patch
>
>
> Precommit started to fail with rubocop install in 'Building base image' phase.
>  The used maven:3.5-jdk8 image was updated ~10h ago, probably related. 
> [https://hub.docker.com/r/library/maven/]
> One of the failed jobs: 
> [https://builds.apache.org/job/PreCommit-HBASE-Build/13105/console] 
> {noformat}
> 14:42:23 Setting up ruby (1:2.3.3) ...
> 14:42:23 Processing triggers for libc-bin (2.24-11+deb9u3) ...
> 14:42:23 Successfully installed rake-12.3.1
> 14:42:23 Parsing documentation for rake-12.3.1
> 14:42:23 Installing ri documentation for rake-12.3.1
> 14:42:23 Done installing documentation for rake after 1 seconds
> 14:42:23 Building native extensions.  This could take a while...
> 14:42:23 [91mERROR:  Error installing rubocop:
> 14:42:23  ERROR: Failed to build gem native extension.
> 14:42:23 
> 14:42:23 current directory: 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0/ext/jaro_winkler
> 14:42:23 /usr/bin/ruby2.3 -r ./siteconf20180606-949-99wdpz.rb extconf.rb
> 14:42:23 mkmf.rb can't find header files for ruby at 
> /usr/lib/ruby/include/ruby.h
> 14:42:23 
> 14:42:23 extconf failed, exit code 1
> 14:42:23 
> 14:42:23 Gem files will remain installed in 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0 for inspection.
> 14:42:23 Results logged to 
> /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/jaro_winkler-1.4.0/gem_make.out
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20689) Docker fails to install rubocop for precommit

2018-06-09 Thread Peter Somogyi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi updated HBASE-20689:
--
Attachment: HBASE-20689.branch-1.001.patch

> Docker fails to install rubocop for precommit
> -
>
> Key: HBASE-20689
> URL: https://issues.apache.org/jira/browse/HBASE-20689
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20689.branch-1.001.patch, 
> HBASE-20689.master.001.patch
>
>
> Precommit started to fail with rubocop install in 'Building base image' phase.
>  The used maven:3.5-jdk8 image was updated ~10h ago, probably related. 
> [https://hub.docker.com/r/library/maven/]
> One of the failed jobs: 
> [https://builds.apache.org/job/PreCommit-HBASE-Build/13105/console] 
> {noformat}
> 14:42:23 Setting up ruby (1:2.3.3) ...
> 14:42:23 Processing triggers for libc-bin (2.24-11+deb9u3) ...
> 14:42:23 Successfully installed rake-12.3.1
> 14:42:23 Parsing documentation for rake-12.3.1
> 14:42:23 Installing ri documentation for rake-12.3.1
> 14:42:23 Done installing documentation for rake after 1 seconds
> 14:42:23 Building native extensions.  This could take a while...
> 14:42:23 [91mERROR:  Error installing rubocop:
> 14:42:23  ERROR: Failed to build gem native extension.
> 14:42:23 
> 14:42:23 current directory: 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0/ext/jaro_winkler
> 14:42:23 /usr/bin/ruby2.3 -r ./siteconf20180606-949-99wdpz.rb extconf.rb
> 14:42:23 mkmf.rb can't find header files for ruby at 
> /usr/lib/ruby/include/ruby.h
> 14:42:23 
> 14:42:23 extconf failed, exit code 1
> 14:42:23 
> 14:42:23 Gem files will remain installed in 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0 for inspection.
> 14:42:23 Results logged to 
> /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/jaro_winkler-1.4.0/gem_make.out
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HBASE-20689) Docker fails to install rubocop for precommit

2018-06-09 Thread Peter Somogyi (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi reopened HBASE-20689:
---

On branch-1 [~mihir6692] noticed the same problem. Reopening.

> Docker fails to install rubocop for precommit
> -
>
> Key: HBASE-20689
> URL: https://issues.apache.org/jira/browse/HBASE-20689
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20689.master.001.patch
>
>
> Precommit started to fail with rubocop install in 'Building base image' phase.
>  The used maven:3.5-jdk8 image was updated ~10h ago, probably related. 
> [https://hub.docker.com/r/library/maven/]
> One of the failed jobs: 
> [https://builds.apache.org/job/PreCommit-HBASE-Build/13105/console] 
> {noformat}
> 14:42:23 Setting up ruby (1:2.3.3) ...
> 14:42:23 Processing triggers for libc-bin (2.24-11+deb9u3) ...
> 14:42:23 Successfully installed rake-12.3.1
> 14:42:23 Parsing documentation for rake-12.3.1
> 14:42:23 Installing ri documentation for rake-12.3.1
> 14:42:23 Done installing documentation for rake after 1 seconds
> 14:42:23 Building native extensions.  This could take a while...
> 14:42:23 [91mERROR:  Error installing rubocop:
> 14:42:23  ERROR: Failed to build gem native extension.
> 14:42:23 
> 14:42:23 current directory: 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0/ext/jaro_winkler
> 14:42:23 /usr/bin/ruby2.3 -r ./siteconf20180606-949-99wdpz.rb extconf.rb
> 14:42:23 mkmf.rb can't find header files for ruby at 
> /usr/lib/ruby/include/ruby.h
> 14:42:23 
> 14:42:23 extconf failed, exit code 1
> 14:42:23 
> 14:42:23 Gem files will remain installed in 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0 for inspection.
> 14:42:23 Results logged to 
> /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/jaro_winkler-1.4.0/gem_make.out
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20689) Docker fails to install rubocop for precommit

2018-06-09 Thread Monani Mihir (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506907#comment-16506907
 ] 

Monani Mihir commented on HBASE-20689:
--

I think it is failing for branch-1.3 build also. 
https://builds.apache.org/job/PreCommit-HBASE-Build/13160/console
{code:java}
15:01:11 Step 39/46 : RUN gem install rubocop
15:01:11  ---> Running in 7bb020b5e66d
15:01:12 Successfully installed unicode-display_width-1.4.0
15:01:12 Successfully installed ruby-progressbar-1.9.0
15:01:12 Successfully installed rainbow-3.0.0
15:01:12 Successfully installed powerpack-0.1.1
15:01:12 Successfully installed ast-2.4.0
15:01:12 Successfully installed parser-2.5.1.0
15:01:12 Successfully installed parallel-1.12.1
15:01:12 Building native extensions.  This could take a while...
15:01:13 [91mERROR:  Error installing rubocop:
15:01:13ERROR: Failed to build gem native extension.
15:01:13 
15:01:13 /usr/bin/ruby2.2 -r ./siteconf20180608-7-1r8czcg.rb extconf.rb
15:01:13 mkmf.rb can't find header files for ruby at 
/usr/lib/ruby/include/ruby.h
15:01:13 
15:01:13 extconf failed, exit code 1
15:01:13 
15:01:13 Gem files will remain installed in 
/var/lib/gems/2.2.0/gems/jaro_winkler-1.5.1 for inspection.
15:01:13 Results logged to 
/var/lib/gems/2.2.0/extensions/x86_64-linux/2.2.0/jaro_winkler-1.5.1/gem_make.out
15:01:13 [0mThe command '/bin/sh -c gem install rubocop' returned a non-zero 
code: 1
15:01:13 
15:01:13 Total Elapsed time:   0m  2s
15:01:13 
15:01:13 ERROR: Docker failed to build image.
{code}
 

> Docker fails to install rubocop for precommit
> -
>
> Key: HBASE-20689
> URL: https://issues.apache.org/jira/browse/HBASE-20689
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20689.master.001.patch
>
>
> Precommit started to fail with rubocop install in 'Building base image' phase.
>  The used maven:3.5-jdk8 image was updated ~10h ago, probably related. 
> [https://hub.docker.com/r/library/maven/]
> One of the failed jobs: 
> [https://builds.apache.org/job/PreCommit-HBASE-Build/13105/console] 
> {noformat}
> 14:42:23 Setting up ruby (1:2.3.3) ...
> 14:42:23 Processing triggers for libc-bin (2.24-11+deb9u3) ...
> 14:42:23 Successfully installed rake-12.3.1
> 14:42:23 Parsing documentation for rake-12.3.1
> 14:42:23 Installing ri documentation for rake-12.3.1
> 14:42:23 Done installing documentation for rake after 1 seconds
> 14:42:23 Building native extensions.  This could take a while...
> 14:42:23 [91mERROR:  Error installing rubocop:
> 14:42:23  ERROR: Failed to build gem native extension.
> 14:42:23 
> 14:42:23 current directory: 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0/ext/jaro_winkler
> 14:42:23 /usr/bin/ruby2.3 -r ./siteconf20180606-949-99wdpz.rb extconf.rb
> 14:42:23 mkmf.rb can't find header files for ruby at 
> /usr/lib/ruby/include/ruby.h
> 14:42:23 
> 14:42:23 extconf failed, exit code 1
> 14:42:23 
> 14:42:23 Gem files will remain installed in 
> /var/lib/gems/2.3.0/gems/jaro_winkler-1.4.0 for inspection.
> 14:42:23 Results logged to 
> /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/jaro_winkler-1.4.0/gem_make.out
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20706) [hack] Don't add known not-OPEN regions in reopen phase of MTP

2018-06-09 Thread Duo Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506890#comment-16506890
 ] 

Duo Zhang commented on HBASE-20706:
---

I guess this is not safe for reopening a region. If a region is in OPENING 
state, then it may have already loaded everything at the RS side, so if we do 
not reopen it then it will miss the new configurations which have just 
modified...

So I think we'd better introduce a new procedure called ReopenRegionProcedure, 
the different between this procedure and MRP is that, it never fails. And in 
the parent issue, we can use something like the openSeqNum to determine whether 
the region has been reopened, and if not, we will schedule a reopen procedure 
for it again.

> [hack] Don't add known not-OPEN regions in reopen phase of MTP
> --
>
> Key: HBASE-20706
> URL: https://issues.apache.org/jira/browse/HBASE-20706
> Project: HBase
>  Issue Type: Sub-task
>  Components: amv2
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20706.001.branch-2.0.patch
>
>
> Shake-down of ModifyTableProcedure, talked this one out with Stack – "proper" 
> fix is likely pending in HBASE-20682. Using MoveRegionProcedure is likely the 
> wrong construct, we would want something specific to reopen (e.g. a 
> ReopenProcedure).
> However, we're in a really bad state right now. If there are non-open regions 
> for a table which has a modify submitted against it, the entire system locks 
> up in a fast-spin while holding the table's lock. This fills up HDFS with PV2 
> wals, and prevents you from doing anything in the hbase shell to try to fix 
> those unassigned regions. You'll see spam in the master log like:
> {noformat}
> 2018-06-07 03:21:29,448 WARN  [PEWorker-1] procedure.ModifyTableProcedure: 
> Retriable error trying to modify table=METRIC_RECORD_HOURLY_UUID (in 
> state=MODIFY_TABLE_REOPEN_ALL_REGIONS)
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: 
> a3dc333606d38aeb6e2ab4b94233cfbc is not OPEN
>     at 
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:193)
>     at 
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:67)
>     at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:767)
>     at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createReopenProcedures(AssignmentManager.java:705)
>     at 
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:128)
>     at 
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:50)
>     at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
>     at 
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
>     at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1472)
>     at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240)
>     at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
>     at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
> {noformat}
> We unstuck out internal test cluster giving the following change on top of 
> Sergey's HBASE-20657. When choosing the regions to reopen, if we filter out a 
> table's regions to only be those which are currently OPEN. There may be some 
> transient failures here as well, but a subsequent retry of the reopen step 
> should filter out that change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20700) Move meta region when server crash can cause the procedure to be stuck

2018-06-09 Thread Duo Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506885#comment-16506885
 ] 

Duo Zhang commented on HBASE-20700:
---

Any other concerns? [~stack] Thanks.

> Move meta region when server crash can cause the procedure to be stuck
> --
>
> Key: HBASE-20700
> URL: https://issues.apache.org/jira/browse/HBASE-20700
> Project: HBase
>  Issue Type: Sub-task
>  Components: master, proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20700-UT.patch, HBASE-20700-v1.patch, 
> HBASE-20700-v2.patch, HBASE-20700.patch
>
>
> As said in HBASE-20682.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Guanghao Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang reopened HBASE-20698:


Reopen this as I found another problem... When a region server expired, it will 
be removed from onlineServers. Now getServerVersion may return 0 when the 
server is not in onlineServers. RSProcedureDispatcher is a ServerListener and 
there are race between ServerManager and RSProcedureDispatcher. For a 
RefreshPeerProcedure which target server expired, addOperationToNode may 
succeed but may get version 0 when remoteDispatch. Then this 
RefreshPeerProcedure will fail to dispatch...

> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20698) Master don't record right server version until new started region server call regionServerReport method

2018-06-09 Thread Guanghao Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-20698:
---
   Resolution: Fixed
Fix Version/s: 2.0.1
   Status: Resolved  (was: Patch Available)

Pushed to master, branch-2, and branch-2.0. Thanks [~stack] and [~Apache9] for 
reviewing.

> Master don't record right server version until new started region server call 
> regionServerReport method
> ---
>
> Key: HBASE-20698
> URL: https://issues.apache.org/jira/browse/HBASE-20698
> Project: HBase
>  Issue Type: Bug
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20698.master.001.patch, 
> HBASE-20698.master.002.patch, HBASE-20698.master.003.patch
>
>
> When a new region server started, it will call regionServerStartup first. 
> Master will record this server as a new online server and may dispath 
> RemoteProcedure to the new server. But master only record the server version 
> when the new region server call regionServerReport method. Dispatch a new 
> RemoteProcedure to this new regionserver will fail if version is not right.
> {code:java}
>   @Override
>   protected void remoteDispatch(final ServerName serverName,
>   final Set remoteProcedures) {
> final int rsVersion = 
> master.getAssignmentManager().getServerVersion(serverName);
> if (rsVersion >= RS_VERSION_WITH_EXEC_PROCS) {
>   LOG.trace("Using procedure batch rpc execution for serverName={} 
> version={}",
> serverName, rsVersion);
>   submitTask(new ExecuteProceduresRemoteCall(serverName, 
> remoteProcedures));
> } else {
>   LOG.info(String.format(
> "Fallback to compat rpc execution for serverName=%s version=%s",
> serverName, rsVersion));
>   submitTask(new CompatRemoteProcedureResolver(serverName, 
> remoteProcedures));
> }
>   }
> {code}
> The above code use version to resolve compatibility problem. So dispatch will 
> work right for old version region server. But for RefreshPeerProcedure, it is 
> new since hbase 2.0. So RefreshPeerProcedure don't need this. But the new 
> region server version is not right, it will use CompatRemoteProcedureResolver 
> for RefreshPeerProcedure, too. So the RefreshPeerProcedure can't be executed 
> rightly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20695) Implement table level RegionServer replication metrics

2018-06-09 Thread Guanghao Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506846#comment-16506846
 ] 

Guanghao Zhang commented on HBASE-20695:


{code:java}
source.getSourceMetrics().setAgeOfLastShippedOpByTable(entry.getKey().getTableName(),
 entry.getKey().getWriteTime());
{code}
I mean that only keep this in ReplicationSourceShipper and move other logic to 
MetricsSource. And use TableName as parameter.

> Implement table level RegionServer replication metrics 
> ---
>
> Key: HBASE-20695
> URL: https://issues.apache.org/jira/browse/HBASE-20695
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-20695.master.001.patch, 
> HBASE-20695.master.002.patch, HBASE-20695.master.003.patch
>
>
> Region server metrics now are mainly global metrics. It would be nice to have 
> table level metrics such as table level source.AgeOfLastShippedOp to indicate 
> operators which table's replication is lagging behind.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

68 matches

Mail list logo