[jira] [Commented] (HBASE-21045) Make a 2.0.2 release
[ https://issues.apache.org/jira/browse/HBASE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595955#comment-16595955 ] stack commented on HBASE-21045: --- I put up an RC1. Refreshed docs and CHANGES/RELEASENOTES. > Make a 2.0.2 release > > > Key: HBASE-21045 > URL: https://issues.apache.org/jira/browse/HBASE-21045 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Fix For: 2.0.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21123) Commit 2.0.2 RELEASENOTES and CHANGES
[
https://issues.apache.org/jira/browse/HBASE-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-21123.
---
Resolution: Fixed
Fix Version/s: 2.0.2
I pushed 2.0.2 release notes and changes a few days ago. Let me resolve.
> Commit 2.0.2 RELEASENOTES and CHANGES
> -
>
> Key: HBASE-21123
> URL: https://issues.apache.org/jira/browse/HBASE-21123
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.2
>
>
> Ran {{./release-doc-maker/releasedocmaker.py -p HBASE --fileversions -v 2.0.2
> -l --sortorder=newer --skip-credits}} then carefully stitched the product
> into the current CHANGES.md and RELEASENOTES.md files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-21123) Commit 2.0.2 RELEASENOTES and CHANGES
[
https://issues.apache.org/jira/browse/HBASE-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reopened HBASE-21123:
---
Now reopen so I can do an update because we're cutting new RC.
> Commit 2.0.2 RELEASENOTES and CHANGES
> -
>
> Key: HBASE-21123
> URL: https://issues.apache.org/jira/browse/HBASE-21123
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.2
>
>
> Ran {{./release-doc-maker/releasedocmaker.py -p HBASE --fileversions -v 2.0.2
> -l --sortorder=newer --skip-credits}} then carefully stitched the product
> into the current CHANGES.md and RELEASENOTES.md files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-21054) Copy down docs, amend to suite branch-2.0, and then commit
[ https://issues.apache.org/jira/browse/HBASE-21054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-21054. --- Resolution: Fixed Re-resolve after pushing a refresh. > Copy down docs, amend to suite branch-2.0, and then commit > -- > > Key: HBASE-21054 > URL: https://issues.apache.org/jira/browse/HBASE-21054 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: HBASE-21054.branch-2.0.001.patch > > > Freshen the doc in branch-2.0 by pulling from master branch. Purge backup, > spark, and serial replication mentions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-21054) Copy down docs, amend to suite branch-2.0, and then commit
[ https://issues.apache.org/jira/browse/HBASE-21054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reopened HBASE-21054: --- Reopening to do a refresh copy-down from master branch. > Copy down docs, amend to suite branch-2.0, and then commit > -- > > Key: HBASE-21054 > URL: https://issues.apache.org/jira/browse/HBASE-21054 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: HBASE-21054.branch-2.0.001.patch > > > Freshen the doc in branch-2.0 by pulling from master branch. Purge backup, > spark, and serial replication mentions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21075) Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after HBASE-20881
[ https://issues.apache.org/jira/browse/HBASE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595881#comment-16595881 ] Duo Zhang commented on HBASE-21075: --- Fine. I think we need more tests for it. Let's not block 2.0.2. > Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after > HBASE-20881 > - > > Key: HBASE-21075 > URL: https://issues.apache.org/jira/browse/HBASE-21075 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.1.1, 2.0.3 > > Attachments: HBASE-21075.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21075) Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after HBASE-20881
[ https://issues.apache.org/jira/browse/HBASE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595880#comment-16595880 ] stack commented on HBASE-21075: --- Ok. I can do that. I won't commit this patch to branch-2.0 just yet. Let me roll 2.0.2. Can add in the zk thingy for 2.0.3 and 2.1.1. > Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after > HBASE-20881 > - > > Key: HBASE-21075 > URL: https://issues.apache.org/jira/browse/HBASE-21075 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.1.1, 2.0.3 > > Attachments: HBASE-21075.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[ https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21083: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-2.0+. Thanks [~allan163] Lets add a client that can call this over in hbase-operator-tools. > Introduce a mechanism to bypass the execution of a stuck procedure > -- > > Key: HBASE-21083 > URL: https://issues.apache.org/jira/browse/HBASE-21083 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.1.1, 2.0.2 > > Attachments: HBASE-21083.branch-2.0.001.patch, > HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.0.003.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.1.001.patch > > > Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to > introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can > continue running. > we still have some unrevealed bugs hiding in our AMv2 and procedureV2 > system, we need something to interfere with stuck procedures before HBCK2 can > work. This is very crucial for a production ready system. > For now, we have little ways to interfere with running procedures. Aborting > them is not a good choice, since some procedures are not abort-able. And some > procedure may have overridden the abort() method, which will ignore the abort > request. > So, here, I will introduce a mechanism to bypass the execution of a stuck > procedure. > Basically, I added a field called 'bypass' to Procedure class. If we set this > field to true, all the logic in execute/rollback will be skipped, letting > this procedure and its ancestors complete normally and releasing the lock > resources at last. > Notice that bypassing a procedure may leave the cluster in a middle state, > e.g. the region not assigned, or some hdfs files left behind. > The Operators need know the side effect of bypassing and recover the > inconsistent state of the cluster themselves, like issuing new procedures to > assign the regions. > A patch will be uploaded and review board will be open. For now, only APIs in > ProcedureExecutor are provided. If anything is fine, I will add it to master > service and add a shell command to bypass a procedure. Or, maybe we can use > dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[ https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21083: -- Fix Version/s: (was: 2.0.3) 2.0.2 > Introduce a mechanism to bypass the execution of a stuck procedure > -- > > Key: HBASE-21083 > URL: https://issues.apache.org/jira/browse/HBASE-21083 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.1.1, 2.0.2 > > Attachments: HBASE-21083.branch-2.0.001.patch, > HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.0.003.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.1.001.patch > > > Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to > introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can > continue running. > we still have some unrevealed bugs hiding in our AMv2 and procedureV2 > system, we need something to interfere with stuck procedures before HBCK2 can > work. This is very crucial for a production ready system. > For now, we have little ways to interfere with running procedures. Aborting > them is not a good choice, since some procedures are not abort-able. And some > procedure may have overridden the abort() method, which will ignore the abort > request. > So, here, I will introduce a mechanism to bypass the execution of a stuck > procedure. > Basically, I added a field called 'bypass' to Procedure class. If we set this > field to true, all the logic in execute/rollback will be skipped, letting > this procedure and its ancestors complete normally and releasing the lock > resources at last. > Notice that bypassing a procedure may leave the cluster in a middle state, > e.g. the region not assigned, or some hdfs files left behind. > The Operators need know the side effect of bypassing and recover the > inconsistent state of the cluster themselves, like issuing new procedures to > assign the regions. > A patch will be uploaded and review board will be open. For now, only APIs in > ProcedureExecutor are provided. If anything is fine, I will add it to master > service and add a shell command to bypass a procedure. Or, maybe we can use > dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21126) Add ability for HBase Canary to ignore a configurable number of ZooKeeper down nodes
[
https://issues.apache.org/jira/browse/HBASE-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595833#comment-16595833
]
David Manning commented on HBASE-21126:
---
I'm working on a patch. I don't think I can assign to myself though.
> Add ability for HBase Canary to ignore a configurable number of ZooKeeper
> down nodes
>
>
> Key: HBASE-21126
> URL: https://issues.apache.org/jira/browse/HBASE-21126
> Project: HBase
> Issue Type: Improvement
> Components: canary, Zookeeper
>Affects Versions: 1.0.0, 3.0.0, 2.0.0
>Reporter: David Manning
>Priority: Minor
> Fix For: 3.0.0, 1.3.0, 2.0.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> When running org.apache.hadoop.hbase.tool.Canary with args -zookeeper
> -treatFailureAsError, the Canary will try to get a znode from each ZooKeeper
> server in the ensemble. If any server is unavailable or unresponsive, the
> canary will exit with a failure code.
> If we use the Canary to gauge server health, and alert accordingly, this can
> be too strict. For example, in a 5-node ZooKeeper cluster, having one node
> down is safe and expected in rolling upgrades/patches.
> This is a request to allow the Canary to take another parameter
> {code:java}
> -permittedZookeeperFailures {code}
> If N=1, in the 5-node ZooKeeper ensemble example, then the Canary will still
> pass if 4 ZooKeeper nodes are reachable, but fail if 3 or fewer are reachable.
> (This is my first Jira posting... sorry if I messed anything up.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21126) Add ability for HBase Canary to ignore a configurable number of ZooKeeper down nodes
David Manning created HBASE-21126:
-
Summary: Add ability for HBase Canary to ignore a configurable
number of ZooKeeper down nodes
Key: HBASE-21126
URL: https://issues.apache.org/jira/browse/HBASE-21126
Project: HBase
Issue Type: Improvement
Components: canary, Zookeeper
Affects Versions: 2.0.0, 1.0.0, 3.0.0
Reporter: David Manning
Fix For: 3.0.0, 2.0.0, 1.3.0
When running org.apache.hadoop.hbase.tool.Canary with args -zookeeper
-treatFailureAsError, the Canary will try to get a znode from each ZooKeeper
server in the ensemble. If any server is unavailable or unresponsive, the
canary will exit with a failure code.
If we use the Canary to gauge server health, and alert accordingly, this can be
too strict. For example, in a 5-node ZooKeeper cluster, having one node down is
safe and expected in rolling upgrades/patches.
This is a request to allow the Canary to take another parameter
{code:java}
-permittedZookeeperFailures {code}
If N=1, in the 5-node ZooKeeper ensemble example, then the Canary will still
pass if 4 ZooKeeper nodes are reachable, but fail if 3 or fewer are reachable.
(This is my first Jira posting... sorry if I messed anything up.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21075) Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after HBASE-20881
[ https://issues.apache.org/jira/browse/HBASE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595831#comment-16595831 ] Duo Zhang commented on HBASE-21075: --- Using zk is because that, I do not want to add a new method for master, which may introduce compatibility issues... > Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after > HBASE-20881 > - > > Key: HBASE-21075 > URL: https://issues.apache.org/jira/browse/HBASE-21075 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.1.1, 2.0.3 > > Attachments: HBASE-21075.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21075) Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after HBASE-20881
[ https://issues.apache.org/jira/browse/HBASE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595829#comment-16595829 ] Duo Zhang commented on HBASE-21075: --- OK, maybe a external tool, that write a special file on zk, and when master think it is safe to quit then it quits. The tool will also monitor the status of master, if the master quits, it will check the proc wals to see if there are outstanding unsupported procedures. If not, will output a message to say that it is safe to upgrade, otherwise you should restart from the beginning and try again. > Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after > HBASE-20881 > - > > Key: HBASE-21075 > URL: https://issues.apache.org/jira/browse/HBASE-21075 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.1.1, 2.0.3 > > Attachments: HBASE-21075.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595824#comment-16595824 ] Duo Zhang commented on HBASE-20942: --- +1 for 2.1. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18549) Unclaimed replication queues can go undetected
[ https://issues.apache.org/jira/browse/HBASE-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595793#comment-16595793 ] Xu Cang commented on HBASE-18549: - [~apurtell] > Unclaimed replication queues can go undetected > -- > > Key: HBASE-18549 > URL: https://issues.apache.org/jira/browse/HBASE-18549 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Ashu Pachauri >Assignee: Xu Cang >Priority: Critical > Fix For: 1.5.0, 1.3.3, 1.4.8 > > Attachments: HBASE-18549-.master.001.patch, > HBASE-18549-.master.002.patch, HBASE-18549-.master.003.patch > > > We have come across this situation multiple times where a zookeeper issues > can cause NodeFailoverWorker to fail picking up replication queue for a dead > region server silently. One example is when the znode size for a particular > queue exceed jute.maxBuffer value. > There can be other situations that may lead to this and just go undetected. > We need to have a metric for number of unclaimed replication queues. This > will help in mitigating the problem through alerting on the metric and > identifying underlying issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[
https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595790#comment-16595790
]
Hadoop QA commented on HBASE-21083:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
38s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
47s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
18s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 1s{color}
| {color:red} hbase-protocol-shaded generated 2 new + 98 unchanged - 2 fixed =
100 total (was 100) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
51s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 22s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
31s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
3s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}175m
8s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}254m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:6f01af0 |
| JIRA Issue | HBASE-21083 |
| JIRA Patch URL |
[jira] [Updated] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-15728: External issue URL: (was: https://reviews.apache.org/r/68551/diff/1#index_header) > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 1.5.0 > > Attachments: HBASE-15728.branch-1.001.patch, > HBASE-15728.branch-1.001.patch, HBASE-15728.branch-1.002.patch, > hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-15728: External issue URL: https://reviews.apache.org/r/68551/diff/1#index_header > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 1.5.0 > > Attachments: HBASE-15728.branch-1.001.patch, > HBASE-15728.branch-1.001.patch, HBASE-15728.branch-1.002.patch, > hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595782#comment-16595782 ] Xu Cang commented on HBASE-15728: - uploaded v2 patch for branch-1. Current master branch code is quite different regarding region server metrics. Will take more time to finish that. > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 1.5.0 > > Attachments: HBASE-15728.branch-1.001.patch, > HBASE-15728.branch-1.001.patch, HBASE-15728.branch-1.002.patch, > hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-15728: Attachment: HBASE-15728.branch-1.002.patch > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 1.5.0 > > Attachments: HBASE-15728.branch-1.001.patch, > HBASE-15728.branch-1.001.patch, HBASE-15728.branch-1.002.patch, > hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595779#comment-16595779
]
Hadoop QA commented on HBASE-20734:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
1s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
29s{color} | {color:red} hbase-server: The patch generated 17 new + 658
unchanged - 13 fixed = 675 total (was 671) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
44s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 40s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 48s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.coprocessor.TestMetaTableMetrics |
| | hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
| | hadoop.hbase.regionserver.TestRecoveredEdits |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-20734 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12937517/HBASE-20734.branch-1.003.patch
|
| Optional Tests |
[jira] [Comment Edited] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595769#comment-16595769 ] Daisuke Kobayashi edited comment on HBASE-6721 at 8/29/18 12:09 AM: bq. Open subtask or new issue? Ok, I will open a new subtask of this. bq. Have you tried it sir? Due to a build issue on my local instance, I have not yet completed it. Thanks [~stack]! was (Author: daisuke.kobayashi): bq. Open subtask or new issue? Ok, I will open a new subtask of this. bq. Have you tried it sir? Due to a build issue on my local instance, I have yet completed it. Thanks [~stack]! > RegionServer Group based Assignment > --- > > Key: HBASE-6721 > URL: https://issues.apache.org/jira/browse/HBASE-6721 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Labels: hbase-6721 > Fix For: 2.0.0 > > Attachments: 6721-master-webUI.patch, HBASE-6721 > GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, > HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, > HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch, > HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch, > HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, > HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, > HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, > HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, > HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch, > HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch, > HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, > HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg, > hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, hbase-6721-v17.patch, > hbase-6721-v18.patch, hbase-6721-v19.patch, hbase-6721-v20.patch, > hbase-6721-v21.patch, hbase-6721-v22.patch, hbase-6721-v23.patch, > hbase-6721-v25.patch, hbase-6721-v26.patch, hbase-6721-v26_draft1.patch, > hbase-6721-v27.patch, hbase-6721-v27.patch, hbase-6721-v27.patch.txt, > hbase-6721-v28.patch, hbase-6721-v28.patch, hbase-6721-v29.patch, > immediateAssignments Sequence Diagram.svg, randomAssignment Sequence > Diagram.svg, retainAssignment Sequence Diagram.svg, roundRobinAssignment > Sequence Diagram.svg > > > In multi-tenant deployments of HBase, it is likely that a RegionServer will > be serving out regions from a number of different tables owned by various > client applications. Being able to group a subset of running RegionServers > and assign specific tables to it, provides a client application a level of > isolation and resource allocation. > The proposal essentially is to have an AssignmentManager which is aware of > RegionServer groups and assigns tables to region servers based on groupings. > Load balancing will occur on a per group basis as well. > This is essentially a simplification of the approach taken in HBASE-4120. See > attached document. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595769#comment-16595769 ] Daisuke Kobayashi commented on HBASE-6721: -- bq. Open subtask or new issue? Ok, I will open a new subtask of this. bq. Have you tried it sir? Due to a build issue on my local instance, I have yet completed it. Thanks [~stack]! > RegionServer Group based Assignment > --- > > Key: HBASE-6721 > URL: https://issues.apache.org/jira/browse/HBASE-6721 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Major > Labels: hbase-6721 > Fix For: 2.0.0 > > Attachments: 6721-master-webUI.patch, HBASE-6721 > GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, > HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, > HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch, > HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch, > HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, > HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, > HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, > HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, > HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch, > HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch, > HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, > HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg, > hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, hbase-6721-v17.patch, > hbase-6721-v18.patch, hbase-6721-v19.patch, hbase-6721-v20.patch, > hbase-6721-v21.patch, hbase-6721-v22.patch, hbase-6721-v23.patch, > hbase-6721-v25.patch, hbase-6721-v26.patch, hbase-6721-v26_draft1.patch, > hbase-6721-v27.patch, hbase-6721-v27.patch, hbase-6721-v27.patch.txt, > hbase-6721-v28.patch, hbase-6721-v28.patch, hbase-6721-v29.patch, > immediateAssignments Sequence Diagram.svg, randomAssignment Sequence > Diagram.svg, retainAssignment Sequence Diagram.svg, roundRobinAssignment > Sequence Diagram.svg > > > In multi-tenant deployments of HBase, it is likely that a RegionServer will > be serving out regions from a number of different tables owned by various > client applications. Being able to group a subset of running RegionServers > and assign specific tables to it, provides a client application a level of > isolation and resource allocation. > The proposal essentially is to have an AssignmentManager which is aware of > RegionServer groups and assigns tables to region servers based on groupings. > Load balancing will occur on a per group basis as well. > This is essentially a simplification of the approach taken in HBASE-4120. See > attached document. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20649) Validate HFiles do not have PREFIX_TREE DataBlockEncoding
[
https://issues.apache.org/jira/browse/HBASE-20649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595746#comment-16595746
]
Hudson commented on HBASE-20649:
Results for branch branch-2.1
[build #256 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/256/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/256//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/256//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/256//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Validate HFiles do not have PREFIX_TREE DataBlockEncoding
> -
>
> Key: HBASE-20649
> URL: https://issues.apache.org/jira/browse/HBASE-20649
> Project: HBase
> Issue Type: New Feature
> Components: Operability, tooling
>Reporter: Peter Somogyi
>Assignee: Balazs Meszaros
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.2
>
> Attachments: HBASE-20649.master.001.patch,
> HBASE-20649.master.002.patch, HBASE-20649.master.003.patch,
> HBASE-20649.master.004.patch, HBASE-20649.master.005.patch,
> HBASE-20649.master.006.patch
>
>
> HBASE-20592 adds a tool to check column families on the cluster do not have
> PREFIX_TREE encoding.
> Since it is possible that DataBlockEncoding was already changed but HFiles
> are not rewritten yet we would need a tool that can verify the content of
> hfiles in the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21070) SnapshotFileCache won't update for snapshots stored in S3
[ https://issues.apache.org/jira/browse/HBASE-21070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595745#comment-16595745 ] Zach York commented on HBASE-21070: --- Good catch on readLock, missed removing that. The volatile keyword is still needed because even if these methods are synchronized, a different thread might have an old reference (which I believe is what happened in my testing). Let me think again on a UT... I couldn't find a good way to test this functionality, but maybe I can extend FileSystem to make a mock filesystem that can behave how I want it. Thanks for the review [~liuml07]! > SnapshotFileCache won't update for snapshots stored in S3 > - > > Key: HBASE-21070 > URL: https://issues.apache.org/jira/browse/HBASE-21070 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0, 2.1.1, 1.4.7 >Reporter: Zach York >Assignee: Zach York >Priority: Critical > Labels: FSRedo > Attachments: HBASE-21070.master.001.patch, > HBASE-21070.master.002.patch > > > The SnapshotFileCache depends on last modified time to determine whether to > update the Snapshot HFile cache. However, in S3, real 'folders' don't exist. > S3 filesystems create a dummy file in place of a folder, but the dummy file > last modified time is not updated when files are changed 'under' it. This > means that the SnapshotFileCache doesn't pick up new snapshot HFiles and > these files aren't removed from the HFileCleaner and can be eligible for > deletion. > > My patch removes the lastmodified assumption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20940) HStore.cansplit should not allow split to happen if it has references
[
https://issues.apache.org/jira/browse/HBASE-20940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595739#comment-16595739
]
Hudson commented on HBASE-20940:
Results for branch branch-1.3
[build #446 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/446/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/446//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/446//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/446//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HStore.cansplit should not allow split to happen if it has references
> -
>
> Key: HBASE-20940
> URL: https://issues.apache.org/jira/browse/HBASE-20940
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 1.4.7, 2.0.3
>
> Attachments: HBASE-20940-branch-1-addendum.patch,
> HBASE-20940.branch-1.3.v1.patch, HBASE-20940.branch-1.3.v2.patch,
> HBASE-20940.branch-1.v1.patch, HBASE-20940.branch-1.v2.patch,
> HBASE-20940.branch-1.v3.patch, HBASE-20940.branch-1.v5.patch,
> HBASE-20940.v1.patch, HBASE-20940.v2.patch, HBASE-20940.v3.patch,
> HBASE-20940.v4.patch, result_HBASE-20940.branch-1.v2.log
>
>
> When split happens and immediately another split happens, it may result into
> a split of a region who still has references to its parent. More details
> about scenario can be found here HBASE-20933
> HStore.hasReferences should check from fs.storefile rather than in memory
> objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[
https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595738#comment-16595738
]
Hudson commented on HBASE-20942:
Results for branch branch-1.3
[build #446 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/446/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/446//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/446//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/446//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Improve RpcServer TRACE logging
> ---
>
> Key: HBASE-20942
> URL: https://issues.apache.org/jira/browse/HBASE-20942
> Project: HBase
> Issue Type: Task
> Components: Operability
>Affects Versions: 2.1.0, 1.4.6
>Reporter: Esteban Gutierrez
>Assignee: Krish Dey
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7
>
> Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch,
> HBASE-20942.004.patch, HBASE-20942.005.patch
>
>
> Two things:
> * We truncate RpcServer output to 1000 characters for trace logging. Would
> be better if that value was configurable.
> * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE
> log message.
> Esteban mentioned this to me earlier, so I'm crediting him as the reporter.
> cc: [~elserj]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21098) Improve Snapshot Performance with Temporary Snapshot Directory when rootDir on S3
[
https://issues.apache.org/jira/browse/HBASE-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595734#comment-16595734
]
Mingliang Liu commented on HBASE-21098:
---
Making temporary directory for snapshort configurable (to use HDFS) makes sense
to the use case of S3 mode. The rename in patch from {{fs}} to {{rootFs}} is
relevant and good.
# The unit test problem in previous comment is also in
{{TestSnapshotDFSTemporaryDirectory}}.
# {{SnapshotDescriptionUtils::isWithinDefaultWorkingDir()}} is flaky as it
should check is subdirectory of {{getDefaultWorkingSnapshotDir()}} instead of
{{rootdir}}.
{code}
270 /**
271* Determines if the given workingDir is a subdirectory of the
default working snapshot directory
272* @param workingDir a directory to check
273* @param conf configuration for the HBase cluster
274* @return true if the given workingDir is a subdirectory of the
default working directory for
275* snapshots, false otherwise
276*/
277 public static boolean isWithinDefaultWorkingDir(final Path
workingDir, Configuration conf) {
278 return isSubDirectoryOf(workingDir, new
Path(conf.get(HConstants.HBASE_DIR)));
279 }
{code}
# The {{FileSystem::exist()}} before {{FileSystem::delete()}} is useless and
redundant per [HADOOP-13427]. I think this is the same case here. We can remove
it from {{SnapshortManager}}.
{code}
if (tmpFs.exists(tmpdir)) {
if (!tmpFs.delete(tmpdir, true)) {
{code}
# Document the newly added configuration "hbase.snapshot.working.dir" to
{{hbase-default.xml|}}
# Is it possible to add this in {{branch-1}}?
# Nit: it's better to put path definition and assertion together for path1 ~
path 10 in {{TestSnapshotDescriptionUtils::testIsSubDirectoryWorks}}
> Improve Snapshot Performance with Temporary Snapshot Directory when rootDir
> on S3
> -
>
> Key: HBASE-21098
> URL: https://issues.apache.org/jira/browse/HBASE-21098
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.1.1
>Reporter: Tyler Mi
>Priority: Major
> Attachments: HBASE-21098.master.001.patch,
> HBASE-21098.master.002.patch, HBASE-21098.master.003.patch,
> HBASE-21098.master.004.patch, HBASE-21098.master.005.patch,
> HBASE-21098.master.006.patch
>
>
> When using Apache HBase, the snapshot feature can be used to make a point in
> time recovery. To do this, HBase creates a manifest of all the files in all
> of the Regions so that those files can be referenced again when a user
> restores a snapshot. With HBase's S3 storage mode, developers can store their
> data off-cluster on Amazon S3. However, utilizing S3 as a file system is
> inefficient in some operations, namely renames. Most Hadoop ecosystem
> applications use an atomic rename as a method of committing data. However,
> with S3, a rename is a separate copy and then a delete of every file which is
> no longer atomic and, in fact, quite costly. In addition, puts and deletes on
> S3 have latency issues that traditional filesystems do not encounter when
> manipulating the region snapshots to consolidate into a single manifest. When
> HBase on S3 users have a significant amount of regions, puts, deletes, and
> renames (the final commit stage of the snapshot) become the bottleneck
> causing snapshots to take many minutes or even hours to complete.
> The purpose of this patch is to increase the overall performance of snapshots
> while utilizing HBase on S3 through the use of a temporary directory for the
> snapshots that exists on a traditional filesystem like HDFS to circumvent the
> bottlenecks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[
https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595726#comment-16595726
]
Hadoop QA commented on HBASE-15728:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 6m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
44s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
42s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hbase-hadoop-compat passed checkstyle
{color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hbase-hadoop2-compat: The patch generated 3 new + 16
unchanged - 1 fixed = 19 total (was 17) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} hbase-server: The patch generated 0 new + 457
unchanged - 21 fixed = 457 total (was 478) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 37s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} |
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[
https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595715#comment-16595715
]
Hudson commented on HBASE-20942:
Results for branch branch-2
[build #1178 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1178/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1178//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1178//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1178//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve RpcServer TRACE logging
> ---
>
> Key: HBASE-20942
> URL: https://issues.apache.org/jira/browse/HBASE-20942
> Project: HBase
> Issue Type: Task
> Components: Operability
>Affects Versions: 2.1.0, 1.4.6
>Reporter: Esteban Gutierrez
>Assignee: Krish Dey
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7
>
> Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch,
> HBASE-20942.004.patch, HBASE-20942.005.patch
>
>
> Two things:
> * We truncate RpcServer output to 1000 characters for trace logging. Would
> be better if that value was configurable.
> * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE
> log message.
> Esteban mentioned this to me earlier, so I'm crediting him as the reporter.
> cc: [~elserj]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-14783) Proc-V2: Master aborts when downgrading from 1.3 to 1.1
[
https://issues.apache.org/jira/browse/HBASE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-14783.
Resolution: Later
> Proc-V2: Master aborts when downgrading from 1.3 to 1.1
> ---
>
> Key: HBASE-14783
> URL: https://issues.apache.org/jira/browse/HBASE-14783
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
>Priority: Major
>
> I was running ITBLL with 1.3 deployed on a 6 node cluster.
> Then I stopped the cluster, deployed 1.1 release and tried to start cluster.
> However, master failed to start due to:
> {code}
> 2015-11-06 00:58:40,351 FATAL [eval-test-2:2.activeMasterManager]
> master.HMaster: Failed to become active master
> java.io.IOException: The procedure class
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be
> accessible and have an empty constructor
> at
> org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:548)
> at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:640)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:105)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:82)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:298)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:275)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:434)
> at
> org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1208)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1107)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:694)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:186)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1713)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:191)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:536)
> ... 12 more
> {code}
> The cause was that ServerCrashProcedure, written in some WAL file under
> MasterProcWALs from first run, was absent in 1.1 release.
> After a brief discussion with Stephen, I am logging this JIRA to solicit
> discussion on how customer experience can be improved if downgrade of hbase
> is performed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-14716) Detection of orphaned table znode should cover table in Enabled state
[
https://issues.apache.org/jira/browse/HBASE-14716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-14716.
Resolution: Later
> Detection of orphaned table znode should cover table in Enabled state
> -
>
> Key: HBASE-14716
> URL: https://issues.apache.org/jira/browse/HBASE-14716
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Labels: hbck
> Attachments: 14716-branch-1-v1.txt, 14716.branch-1.v4.txt
>
>
> HBASE-12070 introduced fix for orphaned table znode where table doesn't have
> entry in hbase:meta
> When Stephen and I investigated rolling upgrade failure,
> {code}
> 2015-10-27 18:21:10,668 WARN [ProcedureExecutorThread-3]
> procedure.CreateTableProcedure: The table smoketest does not exist in meta
> but has a znode. run hbck to fix inconsistencies.
> {code}
> we found that the orphaned table znode corresponded to table in Enabled state.
> Therefore running hbck didn't report the inconsistency.
> Detection for orphaned table znode should cover this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[
https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595674#comment-16595674
]
Hudson commented on HBASE-20942:
Results for branch branch-2.0
[build #745 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/745/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/745//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/745//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/745//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Improve RpcServer TRACE logging
> ---
>
> Key: HBASE-20942
> URL: https://issues.apache.org/jira/browse/HBASE-20942
> Project: HBase
> Issue Type: Task
> Components: Operability
>Affects Versions: 2.1.0, 1.4.6
>Reporter: Esteban Gutierrez
>Assignee: Krish Dey
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7
>
> Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch,
> HBASE-20942.004.patch, HBASE-20942.005.patch
>
>
> Two things:
> * We truncate RpcServer output to 1000 characters for trace logging. Would
> be better if that value was configurable.
> * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE
> log message.
> Esteban mentioned this to me earlier, so I'm crediting him as the reporter.
> cc: [~elserj]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20649) Validate HFiles do not have PREFIX_TREE DataBlockEncoding
[
https://issues.apache.org/jira/browse/HBASE-20649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595673#comment-16595673
]
Hudson commented on HBASE-20649:
Results for branch branch-2.0
[build #745 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/745/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/745//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/745//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/745//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Validate HFiles do not have PREFIX_TREE DataBlockEncoding
> -
>
> Key: HBASE-20649
> URL: https://issues.apache.org/jira/browse/HBASE-20649
> Project: HBase
> Issue Type: New Feature
> Components: Operability, tooling
>Reporter: Peter Somogyi
>Assignee: Balazs Meszaros
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.2
>
> Attachments: HBASE-20649.master.001.patch,
> HBASE-20649.master.002.patch, HBASE-20649.master.003.patch,
> HBASE-20649.master.004.patch, HBASE-20649.master.005.patch,
> HBASE-20649.master.006.patch
>
>
> HBASE-20592 adds a tool to check column families on the cluster do not have
> PREFIX_TREE encoding.
> Since it is possible that DataBlockEncoding was already changed but HFiles
> are not rewritten yet we would need a tool that can verify the content of
> hfiles in the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[ https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21083: -- Attachment: HBASE-21083.branch-2.0.003.patch > Introduce a mechanism to bypass the execution of a stuck procedure > -- > > Key: HBASE-21083 > URL: https://issues.apache.org/jira/browse/HBASE-21083 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21083.branch-2.0.001.patch, > HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.0.003.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.1.001.patch > > > Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to > introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can > continue running. > we still have some unrevealed bugs hiding in our AMv2 and procedureV2 > system, we need something to interfere with stuck procedures before HBCK2 can > work. This is very crucial for a production ready system. > For now, we have little ways to interfere with running procedures. Aborting > them is not a good choice, since some procedures are not abort-able. And some > procedure may have overridden the abort() method, which will ignore the abort > request. > So, here, I will introduce a mechanism to bypass the execution of a stuck > procedure. > Basically, I added a field called 'bypass' to Procedure class. If we set this > field to true, all the logic in execute/rollback will be skipped, letting > this procedure and its ancestors complete normally and releasing the lock > resources at last. > Notice that bypassing a procedure may leave the cluster in a middle state, > e.g. the region not assigned, or some hdfs files left behind. > The Operators need know the side effect of bypassing and recover the > inconsistent state of the cluster themselves, like issuing new procedures to > assign the regions. > A patch will be uploaded and review board will be open. For now, only APIs in > ProcedureExecutor are provided. If anything is fine, I will add it to master > service and add a shell command to bypass a procedure. Or, maybe we can use > dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20734: -- Attachment: HBASE-20734.master.009.patch > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Assignee: Zach York >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20734.branch-1.001.patch, > HBASE-20734.branch-1.002.patch, HBASE-20734.branch-1.003.patch, > HBASE-20734.master.001.patch, HBASE-20734.master.002.patch, > HBASE-20734.master.003.patch, HBASE-20734.master.004.patch, > HBASE-20734.master.005.patch, HBASE-20734.master.006.patch, > HBASE-20734.master.007.patch, HBASE-20734.master.008.patch, > HBASE-20734.master.009.patch > > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21098) Improve Snapshot Performance with Temporary Snapshot Directory when rootDir on S3
[
https://issues.apache.org/jira/browse/HBASE-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595658#comment-16595658
]
Mingliang Liu commented on HBASE-21098:
---
I did not finish, but my review is not binding, so feel free to commit if it
get binding +1s. CC [~yuzhih...@gmail.com] and [~zyork].
One comment is that,
{{TestExportSnapshotWithTemporaryDirectory::setUpBaseConf()}} is to override
the method in base class {{TestExportSnapshot}}. However this method is
{{static}}, which means the overriding will not take effect. So it's basically
not used. We will have to set up the "{{@BeforeClass public static void
setUpBeforeClass() throws Exception...}}" in
{{TestExportSnapshotWithTemporaryDirectory}}. After that, you may see unit test
failures.
> Improve Snapshot Performance with Temporary Snapshot Directory when rootDir
> on S3
> -
>
> Key: HBASE-21098
> URL: https://issues.apache.org/jira/browse/HBASE-21098
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.1.1
>Reporter: Tyler Mi
>Priority: Major
> Attachments: HBASE-21098.master.001.patch,
> HBASE-21098.master.002.patch, HBASE-21098.master.003.patch,
> HBASE-21098.master.004.patch, HBASE-21098.master.005.patch,
> HBASE-21098.master.006.patch
>
>
> When using Apache HBase, the snapshot feature can be used to make a point in
> time recovery. To do this, HBase creates a manifest of all the files in all
> of the Regions so that those files can be referenced again when a user
> restores a snapshot. With HBase's S3 storage mode, developers can store their
> data off-cluster on Amazon S3. However, utilizing S3 as a file system is
> inefficient in some operations, namely renames. Most Hadoop ecosystem
> applications use an atomic rename as a method of committing data. However,
> with S3, a rename is a separate copy and then a delete of every file which is
> no longer atomic and, in fact, quite costly. In addition, puts and deletes on
> S3 have latency issues that traditional filesystems do not encounter when
> manipulating the region snapshots to consolidate into a single manifest. When
> HBase on S3 users have a significant amount of regions, puts, deletes, and
> renames (the final commit stage of the snapshot) become the bottleneck
> causing snapshots to take many minutes or even hours to complete.
> The purpose of this patch is to increase the overall performance of snapshots
> while utilizing HBase on S3 through the use of a temporary directory for the
> snapshots that exists on a traditional filesystem like HDFS to circumvent the
> bottlenecks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-15728: Attachment: HBASE-15728.branch-1.001.patch Status: Patch Available (was: Open) > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 1.5.0 > > Attachments: HBASE-15728.branch-1.001.patch, > HBASE-15728.branch-1.001.patch, hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-15728: Attachment: HBASE-15728.branch-1.001.patch > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 1.5.0 > > Attachments: HBASE-15728.branch-1.001.patch, hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595644#comment-16595644 ] Zach York commented on HBASE-20734: --- Also FYI [~apurtell], the branch-1 patch contains the backwards compatibility logic. > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Assignee: Zach York >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20734.branch-1.001.patch, > HBASE-20734.branch-1.002.patch, HBASE-20734.branch-1.003.patch, > HBASE-20734.master.001.patch, HBASE-20734.master.002.patch, > HBASE-20734.master.003.patch, HBASE-20734.master.004.patch, > HBASE-20734.master.005.patch, HBASE-20734.master.006.patch, > HBASE-20734.master.007.patch, HBASE-20734.master.008.patch > > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20734: -- Attachment: HBASE-20734.branch-1.003.patch > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Assignee: Zach York >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20734.branch-1.001.patch, > HBASE-20734.branch-1.002.patch, HBASE-20734.branch-1.003.patch, > HBASE-20734.master.001.patch, HBASE-20734.master.002.patch, > HBASE-20734.master.003.patch, HBASE-20734.master.004.patch, > HBASE-20734.master.005.patch, HBASE-20734.master.006.patch, > HBASE-20734.master.007.patch, HBASE-20734.master.008.patch > > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595641#comment-16595641
]
Hadoop QA commented on HBASE-20734:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
7s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
8s{color} | {color:red} hbase-server: The patch generated 10 new + 387
unchanged - 5 fixed = 397 total (was 392) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
4s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 48s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
32s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}217m 46s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}260m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestMasterReplication |
| | hadoop.hbase.client.TestFromClientSide |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20734 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12937479/HBASE-20734.master.007.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1db2842c84b7 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20734: -- Attachment: HBASE-20734.master.008.patch > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Assignee: Zach York >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20734.branch-1.001.patch, > HBASE-20734.branch-1.002.patch, HBASE-20734.master.001.patch, > HBASE-20734.master.002.patch, HBASE-20734.master.003.patch, > HBASE-20734.master.004.patch, HBASE-20734.master.005.patch, > HBASE-20734.master.006.patch, HBASE-20734.master.007.patch, > HBASE-20734.master.008.patch > > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595631#comment-16595631 ] Zach York commented on HBASE-20734: --- [~yuzhih...@gmail.com] Thanks for the review. I'll change those things, but the previous diff rb is up. > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Assignee: Zach York >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20734.branch-1.001.patch, > HBASE-20734.branch-1.002.patch, HBASE-20734.master.001.patch, > HBASE-20734.master.002.patch, HBASE-20734.master.003.patch, > HBASE-20734.master.004.patch, HBASE-20734.master.005.patch, > HBASE-20734.master.006.patch, HBASE-20734.master.007.patch > > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21107) add a metrics for netty direct memory
[
https://issues.apache.org/jira/browse/HBASE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595608#comment-16595608
]
Hadoop QA commented on HBASE-21107:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
11s{color} | {color:red} hbase-server: The patch generated 2 new + 2 unchanged
- 0 fixed = 4 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
27s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 45s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}118m
14s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21107 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12937489/HBASE-21107-master-v1.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9f0a170d7ab4 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595602#comment-16595602 ] Andrew Purtell commented on HBASE-15728: Cool, thanks [~xucang] We will need a patch for branch-1 and one for trunk, that should do it. > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 1.5.0 > > Attachments: hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[
https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595591#comment-16595591
]
Xu Cang edited comment on HBASE-15728 at 8/28/18 8:57 PM:
--
Ported this patch to current branch-1 (It's been 2 years since previous patch.
I am picking it up now.)
Result from JMX:
{{ }}{{{ "name" : "Hadoop:service=HBase,name=RegionServer,sub=Tables",
"modelerType" : "RegionServer,sub=Tables", "tag.Context" : "regionserver",
"tag.Hostname" : "xcang-wsl",
"Namespace_hbase_table_meta_metric_readRequestCount" : 1,
"Namespace_hbase_table_meta_metric_filteredReadRequestCount" : 0,
"Namespace_hbase_table_meta_metric_writeRequestCount" : 2,
"Namespace_hbase_table_meta_metric_totalRequestCount" : 3,
"Namespace_hbase_table_meta_metric_memStoreSize" : 1272,
"Namespace_hbase_table_meta_metric_storeFileCount" : 0,
"Namespace_hbase_table_meta_metric_storeFileSize" : 0,
"Namespace_hbase_table_meta_metric_tableSize" : 1272,
"Namespace_hbase_table_meta_metric_averageRegionSize" : 1272,
"Namespace_hbase_table_meta_metric_regionCount" : 1,
"Namespace_hbase_table_meta_metric_storeCount" : 1,
"Namespace_hbase_table_meta_metric_maxStoreFileAge" : 0,
"Namespace_hbase_table_meta_metric_minStoreFileAge" : 0,
"Namespace_hbase_table_meta_metric_avgStoreFileAge" : 0,
"Namespace_hbase_table_meta_metric_numReferenceFiles" : 0,
"Namespace_hbase_table_namespace_metric_readRequestCount" : 2,
"Namespace_hbase_table_namespace_metric_filteredReadRequestCount" : 0,
"Namespace_hbase_table_namespace_metric_writeRequestCount" : 2,
"Namespace_hbase_table_namespace_metric_totalRequestCount" : 4,
"Namespace_hbase_table_namespace_metric_memStoreSize" : 520,
"Namespace_hbase_table_namespace_metric_storeFileCount" : 0,
"Namespace_hbase_table_namespace_metric_storeFileSize" : 0,
"Namespace_hbase_table_namespace_metric_tableSize" : 520,
"Namespace_hbase_table_namespace_metric_averageRegionSize" : 520,
"Namespace_hbase_table_namespace_metric_regionCount" : 1,
"Namespace_hbase_table_namespace_metric_storeCount" : 1,
"Namespace_hbase_table_namespace_metric_maxStoreFileAge" : 0,
"Namespace_hbase_table_namespace_metric_minStoreFileAge" : 0,
"Namespace_hbase_table_namespace_metric_avgStoreFileAge" : 0,
"Namespace_hbase_table_namespace_metric_numReferenceFiles" : 0, "numTables" :
2, "Namespace_hbase_table_meta_metric_splitRequestCount" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_num_ops" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_min" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_max" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_mean" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_25th_percentile"
: 0, "Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_median"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_75th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_90th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_95th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_98th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_99th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_99.9th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_num_ops" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_min" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_max" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_mean" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_25th_percentile"
: 0, "Namespace_hbase_table_namespace_metric_compactionOutputFileCount_median"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_75th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_90th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_95th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_98th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_99th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_99.9th_percentile"
: 0, "Namespace_hbase_table_namespace_metric_flushTime_num_ops" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_min" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_max" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_mean" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_25th_percentile" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_median" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_75th_percentile" : 0,
[jira] [Comment Edited] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[
https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595591#comment-16595591
]
Xu Cang edited comment on HBASE-15728 at 8/28/18 8:58 PM:
--
Ported this patch to current branch-1 (It's been 2 years since previous patch.
I am picking it up now.)
Result from JMX:
{code:java}
{ "name" : "Hadoop:service=HBase,name=RegionServer,sub=Tables", "modelerType" :
"RegionServer,sub=Tables", "tag.Context" : "regionserver", "tag.Hostname" :
"xcang-wsl", "Namespace_hbase_table_meta_metric_readRequestCount" : 1,
"Namespace_hbase_table_meta_metric_filteredReadRequestCount" : 0,
"Namespace_hbase_table_meta_metric_writeRequestCount" : 2,
"Namespace_hbase_table_meta_metric_totalRequestCount" : 3,
"Namespace_hbase_table_meta_metric_memStoreSize" : 1272,
"Namespace_hbase_table_meta_metric_storeFileCount" : 0,
"Namespace_hbase_table_meta_metric_storeFileSize" : 0,
"Namespace_hbase_table_meta_metric_tableSize" : 1272,
"Namespace_hbase_table_meta_metric_averageRegionSize" : 1272,
"Namespace_hbase_table_meta_metric_regionCount" : 1,
"Namespace_hbase_table_meta_metric_storeCount" : 1,
"Namespace_hbase_table_meta_metric_maxStoreFileAge" : 0,
"Namespace_hbase_table_meta_metric_minStoreFileAge" : 0,
"Namespace_hbase_table_meta_metric_avgStoreFileAge" : 0,
"Namespace_hbase_table_meta_metric_numReferenceFiles" : 0,
"Namespace_hbase_table_namespace_metric_readRequestCount" : 2,
"Namespace_hbase_table_namespace_metric_filteredReadRequestCount" : 0,
"Namespace_hbase_table_namespace_metric_writeRequestCount" : 2,
"Namespace_hbase_table_namespace_metric_totalRequestCount" : 4,
"Namespace_hbase_table_namespace_metric_memStoreSize" : 520,
"Namespace_hbase_table_namespace_metric_storeFileCount" : 0,
"Namespace_hbase_table_namespace_metric_storeFileSize" : 0,
"Namespace_hbase_table_namespace_metric_tableSize" : 520,
"Namespace_hbase_table_namespace_metric_averageRegionSize" : 520,
"Namespace_hbase_table_namespace_metric_regionCount" : 1,
"Namespace_hbase_table_namespace_metric_storeCount" : 1,
"Namespace_hbase_table_namespace_metric_maxStoreFileAge" : 0,
"Namespace_hbase_table_namespace_metric_minStoreFileAge" : 0,
"Namespace_hbase_table_namespace_metric_avgStoreFileAge" : 0,
"Namespace_hbase_table_namespace_metric_numReferenceFiles" : 0, "numTables" :
2, "Namespace_hbase_table_meta_metric_splitRequestCount" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_num_ops" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_min" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_max" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_mean" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_25th_percentile"
: 0, "Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_median"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_75th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_90th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_95th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_98th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_99th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_99.9th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_num_ops" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_min" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_max" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_mean" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_25th_percentile"
: 0, "Namespace_hbase_table_namespace_metric_compactionOutputFileCount_median"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_75th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_90th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_95th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_98th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_99th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_99.9th_percentile"
: 0, "Namespace_hbase_table_namespace_metric_flushTime_num_ops" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_min" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_max" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_mean" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_25th_percentile" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_median" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_75th_percentile" :
[jira] [Commented] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[
https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595591#comment-16595591
]
Xu Cang commented on HBASE-15728:
-
Ported this patch to current branch-1 (It's been 2 years since previous patch.
I am picking it up now.)
Result from JMX:
{
"name" : "Hadoop:service=HBase,name=RegionServer,sub=Tables",
"modelerType" : "RegionServer,sub=Tables",
"tag.Context" : "regionserver",
"tag.Hostname" : "xcang-wsl",
"Namespace_hbase_table_meta_metric_readRequestCount" : 1,
"Namespace_hbase_table_meta_metric_filteredReadRequestCount" : 0,
"Namespace_hbase_table_meta_metric_writeRequestCount" : 2,
"Namespace_hbase_table_meta_metric_totalRequestCount" : 3,
"Namespace_hbase_table_meta_metric_memStoreSize" : 1272,
"Namespace_hbase_table_meta_metric_storeFileCount" : 0,
"Namespace_hbase_table_meta_metric_storeFileSize" : 0,
"Namespace_hbase_table_meta_metric_tableSize" : 1272,
"Namespace_hbase_table_meta_metric_averageRegionSize" : 1272,
"Namespace_hbase_table_meta_metric_regionCount" : 1,
"Namespace_hbase_table_meta_metric_storeCount" : 1,
"Namespace_hbase_table_meta_metric_maxStoreFileAge" : 0,
"Namespace_hbase_table_meta_metric_minStoreFileAge" : 0,
"Namespace_hbase_table_meta_metric_avgStoreFileAge" : 0,
"Namespace_hbase_table_meta_metric_numReferenceFiles" : 0,
"Namespace_hbase_table_namespace_metric_readRequestCount" : 2,
"Namespace_hbase_table_namespace_metric_filteredReadRequestCount" : 0,
"Namespace_hbase_table_namespace_metric_writeRequestCount" : 2,
"Namespace_hbase_table_namespace_metric_totalRequestCount" : 4,
"Namespace_hbase_table_namespace_metric_memStoreSize" : 520,
"Namespace_hbase_table_namespace_metric_storeFileCount" : 0,
"Namespace_hbase_table_namespace_metric_storeFileSize" : 0,
"Namespace_hbase_table_namespace_metric_tableSize" : 520,
"Namespace_hbase_table_namespace_metric_averageRegionSize" : 520,
"Namespace_hbase_table_namespace_metric_regionCount" : 1,
"Namespace_hbase_table_namespace_metric_storeCount" : 1,
"Namespace_hbase_table_namespace_metric_maxStoreFileAge" : 0,
"Namespace_hbase_table_namespace_metric_minStoreFileAge" : 0,
"Namespace_hbase_table_namespace_metric_avgStoreFileAge" : 0,
"Namespace_hbase_table_namespace_metric_numReferenceFiles" : 0,
"numTables" : 2,
"Namespace_hbase_table_meta_metric_splitRequestCount" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_num_ops"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_min" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_max" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_mean" : 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_25th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_median" :
0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_75th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_90th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_95th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_98th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_99th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_majorCompactionOutputSize_99.9th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_num_ops"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_min" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_max" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_mean" : 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_25th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_median" :
0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_75th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_90th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_95th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_98th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_99th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_compactionOutputFileCount_99.9th_percentile"
: 0,
"Namespace_hbase_table_namespace_metric_flushTime_num_ops" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_min" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_max" : 0,
"Namespace_hbase_table_namespace_metric_flushTime_mean" : 0,
[jira] [Assigned] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang reassigned HBASE-15728: --- Assignee: Xu Cang (was: Enis Soztutar) > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 1.5.0 > > Attachments: hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21098) Improve Snapshot Performance with Temporary Snapshot Directory when rootDir on S3
[ https://issues.apache.org/jira/browse/HBASE-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595581#comment-16595581 ] Tyler Mi commented on HBASE-21098: -- I will be unavailable to continue working on this starting September 10th as I will be traveling. If any reviewers planning on looking through the code could review it by this weekend, I would greatly appreciate it! > Improve Snapshot Performance with Temporary Snapshot Directory when rootDir > on S3 > - > > Key: HBASE-21098 > URL: https://issues.apache.org/jira/browse/HBASE-21098 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.1.1 >Reporter: Tyler Mi >Priority: Major > Attachments: HBASE-21098.master.001.patch, > HBASE-21098.master.002.patch, HBASE-21098.master.003.patch, > HBASE-21098.master.004.patch, HBASE-21098.master.005.patch, > HBASE-21098.master.006.patch > > > When using Apache HBase, the snapshot feature can be used to make a point in > time recovery. To do this, HBase creates a manifest of all the files in all > of the Regions so that those files can be referenced again when a user > restores a snapshot. With HBase's S3 storage mode, developers can store their > data off-cluster on Amazon S3. However, utilizing S3 as a file system is > inefficient in some operations, namely renames. Most Hadoop ecosystem > applications use an atomic rename as a method of committing data. However, > with S3, a rename is a separate copy and then a delete of every file which is > no longer atomic and, in fact, quite costly. In addition, puts and deletes on > S3 have latency issues that traditional filesystems do not encounter when > manipulating the region snapshots to consolidate into a single manifest. When > HBase on S3 users have a significant amount of regions, puts, deletes, and > renames (the final commit stage of the snapshot) become the bottleneck > causing snapshots to take many minutes or even hours to complete. > The purpose of this patch is to increase the overall performance of snapshots > while utilizing HBase on S3 through the use of a temporary directory for the > snapshots that exists on a traditional filesystem like HDFS to circumvent the > bottlenecks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[
https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595558#comment-16595558
]
Hudson commented on HBASE-20942:
Results for branch branch-1.4
[build #441 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/441/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/441//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/441//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/441//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Improve RpcServer TRACE logging
> ---
>
> Key: HBASE-20942
> URL: https://issues.apache.org/jira/browse/HBASE-20942
> Project: HBase
> Issue Type: Task
> Components: Operability
>Affects Versions: 2.1.0, 1.4.6
>Reporter: Esteban Gutierrez
>Assignee: Krish Dey
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7
>
> Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch,
> HBASE-20942.004.patch, HBASE-20942.005.patch
>
>
> Two things:
> * We truncate RpcServer output to 1000 characters for trace logging. Would
> be better if that value was configurable.
> * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE
> log message.
> Esteban mentioned this to me earlier, so I'm crediting him as the reporter.
> cc: [~elserj]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[
https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595520#comment-16595520
]
Hadoop QA commented on HBASE-21083:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
1s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
20s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
8s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 44s{color}
| {color:red} hbase-protocol-shaded generated 2 new + 98 unchanged - 2 fixed =
100 total (was 100) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
38s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 21s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
35s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
59s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}181m 3s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}257m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestSplitOrMergeStatus |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce
[jira] [Commented] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[ https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595503#comment-16595503 ] stack commented on HBASE-21083: --- Retry. > Introduce a mechanism to bypass the execution of a stuck procedure > -- > > Key: HBASE-21083 > URL: https://issues.apache.org/jira/browse/HBASE-21083 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21083.branch-2.0.001.patch, > HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.0.003.patch, HBASE-21083.branch-2.1.001.patch > > > Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to > introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can > continue running. > we still have some unrevealed bugs hiding in our AMv2 and procedureV2 > system, we need something to interfere with stuck procedures before HBCK2 can > work. This is very crucial for a production ready system. > For now, we have little ways to interfere with running procedures. Aborting > them is not a good choice, since some procedures are not abort-able. And some > procedure may have overridden the abort() method, which will ignore the abort > request. > So, here, I will introduce a mechanism to bypass the execution of a stuck > procedure. > Basically, I added a field called 'bypass' to Procedure class. If we set this > field to true, all the logic in execute/rollback will be skipped, letting > this procedure and its ancestors complete normally and releasing the lock > resources at last. > Notice that bypassing a procedure may leave the cluster in a middle state, > e.g. the region not assigned, or some hdfs files left behind. > The Operators need know the side effect of bypassing and recover the > inconsistent state of the cluster themselves, like issuing new procedures to > assign the regions. > A patch will be uploaded and review board will be open. For now, only APIs in > ProcedureExecutor are provided. If anything is fine, I will add it to master > service and add a shell command to bypass a procedure. Or, maybe we can use > dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[ https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21083: -- Attachment: HBASE-21083.branch-2.0.003.patch > Introduce a mechanism to bypass the execution of a stuck procedure > -- > > Key: HBASE-21083 > URL: https://issues.apache.org/jira/browse/HBASE-21083 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21083.branch-2.0.001.patch, > HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.0.003.patch, HBASE-21083.branch-2.1.001.patch > > > Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to > introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can > continue running. > we still have some unrevealed bugs hiding in our AMv2 and procedureV2 > system, we need something to interfere with stuck procedures before HBCK2 can > work. This is very crucial for a production ready system. > For now, we have little ways to interfere with running procedures. Aborting > them is not a good choice, since some procedures are not abort-able. And some > procedure may have overridden the abort() method, which will ignore the abort > request. > So, here, I will introduce a mechanism to bypass the execution of a stuck > procedure. > Basically, I added a field called 'bypass' to Procedure class. If we set this > field to true, all the logic in execute/rollback will be skipped, letting > this procedure and its ancestors complete normally and releasing the lock > resources at last. > Notice that bypassing a procedure may leave the cluster in a middle state, > e.g. the region not assigned, or some hdfs files left behind. > The Operators need know the side effect of bypassing and recover the > inconsistent state of the cluster themselves, like issuing new procedures to > assign the regions. > A patch will be uploaded and review board will be open. For now, only APIs in > ProcedureExecutor are provided. If anything is fine, I will add it to master > service and add a shell command to bypass a procedure. Or, maybe we can use > dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20940) HStore.cansplit should not allow split to happen if it has references
[
https://issues.apache.org/jira/browse/HBASE-20940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595498#comment-16595498
]
Hadoop QA commented on HBASE-20940:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} hbase-server: The patch generated 0 new + 69
unchanged - 4 fixed = 69 total (was 73) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
0s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
2m 5s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}120m 35s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-20940 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12937476/HBASE-20940-branch-1-addendum.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a77c8970dc93 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64
[jira] [Commented] (HBASE-20940) HStore.cansplit should not allow split to happen if it has references
[ https://issues.apache.org/jira/browse/HBASE-20940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595495#comment-16595495 ] Hudson commented on HBASE-20940: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #466 (See [https://builds.apache.org/job/HBase-1.3-IT/466/]) Amend HBASE-20940 HStore.cansplit should not allow split to happen if it (apurtell: rev 0ebcd11da0bfc83cc73026c477e78266911f24bf) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > HStore.cansplit should not allow split to happen if it has references > - > > Key: HBASE-20940 > URL: https://issues.apache.org/jira/browse/HBASE-20940 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 1.4.7, 2.0.3 > > Attachments: HBASE-20940-branch-1-addendum.patch, > HBASE-20940.branch-1.3.v1.patch, HBASE-20940.branch-1.3.v2.patch, > HBASE-20940.branch-1.v1.patch, HBASE-20940.branch-1.v2.patch, > HBASE-20940.branch-1.v3.patch, HBASE-20940.branch-1.v5.patch, > HBASE-20940.v1.patch, HBASE-20940.v2.patch, HBASE-20940.v3.patch, > HBASE-20940.v4.patch, result_HBASE-20940.branch-1.v2.log > > > When split happens and immediately another split happens, it may result into > a split of a region who still has references to its parent. More details > about scenario can be found here HBASE-20933 > HStore.hasReferences should check from fs.storefile rather than in memory > objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[ https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595494#comment-16595494 ] Umesh Agashe commented on HBASE-21083: -- Thanks for addressing the review comments, [~stack]! Thanks [~allan163] for the changes! > Introduce a mechanism to bypass the execution of a stuck procedure > -- > > Key: HBASE-21083 > URL: https://issues.apache.org/jira/browse/HBASE-21083 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21083.branch-2.0.001.patch, > HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.1.001.patch > > > Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to > introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can > continue running. > we still have some unrevealed bugs hiding in our AMv2 and procedureV2 > system, we need something to interfere with stuck procedures before HBCK2 can > work. This is very crucial for a production ready system. > For now, we have little ways to interfere with running procedures. Aborting > them is not a good choice, since some procedures are not abort-able. And some > procedure may have overridden the abort() method, which will ignore the abort > request. > So, here, I will introduce a mechanism to bypass the execution of a stuck > procedure. > Basically, I added a field called 'bypass' to Procedure class. If we set this > field to true, all the logic in execute/rollback will be skipped, letting > this procedure and its ancestors complete normally and releasing the lock > resources at last. > Notice that bypassing a procedure may leave the cluster in a middle state, > e.g. the region not assigned, or some hdfs files left behind. > The Operators need know the side effect of bypassing and recover the > inconsistent state of the cluster themselves, like issuing new procedures to > assign the regions. > A patch will be uploaded and review board will be open. For now, only APIs in > ProcedureExecutor are provided. If anything is fine, I will add it to master > service and add a shell command to bypass a procedure. Or, maybe we can use > dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[
https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595483#comment-16595483
]
Hadoop QA commented on HBASE-21083:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
42s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 4s{color}
| {color:red} hbase-protocol-shaded generated 2 new + 98 unchanged - 2 fixed =
100 total (was 100) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
22s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
6m 45s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
39s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
0s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}194m 37s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}270m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAsyncTableGetMultiThreaded |
| | hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595462#comment-16595462 ] Andrew Purtell commented on HBASE-20942: That would be great, thank you. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595460#comment-16595460 ] Josh Elser commented on HBASE-20942: bq. I think the latter has worse outcomes from the former, would you not agree? I don't think I am asking for too much here. I honestly don't know what the right outcome is. I don't like the idea of spinning out X Jira issues just to get a single change committed (the hoops to get something committed everywhere is already pedantic). My plan of action will be to just spin out clones for all branches requiring RM-response going forward. I think this will avoid causing more heartburn for the situation you described. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595448#comment-16595448 ] Andrew Purtell edited comment on HBASE-20942 at 8/28/18 7:05 PM: - I don't want to argue about it either, but it happens to me every time I try to make a release, and makes doing the release hard. An alternative is we stop relying on JIRA to produce change logs. bq. For something which will be resolved in a few hours You say this, but you well know delays here are arbitrarily long. Maybe in a few hours. This time. Meanwhile I've been spinning trying to make a release for four days. Our time collectively is precious and we interact on JIRA in asynchronous ways. I think we can either keep JIRA state up to date with commit status by following my above suggestion, or not. If not, fine, then JIRA will not be an authoritative source for creating change logs and RMs will have to do something else. I think the latter has worse outcomes from the former, would you not agree? I don't think I am asking for too much here. was (Author: apurtell): I don't want to argue about it either, but it happens to me every time I try to make a release, and makes doing the release hard. An alternative is we stop relying on JIRA to produce change logs. bq. For something which will be resolved in a few hours You say this, but you well know delays here are arbitrarily long. Maybe in a few hours. This time. Meanwhile I've been spinning trying to make a release for four days. Our time collectively is precious and we interact on JIRA in asynchronous ways. I think we can either keep JIRA state up to date with commit status by following my above suggestion, or not. If not, fine, then JIRA will not be an authoritative source for creating commit logs and RMs will have to do something else. I think the latter has worse outcomes from the former, would you not agree? I don't think I am asking for too much here. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595448#comment-16595448 ] Andrew Purtell edited comment on HBASE-20942 at 8/28/18 7:03 PM: - I don't want to argue about it either, but it happens to me every time I try to make a release, and makes doing the release hard. An alternative is we stop relying on JIRA to produce change logs. bq. For something which will be resolved in a few hours You say this, but you well know delays here are arbitrarily long. Maybe in a few hours. This time. Meanwhile I've been spinning trying to make a release for four days. Our time collectively is precious and we interact on JIRA in asynchronous ways. I think we can either keep JIRA state up to date with commit status by following my above suggestion, or not. If not, fine, then JIRA will not be an authoritative source for creating commit logs and RMs will have to do something else. I think the latter has worse outcomes from the former, would you not agree? I don't think I am asking for too much here. was (Author: apurtell): I don't want to argue about it either, but it happens to me every time I try to make a release, and makes doing the release hard. An alternative is we stop relying on JIRA to produce change logs. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595448#comment-16595448 ] Andrew Purtell commented on HBASE-20942: I don't want to argue about it either, but it happens to me every time I try to make a release, and makes doing the release hard. An alternative is we stop relying on JIRA to produce change logs. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21125) CLONE - Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser reassigned HBASE-21125: -- Assignee: Duo Zhang (was: Krish Dey) [~Apache9], waiting on your approval. > CLONE - Improve RpcServer TRACE logging > --- > > Key: HBASE-21125 > URL: https://issues.apache.org/jira/browse/HBASE-21125 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Duo Zhang >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-20942: --- Fix Version/s: (was: 2.1.1) > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21125) CLONE - Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-21125: --- Fix Version/s: (was: 1.4.7) (was: 2.2.0) (was: 2.0.2) (was: 1.3.3) (was: 1.5.0) (was: 3.0.0) 2.1.1 > CLONE - Improve RpcServer TRACE logging > --- > > Key: HBASE-21125 > URL: https://issues.apache.org/jira/browse/HBASE-21125 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21125) CLONE - Improve RpcServer TRACE logging
Josh Elser created HBASE-21125: -- Summary: CLONE - Improve RpcServer TRACE logging Key: HBASE-21125 URL: https://issues.apache.org/jira/browse/HBASE-21125 Project: HBase Issue Type: Task Components: Operability Affects Versions: 2.1.0, 1.4.6 Reporter: Esteban Gutierrez Assignee: Krish Dey Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.2, 1.4.7 Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, HBASE-20942.004.patch, HBASE-20942.005.patch Two things: * We truncate RpcServer output to 1000 characters for trace logging. Would be better if that value was configurable. * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE log message. Esteban mentioned this to me earlier, so I'm crediting him as the reporter. cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595444#comment-16595444 ] Josh Elser commented on HBASE-20942: bq. Then the RM comes along and tries to make a release and JIRA state is wrong. This strikes me as the exception, not the rule. bq. I'd be happy to revert all commits prior to rolling the RC that correspond to unresolved JIRAs if you'd prefer. Without snark, I really don't care. You're the RM, so I assume you will do the right thing, given how you are tracking releases. bq. Commit what you can, close the JIRA. Open new JIRAs for remaining commits. Just what I said above. This is not difficult. We just need to do it. For something which will be resolved in a few hours, I see this as creating unnecessary work and convoluting what is already a convoluted "what release has a fix HBASE-X". That said, I don't have the interest to argue about it either. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595439#comment-16595439 ] Andrew Purtell commented on HBASE-20942: What we can't abide is something committed somewhere with the JIRA still open. This is incorrect. The change has been committed (at least somewhere). Then the RM comes along and tries to make a release and JIRA state is wrong. I'd be happy to revert all commits prior to rolling the RC that correspond to unresolved JIRAs if you'd prefer. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595437#comment-16595437 ] Andrew Purtell commented on HBASE-20942: Commit what you can, close the JIRA. Open new JIRAs for remaining commits. Just what I said above. This is not difficult. We just need to do it. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595434#comment-16595434 ] Josh Elser commented on HBASE-20942: bq. It would be great if we can stamp out this practice If you want to stamp out this practice, you need to stamp out the "asking for approval" on certain branches. You can't have both. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21076) TestTableResource fails with NPE
[
https://issues.apache.org/jira/browse/HBASE-21076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-21076:
---
Fix Version/s: (was: 1.4.8)
1.4.7
> TestTableResource fails with NPE
>
>
> Key: HBASE-21076
> URL: https://issues.apache.org/jira/browse/HBASE-21076
> Project: HBase
> Issue Type: Test
> Components: REST, test
>Affects Versions: 3.0.0, 1.5.0, 1.3.3, 2.1.1, 2.0.2, 1.4.7
>Reporter: Ted Yu
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.2.7, 2.1.1, 2.0.2, 1.4.7
>
> Attachments: HBASE-21076.0.patch, HBASE-21076.1.patch
>
>
> The following can be observed in master branch:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.rest.TestTableResource.setUpBeforeClass(TestTableResource.java:134)
> {code}
> The NPE comes from the following in TestEndToEndSplitTransaction :
> {code}
> compactAndBlockUntilDone(TEST_UTIL.getAdmin(),
> TEST_UTIL.getMiniHBaseCluster().getRegionServer(0),
> daughterA.getRegionName());
> {code}
> Initial check of the code shows that TestEndToEndSplitTransaction uses
> TEST_UTIL instance which is created within TestEndToEndSplitTransaction.
> However, TestTableResource creates its own instance of HBaseTestingUtility.
> Meaning TEST_UTIL.getMiniHBaseCluster() would return null, since the instance
> created by TestEndToEndSplitTransaction has hbaseCluster as null.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595432#comment-16595432 ] Josh Elser commented on HBASE-20942: [~apurtell], this is *not* committed to all branches. I'm still waiting on a reply from Duo to commit it to his branch. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20942: --- Resolution: Fixed Status: Resolved (was: Patch Available) I'm resolving this so the release notes for 1.4.7 are not incorrect > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595429#comment-16595429 ] Andrew Purtell commented on HBASE-20942: Hey! Another issue committed to some branches but still left in unresolved state. This makes it difficult for an RM to compile release notes. It would be great if we can stamp out this practice. Commit everywhere at once, resolve the JIRA. If you can't commit to all branches, commit to where you can and then resolve the current issue, and open new issues for forward or backwards ports as needed. > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20942: --- Fix Version/s: (was: 1.4.8) 1.4.7 > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20942: --- Fix Version/s: (was: 1.4.7) 1.4.8 > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.2 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20940) HStore.cansplit should not allow split to happen if it has references
[ https://issues.apache.org/jira/browse/HBASE-20940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20940: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed up the addendum. Thanks [~vishk] > HStore.cansplit should not allow split to happen if it has references > - > > Key: HBASE-20940 > URL: https://issues.apache.org/jira/browse/HBASE-20940 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 1.4.7, 2.0.3 > > Attachments: HBASE-20940-branch-1-addendum.patch, > HBASE-20940.branch-1.3.v1.patch, HBASE-20940.branch-1.3.v2.patch, > HBASE-20940.branch-1.v1.patch, HBASE-20940.branch-1.v2.patch, > HBASE-20940.branch-1.v3.patch, HBASE-20940.branch-1.v5.patch, > HBASE-20940.v1.patch, HBASE-20940.v2.patch, HBASE-20940.v3.patch, > HBASE-20940.v4.patch, result_HBASE-20940.branch-1.v2.log > > > When split happens and immediately another split happens, it may result into > a split of a region who still has references to its parent. More details > about scenario can be found here HBASE-20933 > HStore.hasReferences should check from fs.storefile rather than in memory > objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21107) add a metrics for netty direct memory
[ https://issues.apache.org/jira/browse/HBASE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-21107: - Attachment: HBASE-21107-master-v1.patch > add a metrics for netty direct memory > - > > Key: HBASE-21107 > URL: https://issues.apache.org/jira/browse/HBASE-21107 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-21107-master-v1.patch, > HBASE-21107-master-v1.patch, Screen Shot 2018-08-23 at 4.55.01 PM.png > > > This is separated from HBASE-20913 as I realized that netty direct memory > usage needs to show up in metrics instead of the web page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21107) add a metrics for netty direct memory
[ https://issues.apache.org/jira/browse/HBASE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595386#comment-16595386 ] huaxiang sun commented on HBASE-21107: -- reattach the same patch to trigger a QA run. > add a metrics for netty direct memory > - > > Key: HBASE-21107 > URL: https://issues.apache.org/jira/browse/HBASE-21107 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-21107-master-v1.patch, > HBASE-21107-master-v1.patch, Screen Shot 2018-08-23 at 4.55.01 PM.png > > > This is separated from HBASE-20913 as I realized that netty direct memory > usage needs to show up in metrics instead of the web page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21122) bulkload multi-version Cell supported in hbase-spark module
[
https://issues.apache.org/jira/browse/HBASE-21122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595329#comment-16595329
]
Ted Yu edited comment on HBASE-21122 at 8/28/18 5:19 PM:
-
I tried to run the added test alone:
{code}
[ERROR]
/hbase/hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/BulkLoadMultiVersionCellSuite.scala:103:
error: too many arguments for constructor KeyFamilyQualifier: (rowKey:
Array[Byte], family: Array[Byte], qualifier:
Array[Byte])org.apache.hadoop.hbase.spark.KeyFamilyQualifier
[ERROR] val keyFamilyQualifier = new
KeyFamilyQualifier(rowKey, cf, column, timestamp)
[ERROR] ^
[ERROR]
/hbase/hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/BulkLoadMultiVersionCellSuite.scala:114:
error: constructor HTable in class HTable cannot be accessed in class
BulkLoadMultiVersionCellSuite
[ERROR] Access to protected constructor HTable not permitted because
[ERROR] enclosing class BulkLoadMultiVersionCellSuite in package spark is not
a subclass of
[ERROR] class HTable in package client where target is defined
[ERROR] new HTable(hbaseConf, tableName),
{code}
Can you :
* combine the fix with the test
* the test should compile and fail without fix
* the test should compile and pass with fix
* press 'Submit Patch' button after attaching next patch
was (Author: yuzhih...@gmail.com):
I tried to run the added test alone:
{code}
[ERROR]
/mnt/disk2/a/hbase/hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/BulkLoadMultiVersionCellSuite.scala:103:
error: too many arguments for constructor KeyFamilyQualifier: (rowKey:
Array[Byte], family: Array[Byte], qualifier:
Array[Byte])org.apache.hadoop.hbase.spark.KeyFamilyQualifier
[ERROR] val keyFamilyQualifier = new
KeyFamilyQualifier(rowKey, cf, column, timestamp)
[ERROR] ^
[ERROR]
/mnt/disk2/a/hbase/hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/BulkLoadMultiVersionCellSuite.scala:114:
error: constructor HTable in class HTable cannot be accessed in class
BulkLoadMultiVersionCellSuite
[ERROR] Access to protected constructor HTable not permitted because
[ERROR] enclosing class BulkLoadMultiVersionCellSuite in package spark is not
a subclass of
[ERROR] class HTable in package client where target is defined
[ERROR] new HTable(hbaseConf, tableName),
{code}
Can you :
* combine the fix with the test
* the test should compile and fail without fix
* the test should compile and pass with fix
* press 'Submit Patch' button after attaching next patch
> bulkload multi-version Cell supported in hbase-spark module
>
>
> Key: HBASE-21122
> URL: https://issues.apache.org/jira/browse/HBASE-21122
> Project: HBase
> Issue Type: Improvement
> Components: spark
>Affects Versions: 1.1.2, 2.1.1
>Reporter: zhangyanshan
>Priority: Major
> Labels: features
> Attachments: HBASE-21122.000.patch, HBASE-21122.001.patch,
> HBASE-21122.002.patch
>
>
> bulkload multi-version Cell not supported in HBaseContext.scala, this issue
> will add function of bulkload multi-version Cell
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21122) bulkload multi-version Cell supported in hbase-spark module
[
https://issues.apache.org/jira/browse/HBASE-21122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595329#comment-16595329
]
Ted Yu commented on HBASE-21122:
I tried to run the added test alone:
{code}
[ERROR]
/mnt/disk2/a/hbase/hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/BulkLoadMultiVersionCellSuite.scala:103:
error: too many arguments for constructor KeyFamilyQualifier: (rowKey:
Array[Byte], family: Array[Byte], qualifier:
Array[Byte])org.apache.hadoop.hbase.spark.KeyFamilyQualifier
[ERROR] val keyFamilyQualifier = new
KeyFamilyQualifier(rowKey, cf, column, timestamp)
[ERROR] ^
[ERROR]
/mnt/disk2/a/hbase/hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/BulkLoadMultiVersionCellSuite.scala:114:
error: constructor HTable in class HTable cannot be accessed in class
BulkLoadMultiVersionCellSuite
[ERROR] Access to protected constructor HTable not permitted because
[ERROR] enclosing class BulkLoadMultiVersionCellSuite in package spark is not
a subclass of
[ERROR] class HTable in package client where target is defined
[ERROR] new HTable(hbaseConf, tableName),
{code}
Can you :
* combine the fix with the test
* the test should compile and fail without fix
* the test should compile and pass with fix
* press 'Submit Patch' button after attaching next patch
> bulkload multi-version Cell supported in hbase-spark module
>
>
> Key: HBASE-21122
> URL: https://issues.apache.org/jira/browse/HBASE-21122
> Project: HBase
> Issue Type: Improvement
> Components: spark
>Affects Versions: 1.1.2, 2.1.1
>Reporter: zhangyanshan
>Priority: Major
> Labels: features
> Attachments: HBASE-21122.000.patch, HBASE-21122.001.patch,
> HBASE-21122.002.patch
>
>
> bulkload multi-version Cell not supported in HBaseContext.scala, this issue
> will add function of bulkload multi-version Cell
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20734: -- Attachment: HBASE-20734.master.007.patch > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Assignee: Zach York >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20734.branch-1.001.patch, > HBASE-20734.branch-1.002.patch, HBASE-20734.master.001.patch, > HBASE-20734.master.002.patch, HBASE-20734.master.003.patch, > HBASE-20734.master.004.patch, HBASE-20734.master.005.patch, > HBASE-20734.master.006.patch, HBASE-20734.master.007.patch > > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20175) hbase-spark needs scala dependency convergance
[
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595316#comment-16595316
]
Hadoop QA commented on HBASE-20175:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
9s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
12s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 35s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
59s{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20175 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12937466/HBASE-20175.v06.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile |
| uname | Linux d4c534d7f3eb 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / fcd883b5dd |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14242/testReport/ |
| Max. process+thread count | 795 (vs. ulimit of 1) |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14242/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
> Project: HBase
> Issue Type: Bug
> Components: dependencies, spark
>Reporter: Mike Drob
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-20175.v01.patch, HBASE-20175.v04.patch,
>
[jira] [Updated] (HBASE-20940) HStore.cansplit should not allow split to happen if it has references
[ https://issues.apache.org/jira/browse/HBASE-20940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal Khandelwal updated HBASE-20940: -- Attachment: HBASE-20940-branch-1-addendum.patch > HStore.cansplit should not allow split to happen if it has references > - > > Key: HBASE-20940 > URL: https://issues.apache.org/jira/browse/HBASE-20940 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 1.4.7, 2.0.3 > > Attachments: HBASE-20940-branch-1-addendum.patch, > HBASE-20940.branch-1.3.v1.patch, HBASE-20940.branch-1.3.v2.patch, > HBASE-20940.branch-1.v1.patch, HBASE-20940.branch-1.v2.patch, > HBASE-20940.branch-1.v3.patch, HBASE-20940.branch-1.v5.patch, > HBASE-20940.v1.patch, HBASE-20940.v2.patch, HBASE-20940.v3.patch, > HBASE-20940.v4.patch, result_HBASE-20940.branch-1.v2.log > > > When split happens and immediately another split happens, it may result into > a split of a region who still has references to its parent. More details > about scenario can be found here HBASE-20933 > HStore.hasReferences should check from fs.storefile rather than in memory > objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20940) HStore.cansplit should not allow split to happen if it has references
[ https://issues.apache.org/jira/browse/HBASE-20940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595281#comment-16595281 ] Andrew Purtell commented on HBASE-20940: I should have some time later today. I can revert your old change, apply your latest patch, then make an addendum diff by comparing it with the upstream tracking branch. (FYI, if you want to try this trick going forward). If all tests pass for me I will commit the addendum. Will be back later with an update. > HStore.cansplit should not allow split to happen if it has references > - > > Key: HBASE-20940 > URL: https://issues.apache.org/jira/browse/HBASE-20940 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 1.4.7, 2.0.3 > > Attachments: HBASE-20940.branch-1.3.v1.patch, > HBASE-20940.branch-1.3.v2.patch, HBASE-20940.branch-1.v1.patch, > HBASE-20940.branch-1.v2.patch, HBASE-20940.branch-1.v3.patch, > HBASE-20940.branch-1.v5.patch, HBASE-20940.v1.patch, HBASE-20940.v2.patch, > HBASE-20940.v3.patch, HBASE-20940.v4.patch, result_HBASE-20940.branch-1.v2.log > > > When split happens and immediately another split happens, it may result into > a split of a region who still has references to its parent. More details > about scenario can be found here HBASE-20933 > HStore.hasReferences should check from fs.storefile rather than in memory > objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20940) HStore.cansplit should not allow split to happen if it has references
[ https://issues.apache.org/jira/browse/HBASE-20940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595278#comment-16595278 ] Andrew Purtell commented on HBASE-20940: [~vishk] We need a patch against current branch-1, just the new changes, which we can then commit as an addendum. > HStore.cansplit should not allow split to happen if it has references > - > > Key: HBASE-20940 > URL: https://issues.apache.org/jira/browse/HBASE-20940 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 1.4.7, 2.0.3 > > Attachments: HBASE-20940.branch-1.3.v1.patch, > HBASE-20940.branch-1.3.v2.patch, HBASE-20940.branch-1.v1.patch, > HBASE-20940.branch-1.v2.patch, HBASE-20940.branch-1.v3.patch, > HBASE-20940.branch-1.v5.patch, HBASE-20940.v1.patch, HBASE-20940.v2.patch, > HBASE-20940.v3.patch, HBASE-20940.v4.patch, result_HBASE-20940.branch-1.v2.log > > > When split happens and immediately another split happens, it may result into > a split of a region who still has references to its parent. More details > about scenario can be found here HBASE-20933 > HStore.hasReferences should check from fs.storefile rather than in memory > objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20175) hbase-spark needs scala dependency convergance
[
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595238#comment-16595238
]
Artem Ervits commented on HBASE-20175:
--
[~mdrob] based on the vote
[https://mail-archives.apache.org/mod_mbox/hbase-dev/201808.mbox/%3CCAFAMeYLNXhyGzHyNBKYjjVWBbVgagpw35nTPOL=umkan5ni...@mail.gmail.com%3E]
I'm attaching an exclusion pattern approach in v06 patch. Please review.
> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
> Project: HBase
> Issue Type: Bug
> Components: dependencies, spark
>Reporter: Mike Drob
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-20175.v01.patch, HBASE-20175.v04.patch,
> HBASE-20175.v05.patch, HBASE-20175.v06.patch
>
>
> This is a follow-on to HBASE-16179 - I think we might need to specify an
> exclude in the dependency management.
> {noformat}
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
> hbase-spark ---
> [WARNING] Expected all dependencies to require Scala version: 2.11.8
> [WARNING] org.apache.hbase:hbase-spark:3.0.0-SNAPSHOT requires scala
> version: 2.11.8
> [WARNING] org.apache.spark:spark-streaming_2.11:2.1.1 requires scala
> version: 2.11.8
> [WARNING] org.apache.spark:spark-streaming_2.11:2.1.1 requires scala
> version: 2.11.8
> [WARNING] org.scalatest:scalatest_2.11:2.2.4 requires scala version: 2.11.2
> {noformat}
> [~tedyu] - since you're already fiddling in this area, do you want to take a
> look?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20175) hbase-spark needs scala dependency convergance
[
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artem Ervits updated HBASE-20175:
-
Attachment: HBASE-20175.v06.patch
> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
> Project: HBase
> Issue Type: Bug
> Components: dependencies, spark
>Reporter: Mike Drob
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-20175.v01.patch, HBASE-20175.v04.patch,
> HBASE-20175.v05.patch, HBASE-20175.v06.patch
>
>
> This is a follow-on to HBASE-16179 - I think we might need to specify an
> exclude in the dependency management.
> {noformat}
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
> hbase-spark ---
> [WARNING] Expected all dependencies to require Scala version: 2.11.8
> [WARNING] org.apache.hbase:hbase-spark:3.0.0-SNAPSHOT requires scala
> version: 2.11.8
> [WARNING] org.apache.spark:spark-streaming_2.11:2.1.1 requires scala
> version: 2.11.8
> [WARNING] org.apache.spark:spark-streaming_2.11:2.1.1 requires scala
> version: 2.11.8
> [WARNING] org.scalatest:scalatest_2.11:2.2.4 requires scala version: 2.11.2
> {noformat}
> [~tedyu] - since you're already fiddling in this area, do you want to take a
> look?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21075) Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after HBASE-20881
[ https://issues.apache.org/jira/browse/HBASE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595200#comment-16595200 ] stack commented on HBASE-21075: --- Trying to figure something that might be a little easier on the operator. A clean shutdown should also work? Wait for quiescent amv2.. no assigning, crashing, then do shutdown. Let me check if this makes for any assign/unassigns. A tool could look at master wal proc and report if any assign/unassigns outstanding. Or, tool could look at master proc wals to see if outstanding assign/unassigns. If none, kill masters (standby first). Re-run tool to be sure. Then start 2.2? > Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after > HBASE-20881 > - > > Key: HBASE-21075 > URL: https://issues.apache.org/jira/browse/HBASE-21075 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.1.1, 2.0.3 > > Attachments: HBASE-21075.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595180#comment-16595180 ] Hudson commented on HBASE-20942: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #465 (See [https://builds.apache.org/job/HBase-1.3-IT/465/]) HBASE-20942 Fix ArrayIndexOutOfBoundsException for RpcServer TRACE (elserj: rev a2ea72dabaca836be5572d1e93c68ae050390c97) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestRpcServerTraceLogging.java > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[ https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595173#comment-16595173 ] stack commented on HBASE-21083: --- Thanks [~allan163]. > Introduce a mechanism to bypass the execution of a stuck procedure > -- > > Key: HBASE-21083 > URL: https://issues.apache.org/jira/browse/HBASE-21083 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21083.branch-2.0.001.patch, > HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.1.001.patch > > > Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to > introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can > continue running. > we still have some unrevealed bugs hiding in our AMv2 and procedureV2 > system, we need something to interfere with stuck procedures before HBCK2 can > work. This is very crucial for a production ready system. > For now, we have little ways to interfere with running procedures. Aborting > them is not a good choice, since some procedures are not abort-able. And some > procedure may have overridden the abort() method, which will ignore the abort > request. > So, here, I will introduce a mechanism to bypass the execution of a stuck > procedure. > Basically, I added a field called 'bypass' to Procedure class. If we set this > field to true, all the logic in execute/rollback will be skipped, letting > this procedure and its ancestors complete normally and releasing the lock > resources at last. > Notice that bypassing a procedure may leave the cluster in a middle state, > e.g. the region not assigned, or some hdfs files left behind. > The Operators need know the side effect of bypassing and recover the > inconsistent state of the cluster themselves, like issuing new procedures to > assign the regions. > A patch will be uploaded and review board will be open. For now, only APIs in > ProcedureExecutor are provided. If anything is fine, I will add it to master > service and add a shell command to bypass a procedure. Or, maybe we can use > dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[ https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595166#comment-16595166 ] Allan Yang commented on HBASE-21083: Uploaded a HBASE-21083.branch-2.0.003 patch based on [~stack]'s .001 against branch-2.1. And uploaded it to review board too. Thanks all for reviewing. The most significant change in this path is that according to [~Apache9]'s review comment, we need to count the wait time in tryLockEntry(long id, long time) our self , since JVM may wake the waiting thread even time is not up. > Introduce a mechanism to bypass the execution of a stuck procedure > -- > > Key: HBASE-21083 > URL: https://issues.apache.org/jira/browse/HBASE-21083 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21083.branch-2.0.001.patch, > HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.1.001.patch > > > Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to > introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can > continue running. > we still have some unrevealed bugs hiding in our AMv2 and procedureV2 > system, we need something to interfere with stuck procedures before HBCK2 can > work. This is very crucial for a production ready system. > For now, we have little ways to interfere with running procedures. Aborting > them is not a good choice, since some procedures are not abort-able. And some > procedure may have overridden the abort() method, which will ignore the abort > request. > So, here, I will introduce a mechanism to bypass the execution of a stuck > procedure. > Basically, I added a field called 'bypass' to Procedure class. If we set this > field to true, all the logic in execute/rollback will be skipped, letting > this procedure and its ancestors complete normally and releasing the lock > resources at last. > Notice that bypassing a procedure may leave the cluster in a middle state, > e.g. the region not assigned, or some hdfs files left behind. > The Operators need know the side effect of bypassing and recover the > inconsistent state of the cluster themselves, like issuing new procedures to > assign the regions. > A patch will be uploaded and review board will be open. For now, only APIs in > ProcedureExecutor are provided. If anything is fine, I will add it to master > service and add a shell command to bypass a procedure. Or, maybe we can use > dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[ https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595162#comment-16595162 ] stack commented on HBASE-21083: --- [~allan163] no worries. I figured you were busy. Nothing special about 2.1. What is in your .003 patch? > Introduce a mechanism to bypass the execution of a stuck procedure > -- > > Key: HBASE-21083 > URL: https://issues.apache.org/jira/browse/HBASE-21083 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21083.branch-2.0.001.patch, > HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.1.001.patch > > > Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to > introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can > continue running. > we still have some unrevealed bugs hiding in our AMv2 and procedureV2 > system, we need something to interfere with stuck procedures before HBCK2 can > work. This is very crucial for a production ready system. > For now, we have little ways to interfere with running procedures. Aborting > them is not a good choice, since some procedures are not abort-able. And some > procedure may have overridden the abort() method, which will ignore the abort > request. > So, here, I will introduce a mechanism to bypass the execution of a stuck > procedure. > Basically, I added a field called 'bypass' to Procedure class. If we set this > field to true, all the logic in execute/rollback will be skipped, letting > this procedure and its ancestors complete normally and releasing the lock > resources at last. > Notice that bypassing a procedure may leave the cluster in a middle state, > e.g. the region not assigned, or some hdfs files left behind. > The Operators need know the side effect of bypassing and recover the > inconsistent state of the cluster themselves, like issuing new procedures to > assign the regions. > A patch will be uploaded and review board will be open. For now, only APIs in > ProcedureExecutor are provided. If anything is fine, I will add it to master > service and add a shell command to bypass a procedure. Or, maybe we can use > dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21083) Introduce a mechanism to bypass the execution of a stuck procedure
[ https://issues.apache.org/jira/browse/HBASE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-21083: --- Attachment: HBASE-21083.branch-2.0.003.patch > Introduce a mechanism to bypass the execution of a stuck procedure > -- > > Key: HBASE-21083 > URL: https://issues.apache.org/jira/browse/HBASE-21083 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21083.branch-2.0.001.patch, > HBASE-21083.branch-2.0.002.patch, HBASE-21083.branch-2.0.003.patch, > HBASE-21083.branch-2.1.001.patch > > > Offline discussed with [~stack] and [~Apache9]. We all agreed that we need to > introduce a mechanism to 'force complete' a stuck procedure, so the AMv2 can > continue running. > we still have some unrevealed bugs hiding in our AMv2 and procedureV2 > system, we need something to interfere with stuck procedures before HBCK2 can > work. This is very crucial for a production ready system. > For now, we have little ways to interfere with running procedures. Aborting > them is not a good choice, since some procedures are not abort-able. And some > procedure may have overridden the abort() method, which will ignore the abort > request. > So, here, I will introduce a mechanism to bypass the execution of a stuck > procedure. > Basically, I added a field called 'bypass' to Procedure class. If we set this > field to true, all the logic in execute/rollback will be skipped, letting > this procedure and its ancestors complete normally and releasing the lock > resources at last. > Notice that bypassing a procedure may leave the cluster in a middle state, > e.g. the region not assigned, or some hdfs files left behind. > The Operators need know the side effect of bypassing and recover the > inconsistent state of the cluster themselves, like issuing new procedures to > assign the regions. > A patch will be uploaded and review board will be open. For now, only APIs in > ProcedureExecutor are provided. If anything is fine, I will add it to master > service and add a shell command to bypass a procedure. Or, maybe we can use > dynamically compiled JSPs to execute those APIs as mentioned in HBASE-20679. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-20942: --- Release Note: Allows configuration of the length of RPC messages printed to the log at TRACE level via "hbase.ipc.trace.param.size" in RpcServer. (was: Made it configurable with the parameter hbase.ipc.trace.param.size ) > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20942: -- Component/s: Operability > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595156#comment-16595156 ] stack commented on HBASE-20942: --- +1 for branch-2.0. Lgtm > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task > Components: Operability >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20942) Improve RpcServer TRACE logging
[ https://issues.apache.org/jira/browse/HBASE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595107#comment-16595107 ] Josh Elser commented on HBASE-20942: [~stack] and [~Apache9], please advise for your branches (simple "yes" if you ask me) > Improve RpcServer TRACE logging > --- > > Key: HBASE-20942 > URL: https://issues.apache.org/jira/browse/HBASE-20942 > Project: HBase > Issue Type: Task >Affects Versions: 2.1.0, 1.4.6 >Reporter: Esteban Gutierrez >Assignee: Krish Dey >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 2.0.2, 1.4.7 > > Attachments: HBASE-20942.002.patch, HBASE-20942.003.patch, > HBASE-20942.004.patch, HBASE-20942.005.patch > > > Two things: > * We truncate RpcServer output to 1000 characters for trace logging. Would > be better if that value was configurable. > * There is the chance for an ArrayIndexOutOfBounds when truncating the TRACE > log message. > Esteban mentioned this to me earlier, so I'm crediting him as the reporter. > cc: [~elserj] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
