[jira] [Commented] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[
https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590907#comment-15590907
]
ramkrishna.s.vasudevan commented on HBASE-16414:
ACtually the back ground is that we have already doing this work of creating
DBB here.
The only aspect is that in case of SASL we needed onheap byte[] only. So in
that case we were creating onheap BB.
So the point was if we pass DBB to the new Cryto code can your new AES wrap
directly work on it? You can raise an IA to do that.
{code}
plaintextData = cryptoAES.unwrap(b, saslToken.position(), saslToken.limit());
{code}
> Improve performance for RPC encryption with Apache Common Crypto
>
>
> Key: HBASE-16414
> URL: https://issues.apache.org/jira/browse/HBASE-16414
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch,
> HBASE-16414.003.patch, HBASE-16414.004.patch, HBASE-16414.005.patch,
> HBASE-16414.006.patch, HBASE-16414.007.patch, HBASE-16414.008.patch,
> HBASE-16414.009.patch, HbaseRpcEncryptionWithCrypoto.docx
>
>
> Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to
> "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms
> for secure authentication and data protection. For DIGEST-MD5, it uses DES,
> 3DES or RC4 to do encryption and it is very slow, especially for Scan. This
> will become the bottleneck of the RPC throughput.
> Apache Commons Crypto is a cryptographic library optimized with AES-NI. It
> provides Java API for both cipher level and Java stream level. Developers can
> use it to implement high performance AES encryption/decryption with the
> minimum code and effort. Compare with the current implementation of
> org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher
> and OpenSSL Cipher which is better performance than JCE Cipher. User can
> configure the cipher type and the default is JCE Cipher.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16870:
---
Attachment: (was: HBASE-16870-ADDENDUM.patch)
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-ADDENDUM.patch,
> HBASE-16870-branch-1.1.patch, HBASE-16870-branch-1.3.patch,
> HBASE-16870-branch-1.patch, HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16887) Allow setting different hadoopcheck versions in precommit for different branches
Duo Zhang created HBASE-16887: - Summary: Allow setting different hadoopcheck versions in precommit for different branches Key: HBASE-16887 URL: https://issues.apache.org/jira/browse/HBASE-16887 Project: HBase Issue Type: Bug Components: build Reporter: Duo Zhang http://hbase.apache.org/book.html#hadoop The supportted hadoop versions are different for different HBase versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16870:
---
Attachment: HBASE-16870-ADDENDUM.patch
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-ADDENDUM.patch, HBASE-16870-ADDENDUM.patch,
> HBASE-16870-branch-1.1.patch, HBASE-16870-branch-1.3.patch,
> HBASE-16870-branch-1.patch, HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[
https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590881#comment-15590881
]
Colin Ma commented on HBASE-16414:
--
[~ram_krish], currently, RpcServer wrap the data with:
{quote}
data = ByteBuffer.allocate(dataLength)
{quote}
I think this can be updated with direct byte buffers, and release memory after
call. I'll create a JIRA to track this improvement.
> Improve performance for RPC encryption with Apache Common Crypto
>
>
> Key: HBASE-16414
> URL: https://issues.apache.org/jira/browse/HBASE-16414
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch,
> HBASE-16414.003.patch, HBASE-16414.004.patch, HBASE-16414.005.patch,
> HBASE-16414.006.patch, HBASE-16414.007.patch, HBASE-16414.008.patch,
> HBASE-16414.009.patch, HbaseRpcEncryptionWithCrypoto.docx
>
>
> Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to
> "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms
> for secure authentication and data protection. For DIGEST-MD5, it uses DES,
> 3DES or RC4 to do encryption and it is very slow, especially for Scan. This
> will become the bottleneck of the RPC throughput.
> Apache Commons Crypto is a cryptographic library optimized with AES-NI. It
> provides Java API for both cipher level and Java stream level. Developers can
> use it to implement high performance AES encryption/decryption with the
> minimum code and effort. Compare with the current implementation of
> org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher
> and OpenSSL Cipher which is better performance than JCE Cipher. User can
> configure the cipher type and the default is JCE Cipher.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16870:
---
Attachment: (was: HBASE-16870-ADDENDUM.patch)
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-ADDENDUM.patch,
> HBASE-16870-branch-1.1.patch, HBASE-16870-branch-1.3.patch,
> HBASE-16870-branch-1.patch, HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16870:
---
Attachment: HBASE-16870-ADDENDUM.patch
Attach a new ADDENDUM whihc add licenses for master branch.
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-ADDENDUM.patch,
> HBASE-16870-branch-1.1.patch, HBASE-16870-branch-1.3.patch,
> HBASE-16870-branch-1.patch, HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16884) Add HBase-2.0.x to the hadoop version support matrix in our documentation
[ https://issues.apache.org/jira/browse/HBASE-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16884: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~stack] for reviewing. > Add HBase-2.0.x to the hadoop version support matrix in our documentation > - > > Key: HBASE-16884 > URL: https://issues.apache.org/jira/browse/HBASE-16884 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16884.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16884) Add HBase-2.0.x to the hadoop version support matrix in our documentation
[
https://issues.apache.org/jira/browse/HBASE-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590833#comment-15590833
]
Hadoop QA commented on HBASE-16884:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 10s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 149m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.constraint.TestConstraint |
| | org.apache.hadoop.hbase.client.TestBlockEvictionFromClient |
| | org.apache.hadoop.hbase.TestPartialResultsFromClientSide |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834305/HBASE-16884.patch |
| JIRA Issue | HBASE-16884 |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux dfad8887c36f 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ddc8435 |
| Default Java | 1.8.0_101 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4111/artifact/patchprocess/patch-unit-root.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4111/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4111/testReport/ |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4111/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add HBase-2.0.x to the hadoop version support matrix in our documentation
> -
>
> Key: HBASE-16884
> URL: https://issues.apache.org/jira/browse/HBASE-16884
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16884.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590828#comment-15590828 ] Anoop Sam John commented on HBASE-16608: It is here https://reviews.apache.org/r/53025/ > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590809#comment-15590809 ] ramkrishna.s.vasudevan commented on HBASE-16414: bq.plaintextData = cryptoAES.unwrap(b, saslToken.position(), saslToken.limit()); This line of code works with on heap byte[]. Can this be later worked upon as an IA to work with Direct byte buffers? [~colinma] - What do you think? Ok I shall go ahead with the commit later in the day as we have 2 +1s. Pls speak up if there are any objections. Ping [~apurtell], [~anoop.hbase] [~saint@gmail.com]. > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch, > HBASE-16414.003.patch, HBASE-16414.004.patch, HBASE-16414.005.patch, > HBASE-16414.006.patch, HBASE-16414.007.patch, HBASE-16414.008.patch, > HBASE-16414.009.patch, HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590800#comment-15590800 ] stack commented on HBASE-16608: --- [~anoop.hbase] You updated [~anastas]'s patch w/ collection-based on RB? Want to put link here boss? Thanks. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16463) Improve transparent table/CF encryption with Commons Crypto
[ https://issues.apache.org/jira/browse/HBASE-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590790#comment-15590790 ] ramkrishna.s.vasudevan commented on HBASE-16463: I just skimmed with the patch. Looks good to me. Reg the algo I am not aware of that but as the impl it looks to be well abstracted out. Ping [~Apache9], [~saint@gmail.com], [~anoop.hbase] and [~apurtell]. > Improve transparent table/CF encryption with Commons Crypto > --- > > Key: HBASE-16463 > URL: https://issues.apache.org/jira/browse/HBASE-16463 > Project: HBase > Issue Type: New Feature > Components: encryption >Affects Versions: 2.0.0 >Reporter: Dapeng Sun > Attachments: HBASE-16463.001.patch, HBASE-16463.002.patch, > HBASE-16463.003.patch, HBASE-16463.004.patch, HBASE-16463.004.patch > > > Apache Commons Crypto > (https://commons.apache.org/proper/commons-crypto/index.html) is a > cryptographic library optimized with AES-NI. > HBASE-7544 introduces a framework for transparent encryption feature for > protecting HFile and WAL data at rest. Currently JCE cipher is used bu > default, the improvement will use Commons Crypto to accelerate the > transparent encryption of HBase. new crypto provider with Commons CRYPTO will > be provided for Transparent encryption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590784#comment-15590784
]
Hadoop QA commented on HBASE-16870:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in branch-1.1 has 80 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_101.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 16s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 24s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_101.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 2s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 24s
{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 118m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | hadoop.hbase.replication.TestReplicationStatus |
| | hadoop.hbase.util.TestHBaseFsck |
| Timed out junit tests |
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.TestIOFencing |
| | org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:35e2245 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590769#comment-15590769
]
Hadoop QA commented on HBASE-16870:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 46s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 41s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 31s
{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 119m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:b2c5d84 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834304/HBASE-16870-branch-1.patch
|
| JIRA Issue | HBASE-16870 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
|
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590758#comment-15590758
]
Ted Yu commented on HBASE-16870:
Add license header to TestReplicationStatus
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-ADDENDUM.patch,
> HBASE-16870-branch-1.1.patch, HBASE-16870-branch-1.3.patch,
> HBASE-16870-branch-1.patch, HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16870:
---
Attachment: HBASE-16870-ADDENDUM.patch
ADDENDUM is for master branch. This will fix the failed
TestReplicationSmallTests. The unit tests about Replication passed on my local
computer.
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-ADDENDUM.patch,
> HBASE-16870-branch-1.1.patch, HBASE-16870-branch-1.3.patch,
> HBASE-16870-branch-1.patch, HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true get no result.
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590729#comment-15590729 ] huzheng commented on HBASE-16886: - So small = true is not set, things work as expected. --> That's right. U want to make a patch? --> With my pleasure. > hbase-client: scanner with reversed=true and small=true get no result. > --- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Reporter: huzheng > Attachments: TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[
https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590727#comment-15590727
]
Ted Yu commented on HBASE-16827:
{code}
message BackupManifest {
- required string version = 1;
- required string backup_id = 2;
- required BackupType type = 3;
+ //optional string version = 1;
+ //optional string backup_id = 2;
+ //optional BackupType type = 3;
{code}
Please drop commented out fields.
> Backup.proto refactoring
>
>
> Key: HBASE-16827
> URL: https://issues.apache.org/jira/browse/HBASE-16827
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch
>
>
> Make server_name type ServerName (from HBase.proto)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true get no result.
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590722#comment-15590722 ] Anoop Sam John commented on HBASE-16886: BTW in general creating a small scan with out setting start and stop rows looks odd.. But any way when one want to do reverse small scan from end of the table ie. Start row will be empty byte array,This problem will come in. Just saying. > hbase-client: scanner with reversed=true and small=true get no result. > --- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Reporter: huzheng > Attachments: TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true get no result.
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590714#comment-15590714 ] Anoop Sam John commented on HBASE-16886: So small = true is not set, things work as expected. U want to make a patch? > hbase-client: scanner with reversed=true and small=true get no result. > --- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Reporter: huzheng > Attachments: TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16885) Deprecate public classes which are removed in 2.0
[
https://issues.apache.org/jira/browse/HBASE-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590716#comment-15590716
]
Ted Yu commented on HBASE-16885:
StoreFile is @InterfaceAudience.LimitedPrivate("Coprocessor")
> Deprecate public classes which are removed in 2.0
> -
>
> Key: HBASE-16885
> URL: https://issues.apache.org/jira/browse/HBASE-16885
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Attachments: phoenix-compile-against-2.0.txt
>
>
> I tried to compile master branch of Phoenix against 2.0-SNAPSHOT and observed
> some compilation errors.
> /phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java:[31,54]
> cannot find symbol
> symbol: class Reader
> location: class org.apache.hadoop.hbase.regionserver.StoreFile
> /phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/MultiHfileOutputFormat.java:[300,16]
> cannot find symbol
> symbol: class Writer
> location: class org.apache.hadoop.hbase.regionserver.StoreFile
> The above inner classes of StoreFile are marked public as of branch-1.
> We should deprecate them in 1.x release.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16885) Deprecate public classes which are removed in 2.0
[ https://issues.apache.org/jira/browse/HBASE-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590702#comment-15590702 ] Anoop Sam John commented on HBASE-16885: bq.The above inner classes of StoreFile are marked public as of branch-1. Public or CP limited private? Removing public exposed classes with out deprecation cycle may not be acceptable! > Deprecate public classes which are removed in 2.0 > - > > Key: HBASE-16885 > URL: https://issues.apache.org/jira/browse/HBASE-16885 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Attachments: phoenix-compile-against-2.0.txt > > > I tried to compile master branch of Phoenix against 2.0-SNAPSHOT and observed > some compilation errors. > /phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java:[31,54] > cannot find symbol > symbol: class Reader > location: class org.apache.hadoop.hbase.regionserver.StoreFile > /phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/MultiHfileOutputFormat.java:[300,16] > cannot find symbol > symbol: class Writer > location: class org.apache.hadoop.hbase.regionserver.StoreFile > The above inner classes of StoreFile are marked public as of branch-1. > We should deprecate them in 1.x release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true get no result.
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590690#comment-15590690 ] huzheng commented on HBASE-16886: - Cause: when using a Reversed Small Scanner without setting startRow (scan.getStartRow()==null or scan.getStartRow() is EMPTY_BYTE_ARRAY), the current hbase-client code will locate to the first region (for above test case, we'll be located to (-oo, a]), but actually we should locate the last region [d, +oo). > hbase-client: scanner with reversed=true and small=true get no result. > --- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Reporter: huzheng > Attachments: TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[
https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590658#comment-15590658
]
Hadoop QA commented on HBASE-16827:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16827 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834312/HBASE-16827-v2.patch |
| JIRA Issue | HBASE-16827 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4113/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Backup.proto refactoring
>
>
> Key: HBASE-16827
> URL: https://issues.apache.org/jira/browse/HBASE-16827
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch
>
>
> Make server_name type ServerName (from HBase.proto)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16886) hbase-client: scanner with reversed=true and small=true get no result.
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huzheng updated HBASE-16886: Attachment: TestReversedSmallScan.java > hbase-client: scanner with reversed=true and small=true get no result. > --- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Reporter: huzheng > Attachments: TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16886) hbase-client: scanner with reversed=true and small=true get no result.
huzheng created HBASE-16886: --- Summary: hbase-client: scanner with reversed=true and small=true get no result. Key: HBASE-16886 URL: https://issues.apache.org/jira/browse/HBASE-16886 Project: HBase Issue Type: Bug Reporter: huzheng Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all rowKeys are located in region [d, +oo). using a Reversed Small Scanner will get no result. Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16827: -- Attachment: HBASE-16827-v2.patch v2 addresses other RB comments. cc:[~tedyu] > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch > > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590645#comment-15590645
]
Hudson commented on HBASE-16870:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1818 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1818/])
HBASE-16870 Add the metrics of replication sources which were (tedyu: rev
674511875d513ca3c031e63987288c45dacf56d9)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationLoad.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-branch-1.1.patch,
> HBASE-16870-branch-1.3.patch, HBASE-16870-branch-1.patch,
> HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit
[ https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590644#comment-15590644 ] Hudson commented on HBASE-16752: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1818 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1818/]) HBASE-16752 Return error back to rpc client on exceeding rpc request (garyh: rev ee1123b0696e290fe406e3b7de30052f6dd06ba7) * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/AbstractTestIPC.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java > Upgrading from 1.2 to 1.3 can lead to replication failures due to difference > in RPC size limit > -- > > Key: HBASE-16752 > URL: https://issues.apache.org/jira/browse/HBASE-16752 > Project: HBase > Issue Type: Bug > Components: Replication, rpc >Affects Versions: 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, > HBASE-16752.addendum.patch > > > In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by > default to 256 MB. This means that during upgrade scenarios (or when source > is 1.2 peer is already on 1.3), it's possible to encounter a situation where > we try to send an rpc with size greater than 256 MB because we never unroll a > WALEdit while sending replication traffic. > RpcServer throws the underlying exception locally, but closes the connection > with returning the underlying error to the client, and client only sees a > "Broken pipe" error. > I am not sure what is the proper fix here (or if one is needed) to make sure > this does not happen, but we should return the underlying exception to the > RpcClient, because without it, it can be difficult to diagnose the problem, > especially for someone new to HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase
[
https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590640#comment-15590640
]
Ted Yu commented on HBASE-16845:
Looking at http://hbase.apache.org/book.html#integration.tests , I only see
{{-PskipIntegrationTests}} being described.
> Run tests under hbase-spark module in test phase
>
>
> Key: HBASE-16845
> URL: https://issues.apache.org/jira/browse/HBASE-16845
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16845.v1.txt, 16845.v2.txt
>
>
> [~appy] reported over in the discussion thread titled 'Testing and CI' that
> tests under hbase-spark module are not run by QA bot.
> This issue is to run the unit tests in test phase so that we detect build
> breakage early.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16884) Add HBase-2.0.x to the hadoop version support matrix in our documentation
[ https://issues.apache.org/jira/browse/HBASE-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590619#comment-15590619 ] stack commented on HBASE-16884: --- LGTM +1 > Add HBase-2.0.x to the hadoop version support matrix in our documentation > - > > Key: HBASE-16884 > URL: https://issues.apache.org/jira/browse/HBASE-16884 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16884.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590613#comment-15590613
]
stack commented on HBASE-16698:
---
+1 on push to branch-1. Let me know if you want me to do it for you [~carp84],
at your service. [~mantonov] You want this back in 1.3? Can be off by default?
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16885) Deprecate public classes which are removed in 2.0
[ https://issues.apache.org/jira/browse/HBASE-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16885: --- Attachment: phoenix-compile-against-2.0.txt File containing compilation errors. > Deprecate public classes which are removed in 2.0 > - > > Key: HBASE-16885 > URL: https://issues.apache.org/jira/browse/HBASE-16885 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Attachments: phoenix-compile-against-2.0.txt > > > I tried to compile master branch of Phoenix against 2.0-SNAPSHOT and observed > some compilation errors. > /phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java:[31,54] > cannot find symbol > symbol: class Reader > location: class org.apache.hadoop.hbase.regionserver.StoreFile > /phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/MultiHfileOutputFormat.java:[300,16] > cannot find symbol > symbol: class Writer > location: class org.apache.hadoop.hbase.regionserver.StoreFile > The above inner classes of StoreFile are marked public as of branch-1. > We should deprecate them in 1.x release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590606#comment-15590606 ] Duo Zhang commented on HBASE-16414: --- [~ram_krish] Let's move on? I think two +1s is enough. And we can revert the commit if there are objections later. > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch, > HBASE-16414.003.patch, HBASE-16414.004.patch, HBASE-16414.005.patch, > HBASE-16414.006.patch, HBASE-16414.007.patch, HBASE-16414.008.patch, > HBASE-16414.009.patch, HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16885) Deprecate public classes which are removed in 2.0
Ted Yu created HBASE-16885: -- Summary: Deprecate public classes which are removed in 2.0 Key: HBASE-16885 URL: https://issues.apache.org/jira/browse/HBASE-16885 Project: HBase Issue Type: Bug Reporter: Ted Yu I tried to compile master branch of Phoenix against 2.0-SNAPSHOT and observed some compilation errors. /phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexStoreFileScanner.java:[31,54] cannot find symbol symbol: class Reader location: class org.apache.hadoop.hbase.regionserver.StoreFile /phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/MultiHfileOutputFormat.java:[300,16] cannot find symbol symbol: class Writer location: class org.apache.hadoop.hbase.regionserver.StoreFile The above inner classes of StoreFile are marked public as of branch-1. We should deprecate them in 1.x release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16867) Check ACLs for remote HBaseLock
[ https://issues.apache.org/jira/browse/HBASE-16867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16867: - Attachment: HBASE-16867.master.003.patch > Check ACLs for remote HBaseLock > --- > > Key: HBASE-16867 > URL: https://issues.apache.org/jira/browse/HBASE-16867 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16867.master.001.patch, > HBASE-16867.master.002.patch, HBASE-16867.master.003.patch > > > HBaseLock was added in HBASE-16744 to allow clients to take locks on > namespace/table/regions. Check acls for the 2 rpcs that were added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[ https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16744: - Attachment: HBASE-16744.master.011.patch > Procedure V2 - Lock procedures to allow clients to acquire locks on > tables/namespaces/regions > - > > Key: HBASE-16744 > URL: https://issues.apache.org/jira/browse/HBASE-16744 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16744.master.001.patch, > HBASE-16744.master.002.patch, HBASE-16744.master.003.patch, > HBASE-16744.master.004.patch, HBASE-16744.master.005.patch, > HBASE-16744.master.006.patch, HBASE-16744.master.007.patch, > HBASE-16744.master.008.patch, HBASE-16744.master.009.patch, > HBASE-16744.master.010.patch, HBASE-16744.master.011.patch > > > Will help us get rid of ZK locks. > Will be useful for external tools like hbck, future backup manager, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit
[ https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590572#comment-15590572 ] Hudson commented on HBASE-16752: FAILURE: Integrated in Jenkins build HBase-1.4 #479 (See [https://builds.apache.org/job/HBase-1.4/479/]) HBASE-16752 Return error back to rpc client on exceeding rpc request (garyh: rev 33e89fa9cfd4036fe972ff892df957603ae25bbf) * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/AbstractTestIPC.java > Upgrading from 1.2 to 1.3 can lead to replication failures due to difference > in RPC size limit > -- > > Key: HBASE-16752 > URL: https://issues.apache.org/jira/browse/HBASE-16752 > Project: HBase > Issue Type: Bug > Components: Replication, rpc >Affects Versions: 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, > HBASE-16752.addendum.patch > > > In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by > default to 256 MB. This means that during upgrade scenarios (or when source > is 1.2 peer is already on 1.3), it's possible to encounter a situation where > we try to send an rpc with size greater than 256 MB because we never unroll a > WALEdit while sending replication traffic. > RpcServer throws the underlying exception locally, but closes the connection > with returning the underlying error to the client, and client only sees a > "Broken pipe" error. > I am not sure what is the proper fix here (or if one is needed) to make sure > this does not happen, but we should return the underlying exception to the > RpcClient, because without it, it can be difficult to diagnose the problem, > especially for someone new to HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[ https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590571#comment-15590571 ] Hudson commented on HBASE-16345: FAILURE: Integrated in Jenkins build HBase-1.4 #479 (See [https://builds.apache.org/job/HBase-1.4/479/]) HBASE-16345 RpcRetryingCallerWithReadReplicas#call() should catch some (esteban: rev a97aef51635539ea382699495613ebe1bf89e475) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallableWithReplicas.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicaWithCluster.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultBoundedCompletionService.java > RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer > Exceptions > -- > > Key: HBASE-16345 > URL: https://issues.apache.org/jira/browse/HBASE-16345 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.2.3, 1.1.7 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16345-v001.patch, HBASE-16345.branch-1.001.patch, > HBASE-16345.branch-1.001.patch, HBASE-16345.master.001.patch, > HBASE-16345.master.002.patch, HBASE-16345.master.003.patch, > HBASE-16345.master.004.patch, HBASE-16345.master.005.patch, > HBASE-16345.master.005.patch, HBASE-16345.master.006.patch > > > Update for the description. Debugged more at this front based on the comments > from Enis. > The cause is that for the primary replica, if its retry is exhausted too > fast, f.get() [1] returns ExecutionException. This Exception needs to be > ignored and continue with the replicas. > The other issue is that after adding calls for the replicas, if the first > completed task gets ExecutionException (due to the retry exhausted), it > throws the exception to the client[2]. > In this case, it needs to loop through these tasks, waiting for the success > one. If no one succeeds, throw exception. > Similar for the scan as well > [1] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197 > [2] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16870:
---
Attachment: HBASE-16870-branch-1.1.patch
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-branch-1.1.patch,
> HBASE-16870-branch-1.3.patch, HBASE-16870-branch-1.patch,
> HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590568#comment-15590568
]
Guanghao Zhang commented on HBASE-16870:
branch-1.3 patch can be applied to branch-1.2, too. Thanks.
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-branch-1.1.patch,
> HBASE-16870-branch-1.3.patch, HBASE-16870-branch-1.patch,
> HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[ https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-16345: -- Fix Version/s: 1.4.0 2.0.0 > RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer > Exceptions > -- > > Key: HBASE-16345 > URL: https://issues.apache.org/jira/browse/HBASE-16345 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.2.3, 1.1.7 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16345-v001.patch, HBASE-16345.branch-1.001.patch, > HBASE-16345.branch-1.001.patch, HBASE-16345.master.001.patch, > HBASE-16345.master.002.patch, HBASE-16345.master.003.patch, > HBASE-16345.master.004.patch, HBASE-16345.master.005.patch, > HBASE-16345.master.005.patch, HBASE-16345.master.006.patch > > > Update for the description. Debugged more at this front based on the comments > from Enis. > The cause is that for the primary replica, if its retry is exhausted too > fast, f.get() [1] returns ExecutionException. This Exception needs to be > ignored and continue with the replicas. > The other issue is that after adding calls for the replicas, if the first > completed task gets ExecutionException (due to the retry exhausted), it > throws the exception to the client[2]. > In this case, it needs to loop through these tasks, waiting for the success > one. If no one succeeds, throw exception. > Similar for the scan as well > [1] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197 > [2] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16870:
---
Attachment: HBASE-16870-branch-1.3.patch
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-branch-1.3.patch,
> HBASE-16870-branch-1.patch, HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16884) Add HBase-2.0.x to the hadoop version support matrix in our documentation
[ https://issues.apache.org/jira/browse/HBASE-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16884: -- Attachment: HBASE-16884.patch > Add HBase-2.0.x to the hadoop version support matrix in our documentation > - > > Key: HBASE-16884 > URL: https://issues.apache.org/jira/browse/HBASE-16884 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16884.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16884) Add HBase-2.0.x to the hadoop version support matrix in our documentation
[ https://issues.apache.org/jira/browse/HBASE-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16884: -- Status: Patch Available (was: Open) > Add HBase-2.0.x to the hadoop version support matrix in our documentation > - > > Key: HBASE-16884 > URL: https://issues.apache.org/jira/browse/HBASE-16884 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16884.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590555#comment-15590555
]
Yu Li commented on HBASE-16698:
---
[~stack] mind if I push this to branch-1(branch-1.4) first to avoid further
code rebase?
Since YCSB data indicates in both single region and multiple regions scenarios
performance is better w/ patch, I think this also applies for branch-1.2/1.3,
but will wait for your decisions on whether to let it in [~stack] [~mantonov]
Echo the YCSB data here for your convenience:
testing environment:
{noformat}
YCSB 0.7.0
4 physical client nodes, 8 YCSB processes per node, 32 threads per YCSB process
recordcount=3,200,000, fieldcount=1, fieldlength=1024, insertproportion=1,
requestdistribution=uniform
{noformat}
200 regions:
||TestCase||Round||Throughput||AverageLatency(us)||
|w/o patch|Round-1|66554.48|15263.36|
|w/ patch|Round-1|91472.48|11098.85|
|w/o patch|Round-2|66083.53|15382.01|
|w/ patch|Round-2|91420.26|11104.37|
single region:
||TestCase||Throughput||AverageLatency(us)||
|w/o patch|69924.42|14544.38|
|w patch|86373.70|11770.09|
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16870:
---
Attachment: HBASE-16870-branch-1.patch
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-branch-1.patch, HBASE-16870-v1.patch,
> HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16872) Implement mutateRow and checkAndMutate
[ https://issues.apache.org/jira/browse/HBASE-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16872: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~carp84] for reviewing. > Implement mutateRow and checkAndMutate > -- > > Key: HBASE-16872 > URL: https://issues.apache.org/jira/browse/HBASE-16872 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16872-v1.patch, HBASE-16872.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15306) Make RPC call queue length dynamically configurable
[ https://issues.apache.org/jira/browse/HBASE-15306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590465#comment-15590465 ] Appy commented on HBASE-15306: -- More specifically, TestSimpleRpcScheduler#testSoftAndHardQueueLimits starts but never finishes. > Make RPC call queue length dynamically configurable > --- > > Key: HBASE-15306 > URL: https://issues.apache.org/jira/browse/HBASE-15306 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15306-v1.patch, HBASE-15306-v2.patch, > HBASE-15306-v3.patch > > > Sometimes it's useful to be able to modify call queue size on prod cluster > without having to do a rolling restart. > Here setting fixed hard limit on call queue size and soft limit which is > dynamically-updatable. > https://reviews.facebook.net/D54579 for review, will test more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16884) Add HBase-2.0.x to the hadoop version support matrix in our documentation
Duo Zhang created HBASE-16884: - Summary: Add HBase-2.0.x to the hadoop version support matrix in our documentation Key: HBASE-16884 URL: https://issues.apache.org/jira/browse/HBASE-16884 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 2.0.0 Reporter: Duo Zhang Assignee: Duo Zhang Priority: Blocker Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15306) Make RPC call queue length dynamically configurable
[ https://issues.apache.org/jira/browse/HBASE-15306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590455#comment-15590455 ] Appy commented on HBASE-15306: -- It seems like this test hung up and failed the build with "There was a timeout or other error in the fork". :https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/1807/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler-output.txt/*view*/ [~mantonov] ptal. Thanks. > Make RPC call queue length dynamically configurable > --- > > Key: HBASE-15306 > URL: https://issues.apache.org/jira/browse/HBASE-15306 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15306-v1.patch, HBASE-15306-v2.patch, > HBASE-15306-v3.patch > > > Sometimes it's useful to be able to modify call queue size on prod cluster > without having to do a rolling restart. > Here setting fixed hard limit on call queue size and soft limit which is > dynamically-updatable. > https://reviews.facebook.net/D54579 for review, will test more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit
[ https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-16752: -- Attachment: HBASE-16752.addendum.patch Addendum: The original patch introduces an incompatibility in the sense that when the client version is 1.3, the client bails out immediately on receiving the excpetion, but a 1.2 client would keep retrying as it would unwrap the exception only to a RemoteException because it does not know about the RequestTooBigException. The addendum adds a min supported version for the RequestTooBigException and throws a DoNotRetryIOException for older clients. > Upgrading from 1.2 to 1.3 can lead to replication failures due to difference > in RPC size limit > -- > > Key: HBASE-16752 > URL: https://issues.apache.org/jira/browse/HBASE-16752 > Project: HBase > Issue Type: Bug > Components: Replication, rpc >Affects Versions: 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, > HBASE-16752.addendum.patch > > > In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by > default to 256 MB. This means that during upgrade scenarios (or when source > is 1.2 peer is already on 1.3), it's possible to encounter a situation where > we try to send an rpc with size greater than 256 MB because we never unroll a > WALEdit while sending replication traffic. > RpcServer throws the underlying exception locally, but closes the connection > with returning the underlying error to the client, and client only sees a > "Broken pipe" error. > I am not sure what is the proper fix here (or if one is needed) to make sure > this does not happen, but we should return the underlying exception to the > RpcClient, because without it, it can be difficult to diagnose the problem, > especially for someone new to HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16754) Regions failing compaction due to referencing non-existent store file
[
https://issues.apache.org/jira/browse/HBASE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590300#comment-15590300
]
Hudson commented on HBASE-16754:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #46 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/46/])
HBASE-16754 All WALSplitter OutputSinks should process compaction events
(garyh: rev cbdb84f2d33be614c0ee4775d3ea86bc15608e03)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
* (add) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java
> Regions failing compaction due to referencing non-existent store file
> -
>
> Key: HBASE-16754
> URL: https://issues.apache.org/jira/browse/HBASE-16754
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.3
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: HBASE-16754.001.patch, HBASE-16754.branch-1.001.patch,
> HBASE-16754.branch-1.2.001.patch
>
>
> Running a mixed read write workload on a recent build off branch-1.3, we are
> seeing compactions occasionally fail with errors like the following (actual
> filenames replaced with placeholders):
> {noformat}
> 16/09/27 16:57:28 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 116
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
>
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
>
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:498)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:568)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/09/27 17:01:31 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 115
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
>
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590277#comment-15590277
]
Ted Yu commented on HBASE-16870:
Though the patch applies on branch-1, TestReplicationSmallTests keeps failing.
Guanghao
Can you check ?
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[ https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590208#comment-15590208 ] Hudson commented on HBASE-16345: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1817 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1817/]) HBASE-16345 RpcRetryingCallerWithReadReplicas#call() should catch some (esteban: rev 72db95388664fb23314b2e6bb437b961c797f579) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallableWithReplicas.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicaWithCluster.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultBoundedCompletionService.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java > RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer > Exceptions > -- > > Key: HBASE-16345 > URL: https://issues.apache.org/jira/browse/HBASE-16345 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.2.3, 1.1.7 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16345-v001.patch, HBASE-16345.branch-1.001.patch, > HBASE-16345.branch-1.001.patch, HBASE-16345.master.001.patch, > HBASE-16345.master.002.patch, HBASE-16345.master.003.patch, > HBASE-16345.master.004.patch, HBASE-16345.master.005.patch, > HBASE-16345.master.005.patch, HBASE-16345.master.006.patch > > > Update for the description. Debugged more at this front based on the comments > from Enis. > The cause is that for the primary replica, if its retry is exhausted too > fast, f.get() [1] returns ExecutionException. This Exception needs to be > ignored and continue with the replicas. > The other issue is that after adding calls for the replicas, if the first > completed task gets ExecutionException (due to the retry exhausted), it > throws the exception to the client[2]. > In this case, it needs to loop through these tasks, waiting for the success > one. If no one succeeds, throw exception. > Similar for the scan as well > [1] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197 > [2] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16874) Fix TestMasterFailoverWithProcedures and ensure single proc-executor for kill/restart tests
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590206#comment-15590206
]
Hudson commented on HBASE-16874:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1817 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1817/])
HBASE-16874 Fix TestMasterFailoverWithProcedures and ensure single
(matteo.bertozzi: rev d9ee25e82a13d7df1ae097e0f3ae6d12dc9e94fa)
* (edit)
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/ProcedureTestingUtility.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterFailoverWithProcedures.java
> Fix TestMasterFailoverWithProcedures and ensure single proc-executor for
> kill/restart tests
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch, HBASE-16874-v1.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16754) Regions failing compaction due to referencing non-existent store file
[
https://issues.apache.org/jira/browse/HBASE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590197#comment-15590197
]
Hadoop QA commented on HBASE-16754:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 35s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 6s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 114m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
|
| | hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat |
| | hadoop.hbase.coprocessor.TestRowProcessorEndpoint |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpointNoMaster
|
| |
org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager
|
| |
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | org.apache.hadoop.hbase.replication.TestMasterReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Updated] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit
[ https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-16752: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 1.3.0 2.0.0 Status: Resolved (was: Patch Available) Committed to branch-1.3+. Thanks for the patch, Ashu! > Upgrading from 1.2 to 1.3 can lead to replication failures due to difference > in RPC size limit > -- > > Key: HBASE-16752 > URL: https://issues.apache.org/jira/browse/HBASE-16752 > Project: HBase > Issue Type: Bug > Components: Replication, rpc >Affects Versions: 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch > > > In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by > default to 256 MB. This means that during upgrade scenarios (or when source > is 1.2 peer is already on 1.3), it's possible to encounter a situation where > we try to send an rpc with size greater than 256 MB because we never unroll a > WALEdit while sending replication traffic. > RpcServer throws the underlying exception locally, but closes the connection > with returning the underlying error to the client, and client only sees a > "Broken pipe" error. > I am not sure what is the proper fix here (or if one is needed) to make sure > this does not happen, but we should return the underlying exception to the > RpcClient, because without it, it can be difficult to diagnose the problem, > especially for someone new to HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16883) Fix javadoc warnings for mega patch
[ https://issues.apache.org/jira/browse/HBASE-16883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16883: --- Attachment: 16883.v1.txt > Fix javadoc warnings for mega patch > --- > > Key: HBASE-16883 > URL: https://issues.apache.org/jira/browse/HBASE-16883 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Labels: backup > Fix For: HBASE-7912 > > Attachments: 16883.v1.txt > > > There were 14 javadoc warnings for mega patch v29. > This task fixes the warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16883) Fix javadoc warnings for mega patch
Ted Yu created HBASE-16883: -- Summary: Fix javadoc warnings for mega patch Key: HBASE-16883 URL: https://issues.apache.org/jira/browse/HBASE-16883 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor There were 14 javadoc warnings for mega patch v29. This task fixes the warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16870:
---
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16871) Procedure v2 - add waiting procs back to the queue after restart
[ https://issues.apache.org/jira/browse/HBASE-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590012#comment-15590012 ] Appy commented on HBASE-16871: -- +1 > Procedure v2 - add waiting procs back to the queue after restart > > > Key: HBASE-16871 > URL: https://issues.apache.org/jira/browse/HBASE-16871 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16871-v0.patch, HBASE-16871-v1.patch, > HBASE-16871-v1.patch > > > Procs in WAITING_TIMEOUT state don't get re-added to the queue after restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16754) Regions failing compaction due to referencing non-existent store file
[
https://issues.apache.org/jira/browse/HBASE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589972#comment-15589972
]
Hudson commented on HBASE-16754:
FAILURE: Integrated in Jenkins build HBase-1.4 #478 (See
[https://builds.apache.org/job/HBase-1.4/478/])
HBASE-16754 All WALSplitter OutputSinks should process compaction events
(garyh: rev ae151334ba5c20cb2ad4017a25d52aff8bd3994b)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
* (add) hba
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
> Regions failing compaction due to referencing non-existent store file
> -
>
> Key: HBASE-16754
> URL: https://issues.apache.org/jira/browse/HBASE-16754
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.3
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: HBASE-16754.001.patch, HBASE-16754.branch-1.001.patch,
> HBASE-16754.branch-1.2.001.patch
>
>
> Running a mixed read write workload on a recent build off branch-1.3, we are
> seeing compactions occasionally fail with errors like the following (actual
> filenames replaced with placeholders):
> {noformat}
> 16/09/27 16:57:28 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 116
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
>
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
>
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:498)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:568)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/09/27 17:01:31 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 115
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
>
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16870:
---
Summary: Add the metrics of replication sources which were transformed from
other dead rs to ReplicationLoad (was: Add the metrics of replication sources
which were transformed from other died rs to ReplicationLoad)
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other died rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589958#comment-15589958
]
Hadoop QA commented on HBASE-16870:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 15s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 44s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
| | org.apache.hadoop.hbase.filter.TestFilterListOrOperatorWithBlkCnt |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834124/HBASE-16870-v1.patch |
| JIRA Issue | HBASE-16870 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 499de54adf76 3.13.0-98-generic #145-Ubuntu SMP Sat Oct 8
20:13:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 82a2384 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4107/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4107/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
[jira] [Commented] (HBASE-16754) Regions failing compaction due to referencing non-existent store file
[
https://issues.apache.org/jira/browse/HBASE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589942#comment-15589942
]
Gary Helmling commented on HBASE-16754:
---
[~yangzhe1991], yes, I pulled in TestWALSplit and TestWALSplitCompressed for my
commits to branch-1 and branch-1.3. TestWALSplit required some further changes
for branch-1.2, so I've attached a new patch here to get a full test run. Feel
free to take a look as well.
> Regions failing compaction due to referencing non-existent store file
> -
>
> Key: HBASE-16754
> URL: https://issues.apache.org/jira/browse/HBASE-16754
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.3
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: HBASE-16754.001.patch, HBASE-16754.branch-1.001.patch,
> HBASE-16754.branch-1.2.001.patch
>
>
> Running a mixed read write workload on a recent build off branch-1.3, we are
> seeing compactions occasionally fail with errors like the following (actual
> filenames replaced with placeholders):
> {noformat}
> 16/09/27 16:57:28 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 116
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
>
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
>
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:498)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:568)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/09/27 17:01:31 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 115
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
>
[jira] [Updated] (HBASE-16754) Regions failing compaction due to referencing non-existent store file
[
https://issues.apache.org/jira/browse/HBASE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-16754:
--
Attachment: HBASE-16754.branch-1.2.001.patch
Attaching a patch against branch-1.2. There are enough differences between
branch-1.2 and branch-1.3 that I'd like to get a full test run.
> Regions failing compaction due to referencing non-existent store file
> -
>
> Key: HBASE-16754
> URL: https://issues.apache.org/jira/browse/HBASE-16754
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.3
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: HBASE-16754.001.patch, HBASE-16754.branch-1.001.patch,
> HBASE-16754.branch-1.2.001.patch
>
>
> Running a mixed read write workload on a recent build off branch-1.3, we are
> seeing compactions occasionally fail with errors like the following (actual
> filenames replaced with placeholders):
> {noformat}
> 16/09/27 16:57:28 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 116
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
>
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
>
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:498)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:568)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/09/27 17:01:31 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 115
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
>
[jira] [Commented] (HBASE-16754) Regions failing compaction due to referencing non-existent store file
[
https://issues.apache.org/jira/browse/HBASE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589901#comment-15589901
]
Gary Helmling commented on HBASE-16754:
---
Committed to branch-1 and branch-1.3, pulling back in TestWALSplit and
TestWALSplitCompressed.
> Regions failing compaction due to referencing non-existent store file
> -
>
> Key: HBASE-16754
> URL: https://issues.apache.org/jira/browse/HBASE-16754
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.3
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 1.3.0
>
> Attachments: HBASE-16754.001.patch, HBASE-16754.branch-1.001.patch
>
>
> Running a mixed read write workload on a recent build off branch-1.3, we are
> seeing compactions occasionally fail with errors like the following (actual
> filenames replaced with placeholders):
> {noformat}
> 16/09/27 16:57:28 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 116
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
>
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
>
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:498)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:568)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/09/27 17:01:31 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 115
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
>
[jira] [Created] (HBASE-16882) Expose the number of open scanners on region server via metics
Jerry He created HBASE-16882: Summary: Expose the number of open scanners on region server via metics Key: HBASE-16882 URL: https://issues.apache.org/jira/browse/HBASE-16882 Project: HBase Issue Type: Improvement Reporter: Jerry He Assignee: Jerry He Priority: Minor Metrics on the number of open scanners and their live span on the region server are useful in doing server performance diagnostics, and are useful in doing client application diagnostics, profiling as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16881) Address findbugs warnings for backup mega patch
[ https://issues.apache.org/jira/browse/HBASE-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16881: --- Summary: Address findbugs warnings for backup mega patch (was: Address findbugs warnings) > Address findbugs warnings for backup mega patch > --- > > Key: HBASE-16881 > URL: https://issues.apache.org/jira/browse/HBASE-16881 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Labels: backup > Fix For: HBASE-7912 > > Attachments: 16881.v1.txt > > > Multiple findbugs warnings were raised for mega patch v29. > Largely this was due to fixes for findbugs warnings in mega patch v28 were > dropped. > This issue addresses the findbugs warnings for mega patch v29. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16881) Address findbugs warnings for backup mega patch
[ https://issues.apache.org/jira/browse/HBASE-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-16881. Resolution: Fixed Hadoop Flags: Reviewed Thanks, Vlad. > Address findbugs warnings for backup mega patch > --- > > Key: HBASE-16881 > URL: https://issues.apache.org/jira/browse/HBASE-16881 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Labels: backup > Fix For: HBASE-7912 > > Attachments: 16881.v1.txt > > > Multiple findbugs warnings were raised for mega patch v29. > Largely this was due to fixes for findbugs warnings in mega patch v28 were > dropped. > This issue addresses the findbugs warnings for mega patch v29. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16881) Address findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589780#comment-15589780
]
Ted Yu commented on HBASE-16881:
All backup tests passed.
{code}
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 28:56 min
[INFO] Finished at: 2016-10-19T20:32:47+00:00
{code}
> Address findbugs warnings
> -
>
> Key: HBASE-16881
> URL: https://issues.apache.org/jira/browse/HBASE-16881
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Fix For: HBASE-7912
>
> Attachments: 16881.v1.txt
>
>
> Multiple findbugs warnings were raised for mega patch v29.
> Largely this was due to fixes for findbugs warnings in mega patch v28 were
> dropped.
> This issue addresses the findbugs warnings for mega patch v29.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[ https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-16345: - Affects Version/s: (was: 1.3.1) 1.4.0 1.3.0 1.0.3 1.2.3 1.1.7 > RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer > Exceptions > -- > > Key: HBASE-16345 > URL: https://issues.apache.org/jira/browse/HBASE-16345 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.2.3, 1.1.7 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16345-v001.patch, HBASE-16345.branch-1.001.patch, > HBASE-16345.branch-1.001.patch, HBASE-16345.master.001.patch, > HBASE-16345.master.002.patch, HBASE-16345.master.003.patch, > HBASE-16345.master.004.patch, HBASE-16345.master.005.patch, > HBASE-16345.master.005.patch, HBASE-16345.master.006.patch > > > Update for the description. Debugged more at this front based on the comments > from Enis. > The cause is that for the primary replica, if its retry is exhausted too > fast, f.get() [1] returns ExecutionException. This Exception needs to be > ignored and continue with the replicas. > The other issue is that after adding calls for the replicas, if the first > completed task gets ExecutionException (due to the retry exhausted), it > throws the exception to the client[2]. > In this case, it needs to loop through these tasks, waiting for the success > one. If no one succeeds, throw exception. > Similar for the scan as well > [1] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197 > [2] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16881) Address findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589765#comment-15589765 ] Vladimir Rodionov commented on HBASE-16881: --- I am OK as long as all tests pass. > Address findbugs warnings > - > > Key: HBASE-16881 > URL: https://issues.apache.org/jira/browse/HBASE-16881 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Labels: backup > Fix For: HBASE-7912 > > Attachments: 16881.v1.txt > > > Multiple findbugs warnings were raised for mega patch v29. > Largely this was due to fixes for findbugs warnings in mega patch v28 were > dropped. > This issue addresses the findbugs warnings for mega patch v29. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16874) Fix TestMasterFailoverWithProcedures and ensure single proc-executor for kill/restart tests
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-16874:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix TestMasterFailoverWithProcedures and ensure single proc-executor for
> kill/restart tests
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch, HBASE-16874-v1.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-16874:
Issue Type: Sub-task (was: Bug)
Parent: HBASE-16617
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch, HBASE-16874-v1.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16874) Fix TestMasterFailoverWithProcedures and ensure single proc-executor for kill/restart tests
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-16874:
Summary: Fix TestMasterFailoverWithProcedures and ensure single
proc-executor for kill/restart tests (was: Potential NPE from
ProcedureExecutor#stop())
> Fix TestMasterFailoverWithProcedures and ensure single proc-executor for
> kill/restart tests
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch, HBASE-16874-v1.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[ https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-16345: - Affects Version/s: 1.3.1 > RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer > Exceptions > -- > > Key: HBASE-16345 > URL: https://issues.apache.org/jira/browse/HBASE-16345 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.3.1 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16345-v001.patch, HBASE-16345.branch-1.001.patch, > HBASE-16345.branch-1.001.patch, HBASE-16345.master.001.patch, > HBASE-16345.master.002.patch, HBASE-16345.master.003.patch, > HBASE-16345.master.004.patch, HBASE-16345.master.005.patch, > HBASE-16345.master.005.patch, HBASE-16345.master.006.patch > > > Update for the description. Debugged more at this front based on the comments > from Enis. > The cause is that for the primary replica, if its retry is exhausted too > fast, f.get() [1] returns ExecutionException. This Exception needs to be > ignored and continue with the replicas. > The other issue is that after adding calls for the replicas, if the first > completed task gets ExecutionException (due to the retry exhausted), it > throws the exception to the client[2]. > In this case, it needs to loop through these tasks, waiting for the success > one. If no one succeeds, throw exception. > Similar for the scan as well > [1] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197 > [2] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16871) Procedure v2 - add waiting procs back to the queue after restart
[ https://issues.apache.org/jira/browse/HBASE-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16871: Attachment: HBASE-16871-v1.patch > Procedure v2 - add waiting procs back to the queue after restart > > > Key: HBASE-16871 > URL: https://issues.apache.org/jira/browse/HBASE-16871 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16871-v0.patch, HBASE-16871-v1.patch, > HBASE-16871-v1.patch > > > Procs in WAITING_TIMEOUT state don't get re-added to the queue after restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16880) Correct the javadoc/behaviour of the APIs in ByteBufferUtils
[
https://issues.apache.org/jira/browse/HBASE-16880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589032#comment-15589032
]
Anoop Sam John commented on HBASE-16880:
Am +1 to change the code so as to make it behave like ByteBuffer put API. ie
advance pos in the 'in' buf also and correct the javadoc.
> Correct the javadoc/behaviour of the APIs in ByteBufferUtils
>
>
> Key: HBASE-16880
> URL: https://issues.apache.org/jira/browse/HBASE-16880
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>
> There are some issues either with the javadoc or the actual behaviour of some
> APIs in BBUtils.
> For eg,
> BBUtil#copyFromBufferToBuffer() says
> {code}
> /**
>* Copy one buffer's whole data to another. Write starts at the current
> position of 'out' buffer.
>* Note : This will advance the position marker of {@code out} but not
> change the position maker
>* for {@code in}. The position and limit of the {@code in} buffer to be
> set properly by caller.
>* @param in source buffer
>* @param out destination buffer
> {code}
> But this is true in case of UNSAFE.
> {code}
> if (UNSAFE_AVAIL) {
> int length = in.remaining();
> UnsafeAccess.copy(in, in.position(), out, out.position(), length);
> out.position(out.position() + length);
> } else {
> out.put(in);
> }
> {code}
> But in other case where we move the else - the 'in' is also advanced. So we
> need to either correct the behaviour or change the doc and see all the used
> places. This JIRA can be used to correct all the APIs in this util class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16872) Implement mutateRow and checkAndMutate
[
https://issues.apache.org/jira/browse/HBASE-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588971#comment-15588971
]
Hadoop QA commented on HBASE-16872:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 31s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834159/HBASE-16872-v1.patch |
| JIRA Issue | HBASE-16872 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3b98918fc7ac 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 67f1ac1 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4103/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
[jira] [Commented] (HBASE-16871) Procedure v2 - add waiting procs back to the queue after restart
[
https://issues.apache.org/jira/browse/HBASE-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588924#comment-15588924
]
Hadoop QA commented on HBASE-16871:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-16871 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834178/HBASE-16871-v1.patch |
| JIRA Issue | HBASE-16871 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4105/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Procedure v2 - add waiting procs back to the queue after restart
>
>
> Key: HBASE-16871
> URL: https://issues.apache.org/jira/browse/HBASE-16871
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-16871-v0.patch, HBASE-16871-v1.patch
>
>
> Procs in WAITING_TIMEOUT state don't get re-added to the queue after restart.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588909#comment-15588909 ] Anoop Sam John commented on HBASE-16608: Oh Ram reminded me that we dont keep refs to old merged out segments so ya the refs count increase is temp. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16871) Procedure v2 - add waiting procs back to the queue after restart
[ https://issues.apache.org/jira/browse/HBASE-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16871: Status: Patch Available (was: Open) > Procedure v2 - add waiting procs back to the queue after restart > > > Key: HBASE-16871 > URL: https://issues.apache.org/jira/browse/HBASE-16871 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16871-v0.patch, HBASE-16871-v1.patch > > > Procs in WAITING_TIMEOUT state don't get re-added to the queue after restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16871) Procedure v2 - add waiting procs back to the queue after restart
[ https://issues.apache.org/jira/browse/HBASE-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16871: Attachment: HBASE-16871-v1.patch > Procedure v2 - add waiting procs back to the queue after restart > > > Key: HBASE-16871 > URL: https://issues.apache.org/jira/browse/HBASE-16871 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16871-v0.patch, HBASE-16871-v1.patch > > > Procs in WAITING_TIMEOUT state don't get re-added to the queue after restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16864) Procedure v2 - Fix StateMachineProcedure support for child procs at last step
[ https://issues.apache.org/jira/browse/HBASE-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16864: Resolution: Fixed Status: Resolved (was: Patch Available) > Procedure v2 - Fix StateMachineProcedure support for child procs at last step > - > > Key: HBASE-16864 > URL: https://issues.apache.org/jira/browse/HBASE-16864 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16864-v0.patch, HBASE-16864-v1.patch > > > There is a bug in the StateMachineProcedure when we add child procs in the > last step. On recovery we end up spinning on the last step without ever > completing. the fix is to introduce an eof step so recovery knows that we are > already done once the all the children are terminated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-16874:
Attachment: HBASE-16874-v1.patch
v1 adds a check to prevent this situation in the future
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch, HBASE-16874-v1.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[
https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588623#comment-15588623
]
Hadoop QA commented on HBASE-15302:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 0s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 47s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 109m 30s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
| | org.apache.hadoop.hbase.TestHColumnDescriptorDefaultVersions |
| | org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:b2c5d84 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834133/HBASE-15302-branch-1-append-v1.patch
|
| JIRA Issue | HBASE-15302
[jira] [Updated] (HBASE-16872) Implement mutateRow and checkAndMutate
[ https://issues.apache.org/jira/browse/HBASE-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16872: -- Attachment: HBASE-16872-v1.patch Fix javadoc issues. > Implement mutateRow and checkAndMutate > -- > > Key: HBASE-16872 > URL: https://issues.apache.org/jira/browse/HBASE-16872 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16872-v1.patch, HBASE-16872.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588593#comment-15588593 ] ramkrishna.s.vasudevan commented on HBASE-16608: I think once the merge problem is solved with the CollectionBackedMSLAB which is simple and better way to manage segments the gC issue is gone and we get a better run and close to default Memstore performance. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588588#comment-15588588 ] Edward Bortnikov commented on HBASE-16608: -- Thanks again, [~anoop.hbase] and [~ram_krish] for helping us debug the GC issue. Agree that the current merge policy is naive. The number of ref copies and waste memory grows as arithmetic progression with the number of flushes. Let's try to find middle ground between the need to keep the number of segments small and the need to reduce the copy. Looking forward to that conversation. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16837) Implement checkAndPut and checkAndDelete
[ https://issues.apache.org/jira/browse/HBASE-16837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588539#comment-15588539 ] Hudson commented on HBASE-16837: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1814 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1814/]) HBASE-16837 Implement checkAndPut and checkAndDelete (zhangduo: rev acc606571bca4e82aec7136cff64bf6187dd71f7) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTable.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTable.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableImpl.java > Implement checkAndPut and checkAndDelete > > > Key: HBASE-16837 > URL: https://issues.apache.org/jira/browse/HBASE-16837 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16837.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16578) Mob data loss after mob compaction and normal compaction
[
https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588540#comment-15588540
]
Hudson commented on HBASE-16578:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1814 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1814/])
HBASE-16578 Mob data loss after mob compaction and normal compaction
(jingcheng.du: rev 67f1ac1f8e6c33e7917231ef2d541fe7461b4be4)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mob/compactions/TestMobCompactor.java
> Mob data loss after mob compaction and normal compaction
>
>
> Key: HBASE-16578
> URL: https://issues.apache.org/jira/browse/HBASE-16578
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-16578-V2.patch, HBASE-16578-V3.patch,
> HBASE-16578.patch, TestMobCompaction.java, TestMobCompaction.java
>
>
> StoreFileScanners on MOB cells rely on the scannerOrder to find the latest
> cells after mob compaction. The value of scannerOrder is assigned by the
> order of maxSeqId of StoreFile, and this maxSeqId is valued only after the
> reader of the StoreFile is created.
> In {{Compactor.compact}}, the compacted store files are cloned and their
> readers are not created. And in {{StoreFileScanner.getScannersForStoreFiles}}
> the StoreFiles are sorted before the readers are created and at that time the
> maxSeqId for each file is -1 (the default value). This will lead to a chaos
> in scanners in the following normal compaction. Some older cells might be
> chosen during the normal compaction.
> We need to create readers either before the sorting in the method
> {{StoreFileScanner.getScannersForStoreFiles}}, or create readers just after
> the store files are cloned in {{Compactor.compact}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588436#comment-15588436 ] Phil Yang edited comment on HBASE-15302 at 10/19/16 11:06 AM: -- TestAssignmentManager is also missed in branch-1/1.3. And should be added in the patch for branch-1.2 was (Author: yangzhe1991): TestAssignmentManager are also missed in branch-1/1.3. > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.4 > > Attachments: HBASE-15302-branch-1-append-v1.patch, > HBASE-15302-branch-1-v1.patch, HBASE-15302-branch-1.3-append-v1.patch, > HBASE-15302-branch-1.3-append-v1.patch, HBASE-15302-v1.txt, HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16872) Implement mutateRow and checkAndMutate
[
https://issues.apache.org/jira/browse/HBASE-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588437#comment-15588437
]
Yu Li commented on HBASE-16872:
---
Please fix below javadoc warnings as well as the comment mentioned above:
{noformat}
[WARNING]
/testptch/hbase/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTable.java:281:
warning - @param argument "rm" is not a parameter name.
[WARNING]
/testptch/hbase/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTable.java:296:
warning - @param argument "compareOp" is not a parameter name.
{noformat}
Other parts lgtm. +1
> Implement mutateRow and checkAndMutate
> --
>
> Key: HBASE-16872
> URL: https://issues.apache.org/jira/browse/HBASE-16872
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16872.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588436#comment-15588436 ] Phil Yang commented on HBASE-15302: --- TestAssignmentManager are also missed in branch-1/1.3. > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.4 > > Attachments: HBASE-15302-branch-1-append-v1.patch, > HBASE-15302-branch-1-v1.patch, HBASE-15302-branch-1.3-append-v1.patch, > HBASE-15302-branch-1.3-append-v1.patch, HBASE-15302-v1.txt, HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588008#comment-15588008 ] Anoop Sam John edited comment on HBASE-16608 at 10/19/16 10:56 AM: --- Another imp thing which came out now We elsewhere discussed abt the duplication of either cell data or index (In compaction or merge) case and Ram just mentioned it once again. Just case of merge which just copies the cell refs from 2+ arrays into one. It is not temp existance of duplciated refs and it is not just one copy. Let me explain with an eg: 2 segments in pipeline getting merged where each having 100 cells. So each having a cell array with 100 refs. (Each ref is 8 bytes) Now we make a merge and that create a new segment and that contain a cell array with 200 entries. Ya there are no copies of the cells. But new 200 cell refs coming up. So before 1st merge there were 200 cell refs totally and now it is 400. Now say one more segment came in and we merge together this new and old *merged* one. Consider new segment also have 100 cells. So before this merge op there are totally 400 + 100 = 500 refs 2nd merge also wont touch the segments and it creates a new segment and new array will contain 300 refs. So totally 800 refs. Actual cell objects are 300 and 2 merges made the refs# to be 800. So this not even doubling.. Like this if there are so many merge happens before a flush we will waste lots and lots of heap space with filling up refs. was (Author: anoop.hbase): Another imp thing which came out now We elsewhere discussed abt the duplication of either cell data or index (In compaction or merge) case and Ram just mentioned it once again. Just case of merge which just copies the cell refs from 2+ arrays into one. It is not temp existance of duplciated refs and it is not just one copy. Let me explain with an eg: 2 segments in pipeline getting merged where each having 100 cells. So each having a cell array with 100 refs. (Each ref is 8 bytes) Now we make a merge and that create a new segment and that contain a cell array with 200 entries. Ya there are no copies of the cells. But new 200 cell refs coming up. So before 1st merge there were 200 cell refs totally and now it is 400. Now say one more segment came in and we merge together this new and old *merged* one. Consider new segment also have 100 cells. So before this merge op there are totally 400 + 100 = 500 refs 2nd merge also wont touch the segments and it creates a new segment and new array will contain 300 refs. So totally 800 refs. Actual cell objects are 300 and 2 merges made the refs# to be 800. So this not even doubling.. Like this if there are so many merge happens before a flush we will waste lots and lots of heap space with filling up refs. Just test with this attached patch which is not solving the problem correctly. It is a hack and that can not be considered any way. Just to make sure this fix is improving the full GC pattern am sending u. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15302: -- Attachment: HBASE-15302-branch-1.3-append-v1.patch > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.4 > > Attachments: HBASE-15302-branch-1-append-v1.patch, > HBASE-15302-branch-1-v1.patch, HBASE-15302-branch-1.3-append-v1.patch, > HBASE-15302-branch-1.3-append-v1.patch, HBASE-15302-v1.txt, HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588348#comment-15588348
]
Yu Li commented on HBASE-16698:
---
Perf data for one single region:
Test environment (nothing changed but no presplit on the target table):
{noformat}
YCSB 0.7.0
4 physical client nodes, 8 YCSB processes per node, 32 threads per YCSB process
recordcount=3,200,000, fieldcount=1, fieldlength=1024, insertproportion=1,
requestdistribution=uniform
1 single RS, 1 single region (no presplit), handlercount=128,
hbase.wal.storage.policy=ALL_SSD
{noformat}
And the comparison data (one round):
||TestCase||Throughput||AverageLatency(us)||
|w/o patch|69924.42|14544.38|
|w patch|86373.70|11770.09|
>From the result we could see even with one single region, performance w/ patch
>is better under high concurrency, which indicates that the discruptor publish
>and consume processing is more time-costing than the lock.
I could see less CountDownLatch waiting in jstack during testing w/o patch,
which could explain why the throughput is better than that against multiple
regions.
[~chenheng] FYI.
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
