[jira] [Commented] (HBASE-17646) Implement Async getRegion method
[
https://issues.apache.org/jira/browse/HBASE-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871365#comment-15871365
]
Hadoop QA commented on HBASE-17646:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 28s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 134m 48s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853217/HBASE-17646.v1.patch |
| JIRA Issue | HBASE-17646 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 8ed896b8b437 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3fc2889f |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5751/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5751/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5751/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Commented] (HBASE-17472) Correct the semantic of permission grant
[
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871318#comment-15871318
]
Hadoop QA commented on HBASE-17472:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 38s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m 52s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 164m 34s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853204/HBASE-17472.master.v7.patch
|
| JIRA Issue | HBASE-17472 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux 2a942ed5ea3a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 3fc2889f |
|
[jira] [Commented] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871315#comment-15871315
]
Hadoop QA commented on HBASE-17623:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 35s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 58s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 34s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 29s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853213/HBASE-17623.branch-1.v1.patch
|
| JIRA Issue | HBASE-17623 |
| Optional Tests
[jira] [Commented] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871302#comment-15871302
]
ChiaPing Tsai commented on HBASE-17623:
---
bq. Did u get a chance to run this for a longer time (May be use PE with 100GB+
data) and see the GC impacts (Positive or negative)?
copy that.
bq. You test with G1 or CMS?
CMS
Thanks for the feedback. [~anoop.hbase]
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.branch-1.v1.patch, HBASE-17623.v0.patch, HBASE-17623.v1.patch,
> HBASE-17623.v1.patch, memory allocation measurement.xlsx
>
>
> There are two improvements.
> # The uncompressedBlockBytesWithHeader and onDiskBlockBytesWithHeader should
> maintain a bytes array which can be reused when building the hfile.
> # The uncompressedBlockBytesWithHeader/onDiskBlockBytesWithHeader is copied
> to an new bytes array only when we need to cache the block.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17472) Correct the semantic of permission grant
[
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871263#comment-15871263
]
Hadoop QA commented on HBASE-17472:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 38s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
43s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 31s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
19s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 30s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 21s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 125m 39s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
53s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 204m 9s {color}
| {color:black} {color} |
\\
[jira] [Commented] (HBASE-16188) Add EventCounter information to log4j properties file
[
https://issues.apache.org/jira/browse/HBASE-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871258#comment-15871258
]
Hadoop QA commented on HBASE-16188:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 34s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 1s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 46s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-prefix-tree in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m 20s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 33s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hbase-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s
{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 12s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 36s
{color} | {color:green} hbase-client-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 36s
{color} | {color:green} hbase-shaded-client-project in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 126m 9s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 3m
10s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 301m 14s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.thrift.TestThriftServerCmdLine |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853181/HBASE-16188.patch |
| JIRA Issue | HBASE-16188 |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux 7e6405f57ca6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ae840c0 |
|
[jira] [Updated] (HBASE-17646) Implement Async getRegion method
[ https://issues.apache.org/jira/browse/HBASE-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huzheng updated HBASE-17646: Attachment: HBASE-17646.v1.patch > Implement Async getRegion method > > > Key: HBASE-17646 > URL: https://issues.apache.org/jira/browse/HBASE-17646 > Project: HBase > Issue Type: Sub-task > Components: asyncclient >Affects Versions: 2.0.0 >Reporter: huzheng >Assignee: huzheng > Labels: asynchronous > Fix For: 2.0.0 > > Attachments: HBASE-17646.v1.patch, HBASE-17646.v1.patch > > > There are some async admin APIs which depends on async getRegion method. > Such as : > 1. closeRegion. > 2. flushRegion. > 3. compactRegion. > 4. mergeRegion. > 5. splitRegion. > and so on . > So, implement async getRegion method first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871203#comment-15871203
]
Anoop Sam John commented on HBASE-17623:
Did u get a chance to run this for a longer time (May be use PE with 100GB+
data) and see the GC impacts (Positive or negative)? Reuse the byte[] for
building the next blocks also, makes the life of this byte[] longer and make it
promote to old gen? Objects die at young gen is always better wrt GC. That
is why I was asking for a longer run and see the GC. You test with G1 or CMS?
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.branch-1.v1.patch, HBASE-17623.v0.patch, HBASE-17623.v1.patch,
> HBASE-17623.v1.patch, memory allocation measurement.xlsx
>
>
> There are two improvements.
> # The uncompressedBlockBytesWithHeader and onDiskBlockBytesWithHeader should
> maintain a bytes array which can be reused when building the hfile.
> # The uncompressedBlockBytesWithHeader/onDiskBlockBytesWithHeader is copied
> to an new bytes array only when we need to cache the block.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17608) Add suspend support for RawScanResultConsumer
[
https://issues.apache.org/jira/browse/HBASE-17608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871201#comment-15871201
]
Hadoop QA commented on HBASE-17608:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 25s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 1s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 11s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m 26s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
53s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 42s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853194/HBASE-17608-v4.patch |
| JIRA Issue | HBASE-17608 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3124e44db6bd 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f4202a1 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5746/testReport/ |
|
[jira] [Updated] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17623:
--
Attachment: HBASE-17623.branch-1.v1.patch
# fix the findbugs warn.
# TestSimpleRpcScheduler passes locally.
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.branch-1.v1.patch, HBASE-17623.v0.patch, HBASE-17623.v1.patch,
> HBASE-17623.v1.patch, memory allocation measurement.xlsx
>
>
> There are two improvements.
> # The uncompressedBlockBytesWithHeader and onDiskBlockBytesWithHeader should
> maintain a bytes array which can be reused when building the hfile.
> # The uncompressedBlockBytesWithHeader/onDiskBlockBytesWithHeader is copied
> to an new bytes array only when we need to cache the block.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17623:
--
Status: Patch Available (was: Open)
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.branch-1.v1.patch, HBASE-17623.v0.patch, HBASE-17623.v1.patch,
> HBASE-17623.v1.patch, memory allocation measurement.xlsx
>
>
> There are two improvements.
> # The uncompressedBlockBytesWithHeader and onDiskBlockBytesWithHeader should
> maintain a bytes array which can be reused when building the hfile.
> # The uncompressedBlockBytesWithHeader/onDiskBlockBytesWithHeader is copied
> to an new bytes array only when we need to cache the block.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17623:
--
Status: Open (was: Patch Available)
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.v0.patch, HBASE-17623.v1.patch, HBASE-17623.v1.patch, memory
> allocation measurement.xlsx
>
>
> There are two improvements.
> # The uncompressedBlockBytesWithHeader and onDiskBlockBytesWithHeader should
> maintain a bytes array which can be reused when building the hfile.
> # The uncompressedBlockBytesWithHeader/onDiskBlockBytesWithHeader is copied
> to an new bytes array only when we need to cache the block.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly
[
https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-16981:
---
Release Note:
Mob compaction partition policy can be set by
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 100,
MOB_COMPACT_PARTITION_POLICY => 'weekly'}
or
hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 100,
MOB_COMPACT_PARTITION_POLICY => 'monthly'}
Available MOB_COMPACT_PARTITION_POLICY options are "daily", "weekly" and
"monthly", the default is "daily".
When it is "weekly" policy, the mob compaction will try to compact files within
one calendar week into one for a specific partition, similar for "daily" and
"monthly".
With "weekly" policy, one mob file normally is compacted twice during its
lifetime (that is first on daily basis and then all such daily based compacted
files belonging to a week at the weekly interval), for one region, there
normally are 52 files for one year. With "Monthly" policy, one mob file
normally is compacted 3 times during its lifetime (First daily and then weekly
followed by monthly at end of every month) and normally there are 12 files for
one year.
was:
Mob compaction partition policy can be set by
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 100,
MOB_COMPACT_PARTITION_POLICY => 'weekly'}
or
hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 100,
MOB_COMPACT_PARTITION_POLICY => 'monthly'}
Available MOB_COMPACT_PARTITION_POLICY options are "daily", "weekly" and
"monthly", the default is "daily".
When it is "weekly" policy, the mob compaction will try to compact files within
one calendar week into one for a specific partition, similar for "daily" and
"monthly".
With "weekly" policy, one mob file normally is compacted twice during its
lifetime, for one region, there normally are 52 files for one year. With
"Monthly" policy, one mob file normally is compacted 3 times during its
lifetime and normally there are 12 files for one year.
> Expand Mob Compaction Partition policy from daily to weekly, monthly
>
>
> Key: HBASE-16981
> URL: https://issues.apache.org/jira/browse/HBASE-16981
> Project: HBase
> Issue Type: New Feature
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Fix For: 2.0.0
>
> Attachments: HBASE-16981.master.001.patch,
> HBASE-16981.master.002.patch, HBASE-16981.master.003.patch,
> HBASE-16981.master.004.patch, HBASE-16981.master.005.patch,
> HBASE-16981.master.006.patch, HBASE-16981.master.007.patch,
> HBASE-16981.master.008.patch,
> Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf
>
>
> Today the mob region holds all mob files for all regions. With daily
> partition mob compaction policy, after major mob compaction, there is still
> one file per region daily. Given there is 365 days in one year, at least 365
> files per region. Since HDFS has limitation for number of files under one
> folder, this is not going to scale if there are lots of regions. To reduce
> mob file number, we want to introduce other partition policies such as
> weekly, monthly to compact mob files within one week or month into one file.
> This jira is create to track this effort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871157#comment-15871157 ] Anoop Sam John commented on HBASE-16981: A bit of update is done so as to explain the weekly and monthly doing 2 or 3 times IO. Thanks.. Looks good. > Expand Mob Compaction Partition policy from daily to weekly, monthly > > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0 > > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, HBASE-16981.master.005.patch, > HBASE-16981.master.006.patch, HBASE-16981.master.007.patch, > HBASE-16981.master.008.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table
[ https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871151#comment-15871151 ] Ted Yu commented on HBASE-17460: For the future patches, please include trivial change to a file in hbase-server module so that QA can detect test failure early. > enable_table_replication can not perform cyclic replication of a table > -- > > Key: HBASE-17460 > URL: https://issues.apache.org/jira/browse/HBASE-17460 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: NITIN VERMA >Assignee: NITIN VERMA > Labels: replication > Fix For: 2.0.0, 1.4.0 > > Attachments: 17460.branch-1.v3.txt, HBASE-17460.patch, > HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > The enable_table_replication operation is broken for cyclic replication of > HBase table as we compare all the properties of column families (including > REPLICATION_SCOPE). > Below is exactly what happens: > 1. Running "enable_table_replication 'table1' " opeartion on first cluster > will set the REPLICATION_SCOPE of all column families to peer id '1'. This > will also create a table on second cluster where REPLICATION_SCOPE is still > set to peer id '0'. > 2. Now when we run "enable_table_replication 'table1'" on second cluster, we > compare all the properties of table (including REPLICATION_SCOPE_, which > obviously is different now. > I am proposing a fix for this issue where we should avoid comparing > REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially > when replication is not already enabled on the desired table. > I have made that change and it is working. I will submit the patch soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17472) Correct the semantic of permission grant
[
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871148#comment-15871148
]
huzheng commented on HBASE-17472:
-
Upload patch v7 for removing all grant2.
> Correct the semantic of permission grant
> -
>
> Key: HBASE-17472
> URL: https://issues.apache.org/jira/browse/HBASE-17472
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Affects Versions: 2.0.0, 1.4.0
>Reporter: huzheng
>Assignee: huzheng
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17472.branch-1.3.v6.patch,
> HBASE-17472.branch-1.v6.patch, HBASE-17472.branch-1.v7.patch,
> HBASE-17472.master.v6.patch, HBASE-17472.master.v6.patch,
> HBASE-17472.master.v7.patch, HBASE-17472.v1.patch, HBASE-17472.v2.patch,
> HBASE-17472.v3.patch, HBASE-17472.v4.patch, HBASE-17472.v5.patch
>
>
> Currently, HBase grant operation has following semantic:
> {code}
> hbase(main):019:0> grant 'hbase_tst', 'RW', 'ycsb'
> 0 row(s) in 0.0960 seconds
> hbase(main):020:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
>
> hbase_tst default,ycsb,,:
> [Permission:actions=READ,WRITE]
>
>
> 1 row(s) in 0.0550 seconds
> hbase(main):021:0> grant 'hbase_tst', 'CA', 'ycsb'
> 0 row(s) in 0.0820 seconds
> hbase(main):022:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
> hbase_tst default,ycsb,,:
> [Permission: actions=CREATE,ADMIN]
>
>
> 1 row(s) in 0.0490 seconds
> {code}
> Later permission will replace previous granted permissions, which confused
> most of HBase administrator.
> It's seems more reasonable that HBase merge multiple granted permission.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17472) Correct the semantic of permission grant
[
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huzheng updated HBASE-17472:
Attachment: HBASE-17472.master.v7.patch
> Correct the semantic of permission grant
> -
>
> Key: HBASE-17472
> URL: https://issues.apache.org/jira/browse/HBASE-17472
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Affects Versions: 2.0.0, 1.4.0
>Reporter: huzheng
>Assignee: huzheng
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17472.branch-1.3.v6.patch,
> HBASE-17472.branch-1.v6.patch, HBASE-17472.branch-1.v7.patch,
> HBASE-17472.master.v6.patch, HBASE-17472.master.v6.patch,
> HBASE-17472.master.v7.patch, HBASE-17472.v1.patch, HBASE-17472.v2.patch,
> HBASE-17472.v3.patch, HBASE-17472.v4.patch, HBASE-17472.v5.patch
>
>
> Currently, HBase grant operation has following semantic:
> {code}
> hbase(main):019:0> grant 'hbase_tst', 'RW', 'ycsb'
> 0 row(s) in 0.0960 seconds
> hbase(main):020:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
>
> hbase_tst default,ycsb,,:
> [Permission:actions=READ,WRITE]
>
>
> 1 row(s) in 0.0550 seconds
> hbase(main):021:0> grant 'hbase_tst', 'CA', 'ycsb'
> 0 row(s) in 0.0820 seconds
> hbase(main):022:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
> hbase_tst default,ycsb,,:
> [Permission: actions=CREATE,ADMIN]
>
>
> 1 row(s) in 0.0490 seconds
> {code}
> Later permission will replace previous granted permissions, which confused
> most of HBase administrator.
> It's seems more reasonable that HBase merge multiple granted permission.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table
[ https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871147#comment-15871147 ] Ted Yu commented on HBASE-17460: I reverted this from master branch as well : TestReplicationAdminWithClusters consistently failed in the past several trunk builds. > enable_table_replication can not perform cyclic replication of a table > -- > > Key: HBASE-17460 > URL: https://issues.apache.org/jira/browse/HBASE-17460 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: NITIN VERMA >Assignee: NITIN VERMA > Labels: replication > Fix For: 2.0.0, 1.4.0 > > Attachments: 17460.branch-1.v3.txt, HBASE-17460.patch, > HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > The enable_table_replication operation is broken for cyclic replication of > HBase table as we compare all the properties of column families (including > REPLICATION_SCOPE). > Below is exactly what happens: > 1. Running "enable_table_replication 'table1' " opeartion on first cluster > will set the REPLICATION_SCOPE of all column families to peer id '1'. This > will also create a table on second cluster where REPLICATION_SCOPE is still > set to peer id '0'. > 2. Now when we run "enable_table_replication 'table1'" on second cluster, we > compare all the properties of table (including REPLICATION_SCOPE_, which > obviously is different now. > I am proposing a fix for this issue where we should avoid comparing > REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially > when replication is not already enabled on the desired table. > I have made that change and it is working. I will submit the patch soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871146#comment-15871146
]
Hadoop QA commented on HBASE-17623:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 31s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 46s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 54s
{color} | {color:red} hbase-common generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 51s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 16s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-common |
| | org.apache.hadoop.hbase.util.Bytes.equals(Object) checks for operand
being a byte[] At Bytes.java:a byte[] At Bytes.java:[line 267] |
| Failed
[jira] [Commented] (HBASE-17656) Move new Address class from util to net package
[ https://issues.apache.org/jira/browse/HBASE-17656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871133#comment-15871133 ] Hudson commented on HBASE-17656: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2518 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2518/]) HBASE-17656 Move new Address class from util to net package (stack: rev ae840c0ccd123985d5cde186cc2d0a0e92cb2467) * (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ServerName.java * (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * (add) hbase-common/src/main/java/org/apache/hadoop/hbase/net/Address.java * (edit) hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java * (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java * (delete) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Address.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupSerDe.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/Utility.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java * (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java > Move new Address class from util to net package > --- > > Key: HBASE-17656 > URL: https://issues.apache.org/jira/browse/HBASE-17656 > Project: HBase > Issue Type: Sub-task > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17656.master.001.patch > > > I added the new Address type in the wrong package committing the parent JIRA > patch. It should be in net, not util. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17658) Bookkeeping issue in BaseLoadBalancer for Table Skew
[
https://issues.apache.org/jira/browse/HBASE-17658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871130#comment-15871130
]
Hadoop QA commented on HBASE-17658:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 37s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 33s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 124m 53s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 173m 23s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.6 Server=1.12.6 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853175/JIRA-17658.master.001.patch
|
| JIRA Issue | HBASE-17658 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux dfb980fb663c 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18
16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ae840c0 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5743/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5743/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5743/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Bookkeeping issue in BaseLoadBalancer for Table Skew
>
>
> Key: HBASE-17658
> URL: https://issues.apache.org/jira/browse/HBASE-17658
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Reporter: Tim Brown
>Assignee: Tim
[jira] [Commented] (HBASE-17583) Add inclusive/exclusive support for startRow and endRow of scan for sync client
[ https://issues.apache.org/jira/browse/HBASE-17583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871128#comment-15871128 ] Duo Zhang commented on HBASE-17583: --- File a issue to track it? > Add inclusive/exclusive support for startRow and endRow of scan for sync > client > --- > > Key: HBASE-17583 > URL: https://issues.apache.org/jira/browse/HBASE-17583 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17583-branch-1.patch, > HBASE-17583-branch-1-v1.patch, HBASE-17583.patch, HBASE-17583-v1.patch, > HBASE-17583-v2.patch, HBASE-17583-v3.patch > > > Implement the same feature of HBASE-17320 for sync client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17583) Add inclusive/exclusive support for startRow and endRow of scan for sync client
[
https://issues.apache.org/jira/browse/HBASE-17583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871120#comment-15871120
]
Ted Yu edited comment on HBASE-17583 at 2/17/17 3:32 AM:
-
Looks like the following test failure is reproducible:
{code}
testEnableReplicationWhenSlaveClusterDoesntHaveTable(org.apache.hadoop.hbase.client.replication.TestReplicationAdminWithClusters)
Time elapsed: 1.243 sec <<< ERROR!
java.lang.IllegalArgumentException: Table test has replication already enabled
for atleast one Column Family.
at
org.apache.hadoop.hbase.client.replication.TestReplicationAdminWithClusters.testEnableReplicationWhenSlaveClusterDoesntHaveTable(TestReplicationAdminWithClusters.java:83)
{code}
But the test failure was present before this went in.
was (Author: yuzhih...@gmail.com):
Looks like the following test failure is reproducible:
{code}
testEnableReplicationWhenSlaveClusterDoesntHaveTable(org.apache.hadoop.hbase.client.replication.TestReplicationAdminWithClusters)
Time elapsed: 1.243 sec <<< ERROR!
java.lang.IllegalArgumentException: Table test has replication already enabled
for atleast one Column Family.
at
org.apache.hadoop.hbase.client.replication.TestReplicationAdminWithClusters.testEnableReplicationWhenSlaveClusterDoesntHaveTable(TestReplicationAdminWithClusters.java:83)
{code}
> Add inclusive/exclusive support for startRow and endRow of scan for sync
> client
> ---
>
> Key: HBASE-17583
> URL: https://issues.apache.org/jira/browse/HBASE-17583
> Project: HBase
> Issue Type: Sub-task
> Components: Client, scan
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17583-branch-1.patch,
> HBASE-17583-branch-1-v1.patch, HBASE-17583.patch, HBASE-17583-v1.patch,
> HBASE-17583-v2.patch, HBASE-17583-v3.patch
>
>
> Implement the same feature of HBASE-17320 for sync client.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17583) Add inclusive/exclusive support for startRow and endRow of scan for sync client
[
https://issues.apache.org/jira/browse/HBASE-17583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871120#comment-15871120
]
Ted Yu commented on HBASE-17583:
Looks like the following test failure is reproducible:
{code}
testEnableReplicationWhenSlaveClusterDoesntHaveTable(org.apache.hadoop.hbase.client.replication.TestReplicationAdminWithClusters)
Time elapsed: 1.243 sec <<< ERROR!
java.lang.IllegalArgumentException: Table test has replication already enabled
for atleast one Column Family.
at
org.apache.hadoop.hbase.client.replication.TestReplicationAdminWithClusters.testEnableReplicationWhenSlaveClusterDoesntHaveTable(TestReplicationAdminWithClusters.java:83)
{code}
> Add inclusive/exclusive support for startRow and endRow of scan for sync
> client
> ---
>
> Key: HBASE-17583
> URL: https://issues.apache.org/jira/browse/HBASE-17583
> Project: HBase
> Issue Type: Sub-task
> Components: Client, scan
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17583-branch-1.patch,
> HBASE-17583-branch-1-v1.patch, HBASE-17583.patch, HBASE-17583-v1.patch,
> HBASE-17583-v2.patch, HBASE-17583-v3.patch
>
>
> Implement the same feature of HBASE-17320 for sync client.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17608) Add suspend support for RawScanResultConsumer
[ https://issues.apache.org/jira/browse/HBASE-17608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871094#comment-15871094 ] Duo Zhang commented on HBASE-17608: --- Ping [~stack]. > Add suspend support for RawScanResultConsumer > - > > Key: HBASE-17608 > URL: https://issues.apache.org/jira/browse/HBASE-17608 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17608.patch, HBASE-17608-v1.patch, > HBASE-17608-v2.patch, HBASE-17608-v3.patch, HBASE-17608-v4.patch > > > Now for the AsyncResultScanner, we can only close the scanner if we reach the > cache size limit and open a new scanner later. This will breaks the region > level consistency. > For example, you put 10 rows to the region, and open a scanner to scan it. > The scanner returns 5 rows at the first time and the cache is full, so it > closes the background scanner. And before you reopens the scanner to fetch > data, the remaining 5 rows has been deleted and a compaction makes them gone > for ever. Then after you reopen the scanner you can not see the remaining 5 > rows. > So here we should keep the scanner open at RS side to prevent the data below > the mvcc read point of this scanner being deleted. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17623:
--
Status: Patch Available (was: Open)
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.v0.patch, HBASE-17623.v1.patch, HBASE-17623.v1.patch, memory
> allocation measurement.xlsx
>
>
> There are two improvements.
> # The uncompressedBlockBytesWithHeader and onDiskBlockBytesWithHeader should
> maintain a bytes array which can be reused when building the hfile.
> # The uncompressedBlockBytesWithHeader/onDiskBlockBytesWithHeader is copied
> to an new bytes array only when we need to cache the block.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17623:
--
Status: Open (was: Patch Available)
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.v0.patch, HBASE-17623.v1.patch, HBASE-17623.v1.patch, memory
> allocation measurement.xlsx
>
>
> There are two improvements.
> # The uncompressedBlockBytesWithHeader and onDiskBlockBytesWithHeader should
> maintain a bytes array which can be reused when building the hfile.
> # The uncompressedBlockBytesWithHeader/onDiskBlockBytesWithHeader is copied
> to an new bytes array only when we need to cache the block.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17623:
--
Fix Version/s: 1.4.0
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.v0.patch, HBASE-17623.v1.patch, HBASE-17623.v1.patch, memory
> allocation measurement.xlsx
>
>
> There are two improvements.
> # The uncompressedBlockBytesWithHeader and onDiskBlockBytesWithHeader should
> maintain a bytes array which can be reused when building the hfile.
> # The uncompressedBlockBytesWithHeader/onDiskBlockBytesWithHeader is copied
> to an new bytes array only when we need to cache the block.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17623:
--
Attachment: HBASE-17623.branch-1.v0.patch
I borrow some codes from HBASE-11862 and HBASE-15077 for branch-1.
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.v0.patch, HBASE-17623.v1.patch, HBASE-17623.v1.patch, memory
> allocation measurement.xlsx
>
>
> There are two improvements.
> # The uncompressedBlockBytesWithHeader and onDiskBlockBytesWithHeader should
> maintain a bytes array which can be reused when building the hfile.
> # The uncompressedBlockBytesWithHeader/onDiskBlockBytesWithHeader is copied
> to an new bytes array only when we need to cache the block.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17472) Correct the semantic of permission grant
[
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huzheng updated HBASE-17472:
Attachment: HBASE-17472.branch-1.v7.patch
> Correct the semantic of permission grant
> -
>
> Key: HBASE-17472
> URL: https://issues.apache.org/jira/browse/HBASE-17472
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Affects Versions: 2.0.0, 1.4.0
>Reporter: huzheng
>Assignee: huzheng
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17472.branch-1.3.v6.patch,
> HBASE-17472.branch-1.v6.patch, HBASE-17472.branch-1.v7.patch,
> HBASE-17472.master.v6.patch, HBASE-17472.master.v6.patch,
> HBASE-17472.v1.patch, HBASE-17472.v2.patch, HBASE-17472.v3.patch,
> HBASE-17472.v4.patch, HBASE-17472.v5.patch
>
>
> Currently, HBase grant operation has following semantic:
> {code}
> hbase(main):019:0> grant 'hbase_tst', 'RW', 'ycsb'
> 0 row(s) in 0.0960 seconds
> hbase(main):020:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
>
> hbase_tst default,ycsb,,:
> [Permission:actions=READ,WRITE]
>
>
> 1 row(s) in 0.0550 seconds
> hbase(main):021:0> grant 'hbase_tst', 'CA', 'ycsb'
> 0 row(s) in 0.0820 seconds
> hbase(main):022:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
> hbase_tst default,ycsb,,:
> [Permission: actions=CREATE,ADMIN]
>
>
> 1 row(s) in 0.0490 seconds
> {code}
> Later permission will replace previous granted permissions, which confused
> most of HBase administrator.
> It's seems more reasonable that HBase merge multiple granted permission.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17608) Add suspend support for RawScanResultConsumer
[ https://issues.apache.org/jira/browse/HBASE-17608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17608: -- Attachment: HBASE-17608-v4.patch Fix the javadoc warning. > Add suspend support for RawScanResultConsumer > - > > Key: HBASE-17608 > URL: https://issues.apache.org/jira/browse/HBASE-17608 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17608.patch, HBASE-17608-v1.patch, > HBASE-17608-v2.patch, HBASE-17608-v3.patch, HBASE-17608-v4.patch > > > Now for the AsyncResultScanner, we can only close the scanner if we reach the > cache size limit and open a new scanner later. This will breaks the region > level consistency. > For example, you put 10 rows to the region, and open a scanner to scan it. > The scanner returns 5 rows at the first time and the cache is full, so it > closes the background scanner. And before you reopens the scanner to fetch > data, the remaining 5 rows has been deleted and a compaction makes them gone > for ever. Then after you reopen the scanner you can not see the remaining 5 > rows. > So here we should keep the scanner open at RS side to prevent the data below > the mvcc read point of this scanner being deleted. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17472) Correct the semantic of permission grant
[
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871070#comment-15871070
]
huzheng commented on HBASE-17472:
-
Oh... Sorry, Will upload a new patch. Thanks [~enis]
> Correct the semantic of permission grant
> -
>
> Key: HBASE-17472
> URL: https://issues.apache.org/jira/browse/HBASE-17472
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Affects Versions: 2.0.0, 1.4.0
>Reporter: huzheng
>Assignee: huzheng
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17472.branch-1.3.v6.patch,
> HBASE-17472.branch-1.v6.patch, HBASE-17472.master.v6.patch,
> HBASE-17472.master.v6.patch, HBASE-17472.v1.patch, HBASE-17472.v2.patch,
> HBASE-17472.v3.patch, HBASE-17472.v4.patch, HBASE-17472.v5.patch
>
>
> Currently, HBase grant operation has following semantic:
> {code}
> hbase(main):019:0> grant 'hbase_tst', 'RW', 'ycsb'
> 0 row(s) in 0.0960 seconds
> hbase(main):020:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
>
> hbase_tst default,ycsb,,:
> [Permission:actions=READ,WRITE]
>
>
> 1 row(s) in 0.0550 seconds
> hbase(main):021:0> grant 'hbase_tst', 'CA', 'ycsb'
> 0 row(s) in 0.0820 seconds
> hbase(main):022:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
> hbase_tst default,ycsb,,:
> [Permission: actions=CREATE,ADMIN]
>
>
> 1 row(s) in 0.0490 seconds
> {code}
> Later permission will replace previous granted permissions, which confused
> most of HBase administrator.
> It's seems more reasonable that HBase merge multiple granted permission.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17421) Update refguide w.r.t. MOB Sweeper
[ https://issues.apache.org/jira/browse/HBASE-17421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17421: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) > Update refguide w.r.t. MOB Sweeper > -- > > Key: HBASE-17421 > URL: https://issues.apache.org/jira/browse/HBASE-17421 > Project: HBase > Issue Type: Bug > Components: documentation, mob >Reporter: Ted Yu >Assignee: Jingcheng Du > Fix For: 2.0.0 > > Attachments: HBASE-17421.patch > > > The http://hbase.apache.org/book.html#_mob_sweeper section is out of date. > It mentions settings such as hbase.mob.sweep.tool.compaction.mergeable.size > which no longer apply in master branch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17421) Update refguide w.r.t. MOB Sweeper
[ https://issues.apache.org/jira/browse/HBASE-17421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871063#comment-15871063 ] Jingcheng Du commented on HBASE-17421: -- Pushed to the master branch. Thanks [~tedyu] for the review! > Update refguide w.r.t. MOB Sweeper > -- > > Key: HBASE-17421 > URL: https://issues.apache.org/jira/browse/HBASE-17421 > Project: HBase > Issue Type: Bug > Components: documentation, mob >Reporter: Ted Yu >Assignee: Jingcheng Du > Attachments: HBASE-17421.patch > > > The http://hbase.apache.org/book.html#_mob_sweeper section is out of date. > It mentions settings such as hbase.mob.sweep.tool.compaction.mergeable.size > which no longer apply in master branch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Issue Comment Deleted] (HBASE-16188) Add EventCounter information to log4j properties file
[
https://issues.apache.org/jira/browse/HBASE-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopi Krishnan Nambiar updated HBASE-16188:
--
Comment: was deleted
(was: Check HadoopQA)
> Add EventCounter information to log4j properties file
> -
>
> Key: HBASE-16188
> URL: https://issues.apache.org/jira/browse/HBASE-16188
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.1
>Reporter: Lars George
>Assignee: Gopi Krishnan Nambiar
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-16188.patch
>
>
> Hadoop's {{JvmMetrics}}, which HBase also is using in Metrics2 and provides
> it as an MBean, has the ability to count log4j log calls. This is tracked by
> a special {{Appender}} class, also provided by Hadoop, called
> {{EventCounter}}.
> We should add some info how to enable this (or maybe even enable it by
> default?).
> The appender needs to be added in two places, shown here:
> {noformat}
> hbase.root.logger=INFO,console
> ...
> # Define the root logger to the system property "hbase.root.logger".
> log4j.rootLogger=${hbase.root.logger}, EventCounter
> log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter
> {noformat}
> We could simply add this commented out akin to the {{hbase-env.sh}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17312) [JDK8] Use default method for Observer Coprocessors
[
https://issues.apache.org/jira/browse/HBASE-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871048#comment-15871048
]
Appy edited comment on HBASE-17312 at 2/17/17 2:00 AM:
---
So answering [~stack]'s question above:
Right now we have
{noformat}
public interface Foo {
void getBar1();
void getBar2();
}
public class BaseFoo {
void getBar1() { }
void getBar2() { }
}
---
class ThirdParty extends BaseFoo {
void getBar1() { // override }
}
{noformat}
That's changes to:
{noformat}
public interface Foo {
void getBar1() {}
void getBar2() {}
}
--
class ThirdParty implement Foo { // Only one line fix
void getBar1() { // override }
}
{noformat}
One weird case which i saw in tests was, overrides calling base class'
functions. It doesn't make sense because our base classes have no
functionality. They are there to just prevent redundant empty implementations
(or returning default values when hooks have non-void return type).
But that's just our tests, and we should remove these calls ([~zghaobac]).
was (Author: appy):
So answering [~stack]'s question above:
Right now we have
{noformat}
public interface Foo {
void getBar1();
void getBar2();
}
public class BaseFoo {
void getBar1() { }
void getBar2() { }
}
---
class ThirdParty extends BaseFoo {
void getBar1() { // override }
}
{noformat}
That's changes to:
{noformat}
public interface Foo {
void getBar1() {}
void getBar2() {}
}
--
class ThirdParty implement Foo { // Only one line fix
void getBar1() { // override }
}
{noformat}
One weird case which i saw in tests was, overrides calling base class'
functions. It doesn't make sense because our base classes have no
functionality. They are there to just prevent redundant empty implementations
(or returning default values when hooks have non-void return type).
> [JDK8] Use default method for Observer Coprocessors
> ---
>
> Key: HBASE-17312
> URL: https://issues.apache.org/jira/browse/HBASE-17312
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17312.master.001.patch,
> HBASE-17312.master.001.patch
>
>
> Use default method in MasterObserver, RegionObserver, RegionServerObserver
> and WALObserver. And mark the BaseRegionObserver,
> BaseMasterAndRegionObserver, BaseRegionServerObserver and BaseWALObserver.
> User can implement the interface directly and will not break compatibility
> when add new default methods.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17312) [JDK8] Use default method for Observer Coprocessors
[
https://issues.apache.org/jira/browse/HBASE-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871048#comment-15871048
]
Appy commented on HBASE-17312:
--
So answering [~stack]'s question above:
Right now we have
{noformat}
public interface Foo {
void getBar1();
void getBar2();
}
public class BaseFoo {
void getBar1() { }
void getBar2() { }
}
---
class ThirdParty extends BaseFoo {
void getBar1() { // override }
}
{noformat}
That's changes to:
{noformat}
public interface Foo {
void getBar1() {}
void getBar2() {}
}
--
class ThirdParty implement Foo { // Only one line fix
void getBar1() { // override }
}
{noformat}
One weird case which i saw in tests was, overrides calling base class'
functions. It doesn't make sense because our base classes have no
functionality. They are there to just prevent redundant empty implementations
(or returning default values when hooks have non-void return type).
> [JDK8] Use default method for Observer Coprocessors
> ---
>
> Key: HBASE-17312
> URL: https://issues.apache.org/jira/browse/HBASE-17312
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17312.master.001.patch,
> HBASE-17312.master.001.patch
>
>
> Use default method in MasterObserver, RegionObserver, RegionServerObserver
> and WALObserver. And mark the BaseRegionObserver,
> BaseMasterAndRegionObserver, BaseRegionServerObserver and BaseWALObserver.
> User can implement the interface directly and will not break compatibility
> when add new default methods.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871047#comment-15871047 ] Jingcheng Du commented on HBASE-16981: -- Thanks [~huaxiang], it looks good to me. > Expand Mob Compaction Partition policy from daily to weekly, monthly > > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0 > > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, HBASE-16981.master.005.patch, > HBASE-16981.master.006.patch, HBASE-16981.master.007.patch, > HBASE-16981.master.008.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17583) Add inclusive/exclusive support for startRow and endRow of scan for sync client
[ https://issues.apache.org/jira/browse/HBASE-17583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17583: -- Release Note: Now you can include/exlude the startRow and stopRow for a scan. And the new methods to specify startRow and stopRow are withStartRow and withStopRow. The old methods to specify startRow and Row(include constructors) are marked as deprecated as in the old time if startRow and stopRow are equal then we will consider it as a get scan and include the stopRow implicitly. This is strange after we can set inclusiveness explicitly so we add new methods and depredate the old methods. The deprecated methods will be removed in the future. > Add inclusive/exclusive support for startRow and endRow of scan for sync > client > --- > > Key: HBASE-17583 > URL: https://issues.apache.org/jira/browse/HBASE-17583 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17583-branch-1.patch, > HBASE-17583-branch-1-v1.patch, HBASE-17583.patch, HBASE-17583-v1.patch, > HBASE-17583-v2.patch, HBASE-17583-v3.patch > > > Implement the same feature of HBASE-17320 for sync client. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17619) Add async admin Impl which connect to RegionServer and implement close region methods.
[ https://issues.apache.org/jira/browse/HBASE-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17619: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~openinx] for contributing. > Add async admin Impl which connect to RegionServer and implement close region > methods. > -- > > Key: HBASE-17619 > URL: https://issues.apache.org/jira/browse/HBASE-17619 > Project: HBase > Issue Type: Sub-task > Components: asyncclient >Affects Versions: 2.0.0 >Reporter: huzheng >Assignee: huzheng > Labels: asynchronous > Fix For: 2.0.0 > > Attachments: HBASE-17619.v1.patch, HBASE-17619.v1.patch, > HBASE-17619.v2.patch > > > Currently , Async client has async master impl and async htable impl , but > not async admin impl , which connect to RS for managing regions in region > server . > This issue will add async admin which implement AdminService.Interface to > AsyncConnectionImpl, and implement close region method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17654) RSGroup code refactoring
[ https://issues.apache.org/jira/browse/HBASE-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871017#comment-15871017 ] stack commented on HBASE-17654: --- Tests failed. I think Closeable is already removed? There is already a Utility class for rsgroup called Utility so RSGroupUtil clashes. Leave as it was? How does RSGroupStartupWorker get an instance of masterServices now it does not take it in constructor. Any reason to change the ServerEventsListenerThread so it listened on events instead? (I'd rather get rid of that thread if we could). There is a bunch of nice cleanup in here. RB is down. Will give you better review when it comes back. > RSGroup code refactoring > > > Key: HBASE-17654 > URL: https://issues.apache.org/jira/browse/HBASE-17654 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-17654.master.001.patch, > HBASE-17654.master.002.patch, HBASE-17654.master.003.patch > > > TBD -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17654) RSGroup code refactoring
[
https://issues.apache.org/jira/browse/HBASE-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871004#comment-15871004
]
Hadoop QA commented on HBASE-17654:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color}
| {color:red} HBASE-17654 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853184/HBASE-17654.master.003.patch
|
| JIRA Issue | HBASE-17654 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5745/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> RSGroup code refactoring
>
>
> Key: HBASE-17654
> URL: https://issues.apache.org/jira/browse/HBASE-17654
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-17654.master.001.patch,
> HBASE-17654.master.002.patch, HBASE-17654.master.003.patch
>
>
> TBD
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17654) RSGroup code refactoring
[ https://issues.apache.org/jira/browse/HBASE-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17654: - Attachment: HBASE-17654.master.003.patch > RSGroup code refactoring > > > Key: HBASE-17654 > URL: https://issues.apache.org/jira/browse/HBASE-17654 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-17654.master.001.patch, > HBASE-17654.master.002.patch, HBASE-17654.master.003.patch > > > TBD -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17654) RSGroup code refactoring
[ https://issues.apache.org/jira/browse/HBASE-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17654: - Attachment: HBASE-17654.master.002.patch > RSGroup code refactoring > > > Key: HBASE-17654 > URL: https://issues.apache.org/jira/browse/HBASE-17654 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-17654.master.001.patch, > HBASE-17654.master.002.patch > > > TBD -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16188) Add EventCounter information to log4j properties file
[
https://issues.apache.org/jira/browse/HBASE-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopi Krishnan Nambiar updated HBASE-16188:
--
Fix Version/s: (was: 1.3.0)
2.0.0
Status: Patch Available (was: In Progress)
> Add EventCounter information to log4j properties file
> -
>
> Key: HBASE-16188
> URL: https://issues.apache.org/jira/browse/HBASE-16188
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.1
>Reporter: Lars George
>Assignee: Gopi Krishnan Nambiar
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-16188.patch
>
>
> Hadoop's {{JvmMetrics}}, which HBase also is using in Metrics2 and provides
> it as an MBean, has the ability to count log4j log calls. This is tracked by
> a special {{Appender}} class, also provided by Hadoop, called
> {{EventCounter}}.
> We should add some info how to enable this (or maybe even enable it by
> default?).
> The appender needs to be added in two places, shown here:
> {noformat}
> hbase.root.logger=INFO,console
> ...
> # Define the root logger to the system property "hbase.root.logger".
> log4j.rootLogger=${hbase.root.logger}, EventCounter
> log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter
> {noformat}
> We could simply add this commented out akin to the {{hbase-env.sh}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-16188) Add EventCounter information to log4j properties file
[
https://issues.apache.org/jira/browse/HBASE-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopi Krishnan Nambiar updated HBASE-16188:
--
Attachment: HBASE-16188.patch
> Add EventCounter information to log4j properties file
> -
>
> Key: HBASE-16188
> URL: https://issues.apache.org/jira/browse/HBASE-16188
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.1
>Reporter: Lars George
>Assignee: Gopi Krishnan Nambiar
>Priority: Minor
> Labels: beginner
> Fix For: 1.3.0
>
> Attachments: HBASE-16188.patch
>
>
> Hadoop's {{JvmMetrics}}, which HBase also is using in Metrics2 and provides
> it as an MBean, has the ability to count log4j log calls. This is tracked by
> a special {{Appender}} class, also provided by Hadoop, called
> {{EventCounter}}.
> We should add some info how to enable this (or maybe even enable it by
> default?).
> The appender needs to be added in two places, shown here:
> {noformat}
> hbase.root.logger=INFO,console
> ...
> # Define the root logger to the system property "hbase.root.logger".
> log4j.rootLogger=${hbase.root.logger}, EventCounter
> log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter
> {noformat}
> We could simply add this commented out akin to the {{hbase-env.sh}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17653) HBASE-17624 rsgroup synchronizations will (distributed) deadlock
[
https://issues.apache.org/jira/browse/HBASE-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17653:
--
Attachment: HBASE-17653.master.003.patch
> HBASE-17624 rsgroup synchronizations will (distributed) deadlock
>
>
> Key: HBASE-17653
> URL: https://issues.apache.org/jira/browse/HBASE-17653
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Reporter: stack
>Assignee: stack
> Attachments: HBASE-17653.master.001.patch,
> HBASE-17653.master.002.patch, HBASE-17653.master.003.patch
>
>
> Follow-on from HBASE-17624. HBASE-17624 made it so one thread only has access
> to the rsgroup administrator. In tail of HBASE-17624 [~toffer] describes
> scenario under which we may end up in a deadlock (distributed). Let me
> repeat [~toffer] comment...
> {code}
> Both read/write access can't be single threaded. Consider the situation:
> 1. move_rsgroup_servers is called
> 2. while #1 is happening rsgroup region is in transition (rpc thread in #1
> holds monitor lock)
> 3. while #2 is happening meta is in transition.
> Balancer tries to figure out plan for meta region tries to get monitor lock
> but can't. rpc thread task won't release monitor lock since rsgroup region
> never gets assigned. rsgroup region never gets assigned because it can't
> update meta with new state.
> There's a good chance this can be reproduce just by moving both rsgroup and
> meta region onto the same RS and call move_rsgoup_servers on the same RS.
> A bunch different actors will query from group affiliation so we can't have
> writes block reads.
>
> In the code prior to this patch the getter methods that retrieve group
> information (getRSGroup, ofTable, OfServer, etc) don't require the monitor
> lock so the deadlock cycle is broken.
>
> The methods that does mutations and updates to zk and hbase:rsgroup are
> synchronized appropriately. Point me to where the incoherence is?
> {code}
> This issue is about testing/fixing/restoring rsgroup access. Will be back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17658) Bookkeeping issue in BaseLoadBalancer for Table Skew
[ https://issues.apache.org/jira/browse/HBASE-17658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870974#comment-15870974 ] Ted Yu commented on HBASE-17658: +1, pending QA. > Bookkeeping issue in BaseLoadBalancer for Table Skew > > > Key: HBASE-17658 > URL: https://issues.apache.org/jira/browse/HBASE-17658 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Tim Brown >Assignee: Tim Brown > Attachments: JIRA-17658.master.001.patch > > > Currently when numMaxRegionsPerTable of a given table increases above the > current maximum, the value is not properly updated. This means that the cost > for Table Skew cannot get worse for a given region move during the balancer > process leading to an imbalanced cluster with respect to Table Skew. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17653) HBASE-17624 rsgroup synchronizations will (distributed) deadlock
[
https://issues.apache.org/jira/browse/HBASE-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17653:
--
Attachment: HBASE-17653.master.002.patch
> HBASE-17624 rsgroup synchronizations will (distributed) deadlock
>
>
> Key: HBASE-17653
> URL: https://issues.apache.org/jira/browse/HBASE-17653
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Reporter: stack
>Assignee: stack
> Attachments: HBASE-17653.master.001.patch,
> HBASE-17653.master.002.patch
>
>
> Follow-on from HBASE-17624. HBASE-17624 made it so one thread only has access
> to the rsgroup administrator. In tail of HBASE-17624 [~toffer] describes
> scenario under which we may end up in a deadlock (distributed). Let me
> repeat [~toffer] comment...
> {code}
> Both read/write access can't be single threaded. Consider the situation:
> 1. move_rsgroup_servers is called
> 2. while #1 is happening rsgroup region is in transition (rpc thread in #1
> holds monitor lock)
> 3. while #2 is happening meta is in transition.
> Balancer tries to figure out plan for meta region tries to get monitor lock
> but can't. rpc thread task won't release monitor lock since rsgroup region
> never gets assigned. rsgroup region never gets assigned because it can't
> update meta with new state.
> There's a good chance this can be reproduce just by moving both rsgroup and
> meta region onto the same RS and call move_rsgoup_servers on the same RS.
> A bunch different actors will query from group affiliation so we can't have
> writes block reads.
>
> In the code prior to this patch the getter methods that retrieve group
> information (getRSGroup, ofTable, OfServer, etc) don't require the monitor
> lock so the deadlock cycle is broken.
>
> The methods that does mutations and updates to zk and hbase:rsgroup are
> synchronized appropriately. Point me to where the incoherence is?
> {code}
> This issue is about testing/fixing/restoring rsgroup access. Will be back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17658) Bookkeeping issue in BaseLoadBalancer for Table Skew
[ https://issues.apache.org/jira/browse/HBASE-17658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Brown updated HBASE-17658: -- Status: Patch Available (was: In Progress) > Bookkeeping issue in BaseLoadBalancer for Table Skew > > > Key: HBASE-17658 > URL: https://issues.apache.org/jira/browse/HBASE-17658 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Tim Brown >Assignee: Tim Brown > Attachments: JIRA-17658.master.001.patch > > > Currently when numMaxRegionsPerTable of a given table increases above the > current maximum, the value is not properly updated. This means that the cost > for Table Skew cannot get worse for a given region move during the balancer > process leading to an imbalanced cluster with respect to Table Skew. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (HBASE-17658) Bookkeeping issue in BaseLoadBalancer for Table Skew
[ https://issues.apache.org/jira/browse/HBASE-17658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-17658 started by Tim Brown. - > Bookkeeping issue in BaseLoadBalancer for Table Skew > > > Key: HBASE-17658 > URL: https://issues.apache.org/jira/browse/HBASE-17658 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Tim Brown >Assignee: Tim Brown > Attachments: JIRA-17658.master.001.patch > > > Currently when numMaxRegionsPerTable of a given table increases above the > current maximum, the value is not properly updated. This means that the cost > for Table Skew cannot get worse for a given region move during the balancer > process leading to an imbalanced cluster with respect to Table Skew. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17658) Bookkeeping issue in BaseLoadBalancer for Table Skew
[ https://issues.apache.org/jira/browse/HBASE-17658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Brown updated HBASE-17658: -- Attachment: JIRA-17658.master.001.patch > Bookkeeping issue in BaseLoadBalancer for Table Skew > > > Key: HBASE-17658 > URL: https://issues.apache.org/jira/browse/HBASE-17658 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Tim Brown >Assignee: Tim Brown > Attachments: JIRA-17658.master.001.patch > > > Currently when numMaxRegionsPerTable of a given table increases above the > current maximum, the value is not properly updated. This means that the cost > for Table Skew cannot get worse for a given region move during the balancer > process leading to an imbalanced cluster with respect to Table Skew. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17656) Move new Address class from util to net package
[ https://issues.apache.org/jira/browse/HBASE-17656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17656: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed minor issue rename of a class to master branch. > Move new Address class from util to net package > --- > > Key: HBASE-17656 > URL: https://issues.apache.org/jira/browse/HBASE-17656 > Project: HBase > Issue Type: Sub-task > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17656.master.001.patch > > > I added the new Address type in the wrong package committing the parent JIRA > patch. It should be in net, not util. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17653) HBASE-17624 rsgroup synchronizations will (distributed) deadlock
[
https://issues.apache.org/jira/browse/HBASE-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17653:
--
Attachment: HBASE-17653.master.001.patch
> HBASE-17624 rsgroup synchronizations will (distributed) deadlock
>
>
> Key: HBASE-17653
> URL: https://issues.apache.org/jira/browse/HBASE-17653
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Reporter: stack
>Assignee: stack
> Attachments: HBASE-17653.master.001.patch
>
>
> Follow-on from HBASE-17624. HBASE-17624 made it so one thread only has access
> to the rsgroup administrator. In tail of HBASE-17624 [~toffer] describes
> scenario under which we may end up in a deadlock (distributed). Let me
> repeat [~toffer] comment...
> {code}
> Both read/write access can't be single threaded. Consider the situation:
> 1. move_rsgroup_servers is called
> 2. while #1 is happening rsgroup region is in transition (rpc thread in #1
> holds monitor lock)
> 3. while #2 is happening meta is in transition.
> Balancer tries to figure out plan for meta region tries to get monitor lock
> but can't. rpc thread task won't release monitor lock since rsgroup region
> never gets assigned. rsgroup region never gets assigned because it can't
> update meta with new state.
> There's a good chance this can be reproduce just by moving both rsgroup and
> meta region onto the same RS and call move_rsgoup_servers on the same RS.
> A bunch different actors will query from group affiliation so we can't have
> writes block reads.
>
> In the code prior to this patch the getter methods that retrieve group
> information (getRSGroup, ofTable, OfServer, etc) don't require the monitor
> lock so the deadlock cycle is broken.
>
> The methods that does mutations and updates to zk and hbase:rsgroup are
> synchronized appropriately. Point me to where the incoherence is?
> {code}
> This issue is about testing/fixing/restoring rsgroup access. Will be back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17653) HBASE-17624 rsgroup synchronizations will (distributed) deadlock
[
https://issues.apache.org/jira/browse/HBASE-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870919#comment-15870919
]
stack commented on HBASE-17653:
---
[~toffer] Please review. Sorry for making you work.
> HBASE-17624 rsgroup synchronizations will (distributed) deadlock
>
>
> Key: HBASE-17653
> URL: https://issues.apache.org/jira/browse/HBASE-17653
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Reporter: stack
>Assignee: stack
> Attachments: HBASE-17653.master.001.patch
>
>
> Follow-on from HBASE-17624. HBASE-17624 made it so one thread only has access
> to the rsgroup administrator. In tail of HBASE-17624 [~toffer] describes
> scenario under which we may end up in a deadlock (distributed). Let me
> repeat [~toffer] comment...
> {code}
> Both read/write access can't be single threaded. Consider the situation:
> 1. move_rsgroup_servers is called
> 2. while #1 is happening rsgroup region is in transition (rpc thread in #1
> holds monitor lock)
> 3. while #2 is happening meta is in transition.
> Balancer tries to figure out plan for meta region tries to get monitor lock
> but can't. rpc thread task won't release monitor lock since rsgroup region
> never gets assigned. rsgroup region never gets assigned because it can't
> update meta with new state.
> There's a good chance this can be reproduce just by moving both rsgroup and
> meta region onto the same RS and call move_rsgoup_servers on the same RS.
> A bunch different actors will query from group affiliation so we can't have
> writes block reads.
>
> In the code prior to this patch the getter methods that retrieve group
> information (getRSGroup, ofTable, OfServer, etc) don't require the monitor
> lock so the deadlock cycle is broken.
>
> The methods that does mutations and updates to zk and hbase:rsgroup are
> synchronized appropriately. Point me to where the incoherence is?
> {code}
> This issue is about testing/fixing/restoring rsgroup access. Will be back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17656) Move new Address class from util to net package
[
https://issues.apache.org/jira/browse/HBASE-17656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870855#comment-15870855
]
Hadoop QA commented on HBASE-17656:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 47s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 58s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 101m 15s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 48s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 56s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 170m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.replication.regionserver.TestTableBasedReplicationSourceManagerImpl
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853122/HBASE-17656.master.001.patch
|
| JIRA Issue | HBASE-17656 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile rubocop ruby_lint |
| uname
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14123: -- Attachment: Backup-restoreinHBase2.0 (1).pdf > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: HBASE-7912 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v20.txt, 14123-master.v21.txt, > 14123-master.v24.txt, 14123-master.v25.txt, 14123-master.v27.txt, > 14123-master.v28.txt, 14123-master.v29.full.txt, 14123-master.v2.txt, > 14123-master.v30.txt, 14123-master.v31.txt, 14123-master.v32.txt, > 14123-master.v33.txt, 14123-master.v34.txt, 14123-master.v35.txt, > 14123-master.v36.txt, 14123-master.v37.txt, 14123-master.v38.txt, > 14123.master.v39.patch, 14123-master.v3.txt, 14123.master.v40.patch, > 14123.master.v41.patch, 14123.master.v42.patch, 14123.master.v44.patch, > 14123.master.v45.patch, 14123.master.v46.patch, 14123.master.v48.patch, > 14123.master.v49.patch, 14123.master.v50.patch, 14123.master.v51.patch, > 14123.master.v52.patch, 14123.master.v54.patch, 14123.master.v56.patch, > 14123.master.v57.patch, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > Backup-restoreinHBase2.0 (1).pdf, Backup-restoreinHBase2.0.pdf, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v10.patch, HBASE-14123-v11.patch, HBASE-14123-v12.patch, > HBASE-14123-v13.patch, HBASE-14123-v15.patch, HBASE-14123-v16.patch, > HBASE-14123-v1.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870851#comment-15870851
]
Vladimir Rodionov commented on HBASE-14123:
---
{quote}
You should also mention https://issues.apache.org/jira/browse/HBASE-16825 in
the doc.
{quote}
This JIRA is linked to Phase 3 and is not resolved yet - meaning this is what
we need to finish before 2.0 release.
> HBase Backup/Restore Phase 2
>
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
> Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Fix For: HBASE-7912
>
> Attachments: 14123-master.v14.txt, 14123-master.v15.txt,
> 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt,
> 14123-master.v19.txt, 14123-master.v20.txt, 14123-master.v21.txt,
> 14123-master.v24.txt, 14123-master.v25.txt, 14123-master.v27.txt,
> 14123-master.v28.txt, 14123-master.v29.full.txt, 14123-master.v2.txt,
> 14123-master.v30.txt, 14123-master.v31.txt, 14123-master.v32.txt,
> 14123-master.v33.txt, 14123-master.v34.txt, 14123-master.v35.txt,
> 14123-master.v36.txt, 14123-master.v37.txt, 14123-master.v38.txt,
> 14123.master.v39.patch, 14123-master.v3.txt, 14123.master.v40.patch,
> 14123.master.v41.patch, 14123.master.v42.patch, 14123.master.v44.patch,
> 14123.master.v45.patch, 14123.master.v46.patch, 14123.master.v48.patch,
> 14123.master.v49.patch, 14123.master.v50.patch, 14123.master.v51.patch,
> 14123.master.v52.patch, 14123.master.v54.patch, 14123.master.v56.patch,
> 14123.master.v57.patch, 14123-master.v5.txt, 14123-master.v6.txt,
> 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt,
> Backup-restoreinHBase2.0.pdf, HBASE-14123-for-7912-v1.patch,
> HBASE-14123-for-7912-v6.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch,
> HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch,
> HBASE-14123-v16.patch, HBASE-14123-v1.patch, HBASE-14123-v2.patch,
> HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch,
> HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870847#comment-15870847
]
Vladimir Rodionov commented on HBASE-14123:
---
{quote}
Is the patch here contains all of the jiras linked to HBASE-14414 and resolved?
{quote}
Yes, all of them are in the patch. I should have added HBASE-14030 (Phase 1) to
the list of JIRAs as well.
> HBase Backup/Restore Phase 2
>
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
> Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Fix For: HBASE-7912
>
> Attachments: 14123-master.v14.txt, 14123-master.v15.txt,
> 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt,
> 14123-master.v19.txt, 14123-master.v20.txt, 14123-master.v21.txt,
> 14123-master.v24.txt, 14123-master.v25.txt, 14123-master.v27.txt,
> 14123-master.v28.txt, 14123-master.v29.full.txt, 14123-master.v2.txt,
> 14123-master.v30.txt, 14123-master.v31.txt, 14123-master.v32.txt,
> 14123-master.v33.txt, 14123-master.v34.txt, 14123-master.v35.txt,
> 14123-master.v36.txt, 14123-master.v37.txt, 14123-master.v38.txt,
> 14123.master.v39.patch, 14123-master.v3.txt, 14123.master.v40.patch,
> 14123.master.v41.patch, 14123.master.v42.patch, 14123.master.v44.patch,
> 14123.master.v45.patch, 14123.master.v46.patch, 14123.master.v48.patch,
> 14123.master.v49.patch, 14123.master.v50.patch, 14123.master.v51.patch,
> 14123.master.v52.patch, 14123.master.v54.patch, 14123.master.v56.patch,
> 14123.master.v57.patch, 14123-master.v5.txt, 14123-master.v6.txt,
> 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt,
> Backup-restoreinHBase2.0.pdf, HBASE-14123-for-7912-v1.patch,
> HBASE-14123-for-7912-v6.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch,
> HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch,
> HBASE-14123-v16.patch, HBASE-14123-v1.patch, HBASE-14123-v2.patch,
> HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch,
> HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870831#comment-15870831
]
Ted Yu commented on HBASE-17623:
Got some compilation errors applying patch on branch-1:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile)
on project hbase-common: Compilation failure: Compilation failure:
[ERROR]
/Users/tyu/1-hbase/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java:[194,15]
error: constructor Bytes in class Bytes cannot be applied to given types;
[ERROR] int,int
[ERROR] reason: actual and formal argument lists differ in length
[ERROR]
/Users/tyu/1-hbase/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java:[198,15]
error: constructor Bytes in class Bytes cannot be applied to given types;
[ERROR] int,int
[ERROR] reason: actual and formal argument lists differ in length
[ERROR]
/Users/tyu/1-hbase/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java:[212,15]
error: constructor Bytes in class Bytes cannot be applied to given types;
{code}
Chiaping:
Can you attach patch for branch-1 ?
Thanks
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.v0.patch,
> HBASE-17623.v1.patch, HBASE-17623.v1.patch, memory allocation measurement.xlsx
>
>
> There are two improvements.
> # The uncompressedBlockBytesWithHeader and onDiskBlockBytesWithHeader should
> maintain a bytes array which can be reused when building the hfile.
> # The uncompressedBlockBytesWithHeader/onDiskBlockBytesWithHeader is copied
> to an new bytes array only when we need to cache the block.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17172) Optimize mob compaction with _del files
[
https://issues.apache.org/jira/browse/HBASE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870829#comment-15870829
]
Hadoop QA commented on HBASE-17172:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 3s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 16s
{color} | {color:red} hbase-server generated 2 new + 0 unchanged - 0 fixed = 2
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 6s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 6s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Invocation of hashCode on array in
org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactionRequest$CompactionDelPartitionId.hashCode()
At PartitionedMobCompactionRequest.java:in
org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactionRequest$CompactionDelPartitionId.hashCode()
At PartitionedMobCompactionRequest.java:[line 317] |
| | Invocation of hashCode on array in
org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactionRequest$CompactionDelPartitionId.hashCode()
At PartitionedMobCompactionRequest.java:in
org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactionRequest$CompactionDelPartitionId.hashCode()
At PartitionedMobCompactionRequest.java:[line 316] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853127/HBASE-17172.master.002.patch
|
| JIRA Issue | HBASE-17172 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 8910d2e18c5d 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0446ed0 |
| Default Java | 1.8.0_121 |
| findbugs
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870818#comment-15870818 ] stack commented on HBASE-14123: --- Not yet @enis (and [~vrodionov]). It is next on my list. Almost there. Sorry for delay. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: HBASE-7912 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v20.txt, 14123-master.v21.txt, > 14123-master.v24.txt, 14123-master.v25.txt, 14123-master.v27.txt, > 14123-master.v28.txt, 14123-master.v29.full.txt, 14123-master.v2.txt, > 14123-master.v30.txt, 14123-master.v31.txt, 14123-master.v32.txt, > 14123-master.v33.txt, 14123-master.v34.txt, 14123-master.v35.txt, > 14123-master.v36.txt, 14123-master.v37.txt, 14123-master.v38.txt, > 14123.master.v39.patch, 14123-master.v3.txt, 14123.master.v40.patch, > 14123.master.v41.patch, 14123.master.v42.patch, 14123.master.v44.patch, > 14123.master.v45.patch, 14123.master.v46.patch, 14123.master.v48.patch, > 14123.master.v49.patch, 14123.master.v50.patch, 14123.master.v51.patch, > 14123.master.v52.patch, 14123.master.v54.patch, 14123.master.v56.patch, > 14123.master.v57.patch, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > Backup-restoreinHBase2.0.pdf, HBASE-14123-for-7912-v1.patch, > HBASE-14123-for-7912-v6.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v1.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17312) [JDK8] Use default method for Observer Coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870809#comment-15870809 ] Andrew Purtell commented on HBASE-17312: No discussion needed. My answer is based on a lack of discussion actually. Look at what's supported on master. Note the lack of discussion about 2.0. :-) Sure, Phoenix won't be on 1.x forever. When they move to 2.x there will be a lot of work needed regardless, as you point out [~enis] > [JDK8] Use default method for Observer Coprocessors > --- > > Key: HBASE-17312 > URL: https://issues.apache.org/jira/browse/HBASE-17312 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17312.master.001.patch, > HBASE-17312.master.001.patch > > > Use default method in MasterObserver, RegionObserver, RegionServerObserver > and WALObserver. And mark the BaseRegionObserver, > BaseMasterAndRegionObserver, BaseRegionServerObserver and BaseWALObserver. > User can implement the interface directly and will not break compatibility > when add new default methods. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17312) [JDK8] Use default method for Observer Coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870811#comment-15870811 ] Zach York commented on HBASE-17312: --- bq. Phoenix has no support for master branch and no development for same currently planned. Also, no plans to move up to JDK8 even. I think when we get close to releasing HBase 2.0.0, Phoenix HBase branch-2.0.0 work/JDK 8 work should be discussed/start. I think it is okay to create JIRAs for that now in case people want to pick it up. In general, I think using default methods here is a good approach. (Much better to do it and remove the Base classes now than having them deprecated for all of the 2.x line). [~zghaobac] Can you post this on Review Board? I would like to review, but it is hard to review a patch. > [JDK8] Use default method for Observer Coprocessors > --- > > Key: HBASE-17312 > URL: https://issues.apache.org/jira/browse/HBASE-17312 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17312.master.001.patch, > HBASE-17312.master.001.patch > > > Use default method in MasterObserver, RegionObserver, RegionServerObserver > and WALObserver. And mark the BaseRegionObserver, > BaseMasterAndRegionObserver, BaseRegionServerObserver and BaseWALObserver. > User can implement the interface directly and will not break compatibility > when add new default methods. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17657) TestZKAsyncRegistry is flaky
[
https://issues.apache.org/jira/browse/HBASE-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870805#comment-15870805
]
Hadoop QA commented on HBASE-17657:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 4s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 57s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 6s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 52s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853123/17657.v1.txt |
| JIRA Issue | HBASE-17657 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4e5c847f8e1e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 0446ed0 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5739/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5739/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TestZKAsyncRegistry is flaky
> -
>
> Key: HBASE-17657
> URL: https://issues.apache.org/jira/browse/HBASE-17657
> Project: HBase
> Issue
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870799#comment-15870799 ] Enis Soztutar commented on HBASE-14123: --- Thanks for the write up Vladimir. bq. The full list of JIRAs included in this mega patch can be found in two umbrella JIRAs: HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 - all resolved ones) You want to also mention the jiras in the Phase 1 issue? You should also mention https://issues.apache.org/jira/browse/HBASE-16825 in the doc. bq. HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 - all resolved ones) Is the patch here contains all of the jiras linked to HBASE-14414 and resolved? I thought that there are some which are resolved (and in branch), but not in the current patch. I'am +1 for the commit, but we have to create a [VOTE] thread on the dev mailing list for this merge. [~saint@gmail.com] did you get a chance to take a look? > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: HBASE-7912 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v20.txt, 14123-master.v21.txt, > 14123-master.v24.txt, 14123-master.v25.txt, 14123-master.v27.txt, > 14123-master.v28.txt, 14123-master.v29.full.txt, 14123-master.v2.txt, > 14123-master.v30.txt, 14123-master.v31.txt, 14123-master.v32.txt, > 14123-master.v33.txt, 14123-master.v34.txt, 14123-master.v35.txt, > 14123-master.v36.txt, 14123-master.v37.txt, 14123-master.v38.txt, > 14123.master.v39.patch, 14123-master.v3.txt, 14123.master.v40.patch, > 14123.master.v41.patch, 14123.master.v42.patch, 14123.master.v44.patch, > 14123.master.v45.patch, 14123.master.v46.patch, 14123.master.v48.patch, > 14123.master.v49.patch, 14123.master.v50.patch, 14123.master.v51.patch, > 14123.master.v52.patch, 14123.master.v54.patch, 14123.master.v56.patch, > 14123.master.v57.patch, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > Backup-restoreinHBase2.0.pdf, HBASE-14123-for-7912-v1.patch, > HBASE-14123-for-7912-v6.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v1.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870797#comment-15870797 ] huaxiang sun commented on HBASE-16981: -- I updated the release notes, [~jingcheng.du], anything to update? Thanks. > Expand Mob Compaction Partition policy from daily to weekly, monthly > > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0 > > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, HBASE-16981.master.005.patch, > HBASE-16981.master.006.patch, HBASE-16981.master.007.patch, > HBASE-16981.master.008.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly
[
https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huaxiang sun updated HBASE-16981:
-
Release Note:
Mob compaction partition policy can be set by
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 100,
MOB_COMPACT_PARTITION_POLICY => 'weekly'}
or
hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 100,
MOB_COMPACT_PARTITION_POLICY => 'monthly'}
Available MOB_COMPACT_PARTITION_POLICY options are "daily", "weekly" and
"monthly", the default is "daily".
When it is "weekly" policy, the mob compaction will try to compact files within
one calendar week into one for a specific partition, similar for "daily" and
"monthly".
With "weekly" policy, one mob file normally is compacted twice during its
lifetime, for one region, there normally are 52 files for one year. With
"Monthly" policy, one mob file normally is compacted 3 times during its
lifetime and normally there are 12 files for one year.
was:
Mob compaction partition policy can be set by
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 100,
MOB_COMPACT_PARTITION_POLICY => 'weekly'}
or
hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 100,
MOB_COMPACT_PARTITION_POLICY => 'monthly'}
Available MOB_COMPACT_PARTITION_POLICY options are "daily", "weekly" and
"monthly", the default is "daily".
When it is "weekly" policy, the mob compaction will try to compact files within
one calendar week into one for a specific partition, similar for "daily" and
"monthly".
With "weekly" policy, one mob file normally is compacted twice during its
lifetime, for one region, there normally are 52 files for one year. With
"Monthly" policy, one mob file normally is compacted 3 times during its
lifetime and normally the are 12 files for one year.
> Expand Mob Compaction Partition policy from daily to weekly, monthly
>
>
> Key: HBASE-16981
> URL: https://issues.apache.org/jira/browse/HBASE-16981
> Project: HBase
> Issue Type: New Feature
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Fix For: 2.0.0
>
> Attachments: HBASE-16981.master.001.patch,
> HBASE-16981.master.002.patch, HBASE-16981.master.003.patch,
> HBASE-16981.master.004.patch, HBASE-16981.master.005.patch,
> HBASE-16981.master.006.patch, HBASE-16981.master.007.patch,
> HBASE-16981.master.008.patch,
> Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf
>
>
> Today the mob region holds all mob files for all regions. With daily
> partition mob compaction policy, after major mob compaction, there is still
> one file per region daily. Given there is 365 days in one year, at least 365
> files per region. Since HDFS has limitation for number of files under one
> folder, this is not going to scale if there are lots of regions. To reduce
> mob file number, we want to introduce other partition policies such as
> weekly, monthly to compact mob files within one week or month into one file.
> This jira is create to track this effort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly
[
https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huaxiang sun updated HBASE-16981:
-
Release Note:
Mob compaction partition policy can be set by
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 100,
MOB_COMPACT_PARTITION_POLICY => 'weekly'}
or
hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 100,
MOB_COMPACT_PARTITION_POLICY => 'monthly'}
Available MOB_COMPACT_PARTITION_POLICY options are "daily", "weekly" and
"monthly", the default is "daily".
When it is "weekly" policy, the mob compaction will try to compact files within
one calendar week into one for a specific partition, similar for "daily" and
"monthly".
With "weekly" policy, one mob file normally is compacted twice during its
lifetime, for one region, there normally are 52 files for one year. With
"Monthly" policy, one mob file normally is compacted 3 times during its
lifetime and normally the are 12 files for one year.
> Expand Mob Compaction Partition policy from daily to weekly, monthly
>
>
> Key: HBASE-16981
> URL: https://issues.apache.org/jira/browse/HBASE-16981
> Project: HBase
> Issue Type: New Feature
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Fix For: 2.0.0
>
> Attachments: HBASE-16981.master.001.patch,
> HBASE-16981.master.002.patch, HBASE-16981.master.003.patch,
> HBASE-16981.master.004.patch, HBASE-16981.master.005.patch,
> HBASE-16981.master.006.patch, HBASE-16981.master.007.patch,
> HBASE-16981.master.008.patch,
> Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf
>
>
> Today the mob region holds all mob files for all regions. With daily
> partition mob compaction policy, after major mob compaction, there is still
> one file per region daily. Given there is 365 days in one year, at least 365
> files per region. Since HDFS has limitation for number of files under one
> folder, this is not going to scale if there are lots of regions. To reduce
> mob file number, we want to introduce other partition policies such as
> weekly, monthly to compact mob files within one week or month into one file.
> This jira is create to track this effort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17472) Correct the semantic of permission grant
[
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870790#comment-15870790
]
Enis Soztutar commented on HBASE-17472:
---
please remove all mentions of "grant2" from the method names, etc.
> Correct the semantic of permission grant
> -
>
> Key: HBASE-17472
> URL: https://issues.apache.org/jira/browse/HBASE-17472
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Affects Versions: 2.0.0, 1.4.0
>Reporter: huzheng
>Assignee: huzheng
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17472.branch-1.3.v6.patch,
> HBASE-17472.branch-1.v6.patch, HBASE-17472.master.v6.patch,
> HBASE-17472.master.v6.patch, HBASE-17472.v1.patch, HBASE-17472.v2.patch,
> HBASE-17472.v3.patch, HBASE-17472.v4.patch, HBASE-17472.v5.patch
>
>
> Currently, HBase grant operation has following semantic:
> {code}
> hbase(main):019:0> grant 'hbase_tst', 'RW', 'ycsb'
> 0 row(s) in 0.0960 seconds
> hbase(main):020:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
>
> hbase_tst default,ycsb,,:
> [Permission:actions=READ,WRITE]
>
>
> 1 row(s) in 0.0550 seconds
> hbase(main):021:0> grant 'hbase_tst', 'CA', 'ycsb'
> 0 row(s) in 0.0820 seconds
> hbase(main):022:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
> hbase_tst default,ycsb,,:
> [Permission: actions=CREATE,ADMIN]
>
>
> 1 row(s) in 0.0490 seconds
> {code}
> Later permission will replace previous granted permissions, which confused
> most of HBase administrator.
> It's seems more reasonable that HBase merge multiple granted permission.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17312) [JDK8] Use default method for Observer Coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870784#comment-15870784 ] Enis Soztutar commented on HBASE-17312: --- bq. Phoenix has no support for master branch and no development for same currently planned. Also, no plans to move up to JDK8 even. I don't remember seeing that kind of discussion on the mailing list. Regardless, it does not mean that Phoenix will be stuck forever on the 1.x line. My understanding is that you have to fix your coprocessor anyway between 1.x and 2.0 (because of other API changes). Removing the base classes should be fine for now. > [JDK8] Use default method for Observer Coprocessors > --- > > Key: HBASE-17312 > URL: https://issues.apache.org/jira/browse/HBASE-17312 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17312.master.001.patch, > HBASE-17312.master.001.patch > > > Use default method in MasterObserver, RegionObserver, RegionServerObserver > and WALObserver. And mark the BaseRegionObserver, > BaseMasterAndRegionObserver, BaseRegionServerObserver and BaseWALObserver. > User can implement the interface directly and will not break compatibility > when add new default methods. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17649) REST API for scan should return 410 when table is disabled
[ https://issues.apache.org/jira/browse/HBASE-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870768#comment-15870768 ] Hudson commented on HBASE-17649: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #102 (See [https://builds.apache.org/job/HBase-1.2-JDK7/102/]) HBASE-17649 REST API for scan should return 410 when table is disabled (tedyu: rev 81c0f24b9439d3393c7748b62e97e4117c4997d7) * (edit) hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestScannerResource.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java > REST API for scan should return 410 when table is disabled > -- > > Key: HBASE-17649 > URL: https://issues.apache.org/jira/browse/HBASE-17649 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5 > > Attachments: 17649.branch-1.3.v1.txt, 17649.v1.txt > > > Background: > When scanning using REST API, if the table is disabled in the middle of the > scan, code 410 (Gone) should be returned. > Currently in 1.3 and lower releases, ScannerResultGenerator#next() doesn't > handle TableNotEnabledException, leading to the exception being ignored and > code 204 is returned to caller. > This was first spotted by [~Apache9] when investigating HBASE-17603. > Patch for HBASE-17603 contains fix for this problem. > This issue fixes the bug for 1.3 and lower branches and adds test for > branch-1 and master branch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-16431) Add missing method in class HTableWrapper
[ https://issues.apache.org/jira/browse/HBASE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-16431. - Resolution: Cannot Reproduce > Add missing method in class HTableWrapper > - > > Key: HBASE-16431 > URL: https://issues.apache.org/jira/browse/HBASE-16431 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.2 >Reporter: Biao Ma > Fix For: 1.2.2 > > > Class `HTableWrapper` must either declared abstract or implement abstract > method 'setRpcTimeout(int)' in 'Table' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (HBASE-16431) Add missing method in class HTableWrapper
[ https://issues.apache.org/jira/browse/HBASE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reopened HBASE-16431: - > Add missing method in class HTableWrapper > - > > Key: HBASE-16431 > URL: https://issues.apache.org/jira/browse/HBASE-16431 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.2 >Reporter: Biao Ma > Fix For: 1.2.2 > > > Class `HTableWrapper` must either declared abstract or implement abstract > method 'setRpcTimeout(int)' in 'Table' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17649) REST API for scan should return 410 when table is disabled
[ https://issues.apache.org/jira/browse/HBASE-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870716#comment-15870716 ] Hudson commented on HBASE-17649: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #95 (See [https://builds.apache.org/job/HBase-1.2-JDK8/95/]) HBASE-17649 REST API for scan should return 410 when table is disabled (tedyu: rev 81c0f24b9439d3393c7748b62e97e4117c4997d7) * (edit) hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestScannerResource.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java > REST API for scan should return 410 when table is disabled > -- > > Key: HBASE-17649 > URL: https://issues.apache.org/jira/browse/HBASE-17649 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5 > > Attachments: 17649.branch-1.3.v1.txt, 17649.v1.txt > > > Background: > When scanning using REST API, if the table is disabled in the middle of the > scan, code 410 (Gone) should be returned. > Currently in 1.3 and lower releases, ScannerResultGenerator#next() doesn't > handle TableNotEnabledException, leading to the exception being ignored and > code 204 is returned to caller. > This was first spotted by [~Apache9] when investigating HBASE-17603. > Patch for HBASE-17603 contains fix for this problem. > This issue fixes the bug for 1.3 and lower branches and adds test for > branch-1 and master branch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17658) Bookkeeping issue in BaseLoadBalancer for Table Skew
Tim Brown created HBASE-17658: - Summary: Bookkeeping issue in BaseLoadBalancer for Table Skew Key: HBASE-17658 URL: https://issues.apache.org/jira/browse/HBASE-17658 Project: HBase Issue Type: Bug Components: Balancer Reporter: Tim Brown Assignee: Tim Brown Currently when numMaxRegionsPerTable of a given table increases above the current maximum, the value is not properly updated. This means that the cost for Table Skew cannot get worse for a given region move during the balancer process leading to an imbalanced cluster with respect to Table Skew. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17044) Fix merge failed before creating merged region leaves meta inconsistent
[ https://issues.apache.org/jira/browse/HBASE-17044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-17044: Priority: Critical (was: Major) > Fix merge failed before creating merged region leaves meta inconsistent > --- > > Key: HBASE-17044 > URL: https://issues.apache.org/jira/browse/HBASE-17044 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-17044-0.98.patch, HBASE-17044-branch-1.2.patch, > HBASE-17044-branch-1.patch, HBASE-17044.patch > > > Similar to HBASE-16093. Rollback from failed merge should not offline the > target region for the merge if it hasn't been created. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17653) HBASE-17624 rsgroup synchronizations will (distributed) deadlock
[
https://issues.apache.org/jira/browse/HBASE-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870703#comment-15870703
]
stack commented on HBASE-17653:
---
Thanks [~toffer]
bq. BTW you guys could've just asked me instead of trying to read my mind .
I tried:
https://issues.apache.org/jira/browse/HBASE-17624?focusedCommentId=15862239=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15862239
... a few times. I should have written you direct to get your attention .
Sorry about that.
bq. Same reason reads should not be blocked by mutations.
Intent seems to be a CopyOnWrite strategy.
Let me make this explicit and do a pass... Pardon my over-enthusiasm.
> HBASE-17624 rsgroup synchronizations will (distributed) deadlock
>
>
> Key: HBASE-17653
> URL: https://issues.apache.org/jira/browse/HBASE-17653
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Reporter: stack
>Assignee: stack
>
> Follow-on from HBASE-17624. HBASE-17624 made it so one thread only has access
> to the rsgroup administrator. In tail of HBASE-17624 [~toffer] describes
> scenario under which we may end up in a deadlock (distributed). Let me
> repeat [~toffer] comment...
> {code}
> Both read/write access can't be single threaded. Consider the situation:
> 1. move_rsgroup_servers is called
> 2. while #1 is happening rsgroup region is in transition (rpc thread in #1
> holds monitor lock)
> 3. while #2 is happening meta is in transition.
> Balancer tries to figure out plan for meta region tries to get monitor lock
> but can't. rpc thread task won't release monitor lock since rsgroup region
> never gets assigned. rsgroup region never gets assigned because it can't
> update meta with new state.
> There's a good chance this can be reproduce just by moving both rsgroup and
> meta region onto the same RS and call move_rsgoup_servers on the same RS.
> A bunch different actors will query from group affiliation so we can't have
> writes block reads.
>
> In the code prior to this patch the getter methods that retrieve group
> information (getRSGroup, ofTable, OfServer, etc) don't require the monitor
> lock so the deadlock cycle is broken.
>
> The methods that does mutations and updates to zk and hbase:rsgroup are
> synchronized appropriately. Point me to where the incoherence is?
> {code}
> This issue is about testing/fixing/restoring rsgroup access. Will be back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17655) Removing MemStoreScanner and SnapshotScanner
[
https://issues.apache.org/jira/browse/HBASE-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870679#comment-15870679
]
Hadoop QA commented on HBASE-17655:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 4s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 17s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hbase-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 27s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.locking.TestLockManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853117/HBASE-17655-V01.patch
|
| JIRA Issue | HBASE-17655 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c719ff17ad48 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0446ed0 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5738/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5738/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-16850) Run large scale correctness tests for HBASE-14918 (in-memory flushes/compactions)
[ https://issues.apache.org/jira/browse/HBASE-16850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870672#comment-15870672 ] Edward Bortnikov commented on HBASE-16850: -- Large-scale benchmark results are reported in HBASE-16417. Makes sense to redirect/retire this Jira? Thanks. > Run large scale correctness tests for HBASE-14918 (in-memory > flushes/compactions) > - > > Key: HBASE-16850 > URL: https://issues.apache.org/jira/browse/HBASE-16850 > Project: HBase > Issue Type: Sub-task >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Blocker > > As discussed here - > https://issues.apache.org/jira/browse/HBASE-16608?focusedCommentId=15577213=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15577213 > [~stack] [~anastas] [~ram_krish] [~anoop.hbase] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17172) Optimize mob compaction with _del files
[ https://issues.apache.org/jira/browse/HBASE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870654#comment-15870654 ] huaxiang sun commented on HBASE-17172: -- V2 patch addressed Ted's comments and warnings. > Optimize mob compaction with _del files > --- > > Key: HBASE-17172 > URL: https://issues.apache.org/jira/browse/HBASE-17172 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-17172-master-001.patch, > HBASE-17172.master.001.patch, HBASE-17172.master.002.patch > > > Today, when there is a _del file in mobdir, with major mob compaction, every > mob file will be recompacted, this causes lots of IO and slow down major mob > compaction (may take months to finish). This needs to be improved. A few > ideas are: > 1) Do not compact all _del files into one, instead, compact them based on > groups with startKey as the key. Then use firstKey/startKey to make each mob > file to see if the _del file needs to be included for this partition. > 2). Based on the timerange of the _del file, compaction for files after that > timerange does not need to include the _del file as these are newer files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17172) Optimize mob compaction with _del files
[ https://issues.apache.org/jira/browse/HBASE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-17172: - Attachment: HBASE-17172.master.002.patch > Optimize mob compaction with _del files > --- > > Key: HBASE-17172 > URL: https://issues.apache.org/jira/browse/HBASE-17172 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-17172-master-001.patch, > HBASE-17172.master.001.patch, HBASE-17172.master.002.patch > > > Today, when there is a _del file in mobdir, with major mob compaction, every > mob file will be recompacted, this causes lots of IO and slow down major mob > compaction (may take months to finish). This needs to be improved. A few > ideas are: > 1) Do not compact all _del files into one, instead, compact them based on > groups with startKey as the key. Then use firstKey/startKey to make each mob > file to see if the _del file needs to be included for this partition. > 2). Based on the timerange of the _del file, compaction for files after that > timerange does not need to include the _del file as these are newer files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17657) TestZKAsyncRegistry is flaky
[
https://issues.apache.org/jira/browse/HBASE-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17657:
---
Status: Patch Available (was: Open)
> TestZKAsyncRegistry is flaky
> -
>
> Key: HBASE-17657
> URL: https://issues.apache.org/jira/browse/HBASE-17657
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17657.v1.txt
>
>
> TestZKAsyncRegistry showed up in failed tests several times.
> On
> https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
> , TestZKAsyncRegistry is reported flaky 33% of the time.
> e.g.
> https://builds.apache.org/job/PreCommit-HBASE-Build/5708/testReport/org.apache.hadoop.hbase.client/TestZKAsyncRegistry/test/
> Toward the end of test output:
> {code}
> 2017-02-14 20:23:20,779 WARN [main-EventThread] client.ZKAsyncRegistry(198):
> Meta region for replica 2 is in state PENDING_OPEN
> 2017-02-14 20:23:20,800 INFO [98410feec74d:45445.activeMasterManager]
> hbase.MetaTableAccessor(1767): Updated table hbase:meta state to ENABLED in
> META
> 2017-02-14 20:23:20,804 DEBUG [PostOpenDeployTasks:534574363]
> regionserver.HRegionServer(2034): Finished post open deploy task for
> hbase:meta,,1_0001.534574363
> 2017-02-14 20:23:20,811 DEBUG [RS_OPEN_META-98410feec74d:50745-0]
> handler.OpenRegionHandler(126): Opened hbase:meta,,1_0001.534574363 on
> 98410feec74d,50745,1487103793930
> {code}
> Looks like some replica of the hbase:meta table might not have finished
> opening by the time the test asserted region location not being null.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17657) TestZKAsyncRegistry is flaky
[
https://issues.apache.org/jira/browse/HBASE-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17657:
---
Attachment: 17657.v1.txt
> TestZKAsyncRegistry is flaky
> -
>
> Key: HBASE-17657
> URL: https://issues.apache.org/jira/browse/HBASE-17657
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17657.v1.txt
>
>
> TestZKAsyncRegistry showed up in failed tests several times.
> On
> https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
> , TestZKAsyncRegistry is reported flaky 33% of the time.
> e.g.
> https://builds.apache.org/job/PreCommit-HBASE-Build/5708/testReport/org.apache.hadoop.hbase.client/TestZKAsyncRegistry/test/
> Toward the end of test output:
> {code}
> 2017-02-14 20:23:20,779 WARN [main-EventThread] client.ZKAsyncRegistry(198):
> Meta region for replica 2 is in state PENDING_OPEN
> 2017-02-14 20:23:20,800 INFO [98410feec74d:45445.activeMasterManager]
> hbase.MetaTableAccessor(1767): Updated table hbase:meta state to ENABLED in
> META
> 2017-02-14 20:23:20,804 DEBUG [PostOpenDeployTasks:534574363]
> regionserver.HRegionServer(2034): Finished post open deploy task for
> hbase:meta,,1_0001.534574363
> 2017-02-14 20:23:20,811 DEBUG [RS_OPEN_META-98410feec74d:50745-0]
> handler.OpenRegionHandler(126): Opened hbase:meta,,1_0001.534574363 on
> 98410feec74d,50745,1487103793930
> {code}
> Looks like some replica of the hbase:meta table might not have finished
> opening by the time the test asserted region location not being null.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17657) TestZKAsyncRegistry is flaky
[
https://issues.apache.org/jira/browse/HBASE-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870634#comment-15870634
]
Ted Yu commented on HBASE-17657:
[~Apache9]:
Can you take a look at the patch ?
> TestZKAsyncRegistry is flaky
> -
>
> Key: HBASE-17657
> URL: https://issues.apache.org/jira/browse/HBASE-17657
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17657.v1.txt
>
>
> TestZKAsyncRegistry showed up in failed tests several times.
> On
> https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
> , TestZKAsyncRegistry is reported flaky 33% of the time.
> e.g.
> https://builds.apache.org/job/PreCommit-HBASE-Build/5708/testReport/org.apache.hadoop.hbase.client/TestZKAsyncRegistry/test/
> Toward the end of test output:
> {code}
> 2017-02-14 20:23:20,779 WARN [main-EventThread] client.ZKAsyncRegistry(198):
> Meta region for replica 2 is in state PENDING_OPEN
> 2017-02-14 20:23:20,800 INFO [98410feec74d:45445.activeMasterManager]
> hbase.MetaTableAccessor(1767): Updated table hbase:meta state to ENABLED in
> META
> 2017-02-14 20:23:20,804 DEBUG [PostOpenDeployTasks:534574363]
> regionserver.HRegionServer(2034): Finished post open deploy task for
> hbase:meta,,1_0001.534574363
> 2017-02-14 20:23:20,811 DEBUG [RS_OPEN_META-98410feec74d:50745-0]
> handler.OpenRegionHandler(126): Opened hbase:meta,,1_0001.534574363 on
> 98410feec74d,50745,1487103793930
> {code}
> Looks like some replica of the hbase:meta table might not have finished
> opening by the time the test asserted region location not being null.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17656) Move new Address class from util to net package
[ https://issues.apache.org/jira/browse/HBASE-17656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17656: -- Affects Version/s: 2.0.0 Status: Patch Available (was: Open) > Move new Address class from util to net package > --- > > Key: HBASE-17656 > URL: https://issues.apache.org/jira/browse/HBASE-17656 > Project: HBase > Issue Type: Sub-task > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17656.master.001.patch > > > I added the new Address type in the wrong package committing the parent JIRA > patch. It should be in net, not util. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17656) Move new Address class from util to net package
[ https://issues.apache.org/jira/browse/HBASE-17656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17656: -- Attachment: HBASE-17656.master.001.patch > Move new Address class from util to net package > --- > > Key: HBASE-17656 > URL: https://issues.apache.org/jira/browse/HBASE-17656 > Project: HBase > Issue Type: Sub-task > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17656.master.001.patch > > > I added the new Address type in the wrong package committing the parent JIRA > patch. It should be in net, not util. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17657) TestZKAsyncRegistry is flaky
Ted Yu created HBASE-17657:
--
Summary: TestZKAsyncRegistry is flaky
Key: HBASE-17657
URL: https://issues.apache.org/jira/browse/HBASE-17657
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
TestZKAsyncRegistry showed up in failed tests several times.
On
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
, TestZKAsyncRegistry is reported flaky 33% of the time.
e.g.
https://builds.apache.org/job/PreCommit-HBASE-Build/5708/testReport/org.apache.hadoop.hbase.client/TestZKAsyncRegistry/test/
Toward the end of test output:
{code}
2017-02-14 20:23:20,779 WARN [main-EventThread] client.ZKAsyncRegistry(198):
Meta region for replica 2 is in state PENDING_OPEN
2017-02-14 20:23:20,800 INFO [98410feec74d:45445.activeMasterManager]
hbase.MetaTableAccessor(1767): Updated table hbase:meta state to ENABLED in META
2017-02-14 20:23:20,804 DEBUG [PostOpenDeployTasks:534574363]
regionserver.HRegionServer(2034): Finished post open deploy task for
hbase:meta,,1_0001.534574363
2017-02-14 20:23:20,811 DEBUG [RS_OPEN_META-98410feec74d:50745-0]
handler.OpenRegionHandler(126): Opened hbase:meta,,1_0001.534574363 on
98410feec74d,50745,1487103793930
{code}
Looks like some replica of the hbase:meta table might not have finished opening
by the time the test asserted region location not being null.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17656) Move new Address class from util to net package
stack created HBASE-17656: - Summary: Move new Address class from util to net package Key: HBASE-17656 URL: https://issues.apache.org/jira/browse/HBASE-17656 Project: HBase Issue Type: Sub-task Components: rsgroup Reporter: stack Assignee: stack Priority: Minor I added the new Address type in the wrong package committing the parent JIRA patch. It should be in net, not util. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17655) Removing MemStoreScanner and SnapshotScanner
[ https://issues.apache.org/jira/browse/HBASE-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-17655: -- Description: With CompactingMemstore becoming the new default, a store comprises multiple memory segments and not just 1-2. MemStoreScanner encapsulates the scanning of segments in the memory part of the store. SnapshotScanner is used to scan the snapshot segment upon flush to disk. Having the logic of scanners scattered in multiple classes (StoreScanner, SegmentScanner, MemStoreScanner, SnapshotScanner) makes maintainance and debugging challenging tasks, not always for a good reason. For example, MemStoreScanner has a KeyValueHeap (KVH). When creating the store scanner which also has a KVH, this makes a KVH inside a KVH. Reasoning about the correctness of the methods supported by the scanner (seek, next, hasNext, peek, etc.) is hard and debugging them is cumbersome. In addition, by removing the MemStoreScanner layer we allow store scanner to filter out each one of the memory scanners instead of either taking them all (in most cases) or discarding them all (rarely). SnapshotScanner is a simplified version of SegmentScanner as it is used only in a specific context. However it is an additional implementation of the same logic with no real advantage of improved performance. Therefore, I suggest removing both MemStoreScanner and SnapshotScanner. The code is adjusted to handle the list of segment scanners they encapsulate. This fits well with the current code since in most cases at some point a list of scanner is expected, so passing the actual list of segment scanners is more natural than wrapping a single (high level) scanner with Collections.singeltonList(...). was: With CompactingMemstore becoming the new default, a store comprises multiple memory segments and not just 1-2. MemStoreScanner encapsulates the scanning of all the segments in the memory part of the store. SnapshotScanner is used to scan the snapshot segment upon flush to disk. Having the logic of scanners scattered in multiple classes (StoreScanner, SegmentScanner, MemStoreScanner, SnapshotScanner) makes maintainance and debugging challenging tasks, not always for a good reason. For example, MemStoreScanner has a KeyValueHeap (KVH). When creating the store scanner which also has a KVH, this makes a KVH in a KVH. Reasoning about the correctness of the methods supported by the scanner (seek, meet, hasNext, peek, etc.) is hard and debugging them is cumbersome. In addition, by removing the MemStoreScanner layer we allow the store scanner to filter in/out each one of the memory scanners instead of either taking them all (in most cases) or discarding them all (rarely). SnapshotScanner is a simplified version of SegmentScanner as it is used only in a specific context. However it is an additional implementation of the same logic with no real advantage of improved performance. Therefore, I suggest removing both MemStoreScanner and SnapshotScanner. The code is adjusted to handle the list of segment scanners they encapsulate. This is fits well with the current code since in most cases at some point a list of scanner is expected and the current scanners are wrapped with Collections.singeltonList(...). > Removing MemStoreScanner and SnapshotScanner > > > Key: HBASE-17655 > URL: https://issues.apache.org/jira/browse/HBASE-17655 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17655-V01.patch > > > With CompactingMemstore becoming the new default, a store comprises multiple > memory segments and not just 1-2. MemStoreScanner encapsulates the scanning > of segments in the memory part of the store. SnapshotScanner is used to scan > the snapshot segment upon flush to disk. > Having the logic of scanners scattered in multiple classes (StoreScanner, > SegmentScanner, MemStoreScanner, SnapshotScanner) makes maintainance and > debugging challenging tasks, not always for a good reason. > For example, MemStoreScanner has a KeyValueHeap (KVH). When creating the > store scanner which also has a KVH, this makes a KVH inside a KVH. Reasoning > about the correctness of the methods supported by the scanner (seek, next, > hasNext, peek, etc.) is hard and debugging them is cumbersome. > In addition, by removing the MemStoreScanner layer we allow store scanner to > filter out each one of the memory scanners instead of either taking them all > (in most cases) or discarding them all (rarely). > SnapshotScanner is a simplified version of SegmentScanner as it is used only > in a specific context. However it is an additional implementation of the same > logic with no real
[jira] [Commented] (HBASE-17649) REST API for scan should return 410 when table is disabled
[ https://issues.apache.org/jira/browse/HBASE-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870566#comment-15870566 ] Hudson commented on HBASE-17649: FAILURE: Integrated in Jenkins build HBase-1.2-IT #598 (See [https://builds.apache.org/job/HBase-1.2-IT/598/]) HBASE-17649 REST API for scan should return 410 when table is disabled (tedyu: rev 81c0f24b9439d3393c7748b62e97e4117c4997d7) * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java * (edit) hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestScannerResource.java > REST API for scan should return 410 when table is disabled > -- > > Key: HBASE-17649 > URL: https://issues.apache.org/jira/browse/HBASE-17649 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5 > > Attachments: 17649.branch-1.3.v1.txt, 17649.v1.txt > > > Background: > When scanning using REST API, if the table is disabled in the middle of the > scan, code 410 (Gone) should be returned. > Currently in 1.3 and lower releases, ScannerResultGenerator#next() doesn't > handle TableNotEnabledException, leading to the exception being ignored and > code 204 is returned to caller. > This was first spotted by [~Apache9] when investigating HBASE-17603. > Patch for HBASE-17603 contains fix for this problem. > This issue fixes the bug for 1.3 and lower branches and adds test for > branch-1 and master branch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17655) Removing MemStoreScanner and SnapshotScanner
[ https://issues.apache.org/jira/browse/HBASE-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-17655: -- Status: Patch Available (was: Open) > Removing MemStoreScanner and SnapshotScanner > > > Key: HBASE-17655 > URL: https://issues.apache.org/jira/browse/HBASE-17655 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17655-V01.patch > > > With CompactingMemstore becoming the new default, a store comprises multiple > memory segments and not just 1-2. MemStoreScanner encapsulates the scanning > of all the segments in the memory part of the store. SnapshotScanner is used > to scan the snapshot segment upon flush to disk. > Having the logic of scanners scattered in multiple classes (StoreScanner, > SegmentScanner, MemStoreScanner, SnapshotScanner) makes maintainance and > debugging challenging tasks, not always for a good reason. > For example, MemStoreScanner has a KeyValueHeap (KVH). When creating the > store scanner which also has a KVH, this makes a KVH in a KVH. Reasoning > about the correctness of the methods supported by the scanner (seek, meet, > hasNext, peek, etc.) is hard and debugging them is cumbersome. > In addition, by removing the MemStoreScanner layer we allow the store scanner > to filter in/out each one of the memory scanners instead of either taking > them all (in most cases) or discarding them all (rarely). > SnapshotScanner is a simplified version of SegmentScanner as it is used only > in a specific context. However it is an additional implementation of the same > logic with no real advantage of improved performance. > Therefore, I suggest removing both MemStoreScanner and SnapshotScanner. The > code is adjusted to handle the list of segment scanners they encapsulate. > This is fits well with the current code since in most cases at some point a > list of scanner is expected and the current scanners are wrapped with > Collections.singeltonList(...). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17655) Removing MemStoreScanner and SnapshotScanner
[ https://issues.apache.org/jira/browse/HBASE-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-17655: -- Attachment: HBASE-17655-V01.patch > Removing MemStoreScanner and SnapshotScanner > > > Key: HBASE-17655 > URL: https://issues.apache.org/jira/browse/HBASE-17655 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17655-V01.patch > > > With CompactingMemstore becoming the new default, a store comprises multiple > memory segments and not just 1-2. MemStoreScanner encapsulates the scanning > of all the segments in the memory part of the store. SnapshotScanner is used > to scan the snapshot segment upon flush to disk. > Having the logic of scanners scattered in multiple classes (StoreScanner, > SegmentScanner, MemStoreScanner, SnapshotScanner) makes maintainance and > debugging challenging tasks, not always for a good reason. > For example, MemStoreScanner has a KeyValueHeap (KVH). When creating the > store scanner which also has a KVH, this makes a KVH in a KVH. Reasoning > about the correctness of the methods supported by the scanner (seek, meet, > hasNext, peek, etc.) is hard and debugging them is cumbersome. > In addition, by removing the MemStoreScanner layer we allow the store scanner > to filter in/out each one of the memory scanners instead of either taking > them all (in most cases) or discarding them all (rarely). > SnapshotScanner is a simplified version of SegmentScanner as it is used only > in a specific context. However it is an additional implementation of the same > logic with no real advantage of improved performance. > Therefore, I suggest removing both MemStoreScanner and SnapshotScanner. The > code is adjusted to handle the list of segment scanners they encapsulate. > This is fits well with the current code since in most cases at some point a > list of scanner is expected and the current scanners are wrapped with > Collections.singeltonList(...). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17312) [JDK8] Use default method for Observer Coprocessors
[ https://issues.apache.org/jira/browse/HBASE-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870543#comment-15870543 ] Andrew Purtell commented on HBASE-17312: Phoenix has no support for master branch and no development for same currently planned. Also, no plans to move up to JDK8 even. > [JDK8] Use default method for Observer Coprocessors > --- > > Key: HBASE-17312 > URL: https://issues.apache.org/jira/browse/HBASE-17312 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17312.master.001.patch, > HBASE-17312.master.001.patch > > > Use default method in MasterObserver, RegionObserver, RegionServerObserver > and WALObserver. And mark the BaseRegionObserver, > BaseMasterAndRegionObserver, BaseRegionServerObserver and BaseWALObserver. > User can implement the interface directly and will not break compatibility > when add new default methods. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17025) [Shell] Support space quota get/set via the shell
[ https://issues.apache.org/jira/browse/HBASE-17025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870537#comment-15870537 ] Ted Yu commented on HBASE-17025: +1 > [Shell] Support space quota get/set via the shell > - > > Key: HBASE-17025 > URL: https://issues.apache.org/jira/browse/HBASE-17025 > Project: HBase > Issue Type: Sub-task > Components: shell >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17025.001.patch, HBASE-17025.002.patch, > HBASE-17025.003.HBASE-16961.patch, HBASE-17025.004.HBASE-16961.patch > > > Need to make sure that admins can use the shell to get/set the new space > quotas. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17648) HBase Table-level synchronization fails between two secured(kerberized) clusters
[
https://issues.apache.org/jira/browse/HBASE-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870524#comment-15870524
]
Ted Yu commented on HBASE-17648:
On
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
, TestZKAsyncRegistry is reported flaky 33% of the time.
> HBase Table-level synchronization fails between two secured(kerberized)
> clusters
>
>
> Key: HBASE-17648
> URL: https://issues.apache.org/jira/browse/HBASE-17648
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-17648-V1.patch, HBASE-17648-V2.patch
>
>
> run org.apache.hadoop.hbase.mapreduce.SyncTable’ command.
> Follow the steps in HBASE-15557 between two secured cluster, get errors below:
> {code}
> WARN [hconnection-0x55b5e331-metaLookup-shared--pool6-t1]
> org.apache.hadoop.hbase.ipc.RpcClientImpl: Exception encountered while
> connecting to the server : javax.security.sasl.SaslException: GSS initiate
> failed [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]
> 2017-02-08 14:33:57,465 FATAL
> [hconnection-0x55b5e331-metaLookup-shared--pool6-t1]
> org.apache.hadoop.hbase.ipc.RpcClientImpl: SASL authentication failed. The
> most likely cause is missing or invalid credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
> at
> org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
> at
> org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
>
[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870491#comment-15870491
]
Andrew Purtell commented on HBASE-17069:
Checking, [~Apache9] . It may take a couple of days before I have results.
Great work [~abhishek.chouhan]
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>Reporter: Andrew Purtell
>Assignee: Abhishek Singh Chouhan
>Priority: Blocker
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log,
> HBASE-17069.branch-1.3.001.patch, HBASE-17069.branch-1.3.002.patch,
> HBASE-17069.master.001.patch, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63,
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320,
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,717 INFO
[jira] [Commented] (HBASE-17649) REST API for scan should return 410 when table is disabled
[ https://issues.apache.org/jira/browse/HBASE-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870488#comment-15870488 ] Hudson commented on HBASE-17649: FAILURE: Integrated in Jenkins build HBase-1.3-IT #832 (See [https://builds.apache.org/job/HBase-1.3-IT/832/]) HBASE-17649 REST API for scan should return 410 when table is disabled (tedyu: rev 81c0f24b9439d3393c7748b62e97e4117c4997d7) * (edit) hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestScannerResource.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java > REST API for scan should return 410 when table is disabled > -- > > Key: HBASE-17649 > URL: https://issues.apache.org/jira/browse/HBASE-17649 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5 > > Attachments: 17649.branch-1.3.v1.txt, 17649.v1.txt > > > Background: > When scanning using REST API, if the table is disabled in the middle of the > scan, code 410 (Gone) should be returned. > Currently in 1.3 and lower releases, ScannerResultGenerator#next() doesn't > handle TableNotEnabledException, leading to the exception being ignored and > code 204 is returned to caller. > This was first spotted by [~Apache9] when investigating HBASE-17603. > Patch for HBASE-17603 contains fix for this problem. > This issue fixes the bug for 1.3 and lower branches and adds test for > branch-1 and master branch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17653) HBASE-17624 rsgroup synchronizations will (distributed) deadlock
[
https://issues.apache.org/jira/browse/HBASE-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870481#comment-15870481
]
Francis Liu commented on HBASE-17653:
-
Thanks [~stack]. Here are my responses:
{quote}
Throughout this class we're inconsistent with locking on the object returned
from this call. From reading both this class' use, the interface
RSGroupInfoManager, and the default implementation it's not clear to me what
correct synchronized handling is supposed to look like.
{quote}
The strategy is the same as RSgroupInfoManager. Reads don't lock while
mutations do.
{quote}
However, we synchronize for removing groups but not adding them and sometimes
when retrieving them, so my intuition is that something is incorrect.
{quote}
This is just cosmetic. The method being called acquires the same lock. You can
wrap it in the same synchronized(manager) block in the class the effect is the
same. Or did I miss something?
{quote}
if it's rsGroupMap, we still have methods that make use of the map without a
proper synchronization call.
{quote}
Same reason reads should not be blocked by mutations.
{quote}
reading through things, I think it's trying to keep updates to the reference
for rsGroupMap atomic (since I *think* all the things written there are
unmodifiable, which would make the unsynchronized access safe). It would be
much clearer if we did that via an AtomicReference or atleast documented that
this is what our intention is for all these synchronized methods.
{quote}
Yep correct making all updates to the map atomic so invariants aren't broken
(ie tables or servers don't popup in more than one group). Not sure why it'd be
much clearer with AtomicReference, volatile should suffice IMHO. Yep document
should be enough for this?
{quote}
Then, in rb Duo Zhang questioned synchronization here
https://reviews.apache.org/r/56570/diff/1?file=1630642#file1630642line221
RSGroupInfoManagerImpl is used by RSGroupAdminServer. It holds the monitor for
RSGroupInfoManagerImpl across a set of operations making the reviewer think
this method does not need synchronization.
{quote}
Seems to me he is making a comment on your addition of the synchronized method
not the code prior to this patch?
All in all the changes seem to be minor? Add explicit synchronized blocks in
the places that does mutations to make it more readable and document the intent
of why not synchronizing for reads? BTW you guys could've just asked me instead
of trying to read my mind ;-).
> HBASE-17624 rsgroup synchronizations will (distributed) deadlock
>
>
> Key: HBASE-17653
> URL: https://issues.apache.org/jira/browse/HBASE-17653
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Reporter: stack
>Assignee: stack
>
> Follow-on from HBASE-17624. HBASE-17624 made it so one thread only has access
> to the rsgroup administrator. In tail of HBASE-17624 [~toffer] describes
> scenario under which we may end up in a deadlock (distributed). Let me
> repeat [~toffer] comment...
> {code}
> Both read/write access can't be single threaded. Consider the situation:
> 1. move_rsgroup_servers is called
> 2. while #1 is happening rsgroup region is in transition (rpc thread in #1
> holds monitor lock)
> 3. while #2 is happening meta is in transition.
> Balancer tries to figure out plan for meta region tries to get monitor lock
> but can't. rpc thread task won't release monitor lock since rsgroup region
> never gets assigned. rsgroup region never gets assigned because it can't
> update meta with new state.
> There's a good chance this can be reproduce just by moving both rsgroup and
> meta region onto the same RS and call move_rsgoup_servers on the same RS.
> A bunch different actors will query from group affiliation so we can't have
> writes block reads.
>
> In the code prior to this patch the getter methods that retrieve group
> information (getRSGroup, ofTable, OfServer, etc) don't require the monitor
> lock so the deadlock cycle is broken.
>
> The methods that does mutations and updates to zk and hbase:rsgroup are
> synchronized appropriately. Point me to where the incoherence is?
> {code}
> This issue is about testing/fixing/restoring rsgroup access. Will be back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17648) HBase Table-level synchronization fails between two secured(kerberized) clusters
[
https://issues.apache.org/jira/browse/HBASE-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17648:
-
Status: Patch Available (was: Open)
> HBase Table-level synchronization fails between two secured(kerberized)
> clusters
>
>
> Key: HBASE-17648
> URL: https://issues.apache.org/jira/browse/HBASE-17648
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-17648-V1.patch, HBASE-17648-V2.patch
>
>
> run org.apache.hadoop.hbase.mapreduce.SyncTable’ command.
> Follow the steps in HBASE-15557 between two secured cluster, get errors below:
> {code}
> WARN [hconnection-0x55b5e331-metaLookup-shared--pool6-t1]
> org.apache.hadoop.hbase.ipc.RpcClientImpl: Exception encountered while
> connecting to the server : javax.security.sasl.SaslException: GSS initiate
> failed [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]
> 2017-02-08 14:33:57,465 FATAL
> [hconnection-0x55b5e331-metaLookup-shared--pool6-t1]
> org.apache.hadoop.hbase.ipc.RpcClientImpl: SASL authentication failed. The
> most likely cause is missing or invalid credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1697)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
> at
> org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
> at
> org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at
>
